2,652
edits
m (→The script: set LC_ALL) |
(→The script: KD 27.4.22 always print out detector serial number) |
||
(15 intermediate revisions by the same user not shown) | |||
Line 120: | Line 120: | ||
# revision 1.01 . KD 07/12/2020 ROTATION_AXIS=0 -1 0 for Pilatus3 2M, S/N 24-0118 at ID23-2 (http://www.globalphasing.com/autoproc/wiki/index.cgi?BeamlineSettings) | # revision 1.01 . KD 07/12/2020 ROTATION_AXIS=0 -1 0 for Pilatus3 2M, S/N 24-0118 at ID23-2 (http://www.globalphasing.com/autoproc/wiki/index.cgi?BeamlineSettings) | ||
# revision 1.02 . KD 11/01/2021 recognize mar555 detector as mar345 (thanks to Thomas Hauß, HZB) | # revision 1.02 . KD 11/01/2021 recognize mar555 detector as mar345 (thanks to Thomas Hauß, HZB) | ||
# revision 1.03 . Feng | # revision 1.03 . Feng Yu 18/7/2021 fix/expand information about Shanghai Synchrotron Radiation Facility (SSRF) | ||
# revision 1.04 . Zhipu Luo 13/08/2021 extract ORGX,ORGY for electron diffraction from SMV header if the wavelength value starts with 0.0 | # revision 1.04 . Zhipu Luo 13/08/2021 extract ORGX,ORGY for electron diffraction from SMV header if the wavelength value starts with 0.0 | ||
# revision 1.05 . KD specialcase extraction of number of images for BNL detectors E-32-0101 and E-18-0104 | # revision 1.05 . KD specialcase extraction of number of images for BNL detectors E-32-0101 and E-18-0104 | ||
# revision 1.06 . KD fix URLs in output. | # revision 1.06 . KD fix URLs in output. Availability of Apple M1 processor dectris-neggia-Apple-arm64.so . LC_ALL=C . | ||
REVISION="1. | # revision 1.07 . KD, Thomas Hauß, Gleb Bourenkov 25/10/2021. Detector moved from Petra P14 to P13 | ||
# revision 1.08 . KD, Thomas Hauß, Feng Yu 7/11/2021 fix TZ in timestamps for SSRF detectors | |||
# revision 1.09 . KD, Feng Yu 7/11/2021 fix detector number BNL E-18-0121 to be E-18-0104 | |||
# revision 1.10 . KD NeXus header for Eiger | |||
# revision 1.11 . KD / Helena Taberman ROTATION_AXIS=-1 0 0 for Eiger .cbf data from Petra P14 | |||
# revision 1.12 . KD / Helena Taberman correct rev 1.11 to be 0 -1 0 | |||
REVISION="1.12 (25-MAR-2022)" | |||
# | # | ||
Line 132: | Line 138: | ||
# | # | ||
# known problems: | # known problems: | ||
# revision 1.10 . KD implement NeXus for Eiger | |||
# - for ADSC detectors, there are at least three ways to obtain ORGX and ORGY values from the header (see below); | # - for ADSC detectors, there are at least three ways to obtain ORGX and ORGY values from the header (see below); | ||
# - the same might be a problem for MAR headers, too (not sure about this) | # - the same might be a problem for MAR headers, too (not sure about this) | ||
Line 137: | Line 144: | ||
# | # | ||
# notes for debugging of the script: | # notes for debugging of the script: | ||
# - add the -x option to | # - add the -x option to #!/bin/bash in first line, to see where an error occurs | ||
# - comment out the removal of tmp1 and tmp2 in the last line | # - comment out the removal of tmp1 and tmp2 in the last line | ||
# | # | ||
Line 173: | Line 180: | ||
# default DIRECTION_OF_DETECTOR_X-AXIS | # default DIRECTION_OF_DETECTOR_X-AXIS | ||
DIRECTION_OF_DETECTOR_X_AXIS="1 0 0" | DIRECTION_OF_DETECTOR_X_AXIS="1 0 0" | ||
# default DIRECTION_OF_DETECTOR_Y-AXIS | |||
DIRECTION_OF_DETECTOR_Y_AXIS="0 1 0" | |||
# default FRACTION_OF_POLARIZATION | # default FRACTION_OF_POLARIZATION | ||
pol_frac=0.98 | pol_frac=0.98 | ||
Line 469: | Line 478: | ||
DETECTOR_DISTANCE=`grep ^DISTANCE tmp2 | sed s/DISTANCE=/-/` | DETECTOR_DISTANCE=`grep ^DISTANCE tmp2 | sed s/DISTANCE=/-/` | ||
OSCILLATION_RANGE=`grep OMEGA_DELTA tmp2 | sed s/OMEGA_DELTA=//` | OSCILLATION_RANGE=`grep OMEGA_DELTA tmp2 | sed s/OMEGA_DELTA=//` | ||
DETECTOR_X_AXIS="-1 0 0" | |||
elif [ "$DET" == "experimental-ED" ]; then | elif [ "$DET" == "experimental-ED" ]; then | ||
Line 514: | Line 523: | ||
# find SENSOR_THICKNESS: | # find SENSOR_THICKNESS: | ||
SENSOR_THICKNESS=`grep thickness tmp2 | sed -e s/' | SENSOR_THICKNESS=`grep thickness tmp2 | sed -e s/'Si.* sensor, thickness'// | awk '{print $1*1000}'` | ||
# find X_RAY_WAVELENGTH: | # find X_RAY_WAVELENGTH: | ||
X_RAY_WAVELENGTH=`grep Wavelength tmp2 | sed -e s/Wavelength// -e s/A// | awk '{print $1}'` | X_RAY_WAVELENGTH=`grep Wavelength tmp2 | sed -e s/Wavelength// -e s/A// | awk '{print $1}'` | ||
Line 566: | Line 575: | ||
fi | fi | ||
# PETRA P14: raw data from Eiger are stored as CBF files so this is treated as Pilatus | # PETRA P14: raw data from Eiger are stored as CBF files so this is treated as Pilatus | ||
if [ "$DET_SN" == "Dectris Eiger | if [ "$DET_SN" == "Dectris Eiger 4M, E-08-0107" -o "$DET_SN" == "PILATUS 6M-F, S/N 60-0115-F" \ | ||
-o "$DET_SN" == "Dectris EIGER2 CdTe 16M, E-32-0129" ] ; then | |||
rotation_axis="0 -1 0" | rotation_axis="0 -1 0" | ||
echo ROTATION_AXIS="0 -1 0" at PETRA P14 | echo ROTATION_AXIS="0 -1 0" at PETRA P14 | ||
fi | |||
# similarly, Eiger 16M at PETRA P14 until May 22, 2021 | |||
if [ "$DET_SN" == "Dectris Eiger 16M, E-32-0107" ] ; then | |||
egrep -q '^# 2020-|^# 2021-0[1-5]|^# 2021-06-[01]|^# 2021-06-2[01]' tmp2 || isatP13=1 | |||
if [ "$isatP13" == 1 ] ; then | |||
echo using the default ROTATION_AXIS=1 0 0 at PETRA P13 | |||
else | |||
rotation_axis="0 -1 0" | |||
echo ROTATION_AXIS="0 -1 0" at PETRA P14 | |||
fi | |||
fi | fi | ||
# ESRF ID23-2: | # ESRF ID23-2: | ||
Line 580: | Line 600: | ||
DATA_RANGE="1 $nframes" | DATA_RANGE="1 $nframes" | ||
SPOT_RANGE="1 `echo "scale=0; if (${nframes}<2) 1; if (${nframes}>1) ${nframes}/2"|bc -l`" | SPOT_RANGE="1 `echo "scale=0; if (${nframes}<2) 1; if (${nframes}>1) ${nframes}/2"|bc -l`" | ||
SN=`h5dump -d "/entry/instrument/detector/detector_number" $FIRSTFRAME | awk '/\(0\): /{print $2}' | sed s/\"//g` | |||
echo detector serial number is $SN | |||
# find out if HDF5 from Diamond (DLS=1) or Dectris (DLS=0) | # find out if HDF5 from Diamond (DLS=1) or Dectris (DLS=0) | ||
Line 594: | Line 616: | ||
# rotation_axis=`h5dump -a "/entry/sample/transformations/omega/vector" $FIRSTFRAME 2>/dev/null | grep "(0):" | sed -e "s/^.*://; s/,//g"` | # rotation_axis=`h5dump -a "/entry/sample/transformations/omega/vector" $FIRSTFRAME 2>/dev/null | grep "(0):" | sed -e "s/^.*://; s/,//g"` | ||
# the above gives -1 0 0 for DLS data instead of the correct 1 0 0, so commented out for now | # the above gives -1 0 0 for DLS data instead of the correct 1 0 0, so commented out for now | ||
# comment while implementing rev 1.10: this is because the DLS headers are NeXus, so DIRECTION_OF_DETECTOR_X/Y_AXIS must be adjusted. | |||
else | else | ||
echo Eiger HDF5 from Dectris | echo Eiger HDF5 from Dectris | ||
Line 606: | Line 629: | ||
fi | fi | ||
echo "STARTING_ANGLE=$STARTING_ANGLE" | echo "STARTING_ANGLE=$STARTING_ANGLE" | ||
# If rotation | # If rotation axis set (NeXus) | ||
rotation_axis=`h5dump -a "/entry/sample/transformations/omega/vector" $FIRSTFRAME 2>/dev/null | grep "(0):" | sed -e "s/^.*://; s/,//g"` | rotation_axis=`h5dump -a "/entry/sample/transformations/omega/vector" $FIRSTFRAME 2>/dev/null | grep "(0):" | sed -e "s/^.*://; s/,//g"` | ||
# EIGER2 16M CHESS ID7B2 has S/N E-32-0123 (A. Finke 2020-11-07) v0.99 | # EIGER2 16M CHESS ID7B2 has S/N E-32-0123 (A. Finke 2020-11-07) v0.99 | ||
if [ "$SN" == "E-32-0123" ]; then | if [ "$SN" == "E-32-0123" ]; then | ||
rotation_axis="-1 0 0" | rotation_axis="-1 0 0" | ||
Line 616: | Line 638: | ||
# SSRF BL17U1 and SSRF BL10U2 (Feng YU 2021-07-18) | # SSRF BL17U1 and SSRF BL10U2 (Feng YU 2021-07-18) | ||
# Eiger X 16M (S/N E-32-0111) was installed at SSRF BL17U1 from Oct 2017 to Feb 2021. After Feb 2021, it was moved back to SSRF BL10U2. | # Eiger X 16M (S/N E-32-0111) was installed at SSRF BL17U1 from Oct 2017 to Feb 2021. After Feb 2021, it was moved back to SSRF BL10U2. | ||
# 2021- | # 2021-03-01 00:00:00 time stamp is 1614528000 | ||
if [ "$SN" == "E-32-0111" ]; then | if [ "$SN" == "E-32-0111" ]; then | ||
collection_time=`h5dump -d "/entry/instrument/detector/detectorSpecific/data_collection_date" $FIRSTFRAME | grep "(0):" | awk '{print $2}' | sed -e "s/\.\(.*\)/ | collection_time=`h5dump -d "/entry/instrument/detector/detectorSpecific/data_collection_date" $FIRSTFRAME | grep "(0):" | awk '{print $2}' | sed -e "s/\.\(.*\)//g; s/\"//g"` | ||
if [ `uname -s` == "Darwin" ]; then | if [ `uname -s` == "Darwin" ]; then | ||
collection_timestamp=`date -j -f "%Y-%m-%dT%H:%M:%S | collection_timestamp=`date -j -f "%Y-%m-%dT%H:%M:%S" $collection_time +%s` | ||
elif [ `uname -s` == "Linux" ]; then | elif [ `uname -s` == "Linux" ]; then | ||
collection_timestamp=`date -d $collection_time +%s` | collection_timestamp=`date -d $collection_time +%s` | ||
Line 635: | Line 657: | ||
rotation_axis="-1 0 0 ! Cannot determine rotation axis. SSRF BL17U1: -1 0 0; SSRF BL02U1: 0 -1 0" | rotation_axis="-1 0 0 ! Cannot determine rotation axis. SSRF BL17U1: -1 0 0; SSRF BL02U1: 0 -1 0" | ||
fi | fi | ||
elif [ $collection_timestamp -ge 0 ] && [ $collection_timestamp -le | elif [ $collection_timestamp -ge 0 ] && [ $collection_timestamp -le 1614528000 ]; then | ||
# SSRF BL17U1 | # SSRF BL17U1 | ||
rotation_axis="-1 0 0" | rotation_axis="-1 0 0" | ||
Line 666: | Line 688: | ||
fi | fi | ||
# revision 1.05 specialcase nframes for Eiger detectors at BNL | # revision 1.05 specialcase nframes for Eiger detectors at BNL | ||
if [ "$SN" == "E-18- | if [ "$SN" == "E-18-0104" -o "$SN" == "E-32-0101" ]; then | ||
echo specialcase nframes for Eiger detectors at BNL: | echo specialcase nframes for Eiger detectors at BNL: | ||
nframes=`h5dump -A -g "/entry/data" $FIRSTFRAME | grep "DATASPACE SIMPLE" | sed -e "s/,.*//" | awk '{a+=$5}END{print a}'` | nframes=`h5dump -A -g "/entry/data" $FIRSTFRAME | grep "DATASPACE SIMPLE" | sed -e "s/,.*//" | awk '{a+=$5}END{print a}'` | ||
Line 672: | Line 694: | ||
SPOT_RANGE="1 `echo "scale=0; if (${nframes}<2) 1; if (${nframes}>1) ${nframes}/2"|bc -l`" | SPOT_RANGE="1 `echo "scale=0; if (${nframes}<2) 1; if (${nframes}>1) ${nframes}/2"|bc -l`" | ||
fi | fi | ||
fi | |||
# rev 1.10: check for NeXus header. If found, its geometry will overwrite any rotation_axis set until here. | |||
NeXus=0 | |||
h5dump -d "/entry/definition" $FIRSTFRAME | grep -q NXmx && NeXus=1 | |||
if [ "$NeXus" == "1" ]; then | |||
echo NeXus header found. This defines DIRECTION_OF_DETECTOR_X/Y-AXIS and ROTATION_AXIS. | |||
DIRECTION_OF_DETECTOR_X_AXIS=$(h5dump -d "/entry/instrument/detector/module/fast_pixel_direction" $FIRSTFRAME 2>/dev/null | grep "(0):" | tail -1 | sed -e "s/^.*://; s/,//g") | |||
DIRECTION_OF_DETECTOR_Y_AXIS=$(h5dump -d "/entry/instrument/detector/module/slow_pixel_direction" $FIRSTFRAME 2>/dev/null | grep "(0):" | tail -1 | sed -e "s/^.*://; s/,//g") | |||
rotation_axis=$(h5dump -a "/entry/sample/transformations/omega/vector" $FIRSTFRAME 2>/dev/null | grep "(0):" | sed -e "s/^.*://; s/,//g") | |||
else | |||
echo no NeXus header found. | |||
fi | fi | ||
echo DATA_RANGE=$DATA_RANGE | echo DATA_RANGE=$DATA_RANGE | ||
Line 701: | Line 734: | ||
#let SKIP=768 | #let SKIP=768 | ||
#NX=$(od -t x -j $SKIP -N 4 $FIRSTFRAME |awk 'NR==1{print toupper($2)}'|perl -nle '@array= $_ =~/.{2}/g; print "ibase=16;obase=A;".join("",reverse @array)'|bc) | #NX=$(od -t x -j $SKIP -N 4 $FIRSTFRAME |awk 'NR==1{print toupper($2)}'|perl -nle '@array= $_ =~/.{2}/g; print "ibase=16;obase=A;".join("",reverse @array)'|bc) | ||
NX=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(768);print("%.4d"%struct.unpack(">i",f.read(4)))') | NX=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(768);print("%.4d"%struct.unpack(">i",f.read(4)))') | ||
NY=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(772);print("%.4d"%struct.unpack(">i",f.read(4)))') | NY=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(772);print("%.4d"%struct.unpack(">i",f.read(4)))') | ||
DETECTOR_DISTANCE=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(344);print("-%.4f"%struct.unpack(">f",f.read(4)))') | DETECTOR_DISTANCE=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(344);print("-%.4f"%struct.unpack(">f",f.read(4)))') | ||
ORGX=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(540);print("%.4f"%struct.unpack(">f",f.read(4)))') | ORGX=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(540);print("%.4f"%struct.unpack(">f",f.read(4)))') | ||
ORGY=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(544);print("%.4f"%struct.unpack(">f",f.read(4)))') | ORGY=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(544);print("%.4f"%struct.unpack(">f",f.read(4)))') | ||
OSCILLATION_RANGE=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(524);phis,phie=struct.unpack(">ff",f.read(8));print("%.4f"%(phie-phis))') | OSCILLATION_RANGE=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(524);phis,phie=struct.unpack(">ff",f.read(8));print("%.4f"%(phie-phis))') | ||
QX=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(776);print("%.6f"%struct.unpack(">f",f.read(4)))') | QX=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(776);print("%.6f"%struct.unpack(">f",f.read(4)))') | ||
QY=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(780);print("%.6f"%struct.unpack(">f",f.read(4)))') | QY=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(780);print("%.6f"%struct.unpack(">f",f.read(4)))') | ||
X_RAY_WAVELENGTH=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(292);print("%.6f"%struct.unpack(">f",f.read(4)))') | X_RAY_WAVELENGTH=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(292);print("%.6f"%struct.unpack(">f",f.read(4)))') | ||
Line 988: | Line 1,007: | ||
if [ -e /usr/local/lib64/dectris-neggia.so ]; then | if [ -e /usr/local/lib64/dectris-neggia.so ]; then | ||
echo LIB=/usr/local/lib64/dectris-neggia.so >> XDS.INP | echo LIB=/usr/local/lib64/dectris-neggia.so >> XDS.INP | ||
echo LIB= line was written to XDS.INP . For Apple processors you must modify the name. | echo LIB= line was written to XDS.INP . For Apple ARM64 processors, you must modify the name. | ||
else | else | ||
echo !LIB=/usr/local/lib64/dectris-neggia.so >> XDS.INP | echo !LIB=/usr/local/lib64/dectris-neggia.so >> XDS.INP | ||
Line 1,029: | Line 1,048: | ||
cat >> XDS.INP << eof | cat >> XDS.INP << eof | ||
DIRECTION_OF_DETECTOR_X-AXIS=$DIRECTION_OF_DETECTOR_X_AXIS | DIRECTION_OF_DETECTOR_X-AXIS=$DIRECTION_OF_DETECTOR_X_AXIS | ||
DIRECTION_OF_DETECTOR_Y-AXIS= | DIRECTION_OF_DETECTOR_Y-AXIS=$DIRECTION_OF_DETECTOR_Y_AXIS | ||
eof | eof | ||
fi | fi | ||
Line 1,199: | Line 1,218: | ||
echo spots in FRAME.cbf! | echo spots in FRAME.cbf! | ||
rm -f tmp1 tmp2 | rm -f tmp1 tmp2 | ||
</pre> | </pre> | ||
Line 1,282: | Line 1,300: | ||
* At some beamlines, the ROTATION_AXIS should be -1 0 0 ("backwards") instead of the usual 1 0 0 ("horizontal"), or even 0 1 0 ("vertical") like at one of the PETRA Hamburg BLs. The frame headers do not have this information, so the default chosen by [[generate_XDS.INP]] may be wrong and need manual correction. Pls also see the article [[Beamline notes]]. | * At some beamlines, the ROTATION_AXIS should be -1 0 0 ("backwards") instead of the usual 1 0 0 ("horizontal"), or even 0 1 0 ("vertical") like at one of the PETRA Hamburg BLs. The frame headers do not have this information, so the default chosen by [[generate_XDS.INP]] may be wrong and need manual correction. Pls also see the article [[Beamline notes]]. | ||
* Sometimes, the x- and y- values of the primary beam position recorded in the header should be used for ORGY and ORGX (i.e reversed) instead of as ORGX and ORGY. For ADSC, this has been implemented in the script for a number of beamlines. | * Sometimes, the x- and y- values of the primary beam position recorded in the header should be used for ORGY and ORGX (i.e reversed) instead of as ORGX and ORGY. For ADSC, this has been implemented in the script for a number of beamlines. | ||
* there are apparently several flavours of HDF5 files produced at Diamond Light Source. They differ e.g. in the naming of the header items. This means that items like NX, NY, DETECTOR_DISTANCE and number of images cannot be | * there are apparently several flavours of HDF5 files produced at Diamond Light Source. They differ e.g. in the naming of the header items. This means that items like NX, NY, DETECTOR_DISTANCE and number of images cannot always be determined by the [[generate_XDS.INP]] script. Example: The data at /dls/i04-1/data/2021/mx28114-9/processing/Lenye_Diamini/ThiL/ThiL found during the CCP4 2021 online Cape Town workshop. A workaround is to use e.g. xia2 pipeline=3dii to process these files, and - if needed - extract those items from its output files, e.g. from DEFAULT/NATIVE/SWEEP1/index/XDS.INP . | ||
== See also == | == See also == |