Generate XDS.INP: Difference between revisions

Generate XDS.INP (view source)

Revision as of 17:38, 27 April 2022

981 bytes added , 27 April 2022

→‎The script: KD 27.4.22 always print out detector serial number

Kay

Bureaucrats

2,652

edits

@@ Line 120: / Line 120: @@
 # revision 1.01 . KD 07/12/2020 ROTATION_AXIS=0 -1 0 for Pilatus3 2M, S/N 24-0118 at ID23-2 (http://www.globalphasing.com/autoproc/wiki/index.cgi?BeamlineSettings)
 # revision 1.02 . KD 11/01/2021 recognize mar555 detector as mar345 (thanks to Thomas Hauß, HZB)
-# revision 1.03 . Feng YU 18/7/2021 fix/expand information about Shanghai Synchrotron Radiation Facility (SSRF)
+# revision 1.03 . Feng Yu 18/7/2021 fix/expand information about Shanghai Synchrotron Radiation Facility (SSRF)
 # revision 1.04 . Zhipu Luo 13/08/2021 extract ORGX,ORGY for electron diffraction from SMV header if the wavelength value starts with 0.0
 # revision 1.05 . KD specialcase extraction of number of images for BNL detectors E-32-0101 and E-18-0104
-# revision 1.06 . KD fix URLs in output. Comment availability of Apple processor dectris-neggia-Apple-arm64.so . Set LC_ALL=C .
+# revision 1.06 . KD fix URLs in output. Availability of Apple M1 processor dectris-neggia-Apple-arm64.so . LC_ALL=C .
-REVISION="1.06 (15-OCT-2021)"
+# revision 1.07 . KD, Thomas Hauß, Gleb Bourenkov 25/10/2021. Detector moved from Petra P14 to P13
+# revision 1.08 . KD, Thomas Hauß, Feng Yu 7/11/2021 fix TZ in timestamps for SSRF detectors
+# revision 1.09 . KD, Feng Yu 7/11/2021 fix detector number BNL E-18-0121 to be E-18-0104
+# revision 1.10 . KD NeXus header for Eiger
+# revision 1.11 . KD / Helena Taberman ROTATION_AXIS=-1 0 0 for Eiger .cbf data from Petra P14
+# revision 1.12 . KD / Helena Taberman correct rev 1.11 to be 0 -1 0
+REVISION="1.12 (25-MAR-2022)"
 #
@@ Line 132: / Line 138: @@
 #
 # known problems:
+# revision 1.10 . KD implement NeXus for Eiger
 # - for ADSC detectors, there are at least three ways to obtain ORGX and ORGY values from the header (see below);
 # - the same might be a problem for MAR headers, too (not sure about this)
@@ Line 137: / Line 144: @@
 #
 # notes for debugging of the script:
-# - add the -x option to the first line, to see where an error occurs
+# - add the -x option to #!/bin/bash in first line, to see where an error occurs
 # - comment out the removal of tmp1 and tmp2 in the last line
 #
@@ Line 173: / Line 180: @@
 # default DIRECTION_OF_DETECTOR_X-AXIS
 DIRECTION_OF_DETECTOR_X_AXIS="1 0 0"
+# default DIRECTION_OF_DETECTOR_Y-AXIS
+DIRECTION_OF_DETECTOR_Y_AXIS="0 1 0"
 # default FRACTION_OF_POLARIZATION
 pol_frac=0.98
@@ Line 469: / Line 478: @@
    DETECTOR_DISTANCE=`grep ^DISTANCE tmp2 | sed s/DISTANCE=/-/`
    OSCILLATION_RANGE=`grep OMEGA_DELTA tmp2 | sed s/OMEGA_DELTA=//`
-   DIRECTION_OF_DETECTOR_X_AXIS="-1 0 0"
+   DETECTOR_X_AXIS="-1 0 0"
 elif [ "$DET" == "experimental-ED" ]; then
@@ Line 514: / Line 523: @@
        # find SENSOR_THICKNESS:
-       SENSOR_THICKNESS=`grep thickness tmp2 | sed -e s/'Silicon sensor, thickness'// | awk '{print $1*1000}'`
+       SENSOR_THICKNESS=`grep thickness tmp2 | sed -e s/'Si.* sensor, thickness'// | awk '{print $1*1000}'`
        # find X_RAY_WAVELENGTH:
        X_RAY_WAVELENGTH=`grep Wavelength tmp2 | sed -e s/Wavelength// -e s/A// | awk '{print $1}'`
@@ Line 566: / Line 575: @@
        fi
 # PETRA P14: raw data from Eiger are stored as CBF files so this is treated as Pilatus
-       if [ "$DET_SN" == "Dectris Eiger 16M, E-32-0107" -o "$DET_SN" == "Dectris Eiger 4M, E-08-0107" -o "$DET_SN" == "PILATUS 6M-F, S/N 60-0115-F" ] ; then
+       if [ "$DET_SN" == "Dectris Eiger 4M, E-08-0107" -o "$DET_SN" == "PILATUS 6M-F, S/N 60-0115-F" \
+                                        -o "$DET_SN" == "Dectris EIGER2 CdTe 16M, E-32-0129" ] ; then
          rotation_axis="0 -1 0"
          echo ROTATION_AXIS="0 -1 0" at PETRA P14
+      fi
+# similarly, Eiger 16M at PETRA P14 until May 22, 2021
+      if [ "$DET_SN" == "Dectris Eiger 16M, E-32-0107" ] ; then
+        egrep -q '^# 2020-|^# 2021-0[1-5]|^# 2021-06-[01]|^# 2021-06-2[01]' tmp2 || isatP13=1
+        if [ "$isatP13" == 1 ] ; then
+          echo using the default ROTATION_AXIS=1 0 0 at PETRA P13
+        else
+          rotation_axis="0 -1 0"
+          echo ROTATION_AXIS="0 -1 0" at PETRA P14
+        fi
        fi
 # ESRF ID23-2:
@@ Line 580: / Line 600: @@
   DATA_RANGE="1 $nframes"
   SPOT_RANGE="1 `echo "scale=0; if (${nframes}<2) 1; if (${nframes}>1) ${nframes}/2"|bc -l`"
+ SN=`h5dump -d "/entry/instrument/detector/detector_number" $FIRSTFRAME | awk '/\(0\): /{print $2}' | sed s/\"//g`
+ echo detector serial number is $SN
 # find out if HDF5 from Diamond (DLS=1) or Dectris (DLS=0)
@@ Line 594: / Line 616: @@
    # rotation_axis=`h5dump -a "/entry/sample/transformations/omega/vector" $FIRSTFRAME 2>/dev/null | grep "(0):" | sed -e "s/^.*://; s/,//g"`
    # the above gives -1 0 0 for DLS data instead of the correct 1 0 0, so commented out for now
+  # comment while implementing rev 1.10: this is because the DLS headers are NeXus, so DIRECTION_OF_DETECTOR_X/Y_AXIS must be adjusted.
    else
      echo Eiger HDF5 from Dectris
@@ Line 606: / Line 629: @@
      fi
      echo "STARTING_ANGLE=$STARTING_ANGLE"
-   # If rotation vector set (NeXus)
+   # If rotation axis set (NeXus)
      rotation_axis=`h5dump -a "/entry/sample/transformations/omega/vector" $FIRSTFRAME 2>/dev/null | grep "(0):" | sed -e "s/^.*://; s/,//g"`
    # EIGER2 16M CHESS ID7B2 has S/N E-32-0123 (A. Finke 2020-11-07) v0.99
-    SN=`h5dump -d "/entry/instrument/detector/detector_number" $FIRSTFRAME | awk '/\(0\): /{print $2}' | sed s/\"//g`
      if [ "$SN" == "E-32-0123" ]; then
        rotation_axis="-1 0 0"
@@ Line 616: / Line 638: @@
    # SSRF BL17U1 and SSRF BL10U2 (Feng YU 2021-07-18)
    # Eiger X 16M (S/N E-32-0111) was installed at SSRF BL17U1 from Oct 2017 to Feb 2021. After Feb 2021, it was moved back to SSRF BL10U2.
-	# 2021-02-01 00:00:00 CST time stamp is 1612108800
+	# 2021-03-01 00:00:00 time stamp is 1614528000
      if [ "$SN" == "E-32-0111" ]; then
-       collection_time=`h5dump -d "/entry/instrument/detector/detectorSpecific/data_collection_date" $FIRSTFRAME | grep "(0):" | awk '{print $2}' | sed -e "s/\.\(.*\)/CST/g; s/\"//g"`
+       collection_time=`h5dump -d "/entry/instrument/detector/detectorSpecific/data_collection_date" $FIRSTFRAME | grep "(0):" | awk '{print $2}' | sed -e "s/\.\(.*\)//g; s/\"//g"`
        if [ `uname -s` == "Darwin" ]; then
-         collection_timestamp=`date -j -f "%Y-%m-%dT%H:%M:%S%Z" $collection_time +%s`
+         collection_timestamp=`date -j -f "%Y-%m-%dT%H:%M:%S" $collection_time +%s`
        elif [ `uname -s` == "Linux" ]; then
          collection_timestamp=`date -d $collection_time +%s`
@@ Line 635: / Line 657: @@
            rotation_axis="-1 0 0    ! Cannot determine rotation axis. SSRF BL17U1: -1 0 0; SSRF BL02U1: 0 -1 0"
          fi
-       elif [ $collection_timestamp -ge 0 ] && [ $collection_timestamp -le 1612108800 ]; then
+       elif [ $collection_timestamp -ge 0 ] && [ $collection_timestamp -le 1614528000 ]; then
          # SSRF BL17U1
          rotation_axis="-1 0 0"
@@ Line 666: / Line 688: @@
      fi
 # revision 1.05 specialcase nframes for Eiger detectors at BNL
-     if [ "$SN" == "E-18-0121" -o "$SN" == "E-32-0101" ]; then
+     if [ "$SN" == "E-18-0104" -o "$SN" == "E-32-0101" ]; then
        echo specialcase nframes for Eiger detectors at BNL:
        nframes=`h5dump -A -g "/entry/data" $FIRSTFRAME | grep "DATASPACE  SIMPLE" | sed -e "s/,.*//" | awk '{a+=$5}END{print a}'`
@@ Line 672: / Line 694: @@
        SPOT_RANGE="1 `echo "scale=0; if (${nframes}<2) 1; if (${nframes}>1) ${nframes}/2"|bc -l`"
      fi
+  fi
+# rev 1.10: check for NeXus header. If found, its geometry will overwrite any rotation_axis set until here.
+  NeXus=0
+  h5dump -d "/entry/definition" $FIRSTFRAME | grep -q NXmx && NeXus=1
+  if [ "$NeXus" == "1" ]; then
+    echo NeXus header found. This defines DIRECTION_OF_DETECTOR_X/Y-AXIS and ROTATION_AXIS.
+    DIRECTION_OF_DETECTOR_X_AXIS=$(h5dump -d "/entry/instrument/detector/module/fast_pixel_direction" $FIRSTFRAME 2>/dev/null | grep "(0):" | tail -1 | sed -e "s/^.*://; s/,//g")
+    DIRECTION_OF_DETECTOR_Y_AXIS=$(h5dump -d "/entry/instrument/detector/module/slow_pixel_direction" $FIRSTFRAME 2>/dev/null | grep "(0):" | tail -1 | sed -e "s/^.*://; s/,//g")
+    rotation_axis=$(h5dump -a "/entry/sample/transformations/omega/vector" $FIRSTFRAME 2>/dev/null | grep "(0):" | sed -e "s/^.*://; s/,//g")
+  else
+    echo no NeXus header found.
    fi
    echo DATA_RANGE=$DATA_RANGE
@@ Line 701: / Line 734: @@
    #let SKIP=768
    #NX=$(od -t x -j $SKIP -N 4 $FIRSTFRAME |awk 'NR==1{print toupper($2)}'|perl -nle '@array= $_ =~/.{2}/g; print "ibase=16;obase=A;".join("",reverse @array)'|bc)
-!  NX=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(768);print "%.4d"%struct.unpack(">i",f.read(4))')
-!  NY=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(772);print "%.4d"%struct.unpack(">i",f.read(4))')
    NX=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(768);print("%.4d"%struct.unpack(">i",f.read(4)))')
    NY=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(772);print("%.4d"%struct.unpack(">i",f.read(4)))')
-!  DETECTOR_DISTANCE=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(344);print "-%.4f"%struct.unpack(">f",f.read(4))')
    DETECTOR_DISTANCE=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(344);print("-%.4f"%struct.unpack(">f",f.read(4)))')
-!  ORGX=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(540);print "%.4f"%struct.unpack(">f",f.read(4))')
-!  ORGY=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(544);print "%.4f"%struct.unpack(">f",f.read(4))')
    ORGX=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(540);print("%.4f"%struct.unpack(">f",f.read(4)))')
    ORGY=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(544);print("%.4f"%struct.unpack(">f",f.read(4)))')
-!  OSCILLATION_RANGE=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(524);phis,phie=struct.unpack(">ff",f.read(8));print "%.4f"%(phie-phis)')
    OSCILLATION_RANGE=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(524);phis,phie=struct.unpack(">ff",f.read(8));print("%.4f"%(phie-phis))')
-!  QX=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(776);print "%.6f"%struct.unpack(">f",f.read(4))')
-!  QY=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(780);print "%.6f"%struct.unpack(">f",f.read(4))')
    QX=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(776);print("%.6f"%struct.unpack(">f",f.read(4)))')
    QY=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(780);print("%.6f"%struct.unpack(">f",f.read(4)))')
-!  X_RAY_WAVELENGTH=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(292);print "%.6f"%struct.unpack(">f",f.read(4))')
    X_RAY_WAVELENGTH=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(292);print("%.6f"%struct.unpack(">f",f.read(4)))')
@@ Line 988: / Line 1,007: @@
      if [ -e /usr/local/lib64/dectris-neggia.so ]; then
        echo LIB=/usr/local/lib64/dectris-neggia.so >> XDS.INP
-       echo LIB= line was written to XDS.INP . For Apple processors you must modify the name.
+       echo LIB= line was written to XDS.INP . For Apple ARM64 processors, you must modify the name.
      else
        echo !LIB=/usr/local/lib64/dectris-neggia.so >> XDS.INP
@@ Line 1,029: / Line 1,048: @@
   cat >> XDS.INP << eof
 DIRECTION_OF_DETECTOR_X-AXIS=$DIRECTION_OF_DETECTOR_X_AXIS
-DIRECTION_OF_DETECTOR_Y-AXIS=0 1 0
+DIRECTION_OF_DETECTOR_Y-AXIS=$DIRECTION_OF_DETECTOR_Y_AXIS
 eof
   fi
@@ Line 1,199: / Line 1,218: @@
 echo spots in FRAME.cbf!
 rm -f tmp1 tmp2
 </pre>
@@ Line 1,282: / Line 1,300: @@
 * At some beamlines, the ROTATION_AXIS should be -1 0 0 ("backwards") instead of the usual 1 0 0 ("horizontal"), or even 0 1 0 ("vertical") like at one of the PETRA Hamburg BLs. The frame headers do not have this information, so the default chosen by [[generate_XDS.INP]] may be wrong and need manual correction. Pls also see the article [[Beamline notes]].
 * Sometimes, the x- and y- values of the primary beam position recorded in the header should be used for ORGY and ORGX (i.e reversed) instead of as ORGX and ORGY. For ADSC, this has been implemented in the script for a number of beamlines.
-* there are apparently several flavours of HDF5 files produced at Diamond Light Source. They differ e.g. in the naming of the header items. This means that items like NX, NY, DETECTOR_DISTANCE and number of images cannot be found by the [[generate_XDS.INP]] script. Example: The data at /dls/i04-1/data/2021/mx28114-9/processing/Lenye_Diamini/ThiL/ThiL found during the CCP4 2021 online Cape Town workshop. A workaround is to use e.g. xia2 pipeline=3dii to process these files, and - if needed - extract those items from its output files, e.g. from DEFAULT/NATIVE/SWEEP1/index/XDS.INP .
+* there are apparently several flavours of HDF5 files produced at Diamond Light Source. They differ e.g. in the naming of the header items. This means that items like NX, NY, DETECTOR_DISTANCE and number of images cannot always be determined by the [[generate_XDS.INP]] script. Example: The data at /dls/i04-1/data/2021/mx28114-9/processing/Lenye_Diamini/ThiL/ThiL found during the CCP4 2021 online Cape Town workshop. A workaround is to use e.g. xia2 pipeline=3dii to process these files, and - if needed - extract those items from its output files, e.g. from DEFAULT/NATIVE/SWEEP1/index/XDS.INP .
 == See also ==

Generate XDS.INP: Difference between revisions

Generate XDS.INP (view source)

Revision as of 17:38, 27 April 2022

Navigation menu

Search