Generate XDS.INP: Difference between revisions

→‎The script: take away POSITION from REFINE(CORRECT) if electron diffraction data
m (fix quotation marks in Eiger2 section at the bottom)
(→‎The script: take away POSITION from REFINE(CORRECT) if electron diffraction data)
(7 intermediate revisions by the same user not shown)
Line 10: Line 10:


== The script ==
== The script ==
<pre>
<pre>


Line 119: Line 120:
# revision 1.01 . KD 07/12/2020 ROTATION_AXIS=0 -1 0 for Pilatus3 2M, S/N 24-0118 at ID23-2 (http://www.globalphasing.com/autoproc/wiki/index.cgi?BeamlineSettings)
# revision 1.01 . KD 07/12/2020 ROTATION_AXIS=0 -1 0 for Pilatus3 2M, S/N 24-0118 at ID23-2 (http://www.globalphasing.com/autoproc/wiki/index.cgi?BeamlineSettings)
# revision 1.02 . KD 11/01/2021 recognize mar555 detector as mar345 (thanks to Thomas Hauß, HZB)  
# revision 1.02 . KD 11/01/2021 recognize mar555 detector as mar345 (thanks to Thomas Hauß, HZB)  
REVISION="1.02 (11-Jan-2021)"
# revision 1.03 . Feng YU 18/7/2021 fix/expand information about Shanghai Synchrotron Radiation Facility (SSRF)
# revision 1.04 . Zhipu Luo 13/08/2021 extract ORGX,ORGY for electron diffraction from SMV header if the wavelength value starts with 0.0
REVISION="1.04 (13-AUG-2021)"


#                                                                                                             
#                                                                                                             
Line 172: Line 175:
SEPMIN=7.0
SEPMIN=7.0
CLUSTER_RADIUS=3.5
CLUSTER_RADIUS=3.5
REFINE_CORRECT="CELL BEAM ORIENTATION AXIS POSITION ! Default is: refine everything"


dname=`echo "$1" | xargs dirname`
dname=`echo "$1" | xargs dirname`
Line 248: Line 252:
  grep -q BRUKER tmp2 && grep -q CBF tmp2  && DET=Bruker-cbf   
  grep -q BRUKER tmp2 && grep -q CBF tmp2  && DET=Bruker-cbf   
  grep -q CMOS1 tmp2 && DET=adsc-CMOS1   
  grep -q CMOS1 tmp2 && DET=adsc-CMOS1   
  grep -q MARCONTROL tmp2 && DET=OLDMAR                  
  grep -q MARCONTROL tmp2 && DET=OLDMAR
grep -q "WAVELENGTH=0.0" tmp2 && DET=experimental-ED                                   
else
else
  h5dump -d "/entry/instrument/detector/description" $FIRSTFRAME | grep -i Eiger > /dev/null && DET=eiger
  h5dump -d "/entry/instrument/detector/description" $FIRSTFRAME | grep -i Eiger > /dev/null && DET=eiger
Line 468: Line 473:
   DETECTOR_DISTANCE=`grep ^DISTANCE tmp2 | sed s/DISTANCE=/-/`                                                                     
   DETECTOR_DISTANCE=`grep ^DISTANCE tmp2 | sed s/DISTANCE=/-/`                                                                     
   OSCILLATION_RANGE=`grep OMEGA_DELTA tmp2 | sed s/OMEGA_DELTA=//`   
   OSCILLATION_RANGE=`grep OMEGA_DELTA tmp2 | sed s/OMEGA_DELTA=//`   
   DIRECTION_OF_DETECTOR_X_AXIS="-1 0 0"                                                                 
   DIRECTION_OF_DETECTOR_X_AXIS="-1 0 0"  
                                                                  
elif [ "$DET" == "experimental-ED" ]; then
  DETECTOR="ADSC MINIMUM_VALID_PIXEL_VALUE= 1 OVERLOAD= 65000  ! OVERLOAD is really unknown "
  echo --- Electron diffraction data in SMV format!
  SENSOR_THICKNESS=0.01                                                       
  sed s/\;// tmp2 > tmp1                                                     
  mv tmp1 tmp2                                                               
  X_RAY_WAVELENGTH=`grep WAVELENGTH tmp2 | head -1 | sed s/WAVELENGTH=//`
  NX=`grep SIZE1 tmp2 | tail -1 | sed s/SIZE1=//`
  QX=`grep PIXEL_SIZE tmp2 | sed s/PIXEL_SIZE=//`
  NY=`grep SIZE2 tmp2 | tail -1 | sed s/SIZE2=//`                                           
  QY=$QX 
  BEAM_CENTER_X=`grep BEAM_CENTER_X tmp2 | sed s/BEAM_CENTER_X=//`
  BEAM_CENTER_Y=`grep BEAM_CENTER_Y tmp2 | sed s/BEAM_CENTER_Y=//`
  QXY=`echo "scale=1; $QX*($BEAM_CENTER_X+$BEAM_CENTER_Y)" | bc -l`
      if [ "$QXY" "<" "$BEAM_CENTER_X" ] ; then
      ORGX=`echo "scale=2; $BEAM_CENTER_Y/$QX" | bc -l`
      ORGY=`echo "scale=2; $BEAM_CENTER_X/$QX" | bc -l`
      else
      ORGX=`echo "scale=2; $BEAM_CENTER_X/1" | bc -l`
      ORGY=`echo "scale=2; $BEAM_CENTER_Y/1" | bc -l`
      fi
  DETECTOR_DISTANCE=`grep ^DISTANCE tmp2 | sed s/DISTANCE=//`                                                                   
  OSCILLATION_RANGE=`grep OSC_RANGE tmp2 | sed s/OSC_RANGE=//`
  STARTING_ANGLE=`grep OSC_START tmp2 | sed s/OSC_START=//`
  echo --- For TIMEPIX detector, please fix ROTATION_AXIS yourself!
  DIRECTION_OF_DETECTOR_X_AXIS="1 0 0"
  REFINE_CORRECT="ORIENTATION CELL AXIS BEAM ! for ED, no POSITION when CELL is refined"


elif [ "$DET" == "pilatus" ]; then
elif [ "$DET" == "pilatus" ]; then
Line 567: Line 600:
   # STARTING_ANGLE:  the \- was introduced in version 0.91 to allow negative values :
   # STARTING_ANGLE:  the \- was introduced in version 0.91 to allow negative values :
     STARTING_ANGLE=`h5dump -d "/entry/sample/goniometer/omega_start" $FIRSTFRAME | awk '/\(0\): [\-0-9]/{print $2}'`
     STARTING_ANGLE=`h5dump -d "/entry/sample/goniometer/omega_start" $FIRSTFRAME | awk '/\(0\): [\-0-9]/{print $2}'`
  # /entry/sample/goniometer/omega_start is missing in some eiger2 detectors (e.g. Eiger2 9M with fw version release-2020.2.1 and SIMPLON API 1.8) (Feng YU 2021-07-18)
    if [ "$STARTING_ANGLE" == "" ]; then
  echo "/entry/sample/goniometer/omega_start not found, try /entry/sample/goniometer/omega"
  STARTING_ANGLE=`h5dump -d /entry/sample/goniometer/omega $FIRSTFRAME | grep "(0):" | head -n 1 | awk '{print $2}' | sed -e "s/,//g"`
    fi
    echo "STARTING_ANGLE=$STARTING_ANGLE"
   # If rotation vector set (NeXus)
   # If rotation vector set (NeXus)
     rotation_axis=`h5dump -a "/entry/sample/transformations/omega/vector" $FIRSTFRAME 2>/dev/null | grep "(0):" | sed -e "s/^.*://; s/,//g"`
     rotation_axis=`h5dump -a "/entry/sample/transformations/omega/vector" $FIRSTFRAME 2>/dev/null | grep "(0):" | sed -e "s/^.*://; s/,//g"`
   # Eiger 16M SSRF BL17U1 has S/N E-32-0111;  EIGER2 16M CHESS ID7B2 has S/N E-32-0123 (A. Finke 2020-11-07) v0.99
   # EIGER2 16M CHESS ID7B2 has S/N E-32-0123 (A. Finke 2020-11-07) v0.99
     SN=`h5dump -d "/entry/instrument/detector/detector_number" $FIRSTFRAME | awk '/\(0\): /{print $2}' | sed s/\"//g`
     SN=`h5dump -d "/entry/instrument/detector/detector_number" $FIRSTFRAME | awk '/\(0\): /{print $2}' | sed s/\"//g`
     if [ "$SN" == "E-32-0111" ] || [ "$SN" == "E-32-0123" ]; then
     if [ "$SN" == "E-32-0123" ]; then
       rotation_axis="-1 0 0"
       rotation_axis="-1 0 0"
       echo SSRF BL17U1 or CHESS ID7B2 with inverted rotation axis
       echo CHESS ID7B2 with inverted rotation axis
     fi  
    fi
  # SSRF BL17U1 and SSRF BL10U2 (Feng YU 2021-07-18)
  # Eiger X 16M (S/N E-32-0111) was installed at SSRF BL17U1 from Oct 2017 to Feb 2021. After Feb 2021, it was moved back to SSRF BL10U2.
# 2021-02-01 00:00:00 CST time stamp is 1612108800
    if [ "$SN" == "E-32-0111" ]; then
      collection_time=`h5dump -d "/entry/instrument/detector/detectorSpecific/data_collection_date" $FIRSTFRAME | grep "(0):" | awk '{print $2}' | sed -e "s/\.\(.*\)/CST/g; s/\"//g"`
      if [ `uname -s` == "Darwin" ]; then
        collection_timestamp=`date -j -f "%Y-%m-%dT%H:%M:%S%Z" $collection_time +%s`
      elif [ `uname -s` == "Linux" ]; then
        collection_timestamp=`date -d $collection_time +%s`
      else
        collection_timestamp=0
      fi
      if [ $collection_timestamp -eq 0 ]; then
        is_rotation_axis_set=1
        rotation_axis=`h5dump -d "/SSRF/RotationAxis" $FIRSTFRAME 2>/dev/null || is_rotation_axis_set=0`
        if [ "$is_rotation_axis_set" == "1" ]; then
          rotation_axis=`h5dump -d "/SSRF/RotationAxis" $FIRSTFRAME | grep "(0):" | sed -e "s/^.*://; s/,//g"`
          echo "The rotation axis of SSRF BL10U2 (Eiger X 16M) is $rotation_axis"
        else
          rotation_axis="-1 0 0    ! Cannot determine rotation axis. SSRF BL17U1: -1 0 0; SSRF BL02U1: 0 -1 0"
        fi
      elif [ $collection_timestamp -ge 0 ] && [ $collection_timestamp -le 1612108800 ]; then
        # SSRF BL17U1
        rotation_axis="-1 0 0"
        echo "SSRF BL17U1 (Eiger X 16M) with inverted rotation axis"
      else
        # SSRF BL10U2
        is_rotation_axis_set=1
        rotation_axis=`h5dump -d "/SSRF/RotationAxis" $FIRSTFRAME 2>/dev/null || is_rotation_axis_set=0`
        if [ "$is_rotation_axis_set" == "1" ]; then
          rotation_axis=`h5dump -d "/SSRF/RotationAxis" $FIRSTFRAME | grep "(0):" | sed -e "s/^.*://; s/,//g"`
          echo "The rotation axis of SSRF BL10U2 (Eiger X 16M) is $rotation_axis"
        else
          rotation_axis="0 -1 0"
          echo "SSRF BL10U2 (Eiger X 16M) with vertical rotation axis"
        fi
      fi
    fi
  # SSRF BL02U1 (Feng YU 2021-07-18)
  # EIGER2 S 9M SSRF BL02U1 has S/N E-18-0121
    if [ "$SN" == "E-18-0121" ]; then
      is_rotation_axis_set=1
      rotation_axis=`h5dump -d "/SSRF/RotationAxis" $FIRSTFRAME 2>/dev/null || is_rotation_axis_set=0`
      if [ "$is_rotation_axis_set" == "1" ]; then
        rotation_axis=`h5dump -d "/SSRF/RotationAxis" $FIRSTFRAME | grep "(0):" | sed -e "s/^.*://; s/,//g"`
        echo "The rotation axis of SSRF BL02U1 (Eiger2 S 9M) is $rotation_axis"
      else
        rotation_axis="1 0 0"
        echo "SSRF BL02U1 (Eiger2 S 9M) with horizontal rotation axis"
      fi
     fi
   fi
   fi
   DETECTOR="EIGER MINIMUM_VALID_PIXEL_VALUE=0 OVERLOAD= $OVERLOAD"
   DETECTOR="EIGER MINIMUM_VALID_PIXEL_VALUE=0 OVERLOAD= $OVERLOAD"
Line 874: Line 964:
! however, if the spots from COLSPOT extend to 2A then POSITION could, and if 1.5A POSITION should be refined
! however, if the spots from COLSPOT extend to 2A then POSITION could, and if 1.5A POSITION should be refined
REFINE(IDXREF)=CELL BEAM ORIENTATION AXIS  ! add POSITION if high resolution, or DETECTOR_DISTANCE inaccurate
REFINE(IDXREF)=CELL BEAM ORIENTATION AXIS  ! add POSITION if high resolution, or DETECTOR_DISTANCE inaccurate
REFINE(INTEGRATE)= POSITION BEAM ORIENTATION ! AXIS CELL . If 1.5A or higher it is ok to refine CELL
REFINE(INTEGRATE)= POSITION BEAM ORIENTATION ! AXIS CELL . If 1.5A or higher it is ok to refine CELL (unless electron diffraction)
! REFINE(CORRECT)=CELL BEAM ORIENTATION AXIS POSITION ! Default is: refine everything
REFINE(CORRECT)= $REFINE_CORRECT
 
! parameters specifically for this detector and beamline:
! parameters specifically for this detector and beamline:
DETECTOR= $DETECTOR
DETECTOR= $DETECTOR
Line 1,102: Line 1,191:
echo BKGPIX.cbf, and the agreement of predicted and observed spots in FRAME.cbf!
echo BKGPIX.cbf, and the agreement of predicted and observed spots in FRAME.cbf!
rm -f tmp1 tmp2
rm -f tmp1 tmp2
</pre>
</pre>


Line 1,182: Line 1,272:
* The script tries to interpret the header of the frames, so is currently limited to Dectris (Pilatus, Eiger), ADSC (Quantum), Rigaku (several types), MAR (CCD and image plate) detectors, and one Bruker detector. Other detectors need some values to be manually filled into XDS.INP - the relevant places are marked with XXX. These are detector properties (type, pixel size and number, min and max counts in a pixel), and experimental parameters like ROTATION_AXIS, OSCILLATION_RANGE, X-RAY_WAVELENGTH, DETECTOR_DISTANCE, and XORG, YORG. For fine-tuning of detector parameters, see the [http://xds.mpimf-heidelberg.mpg.de/html_doc/xds_prepare.html detector-specific templates].
* The script tries to interpret the header of the frames, so is currently limited to Dectris (Pilatus, Eiger), ADSC (Quantum), Rigaku (several types), MAR (CCD and image plate) detectors, and one Bruker detector. Other detectors need some values to be manually filled into XDS.INP - the relevant places are marked with XXX. These are detector properties (type, pixel size and number, min and max counts in a pixel), and experimental parameters like ROTATION_AXIS, OSCILLATION_RANGE, X-RAY_WAVELENGTH, DETECTOR_DISTANCE, and XORG, YORG. For fine-tuning of detector parameters, see the [http://xds.mpimf-heidelberg.mpg.de/html_doc/xds_prepare.html detector-specific templates].
* The authors have made a "best effort" to provide a XDS.INP that results in the correct sign of the anomalous signal. In the case of one detector type (internally called Rigaku SMV) this requires reversal of one detector axis, and a negative DETECTOR_DISTANCE, as is found in some of the [http://xds.mpimf-heidelberg.mpg.de/html_doc/xds_prepare.html detector-specific templates]. '''For an unusual or unknown detector setup, the correct sign of the anomalous signal needs to be established and verified e.g. with a good dataset from a test crystal that has an anomalous signal.''' The authors do not take any responsibility for problems arising from incorrect sign of the anomalous signal, nor - obviously! - for any other mischief arising in or from data processing.
* The authors have made a "best effort" to provide a XDS.INP that results in the correct sign of the anomalous signal. In the case of one detector type (internally called Rigaku SMV) this requires reversal of one detector axis, and a negative DETECTOR_DISTANCE, as is found in some of the [http://xds.mpimf-heidelberg.mpg.de/html_doc/xds_prepare.html detector-specific templates]. '''For an unusual or unknown detector setup, the correct sign of the anomalous signal needs to be established and verified e.g. with a good dataset from a test crystal that has an anomalous signal.''' The authors do not take any responsibility for problems arising from incorrect sign of the anomalous signal, nor - obviously! - for any other mischief arising in or from data processing.
* At some beamlines, the ROTATION_AXIS should be -1 0 0 ("backwards") instead of the usual 1 0 0 ("horizontal"), or even 0 1 0 ("vertical") like at one of the PETRA Hamburg BLs. We have only just started the article [[Beamline notes]], and the frame headers do not have this information, so the default chosen by [[generate_XDS.INP]] may be wrong and need manual correction.
* At some beamlines, the ROTATION_AXIS should be -1 0 0 ("backwards") instead of the usual 1 0 0 ("horizontal"), or even 0 1 0 ("vertical") like at one of the PETRA Hamburg BLs. The frame headers do not have this information, so the default chosen by [[generate_XDS.INP]] may be wrong and need manual correction. Pls also see the article [[Beamline notes]].
* Sometimes, the x- and y- values of the primary beam position recorded in the header should be used for ORGY and ORGX (i.e reversed) instead of as ORGX and ORGY. For ADSC, this has been implemented in the script for a number of beamlines.
* Sometimes, the x- and y- values of the primary beam position recorded in the header should be used for ORGY and ORGX (i.e reversed) instead of as ORGX and ORGY. For ADSC, this has been implemented in the script for a number of beamlines.
* there are apparently several flavours of HDF5 files produced at Diamond Light Source. They differ e.g. in the naming of the header items. This means that items like NX, NY, DETECTOR_DISTANCE and number of images cannot be found by the [[generate_XDS.INP]] script. Example: The data at /dls/i04-1/data/2021/mx28114-9/processing/Lenye_Diamini/ThiL/ThiL found during the CCP4 2021 online Cape Town workshop. A workaround is to use e.g. xia2 pipeline=3dii to process these files, and - if needed - extract those items from its output files, e.g. from DEFAULT/NATIVE/SWEEP1/index/XDS.INP .


== See also ==
== See also ==
2,684

edits