Generate XDS.INP: Difference between revisions

m
→‎The script: improve wording
(→‎The script: fix upper spot range calculation if nframe=1 and .h5)
m (→‎The script: improve wording)
(76 intermediate revisions by 3 users not shown)
Line 12: Line 12:


<pre>
<pre>
#!/bin/bash                                                                   
#!/bin/bash                                                                   
# purpose: generate XDS.INP                                                   
# purpose: generate XDS.INP                                                   
Line 95: Line 96:
# revision 0.77 . KD 19/12/2017 obtain QX QY from CBF header.
# revision 0.77 . KD 19/12/2017 obtain QX QY from CBF header.
# revision 0.78 . KD 21/12/2017 if possible and sensible, provide LIB= line with hardcoded /usr/local/lib64/dectris-neggia.so .
# revision 0.78 . KD 21/12/2017 if possible and sensible, provide LIB= line with hardcoded /usr/local/lib64/dectris-neggia.so .
REVISION="0.78 (21-Dec-2017)"
# revision 0.79 . KD 16/01/2018 read OVERLOAD from Pilatus miniCBF header instead of fixing at 1048576
# revision 0.80 . KD 13/02/2018 remove DISTANCE keyword from REFINE() list; remove POSITION from REFINE(IDXREF)
# revision 0.81 . KD 21/02/2018 when encountering CBF files from Eiger (ESRF), treat as Pilatus detector
# revision 0.82 . KD 01/03/2018 STARTING_ANGLE for MarCCD/Pilatus/PHOTON, enabling to use dials.rs_mapper with spot2pdb.
# revision 0.83 . KD 25/06/2018 for ADSC detector #458 at APS BM19, revert the definition of ROTATION_AXIS=-1 0 0. See "Beamline notes" in this wiki.
# revision 0.84 . KD 10/10/2018 implement Pilatus detector with d*TREK header
# revision 0.85 . Jie Nan 09/01/2019 STARTING_ANGLE for Eiger
# revision 0.86 . Keitaro 03/05/2019 Add PILATUS3 6M, S/N 60-0123 at SSRF BL18U-1 with ROTATION_AXIS=-1 0 0
# revision 0.87 . KD 12/10/2019 Add PILATUS XXX, S/N XX-XXX at SSRF BL19U1 and MarCCD detector #43 at BL17B1 with ROTATION_AXIS=-1 0 0
# revision 0.88 . KD 16/10/2019 fixes for SSRF, add "-maxdepth 1" to "find -H ..."
# revision 0.89 . KD 21/10/2019 add ADSC S/N 905 at ALS 8.2.1, S/N 928 at Australian Synchrotron MX2 beamline; final SSRF fixes
# revision 0.90 . KD 25/10/2019 add OLDMAR detector type. Tested w/ SBGrid data set 6. Anomalous signal may have wrong hand!
# revision 0.91 . KD 16/01/2020 Allow negative starting angle for Eiger (found -33 at SLS !).
# revision 0.92 . KD 27/02/2020 read *_master.h5 from Diamond Light Source
# revision 0.93 . KD 13/03/2020 print out 2theta for MarCCD (DETECTOR_*_AXIS can be derived from this)
# revision 0.94 . KD 16/03/2020 bugfix for Bruker-cbf to make bc accept e.g. 3.1e-005 by using awk printf "%.5f" instead of awk print
# revision 0.95 . KD 29/07/2020 fix DLS Eiger HDF5 variant OSCILLATION_RANGE, STARTING_ANGLE. Attention: DLS Eiger variant needs h5dump 1.10 for OVERLOAD!
# revision 0.96 . KD 03/10/2020 make script echo detector serial number if inverted ROTATION_AXIS is detected.
# revision 0.97 . KD 19/10/2020 add UNTRUSTED_RECTANGLEs for Eiger2 (which has a few pixels less than Eiger (thanks to Andreas Förster)
# revision 0.98 . Thomas Hauß (HZB) 06/11/2020 switch Python print syntax to Python3
# revision 0.99 . Aaron Finke (CHESS) 07/11/2020 add ROTATION_AXIS=-1 0 0 for EIGER2 16M detector at MX beamline ID7B2 (CHESS)
# revision 1.00 . Gleb Bourenkov 10/11/2020 add ROTATION_AXIS=0 -1 0 for beamline P14 (EMBL Hamburg) detectors Eiger 16M S/N E-32-0107; previously Eiger 4M S/N E-08-0107 and PILATUS 6M-F, S/N 60-0115-F
# revision 1.01 . KD 07/12/2020 ROTATION_AXIS=0 -1 0 for Pilatus3 2M, S/N 24-0118 at ID23-2 (http://www.globalphasing.com/autoproc/wiki/index.cgi?BeamlineSettings)
# revision 1.02 . KD 11/01/2021 recognize mar555 detector as mar345 (thanks to Thomas Hauß, HZB)
# revision 1.03 . Feng YU 18/7/2021 fix/expand information about Shanghai Synchrotron Radiation Facility (SSRF)
# revision 1.04 . Zhipu Luo 13/08/2021 extract ORGX,ORGY for electron diffraction from SMV header if the wavelength value starts with 0.0
# revision 1.05 . KD specialcase extraction of number of images for BNL detectors E-32-0101 and E-18-0104
# revision 1.06 . KD fix URLs in output. Availability of Apple M1 processor dectris-neggia-Apple-arm64.so . LC_ALL=C .
REVISION="1.06 (15-OCT-2021)"


#                                                                                                             
#                                                                                                             
Line 105: Line 134:
# - for ADSC detectors, there are at least three ways to obtain ORGX and ORGY values from the header (see below);
# - for ADSC detectors, there are at least three ways to obtain ORGX and ORGY values from the header (see below);
# - the same might be a problem for MAR headers, too (not sure about this)  
# - the same might be a problem for MAR headers, too (not sure about this)  
# - on Mac OS X, Xcode (from http://developer.apple.com/tools/xcode) might be needed - not sure about this                             
# - on Mac OS X, the Xcode command line tools (from https://developer.apple.com/download/more/) are needed.                           
#                                                                                                                 
#                                                                                                                 
# notes for debugging of the script:                                                                             
# notes for debugging of the script:                                                                             
# - add the -v option to the first line, to see where an error occurs                                             
# - add the -x option to the first line, to see where an error occurs                                             
# - comment out the removal of tmp1 and tmp2 in the last line                                                     
# - comment out the removal of tmp1 and tmp2 in the last line                                                     
#                                                                                                                 
#                                                                                                                 
Line 119: Line 148:
   exit                                                                                                           
   exit                                                                                                           
fi                                                                                                               
fi                                                                                                               
# make sure the locale does not interfere with e.g. awk calculations:
LC_ALL="C";export LC_ALL
#                                                                                                                 
#                                                                                                                 
# defaults:                                                                                                       
# defaults:                                                                                                       
#                                                                                                              
#      
# conversion radian / degrees:
DEGTOR=57.2957795
                                                                                                       
DETECTOR="XXX MINIMUM_VALID_PIXEL_VALUE=XXX OVERLOAD=XXX"                                                         
DETECTOR="XXX MINIMUM_VALID_PIXEL_VALUE=XXX OVERLOAD=XXX"                                                         
REVERSE_PHI="no"
REVERSE_PHI="no"
Line 141: Line 175:
# default FRACTION_OF_POLARIZATION
# default FRACTION_OF_POLARIZATION
pol_frac=0.98
pol_frac=0.98
STARTING_ANGLE=0
dtrek_det=""
SEPMIN=7.0
CLUSTER_RADIUS=3.5
REFINE_CORRECT="CELL BEAM ORIENTATION AXIS POSITION ! Default is: refine everything"


dname=`echo "$1" | xargs dirname`
dname=`echo "$1" | xargs dirname`
Line 151: Line 190:
# cope with blanks in directory / file name
# cope with blanks in directory / file name
IFS=$'\n'
IFS=$'\n'
find -H $dname -name "$bname" -or -name "${bname}.bz2" -or -name "${bname}.gz" -or -name "${bname}.xz" | egrep -v "_00000.cbf|_000.img" | sort > tmp1
find -H $dname -maxdepth 1 -name "$bname" -or -name "${bname}.bz2" -or -name "${bname}.gz" -or -name "${bname}.xz" | egrep -v "_00000.cbf|_000.img" | sort > tmp1
if [ ! -s tmp1 ]
if [ ! -s tmp1 ]
then
then
Line 170: Line 209:
  data_last=`cut -b $pos1-$pos2 tmp1 | tail -n1 | bc`
  data_last=`cut -b $pos1-$pos2 tmp1 | tail -n1 | bc`
  DATA_RANGE="$data_first $data_last"
  DATA_RANGE="$data_first $data_last"
echo DATA_RANGE=$DATA_RANGE
   
   
  # set SPOT_RANGE to first half of DATA_RANGE
  # set SPOT_RANGE to first half of DATA_RANGE
Line 181: Line 221:
FIRSTFRAME=`head -1 tmp1`
FIRSTFRAME=`head -1 tmp1`
echo $FIRSTFRAME | grep "\.h5$" && is_h5=1 || is_h5=0
echo $FIRSTFRAME | grep "\.h5$" && is_h5=1 || is_h5=0
if [ "$is_h5" == 1 ]; then
# nframes=`h5dump -A -g "/entry/data" $FIRSTFRAME | grep "DATASPACE  SIMPLE" | sed -e "s/,.*//" | awk '{a+=$5}END{print a}'`
nframes=`h5dump -d "/entry/instrument/detector/detectorSpecific/nimages" $FIRSTFRAME | awk '/\(0\): [0-9]/{print $2}'`
DATA_RANGE="1 $nframes"
SPOT_RANGE="1 `echo "scale=0; if (${nframes}<2) 1; if (${nframes}>1) ${nframes}/2"|bc -l`"
fi
echo DATA_RANGE=$DATA_RANGE


# find out detector type
# find out detector type
Line 206: Line 237:
  # the reason is that FIRSTFRAME may contain a blank, which makes some commands fail                                                 
  # the reason is that FIRSTFRAME may contain a blank, which makes some commands fail                                                 
  egrep -qi 'marccd|Corrected' tmp2 && DET=mccd                                 
  egrep -qi 'marccd|Corrected' tmp2 && DET=mccd                                 
  grep -q PILATUS tmp2            && DET=pilatus                            
  grep -q PILATUS tmp2            && DET=pilatus  
grep -iq Eiger tmp2              && DET=pilatus                           
  head -n1 tmp2 | grep -q "^RAXIS" && DET=raxis
  head -n1 tmp2 | grep -q "^RAXIS" && DET=raxis
  head -n1 tmp2 | grep -q "^R-AXIS" && DET=raxis
  head -n1 tmp2 | grep -q "^R-AXIS" && DET=raxis
  grep -q "^SOURCE_WAVELENGTH= *1" tmp2 && DET=dtrek
  grep -q "^SOURCE_WAVELENGTH= *1" tmp2 && DET=dtrek
  grep -q BEAM_CENTER_X tmp2      && DET=adsc                                 
  grep -q BEAM_CENTER_X tmp2      && DET=adsc                                 
  grep -q mar345 tmp2 && DET=MAR345    
  grep -q mar345 tmp2 && DET=MAR345
# rev. 1.02: in one case, mar555 data could be processed pretending it is mar345, so:
grep -q mar555 tmp2 && DET=MAR345
  grep -q BRUKER tmp2 && grep -q CBF tmp2  && DET=Bruker-cbf   
  grep -q BRUKER tmp2 && grep -q CBF tmp2  && DET=Bruker-cbf   
  grep -q CMOS1 tmp2 && DET=adsc-CMOS1                  
  grep -q CMOS1 tmp2 && DET=adsc-CMOS1  
grep -q MARCONTROL tmp2 && DET=OLDMAR 
grep -q "WAVELENGTH=0.0" tmp2 && DET=experimental-ED                                   
else
else
  h5dump -d "/entry/instrument/detector/description" $FIRSTFRAME | grep -i Eiger > /dev/null && DET=eiger
  h5dump -d "/entry/instrument/detector/description" $FIRSTFRAME | grep -i Eiger > /dev/null && DET=eiger
Line 229: Line 265:
elif [ "$DET" == "mccd" ]; then  
elif [ "$DET" == "mccd" ]; then  
   echo Data from a MarCCD detector
   echo Data from a MarCCD detector
# http://www.sb.fsu.edu/~xray/Manuals/marCCD165header.html has header information
                                    
                                    
   DETECTOR="CCDCHESS MINIMUM_VALID_PIXEL_VALUE= 1 OVERLOAD= 65500"
   DETECTOR="CCDCHESS MINIMUM_VALID_PIXEL_VALUE= 1 OVERLOAD= 65500"
Line 236: Line 273:
   # Check detector serial number and recognize beamline for reversed-phi setting.
   # Check detector serial number and recognize beamline for reversed-phi setting.
   # Known detectors for reversed-phi in SPring-8: 24: BL26B2 Mar225, 31: BL32XU MX225HE, 38: BL44XU MX225HE, 42: BL44XU MX300HE, 40: BL41XU MX225HE, 106: BL32XU MX225HS
   # Known detectors for reversed-phi in SPring-8: 24: BL26B2 Mar225, 31: BL32XU MX225HE, 38: BL44XU MX225HE, 42: BL44XU MX300HE, 40: BL41XU MX225HE, 106: BL32XU MX225HS
  # same for SSRF: BL17B1 rayonix MX300. As on 2019-10-13, this also needs doubling of ORGX and ORGY. But the beamline staff (Wenming) wants to fix the header.
   REVERSEPHI_SNs="
   REVERSEPHI_SNs="
24
24
Line 242: Line 280:
40
40
42
42
43
106
106
"
"
Line 248: Line 287:
   if echo "${DET_SN}${REVERSEPHI_SNs}" | sort | uniq -d | grep [0-9] > /dev/null; then
   if echo "${DET_SN}${REVERSEPHI_SNs}" | sort | uniq -d | grep [0-9] > /dev/null; then
     REVERSE_PHI="yes"
     REVERSE_PHI="yes"
    echo inverted ROTATION_AXIS since Detector Serial Number is "${DET_SN}"
   fi
   fi


Line 262: Line 302:
   DETECTOR_DISTANCE=$(od -t dI -j $SKIP -N 4 tmp2 | head -1 | awk '{print $2}')
   DETECTOR_DISTANCE=$(od -t dI -j $SKIP -N 4 tmp2 | head -1 | awk '{print $2}')
   DETECTOR_DISTANCE=`echo "scale=3; $DETECTOR_DISTANCE/1000" | bc -l`                                     
   DETECTOR_DISTANCE=`echo "scale=3; $DETECTOR_DISTANCE/1000" | bc -l`                                     
                                                                                                         
 
# Mar 12, 2020 KD
  let SKIP=1724
  TWOTHETA=$(od -t dI -j $SKIP -N 4 tmp2 | head -1 | awk '{print $2}')
  TWOTHETA=`echo "scale=3; $TWOTHETA/1000" | bc -l`
  echo 2THETA= $TWOTHETA
 
  let SKIP=1024+256+128+256+44
  STARTING_ANGLE=$(od -t dI -j $SKIP -N 4 tmp2 | head -1 | awk '{print $2}')
  STARTING_ANGLE=`echo "scale=2; $STARTING_ANGLE/1000" | bc -l `
  echo STARTING_ANGLE= $STARTING_ANGLE 
                                                                                                     
   let SKIP=1024+256+128+256+4                                                                             
   let SKIP=1024+256+128+256+4                                                                             
   ORGX=$(od -t dI -j $SKIP -N 4 tmp2 | head -1 | awk '{print $2}')             
   ORGX=$(od -t dI -j $SKIP -N 4 tmp2 | head -1 | awk '{print $2}')             
Line 321: Line 372:
       BEAM_CENTER_X=`grep BEAM_CENTER_X tmp2 | sed s/BEAM_CENTER_X=//`
       BEAM_CENTER_X=`grep BEAM_CENTER_X tmp2 | sed s/BEAM_CENTER_X=//`
       BEAM_CENTER_Y=`grep BEAM_CENTER_Y tmp2 | sed s/BEAM_CENTER_Y=//`
       BEAM_CENTER_Y=`grep BEAM_CENTER_Y tmp2 | sed s/BEAM_CENTER_Y=//`
# fix 2010-04-26 - tell user about possible ORGX, ORGY alternatives -  
   
       COMMENT_ORGXY="
       COMMENT_ORGXY="
! Following are possible beam center interpretations for ADSC detectors"
! Following are possible beam center interpretations for ADSC detectors"
# at ESRF, PF, ALS 5.0.2 and ... (pls fill in!) the following should be used:         
# at ESRF, PF, ALS 5.0.2, AS MX2 and ... (pls fill in!) the following should be used:         
       ORGX1=`echo "scale=1; $BEAM_CENTER_Y/$QX" | bc -l`
       ORGX1=`echo "scale=1; $BEAM_CENTER_Y/$QX" | bc -l`
       ORGY1=`echo "scale=1; $BEAM_CENTER_X/$QX" | bc -l`
       ORGY1=`echo "scale=1; $BEAM_CENTER_X/$QX" | bc -l`
       echo - at ESRF, PF, APS Argonne BLs use: ORGX=$ORGX1 ORGY=$ORGY1                     
       echo - at ESRF, PF, ALS 8.2.1, APS Argonne BLs use: ORGX=$ORGX1 ORGY=$ORGY1                     
       COMMENT_ORGXY="${COMMENT_ORGXY}
       COMMENT_ORGXY="${COMMENT_ORGXY}
! ORGX= $ORGX1 ORGY= $ORGY1 ! For ESRF, PF, APS Argonne ..."
! ORGX= $ORGX1 ORGY= $ORGY1 ! For ESRF, PF, APS, AS MX2 ..."
# this 2nd alternative convention should be used at the following beamlines (pls complete the list): ALS 5.0.3, ...
# this 2nd alternative convention should be used at the following beamlines (pls complete the list): ALS 5.0.3, ...
       ORGX2=`echo "scale=1; $NX-$BEAM_CENTER_X/$QX" | bc -l `
       ORGX2=`echo "scale=1; $NX-$BEAM_CENTER_X/$QX" | bc -l `
Line 352: Line 403:
       # Decision of beam center convention based on detector serial numbers.
       # Decision of beam center convention based on detector serial numbers.
       DET_SN=`grep DETECTOR_SN tmp2 | sed -e "s/DETECTOR_SN=//"`
       DET_SN=`grep DETECTOR_SN tmp2 | sed -e "s/DETECTOR_SN=//"`
       # For convention 1; Known PF detectors = 449: NW12A Q210, 472: NE3A Q270, 474: BL17A Q270, 912: BL5A Q315, 923: ALS BL5.0.2 Q315, 933: AichiSR BL2S1 Q315, 916: APS 24 IDE
      echo Detector serial number is $DET_SN
       # For convention 1; Known PF detectors = 449: NW12A Q210, 472: NE3A Q270, 474: BL17A Q270, 912: BL5A Q315, 923: ALS BL5.0.2 Q315, 933: AichiSR BL2S1 Q315, 916: APS 24 IDE, 928: AS MX2
       ORG1_SNs="
       ORG1_SNs="
449
449
Line 363: Line 415:
446
446
916
916
905
928
"
"
       ORG4_SNs="
       ORG4_SNs="
Line 370: Line 424:
       ORGX=$ORGX1
       ORGX=$ORGX1
       ORGY=$ORGY1
       ORGY=$ORGY1
      echo the following was chosen based on detector serial number:
       elif echo "${DET_SN}${ORG4_SNs}" | sort | uniq -d | grep [0-9] > /dev/null; then
       elif echo "${DET_SN}${ORG4_SNs}" | sort | uniq -d | grep [0-9] > /dev/null; then
       ORGX=$ORGX4
       ORGX=$ORGX4
       ORGY=$ORGY4
       ORGY=$ORGY4
      echo the following was chosen based on detector serial number:
       else
       else
       ORGX=$ORGX3
       ORGX=$ORGX3
       ORGY=$ORGY3
       ORGY=$ORGY3
      echo the following default was chosen because the detector serial number was not special-cased:
       fi
       fi


       # Check detector serial number and recognize beamline for reversed-phi setting.
       # Check detector serial number and recognize beamline for reversed-phi setting.
       # Known detectors for reversed-phi in SPring-8: 915: BL38B1 Q315; APS 19-ID: 458; BM30A: 924
       # Known detectors for reversed-phi in SPring-8: 915: BL38B1 Q315; APS 19-ID: 458; BM30A: 924
      # 928 is at Australian Beamline MX2
      # revision 0.83 of this script removes 458 from the list!
       REVERSEPHI_SNs="
       REVERSEPHI_SNs="
915
915
458
924
924
928
"
"
       if echo "${DET_SN}${REVERSEPHI_SNs}" | sort | uniq -d | grep [0-9] > /dev/null; then
       if echo "${DET_SN}${REVERSEPHI_SNs}" | sort | uniq -d | grep [0-9] > /dev/null; then
         REVERSE_PHI="yes"
         REVERSE_PHI="yes"
        echo inverted ROTATION_AXIS since detector serial number is ${DET_SN}
       fi
       fi


Line 409: Line 469:
   DETECTOR_DISTANCE=`grep ^DISTANCE tmp2 | sed s/DISTANCE=/-/`                                                                     
   DETECTOR_DISTANCE=`grep ^DISTANCE tmp2 | sed s/DISTANCE=/-/`                                                                     
   OSCILLATION_RANGE=`grep OMEGA_DELTA tmp2 | sed s/OMEGA_DELTA=//`   
   OSCILLATION_RANGE=`grep OMEGA_DELTA tmp2 | sed s/OMEGA_DELTA=//`   
   DIRECTION_OF_DETECTOR_X_AXIS="-1 0 0"                                                                 
   DIRECTION_OF_DETECTOR_X_AXIS="-1 0 0"  
                                                                  
elif [ "$DET" == "experimental-ED" ]; then
  DETECTOR="ADSC MINIMUM_VALID_PIXEL_VALUE= 1 OVERLOAD= 65000  ! OVERLOAD is really unknown "
  echo --- Electron diffraction data in SMV format!
  SENSOR_THICKNESS=0.01                                                       
  sed s/\;// tmp2 > tmp1                                                     
  mv tmp1 tmp2                                                               
  X_RAY_WAVELENGTH=`grep WAVELENGTH tmp2 | head -1 | sed s/WAVELENGTH=//`
  NX=`grep SIZE1 tmp2 | tail -1 | sed s/SIZE1=//`
  QX=`grep PIXEL_SIZE tmp2 | sed s/PIXEL_SIZE=//`
  NY=`grep SIZE2 tmp2 | tail -1 | sed s/SIZE2=//`                                           
  QY=$QX 
  BEAM_CENTER_X=`grep BEAM_CENTER_X tmp2 | sed s/BEAM_CENTER_X=//`
  BEAM_CENTER_Y=`grep BEAM_CENTER_Y tmp2 | sed s/BEAM_CENTER_Y=//`
  QXY=`echo "scale=1; $QX*($BEAM_CENTER_X+$BEAM_CENTER_Y)" | bc -l`
      if [ "$QXY" "<" "$BEAM_CENTER_X" ] ; then
      ORGX=`echo "scale=2; $BEAM_CENTER_Y/$QX" | bc -l`
      ORGY=`echo "scale=2; $BEAM_CENTER_X/$QX" | bc -l`
      else
      ORGX=`echo "scale=2; $BEAM_CENTER_X/1" | bc -l`
      ORGY=`echo "scale=2; $BEAM_CENTER_Y/1" | bc -l`
      fi
  DETECTOR_DISTANCE=`grep ^DISTANCE tmp2 | sed s/DISTANCE=//`                                                                   
  OSCILLATION_RANGE=`grep OSC_RANGE tmp2 | sed s/OSC_RANGE=//`
  STARTING_ANGLE=`grep OSC_START tmp2 | sed s/OSC_START=//`
  echo --- For TIMEPIX detector, please fix ROTATION_AXIS yourself!
  DIRECTION_OF_DETECTOR_X_AXIS="1 0 0"
  REFINE_CORRECT="ORIENTATION CELL AXIS BEAM ! for ED, no POSITION when CELL is refined"


elif [ "$DET" == "pilatus" ]; then
elif [ "$DET" == "pilatus" ]; then
  DETECTOR="PILATUS MINIMUM_VALID_PIXEL_VALUE=0 OVERLOAD= 1048576  !PILATUS"
   echo Data from a Pilatus detector                                         
   echo Data from a Pilatus detector                                         
   sed s/#// tmp2 > tmp1                                                     
   sed s/#// tmp2 > tmp1                                                     
   mv tmp1 tmp2                                                               
   mv tmp1 tmp2                                                               
  OVERLOAD=1048574
  SEPMIN=4
  CLUSTER_RADIUS=2
  grep -q Count_cutoff tmp2 && OVERLOAD=`awk '/Count_cutoff/{print $2}' tmp2`
  DETECTOR="PILATUS MINIMUM_VALID_PIXEL_VALUE=0 OVERLOAD= $OVERLOAD  !PILATUS"
   QX=0.172 QY=0.172                                                         
   QX=0.172 QY=0.172                                                         
# the default above guards against missing Pixel_size line in CBF header
# the default above guards against missing Pixel_size line in CBF header
   grep Pixel_size tmp2 && QX=`awk '/Pixel_size/{print 1000*$2}' tmp2`
   grep -q Pixel_size tmp2 && QX=`awk '/Pixel_size/{print 1000*$2}' tmp2`
   grep -q Pixel_size tmp2 && QY=`awk '/Pixel_size/{print 1000*$5}' tmp2`                                                     
   grep -q Pixel_size tmp2 && QY=`awk '/Pixel_size/{print 1000*$5}' tmp2`                                                     


Line 434: Line 526:
       ORGY=`grep Beam_xy tmp2 | sed -e s/\(// -e s/\)// -e s/\,// | awk '{print $3}'`
       ORGY=`grep Beam_xy tmp2 | sed -e s/\(// -e s/\)// -e s/\,// | awk '{print $3}'`


       # find DETECTOR_DISTANCE and OSCILLATION_RANGE:
       # find DETECTOR_DISTANCE, OSCILLATION_RANGE, and STARTING_ANGLE:
       DETECTOR_DISTANCE=`awk '/distance/{print $2}' tmp2`
       DETECTOR_DISTANCE=`awk '/Detector_distance/{print $2}' tmp2`
       DETECTOR_DISTANCE=`echo "$DETECTOR_DISTANCE*1000" | bc -l`
       DETECTOR_DISTANCE=`echo "$DETECTOR_DISTANCE*1000" | bc -l`


       OSCILLATION_RANGE=`awk '/Angle/{print $2}' tmp2`
       OSCILLATION_RANGE=`awk '/Angle_increment/{print $2}' tmp2`
 
      STARTING_ANGLE=`awk '/Start_angle/{print $2}' tmp2`
      echo STARTING_ANGLE= $STARTING_ANGLE 


       # get detector serial number and check if it is included in the list
       # get detector serial number and check if it is included in the list
Line 445: Line 540:
       # Known detectors for reversed-phi at MX2 beamline (Brazilian Synchrotron National Laboratory - LNLS)
       # Known detectors for reversed-phi at MX2 beamline (Brazilian Synchrotron National Laboratory - LNLS)
       # Known detectors for reversed-phi at CHESS F1 PILATUS3 6M, S/N 60-0127
       # Known detectors for reversed-phi at CHESS F1 PILATUS3 6M, S/N 60-0127
      # Known detectors for reversed-phi at SSRF BL18U1 (S/N 60-0123) and BL19U1 (S/N XX-XXX) (!; 2019-10-19: staff will fix this)
       DET_SN=`grep "Detector:" tmp2 | sed "s/^.*Detector: *//"`
       DET_SN=`grep "Detector:" tmp2 | sed "s/^.*Detector: *//"`
       REVERSEPHI_SNs="
       REVERSEPHI_SNs="
Line 451: Line 547:
PILATUS 2M, S/N 24-0109
PILATUS 2M, S/N 24-0109
PILATUS3 6M, S/N 60-0127
PILATUS3 6M, S/N 60-0127
PILATUS3 6M, S/N 60-0123
"
"
       if echo "${DET_SN}${REVERSEPHI_SNs}" | sort | uniq -d | grep [0-9] > /dev/null; then
       if echo "${DET_SN}${REVERSEPHI_SNs}" | sort | uniq -d | grep [0-9] > /dev/null; then
         REVERSE_PHI="yes"
         REVERSE_PHI="yes"
        echo inverted ROTATION_AXIS since detector serial number is ${DET_SN}
      fi
      if [ "$DET_SN" == "PILATUS XXX, S/N XX-XXX" ] ; then
        REVERSE_PHI="yes"
        echo inverted rotation axis at SSRF BL19U1
       fi
       fi
        
        
Line 460: Line 562:
         if grep -q "Oscillation_axis X.CW +SLOW" tmp2 ; then
         if grep -q "Oscillation_axis X.CW +SLOW" tmp2 ; then
           rotation_axis="0 -1 0"
           rotation_axis="0 -1 0"
          echo ROTATION_AXIS="0 -1 0" at Diamond I24
         fi
         fi
       fi
       fi
# insert similar code for Petra P14 here
# PETRA P14: raw data from Eiger are stored as CBF files so this is treated as Pilatus
      if [ "$DET_SN" == "Dectris Eiger 16M, E-32-0107" -o "$DET_SN" == "Dectris Eiger 4M, E-08-0107" -o "$DET_SN" == "PILATUS 6M-F, S/N 60-0115-F" ] ; then
        rotation_axis="0 -1 0"
        echo ROTATION_AXIS="0 -1 0" at PETRA P14
      fi
# ESRF ID23-2:
      if [ "$DET_SN" == "PILATUS3 2M, S/N 24-0118, ESRF ID23" ] ; then
        rotation_axis="0 -1 0"
        echo ROTATION_AXIS="0 -1 0" at ESRF ID23-2
      fi
 
elif [ "$DET" == "eiger" ]; then
elif [ "$DET" == "eiger" ]; then
   OVERLOAD=`h5dump -d "/entry/instrument/detector/detectorSpecific/countrate_correction_count_cutoff" $FIRSTFRAME | awk '/\(0\):/{print $2}' `
nframes=`h5dump -d "/entry/instrument/detector/detectorSpecific/nimages" $FIRSTFRAME | awk '/\(0\): [0-9]/{print $2}'`
DATA_RANGE="1 $nframes"
SPOT_RANGE="1 `echo "scale=0; if (${nframes}<2) 1; if (${nframes}>1) ${nframes}/2"|bc -l`"
# find out if HDF5 from Diamond (DLS=1) or Dectris (DLS=0)
  DLS=0
  OVERLOAD=`h5dump -d "/entry/instrument/detector/detectorSpecific/countrate_correction_count_cutoff" $FIRSTFRAME 2>/dev/null` || DLS=1
  if [ "$DLS" == 1 ]; then
    echo Eiger HDF5 from Diamond
# unfortunately h5dump 1.10 is required to get this right for the DLS .h5 files. This version is available at DLS but maybe not elsewhere
    OVERLOAD=`h5dump -d "/entry/instrument/detector/saturation_value" $FIRSTFRAME | awk '/\(0\):/{print $2}'`
  # v0.95: fix the next 2 lines by taking care of negative values with the \- , and stop after first "(0)"
    OSCILLATION_RANGE=`h5dump -d "/entry/data/omega" $FIRSTFRAME | awk '/\(0\): [\-0-9]/{print $3-$2;exit}'`
    STARTING_ANGLE=`h5dump    -d "/entry/data/omega" $FIRSTFRAME | awk '/\(0\): [\-0-9]/{print $2;exit}' | sed -e "s/,//"`
    echo OSCILLATION_RANGE=$OSCILLATION_RANGE STARTING_ANGLE=$STARTING_ANGLE
  # rotation_axis=`h5dump -a "/entry/sample/transformations/omega/vector" $FIRSTFRAME 2>/dev/null | grep "(0):" | sed -e "s/^.*://; s/,//g"`
  # the above gives -1 0 0 for DLS data instead of the correct 1 0 0, so commented out for now
   else
    echo Eiger HDF5 from Dectris
    OVERLOAD=`h5dump -d "/entry/instrument/detector/detectorSpecific/countrate_correction_count_cutoff" $FIRSTFRAME | awk '/\(0\):/{print $2}'`  
    OSCILLATION_RANGE=`h5dump -d "/entry/sample/goniometer/omega_range_average" $FIRSTFRAME | awk '/\(0\): [0-9]/{print $2}'`
  # STARTING_ANGLE:  the \- was introduced in version 0.91 to allow negative values :
    STARTING_ANGLE=`h5dump -d "/entry/sample/goniometer/omega_start" $FIRSTFRAME | awk '/\(0\): [\-0-9]/{print $2}'`
  # /entry/sample/goniometer/omega_start is missing in some eiger2 detectors (e.g. Eiger2 9M with fw version release-2020.2.1 and SIMPLON API 1.8) (Feng YU 2021-07-18)
    if [ "$STARTING_ANGLE" == "" ]; then
  echo "/entry/sample/goniometer/omega_start not found, try /entry/sample/goniometer/omega"
  STARTING_ANGLE=`h5dump -d /entry/sample/goniometer/omega $FIRSTFRAME | grep "(0):" | head -n 1 | awk '{print $2}' | sed -e "s/,//g"`
    fi
    echo "STARTING_ANGLE=$STARTING_ANGLE"
  # If rotation vector set (NeXus)
    rotation_axis=`h5dump -a "/entry/sample/transformations/omega/vector" $FIRSTFRAME 2>/dev/null | grep "(0):" | sed -e "s/^.*://; s/,//g"`
  # EIGER2 16M CHESS ID7B2 has S/N E-32-0123 (A. Finke 2020-11-07) v0.99
    SN=`h5dump -d "/entry/instrument/detector/detector_number" $FIRSTFRAME | awk '/\(0\): /{print $2}' | sed s/\"//g`
    if [ "$SN" == "E-32-0123" ]; then
      rotation_axis="-1 0 0"
      echo CHESS ID7B2 with inverted rotation axis
    fi
  # SSRF BL17U1 and SSRF BL10U2 (Feng YU 2021-07-18)
  # Eiger X 16M (S/N E-32-0111) was installed at SSRF BL17U1 from Oct 2017 to Feb 2021. After Feb 2021, it was moved back to SSRF BL10U2.
# 2021-02-01 00:00:00 CST time stamp is 1612108800
    if [ "$SN" == "E-32-0111" ]; then
      collection_time=`h5dump -d "/entry/instrument/detector/detectorSpecific/data_collection_date" $FIRSTFRAME | grep "(0):" | awk '{print $2}' | sed -e "s/\.\(.*\)/CST/g; s/\"//g"`
      if [ `uname -s` == "Darwin" ]; then
        collection_timestamp=`date -j -f "%Y-%m-%dT%H:%M:%S%Z" $collection_time +%s`
      elif [ `uname -s` == "Linux" ]; then
        collection_timestamp=`date -d $collection_time +%s`
      else
        collection_timestamp=0
      fi
      if [ $collection_timestamp -eq 0 ]; then
        is_rotation_axis_set=1
        rotation_axis=`h5dump -d "/SSRF/RotationAxis" $FIRSTFRAME 2>/dev/null || is_rotation_axis_set=0`
        if [ "$is_rotation_axis_set" == "1" ]; then
          rotation_axis=`h5dump -d "/SSRF/RotationAxis" $FIRSTFRAME | grep "(0):" | sed -e "s/^.*://; s/,//g"`
          echo "The rotation axis of SSRF BL10U2 (Eiger X 16M) is $rotation_axis"
        else
          rotation_axis="-1 0 0    ! Cannot determine rotation axis. SSRF BL17U1: -1 0 0; SSRF BL02U1: 0 -1 0"
        fi
      elif [ $collection_timestamp -ge 0 ] && [ $collection_timestamp -le 1612108800 ]; then
        # SSRF BL17U1
        rotation_axis="-1 0 0"
        echo "SSRF BL17U1 (Eiger X 16M) with inverted rotation axis"
      else
        # SSRF BL10U2
        is_rotation_axis_set=1
        rotation_axis=`h5dump -d "/SSRF/RotationAxis" $FIRSTFRAME 2>/dev/null || is_rotation_axis_set=0`
        if [ "$is_rotation_axis_set" == "1" ]; then
          rotation_axis=`h5dump -d "/SSRF/RotationAxis" $FIRSTFRAME | grep "(0):" | sed -e "s/^.*://; s/,//g"`
          echo "The rotation axis of SSRF BL10U2 (Eiger X 16M) is $rotation_axis"
        else
          rotation_axis="0 -1 0"
          echo "SSRF BL10U2 (Eiger X 16M) with vertical rotation axis"
        fi
      fi
    fi
  # SSRF BL02U1 (Feng YU 2021-07-18)
  # EIGER2 S 9M SSRF BL02U1 has S/N E-18-0121
    if [ "$SN" == "E-18-0121" ]; then
      is_rotation_axis_set=1
      rotation_axis=`h5dump -d "/SSRF/RotationAxis" $FIRSTFRAME 2>/dev/null || is_rotation_axis_set=0`
      if [ "$is_rotation_axis_set" == "1" ]; then
        rotation_axis=`h5dump -d "/SSRF/RotationAxis" $FIRSTFRAME | grep "(0):" | sed -e "s/^.*://; s/,//g"`
        echo "The rotation axis of SSRF BL02U1 (Eiger2 S 9M) is $rotation_axis"
      else
        rotation_axis="1 0 0"
        echo "SSRF BL02U1 (Eiger2 S 9M) with horizontal rotation axis"
      fi
    fi
# revision 1.05 specialcase nframes for Eiger detectors at BNL
    if [ "$SN" == "E-18-0121" -o "$SN" == "E-32-0101" ]; then
      echo specialcase nframes for Eiger detectors at BNL:
      nframes=`h5dump -A -g "/entry/data" $FIRSTFRAME | grep "DATASPACE  SIMPLE" | sed -e "s/,.*//" | awk '{a+=$5}END{print a}'`
      DATA_RANGE="1 $nframes"
      SPOT_RANGE="1 `echo "scale=0; if (${nframes}<2) 1; if (${nframes}>1) ${nframes}/2"|bc -l`"
    fi
  fi
  echo DATA_RANGE=$DATA_RANGE
   DETECTOR="EIGER MINIMUM_VALID_PIXEL_VALUE=0 OVERLOAD= $OVERLOAD"
   DETECTOR="EIGER MINIMUM_VALID_PIXEL_VALUE=0 OVERLOAD= $OVERLOAD"
   QX=`h5dump -d "/entry/instrument/detector/x_pixel_size" $FIRSTFRAME | awk '/\(0\): [0-9]/{print $2*1000}'`
   QX=`h5dump -d "/entry/instrument/detector/x_pixel_size" $FIRSTFRAME | awk '/\(0\): [0-9]/{print $2*1000}'`
   QY=`h5dump -d "/entry/instrument/detector/y_pixel_size" $FIRSTFRAME | awk '/\(0\): [0-9]/{print $2*1000}'`
   QY=`h5dump -d "/entry/instrument/detector/y_pixel_size" $FIRSTFRAME | awk '/\(0\): [0-9]/{print $2*1000}'`


   echo Data from a Eiger hdf5
   echo OVERLOAD=$OVERLOAD
   SENSOR_THICKNESS=`h5dump -d "/entry/instrument/detector/sensor_thickness" $FIRSTFRAME | awk '/\(0\): [0-9]/{print $2*1000}'`
   SENSOR_THICKNESS=`h5dump -d "/entry/instrument/detector/sensor_thickness" $FIRSTFRAME | awk '/\(0\): [0-9]/{print $2*1000}'`
   X_RAY_WAVELENGTH=`h5dump -d "/entry/instrument/beam/incident_wavelength" $FIRSTFRAME | awk '/\(0\): [0-9]/{print $2}'`
   X_RAY_WAVELENGTH=`h5dump -d "/entry/instrument/beam/incident_wavelength" $FIRSTFRAME | awk '/\(0\): [0-9]/{print $2}'`
Line 480: Line 689:
   ORGY=`h5dump -d "/entry/instrument/detector/beam_center_y" $FIRSTFRAME | awk '/\(0\): [0-9]/{print $2}'`
   ORGY=`h5dump -d "/entry/instrument/detector/beam_center_y" $FIRSTFRAME | awk '/\(0\): [0-9]/{print $2}'`


   # find DETECTOR_DISTANCE and OSCILLATION_RANGE:
   # find DETECTOR_DISTANCE :
   DETECTOR_DISTANCE=`h5dump -d "/entry/instrument/detector/detector_distance" $FIRSTFRAME | awk '/\(0\): [0-9]/{print $2*1000}'`
   DETECTOR_DISTANCE=`h5dump -d "/entry/instrument/detector/detector_distance" $FIRSTFRAME | awk '/\(0\): [0-9]/{print $2*1000}'`
   OSCILLATION_RANGE=`h5dump -d "/entry/sample/goniometer/omega_range_average" $FIRSTFRAME | awk '/\(0\): [0-9]/{print $2}'`
 
   SEPMIN=4
  CLUSTER_RADIUS=2


  # If rotation vector set (NeXus)
  rotation_axis=`h5dump -a "/entry/sample/transformations/omega/vector" $FIRSTFRAME 2>/dev/null | grep "(0):" | sed -e "s/^.*://; s/,//g"`
 
elif [ "$DET" == "raxis" ]; then
elif [ "$DET" == "raxis" ]; then
   echo Data from a RAXIS detector
   echo Data from a RAXIS detector
Line 493: Line 701:
   #let SKIP=768
   #let SKIP=768
   #NX=$(od -t x -j $SKIP -N 4 $FIRSTFRAME |awk 'NR==1{print toupper($2)}'|perl -nle '@array= $_ =~/.{2}/g; print "ibase=16;obase=A;".join("",reverse @array)'|bc)
   #NX=$(od -t x -j $SKIP -N 4 $FIRSTFRAME |awk 'NR==1{print toupper($2)}'|perl -nle '@array= $_ =~/.{2}/g; print "ibase=16;obase=A;".join("",reverse @array)'|bc)
   NX=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(768);print "%.4d"%struct.unpack(">i",f.read(4))')
!  NX=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(768);print "%.4d"%struct.unpack(">i",f.read(4))')
   NY=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(772);print "%.4d"%struct.unpack(">i",f.read(4))')
!  NY=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(772);print "%.4d"%struct.unpack(">i",f.read(4))')
   NX=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(768);print("%.4d"%struct.unpack(">i",f.read(4)))')
   NY=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(772);print("%.4d"%struct.unpack(">i",f.read(4)))')


   DETECTOR_DISTANCE=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(344);print "-%.4f"%struct.unpack(">f",f.read(4))')
!  DETECTOR_DISTANCE=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(344);print "-%.4f"%struct.unpack(">f",f.read(4))')
   DETECTOR_DISTANCE=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(344);print("-%.4f"%struct.unpack(">f",f.read(4)))')


   ORGX=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(540);print "%.4f"%struct.unpack(">f",f.read(4))')
!  ORGX=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(540);print "%.4f"%struct.unpack(">f",f.read(4))')
   ORGY=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(544);print "%.4f"%struct.unpack(">f",f.read(4))')
!  ORGY=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(544);print "%.4f"%struct.unpack(">f",f.read(4))')
   ORGX=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(540);print("%.4f"%struct.unpack(">f",f.read(4)))')
   ORGY=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(544);print("%.4f"%struct.unpack(">f",f.read(4)))')


   OSCILLATION_RANGE=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(524);phis,phie=struct.unpack(">ff",f.read(8));print "%.4f"%(phie-phis)')
!  OSCILLATION_RANGE=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(524);phis,phie=struct.unpack(">ff",f.read(8));print "%.4f"%(phie-phis)')
   OSCILLATION_RANGE=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(524);phis,phie=struct.unpack(">ff",f.read(8));print("%.4f"%(phie-phis))')


   QX=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(776);print "%.6f"%struct.unpack(">f",f.read(4))')
!  QX=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(776);print "%.6f"%struct.unpack(">f",f.read(4))')
   QY=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(780);print "%.6f"%struct.unpack(">f",f.read(4))')
!  QY=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(780);print "%.6f"%struct.unpack(">f",f.read(4))')
   QX=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(776);print("%.6f"%struct.unpack(">f",f.read(4)))')
   QY=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(780);print("%.6f"%struct.unpack(">f",f.read(4)))')


   X_RAY_WAVELENGTH=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(292);print "%.6f"%struct.unpack(">f",f.read(4))')
!  X_RAY_WAVELENGTH=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(292);print "%.6f"%struct.unpack(">f",f.read(4))')
   X_RAY_WAVELENGTH=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(292);print("%.6f"%struct.unpack(">f",f.read(4)))')


elif [ "$DET" == "dtrek" ]; then
elif [ "$DET" == "dtrek" ]; then
   echo "Data from a RAXIS or Saturn detector with dTREK format"
   echo "Data from a RAXIS or Saturn or Pilatus detector with dTREK format"


   sed s/\;// tmp2 > tmp1
   sed s/\;// tmp2 > tmp1
Line 515: Line 732:


   dname=`grep "DETECTOR_NAMES=" tmp2 | sed -e "s/.*=//"`
   dname=`grep "DETECTOR_NAMES=" tmp2 | sed -e "s/.*=//"`
 
  flip=1
   if [ "$dname" == "CCD_" ]; then
   if [ "$dname" == "CCD_" ]; then
     DETECTOR="SATURN MINIMUM_VALID_PIXEL_VALUE=1"
     DETECTOR="SATURN MINIMUM_VALID_PIXEL_VALUE=1"
Line 532: Line 749:
     echo ""
     echo ""
   fi
   fi
   else # should be RX_
   elif [ "$dname" == "PILT_" ]; then
    DETECTOR="PILATUS MINIMUM_VALID_PIXEL_VALUE=0"
    dtrek_det="pilatus"
    SEPMIN=3
    CLUSTER_RADIUS=1.5
    rotation_axis="0 1 0"  # TODO: read from header
    flip=-1
    SENSOR_THICKNESS=0.45  # TODO: read from header
# 2theta
    TWOTHETA=`awk '/PILT_GONIO_VALUES=/{print $2}' tmp2`
    echo TWOTHETA=$TWOTHETA
    R1=`echo "scale=7;  c($TWOTHETA/$DEGTOR)" | bc -l`
    R3=`echo "scale=7; -s($TWOTHETA/$DEGTOR)" | bc -l`
    DIRECTION_OF_DETECTOR_X_AXIS="$R1 0 $R3"
  elif [ "$dname" == "RX_" ]; then
     DETECTOR="RAXIS MINIMUM_VALID_PIXEL_VALUE=0"
     DETECTOR="RAXIS MINIMUM_VALID_PIXEL_VALUE=0"
     dtrek_det="raxis"
     dtrek_det="raxis"
  else
    DETECTOR="XXX MINIMUM_VALID_PIXEL_VALUE=XXX"
    dtrek_det="unknown"
   fi
   fi
   # find OVERLOAD
   # find OVERLOAD
   SATURATED_VALUE=`grep SATURATED_VALUE tmp2 | head -1 | sed s/SATURATED_VALUE=//`
   SATURATED_VALUE=`grep SATURATED_VALUE tmp2 | head -1 | sed s/SATURATED_VALUE=//`
Line 566: Line 799:
   idx=$((i-1))
   idx=$((i-1))
   if [ "${GONIO_NAMES[$idx]}" == "Distance" ]; then
   if [ "${GONIO_NAMES[$idx]}" == "Distance" ]; then
    DETECTOR_DISTANCE="${GONIO_VALUES[$idx]}"
    # TODO: Check unit!! - ${GONIO_UNITS[$idx]}
  fi
  if [ $flip -gt 0 ]; then
     DETECTOR_DISTANCE="-${GONIO_VALUES[$idx]}"
     DETECTOR_DISTANCE="-${GONIO_VALUES[$idx]}"
     # TODO: Check unit!! - ${GONIO_UNITS[$idx]}
     # TODO: Check unit!! - ${GONIO_UNITS[$idx]}
    echo "using distance <0"
   fi
   fi
   done
   done
Line 580: Line 818:
  DETECTOR="MAR345  MINIMUM_VALID_PIXEL_VALUE=0  OVERLOAD=130000"
  DETECTOR="MAR345  MINIMUM_VALID_PIXEL_VALUE=0  OVERLOAD=130000"
  NX=`awk '/FORMAT/{print $2}' tmp2`
  NX=`awk '/FORMAT/{print $2}' tmp2`
  NY=$NX
# next line is rev 1.02 (previously NY= $NX):
  NY=`awk -v NX=$NX '/FORMAT/{print $4/NX}' tmp2`
  QX=`awk '/PIXEL/{print $3/1000.}' tmp2`
  QX=`awk '/PIXEL/{print $3/1000.}' tmp2`
  QY=`awk '/PIXEL/{print $5/1000.}' tmp2`
  QY=`awk '/PIXEL/{print $5/1000.}' tmp2`
Line 595: Line 834:
  X_RAY_WAVELENGTH=`grep 'WAVELENGTH' tmp2 | awk '{print $2}'`  
  X_RAY_WAVELENGTH=`grep 'WAVELENGTH' tmp2 | awk '{print $2}'`  
  OSCILLATION_RANGE=`grep 'PHI' tmp2 | awk '{print $5-$3}'`     
  OSCILLATION_RANGE=`grep 'PHI' tmp2 | awk '{print $5-$3}'`     
  TRUSTED_REGION="0 0.99"         
  TRUSTED_REGION="0 0.99"  
 
elif [ "$DET" == "OLDMAR" ]; then
 
echo  "Data from old type MAR image plate detector"
DETECTOR="MAR  MINIMUM_VALID_PIXEL_VALUE=0  OVERLOAD=130000"
NX=`awk 'NR==2 {print $2}' tmp2`
NY=$NX
QX=`awk 'NR==2 {print $15}' tmp2`
QY=$QX
ORGX=`awk 'NR==2 {print $19}' tmp2`
ORGY=`awk 'NR==2 {print $20}' tmp2`
DETECTOR_DISTANCE=`awk 'NR==2 {print $22}' tmp2`             
X_RAY_WAVELENGTH=`awk 'NR==2 {print $21}' tmp2`
OSCILLATION_RANGE=`awk 'NR==2 {print $24-$23}' tmp2`   
TRUSTED_REGION="0 0.99" 
rotation_axis="0 1 0" 
echo unsure if sign of anomalous signal is correct - please verify or try both hands!
        
elif [ "$DET" == "Bruker-cbf" ]; then
elif [ "$DET" == "Bruker-cbf" ]; then
    
    
  echo  "Data from a Bruker-cbf detector"
  echo  "Data from a Bruker-cbf detector"
# conversion radian / degrees:
DEGTOR=57.2957795
# MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT:
# MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT:
  MNOPIAS=6  
  MNOPIAS=6  
Line 626: Line 881:


  X_RAY_WAVELENGTH=`awk '/diffrn_radiation_wavelength.wavelength/{print $2}' tmp2`  
  X_RAY_WAVELENGTH=`awk '/diffrn_radiation_wavelength.wavelength/{print $2}' tmp2`  
  OMEGA=`awk '/OMEGA \? \? \?/{print $5}' tmp2`
# fix 16.3.20: instead of print, use printf "%.5f", because bc does not accept e.g. 3.1e-005
  DELTAOMEGA=`awk '/OMEGA \? \? \?/{print $6}' tmp2`
  OMEGA=`awk '/OMEGA \? \? \?/{printf "%.5f",$5}' tmp2`
  PHI=`awk '/PHI \? \? \?/{print $5}' tmp2`
  DELTAOMEGA=`awk '/OMEGA \? \? \?/{printf "%.5f",$6}' tmp2`
  DELTAPHI=`awk '/PHI \? \? \?/{printf "%5.4f",$6}' tmp2`
  PHI=`awk '/PHI \? \? \?/{printf "%.5f",$5}' tmp2`
  KAPPA=`awk '/CHI \? \? \?/{print $5}' tmp2`
  DELTAPHI=`awk '/PHI \? \? \?/{printf "%.5f",$6}' tmp2`
echo OMEGA DELTAOMEGA PHI DELTAPHI KAPPA= $OMEGA $DELTAOMEGA $PHI $DELTAPHI $KAPPA
  KAPPA=`awk '/CHI \? \? \?/{printf "%.5f",$5}' tmp2`
# echo OMEGA DELTAOMEGA PHI DELTAPHI KAPPA= $OMEGA $DELTAOMEGA $PHI $DELTAPHI $KAPPA
 
# test whether the absolute value of deltaphi is > absolute value of deltaomega
# test whether the absolute value of deltaphi is > absolute value of deltaomega
  if (( $(echo "${DELTAPHI}^2 > ${DELTAOMEGA}^2"|bc -l) )); then
  if (( $(echo "${DELTAPHI}^2 > ${DELTAOMEGA}^2"|bc -l) )); then
Line 640: Line 897:
     rotation_axis="$R1 $R2 $R3"
     rotation_axis="$R1 $R2 $R3"
     OSCILLATION_RANGE=${DELTAPHI}
     OSCILLATION_RANGE=${DELTAPHI}
    STARTING_ANGLE=${PHI}
# here we could check if DELTAPHI is <0, and if so, negate it and rotation_axis
# here we could check if DELTAPHI is <0, and if so, negate it and rotation_axis
  else
  else
Line 645: Line 903:
     rotation_axis="0 -1 0"
     rotation_axis="0 -1 0"
     OSCILLATION_RANGE=${DELTAOMEGA}
     OSCILLATION_RANGE=${DELTAOMEGA}
    STARTING_ANGLE=${OMEGA}
# here we could check if DELTAOMEGA is <0, and if so, negate it and rotation_axis
# here we could check if DELTAOMEGA is <0, and if so, negate it and rotation_axis
  fi
  fi
echo STARTING_ANGLE= $STARTING_ANGLE        ! only read by IDXREF               
# 2theta
# 2theta
   TWOTHETA=`awk '/TWOTHETA \? \? \?/{print $5}' tmp2`
   TWOTHETA=`awk '/TWOTHETA \? \? \?/{print $5}' tmp2`
Line 661: Line 922:
echo ORGX= $ORGX ORGY= $ORGY - check these values with adxv !
echo ORGX= $ORGX ORGY= $ORGY - check these values with adxv !
echo DETECTOR_DISTANCE= $DETECTOR_DISTANCE  ! only read by XYCORR, IDXREF                     
echo DETECTOR_DISTANCE= $DETECTOR_DISTANCE  ! only read by XYCORR, IDXREF                     
echo OSCILLATION_RANGE= $OSCILLATION_RANGE  ! only read by IDXREF                
echo OSCILLATION_RANGE= $OSCILLATION_RANGE  ! only read by IDXREF  
echo X-RAY_WAVELENGTH= $X_RAY_WAVELENGTH    ! only read by IDXREF                   
echo X-RAY_WAVELENGTH= $X_RAY_WAVELENGTH    ! only read by IDXREF                   


Line 672: Line 933:
$COMMENT_ORGXY
$COMMENT_ORGXY
DETECTOR_DISTANCE= $DETECTOR_DISTANCE !read by XYCORR, IDXREF. Negative if detector normal points to crystal.               
DETECTOR_DISTANCE= $DETECTOR_DISTANCE !read by XYCORR, IDXREF. Negative if detector normal points to crystal.               
OSCILLATION_RANGE= $OSCILLATION_RANGE                  
OSCILLATION_RANGE= $OSCILLATION_RANGE  
STARTING_ANGLE= $STARTING_ANGLE               
X-RAY_WAVELENGTH= $X_RAY_WAVELENGTH                       
X-RAY_WAVELENGTH= $X_RAY_WAVELENGTH                       
NAME_TEMPLATE_OF_DATA_FRAMES=$NAME_TEMPLATE_OF_DATA_FRAMES
NAME_TEMPLATE_OF_DATA_FRAMES=$NAME_TEMPLATE_OF_DATA_FRAMES
Line 704: Line 966:
STRONG_PIXEL=4          ! COLSPOT: only use strong reflections (default is 3)
STRONG_PIXEL=4          ! COLSPOT: only use strong reflections (default is 3)
MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT=$MNOPIAS ! default of 6 is sometimes too high
MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT=$MNOPIAS ! default of 6 is sometimes too high
! close spots/long cell axis: reduce SEPMIN and CLUSTER_RADIUS from their defaults of 6 and 3
! close spots/long cell axis: reduce SEPMIN and CLUSTER_RADIUS from their defaults of 7 and 3.5
! SEPMIN=4 CLUSTER_RADIUS=2 ! should be default for Pilatus and other detectors with low PSF
SEPMIN=$SEPMIN CLUSTER_RADIUS=$CLUSTER_RADIUS ! 4 and 2 for Pixel Array Detectors
! since XDS 01-MAR-2015, POSITION supersedes DISTANCE. DISTANCE only for compatibility w/ old XDS.
! since XDS 01-MAR-2015, POSITION supersedes DISTANCE.
! nowadays headers are usually correct so refine DISTANCE/POSITION in INTEGRATE but not IDXREF
! nowadays headers are usually correct so refine POSITION in INTEGRATE but not IDXREF if low to medium resolution
REFINE(IDXREF)=CELL BEAM ORIENTATION AXIS POSITION ! works best with XDS >= BUILT=20170720
! however, if the spots from COLSPOT extend to 2A then POSITION could, and if 1.5A POSITION should be refined
REFINE(INTEGRATE)= DISTANCE POSITION BEAM ORIENTATION ! AXIS CELL
REFINE(IDXREF)=CELL BEAM ORIENTATION AXIS  ! add POSITION if high resolution, or DETECTOR_DISTANCE inaccurate
! REFINE(CORRECT)=CELL BEAM ORIENTATION AXIS DISTANCE POSITION ! Default is: refine everything
REFINE(INTEGRATE)= POSITION BEAM ORIENTATION ! AXIS CELL . If 1.5A or higher it is ok to refine CELL (unless electron diffraction)
 
REFINE(CORRECT)= $REFINE_CORRECT
! parameters specifically for this detector and beamline:
! parameters specifically for this detector and beamline:
DETECTOR= $DETECTOR
DETECTOR= $DETECTOR
Line 723: Line 985:


if [ "$DET" == "eiger" ] && [ "$is_h5" == 1 ]; then
if [ "$DET" == "eiger" ] && [ "$is_h5" == 1 ]; then
   if [ -e /usr/local/lib64/dectris-neggia.so ]; then
   if [ "$DLS" == 0 ] ; then
    echo LIB=/usr/local/lib64/dectris-neggia.so >> XDS.INP
    if [ -e /usr/local/lib64/dectris-neggia.so ]; then
    echo LIB= line was written to XDS.INP
      echo LIB=/usr/local/lib64/dectris-neggia.so >> XDS.INP
      echo LIB= line was written to XDS.INP . For Apple ARM64 processors, you must modify the name.
    else
      echo !LIB=/usr/local/lib64/dectris-neggia.so >> XDS.INP
      echo /usr/local/lib64/dectris-neggia.so was not found - specify location manually!
    fi
   else
   else
     echo !LIB=/usr/local/lib64/dectris-neggia.so >> XDS.INP
     if [ -e /usr/local/lib64/durin-plugin.so ]; then
    echo /usr/local/lib64/dectris-neggia.so was not found!
      echo LIB=/usr/local/lib64/durin-plugin.so >> XDS.INP
   fi
      echo LIB= line was written to XDS.INP
    else
      echo !LIB=/usr/local/lib64/durin-plugin.so >> XDS.INP
      echo /usr/local/lib64/durin-plugin.so was not found - specify location manually!
    fi
   fi  
fi  
fi  


Line 745: Line 1,017:
   echo "ROTATION_AXIS= $rotation_axis  ! only read by IDXREF" >> XDS.INP
   echo "ROTATION_AXIS= $rotation_axis  ! only read by IDXREF" >> XDS.INP
  elif [ "$REVERSE_PHI" == "no" ]; then
  elif [ "$REVERSE_PHI" == "no" ]; then
   echo 'ROTATION_AXIS=1 0 0  ! Australian Synchrotron, SERCAT ID-22 (?), APS 19-ID (?), ESRF BM30A, SPring-8, SSRF BL17U need -1 0 0. Diamond ID24 needs 0 -1 0' >> XDS.INP
   echo 'ROTATION_AXIS=1 0 0  ! Australian Synchrotron, SERCAT ID-22 (?), APS 19-ID (?), ESRF BM30A, SPring-8, SSRF need -1 0 0. Diamond ID24 needs 0 -1 0' >> XDS.INP
  else
  else
   echo 'ROTATION_AXIS=-1 0 0  ! if this is wrong, please contact author.' >> XDS.INP
   echo 'ROTATION_AXIS=-1 0 0  ! if this is wrong, please contact author.' >> XDS.INP
Line 789: Line 1,061:
elif [ "$DET" == "pilatus" -o "$DET" == "eiger" ]; then
elif [ "$DET" == "pilatus" -o "$DET" == "eiger" ]; then
cat >> XDS.INP << eof
cat >> XDS.INP << eof
SEPMIN=4 CLUSTER_RADIUS=2
NUMBER_OF_PROFILE_GRID_POINTS_ALONG_ALPHA/BETA=13 ! Default is 9 - Increasing may improve data  
NUMBER_OF_PROFILE_GRID_POINTS_ALONG_ALPHA/BETA=13 ! Default is 9 - Increasing may improve data  
NUMBER_OF_PROFILE_GRID_POINTS_ALONG_GAMMA=13      ! accuracy, particularly if finely-sliced on phi,  
NUMBER_OF_PROFILE_GRID_POINTS_ALONG_GAMMA=13      ! accuracy, particularly if finely-sliced on phi,  
!                                                  and does not seem to have any downsides.  
!                                                  and does not seem to have any downsides.  
eof
eof
   if [ $NX == "1475" ]; then
   if [ "$NX" == "1028"  -a "$NY" == "1062" ]; then
# Eiger2 1M ; v0.97 numbers from Andreas Förster
    cat >> XDS.INP << eof
UNTRUSTED_RECTANGLE=    0 1029    512  551
eof
  elif [ "$NX" == "1475" ]; then
     if ! grep -q Flat_field tmp2 ; then
     if ! grep -q Flat_field tmp2 ; then
     cat >> XDS.INP << eof
     cat >> XDS.INP << eof
Line 827: Line 1,103:
eof
eof
     fi
     fi
   elif [ $NX == "2463" ]; then
   elif [ "$NX" == "2068"  -a "$NY" == "2162" ]; then
# Eiger2 4M ; v0.97 numbers from Andreas Förster
    cat >> XDS.INP << eof
!EXCLUSION OF VERTICAL DEAD AREAS OF THE EIGER2 4M DETECTOR
UNTRUSTED_RECTANGLE= 1028 1041      0 2163
!EXCLUSION OF HORIZONTAL DEAD AREAS OF THE EIGER 4M DETECTOR
UNTRUSTED_RECTANGLE=    0 2069    512  551
UNTRUSTED_RECTANGLE=    0 2069  1062 1101
UNTRUSTED_RECTANGLE=    0 2069  1612 1651
eof
  elif [ "$NX" == "2463" ]; then
# Pilatus 6M
# Pilatus 6M
# FIXME: here we could test if a Flat_field correction was applied like we do for 2M
# FIXME: here we could test if a Flat_field correction was applied like we do for 2M
Line 847: Line 1,133:
UNTRUSTED_RECTANGLE=  0 2464  2315 2333
UNTRUSTED_RECTANGLE=  0 2464  2315 2333
eof
eof
   elif [ $NX == "3110"  -a $NY == "3269" ]; then
   elif [ "$NX" == "3110"  -a "$NY" == "3269" ]; then
# Eiger 9M
# Eiger 9M
     cat >> XDS.INP << eof
     cat >> XDS.INP << eof
Line 860: Line 1,146:
UNTRUSTED_RECTANGLE= 0 3110 2717 2757
UNTRUSTED_RECTANGLE= 0 3110 2717 2757
eof
eof
   elif [ $NX == "4150" -a $NY == "4371" ]; then
   elif [ "$NX" == "3108"  -a "$NY" == "3262" ]; then
# Eiger2 9M ; v0.97 numbers from Andreas Förster
    cat >> XDS.INP << eof
!EXCLUSION OF VERTICAL DEAD AREAS OF THE EIGER2 9M DETECTOR
UNTRUSTED_RECTANGLE= 1028 1041      0 3262
UNTRUSTED_RECTANGLE= 2068 2081      0 3263
!EXCLUSION OF HORIZONTAL DEAD AREAS OF THE EIGER2 9M DETECTOR
UNTRUSTED_RECTANGLE=    0 3109    512  551
UNTRUSTED_RECTANGLE=    0 3109  1062 1101
UNTRUSTED_RECTANGLE=    0 3109  1612 1651
UNTRUSTED_RECTANGLE=    0 3109  2162 2201
UNTRUSTED_RECTANGLE=    0 3109  2712 2751
eof
  elif [ "$NX" == "4150" -a "$NY" == "4371" ]; then
# Eiger 16M
# Eiger 16M
     cat >> XDS.INP << eof
     cat >> XDS.INP << eof
Line 875: Line 1,174:
  UNTRUSTED_RECTANGLE= 2069 2082      0 4371  
  UNTRUSTED_RECTANGLE= 2069 2082      0 4371  
  UNTRUSTED_RECTANGLE= 3109 3122      0 4371
  UNTRUSTED_RECTANGLE= 3109 3122      0 4371
eof
  elif [ "$NX" == "4148" -a "$NY" == "4362" ]; then
# Eiger2 16M ; v0.97 numbers from Andreas Förster
    cat >> XDS.INP << eof
!EXCLUSION OF HORIZONTAL DEAD AREAS OF THE EIGER2 16M DETECTOR
UNTRUSTED_RECTANGLE= 1028 1041      0 4363
UNTRUSTED_RECTANGLE= 2068 2081      0 4363
UNTRUSTED_RECTANGLE= 3108 3121      0 4363
!EXCLUSION OF VERTICAL DEAD AREAS OF THE EIGER2 16M DETECTOR
UNTRUSTED_RECTANGLE=    0 4149    512  551
UNTRUSTED_RECTANGLE=    0 4149  1062 1101
UNTRUSTED_RECTANGLE=    0 4149  1612 1651
UNTRUSTED_RECTANGLE=    0 4149  2162 2201
UNTRUSTED_RECTANGLE=    0 4149  2712 2751
UNTRUSTED_RECTANGLE=    0 4149  3262 3301
UNTRUSTED_RECTANGLE=    0 4149  3812 3851
eof
eof
   fi
   fi
fi
fi
echo XDS.INP is ready for use. The file has only the most important keywords.
echo XDS.INP is ready for use. The file has only the most important keywords.
echo Full documentation, including complete detector templates, is at
echo Full documentation, including complete detector templates, at xds.mr.mpg.de .
echo http://www.mpimf-heidelberg.mpg.de/~kabsch/xds . More documentation in XDSwiki
echo More documentation in strucbio.biologie.uni-konstanz.de/xdswiki/index.php .
echo After running xds, inspect, using XDS-Viewer, at least the beamstop mask in
echo After running xds, inspect at least the agreement of predicted and observed  
echo BKGPIX.cbf, and the agreement of predicted and observed spots in FRAME.cbf!
echo spots in FRAME.cbf!
rm -f tmp1 tmp2
rm -f tmp1 tmp2
# end of generate_XDS.INP
 
</pre>
</pre>


Line 897: Line 1,212:
By using your own file, you can easily update to the latest revision, or even change the script, without having to bother the system administrator.
By using your own file, you can easily update to the latest revision, or even change the script, without having to bother the system administrator.


On Mac OS X, installation of the "Command Line Tools" (from http://developer.apple.com/downloads; requires Apple ID) is required. These are also part of the (larger, but also free) [http://developer.apple.com/tools/xcode Xcode] package. This package comes with a license that has to be accepted by the user when running a Command Line Tool for the first time.
See also [[Generate_XDS.INP#Dependencies]] below, and the [[Installation]] article.


See also the [[Installation]] article.
== Copying generate_XDS.INP from XDSwiki webserver ==
On Linux:
wget https://strucbio.biologie.uni-konstanz.de/pub/linux_bin/generate_XDS.INP
chmod a+x generate_XDS.INP
On Mac:
curl -o generate_XDS.INP https://strucbio.biologie.uni-konstanz.de/pub/linux_bin/generate_XDS.INP
chmod a+x generate_XDS.INP
See also [[Installation]].


== Obtaining generate_XDS.INP from this webpage ==
== Obtaining generate_XDS.INP from this webpage ==
Line 935: Line 1,257:
The script makes use of many GNU commands, like <code>ls, grep, egrep, awk, cut, cat, echo, wc, bc, head, sed, tail, cp, od, python</code>. Some of them (like <code>od</code> and <code>python</code>) are only used in case of specific detectors (MarCCD and RAXIS, respectively).  
The script makes use of many GNU commands, like <code>ls, grep, egrep, awk, cut, cat, echo, wc, bc, head, sed, tail, cp, od, python</code>. Some of them (like <code>od</code> and <code>python</code>) are only used in case of specific detectors (MarCCD and RAXIS, respectively).  
The script will only work if all the required commands are available. They reside in either the <code>coreutils</code> RPM, or specific RPMs (<code>gawk, sed, bc, grep, python</code> ...). Please note that to get the <code>strings</code> command on some Linux distributions (e.g. FC23) you need to install the <code>binutils</code> RPM package.
The script will only work if all the required commands are available. They reside in either the <code>coreutils</code> RPM, or specific RPMs (<code>gawk, sed, bc, grep, python</code> ...). Please note that to get the <code>strings</code> command on some Linux distributions (e.g. FC23) you need to install the <code>binutils</code> RPM package.
For Eiger data processing, the <code>h5dump</code> program must be installed. This is part of <code>hdf5-tools</code> (Ubuntu) or <code>hdf5</code> (RHEL).
For Eiger data processing, the <code>h5dump</code> program must be installed. This is part of <code>hdf5-tools</code> (Ubuntu) or <code>hdf5</code> (RHEL). The .h5 files collected at Diamond Light Source require a very new version of h5dump (namely h5dump 1.10) to extract the OVERLOAD parameter from the .h5 file; this version is available by default in Ubuntu 20.04 and RHEL/CentOS 8.


An easy way to check for missing programs is
On Mac OS X, installation of the "Command Line Tools" (from http://developer.apple.com/downloads; requires Apple ID) is required (open a terminal and type <code>xcode-select --install</code>). These are also part of the (larger, but also free) [http://developer.apple.com/tools/xcode Xcode] package. This package comes with a license that has to be accepted by the user when running a Command Line Tool (e.g. <code>strings</code>) for the first time.
  which ls grep egrep awk cut cat echo wc bc head sed tail cp od python strings h5dump
 
One way to check for missing programs is
  #!/bin/bash
for i in  ls grep egrep awk cut cat echo wc bc head sed tail cp od python strings h5dump ; do
    if [ ! -x /bin/$i ] && [ ! -x /usr/bin/$i ]; then
      echo $i not found
    fi
done


A command that should go a long way in providing all these tools for RedHat-derived distros is (as root)
A command that should go a long way in providing all these tools for RedHat-derived distros is (as root)
Line 944: Line 1,273:
and for Ubuntu this would be (untested!)
and for Ubuntu this would be (untested!)
  sudo apt-get install coreutils binutils gawk sed bc grep python hdf5-tools
  sudo apt-get install coreutils binutils gawk sed bc grep python hdf5-tools
See also [[Installation]].


== Limitations ==
== Limitations ==


* The script tries to interpret the header of the frames, so is currently limited to Dectris (Pilatus, Eiger), ADSC (Quantum), Rigaku (several types), MAR (CCD and image plate) detectors, and one Bruker detector. Other detectors need some values to be manually filled into XDS.INP - the relevant places are marked with XXX. These are detector properties (type, pixel size and number, min and max counts in a pixel), and experimental parameters like oscillation range, wavelength, distance, and direct beam position (or rather: point of detector that is closest to the crystal). For fine-tuning of detector parameters, see the [http://xds.mpimf-heidelberg.mpg.de/html_doc/xds_prepare.html detector-specific templates].
* The script tries to interpret the header of the frames, so is currently limited to Dectris (Pilatus, Eiger), ADSC (Quantum), Rigaku (several types), MAR (CCD and image plate) detectors, and one Bruker detector. Other detectors need some values to be manually filled into XDS.INP - the relevant places are marked with XXX. These are detector properties (type, pixel size and number, min and max counts in a pixel), and experimental parameters like ROTATION_AXIS, OSCILLATION_RANGE, X-RAY_WAVELENGTH, DETECTOR_DISTANCE, and XORG, YORG. For fine-tuning of detector parameters, see the [http://xds.mpimf-heidelberg.mpg.de/html_doc/xds_prepare.html detector-specific templates].
* The authors have made a "best effort" to provide a XDS.INP that results in the correct sign of the anomalous signal. In the case of one detector type (internally called Rigaku SMV) this requires reversal of one detector axis, and a negative DETECTOR_DISTANCE, as is found in some of the [http://xds.mpimf-heidelberg.mpg.de/html_doc/xds_prepare.html detector-specific templates]. '''For an unusual or unknown detector setup, the correct sign of the anomalous signal needs to be established and verified e.g. with a good dataset from a test crystal that has a anomalous signal.''' The authors do not take any responsibility for problems arising from incorrect sign of the anomalous signal, nor - obviously! - for any other mischief arising in or from data processing.
* The authors have made a "best effort" to provide a XDS.INP that results in the correct sign of the anomalous signal. In the case of one detector type (internally called Rigaku SMV) this requires reversal of one detector axis, and a negative DETECTOR_DISTANCE, as is found in some of the [http://xds.mpimf-heidelberg.mpg.de/html_doc/xds_prepare.html detector-specific templates]. '''For an unusual or unknown detector setup, the correct sign of the anomalous signal needs to be established and verified e.g. with a good dataset from a test crystal that has an anomalous signal.''' The authors do not take any responsibility for problems arising from incorrect sign of the anomalous signal, nor - obviously! - for any other mischief arising in or from data processing.
* At some beamlines, the ROTATION_AXIS should be -1 0 0 ("backwards") instead of the usual 1 0 0 ("horizontal"), or even 0 1 0 ("vertical") like at one of the PETRA Hamburg BLs. We have only just started the article [[Beamline notes]], and the frame headers do not have this information, so the default chosen by [[generate_XDS.INP]] may be wrong and need manual correction.
* At some beamlines, the ROTATION_AXIS should be -1 0 0 ("backwards") instead of the usual 1 0 0 ("horizontal"), or even 0 1 0 ("vertical") like at one of the PETRA Hamburg BLs. The frame headers do not have this information, so the default chosen by [[generate_XDS.INP]] may be wrong and need manual correction. Pls also see the article [[Beamline notes]].
* Sometimes, the x- and y- values of the primary beam position recorded in the header should be used for ORGY and ORGX (i.e reversed) instead of as ORGX and ORGY. For ADSC, this has been implemented in the script for a number of beamlines.
* Sometimes, the x- and y- values of the primary beam position recorded in the header should be used for ORGY and ORGX (i.e reversed) instead of as ORGX and ORGY. For ADSC, this has been implemented in the script for a number of beamlines.
* there are apparently several flavours of HDF5 files produced at Diamond Light Source. They differ e.g. in the naming of the header items. This means that items like NX, NY, DETECTOR_DISTANCE and number of images cannot be found by the [[generate_XDS.INP]] script. Example: The data at /dls/i04-1/data/2021/mx28114-9/processing/Lenye_Diamini/ThiL/ThiL found during the CCP4 2021 online Cape Town workshop. A workaround is to use e.g. xia2 pipeline=3dii to process these files, and - if needed - extract those items from its output files, e.g. from DEFAULT/NATIVE/SWEEP1/index/XDS.INP .


== See also ==
== See also ==
Line 956: Line 1,288:
* GlobalPhasing's list of beamlines and their settings is at http://www.globalphasing.com/autoproc/wiki/index.cgi?BeamlineSettings .
* GlobalPhasing's list of beamlines and their settings is at http://www.globalphasing.com/autoproc/wiki/index.cgi?BeamlineSettings .
* [[Beamline notes]] to collect information about specific settings
* [[Beamline notes]] to collect information about specific settings
* [[spot2pdb]] for visualizing reciprocal space
2,684

edits