Generate XDS.INP: Difference between revisions

Revision as of 17:25, 7 March 2022

This script generates XDS.INP based on a list of frame names supplied on the commandline. It currently works for MarCCD, ADSC, Pilatus, Eiger, some Rigaku and one Bruker detector(s); since this is just a bash script, extension to other detectors should be very easy.

Usage

Usage is just (don't forget the quotation marks!):

generate_XDS.INP "/home/myname/frms/mydata_1_???.img"

XDS supports bzip2-ed frames. Thus, when specifying the frame name parameter of the script, you should leave out any .bz2 extension.

For improved interaction with XDSGUI, it is advantageous to provide an absolute filename for the data files - one that starts with a slash ("/").

The script

!/bin/bash
purpose: generate XDS.INP
tested with some datasets from ALS, SSRL, SLS, ESRF, BESSY, SPring-8 and PF; only MAR, ADSC/SMV, PILATUS, Eiger, RAXIS (in-house), Bruker (PHOTON II) detectors;
for other detectors, values marked with XXX must be manually filled in.
revision 0.03 . Kay Diederichs 2/2010
revision 0.04 . Kay Diederichs 4/2010 - include alternative ORGX, ORGY calculations for ADSC
revision 0.05 . Kay Diederichs 5/2010 - grep for "Corrected" in addition to "marccd"; needed for BESSY
revision 0.06 . KD 6/2010 - add UNTRUSTED_RECTANGLE and UNTRUSTED_ELLIPSE; use `whereis catmar` and so on
revision 0.07 . KD 6/2010 - decide about ORGX/Y info in MAR header being pixels or mm; other fixes
revision 0.08 . KD 6/2010 - fixes for Pilatus 6M
revision 0.09 . KD 6/2010 - get rid of requirement for mccd_xdsparams.pl and/or catmar; rather use "od"
revision 0.10 . Tim Gruene 7/2010 - set link 'images' to image directory if path exceeds 72 characters
revision 0.11 . KD 7/2010 - for MarCCD: look for distance info at different byte position
revision 0.12 . KD 7/2010 - fix for negative PHISTART
revision 0.13 . KD 8/2010 - store correct NX NY QX QY in XDS.INP
revision 0.14 . KD 1/2011 - SENSOR_THICKNESS for Pilatus; MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT=3
revision 0.15 . KD 2/2011 - add comment for -ive sign of APS 19-ID and Australian Synchrotron rotation axis
revision 0.16 . KD 3/2011 - SENSOR_THICKNESS=0.01 for ADSC and MarCCD. Add comment about SILICON=
revision 0.17 . KD 3/2011 - make it work for .bz2 frames; improve screen output
revision 0.18 . KD 4/2011 - faster by doing "strings" only once; revert "images/${1##/*/}" "correction"
revision 0.19 . KD 6/2011 - bugfix for 0.18
revision 0.20 . KD 7/2011 - redirect stderr of /bin/ls to /dev/null
revision 0.21 . KD 11/2011 - SEPMIN, CLUSTER_RADIUS hints; read NX NY from header (for Pilatus 2M)
revision 0.22 . KD 12/2011 - Pilatus 2M UNTRUSTED_RECTANGLE lines, SENSOR_THICKNESS from header
revision 0.23 . KD 1/2012 - add UNTRUSTED_QUADRILATERAL, remove MINIMUM_ZETA (0.05 is default now)
revision 0.24 . KD 3/2012 - remove revision 0.10 since XDS now takes much longer paths
revision 0.25 . KD 3/2012 - remove revision 0.22 for PSI Pilatus 2M; see http://www.globalphasing.com/autoproc/wiki/index.cgi?TroubleShootingKnownIssues
revision 0.26 . KD 7/2012 - Mac-compatibility: replace od flags --skip-bytes= and --read-bytes= with -j and -N (thanks to Oliver Clarke for working this out!)
revision 0.27 . KD 11/2012 - EXCLUDE_RESOLUTION_RANGE lines and generic Pilatus Flat_field test
revision 0.28 . Keitaro 11/2012 - for MarCCD: read oscillation range from the position 1024+736 (fix for omega rotation)
revision 0.29 . KD 1/2013 - include UNTRUSTED_RECTANGLEs for Pilatus 6M; never hurts but needed if the beamline software does not mark them with -2 or such
revision 0.30 . Keitaro 3/2013 - for ADSC: write all possible beam center conventions in XDS.INP as comments
revision 0.31 . Keitaro 3/2013 - add comment for reversed phi for SPring-8
revision 0.32 . Keitaro 3/2013 - add RAXIS support. only tested with RAXIS IV++ and VII.
revision 0.33 . Keitaro 5/2013 - automatically set ROTATION_AXIS=-1 0 0 for SPring-8 BL32XU/41XU/44XU beamlines based on detector serial numbers.
revision 0.34 . Keitaro 5/2013 - recognize ADSC detectors in Photon Factory and choose correct beam center convention based on detector serial numbers.
revision 0.35 . KD 6/2013 - reduce 7000 to 6000 for shadow detection; insert comment about *_RESOLUTION_RANGE lines
revision 0.36 . KD 6/2013 - insert NUMBER_OF_PROFILE_GRID* lines for Pilatus (suggested by C.Vonrhein)
revision 0.37 . Keitaro 10/2013 - fix for MX225HS detector on SPring-8 BL32XU (Ignore case when matching marccd in header); see also rev-0.39
revision 0.38 . KD 2/2014 - change defaults for REFINE(IDXREF) and REFINE(INTEGRATE) such that more stable results are obtained for difficult datasets
revision 0.39 . Keitaro 4/2014 - automatically set ROTATION_AXIS=-1 0 0 for MX225HS at SPring-8 BL32XU.
revision 0.40 . Jan Gebauer /KD 4/2014 - simple implementation of MAR345 detector
revision 0.41 . recognize header starting with R-AXIS instead of RAXIS
revision 0.42 . Keitaro 5/2014 automatically set ROTATION_AXIS=-1 0 0 for Q315 at SPring-8 BL38B1.
revision 0.43 . Keitaro 5/2014 add experimental support of dTREK format (raxis_smv)
revision 0.44 . Keitaro 5/2014 automatically set ROTATION_AXIS=-1 0 0 for PILATUS3 at SPring-8 BL41XU.
revision 0.45 . KD cope with blanks in filenames
revision 0.46 . Keitaro 6/2014 automatically set ROTATION_AXIS=-1 0 0 for Mar225 at SPring-8 BL26B2.
revision 0.47 . Keitaro 7/2014 more generic dTREK format support (Saturn and RAXIS)
revision 0.48 . Kip Guja 11/2014 add detector serial number for ALS 5.0.2 to beam center convention 1
revision 0.49 . Nobuhisa 2/2015 add detector serial number for AichiSR BL2S1 to beam center convention 1
revision 0.50 . KD 03/2015 workaround for Mar-1 change of parameter name "DISTANCE" to "POSITION" in REFINE(*) keywords
revision 0.51 . Keitaro 03/2015 add .gz and .xz support and remove limitation - frame numbers can start with any.
revision 0.52 . Keitaro 05/2015 fix ADSC beam center convention for SPring-8 and DET_SN acquisition for PILATUS (didn't work on Mac)
revision 0.53 . KD add ADSC beam center convention for APS Argonne but only as commented line in XDS.INP (not detector serial no)
revision 0.54 . KD add ADSC S/N 911 for APS Argonne, and fix spurious output arising from THETADISTANCE (?!)
revision 0.55 . KD add ADSC S/N 446 for APS, and check w/ 12 datasets from data.sbgrid.org. No rule found: S/N 916 @ APS 24_ID_E !
revision 0.56 . Keitaro 12/2015 show error message when user's input didn't match any files
revision 0.57 . KD 12/2015 start to take care of vertical ROTATION_AXIS at Diamond I24 - for now only introduce comment
revision 0.58 . Keitaro 01/2016 fix for dTREK image: take 'rotation axis' information from header
revision 0.59 . KD 04/04/2016 check for ADSC detector _after_ dtrek detector, to correct wrong choice for https://zenodo.org/record/45756
revision 0.60 . KD 04/04/2016 ADSC detector SN=458 at APS 19-ID has reverse phi (https://zenodo.org/record/45756)
revision 0.61 . Keitaro 10/04/2016 Add Eiger hdf5 support (may be incomplete; UNTRUSTED_RECTANGLE=s not set) NEED h5dump.
revision 0.62 . Keitaro 11/04/2016 Can give foo_master.h5 instead of foo_??????.h5.
revision 0.63 . Keitaro 13/04/2016 Set UNTRUSTED_RECTANGLE=s for EIGER 9M and 16M (KD).
revision 0.64 . KD 16/06/2016 reverse phi @APS 19ID (reported by Wolfram Tempel)
revision 0.65 . Keitaro 07/09/2016 Fix for "too many arguments" problem in ls
revision 0.66 . KD 15/09/2016 add Bruker PHOTON II with .cbf frames
revision 0.67 . KD 02/10/2016 add BM30A (ADSC SERIAL 924) reverse_phi
revision 0.68 . KD 24/10/2016 add -H option (follow symlinks) to "find" command (thanks to Jan Gebauer!)
revision 0.69 . KD 04/11/2016 add CMOS-1 MBC Detector at ALS 4.2.2
revision 0.70beta . KD 08/12/2016 ROTATION_AXIS=0 -1 0 at Diamond I24; depending on CBF header
revision 0.70 . KD 12/01/2017 remove error message if h5dump does not find /entry/sample/transformations/omega/vector
revision 0.71 . KD 27/02/2017 implement rule for S/N 916 @ APS 24_ID_E
revision 0.72 . KD 8/03/2017 fix nframes lookup in Eiger master file
revision 0.73 . KD 18/05/2017 for Andrey Nascimento: add Pilatus 2M S/N 24-0109 with ROTATION_AXIS=-1 0 0
revision 0.74 . Keitaro 02/08/2017 Add PILATUS3 6M, S/N 60-0127 at CHESS F1 with ROTATION_AXIS=-1 0 0
revision 0.75 . KD 30/08/2017 reversed ORGX and ORGY for marCCD @ BM14 (Indian beamline @ ESRF)
revision 0.76 . KD 4/09/2017 include POSITION into REFINE(IDXREF) because latest XDS is more robust. Add comments to keywords.
revision 0.77 . KD 19/12/2017 obtain QX QY from CBF header.
revision 0.78 . KD 21/12/2017 if possible and sensible, provide LIB= line with hardcoded /usr/local/lib64/dectris-neggia.so .
revision 0.79 . KD 16/01/2018 read OVERLOAD from Pilatus miniCBF header instead of fixing at 1048576
revision 0.80 . KD 13/02/2018 remove DISTANCE keyword from REFINE() list; remove POSITION from REFINE(IDXREF)
revision 0.81 . KD 21/02/2018 when encountering CBF files from Eiger (ESRF), treat as Pilatus detector
revision 0.82 . KD 01/03/2018 STARTING_ANGLE for MarCCD/Pilatus/PHOTON, enabling to use dials.rs_mapper with spot2pdb.
revision 0.83 . KD 25/06/2018 for ADSC detector #458 at APS BM19, revert the definition of ROTATION_AXIS=-1 0 0. See "Beamline notes" in this wiki.
revision 0.84 . KD 10/10/2018 implement Pilatus detector with d*TREK header
revision 0.85 . Jie Nan 09/01/2019 STARTING_ANGLE for Eiger
revision 0.86 . Keitaro 03/05/2019 Add PILATUS3 6M, S/N 60-0123 at SSRF BL18U-1 with ROTATION_AXIS=-1 0 0
revision 0.87 . KD 12/10/2019 Add PILATUS XXX, S/N XX-XXX at SSRF BL19U1 and MarCCD detector #43 at BL17B1 with ROTATION_AXIS=-1 0 0
revision 0.88 . KD 16/10/2019 fixes for SSRF, add "-maxdepth 1" to "find -H ..."
revision 0.89 . KD 21/10/2019 add ADSC S/N 905 at ALS 8.2.1, S/N 928 at Australian Synchrotron MX2 beamline; final SSRF fixes
revision 0.90 . KD 25/10/2019 add OLDMAR detector type. Tested w/ SBGrid data set 6. Anomalous signal may have wrong hand!
revision 0.91 . KD 16/01/2020 Allow negative starting angle for Eiger (found -33 at SLS !).
revision 0.92 . KD 27/02/2020 read *_master.h5 from Diamond Light Source
revision 0.93 . KD 13/03/2020 print out 2theta for MarCCD (DETECTOR_*_AXIS can be derived from this)
revision 0.94 . KD 16/03/2020 bugfix for Bruker-cbf to make bc accept e.g. 3.1e-005 by using awk printf "%.5f" instead of awk print
revision 0.95 . KD 29/07/2020 fix DLS Eiger HDF5 variant OSCILLATION_RANGE, STARTING_ANGLE. Attention: DLS Eiger variant needs h5dump 1.10 for OVERLOAD!
revision 0.96 . KD 03/10/2020 make script echo detector serial number if inverted ROTATION_AXIS is detected.
revision 0.97 . KD 19/10/2020 add UNTRUSTED_RECTANGLEs for Eiger2 (which has a few pixels less than Eiger (thanks to Andreas Förster)
revision 0.98 . Thomas Hauß (HZB) 06/11/2020 switch Python print syntax to Python3
revision 0.99 . Aaron Finke (CHESS) 07/11/2020 add ROTATION_AXIS=-1 0 0 for EIGER2 16M detector at MX beamline ID7B2 (CHESS)
revision 1.00 . Gleb Bourenkov 10/11/2020 add ROTATION_AXIS=0 -1 0 for beamline P14 (EMBL Hamburg) detectors Eiger 16M S/N E-32-0107; previously Eiger 4M S/N E-08-0107 and PILATUS 6M-F, S/N 60-0115-F
revision 1.01 . KD 07/12/2020 ROTATION_AXIS=0 -1 0 for Pilatus3 2M, S/N 24-0118 at ID23-2 (http://www.globalphasing.com/autoproc/wiki/index.cgi?BeamlineSettings)
revision 1.02 . KD 11/01/2021 recognize mar555 detector as mar345 (thanks to Thomas Hauß, HZB)
revision 1.03 . Feng Yu 18/7/2021 fix/expand information about Shanghai Synchrotron Radiation Facility (SSRF)
revision 1.04 . Zhipu Luo 13/08/2021 extract ORGX,ORGY for electron diffraction from SMV header if the wavelength value starts with 0.0
revision 1.05 . KD specialcase extraction of number of images for BNL detectors E-32-0101 and E-18-0104
revision 1.06 . KD fix URLs in output. Availability of Apple M1 processor dectris-neggia-Apple-arm64.so . LC_ALL=C .
revision 1.07 . KD, Thomas Hauß, Gleb Bourenkov 25/10/2021. Detector moved from Petra P14 to P13
revision 1.08 . KD, Thomas Hauß, Feng Yu 7/11/2021 fix TZ in timestamps for SSRF detectors
revision 1.09 . KD, Feng Yu 7/11/2021 fix detector number BNL E-18-0121 to be E-18-0104
revision 1.10 . KD NeXus header for Eiger

REVISION="1.10 (7-MAR-2022)"

usage: e.g. generate_XDS.INP "/file/system/frms/mydata_1_???.img"
make sure to have the two quotation marks !
the ? are wildcards for the frame numbers.
known problems:
revision 1.10 . KD implement NeXus for Eiger
- for ADSC detectors, there are at least three ways to obtain ORGX and ORGY values from the header (see below);
- the same might be a problem for MAR headers, too (not sure about this)
- on Mac OS X, the Xcode command line tools (from https://developer.apple.com/download/more/) are needed.
notes for debugging of the script:
- add the -x option to the first line, to see where an error occurs
- comment out the removal of tmp1 and tmp2 in the last line
====== Start of script ======

echo generate_XDS.INP version $REVISION . Obtain the latest version from echo http://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/generate_XDS.INP if [ "$1" == "help" ] || [ "$1" == "-help" ] || [ "$1" == "-h" ]; then

 echo usage: generate_XDS.INP \"/file/system/frms/mydata_1_???.img\"   \(_with_ the quotation marks!\)                       
 echo if the frames are compressed with bzip2, leave out the .bz2 extension!                                    
 exit

fi

make sure the locale does not interfere with e.g. awk calculations:

LC_ALL="C";export LC_ALL

defaults:
conversion radian / degrees:

DEGTOR=57.2957795

DETECTOR="XXX MINIMUM_VALID_PIXEL_VALUE=XXX OVERLOAD=XXX" REVERSE_PHI="no" ORGX=XXX ORGY=XXX DETECTOR_DISTANCE=XXX OSCILLATION_RANGE=XXX X_RAY_WAVELENGTH=XXX QX=XXX QY=XXX NX=XXX NY=XXX SENSOR_THICKNESS=0 TRUSTED_REGION="0.0 1.2 ! partially use corners of detector (0 1.4143: use all pixels)"

default MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT

MNOPIAS=3

default DIRECTION_OF_DETECTOR_X-AXIS

DIRECTION_OF_DETECTOR_X_AXIS="1 0 0"

default DIRECTION_OF_DETECTOR_Y-AXIS

DIRECTION_OF_DETECTOR_Y_AXIS="0 1 0"

default FRACTION_OF_POLARIZATION

pol_frac=0.98 STARTING_ANGLE=0 dtrek_det="" SEPMIN=7.0 CLUSTER_RADIUS=3.5 REFINE_CORRECT="CELL BEAM ORIENTATION AXIS POSITION ! Default is: refine everything"

dname=`echo "$1" | xargs dirname` test "${dname}" == "" && dname="." bname=`echo "$1" | xargs basename`

see how we are called:

NAME_TEMPLATE_OF_DATA_FRAMES="${dname}/${bname}"

list frames matching the wildcards in NAME_TEMPLATE_OF_DATA_FRAMES
don't accept the "direct beam" shot at SLS/Pilatus PX-I and PX-II
cope with blanks in directory / file name

IFS=$'\n' find -H $dname -maxdepth 1 -name "$bname" -or -name "${bname}.bz2" -or -name "${bname}.gz" -or -name "${bname}.xz" | egrep -v "_00000.cbf|_000.img" | sort > tmp1 if [ ! -s tmp1 ] then

echo "Error! No files matched: $1"
rm -f tmp1
exit 1

fi unset IFS

we can continue - the frames are found

if echo $NAME_TEMPLATE_OF_DATA_FRAMES | grep '_master.h5$' > /dev/null; then

NAME_TEMPLATE_OF_DATA_FRAMES=`echo "$NAME_TEMPLATE_OF_DATA_FRAMES" | sed -e 's/_master.h5$/_??????.h5/'`

else

# Find the first '?' position and the number of '?' to determine DATA_RANGE=.
pos1=`echo "$NAME_TEMPLATE_OF_DATA_FRAMES" | awk '{print index($0, "?")}'`
pos2=`echo "$NAME_TEMPLATE_OF_DATA_FRAMES" | sed -e "s/[^\?]//g" | awk '{print length+'$pos1' - 1}'`
data_first=`cut -b $pos1-$pos2 tmp1 | head -n1 | bc`
data_last=`cut -b $pos1-$pos2 tmp1 | tail -n1 | bc`
DATA_RANGE="$data_first $data_last"
echo DATA_RANGE=$DATA_RANGE

# set SPOT_RANGE to first half of DATA_RANGE
data_num=`wc -l tmp1 | awk '{print $1}'`
data_half=`echo "scale=0; $data_num/2" | bc -l`                        
data_half=`echo "if ($data_half<=1) 1;if ($data_half>1) $data_half" | bc -l`
spot_last=`echo "scale=0; $data_first+$data_half-1" | bc -l`
SPOT_RANGE="$data_first $spot_last"

fi

FIRSTFRAME=`head -1 tmp1` echo $FIRSTFRAME | grep "\.h5$" && is_h5=1 || is_h5=0

find out detector type

DET=XXX IFS=$'\n' echo $FIRSTFRAME | grep -q '\.bz2$' && bzcat $FIRSTFRAME > tmp1 && FIRSTFRAME=tmp1

for mac/linux compatibility. zcat foo.gz doesn't work on mac.

echo $FIRSTFRAME | grep -q '\.gz$' && zcat < $FIRSTFRAME > tmp1 && FIRSTFRAME=tmp1 echo $FIRSTFRAME | grep -q '\.xz$' && xzcat $FIRSTFRAME > tmp1 && FIRSTFRAME=tmp1

unset IFS if [ "$is_h5" == 0 ]; then

strings $FIRSTFRAME > tmp2   
# TODO: whenever FIRSTFRAME is used below, it should be copied to tmp2 (using IFS as above), and tmp2 should be used instead
# this was done for "mccd", but still has to be done for the "raxis" detector types
# the reason is that FIRSTFRAME may contain a blank, which makes some commands fail                                                
egrep -qi 'marccd|Corrected' tmp2 && DET=mccd                                 
grep -q PILATUS tmp2             && DET=pilatus 
grep -iq Eiger tmp2              && DET=pilatus                             
head -n1 tmp2 | grep -q "^RAXIS" && DET=raxis
head -n1 tmp2 | grep -q "^R-AXIS" && DET=raxis
grep -q "^SOURCE_WAVELENGTH= *1" tmp2 && DET=dtrek
grep -q BEAM_CENTER_X tmp2       && DET=adsc                                 
grep -q mar345 tmp2		 && DET=MAR345
# rev. 1.02: in one case, mar555 data could be processed pretending it is mar345, so:
grep -q mar555 tmp2		 && DET=MAR345
grep -q BRUKER tmp2 && grep -q CBF tmp2  && DET=Bruker-cbf   
grep -q CMOS1 tmp2 && DET=adsc-CMOS1   
grep -q MARCONTROL tmp2 && DET=OLDMAR  
grep -q "WAVELENGTH=0.0" tmp2 && DET=experimental-ED

else

h5dump -d "/entry/instrument/detector/description" $FIRSTFRAME | grep -i Eiger > /dev/null && DET=eiger

fi

identify other detector types in the same way

parse ASCII header of first frame

if [ "$DET" == "XXX" ]; then

 echo "this is not a MAR, ADSC/SMV or PILATUS detector - fill in XXX values manually!"
 DETECTOR="XXX MINIMUM_VALID_PIXEL_VALUE=XXX OVERLOAD=XXX"

find parameters of first frame

elif [ "$DET" == "mccd" ]; then

 echo Data from a MarCCD detector

http://www.sb.fsu.edu/~xray/Manuals/marCCD165header.html has header information

 DETECTOR="CCDCHESS MINIMUM_VALID_PIXEL_VALUE= 1 OVERLOAD= 65500"
 SENSOR_THICKNESS=0.01                                           
 # use first frame of dataset to obtain parameters

 # Check detector serial number and recognize beamline for reversed-phi setting.
 # Known detectors for reversed-phi in SPring-8: 24: BL26B2 Mar225, 31: BL32XU MX225HE, 38: BL44XU MX225HE, 42: BL44XU MX300HE, 40: BL41XU MX225HE, 106: BL32XU MX225HS
 # same for SSRF: BL17B1 rayonix MX300. As on 2019-10-13, this also needs doubling of ORGX and ORGY. But the beamline staff (Wenming) wants to fix the header.
 REVERSEPHI_SNs="

24 31 38 40 42 43 106 "

 # get detector serial number and check if it is included in the list
 DET_SN=`grep "Detector Serial Number =" tmp2 | sed "s/Detector Serial Number = //"`
 if echo "${DET_SN}${REVERSEPHI_SNs}" | sort | uniq -d | grep [0-9] > /dev/null; then
   REVERSE_PHI="yes"
   echo inverted ROTATION_AXIS since Detector Serial Number is "${DET_SN}"
 fi

 # offsets are documented; values can be found in mccd_xdsparams.pl script
 IFS=$'\n'
 cp $FIRSTFRAME tmp2
 unset IFS
 let SKIP=1024+80                                                        
 NX=$(od -t dI -j $SKIP -N 4 tmp2 | head -1 | awk '{print $2}')
 let SKIP=$SKIP+4                                                                         
 NY=$(od -t dI -j $SKIP -N 4 tmp2 | head -1 | awk '{print $2}')

 let SKIP=1720
 DETECTOR_DISTANCE=$(od -t dI -j $SKIP -N 4 tmp2 | head -1 | awk '{print $2}')
 DETECTOR_DISTANCE=`echo "scale=3; $DETECTOR_DISTANCE/1000" | bc -l`

Mar 12, 2020 KD

 let SKIP=1724
 TWOTHETA=$(od -t dI -j $SKIP -N 4 tmp2 | head -1 | awk '{print $2}')
 TWOTHETA=`echo "scale=3; $TWOTHETA/1000" | bc -l`
 echo 2THETA= $TWOTHETA

 let SKIP=1024+256+128+256+44
 STARTING_ANGLE=$(od -t dI -j $SKIP -N 4 tmp2 | head -1 | awk '{print $2}') 
 STARTING_ANGLE=`echo "scale=2; $STARTING_ANGLE/1000" | bc -l `
 echo STARTING_ANGLE= $STARTING_ANGLE   
                                                                                                      
 let SKIP=1024+256+128+256+4                                                                             
 ORGX=$(od -t dI -j $SKIP -N 4 tmp2 | head -1 | awk '{print $2}')             
 ORGX=`echo "scale=2; $ORGX/1000" | bc -l `                                                              
 let SKIP=$SKIP+4                                                                                        
 ORGY=$(od -t dI -j $SKIP -N 4 tmp2 | head -1 | awk '{print $2}')             
 ORGY=`echo "scale=2; $ORGY/1000" | bc -l `

fixed Aug 30, 2017 after IUCr2017 @ Hyderabad

 if [ "$DET_SN" == "4" ]; then
    TEMP=$ORGY
    ORGY=$ORGX
    ORGX=$TEMP
    echo reversed ORGX and ORGY for marCCD @ ESRF BM14
 fi

 let SKIP=1024+736
 OSCILLATION_RANGE=$(od -t dI -j $SKIP -N 4 tmp2 | head -1 | awk '{print $2}') 
 OSCILLATION_RANGE=`echo "scale=3; $OSCILLATION_RANGE/1000" | bc -l`   
                                                                                                
 let SKIP=1024+256+128+256+128+4                                                                
 QX=$(od -t dI -j $SKIP -N 4 tmp2 | head -1 | awk '{print $2}')      
 QX=`echo "scale=10; $QX/1000000" |bc -l `                                                      
 let SKIP=$SKIP+4                                                                               
 QY=$(od -t dI -j $SKIP -N 4 tmp2 | head -1 | awk '{print $2}')      
 QY=`echo "scale=10; $QY/1000000" |bc -l `

 let SKIP=1024+256+128+256+128+128+12
 X_RAY_WAVELENGTH=$(od -t dI -j $SKIP -N 4 tmp2 | head -1 | awk '{print $2}')
 X_RAY_WAVELENGTH=`echo "scale=5; $X_RAY_WAVELENGTH/100000" | bc -l`

at most BLs, ORGX and ORGY are in pixels, but sometimes in mm ... guess:

 NXBYFOUR=`echo "scale=0; $NX/4" | bc -l `                               
 ORGXINT=`echo "scale=0; $ORGX/1" | bc -l `                              
 if [ $ORGXINT -lt $NXBYFOUR ]; then                                     
    ORGX=`echo "scale=1; $ORGX/$QX" | bc -l`                             
    ORGY=`echo "scale=1; $ORGY/$QY" | bc -l`                             
    echo MARCCD detector: header ORGX, ORGY seem to be in mm ... converting to pixels
 else                                                                                
    echo MARCCD detector: header ORGX, ORGY seem to be in pixel units                
 fi

elif [ "$DET" == "adsc" ]; then

 DETECTOR="ADSC MINIMUM_VALID_PIXEL_VALUE= 1 OVERLOAD= 65000"
 echo Data from ADSC detector. Obtaining ORGX, ORGY depends on beamline setup:
 SENSOR_THICKNESS=0.01                                                        
 sed s/\;// tmp2 > tmp1                                                       
 mv tmp1 tmp2

     # find X_RAY_WAVELENGTH:
     X_RAY_WAVELENGTH=`grep WAVELENGTH tmp2 | head -1 | sed s/WAVELENGTH=//`

     # find NX, QX, ORGX and ORGY:
     NX=`grep SIZE1 tmp2 | tail -1 | sed s/SIZE1=//`
     NY=`grep SIZE2 tmp2 | tail -1 | sed s/SIZE2=//`                                             
     QX=`grep PIXEL_SIZE tmp2 | sed s/PIXEL_SIZE=//`
     QY=$QX                                               
     BEAM_CENTER_X=`grep BEAM_CENTER_X tmp2 | sed s/BEAM_CENTER_X=//`
     BEAM_CENTER_Y=`grep BEAM_CENTER_Y tmp2 | sed s/BEAM_CENTER_Y=//`

     COMMENT_ORGXY="

! Following are possible beam center interpretations for ADSC detectors"

at ESRF, PF, ALS 5.0.2, AS MX2 and ... (pls fill in!) the following should be used:

     ORGX1=`echo "scale=1; $BEAM_CENTER_Y/$QX" | bc -l`
     ORGY1=`echo "scale=1; $BEAM_CENTER_X/$QX" | bc -l`
     echo - at ESRF, PF, ALS 8.2.1, APS Argonne BLs use: ORGX=$ORGX1 ORGY=$ORGY1                    
     COMMENT_ORGXY="${COMMENT_ORGXY}

! ORGX= $ORGX1 ORGY= $ORGY1 ! For ESRF, PF, APS, AS MX2 ..."

this 2nd alternative convention should be used at the following beamlines (pls complete the list): ALS 5.0.3, ...

     ORGX2=`echo "scale=1; $NX-$BEAM_CENTER_X/$QX" | bc -l `
     ORGY2=`echo "scale=1; $BEAM_CENTER_Y/$QX" | bc -l `
     echo - at e.g. ALS 5.0.3 use: ORGX=$ORGX2 ORGY=$ORGY2                                                          
     COMMENT_ORGXY="${COMMENT_ORGXY}

! ORGX= $ORGX2 ORGY= $ORGY2 ! For ALS 5.0.3,.."

this 3rd alternative convention should be used at the following beamlines (pls complete the list): ALS 8.2.2, ...
this alternative is written into the generated XDS.INP ! You have to correct this manually in XDS.INP, or adjust this script.

     ORGX3=`echo "scale=1; $BEAM_CENTER_X/$QX" | bc -l `
     ORGY3=`echo "scale=1; $NX-$BEAM_CENTER_Y/$QX" | bc -l `
     echo - at e.g. ALS 8.2.2 use: ORGX=$ORGX3 ORGY=$ORGY3 - this is written to XDS.INP if beamline is not detected
     COMMENT_ORGXY="${COMMENT_ORGXY}

! ORGX= $ORGX3 ORGY= $ORGY3 ! For ALS 8.2.2,.."

this 4th alternative convention should be used at the following beamlines (pls complete the list): SPring-8, ...

     ORGX4=`echo "scale=1; $BEAM_CENTER_X/$QX" | bc -l `
     ORGY4=`echo "scale=1; $BEAM_CENTER_Y/$QX" | bc -l `
     echo - at e.g. SPring-8 use: ORGX=$ORGX4 ORGY=$ORGY4 
     COMMENT_ORGXY="${COMMENT_ORGXY}

! ORGX= $ORGX4 ORGY= $ORGY4 ! For SPring-8,.."

     # Decision of beam center convention based on detector serial numbers.
     DET_SN=`grep DETECTOR_SN tmp2 | sed -e "s/DETECTOR_SN=//"`
     echo Detector serial number is $DET_SN
     # For convention 1; Known PF detectors = 449: NW12A Q210, 472: NE3A Q270, 474: BL17A Q270, 912: BL5A Q315, 923: ALS BL5.0.2 Q315, 933: AichiSR BL2S1 Q315, 916: APS 24 IDE, 928: AS MX2
     ORG1_SNs="

449 472 474 912 923 933 911 446 916 905 928 "

     ORG4_SNs="

915 "

     if echo "${DET_SN}${ORG1_SNs}" | sort | uniq -d | grep [0-9] > /dev/null; then
      ORGX=$ORGX1
      ORGY=$ORGY1
      echo the following was chosen based on detector serial number:
     elif echo "${DET_SN}${ORG4_SNs}" | sort | uniq -d | grep [0-9] > /dev/null; then
      ORGX=$ORGX4
      ORGY=$ORGY4
      echo the following was chosen based on detector serial number:
     else
      ORGX=$ORGX3
      ORGY=$ORGY3
      echo the following default was chosen because the detector serial number was not special-cased:
     fi

     # Check detector serial number and recognize beamline for reversed-phi setting.
     # Known detectors for reversed-phi in SPring-8: 915: BL38B1 Q315; APS 19-ID: 458; BM30A: 924
     # 928 is at Australian Beamline MX2
     # revision 0.83 of this script removes 458 from the list!
     REVERSEPHI_SNs="

915 924 928 "

     if echo "${DET_SN}${REVERSEPHI_SNs}" | sort | uniq -d | grep [0-9] > /dev/null; then
       REVERSE_PHI="yes"
       echo inverted ROTATION_AXIS since detector serial number is ${DET_SN}
     fi

     # find DETECTOR_DISTANCE and OSCILLATION_RANGE:                                                                                
     DETECTOR_DISTANCE=`grep ^DISTANCE tmp2 | sed s/DISTANCE=//`                                                                     
     OSCILLATION_RANGE=`grep OSC_RANGE tmp2 | sed s/OSC_RANGE=//`

elif [ "$DET" == "adsc-CMOS1" ]; then

 DETECTOR="ADSC MINIMUM_VALID_PIXEL_VALUE= 1 OVERLOAD= 65000"
 echo Data from CMOS1 MBC detector.
 SENSOR_THICKNESS=0.01                                                        
 sed s/\;// tmp2 > tmp1                                                       
 mv tmp1 tmp2                                                                 
 X_RAY_WAVELENGTH=`grep WAVELENGTH tmp2 | head -1 | sed s/WAVELENGTH=//`
 NX=`grep SIZE1 tmp2 | tail -1 | sed s/SIZE1=//`
 QX=`grep PIXEL_SIZE tmp2 | sed s/PIXEL_SIZE=//`
 NY=`grep SIZE2 tmp2 | tail -1 | sed s/SIZE2=//`                                             
 QY=$QX                                               
 ORGX=`grep BEAM_CENTER_X tmp2 | sed s/BEAM_CENTER_X=//`
 ORGY=`grep BEAM_CENTER_Y tmp2 | sed s/BEAM_CENTER_Y=//`
 REVERSE_PHI="yes"
 DETECTOR_DISTANCE=`grep ^DISTANCE tmp2 | sed s/DISTANCE=/-/`                                                                     
 OSCILLATION_RANGE=`grep OMEGA_DELTA tmp2 | sed s/OMEGA_DELTA=//`  
 g_DETECTOR_X_AXIS="-1 0 0"

elif [ "$DET" == "experimental-ED" ]; then

 DETECTOR="ADSC MINIMUM_VALID_PIXEL_VALUE= 1 OVERLOAD= 65000  ! OVERLOAD is really unknown "
 echo --- Electron diffraction data in SMV format!
 SENSOR_THICKNESS=0.01                                                        
 sed s/\;// tmp2 > tmp1                                                       
 mv tmp1 tmp2                                                                 
 X_RAY_WAVELENGTH=`grep WAVELENGTH tmp2 | head -1 | sed s/WAVELENGTH=//`
 NX=`grep SIZE1 tmp2 | tail -1 | sed s/SIZE1=//`
 QX=`grep PIXEL_SIZE tmp2 | sed s/PIXEL_SIZE=//`
 NY=`grep SIZE2 tmp2 | tail -1 | sed s/SIZE2=//`                                             
 QY=$QX  
 BEAM_CENTER_X=`grep BEAM_CENTER_X tmp2 | sed s/BEAM_CENTER_X=//`
 BEAM_CENTER_Y=`grep BEAM_CENTER_Y tmp2 | sed s/BEAM_CENTER_Y=//` 
 QXY=`echo "scale=1; $QX*($BEAM_CENTER_X+$BEAM_CENTER_Y)" | bc -l`
     if [ "$QXY" "<" "$BEAM_CENTER_X" ] ; then
      ORGX=`echo "scale=2; $BEAM_CENTER_Y/$QX" | bc -l`
      ORGY=`echo "scale=2; $BEAM_CENTER_X/$QX" | bc -l`
     else
      ORGX=`echo "scale=2; $BEAM_CENTER_X/1" | bc -l`
      ORGY=`echo "scale=2; $BEAM_CENTER_Y/1" | bc -l`
     fi
 DETECTOR_DISTANCE=`grep ^DISTANCE tmp2 | sed s/DISTANCE=//`                                                                     
 OSCILLATION_RANGE=`grep OSC_RANGE tmp2 | sed s/OSC_RANGE=//` 
 STARTING_ANGLE=`grep OSC_START tmp2 | sed s/OSC_START=//` 
 echo --- For TIMEPIX detector, please fix ROTATION_AXIS yourself!
 DIRECTION_OF_DETECTOR_X_AXIS="1 0 0" 
 REFINE_CORRECT="ORIENTATION CELL AXIS BEAM ! for ED, no POSITION when CELL is refined"

elif [ "$DET" == "pilatus" ]; then

 echo Data from a Pilatus detector                                         
 sed s/#// tmp2 > tmp1                                                     
 mv tmp1 tmp2                                                              
 OVERLOAD=1048574
 SEPMIN=4
 CLUSTER_RADIUS=2
 grep -q Count_cutoff tmp2 && OVERLOAD=`awk '/Count_cutoff/{print $2}' tmp2`
 DETECTOR="PILATUS MINIMUM_VALID_PIXEL_VALUE=0 OVERLOAD= $OVERLOAD  !PILATUS"
 QX=0.172 QY=0.172

the default above guards against missing Pixel_size line in CBF header

 grep -q Pixel_size tmp2 && QX=`awk '/Pixel_size/{print 1000*$2}' tmp2`
 grep -q Pixel_size tmp2 && QY=`awk '/Pixel_size/{print 1000*$5}' tmp2`

     # find SENSOR_THICKNESS:
     SENSOR_THICKNESS=`grep thickness tmp2 | sed -e s/'Si.* sensor, thickness'// | awk '{print $1*1000}'`
     # find X_RAY_WAVELENGTH:                                                                               
     X_RAY_WAVELENGTH=`grep Wavelength tmp2 | sed -e s/Wavelength// -e s/A// | awk '{print $1}'`

     # find NX and NY; 2463/2527 is 6M, 1475/1679 is 2M
     NX=`grep X-Binary-Size-Fastest-Dimension tmp2 | awk '{print $2}'`
     NY=`grep X-Binary-Size-Second-Dimension tmp2 | awk '{print $2}'`

     # find ORGX and ORGY:
     ORGX=`grep Beam_xy tmp2 | sed -e s/\(// -e s/\)// -e s/\,// | awk '{print $2}'`
     ORGY=`grep Beam_xy tmp2 | sed -e s/\(// -e s/\)// -e s/\,// | awk '{print $3}'`

     # find DETECTOR_DISTANCE, OSCILLATION_RANGE, and STARTING_ANGLE:
     DETECTOR_DISTANCE=`awk '/Detector_distance/{print $2}' tmp2`
     DETECTOR_DISTANCE=`echo "$DETECTOR_DISTANCE*1000" | bc -l`

     OSCILLATION_RANGE=`awk '/Angle_increment/{print $2}' tmp2`

     STARTING_ANGLE=`awk '/Start_angle/{print $2}' tmp2`
     echo STARTING_ANGLE= $STARTING_ANGLE

     # get detector serial number and check if it is included in the list
     # Known detectors for reversed-phi in SPring-8: BL41XU PILATUS3 6M 60-0125
     # Known detectors for reversed-phi in APS: 19ID PILATUS3 6M 60-0132
     # Known detectors for reversed-phi at MX2 beamline (Brazilian Synchrotron National Laboratory - LNLS)
     # Known detectors for reversed-phi at CHESS F1 PILATUS3 6M, S/N 60-0127
     # Known detectors for reversed-phi at SSRF BL18U1 (S/N 60-0123) and BL19U1 (S/N XX-XXX) (!; 2019-10-19: staff will fix this)
     DET_SN=`grep "Detector:" tmp2 | sed "s/^.*Detector: *//"`
     REVERSEPHI_SNs="

PILATUS3 6M, S/N 60-0125 PILATUS3 6M, S/N 60-0132 PILATUS 2M, S/N 24-0109 PILATUS3 6M, S/N 60-0127 PILATUS3 6M, S/N 60-0123 "

     if echo "${DET_SN}${REVERSEPHI_SNs}" | sort | uniq -d | grep [0-9] > /dev/null; then
       REVERSE_PHI="yes"
       echo inverted ROTATION_AXIS since detector serial number is ${DET_SN}
     fi
     if [ "$DET_SN" == "PILATUS XXX, S/N XX-XXX" ] ; then
       REVERSE_PHI="yes"
       echo inverted rotation axis at SSRF BL19U1
     fi

Diamond I24:

     if [ "$DET_SN" == "PILATUS3 6M, S/N 60-0119" ] ; then
       if grep -q "Oscillation_axis X.CW +SLOW" tmp2 ; then
         rotation_axis="0 -1 0"
         echo ROTATION_AXIS="0 -1 0" at Diamond I24
       fi
     fi

PETRA P14: raw data from Eiger are stored as CBF files so this is treated as Pilatus

     if [ "$DET_SN" == "Dectris Eiger 4M, E-08-0107" -o "$DET_SN" == "PILATUS 6M-F, S/N 60-0115-F" ] ; then
       rotation_axis="0 -1 0"
       echo ROTATION_AXIS="0 -1 0" at PETRA P14
     fi

similarly, Eiger 16M at PETRA P14 until May 22, 2021

     if [ "$DET_SN" == "Dectris Eiger 16M, E-32-0107" ] ; then
       egrep -q '^# 2020-|^# 2021-0[1-5]|^# 2021-06-[01]|^# 2021-06-2[01]' tmp2 || isatP13=1
       if [ "$isatP13" == 1 ] ; then
         echo using the default ROTATION_AXIS=1 0 0 at PETRA P13
       else
         rotation_axis="0 -1 0"
         echo ROTATION_AXIS="0 -1 0" at PETRA P14
       fi
     fi

ESRF ID23-2:

     if [ "$DET_SN" == "PILATUS3 2M, S/N 24-0118, ESRF ID23" ] ; then
       rotation_axis="0 -1 0"
       echo ROTATION_AXIS="0 -1 0" at ESRF ID23-2
     fi

elif [ "$DET" == "eiger" ]; then

nframes=`h5dump -d "/entry/instrument/detector/detectorSpecific/nimages" $FIRSTFRAME | awk '/\(0\): [0-9]/{print $2}'`
DATA_RANGE="1 $nframes"
SPOT_RANGE="1 `echo "scale=0; if (${nframes}<2) 1; if (${nframes}>1) ${nframes}/2"|bc -l`"

find out if HDF5 from Diamond (DLS=1) or Dectris (DLS=0)

 DLS=0
 OVERLOAD=`h5dump -d "/entry/instrument/detector/detectorSpecific/countrate_correction_count_cutoff" $FIRSTFRAME 2>/dev/null` || DLS=1 
 if [ "$DLS" == 1 ]; then
   echo Eiger HDF5 from Diamond

unfortunately h5dump 1.10 is required to get this right for the DLS .h5 files. This version is available at DLS but maybe not elsewhere

   OVERLOAD=`h5dump -d "/entry/instrument/detector/saturation_value" $FIRSTFRAME | awk '/\(0\):/{print $2}'`
 # v0.95: fix the next 2 lines by taking care of negative values with the \- , and stop after first "(0)"
   OSCILLATION_RANGE=`h5dump -d "/entry/data/omega" $FIRSTFRAME | awk '/\(0\): [\-0-9]/{print $3-$2;exit}'`
   STARTING_ANGLE=`h5dump    -d "/entry/data/omega" $FIRSTFRAME | awk '/\(0\): [\-0-9]/{print $2;exit}' | sed -e "s/,//"`
   echo OSCILLATION_RANGE=$OSCILLATION_RANGE STARTING_ANGLE=$STARTING_ANGLE
 # rotation_axis=`h5dump -a "/entry/sample/transformations/omega/vector" $FIRSTFRAME 2>/dev/null | grep "(0):" | sed -e "s/^.*://; s/,//g"`
 # the above gives -1 0 0 for DLS data instead of the correct 1 0 0, so commented out for now
 # comment while implementing rev 1.10: this is because the DLS headers are NeXus, so DIRECTION_OF_DETECTOR_X/Y_AXIS must be adjusted.
 else
   echo Eiger HDF5 from Dectris
   OVERLOAD=`h5dump -d "/entry/instrument/detector/detectorSpecific/countrate_correction_count_cutoff" $FIRSTFRAME | awk '/\(0\):/{print $2}'`    
   OSCILLATION_RANGE=`h5dump -d "/entry/sample/goniometer/omega_range_average" $FIRSTFRAME | awk '/\(0\): [0-9]/{print $2}'`
 # STARTING_ANGLE:  the \- was introduced in version 0.91 to allow negative values :
   STARTING_ANGLE=`h5dump -d "/entry/sample/goniometer/omega_start" $FIRSTFRAME | awk '/\(0\): [\-0-9]/{print $2}'`
 # /entry/sample/goniometer/omega_start is missing in some eiger2 detectors (e.g. Eiger2 9M with fw version release-2020.2.1 and SIMPLON API 1.8) (Feng YU 2021-07-18)
   if [ "$STARTING_ANGLE" == "" ]; then

echo "/entry/sample/goniometer/omega_start not found, try /entry/sample/goniometer/omega" STARTING_ANGLE=`h5dump -d /entry/sample/goniometer/omega $FIRSTFRAME | grep "(0):" | head -n 1 | awk '{print $2}' | sed -e "s/,//g"`

   fi
   echo "STARTING_ANGLE=$STARTING_ANGLE"
 # If rotation axis set (NeXus)
   rotation_axis=`h5dump -a "/entry/sample/transformations/omega/vector" $FIRSTFRAME 2>/dev/null | grep "(0):" | sed -e "s/^.*://; s/,//g"`
 # EIGER2 16M CHESS ID7B2 has S/N E-32-0123 (A. Finke 2020-11-07) v0.99
   SN=`h5dump -d "/entry/instrument/detector/detector_number" $FIRSTFRAME | awk '/\(0\): /{print $2}' | sed s/\"//g`
   if [ "$SN" == "E-32-0123" ]; then
     rotation_axis="-1 0 0"
     echo CHESS ID7B2 with inverted rotation axis
   fi
 # SSRF BL17U1 and SSRF BL10U2 (Feng YU 2021-07-18)
 # Eiger X 16M (S/N E-32-0111) was installed at SSRF BL17U1 from Oct 2017 to Feb 2021. After Feb 2021, it was moved back to SSRF BL10U2.

# 2021-03-01 00:00:00 time stamp is 1614528000

   if [ "$SN" == "E-32-0111" ]; then
     collection_time=`h5dump -d "/entry/instrument/detector/detectorSpecific/data_collection_date" $FIRSTFRAME | grep "(0):" | awk '{print $2}' | sed -e "s/\.\(.*\)//g; s/\"//g"`
     if [ `uname -s` == "Darwin" ]; then
       collection_timestamp=`date -j -f "%Y-%m-%dT%H:%M:%S" $collection_time +%s`
     elif [ `uname -s` == "Linux" ]; then
       collection_timestamp=`date -d $collection_time +%s`
     else
       collection_timestamp=0
     fi
     if [ $collection_timestamp -eq 0 ]; then
       is_rotation_axis_set=1
       rotation_axis=`h5dump -d "/SSRF/RotationAxis" $FIRSTFRAME 2>/dev/null || is_rotation_axis_set=0`
       if [ "$is_rotation_axis_set" == "1" ]; then
         rotation_axis=`h5dump -d "/SSRF/RotationAxis" $FIRSTFRAME | grep "(0):" | sed -e "s/^.*://; s/,//g"`
         echo "The rotation axis of SSRF BL10U2 (Eiger X 16M) is $rotation_axis"
       else
         rotation_axis="-1 0 0    ! Cannot determine rotation axis. SSRF BL17U1: -1 0 0; SSRF BL02U1: 0 -1 0"
       fi
     elif [ $collection_timestamp -ge 0 ] && [ $collection_timestamp -le 1614528000 ]; then
       # SSRF BL17U1
       rotation_axis="-1 0 0"
       echo "SSRF BL17U1 (Eiger X 16M) with inverted rotation axis"
     else
       # SSRF BL10U2
       is_rotation_axis_set=1
       rotation_axis=`h5dump -d "/SSRF/RotationAxis" $FIRSTFRAME 2>/dev/null || is_rotation_axis_set=0`
       if [ "$is_rotation_axis_set" == "1" ]; then
         rotation_axis=`h5dump -d "/SSRF/RotationAxis" $FIRSTFRAME | grep "(0):" | sed -e "s/^.*://; s/,//g"`
         echo "The rotation axis of SSRF BL10U2 (Eiger X 16M) is $rotation_axis"
       else
         rotation_axis="0 -1 0"
         echo "SSRF BL10U2 (Eiger X 16M) with vertical rotation axis"
       fi
     fi
   fi
 # SSRF BL02U1 (Feng YU 2021-07-18)
 # EIGER2 S 9M SSRF BL02U1 has S/N E-18-0121
   if [ "$SN" == "E-18-0121" ]; then
     is_rotation_axis_set=1
     rotation_axis=`h5dump -d "/SSRF/RotationAxis" $FIRSTFRAME 2>/dev/null || is_rotation_axis_set=0`
     if [ "$is_rotation_axis_set" == "1" ]; then
       rotation_axis=`h5dump -d "/SSRF/RotationAxis" $FIRSTFRAME | grep "(0):" | sed -e "s/^.*://; s/,//g"`
       echo "The rotation axis of SSRF BL02U1 (Eiger2 S 9M) is $rotation_axis"
     else
       rotation_axis="1 0 0"
       echo "SSRF BL02U1 (Eiger2 S 9M) with horizontal rotation axis"
     fi
   fi

revision 1.05 specialcase nframes for Eiger detectors at BNL

   if [ "$SN" == "E-18-0104" -o "$SN" == "E-32-0101" ]; then
     echo specialcase nframes for Eiger detectors at BNL:
     nframes=`h5dump -A -g "/entry/data" $FIRSTFRAME | grep "DATASPACE  SIMPLE" | sed -e "s/,.*//" | awk '{a+=$5}END{print a}'`
     DATA_RANGE="1 $nframes"
     SPOT_RANGE="1 `echo "scale=0; if (${nframes}<2) 1; if (${nframes}>1) ${nframes}/2"|bc -l`"
   fi
 fi

rev 1.10: check for NeXus header. If found, its geometry will overwrite any rotation_axis set until here.

 NeXus=0
 h5dump -d "/entry/definition" $FIRSTFRAME | grep -q NXmx && NeXus=1
 if [ "$NeXus" == "1" ]; then
   echo NeXus header found. This defines DIRECTION_OF_DETECTOR_X/Y-AXIS and ROTATION_AXIS.
   DIRECTION_OF_DETECTOR_X_AXIS=$(h5dump -d "/entry/instrument/detector/module/fast_pixel_direction" $FIRSTFRAME 2>/dev/null | grep "(0):" | tail -1 | sed -e "s/^.*://; s/,//g")
   DIRECTION_OF_DETECTOR_Y_AXIS=$(h5dump -d "/entry/instrument/detector/module/slow_pixel_direction" $FIRSTFRAME 2>/dev/null | grep "(0):" | tail -1 | sed -e "s/^.*://; s/,//g")
   rotation_axis=$(h5dump -a "/entry/sample/transformations/omega/vector" $FIRSTFRAME 2>/dev/null | grep "(0):" | sed -e "s/^.*://; s/,//g")
 else
   echo no NeXus header found.
 fi
 echo DATA_RANGE=$DATA_RANGE
 DETECTOR="EIGER MINIMUM_VALID_PIXEL_VALUE=0 OVERLOAD= $OVERLOAD"
 QX=`h5dump -d "/entry/instrument/detector/x_pixel_size" $FIRSTFRAME | awk '/\(0\): [0-9]/{print $2*1000}'`
 QY=`h5dump -d "/entry/instrument/detector/y_pixel_size" $FIRSTFRAME | awk '/\(0\): [0-9]/{print $2*1000}'`

 echo OVERLOAD=$OVERLOAD
 SENSOR_THICKNESS=`h5dump -d "/entry/instrument/detector/sensor_thickness" $FIRSTFRAME | awk '/\(0\): [0-9]/{print $2*1000}'`
 X_RAY_WAVELENGTH=`h5dump -d "/entry/instrument/beam/incident_wavelength" $FIRSTFRAME | awk '/\(0\): [0-9]/{print $2}'`

 NX=`h5dump -d "/entry/instrument/detector/detectorSpecific/x_pixels_in_detector" $FIRSTFRAME | awk '/\(0\): [0-9]/{print $2}'`
 NY=`h5dump -d "/entry/instrument/detector/detectorSpecific/y_pixels_in_detector" $FIRSTFRAME | awk '/\(0\): [0-9]/{print $2}'`

 # find ORGX and ORGY:
 ORGX=`h5dump -d "/entry/instrument/detector/beam_center_x" $FIRSTFRAME | awk '/\(0\): [0-9]/{print $2}'`
 ORGY=`h5dump -d "/entry/instrument/detector/beam_center_y" $FIRSTFRAME | awk '/\(0\): [0-9]/{print $2}'`

 # find DETECTOR_DISTANCE :
 DETECTOR_DISTANCE=`h5dump -d "/entry/instrument/detector/detector_distance" $FIRSTFRAME | awk '/\(0\): [0-9]/{print $2*1000}'`

 SEPMIN=4
 CLUSTER_RADIUS=2

elif [ "$DET" == "raxis" ]; then

 echo Data from a RAXIS detector

 DETECTOR="RAXIS MINIMUM_VALID_PIXEL_VALUE=0  OVERLOAD=2000000"
 #let SKIP=768
 #NX=$(od -t x -j $SKIP -N 4 $FIRSTFRAME |awk 'NR==1{print toupper($2)}'|perl -nle '@array= $_ =~/.{2}/g; print "ibase=16;obase=A;".join("",reverse @array)'|bc)
 NX=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(768);print("%.4d"%struct.unpack(">i",f.read(4)))')
 NY=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(772);print("%.4d"%struct.unpack(">i",f.read(4)))')
 DETECTOR_DISTANCE=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(344);print("-%.4f"%struct.unpack(">f",f.read(4)))')
 ORGX=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(540);print("%.4f"%struct.unpack(">f",f.read(4)))')
 ORGY=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(544);print("%.4f"%struct.unpack(">f",f.read(4)))')
 OSCILLATION_RANGE=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(524);phis,phie=struct.unpack(">ff",f.read(8));print("%.4f"%(phie-phis))')
 QX=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(776);print("%.6f"%struct.unpack(">f",f.read(4)))')
 QY=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(780);print("%.6f"%struct.unpack(">f",f.read(4)))')
 X_RAY_WAVELENGTH=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(292);print("%.6f"%struct.unpack(">f",f.read(4)))')

elif [ "$DET" == "dtrek" ]; then

 echo "Data from a RAXIS or Saturn or Pilatus detector with dTREK format"

 sed s/\;// tmp2 > tmp1
 mv tmp1 tmp2

 dname=`grep "DETECTOR_NAMES=" tmp2 | sed -e "s/.*=//"`
 flip=1
 if [ "$dname" == "CCD_" ]; then
   DETECTOR="SATURN MINIMUM_VALID_PIXEL_VALUE=1"
   dtrek_det="saturn"

   # Find rotation axis. Warning: currently not support flipping (det(tmpmat)<0)
   # I'm not sure this method is really valid - but at least mosflm seems to read this to determine rotation axis.
   tmpmat=(`grep ${dname}SPATIAL_DISTORTION_VECTORS= tmp2 | tail -1 | sed -e "s/.*=//"`)
   rotx=`echo "scale=6; -1.0*${tmpmat[0]}" | bc -l`
   roty=`echo "scale=6; -1.0*${tmpmat[1]}" | bc -l`
   rotation_axis="$rotx $roty 0"
  if [ `echo "(${tmpmat[0]}*${tmpmat[3]}-(${tmpmat[1]}*${tmpmat[2]}))/1"|bc` -lt 0 ]; then
   echo ""
   echo "WARNING!! not-supported SPATIAL_DISTORTION_VECTORS header detected."
   echo "Please report this to XDSwiki author."
   echo ""
  fi
 elif [ "$dname" == "PILT_" ]; then
   DETECTOR="PILATUS MINIMUM_VALID_PIXEL_VALUE=0"
   dtrek_det="pilatus"
   SEPMIN=3
   CLUSTER_RADIUS=1.5
   rotation_axis="0 1 0"   # TODO: read from header
   flip=-1
   SENSOR_THICKNESS=0.45   # TODO: read from header

2theta

   TWOTHETA=`awk '/PILT_GONIO_VALUES=/{print $2}' tmp2`
   echo TWOTHETA=$TWOTHETA
   R1=`echo "scale=7;  c($TWOTHETA/$DEGTOR)" | bc -l`
   R3=`echo "scale=7; -s($TWOTHETA/$DEGTOR)" | bc -l`
   DIRECTION_OF_DETECTOR_X_AXIS="$R1 0 $R3"
 elif [ "$dname" == "RX_" ]; then
   DETECTOR="RAXIS MINIMUM_VALID_PIXEL_VALUE=0"
   dtrek_det="raxis"
 else
   DETECTOR="XXX MINIMUM_VALID_PIXEL_VALUE=XXX"
   dtrek_det="unknown" 
 fi
 # find OVERLOAD
 SATURATED_VALUE=`grep SATURATED_VALUE tmp2 | head -1 | sed s/SATURATED_VALUE=//`
 DETECTOR="${DETECTOR}  OVERLOAD=${SATURATED_VALUE}"

 # find X_RAY_WAVELENGTH:
 X_RAY_WAVELENGTH=(`grep SOURCE_WAVELENGTH tmp2 | head -1 | sed s/SOURCE_WAVELENGTH=//`)
 X_RAY_WAVELENGTH=${X_RAY_WAVELENGTH[1]}

 # find NX,NY,QX,QY
 # NX,NY should be read from *_DETECTOR_DIMENSIONS?
 NX=`grep SIZE1 tmp2 | tail -1 | sed s/SIZE1=//`
 NY=`grep SIZE2 tmp2 | tail -1 | sed s/SIZE2=//`
 DET_SIZE=(`grep ${dname}DETECTOR_SIZE tmp2 | tail -1 | sed s/.*_DETECTOR_SIZE=//`)
 QX=`echo "scale=6; ${DET_SIZE[0]} / $NX" | bc -l`
 QY=`echo "scale=6; ${DET_SIZE[1]} / $NY" | bc -l`

 # find ORGX, ORGY
 SPATIAL_DISTORTION_INFO=(`grep ${dname}SPATIAL_DISTORTION_INFO tmp2 | tail -1 | sed s/.*_SPATIAL_DISTORTION_INFO=//`)
 ORGX=${SPATIAL_DISTORTION_INFO[0]}
 ORGY=${SPATIAL_DISTORTION_INFO[1]}

 # find DETECTOR_DISTANCE 
 GONIO_NAMES=(`grep ${dname}GONIO_NAMES= tmp2 | tail -1 | sed s/.*_GONIO_NAMES=//`)
 GONIO_VALUES=(`grep ${dname}GONIO_VALUES= tmp2 | tail -1 | sed s/.*_GONIO_VALUES=//`)

GONIO_UNITS=(`grep ${dname}GONIO_UNITS= tmp2 | tail -1 | sed s/.*_GONIO_UNITS=//`)

 for i in `seq 1 ${#GONIO_NAMES[*]}`
 do
  idx=$((i-1))
  if [ "${GONIO_NAMES[$idx]}" == "Distance" ]; then
   DETECTOR_DISTANCE="${GONIO_VALUES[$idx]}"
   # TODO: Check unit!! - ${GONIO_UNITS[$idx]}
  fi
  if [ $flip -gt 0 ]; then 
   DETECTOR_DISTANCE="-${GONIO_VALUES[$idx]}"
   # TODO: Check unit!! - ${GONIO_UNITS[$idx]}
   echo "using distance <0"
  fi
 done

 # find OSCILLATION_RANGE
 ROTATION=(`grep "^ROTATION=" tmp2 | tail -1 | sed s/ROTATION=//`)
 OSCILLATION_RANGE=${ROTATION[2]}

elif [ "$DET" == "MAR345" ]; then

echo  "Data from a Mar345 image plate detector"
DETECTOR="MAR345  MINIMUM_VALID_PIXEL_VALUE=0  OVERLOAD=130000"
NX=`awk '/FORMAT/{print $2}' tmp2`

next line is rev 1.02 (previously NY= $NX):

NY=`awk -v NX=$NX '/FORMAT/{print $4/NX}' tmp2`
QX=`awk '/PIXEL/{print $3/1000.}' tmp2`
QY=`awk '/PIXEL/{print $5/1000.}' tmp2`
if grep -q 'CENTER' tmp2; then
  echo Beam center found.
  ORGX=`grep 'CENTER' tmp2 | awk '{print $3}'`
  ORGY=`grep 'CENTER' tmp2 | awk '{print $5}'`
else
  echo No beam center was found. Setting beam center to the middle of the detector.
  ORGX=`echo $NX / 2 | bc`
  ORGY=`echo $NY / 2 | bc`
fi
DETECTOR_DISTANCE=`grep 'DISTANCE' tmp2 | awk '{print $2}'`               
X_RAY_WAVELENGTH=`grep 'WAVELENGTH' tmp2 | awk '{print $2}'` 
OSCILLATION_RANGE=`grep 'PHI' tmp2 | awk '{print $5-$3}'`    
TRUSTED_REGION="0 0.99"

elif [ "$DET" == "OLDMAR" ]; then

echo  "Data from old type MAR image plate detector"
DETECTOR="MAR  MINIMUM_VALID_PIXEL_VALUE=0  OVERLOAD=130000"
NX=`awk 'NR==2 {print $2}' tmp2`
NY=$NX
QX=`awk 'NR==2 {print $15}' tmp2`
QY=$QX
ORGX=`awk 'NR==2 {print $19}' tmp2`
ORGY=`awk 'NR==2 {print $20}' tmp2`
DETECTOR_DISTANCE=`awk 'NR==2 {print $22}' tmp2`               
X_RAY_WAVELENGTH=`awk 'NR==2 {print $21}' tmp2` 
OSCILLATION_RANGE=`awk 'NR==2 {print $24-$23}' tmp2`    
TRUSTED_REGION="0 0.99"   
rotation_axis="0 1 0"  
echo unsure if sign of anomalous signal is correct - please verify or try both hands!

elif [ "$DET" == "Bruker-cbf" ]; then

echo  "Data from a Bruker-cbf detector"

MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT:

MNOPIAS=6

use complete detector including corners:

TRUSTED_REGION="0 1.42"

polarization

pol_frac=`awk '/polarizn_source_ratio/{print $2}' tmp2`
pol_frac=`echo "${pol_frac}+0.5" | bc -l`
OVERLOAD=`awk '/_array_intensities.overload/{print $2}' tmp2`
DETECTOR="BRUKER MINIMUM_VALID_PIXEL_VALUE=0 OVERLOAD=${OVERLOAD}"
NX=`awk '/X-Binary-Size-Fastest-Dimension/{print $2}' tmp2`
NY=`awk '/X-Binary-Size-Second-Dimension/{print $2}' tmp2`

QX=`awk '/ELEMENT_X ELEMENT_X/{print $4}' tmp2`
QY=`awk '/ELEMENT_Y ELEMENT_Y/{print $4}' tmp2`

ORGX/Y-offsets relative to center of detector:

ORGX=`grep " 0 0 ? ? ?" tmp2 | awk '/ H /{print $2}'`
ORGY=`grep " 0 0 ? ? ?" tmp2 | awk '/ V /{print $2}'`

total ORGX/Y

ORGX=`echo "scale=2; ${NX}/2+($ORGX/$QX)" | bc -l`
ORGY=`echo "scale=2; ${NY}/2-($ORGY/$QY)" | bc -l`

DETECTOR_DISTANCE=`grep "0 0 ? ? ?" tmp2 | awk '/DX /{print $2}'`

X_RAY_WAVELENGTH=`awk '/diffrn_radiation_wavelength.wavelength/{print $2}' tmp2`

fix 16.3.20: instead of print, use printf "%.5f", because bc does not accept e.g. 3.1e-005

OMEGA=`awk '/OMEGA \? \? \?/{printf "%.5f",$5}' tmp2`
DELTAOMEGA=`awk '/OMEGA \? \? \?/{printf "%.5f",$6}' tmp2`
PHI=`awk '/PHI \? \? \?/{printf "%.5f",$5}' tmp2`
DELTAPHI=`awk '/PHI \? \? \?/{printf "%.5f",$6}' tmp2`
KAPPA=`awk '/CHI \? \? \?/{printf "%.5f",$5}' tmp2`

echo OMEGA DELTAOMEGA PHI DELTAPHI KAPPA= $OMEGA $DELTAOMEGA $PHI $DELTAPHI $KAPPA

test whether the absolute value of deltaphi is > absolute value of deltaomega

if (( $(echo "${DELTAPHI}^2 > ${DELTAOMEGA}^2"|bc -l) )); then
   echo PHI scan
   R3=`echo "scale=7; s($KAPPA/$DEGTOR)*s($OMEGA/$DEGTOR)" | bc -l`
   R1=`echo "scale=7; s($KAPPA/$DEGTOR)*c($OMEGA/$DEGTOR)" | bc -l`
   R2=`echo "scale=7; c($KAPPA/$DEGTOR)         	  " | bc -l`
   rotation_axis="$R1 $R2 $R3"
   OSCILLATION_RANGE=${DELTAPHI}
   STARTING_ANGLE=${PHI}

here we could check if DELTAPHI is <0, and if so, negate it and rotation_axis

else
   echo OMEGA scan
   rotation_axis="0 -1 0"
   OSCILLATION_RANGE=${DELTAOMEGA}
   STARTING_ANGLE=${OMEGA}

here we could check if DELTAOMEGA is <0, and if so, negate it and rotation_axis

fi
echo STARTING_ANGLE= $STARTING_ANGLE        ! only read by IDXREF

2theta

 TWOTHETA=`awk '/TWOTHETA \? \? \?/{print $5}' tmp2`
 echo TWOTHETA=$TWOTHETA
 R1=`echo "scale=7; c($TWOTHETA/$DEGTOR)" | bc -l`
 R3=`echo "scale=7; s($TWOTHETA/$DEGTOR)" | bc -l`
 DIRECTION_OF_DETECTOR_X_AXIS="$R1 0 $R3"

end of Bruker-cbf section

else

 echo should never come here
 exit 1

fi

echo ORGX= $ORGX ORGY= $ORGY - check these values with adxv ! echo DETECTOR_DISTANCE= $DETECTOR_DISTANCE ! only read by XYCORR, IDXREF echo OSCILLATION_RANGE= $OSCILLATION_RANGE ! only read by IDXREF echo X-RAY_WAVELENGTH= $X_RAY_WAVELENGTH ! only read by IDXREF

now we know everything that is required to generate XDS.INP

cat > XDS.INP << eof ! written by generate_XDS.INP version $REVISION JOB= XYCORR INIT COLSPOT IDXREF DEFPIX INTEGRATE CORRECT ORGX= $ORGX ORGY= $ORGY ! values from frame header; only read by XYCORR, IDXREF $COMMENT_ORGXY DETECTOR_DISTANCE= $DETECTOR_DISTANCE !read by XYCORR, IDXREF. Negative if detector normal points to crystal. OSCILLATION_RANGE= $OSCILLATION_RANGE STARTING_ANGLE= $STARTING_ANGLE X-RAY_WAVELENGTH= $X_RAY_WAVELENGTH NAME_TEMPLATE_OF_DATA_FRAMES=$NAME_TEMPLATE_OF_DATA_FRAMES ! REFERENCE_DATA_SET=xxx/XDS_ASCII.HKL ! e.g. to ensure consistent indexing DATA_RANGE=$DATA_RANGE SPOT_RANGE=$SPOT_RANGE ! BACKGROUND_RANGE=1 10 ! rather use defaults (first 5 degree of rotation)

SPACE_GROUP_NUMBER=0 ! 0 if unknown UNIT_CELL_CONSTANTS= 70 80 90 90 90 90 ! put correct values if known INCLUDE_RESOLUTION_RANGE=50 0 ! after CORRECT, insert high resol limit; re-run CORRECT ! IDXREF now obeys INCLUDE_RESOLUTION_RANGE and EXCLUDE_RESOLUTION_RANGE to exclude ice-rings

FRIEDEL'S_LAW=FALSE ! This acts only on the CORRECT step ! If the anom signal turns out to be, or is known to be, very low or absent, ! use FRIEDEL'S_LAW=TRUE instead (or comment out the line); re-run CORRECT

! remove the "!" in the following line: ! STRICT_ABSORPTION_CORRECTION=TRUE ! if the anomalous signal is strong: in that case, in CORRECT.LP the three ! "CHI^2-VALUE OF FIT OF CORRECTION FACTORS" values are significantly> 1, e.g. 1.5 ! ! exclude (mask) untrusted areas of detector, e.g. beamstop shadow : ! UNTRUSTED_RECTANGLE= 1800 1950 2100 2150 ! x-min x-max y-min y-max ! repeat ! UNTRUSTED_ELLIPSE= 2034 2070 1850 2240 ! x-min x-max y-min y-max ! if needed ! UNTRUSTED_QUADRILATERAL= x1 y1 x2 y2 x3 y3 x4 y4 ! see documentation ! ! parameters with changes wrt default values: TRUSTED_REGION=$TRUSTED_REGION VALUE_RANGE_FOR_TRUSTED_DETECTOR_PIXELS=6000. 30000. ! often 7000 or 8000 is ok STRONG_PIXEL=4 ! COLSPOT: only use strong reflections (default is 3) MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT=$MNOPIAS ! default of 6 is sometimes too high ! close spots/long cell axis: reduce SEPMIN and CLUSTER_RADIUS from their defaults of 7 and 3.5 SEPMIN=$SEPMIN CLUSTER_RADIUS=$CLUSTER_RADIUS ! 4 and 2 for Pixel Array Detectors ! since XDS 01-MAR-2015, POSITION supersedes DISTANCE. ! nowadays headers are usually correct so refine POSITION in INTEGRATE but not IDXREF if low to medium resolution ! however, if the spots from COLSPOT extend to 2A then POSITION could, and if 1.5A POSITION should be refined REFINE(IDXREF)=CELL BEAM ORIENTATION AXIS ! add POSITION if high resolution, or DETECTOR_DISTANCE inaccurate REFINE(INTEGRATE)= POSITION BEAM ORIENTATION ! AXIS CELL . If 1.5A or higher it is ok to refine CELL (unless electron diffraction) REFINE(CORRECT)= $REFINE_CORRECT ! parameters specifically for this detector and beamline: DETECTOR= $DETECTOR SENSOR_THICKNESS= $SENSOR_THICKNESS ! attention CCD detectors: for very high resolution (better than 1A) make sure to specify SILICON ! as about 32* what CORRECT.LP suggests (absorption of phosphor is much higher than that of silicon). ! Better: read the article http://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/SILICON . NX= $NX NY= $NY QX= $QX QY= $QY ! to make CORRECT happy if frames are unavailable

eof

if [ "$DET" == "eiger" ] && [ "$is_h5" == 1 ]; then

 if [ "$DLS" == 0 ] ; then
   if [ -e /usr/local/lib64/dectris-neggia.so ]; then
     echo LIB=/usr/local/lib64/dectris-neggia.so >> XDS.INP
     echo LIB= line was written to XDS.INP . For Apple ARM64 processors, you must modify the name.
   else
     echo !LIB=/usr/local/lib64/dectris-neggia.so >> XDS.INP
     echo /usr/local/lib64/dectris-neggia.so was not found - specify location manually!
   fi
 else
   if [ -e /usr/local/lib64/durin-plugin.so ]; then
     echo LIB=/usr/local/lib64/durin-plugin.so >> XDS.INP
     echo LIB= line was written to XDS.INP
   else
     echo !LIB=/usr/local/lib64/durin-plugin.so >> XDS.INP
     echo /usr/local/lib64/durin-plugin.so was not found - specify location manually!
   fi
 fi

fi

if [ "$DET" == "raxis" -o "$dtrek_det" == "raxis" ]; then

cat >> XDS.INP << eof

DIRECTION_OF_DETECTOR_X-AXIS=1 0 0 DIRECTION_OF_DETECTOR_Y-AXIS=0 -1 0 INCIDENT_BEAM_DIRECTION=0 0 1 ! only read by IDXREF ROTATION_AXIS=0 1 0 ! only read by IDXREF !FRACTION_OF_POLARIZATION=0.98 ! uncomment if synchrotron; only used by CORRECT POLARIZATION_PLANE_NORMAL=1 0 0 ! only used by CORRECT eof else

if [ "$rotation_axis" != "" ]; then
 echo "ROTATION_AXIS= $rotation_axis  ! only read by IDXREF" >> XDS.INP
elif [ "$REVERSE_PHI" == "no" ]; then
 echo 'ROTATION_AXIS=1 0 0  ! Australian Synchrotron, SERCAT ID-22 (?), APS 19-ID (?), ESRF BM30A, SPring-8, SSRF need -1 0 0. Diamond ID24 needs 0 -1 0' >> XDS.INP
else
 echo 'ROTATION_AXIS=-1 0 0  ! if this is wrong, please contact author.' >> XDS.INP
fi
if [ "$dtrek_det" == "saturn" ]; then
cat >> XDS.INP << eof

DIRECTION_OF_DETECTOR_X-AXIS=-1 0 0 DIRECTION_OF_DETECTOR_Y-AXIS= 0 1 0 eof

else
cat >> XDS.INP << eof

DIRECTION_OF_DETECTOR_X-AXIS=$DIRECTION_OF_DETECTOR_X_AXIS DIRECTION_OF_DETECTOR_Y-AXIS=$DIRECTION_OF_DETECTOR_Y_AXIS eof

fi
cat >> XDS.INP << eof

INCIDENT_BEAM_DIRECTION=0 0 1 ! only read by IDXREF FRACTION_OF_POLARIZATION=${pol_frac} ! better value is provided by beamline staff! POLARIZATION_PLANE_NORMAL=0 1 0 ! only used by CORRECT eof fi cat >> XDS.INP << eof !used by DEFPIX and CORRECT to exclude ice-reflections / ice rings - uncomment if necessary !fine-grained list is in Thorn et al http://journals.iucr.org/d/issues/2017/09/00/hi5647/index.html !EXCLUDE_RESOLUTION_RANGE= 3.93 3.87 !ice-ring at 3.897 Angstrom !EXCLUDE_RESOLUTION_RANGE= 3.70 3.64 !ice-ring at 3.669 Angstrom !EXCLUDE_RESOLUTION_RANGE= 3.47 3.41 !ice-ring at 3.441 Angstrom !EXCLUDE_RESOLUTION_RANGE= 2.70 2.64 !ice-ring at 2.671 Angstrom !EXCLUDE_RESOLUTION_RANGE= 2.28 2.22 !ice-ring at 2.249 Angstrom !EXCLUDE_RESOLUTION_RANGE= 2.102 2.042 !ice-ring at 2.072 Angstrom - strong !EXCLUDE_RESOLUTION_RANGE= 1.978 1.918 !ice-ring at 1.948 Angstrom - weak !EXCLUDE_RESOLUTION_RANGE= 1.948 1.888 !ice-ring at 1.918 Angstrom - strong !EXCLUDE_RESOLUTION_RANGE= 1.913 1.853 !ice-ring at 1.883 Angstrom - weak !EXCLUDE_RESOLUTION_RANGE= 1.751 1.691 !ice-ring at 1.721 Angstrom - weak ! additional ice-ring resolution ranges: 1.524 1.519, 1.473 1.470, 1.444 1.440, 1.372 1.368, 1.367 1.363, ! 1.299 1.296, 1.275 1.274, 1.261 1.259, 1.224 1.222, 1.171 1.168, 1.124 1.122 (compiled by GlobalPhasing)

eof if [ "$DET" == "Bruker-cbf" ]; then

 echo "DELPHI=15 ! refine less often than the default of 5" >> XDS.INP

elif [ "$DET" == "adsc-CMOS1" ]; then

 echo UNTRUSTED_RECTANGLE= 0 1468  2451 2631  >> XDS.INP

elif [ "$DET" == "pilatus" -o "$DET" == "eiger" ]; then cat >> XDS.INP << eof NUMBER_OF_PROFILE_GRID_POINTS_ALONG_ALPHA/BETA=13 ! Default is 9 - Increasing may improve data NUMBER_OF_PROFILE_GRID_POINTS_ALONG_GAMMA=13 ! accuracy, particularly if finely-sliced on phi, ! and does not seem to have any downsides. eof

 if [ "$NX" == "1028"  -a "$NY" == "1062" ]; then

Eiger2 1M ; v0.97 numbers from Andreas Förster

   cat >> XDS.INP << eof

UNTRUSTED_RECTANGLE= 0 1029 512 551 eof

 elif [ "$NX" == "1475" ]; then
   if ! grep -q Flat_field tmp2 ; then
   cat >> XDS.INP << eof

! the following specifications are for a detector _without_ proper ! flat_field correction; they cut away one additional pixel adjacent ! to each UNTRUSTED_RECTANGLE !EXCLUSION OF VERTICAL DEAD AREAS OF THE PILATUS 2M DETECTOR UNTRUSTED_RECTANGLE= 486 496 0 1680 UNTRUSTED_RECTANGLE= 980 990 0 1680 !EXCLUSION OF HORIZONTAL DEAD AREAS OF THE PILATUS 2M DETECTOR UNTRUSTED_RECTANGLE= 0 1476 194 214 UNTRUSTED_RECTANGLE= 0 1476 406 426 UNTRUSTED_RECTANGLE= 0 1476 618 638 UNTRUSTED_RECTANGLE= 0 1476 830 850 UNTRUSTED_RECTANGLE= 0 1476 1042 1062 UNTRUSTED_RECTANGLE= 0 1476 1254 1274 UNTRUSTED_RECTANGLE= 0 1476 1466 1486 eof

   else
   cat >> XDS.INP << eof

!EXCLUSION OF VERTICAL DEAD AREAS OF THE PILATUS 2M DETECTOR UNTRUSTED_RECTANGLE= 487 495 0 1680 UNTRUSTED_RECTANGLE= 981 989 0 1680 !EXCLUSION OF HORIZONTAL DEAD AREAS OF THE PILATUS 2M DETECTOR UNTRUSTED_RECTANGLE= 0 1476 195 213 UNTRUSTED_RECTANGLE= 0 1476 407 425 UNTRUSTED_RECTANGLE= 0 1476 619 637 UNTRUSTED_RECTANGLE= 0 1476 831 849 UNTRUSTED_RECTANGLE= 0 1476 1043 1061 UNTRUSTED_RECTANGLE= 0 1476 1255 1273 UNTRUSTED_RECTANGLE= 0 1476 1467 1485 eof

   fi
 elif [ "$NX" == "2068"  -a "$NY" == "2162" ]; then

Eiger2 4M ; v0.97 numbers from Andreas Förster

   cat >> XDS.INP << eof

!EXCLUSION OF VERTICAL DEAD AREAS OF THE EIGER2 4M DETECTOR

UNTRUSTED_RECTANGLE= 1028 1041      0 2163

!EXCLUSION OF HORIZONTAL DEAD AREAS OF THE EIGER 4M DETECTOR

UNTRUSTED_RECTANGLE=    0 2069    512  551
UNTRUSTED_RECTANGLE=    0 2069   1062 1101
UNTRUSTED_RECTANGLE=    0 2069   1612 1651

eof

 elif [ "$NX" == "2463" ]; then

Pilatus 6M
FIXME: here we could test if a Flat_field correction was applied like we do for 2M

   cat >> XDS.INP << eof

UNTRUSTED_RECTANGLE= 487 495 0 2528 UNTRUSTED_RECTANGLE= 981 989 0 2528 UNTRUSTED_RECTANGLE=1475 1483 0 2528 UNTRUSTED_RECTANGLE=1969 1977 0 2528 UNTRUSTED_RECTANGLE= 0 2464 195 213 UNTRUSTED_RECTANGLE= 0 2464 407 425 UNTRUSTED_RECTANGLE= 0 2464 619 637 UNTRUSTED_RECTANGLE= 0 2464 831 849 UNTRUSTED_RECTANGLE= 0 2464 1043 1061 UNTRUSTED_RECTANGLE= 0 2464 1255 1273 UNTRUSTED_RECTANGLE= 0 2464 1467 1485 UNTRUSTED_RECTANGLE= 0 2464 1679 1697 UNTRUSTED_RECTANGLE= 0 2464 1891 1909 UNTRUSTED_RECTANGLE= 0 2464 2103 2121 UNTRUSTED_RECTANGLE= 0 2464 2315 2333 eof

 elif [ "$NX" == "3110"  -a "$NY" == "3269" ]; then

Eiger 9M

   cat >> XDS.INP << eof

!EXCLUSION OF VERTICAL DEAD AREAS OF THE EIGER 9M DETECTOR UNTRUSTED_RECTANGLE= 1029 1042 0 3269 UNTRUSTED_RECTANGLE= 2069 2082 0 3269 !EXCLUSION OF HORIZONTAL DEAD AREAS OF THE EIGER 9M DETECTOR UNTRUSTED_RECTANGLE= 0 3110 513 553 UNTRUSTED_RECTANGLE= 0 3110 1064 1104 UNTRUSTED_RECTANGLE= 0 3110 1615 1655 UNTRUSTED_RECTANGLE= 0 3110 2166 2206 UNTRUSTED_RECTANGLE= 0 3110 2717 2757 eof

 elif [ "$NX" == "3108"  -a "$NY" == "3262" ]; then

Eiger2 9M ; v0.97 numbers from Andreas Förster

   cat >> XDS.INP << eof

!EXCLUSION OF VERTICAL DEAD AREAS OF THE EIGER2 9M DETECTOR UNTRUSTED_RECTANGLE= 1028 1041 0 3262 UNTRUSTED_RECTANGLE= 2068 2081 0 3263 !EXCLUSION OF HORIZONTAL DEAD AREAS OF THE EIGER2 9M DETECTOR UNTRUSTED_RECTANGLE= 0 3109 512 551 UNTRUSTED_RECTANGLE= 0 3109 1062 1101 UNTRUSTED_RECTANGLE= 0 3109 1612 1651 UNTRUSTED_RECTANGLE= 0 3109 2162 2201 UNTRUSTED_RECTANGLE= 0 3109 2712 2751 eof

 elif [ "$NX" == "4150" -a "$NY" == "4371" ]; then

Eiger 16M

   cat >> XDS.INP << eof

!EXCLUSION OF HORIZONTAL DEAD AREAS OF THE EIGER 16M DETECTOR + ONE PIXEL ON EACH SIDE

UNTRUSTED_RECTANGLE=    0 4150    513  553
UNTRUSTED_RECTANGLE=    0 4150   1064 1104
UNTRUSTED_RECTANGLE=    0 4150   1615 1655
UNTRUSTED_RECTANGLE=    0 4150   2166 2206
UNTRUSTED_RECTANGLE=    0 4150   2717 2757
UNTRUSTED_RECTANGLE=    0 4150   3268 3308
UNTRUSTED_RECTANGLE=    0 4150   3819 3859

!EXCLUSION OF VERTICAL DEAD AREAS OF THE EIGER 16M DETECTOR + ONE PIXEL ON EACH SIDE

UNTRUSTED_RECTANGLE= 1029 1042      0 4371
UNTRUSTED_RECTANGLE= 2069 2082      0 4371 
UNTRUSTED_RECTANGLE= 3109 3122      0 4371

eof

 elif [ "$NX" == "4148" -a "$NY" == "4362" ]; then

Eiger2 16M ; v0.97 numbers from Andreas Förster

   cat >> XDS.INP << eof

!EXCLUSION OF HORIZONTAL DEAD AREAS OF THE EIGER2 16M DETECTOR

UNTRUSTED_RECTANGLE= 1028 1041      0 4363
UNTRUSTED_RECTANGLE= 2068 2081      0 4363
UNTRUSTED_RECTANGLE= 3108 3121      0 4363

!EXCLUSION OF VERTICAL DEAD AREAS OF THE EIGER2 16M DETECTOR

UNTRUSTED_RECTANGLE=    0 4149    512  551
UNTRUSTED_RECTANGLE=    0 4149   1062 1101
UNTRUSTED_RECTANGLE=    0 4149   1612 1651
UNTRUSTED_RECTANGLE=    0 4149   2162 2201
UNTRUSTED_RECTANGLE=    0 4149   2712 2751
UNTRUSTED_RECTANGLE=    0 4149   3262 3301
UNTRUSTED_RECTANGLE=    0 4149   3812 3851

eof

fi

fi echo XDS.INP is ready for use. The file has only the most important keywords. echo Full documentation, including complete detector templates, at xds.mr.mpg.de . echo More documentation in strucbio.biologie.uni-konstanz.de/xdswiki/index.php . echo After running xds, inspect at least the agreement of predicted and observed echo spots in FRAME.cbf! rm -f tmp1 tmp2

System-wide or personal installation

Ask your system adminstrator to cut-and-paste the script into e.g. /usr/local/bin/generate_XDS.INP, and to make it "executable".

But you may also cut-and-paste the script from this webpage into a file in e.g. your home directory; the filename should be generate_XDS.INP. After creating the file, make it executable - e.g. if it's in your $HOME, use:

chmod +x ~/generate_XDS.INP

After that, you can just run it in a similar way as if it were installed in your $PATH:

~/generate_XDS.INP "frms/mydata_1_???.img"

By using your own file, you can easily update to the latest revision, or even change the script, without having to bother the system administrator.

See also Generate_XDS.INP#Dependencies below, and the Installation article.

Copying generate_XDS.INP from XDSwiki webserver

On Linux:

wget https://strucbio.biologie.uni-konstanz.de/pub/linux_bin/generate_XDS.INP
chmod a+x generate_XDS.INP

On Mac:

curl -o generate_XDS.INP https://strucbio.biologie.uni-konstanz.de/pub/linux_bin/generate_XDS.INP
chmod a+x generate_XDS.INP

Obtaining generate_XDS.INP from this webpage

Instead of cutting-and-pasting the lines of the script, you (or the system administrator) could just cut-and-paste the following four lines

 wget http://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/generate_XDS.INP -O - | \
   sed -e s/\&nbsp\;/\ /g -e s/\&gt\;/\>/g -e s/\&lt\;/\</g -e s/amp\;//g -e s/\&quot\;/\"/g -e s/\&\#\1\6\0\;/\ /g | \
   sed '/# end of generate_XDS.INP/,$d' | awk '/^#/,/rm -f tmp1 tmp2/' > generate_XDS.INP
 chmod +x generate_XDS.INP

to copy the script from this website into an executable file generate_XDS.INP in your current directory. On a Mac (which does not seem to have wget), one could try

 curl -L -o - http://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/generate_XDS.INP | \
   sed -e s/\&nbsp\;/\ /g -e s/\&gt\;/\>/g -e s/\&lt\;/\</g -e s/amp\;//g -e s/\&quot\;/\"/g -e s/\&\#\1\6\0\;/\ /g | \
   sed '/# end of generate_XDS.INP/,$d' | awk '/^#/,/rm -f tmp1 tmp2/' > generate_XDS.INP
 chmod +x generate_XDS.INP

If you do use cut-and-paste from the webpage, be aware of the following problem report: On the Mac, after loading frames, by clicking “generate XDS.INP”, the program gives some strange symbol “Ã” in XDS.INP. And the more you click “save” button, the more “Ã” appear. This looks like e.g.
SPACE_GROUP_NUMBER=0 Ã ! 0 if unknown
UNIT_CELL_CONSTANTS= 70 80 90 90 90 90 Ã
The problem is due to the “Rich text” format in TextEdit when saving "generate_XDS.INP". It is solved by re-downloading the script, and changing format to Plain - everything should work then.

Calling generate_XDS.INP from a Python script

It is recommended to use the subprocess.Popen() module instead of os.system():

 subprocess.Popen(["generate_XDS.INP",imagepath],stdout=outputfile)

where imagepath is a string containing the path to an image and outputfile is either a chosen variable for an output file or subprocess.PIPE if you're not interested in the output of the script.

The module os.system() internally uses /bin/sh to execute the command and overrides #!/bin/bash at the beginning of the script. While this is not a problem on most operating systems, /bin/sh points to dash instead of bash on Ubuntu, which leads to a program crash with the error message

 sh: 1: Syntax error: Bad fd number

Dependencies

The script makes use of many GNU commands, like ls, grep, egrep, awk, cut, cat, echo, wc, bc, head, sed, tail, cp, od, python. Some of them (like od and python) are only used in case of specific detectors (MarCCD and RAXIS, respectively). The script will only work if all the required commands are available. They reside in either the coreutils RPM, or specific RPMs (gawk, sed, bc, grep, python ...). Please note that to get the strings command on some Linux distributions (e.g. FC23) you need to install the binutils RPM package. For Eiger data processing, the h5dump program must be installed. This is part of hdf5-tools (Ubuntu) or hdf5 (RHEL). The .h5 files collected at Diamond Light Source require a very new version of h5dump (namely h5dump 1.10) to extract the OVERLOAD parameter from the .h5 file; this version is available by default in Ubuntu 20.04 and RHEL/CentOS 8.

On Mac OS X, installation of the "Command Line Tools" (from http://developer.apple.com/downloads; requires Apple ID) is required (open a terminal and type xcode-select --install). These are also part of the (larger, but also free) Xcode package. This package comes with a license that has to be accepted by the user when running a Command Line Tool (e.g. strings) for the first time.

One way to check for missing programs is

#!/bin/bash
for i in  ls grep egrep awk cut cat echo wc bc head sed tail cp od python strings h5dump ; do 
    if [ ! -x /bin/$i ] && [ ! -x /usr/bin/$i ]; then 
      echo $i not found
    fi
done

A command that should go a long way in providing all these tools for RedHat-derived distros is (as root)

yum -y install coreutils binutils gawk sed bc grep python hdf5

and for Ubuntu this would be (untested!)

sudo apt-get install coreutils binutils gawk sed bc grep python hdf5-tools

Limitations

The script tries to interpret the header of the frames, so is currently limited to Dectris (Pilatus, Eiger), ADSC (Quantum), Rigaku (several types), MAR (CCD and image plate) detectors, and one Bruker detector. Other detectors need some values to be manually filled into XDS.INP - the relevant places are marked with XXX. These are detector properties (type, pixel size and number, min and max counts in a pixel), and experimental parameters like ROTATION_AXIS, OSCILLATION_RANGE, X-RAY_WAVELENGTH, DETECTOR_DISTANCE, and XORG, YORG. For fine-tuning of detector parameters, see the detector-specific templates.
The authors have made a "best effort" to provide a XDS.INP that results in the correct sign of the anomalous signal. In the case of one detector type (internally called Rigaku SMV) this requires reversal of one detector axis, and a negative DETECTOR_DISTANCE, as is found in some of the detector-specific templates. For an unusual or unknown detector setup, the correct sign of the anomalous signal needs to be established and verified e.g. with a good dataset from a test crystal that has an anomalous signal. The authors do not take any responsibility for problems arising from incorrect sign of the anomalous signal, nor - obviously! - for any other mischief arising in or from data processing.
At some beamlines, the ROTATION_AXIS should be -1 0 0 ("backwards") instead of the usual 1 0 0 ("horizontal"), or even 0 1 0 ("vertical") like at one of the PETRA Hamburg BLs. The frame headers do not have this information, so the default chosen by generate_XDS.INP may be wrong and need manual correction. Pls also see the article Beamline notes.
Sometimes, the x- and y- values of the primary beam position recorded in the header should be used for ORGY and ORGX (i.e reversed) instead of as ORGX and ORGY. For ADSC, this has been implemented in the script for a number of beamlines.
there are apparently several flavours of HDF5 files produced at Diamond Light Source. They differ e.g. in the naming of the header items. This means that items like NX, NY, DETECTOR_DISTANCE and number of images cannot always be determined by the generate_XDS.INP script. Example: The data at /dls/i04-1/data/2021/mx28114-9/processing/Lenye_Diamini/ThiL/ThiL found during the CCP4 2021 online Cape Town workshop. A workaround is to use e.g. xia2 pipeline=3dii to process these files, and - if needed - extract those items from its output files, e.g. from DEFAULT/NATIVE/SWEEP1/index/XDS.INP .

Known bugs

For data collected with the Eiger detector at beamline X06SA @ SLS @ PSI (Switzerland), the script produces ROTATION_AXIS=-1 0 0 whereas it should be 1 0 0. Unfortunately the master.h5 written at X06SA has no detector_number so there is no easy fix at the moment.

Generate XDS.INP: Difference between revisions

Revision as of 17:25, 7 March 2022

Contents

Usage

The script

System-wide or personal installation

Copying generate_XDS.INP from XDSwiki webserver

Obtaining generate_XDS.INP from this webpage

Calling generate_XDS.INP from a Python script

Dependencies

Limitations

Known bugs

See also

Navigation menu

Generate XDS.INP: Difference between revisions

Revision as of 17:25, 7 March 2022

Usage

The script

System-wide or personal installation

Copying generate_XDS.INP from XDSwiki webserver

Obtaining generate_XDS.INP from this webpage

Calling generate_XDS.INP from a Python script

Dependencies

Limitations

Known bugs

See also

Navigation menu

Search