XDSwiki - User contributions [en]

Generate XDS.INP

2016-04-13T12:49:32Z

Karine: /* System-wide or personal installation */ it is, I had a user today who had trouble using the script because of a missing license agreement

This script generates XDS.INP based on a list of frame names supplied on the commandline. It currently works for MarCCD, ADSC, Pilatus and a few Rigaku detectors; since this is just a bash script, extension to other detectors is very easy.

The [[Eiger]] detector is not yet implemented in the script, but this will eventually come.

== Usage ==

Usage is just (don't forget the quotation marks!):
generate_XDS.INP "/home/myname/frms/mydata_1_???.img"
XDS [http://strucbio.biologie.uni-konstanz.de/~dikay/XDS_html_doc/html_doc/xds_parameters.html#NAME_TEMPLATE_OF_DATA_FRAMES= supports] bzip2-ed frames. Thus, when specifying the frame name parameter of the script, you should leave out any .bz2 extension.

For improved interaction with [[XDSGUI]], it is advantageous to use an ''absolute'' filename - one that starts with a slash ("/").

== The script ==

<pre>
#!/bin/bash
# purpose: generate XDS.INP
#
# tested with some datasets from ALS, SSRL, SLS, ESRF, BESSY, SPring-8 and PF; only MAR, ADSC/SMV, PILATUS, RAXIS (in-house) detectors;
# for other detectors, values marked with XXX must be manually filled in.
#
# revision 0.03 . Kay Diederichs 2/2010
# revision 0.04 . Kay Diederichs 4/2010 - include alternative ORGX, ORGY calculations for ADSC
# revision 0.05 . Kay Diederichs 5/2010 - grep for "Corrected" in addition to "marccd"; needed for BESSY
# revision 0.06 . KD 6/2010 - add UNTRUSTED_RECTANGLE and UNTRUSTED_ELLIPSE; use `whereis catmar` and so on
# revision 0.07 . KD 6/2010 - decide about ORGX/Y info in MAR header being pixels or mm; other fixes
# revision 0.08 . KD 6/2010 - fixes for Pilatus 6M
# revision 0.09 . KD 6/2010 - get rid of requirement for mccd_xdsparams.pl and/or catmar; rather use "od"
# revision 0.10 . Tim Gruene 7/2010 - set link 'images' to image directory if path exceeds 72 characters
# revision 0.11 . KD 7/2010 - for MarCCD: look for distance info at different byte position
# revision 0.12 . KD 7/2010 - fix for negative PHISTART
# revision 0.13 . KD 8/2010 - store correct NX NY QX QY in XDS.INP
# revision 0.14 . KD 1/2011 - SENSOR_THICKNESS for Pilatus; MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT=3
# revision 0.15 . KD 2/2011 - add comment for -ive sign of APS 19-ID and Australian Synchrotron rotation axis
# revision 0.16 . KD 3/2011 - SENSOR_THICKNESS=0.01 for ADSC and MarCCD. Add comment about SILICON=
# revision 0.17 . KD 3/2011 - make it work for .bz2 frames; improve screen output
# revision 0.18 . KD 4/2011 - faster by doing "strings" only once; revert "images/${1##/*/}" "correction"
# revision 0.19 . KD 6/2011 - bugfix for 0.18
# revision 0.20 . KD 7/2011 - redirect stderr of /bin/ls to /dev/null
# revision 0.21 . KD 11/2011 - SEPMIN, CLUSTER_RADIUS hints; read NX NY from header (for Pilatus 2M)
# revision 0.22 . KD 12/2011 - Pilatus 2M UNTRUSTED_RECTANGLE lines, SENSOR_THICKNESS from header
# revision 0.23 . KD 1/2012 - add UNTRUSTED_QUADRILATERAL, remove MINIMUM_ZETA (0.05 is default now)
# revision 0.24 . KD 3/2012 - remove revision 0.10 since XDS now takes much longer paths
# revision 0.25 . KD 3/2012 - remove revision 0.22 for PSI Pilatus 2M; see http://www.globalphasing.com/autoproc/wiki/index.cgi?TroubleShootingKnownIssues
# revision 0.26 . KD 7/2012 - Mac-compatibility: replace od flags --skip-bytes= and --read-bytes= with -j and -N (thanks to Oliver Clarke for working this out!)
# revision 0.27 . KD 11/2012 - EXCLUDE_RESOLUTION_RANGE lines and generic Pilatus Flat_field test
# revision 0.28 . Keitaro 11/2012 - for MarCCD: read oscillation range from the position 1024+736 (fix for omega rotation)
# revision 0.29 . KD 1/2013 - include UNTRUSTED_RECTANGLEs for Pilatus 6M; never hurts but needed if the beamline software does not mark them with -2 or such
# revision 0.30 . Keitaro 3/2013 - for ADSC: write all possible beam center conventions in XDS.INP as comments
# revision 0.31 . Keitaro 3/2013 - add comment for reversed phi for SPring-8
# revision 0.32 . Keitaro 3/2013 - add RAXIS support. only tested with RAXIS IV++ and VII.
# revision 0.33 . Keitaro 5/2013 - automatically set ROTATION_AXIS=-1 0 0 for SPring-8 BL32XU/41XU/44XU beamlines based on detector serial numbers.
# revision 0.34 . Keitaro 5/2013 - recognize ADSC detectors in Photon Factory and choose correct beam center convention based on detector serial numbers.
# revision 0.35 . KD 6/2013 - reduce 7000 to 6000 for shadow detection; insert comment about *_RESOLUTION_RANGE lines
# revision 0.36 . KD 6/2013 - insert NUMBER_OF_PROFILE_GRID* lines for Pilatus (suggested by C.Vonrhein)
# revision 0.37 . Keitaro 10/2013 - fix for MX225HS detector on SPring-8 BL32XU (Ignore case when matching marccd in header); but cannot decide reverse-phi because of missing detector S/N
# revision 0.38 . KD 2/2014 - change defaults for REFINE(IDXREF) and REFINE(INTEGRATE) such that more stable results are obtained for difficult datasets
# revision 0.39 . Keitaro 4/2014 - automatically set ROTATION_AXIS=-1 0 0 for MX225HS at SPring-8 BL32XU.
# revision 0.40 . Jan Gebauer /KD 4/2014 - simple implementation of MAR345 detector
# revision 0.41 . recognize header starting with R-AXIS instead of RAXIS
# revision 0.42 . Keitaro 5/2014 automatically set ROTATION_AXIS=-1 0 0 for Q315 at SPring-8 BL38B1.
# revision 0.43 . Keitaro 5/2014 add experimental support of dTREK format (raxis_smv)
# revision 0.44 . Keitaro 5/2014 automatically set ROTATION_AXIS=-1 0 0 for PILATUS3 at SPring-8 BL41XU.
# revision 0.45 . KD cope with blanks in filenames
# revision 0.46 . Keitaro 6/2014 automatically set ROTATION_AXIS=-1 0 0 for Mar225 at SPring-8 BL26B2.
# revision 0.47 . Keitaro 7/2014 more generic dTREK format support (Saturn and RAXIS)
# revision 0.48 . Kip Guja 11/2014 add detector serial number for ALS 5.0.2 to beam center convention 1
# revision 0.49 . Nobuhisa 2/2015 add detector serial number for AichiSR BL2S1 to beam center convention 1
# revision 0.50 . KD 03/2015 workaround for Mar-1 change of parameter name "DISTANCE" to "POSITION" in REFINE(*) keywords
# revision 0.51 . Keitaro 03/2015 add .gz and .xz support and remove limitation - frame numbers can start with any.
# revision 0.52 . Keitaro 05/2015 fix ADSC beam center convention for SPring-8 and DET_SN acquisition for PILATUS (didn't work on Mac)
# revision 0.53 . KD add ADSC beam center convention for APS Argonne but only as commented line in XDS.INP (not detector serial no)
# revision 0.54 . KD add ADSC S/N 911 for APS Argonne, and fix spurious output arising from THETADISTANCE (?!)
# revision 0.55 . KD add ADSC S/N 446 for APS, and check w/ 12 datasets from data.sbgrid.org. No rule found: S/N 916 @ APS 24_ID_E !
# revision 0.56 . Keitaro 12/2015 show error message when user's input didn't match any files
# revision 0.57 . KD 12/2015 start to take care of vertical ROTATION_AXIS at Diamond I24 - for now only introduce comment
# revision 0.58 . Keitaro 01/2016 fix for dTREK image: take 'rotation axis' information from header
# revision 0.59 . KD 04/04/2016 check for ADSC detector _after_ checking for dtrek detector, to correct wrong choice for https://zenodo.org/record/45756
# revision 0.60 . KD 04/04/2016 ADSC detector SN=458 at APS 19-ID has reverse phi (https://zenodo.org/record/45756)
# revision 0.61 . Keitaro 10/04/2016 Add Eiger hdf5 support (may be incomplete; UNTRUSTED_RECTANGLE=s not set) NEED h5dump.
# revision 0.62 . Keitaro 11/04/2016 Can give foo_master.h5 instead of foo_??????.h5.
# revision 0.63 . Keitaro 13/04/2016 Set UNTRUSTED_RECTANGLE=s for EIGER 9M.
REVISION="0.63 (13-Apr-2016)"
#
# usage: e.g. generate_XDS.INP "/file/system/frms/mydata_1_???.img"
# make sure to have the two quotation marks !
# the ? are wildcards for the frame numbers.
#
# known problems:
# - for ADSC detectors, there are at least three ways to obtain ORGX and ORGY values from the header (see below);
# - the same might be a problem for MAR headers, too (not sure about this)
# - on Mac OS X, Xcode (from http://developer.apple.com/tools/xcode) might be needed - not sure about this
#
# notes for debugging of the script:
# - add the -v option to the first line, to see where an error occurs
# - comment out the removal of tmp1 and tmp2 in the last line
#
# ====== Start of script ======
echo generate_XDS.INP version $REVISION . Obtain the latest version from
echo http://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/generate_XDS.INP
if [ "$1" == "help" ] || [ "$1" == "-help" ] || [ "$1" == "-h" ]; then
echo usage: generate_XDS.INP \"/file/system/frms/mydata_1_???.img\" $_with_ the quotation marks!$
echo if the frames are compressed with bzip2, leave out the .bz2 extension!
exit
fi
#
# defaults:
#
DETECTOR="XXX MINIMUM_VALID_PIXEL_VALUE=XXX OVERLOAD=XXX"
REVERSE_PHI="no"
ORGX=XXX
ORGY=XXX
DETECTOR_DISTANCE=XXX
OSCILLATION_RANGE=XXX
X_RAY_WAVELENGTH=XXX
QX=XXX
QY=XXX
NX=XXX
NY=XXX
SENSOR_THICKNESS=0
TRUSTED_REGION="0.0 1.2 ! partially use corners of detector (0 1.4143: use all pixels)"
# see how we are called:
NAME_TEMPLATE_OF_DATA_FRAMES="$1"
# list frames matching the wildcards in NAME_TEMPLATE_OF_DATA_FRAMES
# don't accept the "direct beam" shot at SLS/Pilatus PX-I and PX-II
# cope with blanks in directory / file name
IFS=$'\n'
/bin/ls -C1 $1 $1.bz2 $1.gz $1.xz 2>/dev/null | egrep -v "_00000.cbf|_000.img" > tmp1
if [ ! -s tmp1 ]
then
echo "Error! No files matched: $1"
rm -f tmp1
exit 1
fi
unset IFS
# we can continue - the frames are found

if echo $NAME_TEMPLATE_OF_DATA_FRAMES | grep '_master.h5$' > /dev/null; then
NAME_TEMPLATE_OF_DATA_FRAMES=`echo "$NAME_TEMPLATE_OF_DATA_FRAMES" | sed -e 's/_master.h5$/_??????.h5/'`
else
# Find the first '?' position and the number of '?' to determine DATA_RANGE=.
pos1=`echo "$1" | awk '{print index($0, "?")}'`
pos2=`echo "$1" | sed -e "s/[^\?]//g" | awk '{print length+'$pos1' - 1'}`
data_first=`cat tmp1 | cut -b $pos1-$pos2 | head -n1|bc`
data_last=`cat tmp1 | cut -b $pos1-$pos2 | tail -n1|bc`
DATA_RANGE="$data_first $data_last"

# set SPOT_RANGE to first half of DATA_RANGE
data_num=`wc -l tmp1 | awk '{print $1}'`
data_half=`echo "scale=0; $data_num/2" | bc -l`
data_half=`echo "if ($data_half<=1) 1;if ($data_half>1) $data_half" | bc -l`
spot_last=`echo "scale=0; $data_first+$data_half-1" | bc -l`
SPOT_RANGE="$data_first $spot_last"
fi

FIRSTFRAME=`head -1 tmp1`
echo $FIRSTFRAME | grep "\.h5$" && is_h5=1 || is_h5=0

if [ "$is_h5" == 1 ]; then
nframes=`h5dump -A -g "/entry/data" $FIRSTFRAME | grep "DATASPACE SIMPLE" | sed -e "s/,.*//" | awk '{a+=$5}END{print a}'`
DATA_RANGE="1 $nframes"
SPOT_RANGE="1 `echo "scale=0; ${nframes}/2"|bc -l`"
fi

echo DATA_RANGE=$DATA_RANGE

# find out detector type
DET=XXX
IFS=$'\n'
echo $FIRSTFRAME | grep -q '\.bz2$' && bzcat $FIRSTFRAME > tmp1 && FIRSTFRAME=tmp1
# for mac/linux compatibility. zcat foo.gz doesn't work on mac.
echo $FIRSTFRAME | grep -q '\.gz$' && zcat < $FIRSTFRAME > tmp1 && FIRSTFRAME=tmp1
echo $FIRSTFRAME | grep -q '\.xz$' && xzcat $FIRSTFRAME > tmp1 && FIRSTFRAME=tmp1

unset IFS
if [ "$is_h5" == 0 ]; then
strings $FIRSTFRAME > tmp2
# TODO: whenever FIRSTFRAME is used below, it should be copied to tmp2 (using IFS as above), and tmp2 should be used instead
# this was done for "mccd", but still has to be done for the "raxis" detector types
# the reason is that FIRSTFRAME may contain a blank, which makes some commands fail
egrep -qi 'marccd|Corrected' tmp2 && DET=mccd
grep -q PILATUS tmp2 && DET=pilatus
head -n1 tmp2 | grep -q "^RAXIS" && DET=raxis
head -n1 tmp2 | grep -q "^R-AXIS" && DET=raxis
grep -q "^SOURCE_WAVELENGTH= *1" tmp2 && DET=dtrek
grep -q BEAM_CENTER_X tmp2 && DET=adsc
grep -q mar345 tmp2 && DET=MAR345
else
h5dump -d "/entry/instrument/detector/description" $FIRSTFRAME | grep -i Eiger > /dev/null && DET=eiger
fi

# identify other detector types in the same way

# parse ASCII header of first frame

if [ "$DET" == "XXX" ]; then
echo "this is not a MAR, ADSC/SMV or PILATUS detector - fill in XXX values manually!"
DETECTOR="XXX MINIMUM_VALID_PIXEL_VALUE=XXX OVERLOAD=XXX"

# find parameters of first frame
elif [ "$DET" == "mccd" ]; then
echo Data from a MarCCD detector

DETECTOR="CCDCHESS MINIMUM_VALID_PIXEL_VALUE= 1 OVERLOAD= 65500"
SENSOR_THICKNESS=0.01
# use first frame of dataset to obtain parameters

# Check detector serial number and recognize beamline for reversed-phi setting.
# Known detectors for reversed-phi in SPring-8: 24: BL26B2 Mar225, 31: BL32XU MX225HE, 38: BL44XU MX225HE, 42: BL44XU MX300HE, 40: BL41XU MX225HE, 106: BL32XU MX225HS
REVERSEPHI_SNs="
24
31
38
40
42
106
"
# get detector serial number and check if it is included in the list
DET_SN=`grep "Detector Serial Number =" tmp2 | sed "s/Detector Serial Number = //"`
if echo "${DET_SN}${REVERSEPHI_SNs}" | sort | uniq -d | grep [0-9] > /dev/null; then
REVERSE_PHI="yes"
fi

# offsets are documented; values can be found in mccd_xdsparams.pl script
IFS=$'\n'
cp $FIRSTFRAME tmp2
unset IFS
let SKIP=1024+80
NX=$(od -t dI -j $SKIP -N 4 tmp2 | head -1 | awk '{print $2}')
let SKIP=$SKIP+4
NY=$(od -t dI -j $SKIP -N 4 tmp2 | head -1 | awk '{print $2}')

let SKIP=1720
DETECTOR_DISTANCE=$(od -t dI -j $SKIP -N 4 tmp2 | head -1 | awk '{print $2}')
DETECTOR_DISTANCE=`echo "scale=3; $DETECTOR_DISTANCE/1000" | bc -l`

let SKIP=1024+256+128+256+4
ORGX=$(od -t dI -j $SKIP -N 4 tmp2 | head -1 | awk '{print $2}')
ORGX=`echo "scale=2; $ORGX/1000" | bc -l `
let SKIP=$SKIP+4
ORGY=$(od -t dI -j $SKIP -N 4 tmp2 | head -1 | awk '{print $2}')
ORGY=`echo "scale=2; $ORGY/1000" | bc -l `
# FIXME 25.5.2015: reverse ORGX and ORGY for BM14 @ ESRF (no serial number in header)

let SKIP=1024+736
OSCILLATION_RANGE=$(od -t dI -j $SKIP -N 4 tmp2 | head -1 | awk '{print $2}')
OSCILLATION_RANGE=`echo "scale=3; $OSCILLATION_RANGE/1000" | bc -l`

let SKIP=1024+256+128+256+128+4
QX=$(od -t dI -j $SKIP -N 4 tmp2 | head -1 | awk '{print $2}')
QX=`echo "scale=10; $QX/1000000" |bc -l `
let SKIP=$SKIP+4
QY=$(od -t dI -j $SKIP -N 4 tmp2 | head -1 | awk '{print $2}')
QY=`echo "scale=10; $QY/1000000" |bc -l `

let SKIP=1024+256+128+256+128+128+12
X_RAY_WAVELENGTH=$(od -t dI -j $SKIP -N 4 tmp2 | head -1 | awk '{print $2}')
X_RAY_WAVELENGTH=`echo "scale=5; $X_RAY_WAVELENGTH/100000" | bc -l`

# at most BLs, ORGX and ORGY are in pixels, but sometimes in mm ... guess:
NXBYFOUR=`echo "scale=0; $NX/4" | bc -l `
ORGXINT=`echo "scale=0; $ORGX/1" | bc -l `
if [ $ORGXINT -lt $NXBYFOUR ]; then
ORGX=`echo "scale=1; $ORGX/$QX" | bc -l`
ORGY=`echo "scale=1; $ORGY/$QY" | bc -l`
echo MARCCD detector: header ORGX, ORGY seem to be in mm ... converting to pixels
else
echo MARCCD detector: header ORGX, ORGY seem to be in pixel units
fi

elif [ "$DET" == "adsc" ]; then

DETECTOR="ADSC MINIMUM_VALID_PIXEL_VALUE= 1 OVERLOAD= 65000"
echo Data from ADSC detector. Obtaining ORGX, ORGY depends on beamline setup:
SENSOR_THICKNESS=0.01
sed s/\;// tmp2 > tmp1
mv tmp1 tmp2

# find X_RAY_WAVELENGTH:
X_RAY_WAVELENGTH=`grep WAVELENGTH tmp2 | head -1 | sed s/WAVELENGTH=//`

# find NX, QX, ORGX and ORGY:
NX=`grep SIZE1 tmp2 | tail -1 | sed s/SIZE1=//`
QX=`grep PIXEL_SIZE tmp2 | sed s/PIXEL_SIZE=//`
# FIXME - next 2 lines should be done properly, from header
NY=$NX
QY=$QX
BEAM_CENTER_X=`grep BEAM_CENTER_X tmp2 | sed s/BEAM_CENTER_X=//`
BEAM_CENTER_Y=`grep BEAM_CENTER_Y tmp2 | sed s/BEAM_CENTER_Y=//`
# fix 2010-04-26 - tell user about possible ORGX, ORGY alternatives -
COMMENT_ORGXY="
! Following are possible beam center interpretations for ADSC detectors"
# at ESRF, PF, ALS 5.0.2 and ... (pls fill in!) the following should be used:
ORGX1=`echo "scale=1; $BEAM_CENTER_Y/$QX" | bc -l`
ORGY1=`echo "scale=1; $BEAM_CENTER_X/$QX" | bc -l`
echo - at ESRF, PF, APS Argonne BLs use: ORGX=$ORGX1 ORGY=$ORGY1
COMMENT_ORGXY="${COMMENT_ORGXY}
! ORGX= $ORGX1 ORGY= $ORGY1 ! For ESRF, PF, APS Argonne ..."
# this 2nd alternative convention should be used at the following beamlines (pls complete the list): ALS 5.0.3, ...
ORGX2=`echo "scale=1; $NX-$BEAM_CENTER_X/$QX" | bc -l `
ORGY2=`echo "scale=1; $BEAM_CENTER_Y/$QX" | bc -l `
echo - at e.g. ALS 5.0.3 use: ORGX=$ORGX2 ORGY=$ORGY2
COMMENT_ORGXY="${COMMENT_ORGXY}
! ORGX= $ORGX2 ORGY= $ORGY2 ! For ALS 5.0.3,.."
# this 3rd alternative convention should be used at the following beamlines (pls complete the list): ALS 8.2.2, ...
# this alternative is written into the generated XDS.INP ! You have to correct this manually in XDS.INP, or adjust this script.
ORGX3=`echo "scale=1; $BEAM_CENTER_X/$QX" | bc -l `
ORGY3=`echo "scale=1; $NX-$BEAM_CENTER_Y/$QX" | bc -l `
echo - at e.g. ALS 8.2.2 use: ORGX=$ORGX3 ORGY=$ORGY3 - this is written to XDS.INP if beamline is not detected
COMMENT_ORGXY="${COMMENT_ORGXY}
! ORGX= $ORGX3 ORGY= $ORGY3 ! For ALS 8.2.2,.."
# this 4th alternative convention should be used at the following beamlines (pls complete the list): SPring-8, ...
ORGX4=`echo "scale=1; $BEAM_CENTER_X/$QX" | bc -l `
ORGY4=`echo "scale=1; $BEAM_CENTER_Y/$QX" | bc -l `
echo - at e.g. SPring-8 use: ORGX=$ORGX4 ORGY=$ORGY4
COMMENT_ORGXY="${COMMENT_ORGXY}
! ORGX= $ORGX4 ORGY= $ORGY4 ! For SPring-8,.."

# Decision of beam center convention based on detector serial numbers.
DET_SN=`grep DETECTOR_SN tmp2 | sed -e "s/DETECTOR_SN=//"`
# For convention 1; Known PF detectors = 449: NW12A Q210, 472: NE3A Q270, 474: BL17A Q270, 912: BL5A Q315, 923: ALS BL5.0.2 Q315, 933: AichiSR BL2S1 Q315
ORG1_SNs="
449
472
474
912
923
933
911
446
"
ORG4_SNs="
915
"
if echo "${DET_SN}${ORG1_SNs}" | sort | uniq -d | grep [0-9] > /dev/null; then
ORGX=$ORGX1
ORGY=$ORGY1
elif echo "${DET_SN}${ORG4_SNs}" | sort | uniq -d | grep [0-9] > /dev/null; then
ORGX=$ORGX4
ORGY=$ORGY4
else
ORGX=$ORGX3
ORGY=$ORGY3
fi

# Check detector serial number and recognize beamline for reversed-phi setting.
# Known detectors for reversed-phi in SPring-8: 915: BL38B1 Q315; APS 19-ID: 458
REVERSEPHI_SNs="
915
458
"
if echo "${DET_SN}${REVERSEPHI_SNs}" | sort | uniq -d | grep [0-9] > /dev/null; then
REVERSE_PHI="yes"
fi

# find DETECTOR_DISTANCE and OSCILLATION_RANGE:
DETECTOR_DISTANCE=`grep ^DISTANCE tmp2 | sed s/DISTANCE=//`
OSCILLATION_RANGE=`grep OSC_RANGE tmp2 | sed s/OSC_RANGE=//`

elif [ "$DET" == "pilatus" ]; then
DETECTOR="PILATUS MINIMUM_VALID_PIXEL_VALUE=0 OVERLOAD= 1048576 !PILATUS"
QX=0.172 QY=0.172
echo Data from a Pilatus detector
sed s/#// tmp2 > tmp1
mv tmp1 tmp2

# find SENSOR_THICKNESS:
SENSOR_THICKNESS=`grep thickness tmp2 | sed -e s/'Silicon sensor, thickness'// | awk '{print $1*1000}'`
# find X_RAY_WAVELENGTH:
X_RAY_WAVELENGTH=`grep Wavelength tmp2 | sed -e s/Wavelength// -e s/A// | awk '{print $1}'`

# find NX and NY; 2463/2527 is 6M, 1475/1679 is 2M
NX=`grep X-Binary-Size-Fastest-Dimension tmp2 | awk '{print $2}'`
NY=`grep X-Binary-Size-Second-Dimension tmp2 | awk '{print $2}'`

# find ORGX and ORGY:
ORGX=`grep Beam_xy tmp2 | sed -e s/$// -e s/$// -e s/\,// | awk '{print $2}'`
ORGY=`grep Beam_xy tmp2 | sed -e s/$// -e s/$// -e s/\,// | awk '{print $3}'`

# find DETECTOR_DISTANCE and OSCILLATION_RANGE:
DETECTOR_DISTANCE=`awk '/distance/{print $2}' tmp2`
DETECTOR_DISTANCE=`echo "$DETECTOR_DISTANCE*1000" | bc -l`

OSCILLATION_RANGE=`awk '/Angle/{print $2}' tmp2`

# get detector serial number and check if it is included in the list
# Known detectors for reversed-phi in SPring-8: BL41XU PILATUS3 6M 60-0125
DET_SN=`grep "Detector:" tmp2 | sed "s/^.*Detector: *//"`
REVERSEPHI_SNs="
PILATUS3 6M, S/N 60-0125
"
if echo "${DET_SN}${REVERSEPHI_SNs}" | sort | uniq -d | grep [0-9] > /dev/null; then
REVERSE_PHI="yes"
fi

elif [ "$DET" == "eiger" ]; then
OVERLOAD=`h5dump -d "/entry/instrument/detector/detectorSpecific/countrate_correction_count_cutoff" $FIRSTFRAME | awk '/$0$:/{print $2}' `
DETECTOR="EIGER MINIMUM_VALID_PIXEL_VALUE=0 OVERLOAD= $OVERLOAD !EIGER"
QX=`h5dump -d "/entry/instrument/detector/x_pixel_size" $FIRSTFRAME | awk '/$0$: [0-9]/{print $2*1000}'`
QY=`h5dump -d "/entry/instrument/detector/y_pixel_size" $FIRSTFRAME | awk '/$0$: [0-9]/{print $2*1000}'`

echo Data from a Eiger hdf5
SENSOR_THICKNESS=`h5dump -d "/entry/instrument/detector/sensor_thickness" $FIRSTFRAME | awk '/$0$: [0-9]/{print $2*1000}'`
X_RAY_WAVELENGTH=`h5dump -d "/entry/instrument/beam/incident_wavelength" $FIRSTFRAME | awk '/$0$: [0-9]/{print $2}'`

NX=`h5dump -d "/entry/instrument/detector/detectorSpecific/x_pixels_in_detector" $FIRSTFRAME | awk '/$0$: [0-9]/{print $2}'`
NY=`h5dump -d "/entry/instrument/detector/detectorSpecific/y_pixels_in_detector" $FIRSTFRAME | awk '/$0$: [0-9]/{print $2}'`

# find ORGX and ORGY:
ORGX=`h5dump -d "/entry/instrument/detector/beam_center_x" $FIRSTFRAME | awk '/$0$: [0-9]/{print $2}'`
ORGY=`h5dump -d "/entry/instrument/detector/beam_center_y" $FIRSTFRAME | awk '/$0$: [0-9]/{print $2}'`

# find DETECTOR_DISTANCE and OSCILLATION_RANGE:
DETECTOR_DISTANCE=`h5dump -d "/entry/instrument/detector/detector_distance" $FIRSTFRAME | awk '/$0$: [0-9]/{print $2*1000}'`
OSCILLATION_RANGE=`h5dump -d "/entry/sample/goniometer/omega_range_average" $FIRSTFRAME | awk '/$0$: [0-9]/{print $2}'`

# If rotation vector set (NeXus)
rotation_axis=`h5dump -a "/entry/sample/transformations/omega/vector" $FIRSTFRAME | grep "(0):" | sed -e "s/^.*://; s/,//g"`

elif [ "$DET" == "raxis" ]; then
echo Data from a RAXIS detector

DETECTOR="RAXIS MINIMUM_VALID_PIXEL_VALUE=0 OVERLOAD=2000000"
#let SKIP=768
#NX=$(od -t x -j $SKIP -N 4 $FIRSTFRAME |awk 'NR==1{print toupper($2)}'|perl -nle '@array= $_ =~/.{2}/g; print "ibase=16;obase=A;".join("",reverse @array)'|bc)
NX=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(768);print "%.4d"%struct.unpack(">i",f.read(4))')
NY=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(772);print "%.4d"%struct.unpack(">i",f.read(4))')

DETECTOR_DISTANCE=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(344);print "-%.4f"%struct.unpack(">f",f.read(4))')

ORGX=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(540);print "%.4f"%struct.unpack(">f",f.read(4))')
ORGY=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(544);print "%.4f"%struct.unpack(">f",f.read(4))')

OSCILLATION_RANGE=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(524);phis,phie=struct.unpack(">ff",f.read(8));print "%.4f"%(phie-phis)')

QX=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(776);print "%.6f"%struct.unpack(">f",f.read(4))')
QY=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(780);print "%.6f"%struct.unpack(">f",f.read(4))')

X_RAY_WAVELENGTH=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(292);print "%.6f"%struct.unpack(">f",f.read(4))')

elif [ "$DET" == "dtrek" ]; then
echo "Data from a RAXIS or Saturn detector with dTREK format"

sed s/\;// tmp2 > tmp1
mv tmp1 tmp2

dname=`grep "DETECTOR_NAMES=" tmp2 | sed -e "s/.*=//"`

if [ "$dname" == "CCD_" ]; then
DETECTOR="SATURN MINIMUM_VALID_PIXEL_VALUE=1"
dtrek_det="saturn"

# Find rotation axis. Warning: currently not support flipping (det(tmpmat)<0)
# I'm not sure this method is really valid - but at least mosflm seems to read this to determine rotation axis.
tmpmat=(`grep ${dname}SPATIAL_DISTORTION_VECTORS= tmp2 | tail -1 | sed -e "s/.*=//"`)
rotx=`echo "scale=6; -1.0*${tmpmat[0]}" | bc -l`
roty=`echo "scale=6; -1.0*${tmpmat[1]}" | bc -l`
rotation_axis="$rotx $roty 0"
if [ `echo "(${tmpmat[0]}*${tmpmat[3]}-(${tmpmat[1]}*${tmpmat[2]}))/1"|bc` -lt 0 ]; then
echo ""
echo "WARNING!! not-supported SPATIAL_DISTORTION_VECTORS header detected."
echo "Please report this to XDSwiki author."
echo ""
fi
else # should be RX_
DETECTOR="RAXIS MINIMUM_VALID_PIXEL_VALUE=0"
dtrek_det="raxis"
fi

# find OVERLOAD
SATURATED_VALUE=`grep SATURATED_VALUE tmp2 | head -1 | sed s/SATURATED_VALUE=//`
DETECTOR="${DETECTOR} OVERLOAD=${SATURATED_VALUE}"

# find X_RAY_WAVELENGTH:
X_RAY_WAVELENGTH=(`grep SOURCE_WAVELENGTH tmp2 | head -1 | sed s/SOURCE_WAVELENGTH=//`)
X_RAY_WAVELENGTH=${X_RAY_WAVELENGTH[1]}

# find NX,NY,QX,QY
# NX,NY should be read from *_DETECTOR_DIMENSIONS?
NX=`grep SIZE1 tmp2 | tail -1 | sed s/SIZE1=//`
NY=`grep SIZE2 tmp2 | tail -1 | sed s/SIZE2=//`
DET_SIZE=(`grep ${dname}DETECTOR_SIZE tmp2 | tail -1 | sed s/.*_DETECTOR_SIZE=//`)
QX=`echo "scale=6; ${DET_SIZE[0]} / $NX" | bc -l`
QY=`echo "scale=6; ${DET_SIZE[1]} / $NY" | bc -l`

# find ORGX, ORGY
SPATIAL_DISTORTION_INFO=(`grep ${dname}SPATIAL_DISTORTION_INFO tmp2 | tail -1 | sed s/.*_SPATIAL_DISTORTION_INFO=//`)
ORGX=${SPATIAL_DISTORTION_INFO[0]}
ORGY=${SPATIAL_DISTORTION_INFO[1]}

# find DETECTOR_DISTANCE
GONIO_NAMES=(`grep ${dname}GONIO_NAMES= tmp2 | tail -1 | sed s/.*_GONIO_NAMES=//`)
GONIO_VALUES=(`grep ${dname}GONIO_VALUES= tmp2 | tail -1 | sed s/.*_GONIO_VALUES=//`)
GONIO_UNITS=(`grep ${dname}GONIO_UNITS= tmp2 | tail -1 | sed s/.*_GONIO_UNITS=//`)
for i in `seq 1 ${#GONIO_NAMES[*]}`
do
idx=$((i-1))
if [ "${GONIO_NAMES[$idx]}" == "Distance" ]; then
DETECTOR_DISTANCE="-${GONIO_VALUES[$idx]}"
# TODO: Check unit!! - ${GONIO_UNITS[$idx]}
fi
done

# find OSCILLATION_RANGE
ROTATION=(`grep "^ROTATION=" tmp2 | tail -1 | sed s/ROTATION=//`)
OSCILLATION_RANGE=${ROTATION[2]}

elif [ "$DET" == "MAR345" ]; then

echo "Data from a Mar345 image plate detector"
DETECTOR="MAR345 MINIMUM_VALID_PIXEL_VALUE=0 OVERLOAD=130000"
NX=`awk '/FORMAT/{print $2}' tmp2`
NY=$NX
QX=`awk '/PIXEL/{print $3/1000.}' tmp2`
QY=`awk '/PIXEL/{print $5/1000.}' tmp2`
if grep -q 'CENTER' tmp2; then
echo Beam center found.
ORGX=`grep 'CENTER' tmp2 | awk '{print $3}'`
ORGY=`grep 'CENTER' tmp2 | awk '{print $5}'`
else
echo No beam center was found. Setting beam center to the middle of the detector.
ORGX=`echo $NX / 2 | bc`
ORGY=`echo $NY / 2 | bc`
fi
DETECTOR_DISTANCE=`grep 'DISTANCE' tmp2 | awk '{print $2}'`
X_RAY_WAVELENGTH=`grep 'WAVELENGTH' tmp2 | awk '{print $2}'`
OSCILLATION_RANGE=`grep 'PHI' tmp2 | awk '{print $5-$3}'`
TRUSTED_REGION="0 0.99"
else
echo should never come here
exit 1
fi

echo ORGX= $ORGX ORGY= $ORGY - check these values with adxv !
echo DETECTOR_DISTANCE= $DETECTOR_DISTANCE
echo OSCILLATION_RANGE= $OSCILLATION_RANGE
echo X-RAY_WAVELENGTH= $X_RAY_WAVELENGTH

# now we know everything that is required to generate XDS.INP

cat > XDS.INP << eof
! written by generate_XDS.INP version $REVISION
JOB= XYCORR INIT COLSPOT IDXREF DEFPIX INTEGRATE CORRECT
ORGX= $ORGX ORGY= $ORGY ! check these values with adxv !\
$COMMENT_ORGXY
DETECTOR_DISTANCE= $DETECTOR_DISTANCE
OSCILLATION_RANGE= $OSCILLATION_RANGE
X-RAY_WAVELENGTH= $X_RAY_WAVELENGTH
NAME_TEMPLATE_OF_DATA_FRAMES=$NAME_TEMPLATE_OF_DATA_FRAMES
! REFERENCE_DATA_SET=xxx/XDS_ASCII.HKL ! e.g. to ensure consistent indexing
DATA_RANGE=$DATA_RANGE
SPOT_RANGE=$SPOT_RANGE
! BACKGROUND_RANGE=1 10 ! rather use defaults (first 5 degree of rotation)

SPACE_GROUP_NUMBER=0 ! 0 if unknown
UNIT_CELL_CONSTANTS= 70 80 90 90 90 90 ! put correct values if known
INCLUDE_RESOLUTION_RANGE=50 0 ! after CORRECT, insert high resol limit; re-run CORRECT
! IDXREF now obeys INCLUDE_RESOLUTION_RANGE and EXCLUDE_RESOLUTION_RANGE to exclude ice-rings

FRIEDEL'S_LAW=FALSE ! This acts only on the CORRECT step
! If the anom signal turns out to be, or is known to be, very low or absent,
! use FRIEDEL'S_LAW=TRUE instead (or comment out the line); re-run CORRECT

! remove the "!" in the following line:
! STRICT_ABSORPTION_CORRECTION=TRUE
! if the anomalous signal is strong: in that case, in CORRECT.LP the three
! "CHI^2-VALUE OF FIT OF CORRECTION FACTORS" values are significantly> 1, e.g. 1.5
!
! exclude (mask) untrusted areas of detector, e.g. beamstop shadow :
! UNTRUSTED_RECTANGLE= 1800 1950 2100 2150 ! x-min x-max y-min y-max ! repeat
! UNTRUSTED_ELLIPSE= 2034 2070 1850 2240 ! x-min x-max y-min y-max ! if needed
! UNTRUSTED_QUADRILATERAL= x1 y1 x2 y2 x3 y3 x4 y4 ! see documentation
!
! parameters with changes wrt default values:
TRUSTED_REGION=$TRUSTED_REGION
VALUE_RANGE_FOR_TRUSTED_DETECTOR_PIXELS=6000. 30000. ! often 7000 or 8000 is ok
STRONG_PIXEL=4 ! COLSPOT: only use strong reflections (default is 3)
MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT=3 ! default of 6 is sometimes too high
! close spots/long cell axis: reduce SEPMIN and CLUSTER_RADIUS from their defaults of 6 and 3
! SEPMIN=4 CLUSTER_RADIUS=2 ! should be default for Pilatus and other detectors with low PSF
! since XDS 01-MAR-2015, POSITION is used instead of DISTANCE. Older versions do it the other way round.
! nowadays headers are usually correct so refine DISTANCE/POSITION in INTEGRATE but not IDXREF
REFINE(IDXREF)=CELL BEAM ORIENTATION AXIS ! DISTANCE POSITION
REFINE(INTEGRATE)= DISTANCE POSITION BEAM ORIENTATION ! AXIS CELL
! REFINE(CORRECT)=CELL BEAM ORIENTATION AXIS DISTANCE POSITION ! Default is: refine everything

! parameters specifically for this detector and beamline:
DETECTOR= $DETECTOR
SENSOR_THICKNESS= $SENSOR_THICKNESS
! attention CCD detectors: for very high resolution (better than 1A) make sure to specify SILICON
! as about 32* what CORRECT.LP suggests (absorption of phosphor is much higher than that of silicon).
! Better: read the article http://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/SILICON .
NX= $NX NY= $NY QX= $QX QY= $QY ! to make CORRECT happy if frames are unavailable

eof
if [ "$DET" == "raxis" -o "$dtrek_det" == "raxis" ]; then
cat >> XDS.INP << eof
DIRECTION_OF_DETECTOR_X-AXIS=1 0 0
DIRECTION_OF_DETECTOR_Y-AXIS=0 -1 0
INCIDENT_BEAM_DIRECTION=0 0 1
ROTATION_AXIS=0 1 0
!FRACTION_OF_POLARIZATION=0.98 ! uncomment if synchrotron
POLARIZATION_PLANE_NORMAL=1 0 0
eof
else
if [ "$rotation_axis" != "" ]; then
echo "ROTATION_AXIS= $rotation_axis" >> XDS.INP
elif [ "$REVERSE_PHI" == "no" ]; then
echo 'ROTATION_AXIS=1 0 0 ! Australian Synchrotron, SERCAT ID-22 (?), APS 19-ID (?), ESRF BM30A, SPring-8, SSRF BL17U need -1 0 0. Diamond ID24 needs 0 -1 0' >> XDS.INP
else
echo 'ROTATION_AXIS=-1 0 0 ! if this is wrong, please contact author.' >> XDS.INP
fi
if [ "$dtrek_det" == "saturn" ]; then
cat >> XDS.INP << eof
DIRECTION_OF_DETECTOR_X-AXIS=-1 0 0
DIRECTION_OF_DETECTOR_Y-AXIS= 0 1 0
eof
else
cat >> XDS.INP << eof
DIRECTION_OF_DETECTOR_X-AXIS=1 0 0
DIRECTION_OF_DETECTOR_Y-AXIS=0 1 0
eof
fi
cat >> XDS.INP << eof
INCIDENT_BEAM_DIRECTION=0 0 1
FRACTION_OF_POLARIZATION=0.98 ! better value is provided by beamline staff!
POLARIZATION_PLANE_NORMAL=0 1 0
eof
fi
cat >> XDS.INP << eof
!used by DEFPIX and CORRECT to exclude ice-reflections / ice rings - uncomment if necessary
!EXCLUDE_RESOLUTION_RANGE= 3.93 3.87 !ice-ring at 3.897 Angstrom
!EXCLUDE_RESOLUTION_RANGE= 3.70 3.64 !ice-ring at 3.669 Angstrom
!EXCLUDE_RESOLUTION_RANGE= 3.47 3.41 !ice-ring at 3.441 Angstrom
!EXCLUDE_RESOLUTION_RANGE= 2.70 2.64 !ice-ring at 2.671 Angstrom
!EXCLUDE_RESOLUTION_RANGE= 2.28 2.22 !ice-ring at 2.249 Angstrom
!EXCLUDE_RESOLUTION_RANGE= 2.102 2.042 !ice-ring at 2.072 Angstrom - strong
!EXCLUDE_RESOLUTION_RANGE= 1.978 1.918 !ice-ring at 1.948 Angstrom - weak
!EXCLUDE_RESOLUTION_RANGE= 1.948 1.888 !ice-ring at 1.918 Angstrom - strong
!EXCLUDE_RESOLUTION_RANGE= 1.913 1.853 !ice-ring at 1.883 Angstrom - weak
!EXCLUDE_RESOLUTION_RANGE= 1.751 1.691 !ice-ring at 1.721 Angstrom - weak
eof
if [ "$DET" == "pilatus" -o "$DET" == "eiger" ]; then
cat >> XDS.INP << eof
SEPMIN=4 CLUSTER_RADIUS=2
NUMBER_OF_PROFILE_GRID_POINTS_ALONG_ALPHA/BETA=13 ! Default is 9 - Increasing may improve data
NUMBER_OF_PROFILE_GRID_POINTS_ALONG_GAMMA=13 ! accuracy, particularly if finely-sliced on phi,
! and does not seem to have any downsides.
eof
if [ $NX == "1475" ]; then
if ! grep -q Flat_field tmp2 ; then
cat >> XDS.INP << eof
! the following specifications are for a detector _without_ proper
! flat_field correction; they cut away one additional pixel adjacent
! to each UNTRUSTED_RECTANGLE
!EXCLUSION OF VERTICAL DEAD AREAS OF THE PILATUS 2M DETECTOR
UNTRUSTED_RECTANGLE= 486 496 0 1680
UNTRUSTED_RECTANGLE= 980 990 0 1680
!EXCLUSION OF HORIZONTAL DEAD AREAS OF THE PILATUS 2M DETECTOR
UNTRUSTED_RECTANGLE= 0 1476 194 214
UNTRUSTED_RECTANGLE= 0 1476 406 426
UNTRUSTED_RECTANGLE= 0 1476 618 638
UNTRUSTED_RECTANGLE= 0 1476 830 850
UNTRUSTED_RECTANGLE= 0 1476 1042 1062
UNTRUSTED_RECTANGLE= 0 1476 1254 1274
UNTRUSTED_RECTANGLE= 0 1476 1466 1486
eof
else
cat >> XDS.INP << eof
!EXCLUSION OF VERTICAL DEAD AREAS OF THE PILATUS 2M DETECTOR
UNTRUSTED_RECTANGLE= 487 495 0 1680
UNTRUSTED_RECTANGLE= 981 989 0 1680
!EXCLUSION OF HORIZONTAL DEAD AREAS OF THE PILATUS 2M DETECTOR
UNTRUSTED_RECTANGLE= 0 1476 195 213
UNTRUSTED_RECTANGLE= 0 1476 407 425
UNTRUSTED_RECTANGLE= 0 1476 619 637
UNTRUSTED_RECTANGLE= 0 1476 831 849
UNTRUSTED_RECTANGLE= 0 1476 1043 1061
UNTRUSTED_RECTANGLE= 0 1476 1255 1273
UNTRUSTED_RECTANGLE= 0 1476 1467 1485
eof
fi
elif [ $NX == "2463" ]; then
# Pilatus 6M
# FIXME: here we could test if a Flat_field correction was applied like we do for 2M
cat >> XDS.INP << eof
UNTRUSTED_RECTANGLE= 487 495 0 2528
UNTRUSTED_RECTANGLE= 981 989 0 2528
UNTRUSTED_RECTANGLE=1475 1483 0 2528
UNTRUSTED_RECTANGLE=1969 1977 0 2528
UNTRUSTED_RECTANGLE= 0 2464 195 213
UNTRUSTED_RECTANGLE= 0 2464 407 425
UNTRUSTED_RECTANGLE= 0 2464 619 637
UNTRUSTED_RECTANGLE= 0 2464 831 849
UNTRUSTED_RECTANGLE= 0 2464 1043 1061
UNTRUSTED_RECTANGLE= 0 2464 1255 1273
UNTRUSTED_RECTANGLE= 0 2464 1467 1485
UNTRUSTED_RECTANGLE= 0 2464 1679 1697
UNTRUSTED_RECTANGLE= 0 2464 1891 1909
UNTRUSTED_RECTANGLE= 0 2464 2103 2121
UNTRUSTED_RECTANGLE= 0 2464 2315 2333
eof
elif [ $NX == "3110" -a $NY == "3269" ]; then
# Eiger 9M
cat >> XDS.INP << eof
!EXCLUSION OF VERTICAL DEAD AREAS OF THE EIGER 9M DETECTOR
UNTRUSTED_RECTANGLE= 1029 1042 0 3269
UNTRUSTED_RECTANGLE= 2069 2082 0 3269
!EXCLUSION OF HORIZONTAL DEAD AREAS OF THE EIGER 9M DETECTOR
UNTRUSTED_RECTANGLE= 0 3110 513 553
UNTRUSTED_RECTANGLE= 0 3110 1064 1104
UNTRUSTED_RECTANGLE= 0 3110 1615 1655
UNTRUSTED_RECTANGLE= 0 3110 2166 2206
UNTRUSTED_RECTANGLE= 0 3110 2717 2757
eof
fi
fi
echo XDS.INP is ready for use. The file has only the most important keywords.
echo Full documentation, including complete detector templates, is at
echo http://www.mpimf-heidelberg.mpg.de/~kabsch/xds . More documentation in XDSwiki
echo After running xds, inspect, using XDS-Viewer, at least the beamstop mask in
echo BKGPIX.cbf, and the agreement of predicted and observed spots in FRAME.cbf!
rm -f tmp1 tmp2
# end of generate_XDS.INP
</pre>

== System-wide or personal installation ==

Ask your system adminstrator to cut-and-paste the script into e.g. /usr/local/bin/generate_XDS.INP, and to make it "executable".

But you may also cut-and-paste the script from this webpage into a file in e.g. your home directory; the filename should be generate_XDS.INP. After creating the file, make it executable - e.g. if it's in your $HOME, use:
chmod +x ~/generate_XDS.INP
After that, you can just run it in a similar way as if it were installed in your $PATH:
~/generate_XDS.INP "frms/mydata_1_???.img"
By using your own file, you can easily update to the latest revision, or even change the script, without having to bother the system administrator.

On Mac OS X, installation of the "Command Line Tools" (from http://developer.apple.com/downloads; requires Apple ID) is required. These are also part of the (larger, but also free) [http://developer.apple.com/tools/xcode Xcode] package. This package comes with a license that has to be accepted by the user when running a Command Line Tool for the first time.

== Generating generate_XDS.INP from this webpage ==

Instead of cutting-and-pasting the lines of the script, you (or the system administrator) could just cut-and-paste the following four lines
<pre>
wget http://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/generate_XDS.INP -O - | \
sed -e s/\&nbsp\;/\ /g -e s/\&gt\;/\>/g -e s/\&lt\;/\</g -e s/amp\;//g -e s/\&quot\;/\"/g -e s/\&\#\1\6\0\;/\ /g | \
sed '/# end of generate_XDS.INP/,$d' | awk '/^#/,/rm -f tmp1 tmp2/' > generate_XDS.INP
chmod +x generate_XDS.INP
</pre>
to copy the script from this website into an executable file generate_XDS.INP in your current directory. On a Mac (which does not seem to have wget), one could try
<pre>
curl -L -o - http://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/generate_XDS.INP | \
sed -e s/\&nbsp\;/\ /g -e s/\&gt\;/\>/g -e s/\&lt\;/\</g -e s/amp\;//g -e s/\&quot\;/\"/g -e s/\&\#\1\6\0\;/\ /g | \
sed '/# end of generate_XDS.INP/,$d' | awk '/^#/,/rm -f tmp1 tmp2/' > generate_XDS.INP
chmod +x generate_XDS.INP
</pre>
If you do use cut-and-paste from the webpage, be aware of the following problem report: On the Mac, after loading frames, by clicking “generate XDS.INP”, the program gives some strange symbol “Ã” in XDS.INP. And the more you click “save” button, the more “Ã” appear. This looks like e.g. SPACE_GROUP_NUMBER=0 Ã ! 0 if unknown UNIT_CELL_CONSTANTS= 70 80 90 90 90 90 Ã ''The problem is due to the “Rich text” format in TextEdit when saving "generate_XDS.INP". It is solved by re-downloading the script, and changing format to Plain - everything should work then.''

== Calling generate_XDS.INP from a Python script ==

It is recommended to use the [http://docs.python.org/2/library/subprocess.html subprocess.Popen()] module instead of [http://docs.python.org/2/library/os.html os.system()]:
<pre>
subprocess.Popen(["generate_XDS.INP",imagepath],stdout=outputfile)
</pre>
where imagepath is a string containing the path to an image and outputfile is either a chosen variable for an output file or subprocess.PIPE if you're not interested in the output of the script.

The module os.system() internally uses /bin/sh to execute the command and overrides #!/bin/bash at the beginning of the script. While this is not a problem on most operating systems, /bin/sh points to dash instead of bash on Ubuntu, which leads to a program crash with the error message
<pre>
sh: 1: Syntax error: Bad fd number
</pre>

== Dependencies ==
The script makes use of many GNU commands, like ls, grep and egrep, awk, cut, cat, echo, wc, bc, head, sed, tail, cp, od, python. Some of them (like od and python) are only used in case of specific detectors.
The script will only work if all the required commands are available. They reside in either the coreutils RPM, or specific RPMs (sed, bc, grep, python ...). Please note that to get the <code>strings</code> command on some Linux distributions (e.g. FC23) you need to install the <code>binutils</code> RPM package.

== Limitations ==

* The script tries to interpret the header of the frames, so is currently limited to Dectris (Pilatus), ADSC (Quantum), Rigaku (several types) and MAR (CCD and image plate) detectors. Other detectors need some values to be manually filled into XDS.INP - the relevant places are marked with XXX. These are detector properties (type, pixel size and number, min and max counts in a pixel), and experimental parameters like oscillation range, wavelength, distance, and direct beam position (or rather: point of detector that is closest to the crystal). For fine-tuning of detector parameters, see the [http://xds.mpimf-heidelberg.mpg.de/html_doc/xds_prepare.html detector-specific templates].
* The authors have made a "best effort" to provide a XDS.INP that results in the correct sign of the anomalous signal. In the case of one detector type (internally called Rigaku SMV) this requires reversal of one detector axis, and a negative DETECTOR_DISTANCE, as is found in some of the [http://xds.mpimf-heidelberg.mpg.de/html_doc/xds_prepare.html detector-specific templates]. '''For an unusual or unknown detector setup, the correct sign of the anomalous signal needs to be established and verified e.g. with a good dataset from a test crystal that has a anomalous signal.''' The authors do not take any responsibility for problems arising from incorrect sign of the anomalous signal, nor - obviously! - for any other mischief arising in or from data processing.
* At some beamlines, the ROTATION_AXIS should be -1 0 0 ("backwards") instead of the usual 1 0 0 ("horizontal"), or even 0 1 0 ("vertical") like at one of the PETRA Hamburg BLs. We have only just started the article [[Beamline notes]], and the frame headers do not have this information, so the default chosen by [[generate_XDS.INP]] may be wrong and need manual correction.
* Sometimes, the x- and y- values of the primary beam position recorded in the header should be used for ORGY and ORGX (i.e reversed) instead of as ORGX and ORGY. For ADSC, this has been implemented in the script for a number of beamlines. A beamline that is also known to have reversed X and Y is BM14 (ESRF; Mar CCD, as of April 2014) but the script does not currently implement the reversal.

== See also ==

* GlobalPhasing's list of beamlines and their settings is at http://www.globalphasing.com/autoproc/wiki/index.cgi?BeamlineSettings .
* [[Beamline notes]] to collect information about specific settings

Problems

2015-09-14T07:02:56Z

Karine: /* IDXREF.LP does not show the expected lattice */ you do not to know -> you do not know

This is a collection of problems and their solutions.

== XDS crashes ==

XDS should never crash (if it terminates with an error message, this does not count as crash). If it does, it is either a bug in the program which should be brought to the attention of Wolfgang Kabsch or Kay Diederichs, and will be fixed, or it is a problem with your computer.

If it crashes for the second reason, these are the things to try/consider:

=== read error ===

... may result in e.g.

forrtl: severe (24): end-of-file during read, unit 2, file bin1_01.tmp

Any error message that XDS prints out afterwards is just a follow-up error. The most likely cause of the problem is that your disk is full, or your quota exceeded. You must fix the root problem before you can continue.

=== huge frames leading to overflow of the shell's stack limit ===

Increase the stack limit of your shell, e.g. (if using csh or tcsh) with
limit stacksize 102400
or (in case of bash)
ulimit -s 102400
The numbers above mean a 10-fold increase over the default, and should be enough. I've found this to be necessary for unusually large frames (32 MB).
xds_par in this case also might need an increase of the environment variable OMP_STACKSIZE (e.g. "setenv OMP_STACKSIZE 128M").

=== memory consumption exceeding the 4GB barrier when using the 32bit binary ===

As pointed out by James Holton, the memory consumed by huge frames in combination with many threads may be larger than 32 bits allow. Execution then stops with
forrtl: severe (41): insufficient virtual memory
The workaround is to limit the [http://homes.mpimf-heidelberg.mpg.de/~kabsch/xds/html_doc/xds_parameters.html MAXIMUM_NUMBER_OF_PROCESSORS], or to use XDS instead of XDS_PAR. An even better solution is to switch to the 64bit binary, but that of course requires a 64bit operating system.

=== Problems with OpenMP ===

If the "xds_par" binary crashed, try "xds". xds_par uses OpenMP for parallelization, which adds complexity. If it works with xds, but not with xds_par, then there is a chance that some environment variable needs to be set/changed. In any case the XDS developers would like to learn about this.

=== limitation of the parallel 32bit binary ===

The 32bit OpenMP binary may not be able to allocate enough storage when using many threads (e.g. >10). The error message might be something like:

OMP: Error #34: System unable to allocate necessary resources for OMP thread:
OMP: System error #11: Resource temporarily unavailable

=== 64bit binary on a 32 bit operating system ===

If the error message is e.g.
xds: Exec format error. Wrong Architecture.
or
-bash: /usr/local/bin/xds: cannot execute binary file
make sure to use the 32bit version of XDS instead - you try to run the 64bit version on a 32bit operating system; this won't work.

=== CPU without SSE2 support ===

If the error message is

forrtl: severe (168): Program Exception - illegal instruction
Image PC Routine Line Source
xds 08055250 Unknown Unknown Unknown
xds 0804B3F6 Unknown Unknown Unknown

this means that unfortunately the CPU of your machine is too old to be supported by XDS (which in this respect is compiled with the default options of the ifort compiler). This may happen e.g. with old AMD CPUs that don't support SSE2 (on Linux check the ''flags'' field of /proc/cpuinfo for the presence of the sse2 flag).

According to [http://en.wikipedia.org/wiki/SSE2#Notable_IA-32_CPUs_not_supporting_SSE2] the following Intel-compatible CPUs did not implement SSE2 after SSE2 was introduced (2001):
* AMD CPUs prior to Athlon 64, including all Socket A-based CPUs
* Intel CPUs prior to Pentium 4
* VIA C3
* Transmeta Crusoe

=== ASSERT VIOLATION ===

If you get e.g.
!!! ERROR !!! ASSERT VIOLATION IN "RESTHKL"
this indicates that an internal error has occurred, i.e. some situation was hit that the program is not prepared to handle.
There must be something very special about your dataset - maybe a very small number of reflections, maybe only negative intensities, maybe something else -
which triggers the bug. If this happens then get in touch with the XDS developers - they will most likely ask to make your XDS.INP and data frames available to them, to fix the program.

== Indexing failures ==

In case of IDXREF problems, you should definitively inspect, using [[XDS-viewer]] or [[adxv]], the FRAME.cbf file which is written by COLSPOT. This file shows the first frame of the SPOT_RANGE, together with strong pixels that COLSPOT found on this frame, marked with white pixels. IDXREF will use those spot positions which have at least as many strong pixels as MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT specifies.

You will only be able to look at this file if you prevent XDS from running INTEGRATE. This can be achieved e.g. by

JOB= XYCORR INIT COLSPOT IDXREF DEFPIX ! INTEGRATE CORRECT

since otherwise, FRAME.cbf is overwritten by INTEGRATE.

To look at '''all''' spot positions found by COLSPOT, you could try
echo "set yrange [] reverse ; plot 'SPOT.XDS' us 1:2 w dots" | gnuplot -persist
This may also help to find shaded regions on the detector.

=== IDXREF ends with !!! ERROR !!! message ===

These messages should really be called !!! WARNING !!!. The two types occuring most often are:

!!! ERROR !!! SOLUTION IS INACCURATE
This message is printed if the "STANDARD DEVIATION OF SPOT POSITION (PIXELS)" is larger than the MAXIMUM_ERROR_OF_SPOT_POSITION= (default of 3).

or

!!! ERROR !!! INSUFFICIENT PERCENTAGE (< 50%) OF INDEXED REFLECTIONS
This message is printed if less than the given percentage of reflections that COLSPOT found was actually indexed. If this occurs, it is clearly a warning sign.

The message continues in both cases:
AUTOMATIC DATA PROCESSING STOPPED. AS THE CRITERIA FOR A GOOD
SOLUTION ARE RATHER STRICT, YOU MAY CHOOSE TO CONTINUE DATA
PROCESSING AFTER CHANGING THE "JOB="-CARD IN "XDS.INP" TO
"JOB= DEFPIX INTEGRATE CORRECT".
IF THE BEST SOLUTION IS REALLY NONSENSE YOU SHOULD FIRST HAVE
A LOOK AT THE ASCII-FILE "SPOT.XDS". THIS FILE CONTAINS THE
INITIAL SPOT LIST SORTED IN DECREASING SPOT INTENSITY. SPOTS
NEAR THE END OF THE FILE MAY BE ARTEFACTS AND SHOULD BE ERASED.
ALTERNATIVELY YOU MAY TRY DIFFERENT VALUES FOR "INDEX_ORIGIN"
AS SUGGESTED IN THE ABOVE LISTING.
IF THE CRYSTAL HAS SLIPPED AT THE BEGINNING OF DATA COLLECTION
YOU MAY CHOOSE TO SKIP SOME OF THE FIRST FRAMES BY CHANGING
THE "DATA_RANGE=" IN FILE "XDS.INP" AND START ALL OVER AGAIN.

This is printed out for you to actually read, and take action accordingly. In most cases you just change the JOBS - line in XDS.INP to read
JOB= DEFPIX INTEGRATE CORRECT
and then continue to run XDS. In other cases you may want to change SPOT.XDS, or other keywords in [[XDS.INP]] (see also below). But in any case this is an important alert that should make you check the correctness of the parameters that describe the data collection (X-RAY_WAVELENGTH, DETECTOR_DISTANCE, ORGX, ORGY, OSCILLATION_RANGE, NAME_TEMPLATE_OF_DATA_FRAMES).

=== IDXREF produces too short cell parameter(s) ===

The first table "CLUSTER COORDINATES AND INDICES WITH RESPECT TO REC. LATTICE BASIS VECTORS" should show close-to-integer difference vectors. The following is an example how it should ''not'' look like:
# COORDINATES OF REC. BASIS VECTOR LENGTH 1/LENGTH

1 -0.0000476-0.0014008-0.0044654 0.0046802 213.67
2 -0.0101960-0.0056194 0.0019466 0.0118037 84.72
3 -0.0145331 0.0238225-0.0071754 0.0288134 34.71

CLUSTER COORDINATES AND INDICES WITH RESPECT TO REC. LATTICE BASIS VECTORS
# COORDINATES OF VECTOR CLUSTER FREQUENCY CLUSTER INDICES
1 0.0004199-0.0012633-0.0043826 2442. 0.97 -0.04 -0.00
2 0.0101757 0.0060701-0.0019630 2175. -0.03 -1.02 0.01
3 -0.0076118 0.0114167-0.0040603 1965. 0.13 0.04 0.50
4 0.0100552 0.0071646 0.0025337 1944. -1.01 -0.99 0.01
5 0.0072840-0.0101455 0.0084405 1841. -1.10 -0.01 -0.49
6 0.0000976 0.0027828 0.0089584 1792. -2.00 0.00 -0.00
7 0.0103103 0.0043851-0.0063902 1790. 0.98 -1.02 0.00
8 0.0025742 0.0163995-0.0098507 1760. 0.95 -0.99 0.51
9 0.0068686-0.0089437 0.0128884 1724. -2.08 0.03 -0.49
10 -0.0174443 0.0123504 0.0161324 1694. -4.01 0.99 0.52
11 0.0272195-0.0050005-0.0136142 1678. 2.99 -1.97 -0.50
...

PARAMETERS OF THE REDUCED CELL (ANGSTROEM & DEGREES)
34.71 84.72 213.67 90.35 90.27 90.06
Do you see the half-integer difference vectors in the last column? This clearly indicates that the third axis above the table (with 1/length=34.71) actually should be twice that size. If this happens, the histogram of indexed spots often has two equally-large subtrees:
SUBTREE POPULATION

1 976
2 972
3 10
4 7
5 6

Remedy: take the reduced cell (as found below the table), but double the 34.71, and stick it into the line
UNIT_CELL_CONSTANTS=69.42 84.72 213.67 90.35 90.27 90.06
in XDS.INP. To make XDS actually ''use'' that line, specify
SPACE_GROUP_NUMBER=1
and leave the space group determination for later.

Alternatively, try several well-separated SPOT_RANGEs instead of 1. For example, instead of
SPOT_RANGE=1 900
try e.g.
SPOT_RANGE=1 100
SPOT_RANGE=801 900
In the cases where I've tried this, this splitting of SPOT_RANGE into several wedges always resulted in IDXREF picking up the correct cell.

The reason for the indexing failure in these cases seems to be the fact that the crystal changed its orientation within the SPOT_RANGE by more than ~ 0.1°.

=== IDXREF produces too long axes ===

This may (rarely) happen when MAXIMUM_NUMBER_OF_JOBS is > 1. In this case, different JOBS' COLSPOT runs may report some reflections twice in SPOT.XDS. Since their phi values are close, they correspond to long unit cell parameters. This happens more easily if the mosaicity is high and therefore reflections extend over the borders between JOBs.

This effect may be mitigated by having as many SPOT_RANGEs as JOBs, and leaving gaps between the SPOT_RANGEs.

=== IDXREF prints !!! WARNING !!! message ===

If you see
!!! WARNING !!! REFINEMENT DID NOT CONVERGE
LAST CORRECTION SHIFT WAS 6.1E-02 (should be < 1.0E-03)
then try the following: find the line
REFINE(IDXREF)= ...
in XDS.INP. If it is commented out with a !, remove the !. Then, change it to have
REFINE(IDXREF)=CELL BEAM ORIENTATION AXIS ! POSITION
i.e. remove POSITION from the list of refinable parameters. Once that is done, save XDS.INP and re-run the IDXREF step.

This problem occurs if the POSITION value (called DISTANCE in former XDS versions) is large, and XDS cannot refine it meaningfully.

If this does not help, try to refine even less items, e.g. leave out AXIS.

=== IDXREF.LP does not show the expected lattice ===

By specifying the spacegroup and unitcell, you tell XDS that it should index based on those reflections that match that spacegroup and unitcell. In a sense, you _force_ that spacegroup and cell. This will discard other (spurious?) reflections, and usually leads to a clean list of Bravais lattice possibilities.

If the data reduction fails nevertheless (in terms of bad R-factors and [[CORRECT#An estimate for the overall quality of an experimental setup|ISa]] in the [[CORRECT]] step), then chances are that you specified some parameter wrongly, or not accurate enough (ORGX and ORGY are the most likely candidates), or that the crystal does not match your idea about its spacegroup and unit cell. Actually the latter happens pretty frequently (which is why it is always the safest way to collect 180° of spindle rotation unless you know your crystals very well; this is also a good strategy in other respects).

If you tell XDS that you do not know the spacegroup (SPACE_GROUP_NUMBER= 0), then [[IDXREF]] takes all observed reflections into account. By design, the spacegroup decision is then postponed until the CORRECT step, or rather to a run of [[pointless]] after CORRECT, and it may be not vital to closely inspect IDXREF.LP, because CORRECT.LP has basically the same information, plus more.

=== none of the lattices in IDXREF.LP (except aP) has a good QUALITY OF FIT ===

It is a good idea to use many frames for the SPOT_RANGE, (e.g. the first half of the DATA_RANGE, as does the [[generate_XDS.INP]] script) for the COLSPOT and IDXREF steps.

It is entirely possible to run COLSPOT for the ''complete'' DATA_RANGE (in that case the parameters of SPOT_RANGE are the same as for the DATA_RANGE), and to try smaller SPOT_RANGEs in IDXREF - this means COLSPOT has to be run only once, and the JOBS= line has e.g.
JOBS= IDXREF
only. Maybe some SPOT_RANGE sub-range gives a clear answer. In that case it may be useful to use
REFINE(INTEGRATE)=! AXIS BEAM ORIENTATION CELL DISTANCE
because otherwise the spurious reflections in the other ranges will probably disturb the on-the-fly refinement of parameters.

I've seen datasets where each reflection had a satellite associated with the main reflection, but separate from it. This is well visible in FRAME.cbf written by COLSPOT. In such a case it helps to use
MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT= 12 ! re-run COLSPOT after changing the parameter !
thus doubling (w.r.t. the default of 6) the required spotsize that makes a reflection be used for indexing. In the latest case this made it possible to index cleanly on a single frame (which is actually not uncommon).

A final possibility: your crystal may really be triclinic - hopefully you collected 180° of data, or even a bit more than that.

== Integration failure ==

If INTEGRATE stops after e.g.
******************************************************************************
PROCESSING OF IMAGES 136 ... 140
******************************************************************************
with
!!! ERROR !!! AUTOMATIC DETERMINATION OF SPOT SIZE PARAMETERS HAS FAILED.
YOU MAY RESTART THIS STEP AFTER SPECIFYING VALUES IN XDS.INP FOR:
REFLECTING_RANGE= REFLECTING_RANGE_E.S.D.=
BEAM_DIVERGENCE= BEAM_DIVERGENCE_E.S.D.=

then you should reduce the upper limit of the DATA_RANGE, to stop before the problematic frames, and re-run INTEGRATE. In this example, you would modify XDS.INP to have
DATA_RANGE=1 135
JOB=INTEGRATE CORRECT
Save XDS.INP, run XDS and inspect INTEGRATE.LP, to find the lines (e.g.)
BEAM_DIVERGENCE= 0.478 BEAM_DIVERGENCE_E.S.D.= 0.048
REFLECTING_RANGE= 1.100 REFLECTING_RANGE_E.S.D.= 0.157
Copy them to XDS.INP. Restore the original DATA_RANGE and continue.

Another error mode of INTEGRATE (in processing of small-molecule data) is ...
IMAGE IER SCALE NBKG NOVL NEWALD NSTRONG NREJ SIGMAB SIGMAR
5 0 0.975 3968463 1 5 1 0 0.1028 1.3486
6 0 0.967 3969949 2 2 2 0 0.1688 0.0000
7 0 0.929 3975306 0 3 0 0 0.0000 0.0000
!!! ERROR !!! CANNOT ALLOCATE MEMORY
YOU COULD RERUN THIS STEP WITH SMALLER VALUES FOR THE PARAMETERS
NUMBER_OF_PROFILE_GRID_POINTS_ALONG_ALPHA/BETA=
NUMBER_OF_PROFILE_GRID_POINTS_ALONG_GAMMA=
The error message is misleading in this case: there are too few reflections to build the average profile. The fix is: restart INTEGRATE after inserting e.g.
DELPHI=20 ! default is 5, so try with e.g. 10, 20, 45, 90, 180
and re-run INTEGRATE.

== See also ==
[[XDS.INP#What can go wrong with this file?]]

[[Low dose data]]

Problems

2015-06-04T14:07:27Z

Karine: /* IDXREF produces too long axes */ corrected typo

This is a collection of problems and their solutions.

== XDS crashes ==

XDS should never crash (if it terminates with an error message, this does not count as crash). If it does, it is either a bug in the program which should be brought to the attention of Wolfgang Kabsch or Kay Diederichs, and will be fixed, or it is a problem with your computer.

If it crashes for the second reason, these are the things to try/consider:

=== read error ===

... may result in e.g.

forrtl: severe (24): end-of-file during read, unit 2, file bin1_01.tmp

Any error message that XDS prints out afterwards is just a follow-up error. The most likely cause of the problem is that your disk is full, or your quota exceeded. You must fix the root problem before you can continue.

=== huge frames leading to overflow of the shell's stack limit ===

Increase the stack limit of your shell, e.g. (if using csh or tcsh) with
limit stacksize 102400
or (in case of bash)
ulimit -s 102400
The numbers above mean a 10-fold increase over the default, and should be enough. I've found this to be necessary for unusually large frames (32 MB).
xds_par in this case also might need an increase of the environment variable OMP_STACKSIZE (e.g. "setenv OMP_STACKSIZE 128M").

=== memory consumption exceeding the 4GB barrier when using the 32bit binary ===

As pointed out by James Holton, the memory consumed by huge frames in combination with many threads may be larger than 32 bits allow. Execution then stops with
forrtl: severe (41): insufficient virtual memory
The workaround is to limit the [http://homes.mpimf-heidelberg.mpg.de/~kabsch/xds/html_doc/xds_parameters.html MAXIMUM_NUMBER_OF_PROCESSORS], or to use XDS instead of XDS_PAR. An even better solution is to switch to the 64bit binary, but that of course requires a 64bit operating system.

=== Problems with OpenMP ===

If the "xds_par" binary crashed, try "xds". xds_par uses OpenMP for parallelization, which adds complexity. If it works with xds, but not with xds_par, then there is a chance that some environment variable needs to be set/changed. In any case the XDS developers would like to learn about this.

=== limitation of the parallel 32bit binary ===

The 32bit OpenMP binary may not be able to allocate enough storage when using many threads (e.g. >10). The error message might be something like:

OMP: Error #34: System unable to allocate necessary resources for OMP thread:
OMP: System error #11: Resource temporarily unavailable

=== 64bit binary on a 32 bit operating system ===

If the error message is e.g.
xds: Exec format error. Wrong Architecture.
or
-bash: /usr/local/bin/xds: cannot execute binary file
make sure to use the 32bit version of XDS instead - you try to run the 64bit version on a 32bit operating system; this won't work.

=== CPU without SSE2 support ===

If the error message is

forrtl: severe (168): Program Exception - illegal instruction
Image PC Routine Line Source
xds 08055250 Unknown Unknown Unknown
xds 0804B3F6 Unknown Unknown Unknown

this means that unfortunately the CPU of your machine is too old to be supported by XDS (which in this respect is compiled with the default options of the ifort compiler). This may happen e.g. with old AMD CPUs that don't support SSE2 (on Linux check the ''flags'' field of /proc/cpuinfo for the presence of the sse2 flag).

According to [http://en.wikipedia.org/wiki/SSE2#Notable_IA-32_CPUs_not_supporting_SSE2] the following Intel-compatible CPUs did not implement SSE2 after SSE2 was introduced (2001):
* AMD CPUs prior to Athlon 64, including all Socket A-based CPUs
* Intel CPUs prior to Pentium 4
* VIA C3
* Transmeta Crusoe

=== ASSERT VIOLATION ===

If you get e.g.
!!! ERROR !!! ASSERT VIOLATION IN "RESTHKL"
this indicates that an internal error has occurred, i.e. some situation was hit that the program is not prepared to handle.
There must be something very special about your dataset - maybe a very small number of reflections, maybe only negative intensities, maybe something else -
which triggers the bug. If this happens then get in touch with the XDS developers - they will most likely ask to make your XDS.INP and data frames available to them, to fix the program.

== Indexing failures ==

In case of IDXREF problems, you should definitively inspect, using [[XDS-viewer]] or [[adxv]], the FRAME.cbf file which is written by COLSPOT. This file shows the first frame of the SPOT_RANGE, together with strong pixels that COLSPOT found on this frame, marked with white pixels. IDXREF will use those spot positions which have at least as many strong pixels as MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT specifies.

You will only be able to look at this file if you prevent XDS from running INTEGRATE. This can be achieved e.g. by

JOB= XYCORR INIT COLSPOT IDXREF DEFPIX ! INTEGRATE CORRECT

since otherwise, FRAME.cbf is overwritten by INTEGRATE.

To look at '''all''' spot positions found by COLSPOT, you could try
echo "set yrange [] reverse ; plot 'SPOT.XDS' us 1:2 w dots" | gnuplot -persist
This may also help to find shaded regions on the detector.

=== IDXREF ends with !!! ERROR !!! message ===

These messages should really be called !!! WARNING !!!. The two types occuring most often are:

!!! ERROR !!! SOLUTION IS INACCURATE
This message is printed if the "STANDARD DEVIATION OF SPOT POSITION (PIXELS)" is larger than the MAXIMUM_ERROR_OF_SPOT_POSITION= (default of 3).

or

!!! ERROR !!! INSUFFICIENT PERCENTAGE (< 50%) OF INDEXED REFLECTIONS
This message is printed if less than the given percentage of reflections that COLSPOT found was actually indexed. If this occurs, it is clearly a warning sign.

The message continues in both cases:
AUTOMATIC DATA PROCESSING STOPPED. AS THE CRITERIA FOR A GOOD
SOLUTION ARE RATHER STRICT, YOU MAY CHOOSE TO CONTINUE DATA
PROCESSING AFTER CHANGING THE "JOB="-CARD IN "XDS.INP" TO
"JOB= DEFPIX INTEGRATE CORRECT".
IF THE BEST SOLUTION IS REALLY NONSENSE YOU SHOULD FIRST HAVE
A LOOK AT THE ASCII-FILE "SPOT.XDS". THIS FILE CONTAINS THE
INITIAL SPOT LIST SORTED IN DECREASING SPOT INTENSITY. SPOTS
NEAR THE END OF THE FILE MAY BE ARTEFACTS AND SHOULD BE ERASED.
ALTERNATIVELY YOU MAY TRY DIFFERENT VALUES FOR "INDEX_ORIGIN"
AS SUGGESTED IN THE ABOVE LISTING.
IF THE CRYSTAL HAS SLIPPED AT THE BEGINNING OF DATA COLLECTION
YOU MAY CHOOSE TO SKIP SOME OF THE FIRST FRAMES BY CHANGING
THE "DATA_RANGE=" IN FILE "XDS.INP" AND START ALL OVER AGAIN.

This is printed out for you to actually read, and take action accordingly. In most cases you just change the JOBS - line in XDS.INP to read
JOB= DEFPIX INTEGRATE CORRECT
and then continue to run XDS. In other cases you may want to change SPOT.XDS, or other keywords in [[XDS.INP]] (see also below). But in any case this is an important alert that should make you check the correctness of the parameters that describe the data collection (X-RAY_WAVELENGTH, DETECTOR_DISTANCE, ORGX, ORGY, OSCILLATION_RANGE, NAME_TEMPLATE_OF_DATA_FRAMES).

=== IDXREF produces too short cell parameter(s) ===

The first table "CLUSTER COORDINATES AND INDICES WITH RESPECT TO REC. LATTICE BASIS VECTORS" should show close-to-integer difference vectors. The following is an example how it should ''not'' look like:
# COORDINATES OF REC. BASIS VECTOR LENGTH 1/LENGTH

1 -0.0000476-0.0014008-0.0044654 0.0046802 213.67
2 -0.0101960-0.0056194 0.0019466 0.0118037 84.72
3 -0.0145331 0.0238225-0.0071754 0.0288134 34.71

CLUSTER COORDINATES AND INDICES WITH RESPECT TO REC. LATTICE BASIS VECTORS
# COORDINATES OF VECTOR CLUSTER FREQUENCY CLUSTER INDICES
1 0.0004199-0.0012633-0.0043826 2442. 0.97 -0.04 -0.00
2 0.0101757 0.0060701-0.0019630 2175. -0.03 -1.02 0.01
3 -0.0076118 0.0114167-0.0040603 1965. 0.13 0.04 0.50
4 0.0100552 0.0071646 0.0025337 1944. -1.01 -0.99 0.01
5 0.0072840-0.0101455 0.0084405 1841. -1.10 -0.01 -0.49
6 0.0000976 0.0027828 0.0089584 1792. -2.00 0.00 -0.00
7 0.0103103 0.0043851-0.0063902 1790. 0.98 -1.02 0.00
8 0.0025742 0.0163995-0.0098507 1760. 0.95 -0.99 0.51
9 0.0068686-0.0089437 0.0128884 1724. -2.08 0.03 -0.49
10 -0.0174443 0.0123504 0.0161324 1694. -4.01 0.99 0.52
11 0.0272195-0.0050005-0.0136142 1678. 2.99 -1.97 -0.50
...

PARAMETERS OF THE REDUCED CELL (ANGSTROEM & DEGREES)
34.71 84.72 213.67 90.35 90.27 90.06
Do you see the half-integer difference vectors in the last column? This clearly indicates that the third axis above the table (with 1/length=34.71) actually should be twice that size. If this happens, the histogram of indexed spots often has two equally-large subtrees:
SUBTREE POPULATION

1 976
2 972
3 10
4 7
5 6

Remedy: take the reduced cell (as found below the table), but double the 34.71, and stick it into the line
UNIT_CELL_CONSTANTS=69.42 84.72 213.67 90.35 90.27 90.06
in XDS.INP. To make XDS actually ''use'' that line, specify
SPACE_GROUP_NUMBER=1
and leave the space group determination for later.

Alternatively, try several well-separated SPOT_RANGEs instead of 1. For example, instead of
SPOT_RANGE=1 900
try e.g.
SPOT_RANGE=1 100
SPOT_RANGE=801 900
In the cases where I've tried this, this splitting of SPOT_RANGE into several wedges always resulted in IDXREF picking up the correct cell.

The reason for the indexing failure in these cases seems to be the fact that the crystal changed its orientation within the SPOT_RANGE by more than ~ 0.1°.

=== IDXREF produces too long axes ===

This may (rarely) happen when MAXIMUM_NUMBER_OF_JOBS is > 1. In this case, different JOBS' COLSPOT runs may report some reflections twice in SPOT.XDS. Since their phi values are close, they correspond to long unit cell parameters. This happens more easily if the mosaicity is high and therefore reflections extend over the borders between JOBs.

This effect may be mitigated by having as many SPOT_RANGEs as JOBs, and leaving gaps between the SPOT_RANGEs.

=== IDXREF prints !!! WARNING !!! message ===

If you see
!!! WARNING !!! REFINEMENT DID NOT CONVERGE
LAST CORRECTION SHIFT WAS 6.1E-02 (should be < 1.0E-03)
then try the following: find the line
REFINE(IDXREF)= ...
in XDS.INP. If it is commented out with a !, remove the !. Then, change it to have
REFINE(IDXREF)=CELL BEAM ORIENTATION AXIS ! DISTANCE
i.e. remove DISTANCE from the list of refinable parameters. Once that is done, save XDS.INP and re-run the IDXREF step.

This problem occurs if the DISTANCE is large, and XDS cannot refine it meaningfully.

If this does not help, try to refine even less items, e.g. leave out AXIS.

=== IDXREF.LP does not show the expected lattice ===

By specifying the spacegroup and unitcell, you tell XDS that it should index based on those reflections that match that spacegroup and unitcell. In a sense, you _force_ that spacegroup and cell. This will discard other (spurious?) reflections, and usually leads to a clean list of Bravais lattice possibilities.

If the data reduction fails nevertheless (in terms of bad R-factors and [[CORRECT#An estimate for the overall quality of an experimental setup|ISa]] in the [[CORRECT]] step), then chances are that you specified some parameter wrongly, or not accurate enough (ORGY and ORGY are the most likely candidates), or that the crystal does not match your idea about its spacegroup and unit cell. Actually the latter happens pretty frequently (which is why it is always the safest way to collect 180° of spindle rotation unless you know your crystals very well; this is also a good strategy in other respects).

If you tell XDS that you do not to know the spacegroup (SPACE_GROUP_NUMBER= 0), then [[IDXREF]] takes all observed reflections into account. By design, the spacegroup decision is then postponed until the CORRECT step, or rather to a run of [[pointless]] after CORRECT, and it may be not vital to closely inspect IDXREF.LP, because CORRECT.LP has basically the same information, plus more.

=== none of the lattices in IDXREF.LP (except aP) has a good QUALITY OF FIT ===

It is a good idea to use many frames for the SPOT_RANGE, (e.g. the first half of the DATA_RANGE, as does the [[generate_XDS.INP]] script) for the COLSPOT and IDXREF steps.

It is entirely possible to run COLSPOT for the ''complete'' DATA_RANGE (in that case the parameters of SPOT_RANGE are the same as for the DATA_RANGE), and to try smaller SPOT_RANGEs in IDXREF - this means COLSPOT has to be run only once, and the JOBS= line has e.g.
JOBS= IDXREF
only. Maybe some SPOT_RANGE sub-range gives a clear answer. In that case it may be useful to use
REFINE(INTEGRATE)=! AXIS BEAM ORIENTATION CELL DISTANCE
because otherwise the spurious reflections in the other ranges will probably disturb the on-the-fly refinement of parameters.

I've seen datasets where each reflection had a satellite associated with the main reflection, but separate from it. This is well visible in FRAME.cbf written by COLSPOT. In such a case it helps to use
MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT= 12 ! re-run COLSPOT after changing the parameter !
thus doubling (w.r.t. the default of 6) the required spotsize that makes a reflection be used for indexing. In the latest case this made it possible to index cleanly on a single frame (which is actually not uncommon).

A final possibility: your crystal may really be triclinic - hopefully you collected 180° of data, or even a bit more than that.

== Integration failure ==

If INTEGRATE stops after e.g.
******************************************************************************
PROCESSING OF IMAGES 136 ... 140
******************************************************************************
with
!!! ERROR !!! AUTOMATIC DETERMINATION OF SPOT SIZE PARAMETERS HAS FAILED.
YOU MAY RESTART THIS STEP AFTER SPECIFYING VALUES IN XDS.INP FOR:
REFLECTING_RANGE= REFLECTING_RANGE_E.S.D.=
BEAM_DIVERGENCE= BEAM_DIVERGENCE_E.S.D.=

then you should reduce the upper limit of the DATA_RANGE, to stop before the problematic frames, and re-run INTEGRATE. In this example, you would modify XDS.INP to have
DATA_RANGE=1 135
JOB=INTEGRATE CORRECT
Save XDS.INP, run XDS and inspect INTEGRATE.LP, to find the lines (e.g.)
BEAM_DIVERGENCE= 0.478 BEAM_DIVERGENCE_E.S.D.= 0.048
REFLECTING_RANGE= 1.100 REFLECTING_RANGE_E.S.D.= 0.157
Copy them to XDS.INP. Restore the original DATA_RANGE and continue.

Another error mode of INTEGRATE (in processing of small-molecule data) is ...
IMAGE IER SCALE NBKG NOVL NEWALD NSTRONG NREJ SIGMAB SIGMAR
5 0 0.975 3968463 1 5 1 0 0.1028 1.3486
6 0 0.967 3969949 2 2 2 0 0.1688 0.0000
7 0 0.929 3975306 0 3 0 0 0.0000 0.0000
!!! ERROR !!! CANNOT ALLOCATE MEMORY
YOU COULD RERUN THIS STEP WITH SMALLER VALUES FOR THE PARAMETERS
NUMBER_OF_PROFILE_GRID_POINTS_ALONG_ALPHA/BETA=
NUMBER_OF_PROFILE_GRID_POINTS_ALONG_GAMMA=
The error message is misleading in this case: there are too few reflections to build the average profile. The fix is: restart INTEGRATE after inserting e.g.
DELPHI=20 ! default is 5, so try with e.g. 10, 20, 45, 90, 180
and re-run INTEGRATE.

== See also ==
[[XDS.INP#What can go wrong with this file?]]

[[Low dose data]]

Scripts and interfaces

2014-09-10T13:39:11Z

Karine: update

There is a number of script and graphical interfaces for the XDS package.

== XDSi ==

the first published interface, by Petri Kursula (XDSi: a graphical interface for the data processing program XDS. (2004) J. Appl. Cryst. 37, 347-348).

http://cc.oulu.fi/~pkursula/xdsi.html gives 2007 as last modification date.

== [[XDSi]] ==

Tcl/Tk interface by Michael Krug (Welte/Diederichs lab). Unfortunately same name. Can process multiple datasets.

== XDSAPP ==

python/Qt version of [[XDSi]], developed at [http://www.helmholtz-berlin.de/forschung/oe/em/soft-matter/forschung/bessy-mx/xdsapp/index_en.html BESSY].

== [[XDSGUI]] ==

to edit input files and run commands.

== ixds ==

python script programmed at BESSY.

== autoxds ==

script in use at SSRL (Stanford) . See [http://smb.slac.stanford.edu/facilities/software/xds/ A Quick XDS Tutorial for SSRL].

Download: http://smb.slac.stanford.edu/templates/xds/autoxds ; download http://smb.slac.stanford.edu/templates/xds/mccd_xdsparams.pl , too!

== autoPROC ==

software package, not exclusively using XDS, available at http://www.globalphasing.com/autoproc/ (Vonrhein ''et al.'' (2011). Acta Cryst. D67, 293-302)

== AutoProcess ==

Pipeline developed by Michel Fodje at Canadian Macromolecular Crystallography Facility (Saskatoon).

Usage:

autoprocess [options] /path/to/set1.img /path/to/set2.img ... /path/to/setn.img

options:
--mad, -m : Process each set, scale together and generate separate reflection files.
--screen, -s : Process a few frames from characterize crystal from each set.
--anom, -a : Process with Friedel's law False
--backup, -b : Backup previous output directory if it exists
--prefix=p1,p2,p3 : comma separated list of prefixes to use for output files.
Default is first part of image name
prefix order should correspond to the order of the data sets
for example for MAD data, use --prefix=peak,infl,remo
--dir=/path : Directory to store processed results. Default is to create a new one in the current directory.
--inputs, -i: generate XDS.INP only and quit
--help, -h : display this message
Default (no option): Process each set, scale together and merge into one reflection file.

Data sets:
Each data set can be represented by any frame from that set.

== xdsme ==

by Pierre Legrand at SOLEIL (French synchrotron): web address: http://code.google.com/p/xdsme/

Here's a short description of the program:

xdsme is a collection of python scripts made to simplify the processing of crystal
diffraction images with the XDS Program Package. Provided that the diffraction
parameters are well recorded in the diffraction image headers, XDS data processing
can be started with a simple command line like:

$ xdsme pos1_1_???.img

Supported detector image format include: ADSC, MARCCD, MAR345 + experimental
for PILATUS, SATURN, RAXIS and MAR555.

* xdsme (XDS.py), xscale.py and xdsconv.py for data processing, scaling and file conversion.
* XOalign.py for the goniometer setting calculation (to work with different type of goniometer including Kappa, mini-Kappa, Euler...).
* xds2mos.py or xds2dnz.py ... (for conversion of orientation matrices)

== [[generate_XDS.INP]] ==

is a script for generating a XDS.INP for a given dataset. Under development; works well for MarCCD, ADSC and Pilatus detectors.

Talk:Generate XDS.INP

2014-09-10T09:00:39Z

Karine: reference?

== Reference ==
Hello,

when mentionning the use of generate_XDS.INP in a paper, which reference should be given? [[User:Karine|Karine]] ([[User talk:Karine|talk]]) 11:00, 10 September 2014 (CEST)

Generate XDS.INP

2014-03-10T10:09:19Z

Karine: Calling generate_XDS.INP from a Python script

This script generates XDS.INP based on a list of frame names supplied on the commandline. It currently works for MarCCD, ADSC, Pilatus and a few Rigaku detectors; since this is just a bash script, extension to other detectors is very easy.

On Mac OS X, installation of the "Command Line Tools" (from http://developer.apple.com/downloads; requires Apple ID) is required (I think). These are a part of the (larger, but also free) [http://developer.apple.com/tools/xcode Xcode] package.

== Usage ==

Usage is just (don't forget the quotation marks!):
generate_XDS.INP "/home/myname/frms/mydata_1_???.img"
XDS [http://strucbio.biologie.uni-konstanz.de/~dikay/XDS_html_doc/html_doc/xds_parameters.html#NAME_TEMPLATE_OF_DATA_FRAMES= supports] bzip2-ed frames. Thus, when specifying the frame name parameter of the script, you should leave out any .bz2 extension.

For improved interaction with [[XDSGUI]], it is advantageous to use an ''absolute'' filename - one that starts with a slash ("/").

== The script ==

<pre>
#!/bin/bash
# purpose: generate XDS.INP
#
# tested with some datasets from ALS, SSRL, SLS, ESRF, BESSY, SPring-8 and PF; only MARCCD, ADSC/SMV, PILATUS, RAXIS (in-house) detectors;
# for other detectors, values marked with XXX must be manually filled in.
#
# revision 0.03 . Kay Diederichs 2/2010
# revision 0.04 . Kay Diederichs 4/2010 - include alternative ORGX, ORGY calculations for ADSC
# revision 0.05 . Kay Diederichs 5/2010 - grep for "Corrected" in addition to "marccd"; needed for BESSY
# revision 0.06 . KD 6/2010 - add UNTRUSTED_RECTANGLE and UNTRUSTED_ELLIPSE; use `whereis catmar` and so on
# revision 0.07 . KD 6/2010 - decide about ORGX/Y info in MAR header being pixels or mm; other fixes
# revision 0.08 . KD 6/2010 - fixes for Pilatus 6M
# revision 0.09 . KD 6/2010 - get rid of requirement for mccd_xdsparams.pl and/or catmar; rather use "od"
# revision 0.10 . Tim Gruene 7/2010 - set link 'images' to image directory if path exceeds 72 characters
# revision 0.11 . KD 7/2010 - for MarCCD: look for distance info at different byte position
# revision 0.12 . KD 7/2010 - fix for negative PHISTART
# revision 0.13 . KD 8/2010 - store correct NX NY QX QY in XDS.INP
# revision 0.14 . KD 1/2011 - SENSOR_THICKNESS for Pilatus; MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT=3
# revision 0.15 . KD 2/2011 - add comment for -ive sign of APS 19-ID and Australian Synchrotron rotation axis
# revision 0.16 . KD 3/2011 - SENSOR_THICKNESS=0.01 for ADSC and MarCCD. Add comment about SILICON=
# revision 0.17 . KD 3/2011 - make it work for .bz2 frames; improve screen output
# revision 0.18 . KD 4/2011 - faster by doing "strings" only once; revert "images/${1##/*/}" "correction"
# revision 0.19 . KD 6/2011 - bugfix for 0.18
# revision 0.20 . KD 7/2011 - redirect stderr of /bin/ls to /dev/null
# revision 0.21 . KD 11/2011 - SEPMIN, CLUSTER_RADIUS hints; read NX NY from header (for Pilatus 2M)
# revision 0.22 . KD 12/2011 - Pilatus 2M UNTRUSTED_RECTANGLE lines, SENSOR_THICKNESS from header
# revision 0.23 . KD 1/2012 - add UNTRUSTED_QUADRILATERAL, remove MINIMUM_ZETA (0.05 is default now)
# revision 0.24 . KD 3/2012 - remove revision 0.10 since XDS now takes much longer paths
# revision 0.25 . KD 3/2012 - remove revision 0.22 for PSI Pilatus 2M; see http://www.globalphasing.com/autoproc/wiki/index.cgi?TroubleShootingKnownIssues
# revision 0.26 . KD 7/2012 - Mac-compatibility: replace od flags --skip-bytes= and --read-bytes= with -j and -N (thanks to Oliver Clarke for working this out!)
# revision 0.27 . KD 11/2012 - EXCLUDE_RESOLUTION_RANGE lines and generic Pilatus Flat_field test
# revision 0.28 . Keitaro 11/2012 - for MarCCD: read oscillation range from the position 1024+736 (fix for omega rotation)
# revision 0.29 . KD 1/2013 - include UNTRUSTED_RECTANGLEs for Pilatus 6M; never hurts but needed if the beamline software does not mark them with -2 or such
# revision 0.30 . Keitaro 3/2013 - for ADSC: write all possible beam center conventions in XDS.INP as comments
# revision 0.31 . Keitaro 3/2013 - add comment for reversed phi for SPring-8
# revision 0.32 . Keitaro 3/2013 - add RAXIS support. only tested with RAXIS IV++ and VII.
# revision 0.33 . Keitaro 5/2013 - automatically set ROTATION_AXIS=-1 0 0 for SPring-8 BL32XU/41XU/44XU beamlines based on detector serial numbers.
# revision 0.34 . Keitaro 5/2013 - recognize ADSC detectors in Photon Factory and choose correct beam center convention based on detector serial numbers.
# revision 0.35 . KD 6/2013 - reduce 7000 to 6000 for shadow detection; insert comment about *_RESOLUTION_RANGE lines
# revision 0.36 . KD 6/2013 - insert NUMBER_OF_PROFILE_GRID* lines for Pilatus (suggested by C.Vonrhein)
# revision 0.37 . Keitaro 10/2013 - fix for MX225HS detector on SPring-8 BL32XU (Ignore case when matching marccd in header); but cannot decide reverse-phi because of missing detector S/N
# revision 0.38 . KD 2/2014 - change defaults for REFINE(IDXREF) and REFINE(INTEGRATE) such that more stable results are obtained for difficult datasets
REVISION="0.38 (15-February-2014)"
#
# usage: e.g. generate_XDS.INP "frms/mydata_1_???.img"
# make sure to have the two quotation marks !
# the ? are wildcards for the frame numbers.
#
# limitations:
# - frame numbers are assumed to start with 1 and run consecutively
#
# known problems:
# - for ADSC detectors, there are at least three ways to obtain ORGX and ORGY values from the header (see below);
# - the same might be a problem for MAR headers, too (not sure about this)
# - on Mac OS X, Xcode (from http://developer.apple.com/tools/xcode) might be needed - not sure about this
#
# notes for debugging of the script:
# - add the -v option to the first line, to see where an error occurs
# - comment out the removal of tmp1 and tmp2 in the last line
#
# ====== Start of script ======
echo generate_XDS.INP version $REVISION . Obtain the latest version from
echo http://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/generate_XDS.INP
if [ "$1" == "help" ] || [ "$1" == "-help" ] || [ "$1" == "-h" ]; then
echo usage: generate_XDS.INP \"frms/mydata_1_???.img\" $_with_ the quotation marks!$
echo if the frames are compressed with bzip2, leave out the .bz2 extension!
exit
fi
#
# defaults:
#
DETECTOR="XXX MINIMUM_VALID_PIXEL_VALUE=XXX OVERLOAD=XXX"
REVERSE_PHI="no"
ORGX=XXX
ORGY=XXX
DETECTOR_DISTANCE=XXX
OSCILLATION_RANGE=XXX
X_RAY_WAVELENGTH=XXX
QX=XXX
QY=XXX
NX=XXX
NY=XXX
SENSOR_THICKNESS=0
# see how we are called:
NAME_TEMPLATE_OF_DATA_FRAMES="$1"
# list frames matching the wildcards in NAME_TEMPLATE_OF_DATA_FRAMES
# don't accept the "direct beam" shot at SLS/Pilatus PX-I and PX-II
/bin/ls -C1 $1 $1.bz2 2>/dev/null | egrep -v "_00000.cbf|_000.img" > tmp1 || exit 1

# we can continue - the frames are found

# set upper limit of DATA_RANGE to number of frames (see "limitations" above)
DATA_RANGE=`wc -l tmp1 | awk '{print $1}'`

# set upper limit of SPOT_RANGE to half of DATA_RANGE, but not less than 1
SPOT_RANGE=`echo "scale=0; $DATA_RANGE/2" | bc -l`
SPOT_RANGE=`echo "if ($SPOT_RANGE<1) 1;if ($SPOT_RANGE>1) $SPOT_RANGE" | bc -l`

echo DATA_RANGE=1 $DATA_RANGE

# find out detector type
DET=XXX
FIRSTFRAME=`head -1 tmp1`
echo $FIRSTFRAME | grep -q bz2 && bzcat $FIRSTFRAME > tmp1 && FIRSTFRAME=tmp1
strings $FIRSTFRAME > tmp2
egrep -qi 'marccd|Corrected' tmp2 && DET=mccd
grep -q PILATUS tmp2 && DET=pilatus
grep -q BEAM_CENTER_X tmp2 && DET=adsc
head -n1 tmp2 | grep -q "^RAXIS" && DET=raxis
# identify other detector types in the same way (MAR IP would be straightforward)

# parse ASCII header of first frame

if [ "$DET" == "XXX" ]; then
echo "this is not a MAR, ADSC/SMV or PILATUS detector - fill in XXX values manually!"
DETECTOR="XXX MINIMUM_VALID_PIXEL_VALUE=XXX OVERLOAD=XXX"

# find parameters of first frame
elif [ "$DET" == "mccd" ]; then
echo Data from a MarCCD detector

DETECTOR="CCDCHESS MINIMUM_VALID_PIXEL_VALUE= 1 OVERLOAD= 65500"
SENSOR_THICKNESS=0.01
# use first frame of dataset to obtain parameters

# Check detector serial number and recognize beamline for reversed-phi setting.
# Known detectors for reversed-phi in SPring-8: 31: BL32XU MX225HE, 38: BL44XU MX225HE, 42: BL44XU MX300HE, 40: BL41XU MX225HE
REVERSEPHI_SNs="
31
38
40
42
"
# get detector serial number and check if it is included in the list
DET_SN=`grep "Detector Serial Number =" tmp2 | sed "s/Detector Serial Number = //"`
if echo "${DET_SN}${REVERSEPHI_SNs}" | sort | uniq -d | grep [0-9] > /dev/null; then
REVERSE_PHI="yes"
fi

# offsets are documented; values can be found in mccd_xdsparams.pl script
let SKIP=1024+80
NX=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
let SKIP=$SKIP+4
NY=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')

let SKIP=1720
DETECTOR_DISTANCE=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
DETECTOR_DISTANCE=`echo "scale=3; $DETECTOR_DISTANCE/1000" | bc -l`

let SKIP=1024+256+128+256+4
ORGX=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
ORGX=`echo "scale=2; $ORGX/1000" | bc -l `
let SKIP=$SKIP+4
ORGY=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
ORGY=`echo "scale=2; $ORGY/1000" | bc -l `

let SKIP=1024+736
OSCILLATION_RANGE=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
OSCILLATION_RANGE=`echo "scale=3; $OSCILLATION_RANGE/1000" | bc -l`

let SKIP=1024+256+128+256+128+4
QX=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
QX=`echo "scale=10; $QX/1000000" |bc -l `
let SKIP=$SKIP+4
QY=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
QY=`echo "scale=10; $QY/1000000" |bc -l `

let SKIP=1024+256+128+256+128+128+12
X_RAY_WAVELENGTH=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
X_RAY_WAVELENGTH=`echo "scale=5; $X_RAY_WAVELENGTH/100000" | bc -l`

# at most BLs, ORGX and ORGY are in pixels, but sometimes in mm ... guess:
NXBYFOUR=`echo "scale=0; $NX/4" | bc -l `
ORGXINT=`echo "scale=0; $ORGX/1" | bc -l `
if [ $ORGXINT -lt $NXBYFOUR ]; then
ORGX=`echo "scale=1; $ORGX/$QX" | bc -l`
ORGY=`echo "scale=1; $ORGY/$QY" | bc -l`
echo MARCCD detector: header ORGX, ORGY seem to be in mm ... converting to pixels
else
echo MARCCD detector: header ORGX, ORGY seem to be in pixel units
fi

elif [ "$DET" == "adsc" ]; then

DETECTOR="ADSC MINIMUM_VALID_PIXEL_VALUE= 1 OVERLOAD= 65000"
echo Data from ADSC detector. Obtaining ORGX, ORGY depends on beamline setup:
SENSOR_THICKNESS=0.01
sed s/\;// tmp2 > tmp1
mv tmp1 tmp2

# find X_RAY_WAVELENGTH:
X_RAY_WAVELENGTH=`grep WAVELENGTH tmp2 | head -1 | sed s/WAVELENGTH=//`

# find NX, QX, ORGX and ORGY:
NX=`grep SIZE1 tmp2 | tail -1 | sed s/SIZE1=//`
QX=`grep PIXEL_SIZE tmp2 | sed s/PIXEL_SIZE=//`
# FIXME - next 2 lines should be done properly, from header
NY=$NX
QY=$QX
BEAM_CENTER_X=`grep BEAM_CENTER_X tmp2 | sed s/BEAM_CENTER_X=//`
BEAM_CENTER_Y=`grep BEAM_CENTER_Y tmp2 | sed s/BEAM_CENTER_Y=//`
# fix 2010-04-26 - tell user about possible ORGX, ORGY alternatives -
COMMENT_ORGXY="
! Following are possible beam center interpretations for ADSC detectors"
# at ESRF, PF, and ... (pls fill in!) the following should be used:
ORGX1=`echo "scale=1; $BEAM_CENTER_Y/$QX" | bc -l`
ORGY1=`echo "scale=1; $BEAM_CENTER_X/$QX" | bc -l`
echo - at ESRF,PF BLs use: ORGX=$ORGX1 ORGY=$ORGY1
COMMENT_ORGXY="${COMMENT_ORGXY}
! ORGX= $ORGX1 ORGY= $ORGY1 ! For ESRF,PF,..."
# this 2nd alternative convention should be used at the following beamlines (pls complete the list): ALS 5.0.3, ...
ORGX2=`echo "scale=1; $NX-$BEAM_CENTER_X/$QX" | bc -l `
ORGY2=`echo "scale=1; $BEAM_CENTER_Y/$QX" | bc -l `
echo - at e.g. ALS 5.0.3 use: ORGX=$ORGX2 ORGY=$ORGY2
COMMENT_ORGXY="${COMMENT_ORGXY}
! ORGX= $ORGX2 ORGY= $ORGY2 ! For ALS 5.0.3,.."
# this 3rd alternative convention should be used at the following beamlines (pls complete the list): ALS 8.2.2, ...
ORGX3=`echo "scale=1; $BEAM_CENTER_X/$QX" | bc -l `
ORGY3=`echo "scale=1; $NX-$BEAM_CENTER_Y/$QX" | bc -l `
echo - at e.g. ALS 8.2.2 use: ORGX=$ORGX3 ORGY=$ORGY3 - this is written to XDS.INP
COMMENT_ORGXY="${COMMENT_ORGXY}
! ORGX= $ORGX3 ORGY= $ORGY3 ! For ALS 8.2.2,.."
# the latter alternative is written into the generated XDS.INP ! You have to correct this manually in XDS.INP, or adjust this script.

# Decision of beam center convention based on detector serial numbers.
DET_SN=`grep DETECTOR_SN tmp2 | sed -e "s/DETECTOR_SN=//"`
# For convention 1; Known PF detectors = 449: NW12A Q210, 472: NE3A Q270, 474: BL17A Q270, 912: BL5A Q315
ORG1_SNs="
449
472
474
912
"
if echo "${DET_SN}${ORG1_SNs}" | sort | uniq -d | grep [0-9] > /dev/null; then
ORGX=$ORGX1
ORGY=$ORGY1
else
ORGX=$ORGX3
ORGY=$ORGY3
fi

# find DETECTOR_DISTANCE and OSCILLATION_RANGE:
DETECTOR_DISTANCE=`grep DISTANCE tmp2 | sed s/DISTANCE=//`
OSCILLATION_RANGE=`grep OSC_RANGE tmp2 | sed s/OSC_RANGE=//`

elif [ "$DET" == "pilatus" ]; then
DETECTOR="PILATUS MINIMUM_VALID_PIXEL_VALUE=0 OVERLOAD= 1048576 !PILATUS"
QX=0.172 QY=0.172
echo Data from a Pilatus detector
sed s/#// tmp2 > tmp1
mv tmp1 tmp2

# find SENSOR_THICKNESS:
SENSOR_THICKNESS=`grep thickness tmp2 | sed -e s/'Silicon sensor, thickness'// | awk '{print $1*1000}'`
# find X_RAY_WAVELENGTH:
X_RAY_WAVELENGTH=`grep Wavelength tmp2 | sed -e s/Wavelength// -e s/A// | awk '{print $1}'`

# find NX and NY; 2463/2527 is 6M, 1475/1679 is 2M
NX=`grep X-Binary-Size-Fastest-Dimension tmp2 | awk '{print $2}'`
NY=`grep X-Binary-Size-Second-Dimension tmp2 | awk '{print $2}'`

# find ORGX and ORGY:
ORGX=`grep Beam_xy tmp2 | sed -e s/$// -e s/$// -e s/\,// | awk '{print $2}'`
ORGY=`grep Beam_xy tmp2 | sed -e s/$// -e s/$// -e s/\,// | awk '{print $3}'`

# find DETECTOR_DISTANCE and OSCILLATION_RANGE:
DETECTOR_DISTANCE=`awk '/distance/{print $2}' tmp2`
DETECTOR_DISTANCE=`echo "$DETECTOR_DISTANCE*1000" | bc -l`

OSCILLATION_RANGE=`awk '/Angle/{print $2}' tmp2`

elif [ "$DET" == "raxis" ]; then
echo Data from a RAXIS detector

DETECTOR="RAXIS MINIMUM_VALID_PIXEL_VALUE=0 OVERLOAD=2000000"
#let SKIP=768
#NX=$(od -t x -j $SKIP -N 4 $FIRSTFRAME |awk 'NR==1{print toupper($2)}'|perl -nle '@array= $_ =~/.{2}/g; print "ibase=16;obase=A;".join("",reverse @array)'|bc)
NX=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(768);print "%.4d"%struct.unpack(">i",f.read(4))')
NY=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(772);print "%.4d"%struct.unpack(">i",f.read(4))')

DETECTOR_DISTANCE=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(344);print "-%.4f"%struct.unpack(">f",f.read(4))')

ORGX=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(540);print "%.4f"%struct.unpack(">f",f.read(4))')
ORGY=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(544);print "%.4f"%struct.unpack(">f",f.read(4))')

OSCILLATION_RANGE=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(524);phis,phie=struct.unpack(">ff",f.read(8));print "%.4f"%(phie-phis)')

QX=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(776);print "%.6f"%struct.unpack(">f",f.read(4))')
QY=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(780);print "%.6f"%struct.unpack(">f",f.read(4))')

X_RAY_WAVELENGTH=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(292);print "%.6f"%struct.unpack(">f",f.read(4))')
else
echo should never come here
exit 1
fi

echo ORGX= $ORGX ORGY= $ORGY - check these values with adxv !
echo DETECTOR_DISTANCE= $DETECTOR_DISTANCE
echo OSCILLATION_RANGE= $OSCILLATION_RANGE
echo X-RAY_WAVELENGTH= $X_RAY_WAVELENGTH

# now we know everything that is required to generate XDS.INP

cat > XDS.INP << eof
! written by generate_XDS.INP version $REVISION
JOB= XYCORR INIT COLSPOT IDXREF DEFPIX INTEGRATE CORRECT
ORGX= $ORGX ORGY= $ORGY ! check these values with adxv !\
$COMMENT_ORGXY
DETECTOR_DISTANCE= $DETECTOR_DISTANCE
OSCILLATION_RANGE= $OSCILLATION_RANGE
X-RAY_WAVELENGTH= $X_RAY_WAVELENGTH
NAME_TEMPLATE_OF_DATA_FRAMES=$NAME_TEMPLATE_OF_DATA_FRAMES
! REFERENCE_DATA_SET=xxx/XDS_ASCII.HKL ! e.g. to ensure consistent indexing
DATA_RANGE=1 $DATA_RANGE
SPOT_RANGE=1 $SPOT_RANGE
! BACKGROUND_RANGE=1 10 ! rather use defaults (first 5 degree of rotation)

SPACE_GROUP_NUMBER=0 ! 0 if unknown
UNIT_CELL_CONSTANTS= 70 80 90 90 90 90 ! put correct values if known
INCLUDE_RESOLUTION_RANGE=50 0 ! after CORRECT, insert high resol limit; re-run CORRECT
! IDXREF now obeys INCLUDE_RESOLUTION_RANGE and EXCLUDE_RESOLUTION_RANGE to exclude ice-rings

FRIEDEL'S_LAW=FALSE ! This acts only on the CORRECT step
! If the anom signal turns out to be, or is known to be, very low or absent,
! use FRIEDEL'S_LAW=TRUE instead (or comment out the line); re-run CORRECT

! remove the "!" in the following line:
! STRICT_ABSORPTION_CORRECTION=TRUE
! if the anomalous signal is strong: in that case, in CORRECT.LP the three
! "CHI^2-VALUE OF FIT OF CORRECTION FACTORS" values are significantly> 1, e.g. 1.5
!
! exclude (mask) untrusted areas of detector, e.g. beamstop shadow :
! UNTRUSTED_RECTANGLE= 1800 1950 2100 2150 ! x-min x-max y-min y-max ! repeat
! UNTRUSTED_ELLIPSE= 2034 2070 1850 2240 ! x-min x-max y-min y-max ! if needed
! UNTRUSTED_QUADRILATERAL= x1 y1 x2 y2 x3 y3 x4 y4 ! see documentation
!
! parameters with changes wrt default values:
TRUSTED_REGION=0.00 1.2 ! partially use corners of detectors; 1.41421=full use
VALUE_RANGE_FOR_TRUSTED_DETECTOR_PIXELS=6000. 30000. ! often 7000 or 8000 is ok
STRONG_PIXEL=4 ! COLSPOT: only use strong reflections (default is 3)
MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT=3 ! default of 6 is sometimes too high
! close spots: reduce SEPMIN and CLUSTER_RADIUS from their defaults of 6 and 3, e.g. to 4 and 2
! nowadays headers are usually correct so refine DISTANCE in INTEGRATE but not IDXREF
REFINE(IDXREF)=CELL BEAM ORIENTATION AXIS ! DISTANCE
REFINE(INTEGRATE)= DISTANCE BEAM ORIENTATION ! AXIS CELL
! REFINE(CORRECT)=CELL BEAM ORIENTATION AXIS DISTANCE ! Default is: refine everything

! parameters specifically for this detector and beamline:
DETECTOR= $DETECTOR
SENSOR_THICKNESS= $SENSOR_THICKNESS
! attention CCD detectors: for very high resolution (better than 1A) make sure to specify SILICON
! as about 32* what CORRECT.LP suggests (absorption of phosphor is much higher than that of silicon)
NX= $NX NY= $NY QX= $QX QY= $QY ! to make CORRECT happy if frames are unavailable

eof
if [ "$DET" == "raxis" ]; then
cat >> XDS.INP << eof
DIRECTION_OF_DETECTOR_X-AXIS=1 0 0
DIRECTION_OF_DETECTOR_Y-AXIS=0 -1 0
INCIDENT_BEAM_DIRECTION=0 0 1
ROTATION_AXIS=0 1 0
!FRACTION_OF_POLARIZATION=0.98 ! uncomment if synchrotron
POLARIZATION_PLANE_NORMAL=1 0 0
eof
else
if [ "$REVERSE_PHI" == "no" ]; then
echo 'ROTATION_AXIS=1 0 0 ! at e.g. Australian Synchrotron, SERCAT ID-22 (?), APS 19-ID (?), ESRF BM30A, SPring-8 this needs to be -1 0 0' >> XDS.INP
else
echo 'ROTATION_AXIS=-1 0 0 ! if this is wrong, please contact author.' >> XDS.INP
fi
cat >> XDS.INP << eof
DIRECTION_OF_DETECTOR_X-AXIS=1 0 0
DIRECTION_OF_DETECTOR_Y-AXIS=0 1 0
INCIDENT_BEAM_DIRECTION=0 0 1
FRACTION_OF_POLARIZATION=0.98 ! better value is provided by beamline staff!
POLARIZATION_PLANE_NORMAL=0 1 0
eof
fi
cat >> XDS.INP << eof
!used by DEFPIX and CORRECT to exclude ice-reflections / ice rings - uncomment if necessary
!EXCLUDE_RESOLUTION_RANGE= 3.93 3.87 !ice-ring at 3.897 Angstrom
!EXCLUDE_RESOLUTION_RANGE= 3.70 3.64 !ice-ring at 3.669 Angstrom
!EXCLUDE_RESOLUTION_RANGE= 3.47 3.41 !ice-ring at 3.441 Angstrom
!EXCLUDE_RESOLUTION_RANGE= 2.70 2.64 !ice-ring at 2.671 Angstrom
!EXCLUDE_RESOLUTION_RANGE= 2.28 2.22 !ice-ring at 2.249 Angstrom
!EXCLUDE_RESOLUTION_RANGE= 2.102 2.042 !ice-ring at 2.072 Angstrom - strong
!EXCLUDE_RESOLUTION_RANGE= 1.978 1.918 !ice-ring at 1.948 Angstrom - weak
!EXCLUDE_RESOLUTION_RANGE= 1.948 1.888 !ice-ring at 1.918 Angstrom - strong
!EXCLUDE_RESOLUTION_RANGE= 1.913 1.853 !ice-ring at 1.883 Angstrom - weak
!EXCLUDE_RESOLUTION_RANGE= 1.751 1.691 !ice-ring at 1.721 Angstrom - weak
eof
if [ "$DET" == "pilatus" ]; then
cat >> XDS.INP << eof
NUMBER_OF_PROFILE_GRID_POINTS_ALONG_ALPHA/BETA=13 ! Default is 9 - Increasing may improve data
NUMBER_OF_PROFILE_GRID_POINTS_ALONG_GAMMA=13 ! accuracy, particularly if finely-sliced on phi,
! and does not seem to have any downsides.
eof
if [ $NX == "1475" ]; then
if ! grep -q Flat_field tmp2 ; then
cat >> XDS.INP << eof
! the following specifications are for a detector _without_ proper
! flat_field correction; they cut away one additional pixel adjacent
! to each UNTRUSTED_RECTANGLE
!EXCLUSION OF VERTICAL DEAD AREAS OF THE PILATUS 2M DETECTOR
UNTRUSTED_RECTANGLE= 486 496 0 1680
UNTRUSTED_RECTANGLE= 980 990 0 1680
!EXCLUSION OF HORIZONTAL DEAD AREAS OF THE PILATUS 2M DETECTOR
UNTRUSTED_RECTANGLE= 0 1476 194 214
UNTRUSTED_RECTANGLE= 0 1476 406 426
UNTRUSTED_RECTANGLE= 0 1476 618 638
UNTRUSTED_RECTANGLE= 0 1476 830 850
UNTRUSTED_RECTANGLE= 0 1476 1042 1062
UNTRUSTED_RECTANGLE= 0 1476 1254 1274
UNTRUSTED_RECTANGLE= 0 1476 1466 1486
eof
else
cat >> XDS.INP << eof
!EXCLUSION OF VERTICAL DEAD AREAS OF THE PILATUS 2M DETECTOR
UNTRUSTED_RECTANGLE= 487 495 0 1680
UNTRUSTED_RECTANGLE= 981 989 0 1680
!EXCLUSION OF HORIZONTAL DEAD AREAS OF THE PILATUS 2M DETECTOR
UNTRUSTED_RECTANGLE= 0 1476 195 213
UNTRUSTED_RECTANGLE= 0 1476 407 425
UNTRUSTED_RECTANGLE= 0 1476 619 637
UNTRUSTED_RECTANGLE= 0 1476 831 849
UNTRUSTED_RECTANGLE= 0 1476 1043 1061
UNTRUSTED_RECTANGLE= 0 1476 1255 1273
UNTRUSTED_RECTANGLE= 0 1476 1467 1485
eof
fi
elif [ $NX == "2463" ]; then
# Pilatus 6M
# FIXME: here we could test if a Flat_field correction was applied like we do for 2M
cat >> XDS.INP << eof
UNTRUSTED_RECTANGLE= 487 495 0 2528
UNTRUSTED_RECTANGLE= 981 989 0 2528
UNTRUSTED_RECTANGLE=1475 1483 0 2528
UNTRUSTED_RECTANGLE=1969 1977 0 2528
UNTRUSTED_RECTANGLE= 0 2464 195 213
UNTRUSTED_RECTANGLE= 0 2464 407 425
UNTRUSTED_RECTANGLE= 0 2464 619 637
UNTRUSTED_RECTANGLE= 0 2464 831 849
UNTRUSTED_RECTANGLE= 0 2464 1043 1061
UNTRUSTED_RECTANGLE= 0 2464 1255 1273
UNTRUSTED_RECTANGLE= 0 2464 1467 1485
UNTRUSTED_RECTANGLE= 0 2464 1679 1697
UNTRUSTED_RECTANGLE= 0 2464 1891 1909
UNTRUSTED_RECTANGLE= 0 2464 2103 2121
UNTRUSTED_RECTANGLE= 0 2464 2315 2333
eof
fi
fi
echo XDS.INP is ready for use. The file has only the most important keywords.
echo Full documentation, including complete detector templates, is at
echo http://www.mpimf-heidelberg.mpg.de/~kabsch/xds . More documentation in XDSwiki
echo After running xds, inspect, using XDS-Viewer, at least the beamstop mask in
echo BKGPIX.cbf, and the agreement of predicted and observed spots in FRAME.cbf!
rm -f tmp1 tmp2
# end of generate_XDS.INP
</pre>

== System-wide or personal installation ==

Ask your system adminstrator to cut-and-paste the script into e.g. /usr/local/bin/generate_XDS.INP, and to make it "executable".

But you may also cut-and-paste the script from this webpage into a file in e.g. your home directory; the filename should be generate_XDS.INP. After creating the file, make it executable - e.g. if it's in your $HOME, use:
chmod +x ~/generate_XDS.INP
After that, you can just run it in a similar way as if it were installed in your $PATH:
~/generate_XDS.INP "frms/mydata_1_???.img"
By using your own file, you can easily update to the latest revision, or even change the script, without having to bother the system administrator.

== Generating generate_XDS.INP from this webpage ==

Instead of cutting-and-pasting the lines of the script, you (or the system administrator) could just cut-and-paste the following four lines
<pre>
wget http://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/generate_XDS.INP -O - | \
sed -e s/\&nbsp\;/\ /g -e s/\&gt\;/\>/g -e s/\&lt\;/\</g -e s/amp\;//g -e s/\&quot\;/\"/g -e s/\&\#\1\6\0\;/\ /g | \
sed '/# end of generate_XDS.INP/,$d' | awk '/^#/,/rm -f tmp1 tmp2/' > generate_XDS.INP
chmod +x generate_XDS.INP
</pre>
to copy the script from this website into an executable file generate_XDS.INP in your current directory.

If you do use cut-and-paste from the webpage, be aware of the following problem report: On the Mac, after loading frames, by clicking “generate XDS.INP”, the program gives some strange symbol “Ã” in XDS.INP. And the more you click “save” button, the more “Ã” appear. This looks like e.g. SPACE_GROUP_NUMBER=0 Ã ! 0 if unknown UNIT_CELL_CONSTANTS= 70 80 90 90 90 90 Ã ''The problem is due to the “Rich text” format in TextEdit when saving "generate_XDS.INP". It is solved by re-downloading the script, and changing format to Plain - everything should work then.''

== Calling generate_XDS.INP from a Python script ==

It is recommended to use the [http://docs.python.org/2/library/subprocess.html subprocess.Popen()] module instead of [http://docs.python.org/2/library/os.html os.system()]:
<pre>
subprocess.Popen(["generate_XDS.INP",imagepath],stdout=outputfile)
</pre>
where imagepath is a string containing the path to an image and outputfile is either a chosen variable for an output file or subprocess.PIPE if you're not interested in the output of the script.

The module os.system() internally uses /bin/sh to execute the command and overrides #!/bin/bash at the beginning of the script. While this is not a problem on most operating systems, /bin/sh points to dash instead of bash on Ubuntu, which leads to a program crash with the error message
<pre>
sh: 1: Syntax error: Bad fd number
</pre>

Main Page

2014-03-10T09:50:47Z

Karine: XDSAPP link

== What is XDS, and how to obtain it? ==
XDS is a program package written for the "reduction" of 2-dimensional data images ("frames") obtained from crystals irradiated with monochromatic X-rays. The package mainly consists of the programs [[XDS]] (which processes a single dataset), [[XSCALE]] (for scaling of several datasets) , [[XDSCONV]] (for conversion of output file to other formats) and [[VIEW]] (for visualizing).

The author of the program package is Wolfgang Kabsch (Wolfgang dot Kabsch at mpimf-heidelberg dot mpg dot de).

The program is free for academic use, and can be downloaded from http://www.mpimf-heidelberg.mpg.de/~kabsch/xds/html_doc/downloading.html (if the download from that site fails, there's a copy at ftp://turn5.biologie.uni-konstanz.de/xds ).

Documentation is at http://www.mpimf-heidelberg.mpg.de/~kabsch/xds or [http://strucbio.biologie.uni-konstanz.de/~dikay/XDS_html_doc/index.html here].

Since version 30-Jan-2009, the format of the control images written by [[XDS]] (and [[XSCALE]]) [http://www.mpimf-heidelberg.mpg.de/~kabsch/xds/html_doc/Release_Notes.html changed]. An open-source viewer [[XDS-Viewer]] is [http://xds-viewer.sourceforge.net available]. Latest versions of [[Adxv]] can also visualize XDS output files.

There is another program available, which serves to produce additional output of tables and plots. This is not part of the official XDS package; its name is [[XDSSTAT]] (written and distributed by Kay Diederichs).

[[XDSGUI]], a graphical user interface, serves to help using XDS.

== Links to XDS papers by Wolfgang Kabsch ==

# [http://dx.doi.org/10.1107/S0021889887009737 ''J. Appl. Cryst.'' (1988). 21, 67-72. Automatic indexing of rotation diffraction patterns.] explaining the diffraction geometry as used in XDS, the local-indexing method used in IDXREF and the refinement of diffraction geometry parameters.
# [http://dx.doi.org/10.1107/S0021889888007903 ''J. Appl. Cryst.'' (1988). 21, 916-924. Evaluation of single-crystal X-ray diffraction data from a position-sensitive detector]. This is ''the'' reference for the integration and scaling algorithms employed in the INTEGRATE and CORRECT steps.
# [http://dx.doi.org/10.1107/S0021889893005588 ''J. Appl. Cryst.'' (1993). 26, 795-800. Automatic processing of rotation diffraction data from crystals of initially unknown symmetry and cell constants]. This builds upon publication 1. and gives further important details of the IDXREF step, in particular about space-group determination.
# [http://it.iucr.org/Fa/contents/ Chapters 11.3 and 25.2.9 in ''International Tables for Crystallography Volume F: Crystallography of Biological Macromolecules'']
# [http://dx.doi.org/10.1107/S0907444909047374 ''Acta Cryst.'' (2010). '''D66''', 133-144. Integration, scaling, space-group assignment and post-refinement.] the definitive reference. Open access.
# [http://dx.doi.org/10.1107/S0907444909047337 ''Acta Cryst.'' (2010). '''D66''', 125-132. XDS] the definitive reference. Open access.

== Purpose of the Wiki ==
The purpose of this Wiki is to provide an accurate resource for XDS users, and to provide explanations and hints for the successful use of XDS. I (Kay Diederichs) try to keep the Wiki up-to-date with XDS versions and insights. I collect [[Tips and Tricks]] for XDS usage, explanations of its output, and recommendations for the parameters in [[XDS.INP]] (if it appears useful to deviate from their defaults).

To see what's there already, go to [[Special:Allpages|List of all pages]]. A good starting point is the article about [[XDS.INP]] where the keywords and parameters for running XDS are described. There is also a [[wishlist]], and [[Known Bugs]], and [[Quality Control]] (under construction).

== Contributing to the Wiki ==
* The MediaWiki software used here works in the same way as the well-known Wikipedia. This means that anyone can read articles, but for editing existing pages, and creating new ones (this happens automatically when you click on a [[red link]]), you need a [[Creating an account|userid and password]]. Due to the amount of spam created by abuse of Google mail accounts, we are no longer allowing account creation for @gmail.com email addresses.
* Short introduction to [[creating an article]]

== Wiki contents ==

* [[Special:Allpages|List of all pages]] - to see which pages exist already ...

* [[Topics]] as an attempt to provide a hierarchy of pages

* new articles: [[adxv]], [[SIM_MX]], [[FAQ]], [[Problems]], [[CORRECT]]

== See also ==
[http://smb.slac.stanford.edu/facilities/software/xds/ A Quick XDS Tutorial for SSRL]

[http://strucbio.biologie.uni-konstanz.de/~dikay/XDS_Diederichs_ACA2011.pdf XDS talk given at ACA2011]

[https://www.researchgate.net/publication/236032099_Practical_use_of_XDS_%28for_data_reduction%29._Presented_in_April_4th_2013_Dept._of_Biochemical_Sciences_Sapienza_University_of_Rome?ev=prf_pub Practical use of XDS (for data reduction) - by Fulvio Saccoccia, Presented in April, 4th 2013; Dept. of Biochemical Sciences and Istituto Pasteur-Fondazione Cenci Bolognetti - Sapienza University of Rome. ]

[http://www.helmholtz-berlin.de/forschung/oe/funkma/soft-matter/forschung/bessy-mx/xdsapp/index_en.html XDSAPP, a graphical user interface for automatic data processing with XDS]