XDSwiki - User contributions [en]

Generate XDS.INP

2012-10-20T14:23:39Z

Olibclarke:

This script generates XDS.INP based on a list of frame names supplied on the commandline. It currently works for MarCCD, ADSC and Pilatus detectors; since this is just a bash script, extension to other detectors is very easy.

On Mac OS X, installation of the "Command Line Tools" (from http://developer.apple.com/downloads; requires Apple ID) is required (I think). These are a part of the (larger, but also free) [http://developer.apple.com/tools/xcode Xcode] package.

== Usage ==

Usage is just (don't forget the quotation marks!):
generate_XDS.INP "frms/mydata_1_???.img"
XDS [http://strucbio.biologie.uni-konstanz.de/~dikay/XDS_html_doc/html_doc/xds_parameters.html#NAME_TEMPLATE_OF_DATA_FRAMES= supports] bzip2-ed frames. Thus, when specifying the frame name parameter of the script, you should leave out any .bz2 extension.

== The script ==

<pre>
#!/bin/bash
# purpose: generate XDS.INP
#
# tested with some datasets from ALS, SSRL, SLS, ESRF and BESSY; only MARCCD, ADSC/SMV, PILATUS detectors;
# for other detectors, values marked with XXX must be manually filled in.
#
# revision 0.03 . Kay Diederichs 2/2010
# revision 0.04 . Kay Diederichs 4/2010 - include alternative ORGX, ORGY calculations for ADSC
# revision 0.05 . Kay Diederichs 5/2010 - grep for "Corrected" in addition to "marccd"; needed for BESSY
# revision 0.06 . KD 6/2010 - add UNTRUSTED_RECTANGLE and UNTRUSTED_ELLIPSE; use `whereis catmar` and so on
# revision 0.07 . KD 6/2010 - decide about ORGX/Y info in MAR header being pixels or mm; other fixes
# revision 0.08 . KD 6/2010 - fixes for Pilatus 6M
# revision 0.09 . KD 6/2010 - get rid of requirement for mccd_xdsparams.pl and/or catmar; rather use "od"
# revision 0.10 . Tim Gruene 7/2010 - set link 'images' to image directory if path exceeds 72 characters
# revision 0.11 . KD 7/2010 - for MarCCD: look for distance info at different byte position
# revision 0.12 . KD 7/2010 - fix for negative PHISTART
# revision 0.13 . KD 8/2010 - store correct NX NY QX QY in XDS.INP
# revision 0.14 . KD 1/2011 - SENSOR_THICKNESS for Pilatus; MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT=3
# revision 0.15 . KD 2/2011 - add comment for -ive sign of APS 19-ID and Australian Synchrotron rotation axis
# revision 0.16 . KD 3/2011 - SENSOR_THICKNESS=0.01 for ADSC and MarCCD. Add comment about SILICON=
# revision 0.17 . KD 3/2011 - make it work for .bz2 frames; improve screen output
# revision 0.18 . KD 4/2011 - faster by doing "strings" only once; revert "images/${1##/*/}" "correction"
# revision 0.19 . KD 6/2011 - bugfix for 0.18
# revision 0.20 . KD 7/2011 - redirect stderr of /bin/ls to /dev/null
# revision 0.21 . KD 11/2011 - SEPMIN, CLUSTER_RADIUS hints; read NX NY from header (for Pilatus 2M)
# revision 0.22 . KD 12/2011 - Pilatus 2M UNTRUSTED_RECTANGLE lines, SENSOR_THICKNESS from header
# revision 0.23 . KD 1/2012 - add UNTRUSTED_QUADRILATERAL, remove MINIMUM_ZETA (0.05 is default now)
# revision 0.24 . KD 3/2012 - remove revision 0.10 since XDS now takes much longer paths
# revision 0.25 . KD 3/2012 - remove revision 0.22 for PSI Pilatus 2M; see http://www.globalphasing.com/autoproc/wiki/index.cgi?TroubleShootingKnownIssues
# revision 0.26 . KD 7/2012 - Mac-compatibility: replace od flags --skip-bytes= and --read-bytes= with -j and -N (thanks to Oliver Clarke for working this out!)
REVISION="0.26 (02-Jul-2012)"
#
# usage: e.g. generate_XDS.INP "frms/mydata_1_???.img"
# make sure to have the two quotation marks !
# the ? are wildcards for the frame numbers.
#
# limitations:
# - frame numbers are assumed to start with 1 and run consecutively
#
# known problems:
# - for ADSC detectors, there are at least three ways to obtain ORGX and ORGY values from the header (see below);
# - the same might be a problem for MAR headers, too (not sure about this)
# - on Mac OS X, Xcode (from http://developer.apple.com/tools/xcode) might be needed - not sure about this
#
# notes for debugging of the script:
# - add the -v option to the first line, to see where an error occurs
# - comment out the removal of tmp1 and tmp2 in the last line
#
# ====== Start of script ======
echo generate_XDS.INP version $REVISION . Obtain the latest version from
echo http://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/generate_XDS.INP
if [ "$1" == "help" ] || [ "$1" == "-help" ] || [ "$1" == "-h" ]; then
echo usage: generate_XDS.INP \"frms/mydata_1_???.img\" $_with_ the quotation marks!$
echo if the frames are compressed with bzip2, leave out the .bz2 extension!
exit
fi
#
# defaults:
#
DETECTOR="XXX MINIMUM_VALID_PIXEL_VALUE=XXX OVERLOAD=XXX"
ORGX=XXX
ORGY=XXX
DETECTOR_DISTANCE=XXX
OSCILLATION_RANGE=XXX
X_RAY_WAVELENGTH=XXX
QX=XXX
QY=XXX
NX=XXX
NY=XXX
SENSOR_THICKNESS=0
# see how we are called:
NAME_TEMPLATE_OF_DATA_FRAMES="$1"
# list frames matching the wildcards in NAME_TEMPLATE_OF_DATA_FRAMES
# don't accept the "direct beam" shot at SLS/Pilatus PX-I and PX-II
/bin/ls -C1 $1 $1.bz2 2>/dev/null | egrep -v "_00000.cbf|_000.img" > tmp1 || exit 1

# we can continue - the frames are found

# set upper limit of DATA_RANGE to number of frames (see "limitations" above)
DATA_RANGE=`wc -l tmp1 | awk '{print $1}'`

# set upper limit of SPOT_RANGE to half of DATA_RANGE, but not less than 1
SPOT_RANGE=`echo "scale=0; $DATA_RANGE/2" | bc -l`
SPOT_RANGE=`echo "if ($SPOT_RANGE<1) 1;if ($SPOT_RANGE>1) $SPOT_RANGE" | bc -l`

echo DATA_RANGE=1 $DATA_RANGE

# find out detector type
DET=XXX
FIRSTFRAME=`head -1 tmp1`
echo $FIRSTFRAME | grep -q bz2 && bzcat $FIRSTFRAME > tmp1 && FIRSTFRAME=tmp1
strings $FIRSTFRAME > tmp2
egrep -q 'marccd|Corrected' tmp2 && DET=mccd
grep -q PILATUS tmp2 && DET=pilatus
grep -q BEAM_CENTER_X tmp2 && DET=adsc
# identify other detector types in the same way (MAR IP would be straightforward)

# parse ASCII header of first frame

if [ "$DET" == "XXX" ]; then
echo "this is not a MAR, ADSC/SMV or PILATUS detector - fill in XXX values manually!"
DETECTOR="XXX MINIMUM_VALID_PIXEL_VALUE=XXX OVERLOAD=XXX"

# find parameters of first frame
elif [ "$DET" == "mccd" ]; then
echo Data from a MarCCD detector

DETECTOR="CCDCHESS MINIMUM_VALID_PIXEL_VALUE= 1 OVERLOAD= 65500"
SENSOR_THICKNESS=0.01
# use first frame of dataset to obtain parameters

# offsets are documented; values can be found in mccd_xdsparams.pl script
let SKIP=1024+80
NX=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
let SKIP=$SKIP+4
NY=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')

let SKIP=1720
DETECTOR_DISTANCE=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
DETECTOR_DISTANCE=`echo "scale=3; $DETECTOR_DISTANCE/1000" | bc -l`

let SKIP=1024+256+128+256+4
ORGX=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
ORGX=`echo "scale=2; $ORGX/1000" | bc -l `
let SKIP=$SKIP+4
ORGY=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
ORGY=`echo "scale=2; $ORGY/1000" | bc -l `

let SKIP=1024+256+128+256+44
PHISTART=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
let SKIP=1024+256+128+256+76
PHIEND=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
OSCILLATION_RANGE=`echo "scale=3; ($PHIEND-($PHISTART))/1000" | bc -l`

let SKIP=1024+256+128+256+128+4
QX=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
QX=`echo "scale=10; $QX/1000000" |bc -l `
let SKIP=$SKIP+4
QY=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
QY=`echo "scale=10; $QY/1000000" |bc -l `

let SKIP=1024+256+128+256+128+128+12
X_RAY_WAVELENGTH=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
X_RAY_WAVELENGTH=`echo "scale=5; $X_RAY_WAVELENGTH/100000" | bc -l`

# at most BLs, ORGX and ORGY are in pixels, but sometimes in mm ... guess:
NXBYFOUR=`echo "scale=0; $NX/4" | bc -l `
ORGXINT=`echo "scale=0; $ORGX/1" | bc -l `
if [ $ORGXINT -lt $NXBYFOUR ]; then
ORGX=`echo "scale=1; $ORGX/$QX" | bc -l`
ORGY=`echo "scale=1; $ORGY/$QY" | bc -l`
echo MARCCD detector: header ORGX, ORGY seem to be in mm ... converting to pixels
else
echo MARCCD detector: header ORGX, ORGY seem to be in pixel units
fi

elif [ "$DET" == "adsc" ]; then

DETECTOR="ADSC MINIMUM_VALID_PIXEL_VALUE= 1 OVERLOAD= 65000"
echo Data from ADSC detector. Obtaining ORGX, ORGY depends on beamline setup:
SENSOR_THICKNESS=0.01
sed s/\;// tmp2 > tmp1
mv tmp1 tmp2

# find X_RAY_WAVELENGTH:
X_RAY_WAVELENGTH=`grep WAVELENGTH tmp2 | head -1 | sed s/WAVELENGTH=//`

# find NX, QX, ORGX and ORGY:
NX=`grep SIZE1 tmp2 | tail -1 | sed s/SIZE1=//`
QX=`grep PIXEL_SIZE tmp2 | sed s/PIXEL_SIZE=//`
# FIXME - next 2 lines should be done properly, from header
NY=$NX
QY=$QX
BEAM_CENTER_X=`grep BEAM_CENTER_X tmp2 | sed s/BEAM_CENTER_X=//`
BEAM_CENTER_Y=`grep BEAM_CENTER_Y tmp2 | sed s/BEAM_CENTER_Y=//`
# fix 2010-04-26 - tell user about possible ORGX, ORGY alternatives -
# at ESRF and ... (pls fill in!) the following should be used:
ORGX=`echo "scale=1; $BEAM_CENTER_Y/$QX" | bc -l`
ORGY=`echo "scale=1; $BEAM_CENTER_X/$QX" | bc -l`
echo - at ESRF BLs use: ORGX=$ORGX ORGY=$ORGY
# this 2nd alternative convention should be used at the following beamlines (pls complete the list): ALS 5.0.3, ...
ORGX=`echo "scale=1; $NX-$BEAM_CENTER_X/$QX" | bc -l `
ORGY=`echo "scale=1; $BEAM_CENTER_Y/$QX" | bc -l `
echo - at e.g. ALS 5.0.3 use: ORGX=$ORGX ORGY=$ORGY
# this 3rd alternative convention should be used at the following beamlines (pls complete the list): ALS 8.2.2, ...
ORGX=`echo "scale=1; $BEAM_CENTER_X/$QX" | bc -l `
ORGY=`echo "scale=1; $NX-$BEAM_CENTER_Y/$QX" | bc -l `
echo - at e.g. ALS 8.2.2 use: ORGX=$ORGX ORGY=$ORGY - this is written to XDS.INP
# the latter alternative is written into the generated XDS.INP ! You have to correct this manually in XDS.INP, or adjust this script.
# find DETECTOR_DISTANCE and OSCILLATION_RANGE:
DETECTOR_DISTANCE=`grep DISTANCE tmp2 | sed s/DISTANCE=//`
OSCILLATION_RANGE=`grep OSC_RANGE tmp2 | sed s/OSC_RANGE=//`

elif [ "$DET" == "pilatus" ]; then
DETECTOR="PILATUS MINIMUM_VALID_PIXEL_VALUE=0 OVERLOAD= 1048576 !PILATUS"
QX=0.172 QY=0.172
echo Data from a Pilatus detector
sed s/#// tmp2 > tmp1
mv tmp1 tmp2

# find SENSOR_THICKNESS:
SENSOR_THICKNESS=`grep thickness tmp2 | sed -e s/'Silicon sensor, thickness'// | awk '{print $1*1000}'`
# find X_RAY_WAVELENGTH:
X_RAY_WAVELENGTH=`grep Wavelength tmp2 | sed -e s/Wavelength// -e s/A// | awk '{print $1}'`

# find NX and NY; 2463/2527 is 6M, 1475/1679 is 2M
NX=`grep X-Binary-Size-Fastest-Dimension tmp2 | awk '{print $2}'`
NY=`grep X-Binary-Size-Second-Dimension tmp2 | awk '{print $2}'`

# find ORGX and ORGY:
ORGX=`grep Beam_xy tmp2 | sed -e s/$// -e s/$// -e s/\,// | awk '{print $2}'`
ORGY=`grep Beam_xy tmp2 | sed -e s/$// -e s/$// -e s/\,// | awk '{print $3}'`

# find DETECTOR_DISTANCE and OSCILLATION_RANGE:
DETECTOR_DISTANCE=`awk '/distance/{print $2}' tmp2`
DETECTOR_DISTANCE=`echo "$DETECTOR_DISTANCE*1000" | bc -l`

OSCILLATION_RANGE=`awk '/Angle/{print $2}' tmp2`

else
echo should never come here
exit 1
fi

echo ORGX= $ORGX ORGY= $ORGY - check these values with adxv !
echo DETECTOR_DISTANCE= $DETECTOR_DISTANCE
echo OSCILLATION_RANGE= $OSCILLATION_RANGE
echo X-RAY_WAVELENGTH= $X_RAY_WAVELENGTH

# now we know everything that is required to generate XDS.INP

cat > XDS.INP << eof
! written by generate_XDS.INP version $REVISION
JOB= XYCORR INIT COLSPOT IDXREF DEFPIX INTEGRATE CORRECT
ORGX= $ORGX ORGY= $ORGY ! check these values with adxv !
DETECTOR_DISTANCE= $DETECTOR_DISTANCE
OSCILLATION_RANGE= $OSCILLATION_RANGE
X-RAY_WAVELENGTH= $X_RAY_WAVELENGTH
NAME_TEMPLATE_OF_DATA_FRAMES=$NAME_TEMPLATE_OF_DATA_FRAMES
! REFERENCE_DATA_SET=xxx/XDS_ASCII.HKL ! e.g. to ensure consistent indexing
DATA_RANGE=1 $DATA_RANGE
SPOT_RANGE=1 $SPOT_RANGE
! BACKGROUND_RANGE=1 10 ! rather use defaults (first 5 degree of rotation)

SPACE_GROUP_NUMBER=0 ! 0 if unknown
UNIT_CELL_CONSTANTS= 70 80 90 90 90 90 ! put correct values if known
INCLUDE_RESOLUTION_RANGE=50 0 ! after CORRECT, insert high resol limit; re-run CORRECT

FRIEDEL'S_LAW=FALSE ! This acts only on the CORRECT step
! If the anom signal turns out to be, or is known to be, very low or absent,
! use FRIEDEL'S_LAW=TRUE instead (or comment out the line); re-run CORRECT

! remove the "!" in the following line:
! STRICT_ABSORPTION_CORRECTION=TRUE
! if the anomalous signal is strong: in that case, in CORRECT.LP the three
! "CHI^2-VALUE OF FIT OF CORRECTION FACTORS" values are significantly> 1, e.g. 1.5
!
! exclude (mask) untrusted areas of detector, e.g. beamstop shadow :
! UNTRUSTED_RECTANGLE= 1800 1950 2100 2150 ! x-min x-max y-min y-max ! repeat
! UNTRUSTED_ELLIPSE= 2034 2070 1850 2240 ! x-min x-max y-min y-max ! if needed
! UNTRUSTED_QUADRILATERAL= x1 y1 x2 y2 x3 y3 x4 y4 ! see documentation
!
! parameters with changes wrt default values:
TRUSTED_REGION=0.00 1.2 ! partially use corners of detectors; 1.41421=full use
VALUE_RANGE_FOR_TRUSTED_DETECTOR_PIXELS=7000. 30000. ! often 8000 is ok
STRONG_PIXEL=4 ! COLSPOT: only use strong reflections (default is 3)
MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT=3 ! default of 6 is sometimes too high
! close spots: reduce SEPMIN and CLUSTER_RADIUS from their defaults of 6 and 3, e.g. to 4 and 2
! for bad or low resolution data remove the "!" in the following line:
! REFINE(IDXREF)=CELL BEAM ORIENTATION AXIS ! DISTANCE
REFINE(INTEGRATE)=CELL BEAM ORIENTATION ! AXIS DISTANCE
! REFINE(CORRECT)=CELL BEAM ORIENTATION AXIS DISTANCE ! Default is: refine everything

! parameters specifically for this detector and beamline:
DETECTOR= $DETECTOR
SENSOR_THICKNESS= $SENSOR_THICKNESS
! attention CCD detectors: for very high resolution (better than 1A) make sure to specify SILICON
! as about 32* what CORRECT.LP suggests (absorption of phosphor is much higher than that of silicon)
NX= $NX NY= $NY QX= $QX QY= $QY ! to make CORRECT happy if frames are unavailable
DIRECTION_OF_DETECTOR_X-AXIS=1 0 0
DIRECTION_OF_DETECTOR_Y-AXIS=0 1 0
INCIDENT_BEAM_DIRECTION=0 0 1
ROTATION_AXIS=1 0 0 ! at e.g. Australian Synchrotron, SERCAT ID-22 (?), APS 19-ID (?) this needs to be -1 0 0
FRACTION_OF_POLARIZATION=0.98 ! better value is provided by beamline staff!
POLARIZATION_PLANE_NORMAL=0 1 0
eof
if [ "$DET" == "pilatus" ]; then
if [ $NX == "1475" ]; then
if ! grep -q FF_p2m0109_E12398_T6199_vrf_m0p20.tif tmp2 ; then
cat >> XDS.INP << eof
!EXCLUSION OF VERTICAL DEAD AREAS OF THE PILATUS 2M DETECTOR
UNTRUSTED_RECTANGLE= 486 496 0 1680
UNTRUSTED_RECTANGLE= 980 990 0 1680
!EXCLUSION OF HORIZONTAL DEAD AREAS OF THE PILATUS 2M DETECTOR
UNTRUSTED_RECTANGLE= 0 1476 194 214
UNTRUSTED_RECTANGLE= 0 1476 406 426
UNTRUSTED_RECTANGLE= 0 1476 618 638
UNTRUSTED_RECTANGLE= 0 1476 830 850
UNTRUSTED_RECTANGLE= 0 1476 1042 1062
UNTRUSTED_RECTANGLE= 0 1476 1254 1274
UNTRUSTED_RECTANGLE= 0 1476 1466 1486
eof
fi
# elif [ $NX == "2463" ]; then
# cat >> XDS.INP << eof
#eof
fi
fi
echo XDS.INP is ready for use. The file has only the most important keywords.
echo Full documentation, including complete detector templates, is at
echo http://www.mpimf-heidelberg.mpg.de/~kabsch/xds . More documentation in XDSwiki
echo After running xds, inspect, using XDS-Viewer, at least the beamstop mask in
echo BKGPIX.cbf, and the agreement of predicted and observed spots in FRAME.cbf!
rm -f tmp1 tmp2
</pre>

== System-wide or personal installation ==

Ask your system adminstrator to cut-and-paste the script into e.g. /usr/local/bin/generate_XDS.INP, and to make it "executable".

But you may also cut-and-paste the script from this webpage into a file in e.g. your home directory; the filename should be generate_XDS.INP. After creating the file, make it executable - e.g. if it's in your $HOME, use:
chmod +x ~/generate_XDS.INP
After that, you can just run it in a similar way as if it were installed in your $PATH:
~/generate_XDS.INP "frms/mydata_1_???.img"
By using your own file, you can easily update to the latest revision, or even change the script, without having to bother the system administrator.

== Generating generate_XDS.INP from this webpage ==

Instead of cutting-and-pasting the lines of the script, you (or the system administrator) could just cut-and-paste the following four lines
<pre>
wget http://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/generate_XDS.INP -O - | \
sed -e s/\&nbsp\;/\ /g -e s/\&gt\;/\>/g -e s/\&lt\;/\</g -e s/amp\;//g -e s/\&quot\;/\"/g -e s/\&\#\1\6\0\;/\ /g | \
awk '/^#/,/rm -f tmp1 tmp2/' > generate_XDS.INP
chmod +x generate_XDS.INP
</pre>
to copy the script from this website into an executable file generate_XDS.INP in your current directory.

==Variant script (xds_generate_all)==
This script includes some minor modifications of the Generate_XDS.INP script - for instance, it tells the user the energy, as well as the wavelength of the X-rays, and gives the user the beam center in mm as well as pixels - values that may be useful when switching between HKL2000, MOSFLM and XDS.

It also generates two executable shell scripts in the same directory as XDS.INP.

The first of these scripts, xds_graph.sh, extracts various statistics from INTEGRATE.LP and CORRECT.LP, pipes them out to a logfile, INTEGRATE_STATS.LP, and plots them using loggraph. This script does not have some of the more useful features of XDSSTAT, such as calculation of per frame Rmeas values, but it does give a quick overview of various data quality parameters by resolution and image number. This script will only work with recent versions of XDS (after calculation of CC(1/2) - not sure exactly which version).

The second of these scripts, xds_to_ccp4.sh, will take XDS_ASCII.HKL and generate a CCP4-format MTZ file, containing anomalous data (as F(+)/F(-) and DANO/SigDANO) and a test set of 5% of reflections for calculation of the free R-factor. It will also generate an unmerged SHELX format HKL file, for input into SHELXC/D/E (e.g. via the HKL2MAP GUI).

Both these shell scripts can by installed system wide by transferring them to a directory in your path, for example the directory containing your XDS binaries.

Here is the script:
<pre>
#!/bin/bash
# purpose: xds_generate_all
#
# tested with some datasets from ALS, SSRL, SLS, ESRF and BESSY; only MARCCD, ADSC/SMV, PILATUS detectors;
# for other detectors, values marked with XXX must be manually filled in.
#
# revision 0.03 . Kay Diederichs 2/2010
# revision 0.04 . Kay Diederichs 4/2010 - include alternative ORGX, ORGY calculations for ADSC
# revision 0.05 . Kay Diederichs 5/2010 - grep for "Corrected" in addition to "marccd"; needed for BESSY
# revision 0.06 . KD 6/2010 - add UNTRUSTED_RECTANGLE and UNTRUSTED_ELLIPSE; use `whereis catmar` and so on
# revision 0.07 . KD 6/2010 - decide about ORGX/Y info in MAR header being pixels or mm; other fixes
# revision 0.08 . KD 6/2010 - fixes for Pilatus 6M
# revision 0.09 . KD 6/2010 - get rid of requirement for mccd_xdsparams.pl and/or catmar; rather use "od"
# revision 0.10 . Tim Gruene 7/2010 - set link 'images' to image directory if path exceeds 72 characters
# revision 0.11 . KD 7/2010 - for MarCCD: look for distance info at different byte position
# revision 0.12 . KD 7/2010 - fix for negative PHISTART
# revision 0.13 . KD 8/2010 - store correct NX NY QX QY in XDS.INP
# revision 0.14 . KD 1/2011 - SENSOR_THICKNESS for Pilatus; MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT=3
# revision 0.15 . KD 2/2011 - add comment for -ive sign of APS 19-ID and Australian Synchrotron rotation axis
# revision 0.16 . KD 3/2011 - SENSOR_THICKNESS=0.01 for ADSC and MarCCD. Add comment about SILICON=
# revision 0.17 . KD 3/2011 - make it work for .bz2 frames; improve screen output
# revision 0.18 . KD 4/2011 - faster by doing "strings" only once; revert "images/${1##/*/}" "correction"
# revision 0.19 . KD 6/2011 - bugfix for 0.18
# revision 0.20 . KD 7/2011 - redirect stderr of /bin/ls to /dev/null
# revision 0.21 . KD 11/2011 - SEPMIN, CLUSTER_RADIUS hints; read NX NY from header (for Pilatus 2M)
# revision 0.22 . KD 12/2011 - Pilatus 2M UNTRUSTED_RECTANGLE lines, SENSOR_THICKNESS from header
# revision 0.23 . KD 1/2012 - add UNTRUSTED_QUADRILATERAL, remove MINIMUM_ZETA (0.05 is default now)
# revision 0.24 . KD 3/2012 - remove revision 0.10 since XDS now takes much longer paths
# revision 0.25 . KD 3/2012 - remove revision 0.22 for PSI Pilatus 2M; see http://www.globalphasing.com/autoproc/wiki/index.cgi?TroubleShootingKnownIssues
# revision 0.26 Oliver Clarke 10/12 - Added generation of shell scripts for conversion to MTZ and visualisation of statistics. Other minor alterations to XDS.INP.
# revision 0.27 Oliver Clarke 10/12 - xds_graph.sh will now run xdsstat, if available, and plot Rmeas and Rd per image.
# revision 0.28 Oliver Clarke 10/12 - xds_graph was not plotting h00, 0k0, 00l reflections when a reference dataset was present. Fixed.
REVISION="0.28 (20-Oct-2012)"
#
# usage: e.g. xds_generate_all "frms/mydata_1_???.img"
# make sure to have the two quotation marks !
# the ? are wildcards for the frame numbers.
#
# limitations:
# - frame numbers are assumed to start with 1 and run consecutively
#
# known problems:
# - for ADSC detectors, there are at least three ways to obtain ORGX and ORGY values from the header (see below);
# - the same might be a problem for MAR headers, too (not sure about this)
#
# notes for debugging of the script:
# - add the -v option to the first line, to see where an error occurs
# - comment out the removal of tmp1 and tmp2 in the last line
#
# ====== Start of script ======
echo generate_XDS.INP version $REVISION . Obtain the latest version from
echo http://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/generate_XDS.INP
if [ "$1" == "help" ] || [ "$1" == "-help" ] || [ "$1" == "-h" ]; then
echo usage: xds_generate_all \"frms/mydata_1_???.img\" $_with_ the quotation marks!$
echo if the frames are compressed with bzip2, leave out the .bz2 extension!
exit
fi
#
# defaults:
#
DETECTOR="XXX MINIMUM_VALID_PIXEL_VALUE=XXX OVERLOAD=XXX"
ORGX=XXX
ORGY=XXX
ORGXMM=XXX
ORGYMM=XXX
DETECTOR_DISTANCE=XXX
OSCILLATION_RANGE=XXX
X_RAY_WAVELENGTH=XXX
X_RAY_ENERGY=XXX
STARTING_ANGLE=XXX
QX=XXX
QY=XXX
NX=XXX
NY=XXX
NXMM=XXX
NYMM=XXX
SENSOR_THICKNESS=0
# see how we are called:
NAME_TEMPLATE_OF_DATA_FRAMES="$1"
# list frames matching the wildcards in NAME_TEMPLATE_OF_DATA_FRAMES
# don't accept the "direct beam" shot at SLS/Pilatus PX-I and PX-II
/bin/ls -C1 $1 $1.bz2 2>/dev/null | egrep -v "_00000.cbf|_000.img" > tmp1 || exit 1

# we can continue - the frames are found

# set upper limit of DATA_RANGE to number of frames (see "limitations" above)
DATA_RANGE=`wc -l tmp1 | awk '{print $1}'`

# set upper limit of SPOT_RANGE to half of DATA_RANGE, but not less than 1
SPOT_RANGE=`echo "scale=0; $DATA_RANGE/2" | bc -l`
SPOT_RANGE=`echo "if ($SPOT_RANGE<1) 1;if ($SPOT_RANGE>1) $SPOT_RANGE" | bc -l`

echo DATA_RANGE=1 $DATA_RANGE

# find out detector type
DET=XXX
FIRSTFRAME=`head -1 tmp1`
echo $FIRSTFRAME | grep -q bz2 && bzcat $FIRSTFRAME > tmp1 && FIRSTFRAME=tmp1
strings $FIRSTFRAME > tmp2
egrep -q 'marccd|Corrected' tmp2 && DET=mccd
grep -q PILATUS tmp2 && DET=pilatus
grep -q BEAM_CENTER_X tmp2 && DET=adsc
# identify other detector types in the same way (MAR IP would be straightforward)

# parse ASCII header of first frame

if [ "$DET" == "XXX" ]; then
echo "this is not a MAR, ADSC/SMV or PILATUS detector - fill in XXX values manually!"
DETECTOR="XXX MINIMUM_VALID_PIXEL_VALUE=XXX OVERLOAD=XXX"

# find parameters of first frame
elif [ "$DET" == "mccd" ]; then
echo Data from a MarCCD detector

DETECTOR="CCDCHESS MINIMUM_VALID_PIXEL_VALUE= 1 OVERLOAD= 65500"
SENSOR_THICKNESS=0.01
# use first frame of dataset to obtain parameters

# offsets are documented; values can be find in mccd_xdsparams.pl script
let SKIP=1024+80
NX=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
let SKIP=$SKIP+4
NY=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')

let SKIP=1720
DETECTOR_DISTANCE=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
DETECTOR_DISTANCE=`echo "scale=3; $DETECTOR_DISTANCE/1000" | bc -l`

let SKIP=1024+256+128+256+4
ORGX=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
ORGX=`echo "scale=2; $ORGX/1000" | bc -l `
let SKIP=$SKIP+4
ORGY=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
ORGY=`echo "scale=2; $ORGY/1000" | bc -l `

let SKIP=1024+256+128+256+44
PHISTART=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
let SKIP=1024+256+128+256+76
PHIEND=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
OSCILLATION_RANGE=`echo "scale=3; ($PHIEND-($PHISTART))/1000" | bc -l`
STARTING_ANGLE=`echo "scale=3; ($PHISTART/1000)" | bc -l`

let SKIP=1024+256+128+256+128+4
QX=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
QX=`echo "scale=10; $QX/1000000" |bc -l `
let SKIP=$SKIP+4
QY=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
QY=`echo "scale=10; $QY/1000000" |bc -l `

let SKIP=1024+256+128+256+128+128+12
X_RAY_WAVELENGTH=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
X_RAY_WAVELENGTH=`echo "scale=5; $X_RAY_WAVELENGTH/100000" | bc -l`
X_RAY_ENERGY=`echo "12398.5/$X_RAY_WAVELENGTH" | bc -l | awk '{printf "%5.1f", $1}'`
# at most BLs, ORGX and ORGY are in pixels, but sometimes in mm ... guess:
NXBYFOUR=`echo "scale=0; $NX/4" | bc -l `
ORGXINT=`echo "scale=0; $ORGX/1" | bc -l `
if [ $ORGXINT -lt $NXBYFOUR ]; then
ORGXMM=$ORGX
ORGYMM=$ORGY
NXMM=`echo "$NX*$QX" | bc -l | awk '{printf "%3.2f", $1}'`
NYMM=`echo "$NY*$QY" | bc -l | awk '{printf "%3.2f", $1}'`
ORGX=`echo "scale=1; $ORGX/$QX" | bc -l`
ORGY=`echo "scale=1; $ORGY/$QY" | bc -l`
echo MARCCD detector: header ORGX, ORGY seem to be in mm ... converting to pixels
else
ORGXMM=`echo "$ORGX*$QX" | bc -l | awk '{printf "%3.2f", $1}'`
ORGYMM=`echo "$ORGY*$QY" | bc -l | awk '{printf "%3.2f", $1}'`
NXMM=`echo "$NX*$QX" | bc -l | awk '{printf "%3.2f", $1}'`
NYMM=`echo "$NY*$QY" | bc -l | awk '{printf "%3.2f", $1}'`
echo MARCCD detector: header ORGX, ORGY seem to be in pixel units
fi

elif [ "$DET" == "adsc" ]; then

DETECTOR="ADSC MINIMUM_VALID_PIXEL_VALUE= 1 OVERLOAD= 65000"
echo Data from ADSC detector. Obtaining ORGX, ORGY depends on beamline setup:
SENSOR_THICKNESS=0.01
sed s/\;// tmp2 > tmp1
mv tmp1 tmp2

# find X_RAY_WAVELENGTH:
X_RAY_WAVELENGTH=`grep WAVELENGTH tmp2 | head -1 | sed s/WAVELENGTH=//`

# find NX, QX, ORGX and ORGY:
X_RAY_ENERGY=`echo "12398.5/$X_RAY_WAVELENGTH" | bc -l | awk '{printf "%5.1f", $1}'`
STARTING_ANGLE=`grep PHI tmp2 | sed s/PHI=//`
NX=`grep SIZE1 tmp2 | tail -1 | sed s/SIZE1=//`
QX=`grep PIXEL_SIZE tmp2 | sed s/PIXEL_SIZE=//`
# FIXME - next 2 lines should be done properly, from header
NY=$NX
QY=$QX
NXMM=`echo "$NX*$QX" | bc -l | awk '{printf "%3.2f", $1}'`
NYMM=`echo "$NY*$QY" | bc -l | awk '{printf "%3.2f", $1}'`
BEAM_CENTER_X=`grep BEAM_CENTER_X tmp2 | sed s/BEAM_CENTER_X=//`
BEAM_CENTER_Y=`grep BEAM_CENTER_Y tmp2 | sed s/BEAM_CENTER_Y=//`
# fix 2010-04-26 - tell user about possible ORGX, ORGY alternatives -
# at ESRF and ... (pls fill in!) the following should be used:
ORGX=`echo "scale=1; $BEAM_CENTER_Y/$QX" | bc -l`
ORGY=`echo "scale=1; $BEAM_CENTER_X/$QX" | bc -l`
echo - at ESRF BLs use: ORGX=$ORGX ORGY=$ORGY
# this 2nd alternative convention should be used at the following beamlines (pls complete the list): ALS 5.0.3, ...
ORGX=`echo "scale=1; $NX-$BEAM_CENTER_X/$QX" | bc -l `
ORGY=`echo "scale=1; $BEAM_CENTER_Y/$QX" | bc -l `
echo - at e.g. ALS 5.0.3 use: ORGX=$ORGX ORGY=$ORGY
# this 3rd alternative convention should be used at the following beamlines (pls complete the list): ALS 8.2.2, ...
ORGX=`echo "scale=1; $BEAM_CENTER_X/$QX" | bc -l `
ORGY=`echo "scale=1; $NX-$BEAM_CENTER_Y/$QX" | bc -l `
ORGXMM=`echo "$ORGX*$QX" | bc -l | awk '{printf "%3.2f", $1}'`
ORGYMM=`echo "$ORGY*$QY" | bc -l | awk '{printf "%3.2f", $1}'`
echo - at e.g. ALS 8.2.2 use: ORGX=$ORGX ORGY=$ORGY - this is written to XDS.INP
# the latter alternative is written into the generated XDS.INP ! You have to correct this manually in XDS.INP, or adjust this script.
# find DETECTOR_DISTANCE and OSCILLATION_RANGE:
DETECTOR_DISTANCE=`grep DISTANCE tmp2 | sed s/DISTANCE=//`
OSCILLATION_RANGE=`grep OSC_RANGE tmp2 | sed s/OSC_RANGE=//`

elif [ "$DET" == "pilatus" ]; then
DETECTOR="PILATUS MINIMUM_VALID_PIXEL_VALUE=0 OVERLOAD= 1048576 !PILATUS"
QX=0.172 QY=0.172
echo Data from a Pilatus detector
sed s/#// tmp2 > tmp1
mv tmp1 tmp2

# find SENSOR_THICKNESS:
SENSOR_THICKNESS=`grep thickness tmp2 | sed -e s/'Silicon sensor, thickness'// | awk '{print $1*1000}'`
# find X_RAY_WAVELENGTH:
X_RAY_WAVELENGTH=`grep Wavelength tmp2 | sed -e s/Wavelength// -e s/A// | awk '{print $1}'`

X_RAY_ENERGY=`echo "12398.5/$X_RAY_WAVELENGTH" | bc -l | awk '{printf "%5.1f", $1}'`
# find NX and NY; 2463/2527 is 6M, 1475/1679 is 2M
NX=`grep X-Binary-Size-Fastest-Dimension tmp2 | awk '{print $2}'`
NY=`grep X-Binary-Size-Second-Dimension tmp2 | awk '{print $2}'`
NXMM=`echo "$NX*$QX" | bc -l | awk '{printf "%3.2f", $1}'`
NYMM=`echo "$NY*$QY" | bc -l | awk '{printf "%3.2f", $1}'`
# find ORGX and ORGY:
ORGX=`grep Beam_xy tmp2 | sed -e s/$// -e s/$// -e s/\,// | awk '{print $2}'`
ORGY=`grep Beam_xy tmp2 | sed -e s/$// -e s/$// -e s/\,// | awk '{print $3}'`
ORGXMM=`echo "$ORGX*$QX" | bc -l | awk '{printf "%3.2f", $1}'`
ORGYMM=`echo "$ORGY*$QY" | bc -l | awk '{printf "%3.2f", $1}'`
# find DETECTOR_DISTANCE and OSCILLATION_RANGE:
DETECTOR_DISTANCE=`awk '/distance/{print $2}' tmp2`
DETECTOR_DISTANCE=`echo "$DETECTOR_DISTANCE*1000" | bc -l`

OSCILLATION_RANGE=`awk '/Angle/{print $2}' tmp2`

else
echo should never come here
exit 1
fi

echo ORGX= $ORGX ORGY= $ORGY - check these values with adxv !
echo DETECTOR_DISTANCE= $DETECTOR_DISTANCE
echo OSCILLATION_RANGE= $OSCILLATION_RANGE
echo X-RAY_WAVELENGTH= $X_RAY_WAVELENGTH
# now we know everything that is required to generate XDS.INP

cat > XDS.INP << eof
!****************************************************************************************
! Written by generate_XDS.INP version $REVISION
!
! A few notes on usage and modification of this input file:
!
! If XDS fails after IDXREF with an error indicating that
! an insufficient percentage of reflections were indexed,
! but you are confident that the indexing solution is
! correct, rerun XDS with the JOB keyword changed to
! "DEFPIX INTEGRATE CORRECT".
!
! After the first round of integration (in a triclinic cell)
! it is often helpful to reintegrate using the globally
! refined cell parameters and the correct space group.
! You can do this by renaming GXPARM.XDS to XPARM.XDS
! - e.g. by the command "cp GXPARM.XDS XPARM.XDS" -
! and rerunning DEFPIX, INTEGRATE and CORRECT.
!
! If you are trying to determine the optimal starting point for
! data collection starting from a single image, first complete
! a run of XDS using default parameters. Then, rerun XDS using
! the correct cell and spacegroup as described above
! ("cp GXPARM.XDS XPARM.XDS"), but on the second run of XDS, give
! the JOB keyword as "JOB=DEFPIX XPLAN".
!
! Be warned that the starting spindle angles given by XPLAN are
! relative to the first image given - XDS assumes that the first image
! given has a phi of 0 unless explicitly told otherwise. If this is not
! the case (e.g. you are indexing from a snapshot taken at phi=90,
! you will need to add the actual phi (as reported in the image header)
! to the values given by XPLAN in order for these values to make sense.
!****************************************************************************************
!
!****************************************************************************************
!General parameters:
!****************************************************************************************

JOB= XYCORR INIT COLSPOT IDXREF DEFPIX INTEGRATE CORRECT

ORGX= $ORGX ORGY= $ORGY ! check these values with adxv!
!In mm, ORGX=$ORGXMM and ORGY=$ORGYMM

DETECTOR_DISTANCE= $DETECTOR_DISTANCE
OSCILLATION_RANGE= $OSCILLATION_RANGE

X-RAY_WAVELENGTH= $X_RAY_WAVELENGTH
!X-ray energy is $X_RAY_ENERGY eV (calculated from wavelength)

NAME_TEMPLATE_OF_DATA_FRAMES=$NAME_TEMPLATE_OF_DATA_FRAMES

!REFERENCE_DATA_SET=xxx/XDS_ASCII.HKL ! e.g. to ensure consistent indexing.
!XDS will *only* use the reference dataset if the space group and cell
!parameters are specified below.

DATA_RANGE=1 $DATA_RANGE
SPOT_RANGE=1 $SPOT_RANGE

!BACKGROUND_RANGE=1 10 !Default is first five degrees.

SPACE_GROUP_NUMBER=0 ! 0 if unknown
UNIT_CELL_CONSTANTS= 70 80 90 90 90 90 ! put correct values if known

INCLUDE_RESOLUTION_RANGE=50 0 ! After CORRECT, insert high resol limit; re-run CORRECT
! Ice rings can be excluded from scaling with the keyword
! EXCLUDE_RESOLUTION_RANGE, e.g. EXCLUDE_RESOLUTION_RANGE=2.28 2.22

!SECONDS=60 !Uncomment this line to tell XDS to wait 1min for the next image before aborting.

!****************************************************************************************
!Parameters important for processing anomalous data:
!****************************************************************************************

FRIEDEL'S_LAW=FALSE ! This acts only on the CORRECT step. Even if the anomalous signal
! in your dataset is very small, it still may be useful for
! calculating an anomalous difference map.

STRICT_ABSORPTION_CORRECTION=FALSE !Change this to TRUE if the anomalous signal
! is strong: in that case, in CORRECT.LP the three
! "CHI^2-VALUE OF FIT OF CORRECTION FACTORS" values
! will be significantly> 1, e.g. 1.5.

WFAC1=1.0 !This parameter controls rejection of misfits during scaling. Sometimes,
! strong anomalous pairs may be rejected as misfits, in which case increasing
! WFAC1 (e.g. to 1.5) may improve anomalous signal.

!****************************************************************************************
!Regions of the detector to be excluded during indexing and integration:
!****************************************************************************************
! exclude (mask) untrusted areas of detector, e.g. beamstop shadow :
!UNTRUSTED_RECTANGLE= !1800 1950 2100 2150 ! x-min x-max y-min y-max ! repeat
!UNTRUSTED_ELLIPSE= !2034 2070 1850 2240 ! x-min x-max y-min y-max ! if needed
!UNTRUSTED_QUADRILATERAL= !x1 y1 x2 y2 x3 y3 x4 y4 ! see documentation

TRUSTED_REGION=0.00 1.2 ! Partially use corners of detectors; 1.41421=full use, 1.0=edge
! It is important to note that TRUSTED_REGION is the only resolution
! cutoff obeyed during indexing and integration;
! Both INCLUDE_RESOLUTION_RANGE and EXCLUDE_RESOLUTION_RANGE are
! ignored during IDXREF and INTEGRATE. So this parameter can be
! important when indexing or integration steps fail.

!****************************************************************************************
!Other parameters:
!****************************************************************************************
VALUE_RANGE_FOR_TRUSTED_DETECTOR_PIXELS=7000. 30000. ! Often 8000 is ok.
STRONG_PIXEL=4 ! COLSPOT: only use strong reflections (default is 3; 6 may be better for strong data)
MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT=3 ! default of 6 is sometimes too high
MAXIMUM_ERROR_OF_SPOT_POSITION=3 ! Increasing may help for poor quality data

MINPK=75 !Increase (e.g. MINPK=98) to improve data accuracy at the expense of completeness.

DELPHI=5 !Size of integration wedge. Increasing to 10 or 20 may be helpful in some cases.

NUMBER_OF_PROFILE_GRID_POINTS_ALONG_ALPHA/BETA=13 ! Default is 9 - Increasing may improve data
NUMBER_OF_PROFILE_GRID_POINTS_ALONG_GAMMA=13 ! accuracy, particularly if finely-sliced on phi,
! and does not seem to have any downsides.

MINIMUM_ZETA=0.05 ! Controls how close to the blind region (about phi) reflections should be
! integrated. 0.05 is the default; increasing MINIMUM_ZETA to 0.15 *may*
! improve data quality by removing unreliable reflections near the phi
! axis, but will reduce completeness in low symmetry space groups.

SEPMIN=6 ! Reduce if spots are close together (due to a long axis). Consider
CLUSTER_RADIUS=3 ! increasing if crystal is split and unit cell dimensions are relatively short.

! For bad or low resolution data remove the "!" in the following line (default is ALL):
! REFINE(IDXREF)=CELL BEAM ORIENTATION AXIS ! DISTANCE
REFINE(INTEGRATE)=CELL BEAM ORIENTATION ! AXIS DISTANCE (If integration is unstable, comment out this line.
! REFINE(CORRECT)=CELL BEAM ORIENTATION AXIS DISTANCE ! Default is: refine everything

ROTATION_AXIS=1 0 0 !At e.g. Australian Synchrotron, SERCAT ID-22 (?), APS 19-ID (?) this needs to be -1 0 0

!****************************************************************************************
! Parameters specifically for this detector and beamline (shouldn't need changing):
!****************************************************************************************
DETECTOR= $DETECTOR
SENSOR_THICKNESS= $SENSOR_THICKNESS
! attention CCD detectors: for very high resolution (better than 1A) make sure to specify SILICON
! as about 32* what CORRECT.LP suggests (absorption of phosphor is much higher than that of silicon)

NX= $NX NY= $NY QX= $QX QY= $QY ! to make CORRECT happy if frames are unavailable
!In mm, NX=$NXMM and NY=$NYMM

DIRECTION_OF_DETECTOR_X-AXIS=1 0 0
DIRECTION_OF_DETECTOR_Y-AXIS=0 1 0
INCIDENT_BEAM_DIRECTION=0 0 1
FRACTION_OF_POLARIZATION=0.98 ! better value is provided by beamline staff!
POLARIZATION_PLANE_NORMAL=0 1 0
eof
if [ "$DET" == "pilatus" ]; then
if [ $NX == "1475" ]; then
if ! grep -q FF_p2m0109_E12398_T6199_vrf_m0p20.tif tmp2 ; then
cat >> XDS.INP << eof
!EXCLUSION OF VERTICAL DEAD AREAS OF THE PILATUS 2M DETECTOR
UNTRUSTED_RECTANGLE= 486 496 0 1680
UNTRUSTED_RECTANGLE= 980 990 0 1680
!EXCLUSION OF HORIZONTAL DEAD AREAS OF THE PILATUS 2M DETECTOR
UNTRUSTED_RECTANGLE= 0 1476 194 214
UNTRUSTED_RECTANGLE= 0 1476 406 426
UNTRUSTED_RECTANGLE= 0 1476 618 638
UNTRUSTED_RECTANGLE= 0 1476 830 850
UNTRUSTED_RECTANGLE= 0 1476 1042 1062
UNTRUSTED_RECTANGLE= 0 1476 1254 1274
UNTRUSTED_RECTANGLE= 0 1476 1466 1486
eof
fi
# elif [ $NX == "2463" ]; then
# cat >> XDS.INP << eof
#eof
fi
fi
touch xds_to_ccp4.sh
echo "#!/bin/bash
#Removes old log file if present
rm xds_to_ccp4.log
#Makes new log file
touch xds_to_ccp4.log
#Pipe stdout and stderr to both logfile and tty
tail -f -n1 xds_to_ccp4.log &
exec > xds_to_ccp4.log 2>&1
echo \"

If this message is all you see,
something has probably gone wrong.
Are you sure XDS_ASCII.HKL is present
in the current directory?

\"
#Make first MTZ file (with DANO/SIGDANO)
echo \"INPUT_FILE=XDS_ASCII.HKL
OUTPUT_FILE=temp.hkl CCP4
MERGE=TRUE
GENERATE_FRACTION_OF_TEST_REFLECTIONS=0.05
FRIEDEL'S_LAW=FALSE\" > XDSCONV.INP
xdsconv
f2mtz HKLOUT temp.mtz<F2MTZ.INP

#Make second MTZ file (with F(+)/F(-))
echo \"INPUT_FILE=XDS_ASCII.HKL
OUTPUT_FILE=temp2.hkl CCP4_F
MERGE=TRUE
GENERATE_FRACTION_OF_TEST_REFLECTIONS=0.05
FRIEDEL'S_LAW=FALSE\" > XDSCONV.INP
xdsconv
f2mtz HKLOUT temp2.mtz<F2MTZ.INP

#Make SHELX format unmerged hkl file
echo \"INPUT_FILE=XDS_ASCII.HKL
OUTPUT_FILE=xds_shelx.hkl SHELX
MERGE=FALSE
FRIEDEL'S_LAW=FALSE\" > XDSCONV.INP
xdsconv

#cad the two previously generated MTZ files together
cad HKLIN1 temp.mtz HKLIN2 temp2.mtz HKLOUT xds_ccp4_merged.mtz<<EOF
LABIN FILE 1 E1=FP E2=SIGFP E3=DANO E4=SIGDANO E5=FreeRflag
LABIN FILE 2 E1=F(+) E2=SIGF(+) E3=F(-) E4=SIGF(-)
END
EOF

#Remove temporary files
rm temp.mtz
rm temp2.mtz
rm temp.hkl
rm temp2.hkl
rm F2MTZ.INP

echo \"

********************************XDS_TO_CCP4**********************************

This script takes XDS_ASCII.HKL and uses XDSCONV and F2MTZ to
generate a merged CCP4-format MTZ file named
\\\"xds_ccp4_merged.mtz\\\" with a test (free) set
labeled \\\"FreeRflag\\\" constituting 5% of reflections.

The space group of the output MTZ file will be the same as
that present in XDS_ASCII.HKL. It is probably worth checking
that this is the correct space group - either manually inspect
the systematic absences in CORRECT.LP, or run XDS_ASCII.HKL
through the CCP4 program POINTLESS (\\\"pointless xdsin XDS_ASCII.HKL\\\")

Anomalous data is kept, if present, and output as both
DANO/SIGDANO and F(+)/F(-).

This script also generates an unmerged SHELX hkl file,
suitable for input into SHELXC/D/E, named \\\"xds_shelx.hkl\\\".

A log of all processes is present in xds_to_ccp4.log.

Parts of this script have been shamelessly copied from
the XDS Wiki, at http://www.strucbio.biologie.uni-konstanz.de/xdswiki/

******************************************************************************
\"" > xds_to_ccp4.sh
touch xds_graph.sh
echo "#!/bin/bash
#Makes new log file
touch INTEGRATE_STATS.LP
#Pipe stdout and stderr to both logfile and tty
tail -f -n1 INTEGRATE_STATS.LP &
exec > INTEGRATE_STATS.LP 2>&1
if command -v xdsstat >/dev/null; then
echo \"\" | xdsstat > XDSSTAT.LP
echo \"\" | xdsstat 100 6 > XDSSTAT.LP_low
echo \"
\\\$TABLE:Rmeas vs image (to 6A):
\\\$SCATTER:Rmeas vs image for low resolution reflections:A:1,2: \\$\\$
IMAGE RMEAS
\\$\\$ \\$\\$
\"
grep ' L\$' XDSSTAT.LP_low | awk '{print \$1,\$9}'
echo \" \\$\\$ \"
echo \"
\\\$TABLE:Rmeas vs image (all):
\\\$SCATTER:Rmeas vs image (all reflections):A:1,2: \\$\\$
IMAGE RMEAS
\\$\\$ \\$\\$
\"
grep ' L$' XDSSTAT.LP | awk '{print \$1,\$9}'
echo \" \\$\\$ \"
echo \"
\\\$TABLE:Rd vs image (to 6A):
\\\$SCATTER:Rd (from XDSSTAT) for low resolution reflections:A:1,2: \\$\\$
IMAGE RD
\\$\\$ \\$\\$
\"
grep 'DIFFERENCE' XDSSTAT.LP_low | awk '{print \$1,\$7}'
echo \" \\$\\$ \"
echo \"
\\\$TABLE:Rd vs image (all):
\\\$SCATTER:Rd (from XDSSTAT) for all reflections:A:1,2: \\$\\$
IMAGE RD
\\$\\$ \\$\\$
\"
grep 'DIFFERENCE' XDSSTAT.LP | awk '{print \$1,\$7}'
echo \" \\$\\$ \"
fi

echo \"
\\\$TABLE: XDS Integration Statistics :
\\\$SCATTER:Scale factors by image:A:1,3: :Number of overloads by image:A:1,5: :Number of strong reflections by image:A:1,7: :Number of rejects by image:A:1,8: :Mosaicity by image:A:1,10: \\$\\$
IMAGE IER SCALE NBKG NOVL NEWALD NSTRONG NREJ SIGMAB SIGMAR
\\$\\$ \\$\\$
\"
egrep \"[0-9]\.[0-9][0-9][0-9][0-9] [0-9]\.[0-9][0-9][0-9][0-9]\" INTEGRATE.LP
echo \" \\$\\$ \"
echo \"
\\\$TABLE: XDS Scaling Statistics by Resolution (X-axis is resolution in Angstroems) :
\\\$GRAPHS:Chi-Squared by resolution:A:2,4: :Observed and expected R-factors by resolution (as %):A:2,5,6: :Rejects by resolution:A:2,9: :Rejects by resolution as a percentage of total reflections:A:2,10: \\$\\$
LOW_RES Resolution(Angstroems) I/Sigma Chi^2 R-FACTOR(OBSERVED) R-FACTOR(EXPECTED) NUMBER_OF_REFLECTIONS ACCEPTED REJECTS REJECT_PERCENT
\\$\\$ \\$\\$
\"
awk '/RESOLUTION RANGE I\/Sigma Chi\^2 R\-FACTOR R\-FACTOR NUMBER ACCEPTED REJECTED/, /\-\-\-\-\-\-\-\-\-/' CORRECT.LP | egrep -v \"[a-z,A-Z,] | \-\-|\-99\.9\" | egrep \"[0-9]\" | awk '{print \$1,\$2,\$3,\$4,\$5,\$6,\$7,\$8,\$9,(\$9/(\$8+\$9))*100}'
echo \" \\$\\$\"
echo \"
\\\$TABLE: Overall data quality by Resolution (X-axis is resolution in Angstroems) :
\\\$GRAPHS:Completeness vs resolution :A:1,6: :Redundancy vs resolution :A:1,4: :I/Sigma by resolution:A:1,10: :Rmeas by resolution:A:1,11: :Anomalous correlation by resolution:A:1,13: :Significance of the anomalous signal by resolution:A:1,14: \\$\\$
Resolution(Angstroems) OBSERVED_REFLECTIONS UNIQUE_REFLECTIONS REDUNDANCY POSSIBLE_REFLECTIONS COMPLETENESS R-FACTOR(OBSERVED) R-FACTOR(EXPECTED) COMPARED I/SIGMA R-meas CC(1/2) Anomalous_Correlation SigAno Nano
\\$\\$ \\$\\$
\"
egrep -B25 \" WILSON STATISTICS \" CORRECT.LP | egrep -v \"[a-z,A-Z]|\*\*\*\" | egrep \" \" | awk '{ gsub(/[%*]/,\" \"); print }' | awk '{print \$1,\$2,\$3,\$2/\$3,\$4,\$5,\$6,\$7,\$8,\$9,\$10,\$11,\$12,\$13,\$14}'
echo \" \\$\\$ \"

echo \"
\\\$TABLE: I/Sigma of h00 reflections (for identification of systematic absences):
\\\$GRAPHS:I/Sigma vs h :A:1,5: \\$\\$
h k l RES I/SIGMA NUMBER
\\$\\$ \\$\\$
\"
awk '/REFLECTIONS OF TYPE H/,/COMP/' CORRECT.LP | egrep \"[1-9]\" | awk '{print \$1,\$2,\$3,\$4,\$7,\$8}' | awk '(\$1 > 0)'
echo \"\\$\\$\"

echo \"
\\\$TABLE: I/Sigma of 0k0 reflections (for identification of systematic absences):
\\\$GRAPHS:I/Sigma vs k :A:2,5: \\$\\$
h k l RES I/SIGMA NUMBER
\\$\\$ \\$\\$
\"
awk '/REFLECTIONS OF TYPE H/,/COMP/' CORRECT.LP | egrep \"[1-9]\" | awk '{print \$1,\$2,\$3,\$4,\$7,\$8}' | awk '(\$2 > 0)'
echo \"\\$\\$\"

echo \"
\\\$TABLE: I/Sigma of 00l reflections (for identification of systematic absences):
\\\$GRAPHS:I/Sigma vs l :A:3,5: \\$\\$
h k l RES I/SIGMA NUMBER
\\$\\$ \\$\\$
\"
awk '/REFLECTIONS OF TYPE H/,/COMP/' CORRECT.LP | egrep \"[1-9]\" | awk '{print \$1,\$2,\$3,\$4,\$7,\$8}' | awk '(\$3 > 0)'
echo \"\\$\\$\"

echo \"
\\\$TABLE:Beam center (ORGX/ORGY) plotted vs wedge (DELPHI):
\\\$GRAPHS:ORGX and ORGY vs batch (DELPHI) :A:1,2,3: \\$\\$
N ORGX ORGY
\\$\\$ \\$\\$
\"
egrep \"DETECTOR ORIGIN\" INTEGRATE.LP | grep -n \"\" | awk '{ gsub(/[a-z,A-Z,),(,:]/,\"\"); print }'
echo \"\\$\\$\"

echo \"
\\\$TABLE:Unit cell parameters plotted vs wedge:
\\\$GRAPHS:Unit cell lengths a, b and c plotted vs wedge:A:1,2,3,4: :Unit cell angles alpha, beta and gamma plotted vs wedge:A:1,5,6,7: \\$\\$
N a b c ALPHA BETA GAMMA
\\$\\$ \\$\\$
\"
egrep \"UNIT CELL PARAMETERS\" INTEGRATE.LP | grep -n \"\" | awk '{ gsub(/[a-z,A-Z,),(,:]/,\"\"); print }'
echo \"\\$\\$\"

echo \"
\\\$TABLE:Standard deviation of spot position (pixels) plotted vs wedge:
\\\$GRAPHS:Standard deviation of spot position plotted vs wedge:A:1,2: \\$\\$
N SD
\\$\\$ \\$\\$
\"
egrep \"SPOT POSITION\" INTEGRATE.LP | grep -n \"\" | awk '{ gsub(/[a-z,A-Z,),(,:]/,\"\"); print }'
echo \"\\$\\$\"

echo \"
\\\$TABLE:Detector distance (mm) plotted vs wedge (DELPHI):
\\\$GRAPHS:Detector distance plotted vs batch:A:1,2: \\$\\$
N DIST
\\$\\$ \\$\\$
\"
egrep \"DETECTOR DISTANCE \$mm\$\" INTEGRATE.LP | grep -n \"\" | awk '{ gsub(/[a-z,A-Z,),(,:]/,\"\"); print }'

echo \"\\$\\$\"

loggraph INTEGRATE_STATS.LP">xds_graph.sh
chmod +x xds_to_ccp4.sh
chmod +x xds_graph.sh
echo "XDS.INP is ready for use. The file has only the most important keywords.
Full documentation, including complete detector templates, is at
http://www.mpimf-heidelberg.mpg.de/~kabsch/xds . More documentation in XDSwiki
After running xds, inspect, using XDS-Viewer, at least the beamstop mask in
BKGPIX.cbf, and the agreement of predicted and observed spots in FRAME.cbf!

Two shell scripts have been generated that may be of use.

The first, xds_graph.sh, will plot various statistics after an XDS run.
After both INTEGRATE and CORRECT have finished, run it by typing \"./xds_graph.sh\"
without the quotation marks. This script uses loggraph to plot data, so you will
need CCP4 to see the graphical output. All the raw data is piped out to a log file,
INTEGRATE_STATS.LP.

If XDSSTAT is present in your path, xds_graph will also plot some of the statistics it
calculates - specifically the Rmeas and Rd per image, for all reflections and only
low resolution reflections.

The second script, xds_to_ccp4.sh, will take XDS_ASCII.HKL and generate both a CCP4
format MTZ file with anomalous data retained (as F(+)/F(-) and DANO/SigDANO)
and an unmerged SHELX format hkl file for input to SHELXC/D/E. Run it after CORRECT
by typing \"./xds_to_ccp4.sh\" at the prompt."

rm -f tmp1 tmp2
</pre>

Use this script in the same manner as Generate_XDS.INP. The two shell scripts created should be executable; If they are not, then make them so (chmod +x ./xxx.sh).

Generate XDS.INP

2012-10-19T18:43:11Z

Olibclarke: xds_graph.sh will now run xdsstat, if available, and display Rmeas and Rd per image.

This script generates XDS.INP based on a list of frame names supplied on the commandline. It currently works for MarCCD, ADSC and Pilatus detectors; since this is just a bash script, extension to other detectors is very easy.

On Mac OS X, installation of the "Command Line Tools" (from http://developer.apple.com/downloads; requires Apple ID) is required (I think). These are a part of the (larger, but also free) [http://developer.apple.com/tools/xcode Xcode] package.

== Usage ==

Usage is just (don't forget the quotation marks!):
generate_XDS.INP "frms/mydata_1_???.img"
XDS [http://strucbio.biologie.uni-konstanz.de/~dikay/XDS_html_doc/html_doc/xds_parameters.html#NAME_TEMPLATE_OF_DATA_FRAMES= supports] bzip2-ed frames. Thus, when specifying the frame name parameter of the script, you should leave out any .bz2 extension.

== The script ==

<pre>
#!/bin/bash
# purpose: generate XDS.INP
#
# tested with some datasets from ALS, SSRL, SLS, ESRF and BESSY; only MARCCD, ADSC/SMV, PILATUS detectors;
# for other detectors, values marked with XXX must be manually filled in.
#
# revision 0.03 . Kay Diederichs 2/2010
# revision 0.04 . Kay Diederichs 4/2010 - include alternative ORGX, ORGY calculations for ADSC
# revision 0.05 . Kay Diederichs 5/2010 - grep for "Corrected" in addition to "marccd"; needed for BESSY
# revision 0.06 . KD 6/2010 - add UNTRUSTED_RECTANGLE and UNTRUSTED_ELLIPSE; use `whereis catmar` and so on
# revision 0.07 . KD 6/2010 - decide about ORGX/Y info in MAR header being pixels or mm; other fixes
# revision 0.08 . KD 6/2010 - fixes for Pilatus 6M
# revision 0.09 . KD 6/2010 - get rid of requirement for mccd_xdsparams.pl and/or catmar; rather use "od"
# revision 0.10 . Tim Gruene 7/2010 - set link 'images' to image directory if path exceeds 72 characters
# revision 0.11 . KD 7/2010 - for MarCCD: look for distance info at different byte position
# revision 0.12 . KD 7/2010 - fix for negative PHISTART
# revision 0.13 . KD 8/2010 - store correct NX NY QX QY in XDS.INP
# revision 0.14 . KD 1/2011 - SENSOR_THICKNESS for Pilatus; MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT=3
# revision 0.15 . KD 2/2011 - add comment for -ive sign of APS 19-ID and Australian Synchrotron rotation axis
# revision 0.16 . KD 3/2011 - SENSOR_THICKNESS=0.01 for ADSC and MarCCD. Add comment about SILICON=
# revision 0.17 . KD 3/2011 - make it work for .bz2 frames; improve screen output
# revision 0.18 . KD 4/2011 - faster by doing "strings" only once; revert "images/${1##/*/}" "correction"
# revision 0.19 . KD 6/2011 - bugfix for 0.18
# revision 0.20 . KD 7/2011 - redirect stderr of /bin/ls to /dev/null
# revision 0.21 . KD 11/2011 - SEPMIN, CLUSTER_RADIUS hints; read NX NY from header (for Pilatus 2M)
# revision 0.22 . KD 12/2011 - Pilatus 2M UNTRUSTED_RECTANGLE lines, SENSOR_THICKNESS from header
# revision 0.23 . KD 1/2012 - add UNTRUSTED_QUADRILATERAL, remove MINIMUM_ZETA (0.05 is default now)
# revision 0.24 . KD 3/2012 - remove revision 0.10 since XDS now takes much longer paths
# revision 0.25 . KD 3/2012 - remove revision 0.22 for PSI Pilatus 2M; see http://www.globalphasing.com/autoproc/wiki/index.cgi?TroubleShootingKnownIssues
# revision 0.26 . KD 7/2012 - Mac-compatibility: replace od flags --skip-bytes= and --read-bytes= with -j and -N (thanks to Oliver Clarke for working this out!)
REVISION="0.26 (02-Jul-2012)"
#
# usage: e.g. generate_XDS.INP "frms/mydata_1_???.img"
# make sure to have the two quotation marks !
# the ? are wildcards for the frame numbers.
#
# limitations:
# - frame numbers are assumed to start with 1 and run consecutively
#
# known problems:
# - for ADSC detectors, there are at least three ways to obtain ORGX and ORGY values from the header (see below);
# - the same might be a problem for MAR headers, too (not sure about this)
# - on Mac OS X, Xcode (from http://developer.apple.com/tools/xcode) might be needed - not sure about this
#
# notes for debugging of the script:
# - add the -v option to the first line, to see where an error occurs
# - comment out the removal of tmp1 and tmp2 in the last line
#
# ====== Start of script ======
echo generate_XDS.INP version $REVISION . Obtain the latest version from
echo http://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/generate_XDS.INP
if [ "$1" == "help" ] || [ "$1" == "-help" ] || [ "$1" == "-h" ]; then
echo usage: generate_XDS.INP \"frms/mydata_1_???.img\" $_with_ the quotation marks!$
echo if the frames are compressed with bzip2, leave out the .bz2 extension!
exit
fi
#
# defaults:
#
DETECTOR="XXX MINIMUM_VALID_PIXEL_VALUE=XXX OVERLOAD=XXX"
ORGX=XXX
ORGY=XXX
DETECTOR_DISTANCE=XXX
OSCILLATION_RANGE=XXX
X_RAY_WAVELENGTH=XXX
QX=XXX
QY=XXX
NX=XXX
NY=XXX
SENSOR_THICKNESS=0
# see how we are called:
NAME_TEMPLATE_OF_DATA_FRAMES="$1"
# list frames matching the wildcards in NAME_TEMPLATE_OF_DATA_FRAMES
# don't accept the "direct beam" shot at SLS/Pilatus PX-I and PX-II
/bin/ls -C1 $1 $1.bz2 2>/dev/null | egrep -v "_00000.cbf|_000.img" > tmp1 || exit 1

# we can continue - the frames are found

# set upper limit of DATA_RANGE to number of frames (see "limitations" above)
DATA_RANGE=`wc -l tmp1 | awk '{print $1}'`

# set upper limit of SPOT_RANGE to half of DATA_RANGE, but not less than 1
SPOT_RANGE=`echo "scale=0; $DATA_RANGE/2" | bc -l`
SPOT_RANGE=`echo "if ($SPOT_RANGE<1) 1;if ($SPOT_RANGE>1) $SPOT_RANGE" | bc -l`

echo DATA_RANGE=1 $DATA_RANGE

# find out detector type
DET=XXX
FIRSTFRAME=`head -1 tmp1`
echo $FIRSTFRAME | grep -q bz2 && bzcat $FIRSTFRAME > tmp1 && FIRSTFRAME=tmp1
strings $FIRSTFRAME > tmp2
egrep -q 'marccd|Corrected' tmp2 && DET=mccd
grep -q PILATUS tmp2 && DET=pilatus
grep -q BEAM_CENTER_X tmp2 && DET=adsc
# identify other detector types in the same way (MAR IP would be straightforward)

# parse ASCII header of first frame

if [ "$DET" == "XXX" ]; then
echo "this is not a MAR, ADSC/SMV or PILATUS detector - fill in XXX values manually!"
DETECTOR="XXX MINIMUM_VALID_PIXEL_VALUE=XXX OVERLOAD=XXX"

# find parameters of first frame
elif [ "$DET" == "mccd" ]; then
echo Data from a MarCCD detector

DETECTOR="CCDCHESS MINIMUM_VALID_PIXEL_VALUE= 1 OVERLOAD= 65500"
SENSOR_THICKNESS=0.01
# use first frame of dataset to obtain parameters

# offsets are documented; values can be found in mccd_xdsparams.pl script
let SKIP=1024+80
NX=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
let SKIP=$SKIP+4
NY=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')

let SKIP=1720
DETECTOR_DISTANCE=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
DETECTOR_DISTANCE=`echo "scale=3; $DETECTOR_DISTANCE/1000" | bc -l`

let SKIP=1024+256+128+256+4
ORGX=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
ORGX=`echo "scale=2; $ORGX/1000" | bc -l `
let SKIP=$SKIP+4
ORGY=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
ORGY=`echo "scale=2; $ORGY/1000" | bc -l `

let SKIP=1024+256+128+256+44
PHISTART=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
let SKIP=1024+256+128+256+76
PHIEND=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
OSCILLATION_RANGE=`echo "scale=3; ($PHIEND-($PHISTART))/1000" | bc -l`

let SKIP=1024+256+128+256+128+4
QX=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
QX=`echo "scale=10; $QX/1000000" |bc -l `
let SKIP=$SKIP+4
QY=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
QY=`echo "scale=10; $QY/1000000" |bc -l `

let SKIP=1024+256+128+256+128+128+12
X_RAY_WAVELENGTH=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
X_RAY_WAVELENGTH=`echo "scale=5; $X_RAY_WAVELENGTH/100000" | bc -l`

# at most BLs, ORGX and ORGY are in pixels, but sometimes in mm ... guess:
NXBYFOUR=`echo "scale=0; $NX/4" | bc -l `
ORGXINT=`echo "scale=0; $ORGX/1" | bc -l `
if [ $ORGXINT -lt $NXBYFOUR ]; then
ORGX=`echo "scale=1; $ORGX/$QX" | bc -l`
ORGY=`echo "scale=1; $ORGY/$QY" | bc -l`
echo MARCCD detector: header ORGX, ORGY seem to be in mm ... converting to pixels
else
echo MARCCD detector: header ORGX, ORGY seem to be in pixel units
fi

elif [ "$DET" == "adsc" ]; then

DETECTOR="ADSC MINIMUM_VALID_PIXEL_VALUE= 1 OVERLOAD= 65000"
echo Data from ADSC detector. Obtaining ORGX, ORGY depends on beamline setup:
SENSOR_THICKNESS=0.01
sed s/\;// tmp2 > tmp1
mv tmp1 tmp2

# find X_RAY_WAVELENGTH:
X_RAY_WAVELENGTH=`grep WAVELENGTH tmp2 | head -1 | sed s/WAVELENGTH=//`

# find NX, QX, ORGX and ORGY:
NX=`grep SIZE1 tmp2 | tail -1 | sed s/SIZE1=//`
QX=`grep PIXEL_SIZE tmp2 | sed s/PIXEL_SIZE=//`
# FIXME - next 2 lines should be done properly, from header
NY=$NX
QY=$QX
BEAM_CENTER_X=`grep BEAM_CENTER_X tmp2 | sed s/BEAM_CENTER_X=//`
BEAM_CENTER_Y=`grep BEAM_CENTER_Y tmp2 | sed s/BEAM_CENTER_Y=//`
# fix 2010-04-26 - tell user about possible ORGX, ORGY alternatives -
# at ESRF and ... (pls fill in!) the following should be used:
ORGX=`echo "scale=1; $BEAM_CENTER_Y/$QX" | bc -l`
ORGY=`echo "scale=1; $BEAM_CENTER_X/$QX" | bc -l`
echo - at ESRF BLs use: ORGX=$ORGX ORGY=$ORGY
# this 2nd alternative convention should be used at the following beamlines (pls complete the list): ALS 5.0.3, ...
ORGX=`echo "scale=1; $NX-$BEAM_CENTER_X/$QX" | bc -l `
ORGY=`echo "scale=1; $BEAM_CENTER_Y/$QX" | bc -l `
echo - at e.g. ALS 5.0.3 use: ORGX=$ORGX ORGY=$ORGY
# this 3rd alternative convention should be used at the following beamlines (pls complete the list): ALS 8.2.2, ...
ORGX=`echo "scale=1; $BEAM_CENTER_X/$QX" | bc -l `
ORGY=`echo "scale=1; $NX-$BEAM_CENTER_Y/$QX" | bc -l `
echo - at e.g. ALS 8.2.2 use: ORGX=$ORGX ORGY=$ORGY - this is written to XDS.INP
# the latter alternative is written into the generated XDS.INP ! You have to correct this manually in XDS.INP, or adjust this script.
# find DETECTOR_DISTANCE and OSCILLATION_RANGE:
DETECTOR_DISTANCE=`grep DISTANCE tmp2 | sed s/DISTANCE=//`
OSCILLATION_RANGE=`grep OSC_RANGE tmp2 | sed s/OSC_RANGE=//`

elif [ "$DET" == "pilatus" ]; then
DETECTOR="PILATUS MINIMUM_VALID_PIXEL_VALUE=0 OVERLOAD= 1048576 !PILATUS"
QX=0.172 QY=0.172
echo Data from a Pilatus detector
sed s/#// tmp2 > tmp1
mv tmp1 tmp2

# find SENSOR_THICKNESS:
SENSOR_THICKNESS=`grep thickness tmp2 | sed -e s/'Silicon sensor, thickness'// | awk '{print $1*1000}'`
# find X_RAY_WAVELENGTH:
X_RAY_WAVELENGTH=`grep Wavelength tmp2 | sed -e s/Wavelength// -e s/A// | awk '{print $1}'`

# find NX and NY; 2463/2527 is 6M, 1475/1679 is 2M
NX=`grep X-Binary-Size-Fastest-Dimension tmp2 | awk '{print $2}'`
NY=`grep X-Binary-Size-Second-Dimension tmp2 | awk '{print $2}'`

# find ORGX and ORGY:
ORGX=`grep Beam_xy tmp2 | sed -e s/$// -e s/$// -e s/\,// | awk '{print $2}'`
ORGY=`grep Beam_xy tmp2 | sed -e s/$// -e s/$// -e s/\,// | awk '{print $3}'`

# find DETECTOR_DISTANCE and OSCILLATION_RANGE:
DETECTOR_DISTANCE=`awk '/distance/{print $2}' tmp2`
DETECTOR_DISTANCE=`echo "$DETECTOR_DISTANCE*1000" | bc -l`

OSCILLATION_RANGE=`awk '/Angle/{print $2}' tmp2`

else
echo should never come here
exit 1
fi

echo ORGX= $ORGX ORGY= $ORGY - check these values with adxv !
echo DETECTOR_DISTANCE= $DETECTOR_DISTANCE
echo OSCILLATION_RANGE= $OSCILLATION_RANGE
echo X-RAY_WAVELENGTH= $X_RAY_WAVELENGTH

# now we know everything that is required to generate XDS.INP

cat > XDS.INP << eof
! written by generate_XDS.INP version $REVISION
JOB= XYCORR INIT COLSPOT IDXREF DEFPIX INTEGRATE CORRECT
ORGX= $ORGX ORGY= $ORGY ! check these values with adxv !
DETECTOR_DISTANCE= $DETECTOR_DISTANCE
OSCILLATION_RANGE= $OSCILLATION_RANGE
X-RAY_WAVELENGTH= $X_RAY_WAVELENGTH
NAME_TEMPLATE_OF_DATA_FRAMES=$NAME_TEMPLATE_OF_DATA_FRAMES
! REFERENCE_DATA_SET=xxx/XDS_ASCII.HKL ! e.g. to ensure consistent indexing
DATA_RANGE=1 $DATA_RANGE
SPOT_RANGE=1 $SPOT_RANGE
! BACKGROUND_RANGE=1 10 ! rather use defaults (first 5 degree of rotation)

SPACE_GROUP_NUMBER=0 ! 0 if unknown
UNIT_CELL_CONSTANTS= 70 80 90 90 90 90 ! put correct values if known
INCLUDE_RESOLUTION_RANGE=50 0 ! after CORRECT, insert high resol limit; re-run CORRECT

FRIEDEL'S_LAW=FALSE ! This acts only on the CORRECT step
! If the anom signal turns out to be, or is known to be, very low or absent,
! use FRIEDEL'S_LAW=TRUE instead (or comment out the line); re-run CORRECT

! remove the "!" in the following line:
! STRICT_ABSORPTION_CORRECTION=TRUE
! if the anomalous signal is strong: in that case, in CORRECT.LP the three
! "CHI^2-VALUE OF FIT OF CORRECTION FACTORS" values are significantly> 1, e.g. 1.5
!
! exclude (mask) untrusted areas of detector, e.g. beamstop shadow :
! UNTRUSTED_RECTANGLE= 1800 1950 2100 2150 ! x-min x-max y-min y-max ! repeat
! UNTRUSTED_ELLIPSE= 2034 2070 1850 2240 ! x-min x-max y-min y-max ! if needed
! UNTRUSTED_QUADRILATERAL= x1 y1 x2 y2 x3 y3 x4 y4 ! see documentation
!
! parameters with changes wrt default values:
TRUSTED_REGION=0.00 1.2 ! partially use corners of detectors; 1.41421=full use
VALUE_RANGE_FOR_TRUSTED_DETECTOR_PIXELS=7000. 30000. ! often 8000 is ok
STRONG_PIXEL=4 ! COLSPOT: only use strong reflections (default is 3)
MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT=3 ! default of 6 is sometimes too high
! close spots: reduce SEPMIN and CLUSTER_RADIUS from their defaults of 6 and 3, e.g. to 4 and 2
! for bad or low resolution data remove the "!" in the following line:
! REFINE(IDXREF)=CELL BEAM ORIENTATION AXIS ! DISTANCE
REFINE(INTEGRATE)=CELL BEAM ORIENTATION ! AXIS DISTANCE
! REFINE(CORRECT)=CELL BEAM ORIENTATION AXIS DISTANCE ! Default is: refine everything

! parameters specifically for this detector and beamline:
DETECTOR= $DETECTOR
SENSOR_THICKNESS= $SENSOR_THICKNESS
! attention CCD detectors: for very high resolution (better than 1A) make sure to specify SILICON
! as about 32* what CORRECT.LP suggests (absorption of phosphor is much higher than that of silicon)
NX= $NX NY= $NY QX= $QX QY= $QY ! to make CORRECT happy if frames are unavailable
DIRECTION_OF_DETECTOR_X-AXIS=1 0 0
DIRECTION_OF_DETECTOR_Y-AXIS=0 1 0
INCIDENT_BEAM_DIRECTION=0 0 1
ROTATION_AXIS=1 0 0 ! at e.g. Australian Synchrotron, SERCAT ID-22 (?), APS 19-ID (?) this needs to be -1 0 0
FRACTION_OF_POLARIZATION=0.98 ! better value is provided by beamline staff!
POLARIZATION_PLANE_NORMAL=0 1 0
eof
if [ "$DET" == "pilatus" ]; then
if [ $NX == "1475" ]; then
if ! grep -q FF_p2m0109_E12398_T6199_vrf_m0p20.tif tmp2 ; then
cat >> XDS.INP << eof
!EXCLUSION OF VERTICAL DEAD AREAS OF THE PILATUS 2M DETECTOR
UNTRUSTED_RECTANGLE= 486 496 0 1680
UNTRUSTED_RECTANGLE= 980 990 0 1680
!EXCLUSION OF HORIZONTAL DEAD AREAS OF THE PILATUS 2M DETECTOR
UNTRUSTED_RECTANGLE= 0 1476 194 214
UNTRUSTED_RECTANGLE= 0 1476 406 426
UNTRUSTED_RECTANGLE= 0 1476 618 638
UNTRUSTED_RECTANGLE= 0 1476 830 850
UNTRUSTED_RECTANGLE= 0 1476 1042 1062
UNTRUSTED_RECTANGLE= 0 1476 1254 1274
UNTRUSTED_RECTANGLE= 0 1476 1466 1486
eof
fi
# elif [ $NX == "2463" ]; then
# cat >> XDS.INP << eof
#eof
fi
fi
echo XDS.INP is ready for use. The file has only the most important keywords.
echo Full documentation, including complete detector templates, is at
echo http://www.mpimf-heidelberg.mpg.de/~kabsch/xds . More documentation in XDSwiki
echo After running xds, inspect, using XDS-Viewer, at least the beamstop mask in
echo BKGPIX.cbf, and the agreement of predicted and observed spots in FRAME.cbf!
rm -f tmp1 tmp2
</pre>

== System-wide or personal installation ==

Ask your system adminstrator to cut-and-paste the script into e.g. /usr/local/bin/generate_XDS.INP, and to make it "executable".

But you may also cut-and-paste the script from this webpage into a file in e.g. your home directory; the filename should be generate_XDS.INP. After creating the file, make it executable - e.g. if it's in your $HOME, use:
chmod +x ~/generate_XDS.INP
After that, you can just run it in a similar way as if it were installed in your $PATH:
~/generate_XDS.INP "frms/mydata_1_???.img"
By using your own file, you can easily update to the latest revision, or even change the script, without having to bother the system administrator.

== Generating generate_XDS.INP from this webpage ==

Instead of cutting-and-pasting the lines of the script, you (or the system administrator) could just cut-and-paste the following four lines
<pre>
wget http://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/generate_XDS.INP -O - | \
sed -e s/\&nbsp\;/\ /g -e s/\&gt\;/\>/g -e s/\&lt\;/\</g -e s/amp\;//g -e s/\&quot\;/\"/g -e s/\&\#\1\6\0\;/\ /g | \
awk '/^#/,/rm -f tmp1 tmp2/' > generate_XDS.INP
chmod +x generate_XDS.INP
</pre>
to copy the script from this website into an executable file generate_XDS.INP in your current directory.

==Variant script (xds_generate_all)==
This script includes some minor modifications of the Generate_XDS.INP script - for instance, it tells the user the energy, as well as the wavelength of the X-rays, and gives the user the beam center in mm as well as pixels - values that may be useful when switching between HKL2000, MOSFLM and XDS.

It also generates two executable shell scripts in the same directory as XDS.INP.

The first of these scripts, xds_graph.sh, extracts various statistics from INTEGRATE.LP and CORRECT.LP, pipes them out to a logfile, INTEGRATE_STATS.LP, and plots them using loggraph. This script does not have some of the more useful features of XDSSTAT, such as calculation of per frame Rmeas values, but it does give a quick overview of various data quality parameters by resolution and image number. This script will only work with recent versions of XDS (after calculation of CC(1/2) - not sure exactly which version).

The second of these scripts, xds_to_ccp4.sh, will take XDS_ASCII.HKL and generate a CCP4-format MTZ file, containing anomalous data (as F(+)/F(-) and DANO/SigDANO) and a test set of 5% of reflections for calculation of the free R-factor. It will also generate an unmerged SHELX format HKL file, for input into SHELXC/D/E (e.g. via the HKL2MAP GUI).

Both these shell scripts can by installed system wide by transferring them to a directory in your path, for example the directory containing your XDS binaries.

Here is the script:
<pre>
#!/bin/bash
# purpose: xds_generate_all
#
# tested with some datasets from ALS, SSRL, SLS, ESRF and BESSY; only MARCCD, ADSC/SMV, PILATUS detectors;
# for other detectors, values marked with XXX must be manually filled in.
#
# revision 0.03 . Kay Diederichs 2/2010
# revision 0.04 . Kay Diederichs 4/2010 - include alternative ORGX, ORGY calculations for ADSC
# revision 0.05 . Kay Diederichs 5/2010 - grep for "Corrected" in addition to "marccd"; needed for BESSY
# revision 0.06 . KD 6/2010 - add UNTRUSTED_RECTANGLE and UNTRUSTED_ELLIPSE; use `whereis catmar` and so on
# revision 0.07 . KD 6/2010 - decide about ORGX/Y info in MAR header being pixels or mm; other fixes
# revision 0.08 . KD 6/2010 - fixes for Pilatus 6M
# revision 0.09 . KD 6/2010 - get rid of requirement for mccd_xdsparams.pl and/or catmar; rather use "od"
# revision 0.10 . Tim Gruene 7/2010 - set link 'images' to image directory if path exceeds 72 characters
# revision 0.11 . KD 7/2010 - for MarCCD: look for distance info at different byte position
# revision 0.12 . KD 7/2010 - fix for negative PHISTART
# revision 0.13 . KD 8/2010 - store correct NX NY QX QY in XDS.INP
# revision 0.14 . KD 1/2011 - SENSOR_THICKNESS for Pilatus; MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT=3
# revision 0.15 . KD 2/2011 - add comment for -ive sign of APS 19-ID and Australian Synchrotron rotation axis
# revision 0.16 . KD 3/2011 - SENSOR_THICKNESS=0.01 for ADSC and MarCCD. Add comment about SILICON=
# revision 0.17 . KD 3/2011 - make it work for .bz2 frames; improve screen output
# revision 0.18 . KD 4/2011 - faster by doing "strings" only once; revert "images/${1##/*/}" "correction"
# revision 0.19 . KD 6/2011 - bugfix for 0.18
# revision 0.20 . KD 7/2011 - redirect stderr of /bin/ls to /dev/null
# revision 0.21 . KD 11/2011 - SEPMIN, CLUSTER_RADIUS hints; read NX NY from header (for Pilatus 2M)
# revision 0.22 . KD 12/2011 - Pilatus 2M UNTRUSTED_RECTANGLE lines, SENSOR_THICKNESS from header
# revision 0.23 . KD 1/2012 - add UNTRUSTED_QUADRILATERAL, remove MINIMUM_ZETA (0.05 is default now)
# revision 0.24 . KD 3/2012 - remove revision 0.10 since XDS now takes much longer paths
# revision 0.25 . KD 3/2012 - remove revision 0.22 for PSI Pilatus 2M; see http://www.globalphasing.com/autoproc/wiki/index.cgi?TroubleShootingKnownIssues
# revision 0.26 Oliver Clarke 10/12 - Added generation of shell scripts for conversion to MTZ and visualisation of statistics. Other minor alterations to XDS.INP.
# revision 0.27 Oliver Clarke 10/12 - xds_graph.sh will now run xdsstat, if available, and plot Rmeas and Rd per image.
REVISION="0.27 (19-Oct-2012)"
#
# usage: e.g. xds_generate_all "frms/mydata_1_???.img"
# make sure to have the two quotation marks !
# the ? are wildcards for the frame numbers.
#
# limitations:
# - frame numbers are assumed to start with 1 and run consecutively
#
# known problems:
# - for ADSC detectors, there are at least three ways to obtain ORGX and ORGY values from the header (see below);
# - the same might be a problem for MAR headers, too (not sure about this)
#
# notes for debugging of the script:
# - add the -v option to the first line, to see where an error occurs
# - comment out the removal of tmp1 and tmp2 in the last line
#
# ====== Start of script ======
echo generate_XDS.INP version $REVISION . Obtain the latest version from
echo http://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/generate_XDS.INP
if [ "$1" == "help" ] || [ "$1" == "-help" ] || [ "$1" == "-h" ]; then
echo usage: xds_generate_all \"frms/mydata_1_???.img\" $_with_ the quotation marks!$
echo if the frames are compressed with bzip2, leave out the .bz2 extension!
exit
fi
#
# defaults:
#
DETECTOR="XXX MINIMUM_VALID_PIXEL_VALUE=XXX OVERLOAD=XXX"
ORGX=XXX
ORGY=XXX
ORGXMM=XXX
ORGYMM=XXX
DETECTOR_DISTANCE=XXX
OSCILLATION_RANGE=XXX
X_RAY_WAVELENGTH=XXX
X_RAY_ENERGY=XXX
STARTING_ANGLE=XXX
QX=XXX
QY=XXX
NX=XXX
NY=XXX
NXMM=XXX
NYMM=XXX
SENSOR_THICKNESS=0
# see how we are called:
NAME_TEMPLATE_OF_DATA_FRAMES="$1"
# list frames matching the wildcards in NAME_TEMPLATE_OF_DATA_FRAMES
# don't accept the "direct beam" shot at SLS/Pilatus PX-I and PX-II
/bin/ls -C1 $1 $1.bz2 2>/dev/null | egrep -v "_00000.cbf|_000.img" > tmp1 || exit 1

# we can continue - the frames are found

# set upper limit of DATA_RANGE to number of frames (see "limitations" above)
DATA_RANGE=`wc -l tmp1 | awk '{print $1}'`

# set upper limit of SPOT_RANGE to half of DATA_RANGE, but not less than 1
SPOT_RANGE=`echo "scale=0; $DATA_RANGE/2" | bc -l`
SPOT_RANGE=`echo "if ($SPOT_RANGE<1) 1;if ($SPOT_RANGE>1) $SPOT_RANGE" | bc -l`

echo DATA_RANGE=1 $DATA_RANGE

# find out detector type
DET=XXX
FIRSTFRAME=`head -1 tmp1`
echo $FIRSTFRAME | grep -q bz2 && bzcat $FIRSTFRAME > tmp1 && FIRSTFRAME=tmp1
strings $FIRSTFRAME > tmp2
egrep -q 'marccd|Corrected' tmp2 && DET=mccd
grep -q PILATUS tmp2 && DET=pilatus
grep -q BEAM_CENTER_X tmp2 && DET=adsc
# identify other detector types in the same way (MAR IP would be straightforward)

# parse ASCII header of first frame

if [ "$DET" == "XXX" ]; then
echo "this is not a MAR, ADSC/SMV or PILATUS detector - fill in XXX values manually!"
DETECTOR="XXX MINIMUM_VALID_PIXEL_VALUE=XXX OVERLOAD=XXX"

# find parameters of first frame
elif [ "$DET" == "mccd" ]; then
echo Data from a MarCCD detector

DETECTOR="CCDCHESS MINIMUM_VALID_PIXEL_VALUE= 1 OVERLOAD= 65500"
SENSOR_THICKNESS=0.01
# use first frame of dataset to obtain parameters

# offsets are documented; values can be find in mccd_xdsparams.pl script
let SKIP=1024+80
NX=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
let SKIP=$SKIP+4
NY=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')

let SKIP=1720
DETECTOR_DISTANCE=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
DETECTOR_DISTANCE=`echo "scale=3; $DETECTOR_DISTANCE/1000" | bc -l`

let SKIP=1024+256+128+256+4
ORGX=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
ORGX=`echo "scale=2; $ORGX/1000" | bc -l `
let SKIP=$SKIP+4
ORGY=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
ORGY=`echo "scale=2; $ORGY/1000" | bc -l `

let SKIP=1024+256+128+256+44
PHISTART=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
let SKIP=1024+256+128+256+76
PHIEND=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
OSCILLATION_RANGE=`echo "scale=3; ($PHIEND-($PHISTART))/1000" | bc -l`
STARTING_ANGLE=`echo "scale=3; ($PHISTART/1000)" | bc -l`

let SKIP=1024+256+128+256+128+4
QX=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
QX=`echo "scale=10; $QX/1000000" |bc -l `
let SKIP=$SKIP+4
QY=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
QY=`echo "scale=10; $QY/1000000" |bc -l `

let SKIP=1024+256+128+256+128+128+12
X_RAY_WAVELENGTH=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
X_RAY_WAVELENGTH=`echo "scale=5; $X_RAY_WAVELENGTH/100000" | bc -l`
X_RAY_ENERGY=`echo "12398.5/$X_RAY_WAVELENGTH" | bc -l | awk '{printf "%5.1f", $1}'`
# at most BLs, ORGX and ORGY are in pixels, but sometimes in mm ... guess:
NXBYFOUR=`echo "scale=0; $NX/4" | bc -l `
ORGXINT=`echo "scale=0; $ORGX/1" | bc -l `
if [ $ORGXINT -lt $NXBYFOUR ]; then
ORGXMM=$ORGX
ORGYMM=$ORGY
NXMM=`echo "$NX*$QX" | bc -l | awk '{printf "%3.2f", $1}'`
NYMM=`echo "$NY*$QY" | bc -l | awk '{printf "%3.2f", $1}'`
ORGX=`echo "scale=1; $ORGX/$QX" | bc -l`
ORGY=`echo "scale=1; $ORGY/$QY" | bc -l`
echo MARCCD detector: header ORGX, ORGY seem to be in mm ... converting to pixels
else
ORGXMM=`echo "$ORGX*$QX" | bc -l | awk '{printf "%3.2f", $1}'`
ORGYMM=`echo "$ORGY*$QY" | bc -l | awk '{printf "%3.2f", $1}'`
NXMM=`echo "$NX*$QX" | bc -l | awk '{printf "%3.2f", $1}'`
NYMM=`echo "$NY*$QY" | bc -l | awk '{printf "%3.2f", $1}'`
echo MARCCD detector: header ORGX, ORGY seem to be in pixel units
fi

elif [ "$DET" == "adsc" ]; then

DETECTOR="ADSC MINIMUM_VALID_PIXEL_VALUE= 1 OVERLOAD= 65000"
echo Data from ADSC detector. Obtaining ORGX, ORGY depends on beamline setup:
SENSOR_THICKNESS=0.01
sed s/\;// tmp2 > tmp1
mv tmp1 tmp2

# find X_RAY_WAVELENGTH:
X_RAY_WAVELENGTH=`grep WAVELENGTH tmp2 | head -1 | sed s/WAVELENGTH=//`

# find NX, QX, ORGX and ORGY:
X_RAY_ENERGY=`echo "12398.5/$X_RAY_WAVELENGTH" | bc -l | awk '{printf "%5.1f", $1}'`
STARTING_ANGLE=`grep PHI tmp2 | sed s/PHI=//`
NX=`grep SIZE1 tmp2 | tail -1 | sed s/SIZE1=//`
QX=`grep PIXEL_SIZE tmp2 | sed s/PIXEL_SIZE=//`
# FIXME - next 2 lines should be done properly, from header
NY=$NX
QY=$QX
NXMM=`echo "$NX*$QX" | bc -l | awk '{printf "%3.2f", $1}'`
NYMM=`echo "$NY*$QY" | bc -l | awk '{printf "%3.2f", $1}'`
BEAM_CENTER_X=`grep BEAM_CENTER_X tmp2 | sed s/BEAM_CENTER_X=//`
BEAM_CENTER_Y=`grep BEAM_CENTER_Y tmp2 | sed s/BEAM_CENTER_Y=//`
# fix 2010-04-26 - tell user about possible ORGX, ORGY alternatives -
# at ESRF and ... (pls fill in!) the following should be used:
ORGX=`echo "scale=1; $BEAM_CENTER_Y/$QX" | bc -l`
ORGY=`echo "scale=1; $BEAM_CENTER_X/$QX" | bc -l`
echo - at ESRF BLs use: ORGX=$ORGX ORGY=$ORGY
# this 2nd alternative convention should be used at the following beamlines (pls complete the list): ALS 5.0.3, ...
ORGX=`echo "scale=1; $NX-$BEAM_CENTER_X/$QX" | bc -l `
ORGY=`echo "scale=1; $BEAM_CENTER_Y/$QX" | bc -l `
echo - at e.g. ALS 5.0.3 use: ORGX=$ORGX ORGY=$ORGY
# this 3rd alternative convention should be used at the following beamlines (pls complete the list): ALS 8.2.2, ...
ORGX=`echo "scale=1; $BEAM_CENTER_X/$QX" | bc -l `
ORGY=`echo "scale=1; $NX-$BEAM_CENTER_Y/$QX" | bc -l `
ORGXMM=`echo "$ORGX*$QX" | bc -l | awk '{printf "%3.2f", $1}'`
ORGYMM=`echo "$ORGY*$QY" | bc -l | awk '{printf "%3.2f", $1}'`
echo - at e.g. ALS 8.2.2 use: ORGX=$ORGX ORGY=$ORGY - this is written to XDS.INP
# the latter alternative is written into the generated XDS.INP ! You have to correct this manually in XDS.INP, or adjust this script.
# find DETECTOR_DISTANCE and OSCILLATION_RANGE:
DETECTOR_DISTANCE=`grep DISTANCE tmp2 | sed s/DISTANCE=//`
OSCILLATION_RANGE=`grep OSC_RANGE tmp2 | sed s/OSC_RANGE=//`

elif [ "$DET" == "pilatus" ]; then
DETECTOR="PILATUS MINIMUM_VALID_PIXEL_VALUE=0 OVERLOAD= 1048576 !PILATUS"
QX=0.172 QY=0.172
echo Data from a Pilatus detector
sed s/#// tmp2 > tmp1
mv tmp1 tmp2

# find SENSOR_THICKNESS:
SENSOR_THICKNESS=`grep thickness tmp2 | sed -e s/'Silicon sensor, thickness'// | awk '{print $1*1000}'`
# find X_RAY_WAVELENGTH:
X_RAY_WAVELENGTH=`grep Wavelength tmp2 | sed -e s/Wavelength// -e s/A// | awk '{print $1}'`

X_RAY_ENERGY=`echo "12398.5/$X_RAY_WAVELENGTH" | bc -l | awk '{printf "%5.1f", $1}'`
# find NX and NY; 2463/2527 is 6M, 1475/1679 is 2M
NX=`grep X-Binary-Size-Fastest-Dimension tmp2 | awk '{print $2}'`
NY=`grep X-Binary-Size-Second-Dimension tmp2 | awk '{print $2}'`
NXMM=`echo "$NX*$QX" | bc -l | awk '{printf "%3.2f", $1}'`
NYMM=`echo "$NY*$QY" | bc -l | awk '{printf "%3.2f", $1}'`
# find ORGX and ORGY:
ORGX=`grep Beam_xy tmp2 | sed -e s/$// -e s/$// -e s/\,// | awk '{print $2}'`
ORGY=`grep Beam_xy tmp2 | sed -e s/$// -e s/$// -e s/\,// | awk '{print $3}'`
ORGXMM=`echo "$ORGX*$QX" | bc -l | awk '{printf "%3.2f", $1}'`
ORGYMM=`echo "$ORGY*$QY" | bc -l | awk '{printf "%3.2f", $1}'`
# find DETECTOR_DISTANCE and OSCILLATION_RANGE:
DETECTOR_DISTANCE=`awk '/distance/{print $2}' tmp2`
DETECTOR_DISTANCE=`echo "$DETECTOR_DISTANCE*1000" | bc -l`

OSCILLATION_RANGE=`awk '/Angle/{print $2}' tmp2`

else
echo should never come here
exit 1
fi

echo ORGX= $ORGX ORGY= $ORGY - check these values with adxv !
echo DETECTOR_DISTANCE= $DETECTOR_DISTANCE
echo OSCILLATION_RANGE= $OSCILLATION_RANGE
echo X-RAY_WAVELENGTH= $X_RAY_WAVELENGTH
# now we know everything that is required to generate XDS.INP

cat > XDS.INP << eof
!****************************************************************************************
! Written by generate_XDS.INP version $REVISION
!
! A few notes on usage and modification of this input file:
!
! If XDS fails after IDXREF with an error indicating that
! an insufficient percentage of reflections were indexed,
! but you are confident that the indexing solution is
! correct, rerun XDS with the JOB keyword changed to
! "DEFPIX INTEGRATE CORRECT".
!
! After the first round of integration (in a triclinic cell)
! it is often helpful to reintegrate using the globally
! refined cell parameters and the correct space group.
! You can do this by renaming GXPARM.XDS to XPARM.XDS
! - e.g. by the command "cp GXPARM.XDS XPARM.XDS" -
! and rerunning DEFPIX, INTEGRATE and CORRECT.
!
! If you are trying to determine the optimal starting point for
! data collection starting from a single image, first complete
! a run of XDS using default parameters. Then, rerun XDS using
! the correct cell and spacegroup as described above
! ("cp GXPARM.XDS XPARM.XDS"), but on the second run of XDS, give
! the JOB keyword as "JOB=DEFPIX XPLAN".
!
! Be warned that the starting spindle angles given by XPLAN are
! relative to the first image given - XDS assumes that the first image
! given has a phi of 0 unless explicitly told otherwise. If this is not
! the case (e.g. you are indexing from a snapshot taken at phi=90,
! you will need to add the actual phi (as reported in the image header)
! to the values given by XPLAN in order for these values to make sense.
!****************************************************************************************
!
!****************************************************************************************
!General parameters:
!****************************************************************************************

JOB= XYCORR INIT COLSPOT IDXREF DEFPIX INTEGRATE CORRECT

ORGX= $ORGX ORGY= $ORGY ! check these values with adxv!
!In mm, ORGX=$ORGXMM and ORGY=$ORGYMM

DETECTOR_DISTANCE= $DETECTOR_DISTANCE
OSCILLATION_RANGE= $OSCILLATION_RANGE

X-RAY_WAVELENGTH= $X_RAY_WAVELENGTH
!X-ray energy is $X_RAY_ENERGY eV (calculated from wavelength)

NAME_TEMPLATE_OF_DATA_FRAMES=$NAME_TEMPLATE_OF_DATA_FRAMES

!REFERENCE_DATA_SET=xxx/XDS_ASCII.HKL ! e.g. to ensure consistent indexing.
!XDS will *only* use the reference dataset if the space group and cell
!parameters are specified below.

DATA_RANGE=1 $DATA_RANGE
SPOT_RANGE=1 $SPOT_RANGE

!BACKGROUND_RANGE=1 10 !Default is first five degrees.

SPACE_GROUP_NUMBER=0 ! 0 if unknown
UNIT_CELL_CONSTANTS= 70 80 90 90 90 90 ! put correct values if known

INCLUDE_RESOLUTION_RANGE=50 0 ! After CORRECT, insert high resol limit; re-run CORRECT
! Ice rings can be excluded from scaling with the keyword
! EXCLUDE_RESOLUTION_RANGE, e.g. EXCLUDE_RESOLUTION_RANGE=2.28 2.22

!SECONDS=60 !Uncomment this line to tell XDS to wait 1min for the next image before aborting.

!****************************************************************************************
!Parameters important for processing anomalous data:
!****************************************************************************************

FRIEDEL'S_LAW=FALSE ! This acts only on the CORRECT step. Even if the anomalous signal
! in your dataset is very small, it still may be useful for
! calculating an anomalous difference map.

STRICT_ABSORPTION_CORRECTION=FALSE !Change this to TRUE if the anomalous signal
! is strong: in that case, in CORRECT.LP the three
! "CHI^2-VALUE OF FIT OF CORRECTION FACTORS" values
! will be significantly> 1, e.g. 1.5.

WFAC1=1.0 !This parameter controls rejection of misfits during scaling. Sometimes,
! strong anomalous pairs may be rejected as misfits, in which case increasing
! WFAC1 (e.g. to 1.5) may improve anomalous signal.

!****************************************************************************************
!Regions of the detector to be excluded during indexing and integration:
!****************************************************************************************
! exclude (mask) untrusted areas of detector, e.g. beamstop shadow :
!UNTRUSTED_RECTANGLE= !1800 1950 2100 2150 ! x-min x-max y-min y-max ! repeat
!UNTRUSTED_ELLIPSE= !2034 2070 1850 2240 ! x-min x-max y-min y-max ! if needed
!UNTRUSTED_QUADRILATERAL= !x1 y1 x2 y2 x3 y3 x4 y4 ! see documentation

TRUSTED_REGION=0.00 1.2 ! Partially use corners of detectors; 1.41421=full use, 1.0=edge
! It is important to note that TRUSTED_REGION is the only resolution
! cutoff obeyed during indexing and integration;
! Both INCLUDE_RESOLUTION_RANGE and EXCLUDE_RESOLUTION_RANGE are
! ignored during IDXREF and INTEGRATE. So this parameter can be
! important when indexing or integration steps fail.

!****************************************************************************************
!Other parameters:
!****************************************************************************************
VALUE_RANGE_FOR_TRUSTED_DETECTOR_PIXELS=7000. 30000. ! Often 8000 is ok.
STRONG_PIXEL=4 ! COLSPOT: only use strong reflections (default is 3; 6 may be better for strong data)
MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT=3 ! default of 6 is sometimes too high
MAXIMUM_ERROR_OF_SPOT_POSITION=3 ! Increasing may help for poor quality data

MINPK=75 !Increase (e.g. MINPK=98) to improve data accuracy at the expense of completeness.

DELPHI=5 !Size of integration wedge. Increasing to 10 or 20 may be helpful in some cases.

NUMBER_OF_PROFILE_GRID_POINTS_ALONG_ALPHA/BETA=13 ! Default is 9 - Increasing may improve data
NUMBER_OF_PROFILE_GRID_POINTS_ALONG_GAMMA=13 ! accuracy, particularly if finely-sliced on phi,
! and does not seem to have any downsides.

MINIMUM_ZETA=0.05 ! Controls how close to the blind region (about phi) reflections should be
! integrated. 0.05 is the default; increasing MINIMUM_ZETA to 0.15 *may*
! improve data quality by removing unreliable reflections near the phi
! axis, but will reduce completeness in low symmetry space groups.

SEPMIN=6 ! Reduce if spots are close together (due to a long axis). Consider
CLUSTER_RADIUS=3 ! increasing if crystal is split and unit cell dimensions are relatively short.

! For bad or low resolution data remove the "!" in the following line (default is ALL):
! REFINE(IDXREF)=CELL BEAM ORIENTATION AXIS ! DISTANCE
REFINE(INTEGRATE)=CELL BEAM ORIENTATION ! AXIS DISTANCE (If integration is unstable, comment out this line.
! REFINE(CORRECT)=CELL BEAM ORIENTATION AXIS DISTANCE ! Default is: refine everything

ROTATION_AXIS=1 0 0 !At e.g. Australian Synchrotron, SERCAT ID-22 (?), APS 19-ID (?) this needs to be -1 0 0

!****************************************************************************************
! Parameters specifically for this detector and beamline (shouldn't need changing):
!****************************************************************************************
DETECTOR= $DETECTOR
SENSOR_THICKNESS= $SENSOR_THICKNESS
! attention CCD detectors: for very high resolution (better than 1A) make sure to specify SILICON
! as about 32* what CORRECT.LP suggests (absorption of phosphor is much higher than that of silicon)

NX= $NX NY= $NY QX= $QX QY= $QY ! to make CORRECT happy if frames are unavailable
!In mm, NX=$NXMM and NY=$NYMM

DIRECTION_OF_DETECTOR_X-AXIS=1 0 0
DIRECTION_OF_DETECTOR_Y-AXIS=0 1 0
INCIDENT_BEAM_DIRECTION=0 0 1
FRACTION_OF_POLARIZATION=0.98 ! better value is provided by beamline staff!
POLARIZATION_PLANE_NORMAL=0 1 0
eof
if [ "$DET" == "pilatus" ]; then
if [ $NX == "1475" ]; then
if ! grep -q FF_p2m0109_E12398_T6199_vrf_m0p20.tif tmp2 ; then
cat >> XDS.INP << eof
!EXCLUSION OF VERTICAL DEAD AREAS OF THE PILATUS 2M DETECTOR
UNTRUSTED_RECTANGLE= 486 496 0 1680
UNTRUSTED_RECTANGLE= 980 990 0 1680
!EXCLUSION OF HORIZONTAL DEAD AREAS OF THE PILATUS 2M DETECTOR
UNTRUSTED_RECTANGLE= 0 1476 194 214
UNTRUSTED_RECTANGLE= 0 1476 406 426
UNTRUSTED_RECTANGLE= 0 1476 618 638
UNTRUSTED_RECTANGLE= 0 1476 830 850
UNTRUSTED_RECTANGLE= 0 1476 1042 1062
UNTRUSTED_RECTANGLE= 0 1476 1254 1274
UNTRUSTED_RECTANGLE= 0 1476 1466 1486
eof
fi
# elif [ $NX == "2463" ]; then
# cat >> XDS.INP << eof
#eof
fi
fi
touch xds_to_ccp4.sh
echo "#!/bin/bash
#Removes old log file if present
rm xds_to_ccp4.log
#Makes new log file
touch xds_to_ccp4.log
#Pipe stdout and stderr to both logfile and tty
tail -f -n1 xds_to_ccp4.log &
exec > xds_to_ccp4.log 2>&1
echo \"

If this message is all you see,
something has probably gone wrong.
Are you sure XDS_ASCII.HKL is present
in the current directory?

\"
#Make first MTZ file (with DANO/SIGDANO)
echo \"INPUT_FILE=XDS_ASCII.HKL
OUTPUT_FILE=temp.hkl CCP4
MERGE=TRUE
GENERATE_FRACTION_OF_TEST_REFLECTIONS=0.05
FRIEDEL'S_LAW=FALSE\" > XDSCONV.INP
xdsconv
f2mtz HKLOUT temp.mtz<F2MTZ.INP

#Make second MTZ file (with F(+)/F(-))
echo \"INPUT_FILE=XDS_ASCII.HKL
OUTPUT_FILE=temp2.hkl CCP4_F
MERGE=TRUE
GENERATE_FRACTION_OF_TEST_REFLECTIONS=0.05
FRIEDEL'S_LAW=FALSE\" > XDSCONV.INP
xdsconv
f2mtz HKLOUT temp2.mtz<F2MTZ.INP

#Make SHELX format unmerged hkl file
echo \"INPUT_FILE=XDS_ASCII.HKL
OUTPUT_FILE=xds_shelx.hkl SHELX
MERGE=FALSE
FRIEDEL'S_LAW=FALSE\" > XDSCONV.INP
xdsconv

#cad the two previously generated MTZ files together
cad HKLIN1 temp.mtz HKLIN2 temp2.mtz HKLOUT xds_ccp4_merged.mtz<<EOF
LABIN FILE 1 E1=FP E2=SIGFP E3=DANO E4=SIGDANO E5=FreeRflag
LABIN FILE 2 E1=F(+) E2=SIGF(+) E3=F(-) E4=SIGF(-)
END
EOF

#Remove temporary files
rm temp.mtz
rm temp2.mtz
rm temp.hkl
rm temp2.hkl
rm F2MTZ.INP

echo \"

********************************XDS_TO_CCP4**********************************

This script takes XDS_ASCII.HKL and uses XDSCONV and F2MTZ to
generate a merged CCP4-format MTZ file named
\\\"xds_ccp4_merged.mtz\\\" with a test (free) set
labeled \\\"FreeRflag\\\" constituting 5% of reflections.

The space group of the output MTZ file will be the same as
that present in XDS_ASCII.HKL. It is probably worth checking
that this is the correct space group - either manually inspect
the systematic absences in CORRECT.LP, or run XDS_ASCII.HKL
through the CCP4 program POINTLESS (\\\"pointless xdsin XDS_ASCII.HKL\\\")

Anomalous data is kept, if present, and output as both
DANO/SIGDANO and F(+)/F(-).

This script also generates an unmerged SHELX hkl file,
suitable for input into SHELXC/D/E, named \\\"xds_shelx.hkl\\\".

A log of all processes is present in xds_to_ccp4.log.

Parts of this script have been shamelessly copied from
the XDS Wiki, at http://www.strucbio.biologie.uni-konstanz.de/xdswiki/

******************************************************************************
\"" > xds_to_ccp4.sh
touch xds_graph.sh
echo "#!/bin/bash
#Makes new log file
touch INTEGRATE_STATS.LP
#Pipe stdout and stderr to both logfile and tty
tail -f -n1 INTEGRATE_STATS.LP &
exec > INTEGRATE_STATS.LP 2>&1
if command -v xdsstat >/dev/null; then
echo \"\" | xdsstat > XDSSTAT.LP
echo \"\" | xdsstat 100 6 > XDSSTAT.LP_low
echo \"
\\\$TABLE:Rmeas vs image (to 6A):
\\\$SCATTER:Rmeas vs image for low resolution reflections:A:1,2: \\$\\$
IMAGE RMEAS
\\$\\$ \\$\\$
\"
grep ' L\$' XDSSTAT.LP_low | awk '{print \$1,\$9}'
echo \" \\$\\$ \"
echo \"
\\\$TABLE:Rmeas vs image (all):
\\\$SCATTER:Rmeas vs image (all reflections):A:1,2: \\$\\$
IMAGE RMEAS
\\$\\$ \\$\\$
\"
grep ' L$' XDSSTAT.LP | awk '{print \$1,\$9}'
echo \" \\$\\$ \"
echo \"
\\\$TABLE:Rd vs image (to 6A):
\\\$SCATTER:Rd (from XDSSTAT) for low resolution reflections:A:1,2: \\$\\$
IMAGE RD
\\$\\$ \\$\\$
\"
grep 'DIFFERENCE' XDSSTAT.LP_low | awk '{print \$1,\$7}'
echo \" \\$\\$ \"
echo \"
\\\$TABLE:Rd vs image (all):
\\\$SCATTER:Rd (from XDSSTAT) for all reflections:A:1,2: \\$\\$
IMAGE RD
\\$\\$ \\$\\$
\"
grep 'DIFFERENCE' XDSSTAT.LP | awk '{print \$1,\$7}'
echo \" \\$\\$ \"
fi

echo \"
\\\$TABLE: XDS Integration Statistics :
\\\$SCATTER:Scale factors by image:A:1,3: :Number of overloads by image:A:1,5: :Number of strong reflections by image:A:1,7: :Number of rejects by image:A:1,8: :Mosaicity by image:A:1,10: \\$\\$
IMAGE IER SCALE NBKG NOVL NEWALD NSTRONG NREJ SIGMAB SIGMAR
\\$\\$ \\$\\$
\"
egrep \"[0-9]\.[0-9][0-9][0-9][0-9] [0-9]\.[0-9][0-9][0-9][0-9]\" INTEGRATE.LP
echo \" \\$\\$ \"
echo \"
\\\$TABLE: XDS Scaling Statistics by Resolution (X-axis is resolution in Angstroems) :
\\\$GRAPHS:Chi-Squared by resolution:A:2,4: :Observed and expected R-factors by resolution (as %):A:2,5,6: :Rejects by resolution:A:2,9: :Rejects by resolution as a percentage of total reflections:A:2,10: \\$\\$
LOW_RES Resolution(Angstroems) I/Sigma Chi^2 R-FACTOR(OBSERVED) R-FACTOR(EXPECTED) NUMBER_OF_REFLECTIONS ACCEPTED REJECTS REJECT_PERCENT
\\$\\$ \\$\\$
\"
awk '/RESOLUTION RANGE I\/Sigma Chi\^2 R\-FACTOR R\-FACTOR NUMBER ACCEPTED REJECTED/, /\-\-\-\-\-\-\-\-\-/' CORRECT.LP | egrep -v \"[a-z,A-Z,] | \-\-|\-99\.9\" | egrep \"[0-9]\" | awk '{print \$1,\$2,\$3,\$4,\$5,\$6,\$7,\$8,\$9,(\$9/(\$8+\$9))*100}'
echo \" \\$\\$\"
echo \"
\\\$TABLE: Overall data quality by Resolution (X-axis is resolution in Angstroems) :
\\\$GRAPHS:Completeness vs resolution :A:1,6: :Redundancy vs resolution :A:1,4: :I/Sigma by resolution:A:1,10: :Rmeas by resolution:A:1,11: :Anomalous correlation by resolution:A:1,13: :Significance of the anomalous signal by resolution:A:1,14: \\$\\$
Resolution(Angstroems) OBSERVED_REFLECTIONS UNIQUE_REFLECTIONS REDUNDANCY POSSIBLE_REFLECTIONS COMPLETENESS R-FACTOR(OBSERVED) R-FACTOR(EXPECTED) COMPARED I/SIGMA R-meas CC(1/2) Anomalous_Correlation SigAno Nano
\\$\\$ \\$\\$
\"
egrep -B25 \" WILSON STATISTICS \" CORRECT.LP | egrep -v \"[a-z,A-Z]|\*\*\*\" | egrep \" \" | awk '{ gsub(/[%*]/,\" \"); print }' | awk '{print \$1,\$2,\$3,\$2/\$3,\$4,\$5,\$6,\$7,\$8,\$9,\$10,\$11,\$12,\$13,\$14}'
echo \" \\$\\$ \"

echo \"
\\\$TABLE: I/Sigma of h00 reflections (for identification of systematic absences):
\\\$GRAPHS:I/Sigma vs h :A:1,5: \\$\\$
h k l RES I/SIGMA NUMBER
\\$\\$ \\$\\$
\"
awk '/REFLECTIONS OF TYPE H/,/COMPLETENESS AND QUALITY/' CORRECT.LP | egrep \"[1-9]\" | awk '{print \$1,\$2,\$3,\$4,\$7,\$8}' | awk '(\$1 > 0)'
echo \"\\$\\$\"

echo \"
\\\$TABLE: I/Sigma of 0k0 reflections (for identification of systematic absences):
\\\$GRAPHS:I/Sigma vs k :A:2,5: \\$\\$
h k l RES I/SIGMA NUMBER
\\$\\$ \\$\\$
\"
awk '/REFLECTIONS OF TYPE H/,/COMPLETENESS AND QUALITY/' CORRECT.LP | egrep \"[1-9]\" | awk '{print \$1,\$2,\$3,\$4,\$7,\$8}' | awk '(\$2 > 0)'
echo \"\\$\\$\"

echo \"
\\\$TABLE: I/Sigma of 00l reflections (for identification of systematic absences):
\\\$GRAPHS:I/Sigma vs l :A:3,5: \\$\\$
h k l RES I/SIGMA NUMBER
\\$\\$ \\$\\$
\"
awk '/REFLECTIONS OF TYPE H/,/COMPLETENESS AND QUALITY/' CORRECT.LP | egrep \"[1-9]\" | awk '{print \$1,\$2,\$3,\$4,\$7,\$8}' | awk '(\$3 > 0)'
echo \"\\$\\$\"

echo \"
\\\$TABLE:Beam center (ORGX/ORGY) plotted vs wedge (DELPHI):
\\\$GRAPHS:ORGX and ORGY vs batch (DELPHI) :A:1,2,3: \\$\\$
N ORGX ORGY
\\$\\$ \\$\\$
\"
egrep \"DETECTOR ORIGIN\" INTEGRATE.LP | grep -n \"\" | awk '{ gsub(/[a-z,A-Z,),(,:]/,\"\"); print }'
echo \"\\$\\$\"

echo \"
\\\$TABLE:Unit cell parameters plotted vs wedge:
\\\$GRAPHS:Unit cell lengths a, b and c plotted vs wedge:A:1,2,3,4: :Unit cell angles alpha, beta and gamma plotted vs wedge:A:1,5,6,7: \\$\\$
N a b c ALPHA BETA GAMMA
\\$\\$ \\$\\$
\"
egrep \"UNIT CELL PARAMETERS\" INTEGRATE.LP | grep -n \"\" | awk '{ gsub(/[a-z,A-Z,),(,:]/,\"\"); print }'
echo \"\\$\\$\"

echo \"
\\\$TABLE:Standard deviation of spot position (pixels) plotted vs wedge:
\\\$GRAPHS:Standard deviation of spot position plotted vs wedge:A:1,2: \\$\\$
N SD
\\$\\$ \\$\\$
\"
egrep \"SPOT POSITION\" INTEGRATE.LP | grep -n \"\" | awk '{ gsub(/[a-z,A-Z,),(,:]/,\"\"); print }'
echo \"\\$\\$\"

echo \"
\\\$TABLE:Detector distance (mm) plotted vs wedge (DELPHI):
\\\$GRAPHS:Detector distance plotted vs batch:A:1,2: \\$\\$
N DIST
\\$\\$ \\$\\$
\"
egrep \"DETECTOR DISTANCE \$mm\$\" INTEGRATE.LP | grep -n \"\" | awk '{ gsub(/[a-z,A-Z,),(,:]/,\"\"); print }'

echo \"\\$\\$\"

loggraph INTEGRATE_STATS.LP">xds_graph.sh
chmod +x xds_to_ccp4.sh
chmod +x xds_graph.sh
echo "XDS.INP is ready for use. The file has only the most important keywords.
Full documentation, including complete detector templates, is at
http://www.mpimf-heidelberg.mpg.de/~kabsch/xds . More documentation in XDSwiki
After running xds, inspect, using XDS-Viewer, at least the beamstop mask in
BKGPIX.cbf, and the agreement of predicted and observed spots in FRAME.cbf!

Two shell scripts have been generated that may be of use.

The first, xds_graph.sh, will plot various statistics after an XDS run.
After both INTEGRATE and CORRECT have finished, run it by typing \"./xds_graph.sh\"
without the quotation marks. This script uses loggraph to plot data, so you will
need CCP4 to see the graphical output. All the raw data is piped out to a log file,
INTEGRATE_STATS.LP.

If XDSSTAT is present in your path, xds_graph will also plot some of the statistics it
calculates - specifically the Rmeas and Rd per image, for all reflections and only
low resolution reflections.

The second script, xds_to_ccp4.sh, will take XDS_ASCII.HKL and generate both a CCP4
format MTZ file with anomalous data retained (as F(+)/F(-) and DANO/SigDANO)
and an unmerged SHELX format hkl file for input to SHELXC/D/E. Run it after CORRECT
by typing \"./xds_to_ccp4.sh\" at the prompt."

rm -f tmp1 tmp2
</pre>

Use this script in the same manner as Generate_XDS.INP. The two shell scripts created should be executable; If they are not, then make them so (chmod +x ./xxx.sh).

Generate XDS.INP

2012-10-19T15:08:31Z

Olibclarke:

This script generates XDS.INP based on a list of frame names supplied on the commandline. It currently works for MarCCD, ADSC and Pilatus detectors; since this is just a bash script, extension to other detectors is very easy.

On Mac OS X, installation of the "Command Line Tools" (from http://developer.apple.com/downloads; requires Apple ID) is required (I think). These are a part of the (larger, but also free) [http://developer.apple.com/tools/xcode Xcode] package.

== Usage ==

Usage is just (don't forget the quotation marks!):
generate_XDS.INP "frms/mydata_1_???.img"
XDS [http://strucbio.biologie.uni-konstanz.de/~dikay/XDS_html_doc/html_doc/xds_parameters.html#NAME_TEMPLATE_OF_DATA_FRAMES= supports] bzip2-ed frames. Thus, when specifying the frame name parameter of the script, you should leave out any .bz2 extension.

== The script ==

<pre>
#!/bin/bash
# purpose: generate XDS.INP
#
# tested with some datasets from ALS, SSRL, SLS, ESRF and BESSY; only MARCCD, ADSC/SMV, PILATUS detectors;
# for other detectors, values marked with XXX must be manually filled in.
#
# revision 0.03 . Kay Diederichs 2/2010
# revision 0.04 . Kay Diederichs 4/2010 - include alternative ORGX, ORGY calculations for ADSC
# revision 0.05 . Kay Diederichs 5/2010 - grep for "Corrected" in addition to "marccd"; needed for BESSY
# revision 0.06 . KD 6/2010 - add UNTRUSTED_RECTANGLE and UNTRUSTED_ELLIPSE; use `whereis catmar` and so on
# revision 0.07 . KD 6/2010 - decide about ORGX/Y info in MAR header being pixels or mm; other fixes
# revision 0.08 . KD 6/2010 - fixes for Pilatus 6M
# revision 0.09 . KD 6/2010 - get rid of requirement for mccd_xdsparams.pl and/or catmar; rather use "od"
# revision 0.10 . Tim Gruene 7/2010 - set link 'images' to image directory if path exceeds 72 characters
# revision 0.11 . KD 7/2010 - for MarCCD: look for distance info at different byte position
# revision 0.12 . KD 7/2010 - fix for negative PHISTART
# revision 0.13 . KD 8/2010 - store correct NX NY QX QY in XDS.INP
# revision 0.14 . KD 1/2011 - SENSOR_THICKNESS for Pilatus; MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT=3
# revision 0.15 . KD 2/2011 - add comment for -ive sign of APS 19-ID and Australian Synchrotron rotation axis
# revision 0.16 . KD 3/2011 - SENSOR_THICKNESS=0.01 for ADSC and MarCCD. Add comment about SILICON=
# revision 0.17 . KD 3/2011 - make it work for .bz2 frames; improve screen output
# revision 0.18 . KD 4/2011 - faster by doing "strings" only once; revert "images/${1##/*/}" "correction"
# revision 0.19 . KD 6/2011 - bugfix for 0.18
# revision 0.20 . KD 7/2011 - redirect stderr of /bin/ls to /dev/null
# revision 0.21 . KD 11/2011 - SEPMIN, CLUSTER_RADIUS hints; read NX NY from header (for Pilatus 2M)
# revision 0.22 . KD 12/2011 - Pilatus 2M UNTRUSTED_RECTANGLE lines, SENSOR_THICKNESS from header
# revision 0.23 . KD 1/2012 - add UNTRUSTED_QUADRILATERAL, remove MINIMUM_ZETA (0.05 is default now)
# revision 0.24 . KD 3/2012 - remove revision 0.10 since XDS now takes much longer paths
# revision 0.25 . KD 3/2012 - remove revision 0.22 for PSI Pilatus 2M; see http://www.globalphasing.com/autoproc/wiki/index.cgi?TroubleShootingKnownIssues
# revision 0.26 . KD 7/2012 - Mac-compatibility: replace od flags --skip-bytes= and --read-bytes= with -j and -N (thanks to Oliver Clarke for working this out!)
REVISION="0.26 (02-Jul-2012)"
#
# usage: e.g. generate_XDS.INP "frms/mydata_1_???.img"
# make sure to have the two quotation marks !
# the ? are wildcards for the frame numbers.
#
# limitations:
# - frame numbers are assumed to start with 1 and run consecutively
#
# known problems:
# - for ADSC detectors, there are at least three ways to obtain ORGX and ORGY values from the header (see below);
# - the same might be a problem for MAR headers, too (not sure about this)
# - on Mac OS X, Xcode (from http://developer.apple.com/tools/xcode) might be needed - not sure about this
#
# notes for debugging of the script:
# - add the -v option to the first line, to see where an error occurs
# - comment out the removal of tmp1 and tmp2 in the last line
#
# ====== Start of script ======
echo generate_XDS.INP version $REVISION . Obtain the latest version from
echo http://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/generate_XDS.INP
if [ "$1" == "help" ] || [ "$1" == "-help" ] || [ "$1" == "-h" ]; then
echo usage: generate_XDS.INP \"frms/mydata_1_???.img\" $_with_ the quotation marks!$
echo if the frames are compressed with bzip2, leave out the .bz2 extension!
exit
fi
#
# defaults:
#
DETECTOR="XXX MINIMUM_VALID_PIXEL_VALUE=XXX OVERLOAD=XXX"
ORGX=XXX
ORGY=XXX
DETECTOR_DISTANCE=XXX
OSCILLATION_RANGE=XXX
X_RAY_WAVELENGTH=XXX
QX=XXX
QY=XXX
NX=XXX
NY=XXX
SENSOR_THICKNESS=0
# see how we are called:
NAME_TEMPLATE_OF_DATA_FRAMES="$1"
# list frames matching the wildcards in NAME_TEMPLATE_OF_DATA_FRAMES
# don't accept the "direct beam" shot at SLS/Pilatus PX-I and PX-II
/bin/ls -C1 $1 $1.bz2 2>/dev/null | egrep -v "_00000.cbf|_000.img" > tmp1 || exit 1

# we can continue - the frames are found

# set upper limit of DATA_RANGE to number of frames (see "limitations" above)
DATA_RANGE=`wc -l tmp1 | awk '{print $1}'`

# set upper limit of SPOT_RANGE to half of DATA_RANGE, but not less than 1
SPOT_RANGE=`echo "scale=0; $DATA_RANGE/2" | bc -l`
SPOT_RANGE=`echo "if ($SPOT_RANGE<1) 1;if ($SPOT_RANGE>1) $SPOT_RANGE" | bc -l`

echo DATA_RANGE=1 $DATA_RANGE

# find out detector type
DET=XXX
FIRSTFRAME=`head -1 tmp1`
echo $FIRSTFRAME | grep -q bz2 && bzcat $FIRSTFRAME > tmp1 && FIRSTFRAME=tmp1
strings $FIRSTFRAME > tmp2
egrep -q 'marccd|Corrected' tmp2 && DET=mccd
grep -q PILATUS tmp2 && DET=pilatus
grep -q BEAM_CENTER_X tmp2 && DET=adsc
# identify other detector types in the same way (MAR IP would be straightforward)

# parse ASCII header of first frame

if [ "$DET" == "XXX" ]; then
echo "this is not a MAR, ADSC/SMV or PILATUS detector - fill in XXX values manually!"
DETECTOR="XXX MINIMUM_VALID_PIXEL_VALUE=XXX OVERLOAD=XXX"

# find parameters of first frame
elif [ "$DET" == "mccd" ]; then
echo Data from a MarCCD detector

DETECTOR="CCDCHESS MINIMUM_VALID_PIXEL_VALUE= 1 OVERLOAD= 65500"
SENSOR_THICKNESS=0.01
# use first frame of dataset to obtain parameters

# offsets are documented; values can be found in mccd_xdsparams.pl script
let SKIP=1024+80
NX=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
let SKIP=$SKIP+4
NY=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')

let SKIP=1720
DETECTOR_DISTANCE=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
DETECTOR_DISTANCE=`echo "scale=3; $DETECTOR_DISTANCE/1000" | bc -l`

let SKIP=1024+256+128+256+4
ORGX=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
ORGX=`echo "scale=2; $ORGX/1000" | bc -l `
let SKIP=$SKIP+4
ORGY=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
ORGY=`echo "scale=2; $ORGY/1000" | bc -l `

let SKIP=1024+256+128+256+44
PHISTART=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
let SKIP=1024+256+128+256+76
PHIEND=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
OSCILLATION_RANGE=`echo "scale=3; ($PHIEND-($PHISTART))/1000" | bc -l`

let SKIP=1024+256+128+256+128+4
QX=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
QX=`echo "scale=10; $QX/1000000" |bc -l `
let SKIP=$SKIP+4
QY=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
QY=`echo "scale=10; $QY/1000000" |bc -l `

let SKIP=1024+256+128+256+128+128+12
X_RAY_WAVELENGTH=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
X_RAY_WAVELENGTH=`echo "scale=5; $X_RAY_WAVELENGTH/100000" | bc -l`

# at most BLs, ORGX and ORGY are in pixels, but sometimes in mm ... guess:
NXBYFOUR=`echo "scale=0; $NX/4" | bc -l `
ORGXINT=`echo "scale=0; $ORGX/1" | bc -l `
if [ $ORGXINT -lt $NXBYFOUR ]; then
ORGX=`echo "scale=1; $ORGX/$QX" | bc -l`
ORGY=`echo "scale=1; $ORGY/$QY" | bc -l`
echo MARCCD detector: header ORGX, ORGY seem to be in mm ... converting to pixels
else
echo MARCCD detector: header ORGX, ORGY seem to be in pixel units
fi

elif [ "$DET" == "adsc" ]; then

DETECTOR="ADSC MINIMUM_VALID_PIXEL_VALUE= 1 OVERLOAD= 65000"
echo Data from ADSC detector. Obtaining ORGX, ORGY depends on beamline setup:
SENSOR_THICKNESS=0.01
sed s/\;// tmp2 > tmp1
mv tmp1 tmp2

# find X_RAY_WAVELENGTH:
X_RAY_WAVELENGTH=`grep WAVELENGTH tmp2 | head -1 | sed s/WAVELENGTH=//`

# find NX, QX, ORGX and ORGY:
NX=`grep SIZE1 tmp2 | tail -1 | sed s/SIZE1=//`
QX=`grep PIXEL_SIZE tmp2 | sed s/PIXEL_SIZE=//`
# FIXME - next 2 lines should be done properly, from header
NY=$NX
QY=$QX
BEAM_CENTER_X=`grep BEAM_CENTER_X tmp2 | sed s/BEAM_CENTER_X=//`
BEAM_CENTER_Y=`grep BEAM_CENTER_Y tmp2 | sed s/BEAM_CENTER_Y=//`
# fix 2010-04-26 - tell user about possible ORGX, ORGY alternatives -
# at ESRF and ... (pls fill in!) the following should be used:
ORGX=`echo "scale=1; $BEAM_CENTER_Y/$QX" | bc -l`
ORGY=`echo "scale=1; $BEAM_CENTER_X/$QX" | bc -l`
echo - at ESRF BLs use: ORGX=$ORGX ORGY=$ORGY
# this 2nd alternative convention should be used at the following beamlines (pls complete the list): ALS 5.0.3, ...
ORGX=`echo "scale=1; $NX-$BEAM_CENTER_X/$QX" | bc -l `
ORGY=`echo "scale=1; $BEAM_CENTER_Y/$QX" | bc -l `
echo - at e.g. ALS 5.0.3 use: ORGX=$ORGX ORGY=$ORGY
# this 3rd alternative convention should be used at the following beamlines (pls complete the list): ALS 8.2.2, ...
ORGX=`echo "scale=1; $BEAM_CENTER_X/$QX" | bc -l `
ORGY=`echo "scale=1; $NX-$BEAM_CENTER_Y/$QX" | bc -l `
echo - at e.g. ALS 8.2.2 use: ORGX=$ORGX ORGY=$ORGY - this is written to XDS.INP
# the latter alternative is written into the generated XDS.INP ! You have to correct this manually in XDS.INP, or adjust this script.
# find DETECTOR_DISTANCE and OSCILLATION_RANGE:
DETECTOR_DISTANCE=`grep DISTANCE tmp2 | sed s/DISTANCE=//`
OSCILLATION_RANGE=`grep OSC_RANGE tmp2 | sed s/OSC_RANGE=//`

elif [ "$DET" == "pilatus" ]; then
DETECTOR="PILATUS MINIMUM_VALID_PIXEL_VALUE=0 OVERLOAD= 1048576 !PILATUS"
QX=0.172 QY=0.172
echo Data from a Pilatus detector
sed s/#// tmp2 > tmp1
mv tmp1 tmp2

# find SENSOR_THICKNESS:
SENSOR_THICKNESS=`grep thickness tmp2 | sed -e s/'Silicon sensor, thickness'// | awk '{print $1*1000}'`
# find X_RAY_WAVELENGTH:
X_RAY_WAVELENGTH=`grep Wavelength tmp2 | sed -e s/Wavelength// -e s/A// | awk '{print $1}'`

# find NX and NY; 2463/2527 is 6M, 1475/1679 is 2M
NX=`grep X-Binary-Size-Fastest-Dimension tmp2 | awk '{print $2}'`
NY=`grep X-Binary-Size-Second-Dimension tmp2 | awk '{print $2}'`

# find ORGX and ORGY:
ORGX=`grep Beam_xy tmp2 | sed -e s/$// -e s/$// -e s/\,// | awk '{print $2}'`
ORGY=`grep Beam_xy tmp2 | sed -e s/$// -e s/$// -e s/\,// | awk '{print $3}'`

# find DETECTOR_DISTANCE and OSCILLATION_RANGE:
DETECTOR_DISTANCE=`awk '/distance/{print $2}' tmp2`
DETECTOR_DISTANCE=`echo "$DETECTOR_DISTANCE*1000" | bc -l`

OSCILLATION_RANGE=`awk '/Angle/{print $2}' tmp2`

else
echo should never come here
exit 1
fi

echo ORGX= $ORGX ORGY= $ORGY - check these values with adxv !
echo DETECTOR_DISTANCE= $DETECTOR_DISTANCE
echo OSCILLATION_RANGE= $OSCILLATION_RANGE
echo X-RAY_WAVELENGTH= $X_RAY_WAVELENGTH

# now we know everything that is required to generate XDS.INP

cat > XDS.INP << eof
! written by generate_XDS.INP version $REVISION
JOB= XYCORR INIT COLSPOT IDXREF DEFPIX INTEGRATE CORRECT
ORGX= $ORGX ORGY= $ORGY ! check these values with adxv !
DETECTOR_DISTANCE= $DETECTOR_DISTANCE
OSCILLATION_RANGE= $OSCILLATION_RANGE
X-RAY_WAVELENGTH= $X_RAY_WAVELENGTH
NAME_TEMPLATE_OF_DATA_FRAMES=$NAME_TEMPLATE_OF_DATA_FRAMES
! REFERENCE_DATA_SET=xxx/XDS_ASCII.HKL ! e.g. to ensure consistent indexing
DATA_RANGE=1 $DATA_RANGE
SPOT_RANGE=1 $SPOT_RANGE
! BACKGROUND_RANGE=1 10 ! rather use defaults (first 5 degree of rotation)

SPACE_GROUP_NUMBER=0 ! 0 if unknown
UNIT_CELL_CONSTANTS= 70 80 90 90 90 90 ! put correct values if known
INCLUDE_RESOLUTION_RANGE=50 0 ! after CORRECT, insert high resol limit; re-run CORRECT

FRIEDEL'S_LAW=FALSE ! This acts only on the CORRECT step
! If the anom signal turns out to be, or is known to be, very low or absent,
! use FRIEDEL'S_LAW=TRUE instead (or comment out the line); re-run CORRECT

! remove the "!" in the following line:
! STRICT_ABSORPTION_CORRECTION=TRUE
! if the anomalous signal is strong: in that case, in CORRECT.LP the three
! "CHI^2-VALUE OF FIT OF CORRECTION FACTORS" values are significantly> 1, e.g. 1.5
!
! exclude (mask) untrusted areas of detector, e.g. beamstop shadow :
! UNTRUSTED_RECTANGLE= 1800 1950 2100 2150 ! x-min x-max y-min y-max ! repeat
! UNTRUSTED_ELLIPSE= 2034 2070 1850 2240 ! x-min x-max y-min y-max ! if needed
! UNTRUSTED_QUADRILATERAL= x1 y1 x2 y2 x3 y3 x4 y4 ! see documentation
!
! parameters with changes wrt default values:
TRUSTED_REGION=0.00 1.2 ! partially use corners of detectors; 1.41421=full use
VALUE_RANGE_FOR_TRUSTED_DETECTOR_PIXELS=7000. 30000. ! often 8000 is ok
STRONG_PIXEL=4 ! COLSPOT: only use strong reflections (default is 3)
MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT=3 ! default of 6 is sometimes too high
! close spots: reduce SEPMIN and CLUSTER_RADIUS from their defaults of 6 and 3, e.g. to 4 and 2
! for bad or low resolution data remove the "!" in the following line:
! REFINE(IDXREF)=CELL BEAM ORIENTATION AXIS ! DISTANCE
REFINE(INTEGRATE)=CELL BEAM ORIENTATION ! AXIS DISTANCE
! REFINE(CORRECT)=CELL BEAM ORIENTATION AXIS DISTANCE ! Default is: refine everything

! parameters specifically for this detector and beamline:
DETECTOR= $DETECTOR
SENSOR_THICKNESS= $SENSOR_THICKNESS
! attention CCD detectors: for very high resolution (better than 1A) make sure to specify SILICON
! as about 32* what CORRECT.LP suggests (absorption of phosphor is much higher than that of silicon)
NX= $NX NY= $NY QX= $QX QY= $QY ! to make CORRECT happy if frames are unavailable
DIRECTION_OF_DETECTOR_X-AXIS=1 0 0
DIRECTION_OF_DETECTOR_Y-AXIS=0 1 0
INCIDENT_BEAM_DIRECTION=0 0 1
ROTATION_AXIS=1 0 0 ! at e.g. Australian Synchrotron, SERCAT ID-22 (?), APS 19-ID (?) this needs to be -1 0 0
FRACTION_OF_POLARIZATION=0.98 ! better value is provided by beamline staff!
POLARIZATION_PLANE_NORMAL=0 1 0
eof
if [ "$DET" == "pilatus" ]; then
if [ $NX == "1475" ]; then
if ! grep -q FF_p2m0109_E12398_T6199_vrf_m0p20.tif tmp2 ; then
cat >> XDS.INP << eof
!EXCLUSION OF VERTICAL DEAD AREAS OF THE PILATUS 2M DETECTOR
UNTRUSTED_RECTANGLE= 486 496 0 1680
UNTRUSTED_RECTANGLE= 980 990 0 1680
!EXCLUSION OF HORIZONTAL DEAD AREAS OF THE PILATUS 2M DETECTOR
UNTRUSTED_RECTANGLE= 0 1476 194 214
UNTRUSTED_RECTANGLE= 0 1476 406 426
UNTRUSTED_RECTANGLE= 0 1476 618 638
UNTRUSTED_RECTANGLE= 0 1476 830 850
UNTRUSTED_RECTANGLE= 0 1476 1042 1062
UNTRUSTED_RECTANGLE= 0 1476 1254 1274
UNTRUSTED_RECTANGLE= 0 1476 1466 1486
eof
fi
# elif [ $NX == "2463" ]; then
# cat >> XDS.INP << eof
#eof
fi
fi
echo XDS.INP is ready for use. The file has only the most important keywords.
echo Full documentation, including complete detector templates, is at
echo http://www.mpimf-heidelberg.mpg.de/~kabsch/xds . More documentation in XDSwiki
echo After running xds, inspect, using XDS-Viewer, at least the beamstop mask in
echo BKGPIX.cbf, and the agreement of predicted and observed spots in FRAME.cbf!
rm -f tmp1 tmp2
</pre>

== System-wide or personal installation ==

Ask your system adminstrator to cut-and-paste the script into e.g. /usr/local/bin/generate_XDS.INP, and to make it "executable".

But you may also cut-and-paste the script from this webpage into a file in e.g. your home directory; the filename should be generate_XDS.INP. After creating the file, make it executable - e.g. if it's in your $HOME, use:
chmod +x ~/generate_XDS.INP
After that, you can just run it in a similar way as if it were installed in your $PATH:
~/generate_XDS.INP "frms/mydata_1_???.img"
By using your own file, you can easily update to the latest revision, or even change the script, without having to bother the system administrator.

== Generating generate_XDS.INP from this webpage ==

Instead of cutting-and-pasting the lines of the script, you (or the system administrator) could just cut-and-paste the following four lines
<pre>
wget http://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/generate_XDS.INP -O - | \
sed -e s/\&nbsp\;/\ /g -e s/\&gt\;/\>/g -e s/\&lt\;/\</g -e s/amp\;//g -e s/\&quot\;/\"/g -e s/\&\#\1\6\0\;/\ /g | \
awk '/^#/,/rm -f tmp1 tmp2/' > generate_XDS.INP
chmod +x generate_XDS.INP
</pre>
to copy the script from this website into an executable file generate_XDS.INP in your current directory.

==Variant script (xds_generate_all)==
This script includes some minor modifications of the Generate_XDS.INP script - for instance, it tells the user the energy, as well as the wavelength of the X-rays, and gives the user the beam center in mm as well as pixels - values that may be useful when switching between HKL2000, MOSFLM and XDS.

It also generates two executable shell scripts in the same directory as XDS.INP.

The first of these scripts, xds_graph.sh, extracts various statistics from INTEGRATE.LP and CORRECT.LP, pipes them out to a logfile, INTEGRATE_STATS.LP, and plots them using loggraph. This script does not have some of the more useful features of XDSSTAT, such as calculation of per frame Rmeas values, but it does give a quick overview of various data quality parameters by resolution and image number. This script will only work with recent versions of XDS (after calculation of CC(1/2) - not sure exactly which version).

The second of these scripts, xds_to_ccp4.sh, will take XDS_ASCII.HKL and generate a CCP4-format MTZ file, containing anomalous data (as F(+)/F(-) and DANO/SigDANO) and a test set of 5% of reflections for calculation of the free R-factor. It will also generate an unmerged SHELX format HKL file, for input into SHELXC/D/E (e.g. via the HKL2MAP GUI).

Both these shell scripts can by installed system wide by transferring them to a directory in your path, for example the directory containing your XDS binaries.

Here is the script:
<pre>
#!/bin/bash
# purpose: xds_generate_all
#
# tested with some datasets from ALS, SSRL, SLS, ESRF and BESSY; only MARCCD, ADSC/SMV, PILATUS detectors;
# for other detectors, values marked with XXX must be manually filled in.
#
# revision 0.03 . Kay Diederichs 2/2010
# revision 0.04 . Kay Diederichs 4/2010 - include alternative ORGX, ORGY calculations for ADSC
# revision 0.05 . Kay Diederichs 5/2010 - grep for "Corrected" in addition to "marccd"; needed for BESSY
# revision 0.06 . KD 6/2010 - add UNTRUSTED_RECTANGLE and UNTRUSTED_ELLIPSE; use `whereis catmar` and so on
# revision 0.07 . KD 6/2010 - decide about ORGX/Y info in MAR header being pixels or mm; other fixes
# revision 0.08 . KD 6/2010 - fixes for Pilatus 6M
# revision 0.09 . KD 6/2010 - get rid of requirement for mccd_xdsparams.pl and/or catmar; rather use "od"
# revision 0.10 . Tim Gruene 7/2010 - set link 'images' to image directory if path exceeds 72 characters
# revision 0.11 . KD 7/2010 - for MarCCD: look for distance info at different byte position
# revision 0.12 . KD 7/2010 - fix for negative PHISTART
# revision 0.13 . KD 8/2010 - store correct NX NY QX QY in XDS.INP
# revision 0.14 . KD 1/2011 - SENSOR_THICKNESS for Pilatus; MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT=3
# revision 0.15 . KD 2/2011 - add comment for -ive sign of APS 19-ID and Australian Synchrotron rotation axis
# revision 0.16 . KD 3/2011 - SENSOR_THICKNESS=0.01 for ADSC and MarCCD. Add comment about SILICON=
# revision 0.17 . KD 3/2011 - make it work for .bz2 frames; improve screen output
# revision 0.18 . KD 4/2011 - faster by doing "strings" only once; revert "images/${1##/*/}" "correction"
# revision 0.19 . KD 6/2011 - bugfix for 0.18
# revision 0.20 . KD 7/2011 - redirect stderr of /bin/ls to /dev/null
# revision 0.21 . KD 11/2011 - SEPMIN, CLUSTER_RADIUS hints; read NX NY from header (for Pilatus 2M)
# revision 0.22 . KD 12/2011 - Pilatus 2M UNTRUSTED_RECTANGLE lines, SENSOR_THICKNESS from header
# revision 0.23 . KD 1/2012 - add UNTRUSTED_QUADRILATERAL, remove MINIMUM_ZETA (0.05 is default now)
# revision 0.24 . KD 3/2012 - remove revision 0.10 since XDS now takes much longer paths
# revision 0.25 . KD 3/2012 - remove revision 0.22 for PSI Pilatus 2M; see http://www.globalphasing.com/autoproc/wiki/index.cgi?TroubleShootingKnownIssues
# revision 0.26 Oliver Clarke 10/12 - Added generation of shell scripts for conversion to MTZ and visualisation of statistics. Other minor alterations to XDS.INP.
REVISION="0.26 (18-Oct-2012)"
#
# usage: e.g. xds_generate_all "frms/mydata_1_???.img"
# make sure to have the two quotation marks !
# the ? are wildcards for the frame numbers.
#
# limitations:
# - frame numbers are assumed to start with 1 and run consecutively
#
# known problems:
# - for ADSC detectors, there are at least three ways to obtain ORGX and ORGY values from the header (see below);
# - the same might be a problem for MAR headers, too (not sure about this)
#
# notes for debugging of the script:
# - add the -v option to the first line, to see where an error occurs
# - comment out the removal of tmp1 and tmp2 in the last line
#
# ====== Start of script ======
echo generate_XDS.INP version $REVISION . Obtain the latest version from
echo http://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/generate_XDS.INP
if [ "$1" == "help" ] || [ "$1" == "-help" ] || [ "$1" == "-h" ]; then
echo usage: xds_generate_all \"frms/mydata_1_???.img\" $_with_ the quotation marks!$
echo if the frames are compressed with bzip2, leave out the .bz2 extension!
exit
fi
#
# defaults:
#
DETECTOR="XXX MINIMUM_VALID_PIXEL_VALUE=XXX OVERLOAD=XXX"
ORGX=XXX
ORGY=XXX
ORGXMM=XXX
ORGYMM=XXX
DETECTOR_DISTANCE=XXX
OSCILLATION_RANGE=XXX
X_RAY_WAVELENGTH=XXX
X_RAY_ENERGY=XXX
QX=XXX
QY=XXX
NX=XXX
NY=XXX
NXMM=XXX
NYMM=XXX
SENSOR_THICKNESS=0
# see how we are called:
NAME_TEMPLATE_OF_DATA_FRAMES="$1"
# list frames matching the wildcards in NAME_TEMPLATE_OF_DATA_FRAMES
# don't accept the "direct beam" shot at SLS/Pilatus PX-I and PX-II
/bin/ls -C1 $1 $1.bz2 2>/dev/null | egrep -v "_00000.cbf|_000.img" > tmp1 || exit 1

# we can continue - the frames are found

# set upper limit of DATA_RANGE to number of frames (see "limitations" above)
DATA_RANGE=`wc -l tmp1 | awk '{print $1}'`

# set upper limit of SPOT_RANGE to half of DATA_RANGE, but not less than 1
SPOT_RANGE=`echo "scale=0; $DATA_RANGE/2" | bc -l`
SPOT_RANGE=`echo "if ($SPOT_RANGE<1) 1;if ($SPOT_RANGE>1) $SPOT_RANGE" | bc -l`

echo DATA_RANGE=1 $DATA_RANGE

# find out detector type
DET=XXX
FIRSTFRAME=`head -1 tmp1`
echo $FIRSTFRAME | grep -q bz2 && bzcat $FIRSTFRAME > tmp1 && FIRSTFRAME=tmp1
strings $FIRSTFRAME > tmp2
egrep -q 'marccd|Corrected' tmp2 && DET=mccd
grep -q PILATUS tmp2 && DET=pilatus
grep -q BEAM_CENTER_X tmp2 && DET=adsc
# identify other detector types in the same way (MAR IP would be straightforward)

# parse ASCII header of first frame

if [ "$DET" == "XXX" ]; then
echo "this is not a MAR, ADSC/SMV or PILATUS detector - fill in XXX values manually!"
DETECTOR="XXX MINIMUM_VALID_PIXEL_VALUE=XXX OVERLOAD=XXX"

# find parameters of first frame
elif [ "$DET" == "mccd" ]; then
echo Data from a MarCCD detector

DETECTOR="CCDCHESS MINIMUM_VALID_PIXEL_VALUE= 1 OVERLOAD= 65500"
SENSOR_THICKNESS=0.01
# use first frame of dataset to obtain parameters

# offsets are documented; values can be find in mccd_xdsparams.pl script
let SKIP=1024+80
NX=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
let SKIP=$SKIP+4
NY=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')

let SKIP=1720
DETECTOR_DISTANCE=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
DETECTOR_DISTANCE=`echo "scale=3; $DETECTOR_DISTANCE/1000" | bc -l`

let SKIP=1024+256+128+256+4
ORGX=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
ORGX=`echo "scale=2; $ORGX/1000" | bc -l `
let SKIP=$SKIP+4
ORGY=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
ORGY=`echo "scale=2; $ORGY/1000" | bc -l `

let SKIP=1024+256+128+256+44
PHISTART=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
let SKIP=1024+256+128+256+76
PHIEND=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
OSCILLATION_RANGE=`echo "scale=3; ($PHIEND-($PHISTART))/1000" | bc -l`

let SKIP=1024+256+128+256+128+4
QX=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
QX=`echo "scale=10; $QX/1000000" |bc -l `
let SKIP=$SKIP+4
QY=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
QY=`echo "scale=10; $QY/1000000" |bc -l `

let SKIP=1024+256+128+256+128+128+12
X_RAY_WAVELENGTH=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
X_RAY_WAVELENGTH=`echo "scale=5; $X_RAY_WAVELENGTH/100000" | bc -l`
X_RAY_ENERGY=`echo "12398.5/$X_RAY_WAVELENGTH" | bc -l | awk '{printf "%5.1f", $1}'`
# at most BLs, ORGX and ORGY are in pixels, but sometimes in mm ... guess:
NXBYFOUR=`echo "scale=0; $NX/4" | bc -l `
ORGXINT=`echo "scale=0; $ORGX/1" | bc -l `
if [ $ORGXINT -lt $NXBYFOUR ]; then
ORGXMM=$ORGX
ORGYMM=$ORGY
NXMM=`echo "$NX*$QX" | bc -l | awk '{printf "%3.2f", $1}'`
NYMM=`echo "$NY*$QY" | bc -l | awk '{printf "%3.2f", $1}'`
ORGX=`echo "scale=1; $ORGX/$QX" | bc -l`
ORGY=`echo "scale=1; $ORGY/$QY" | bc -l`
echo MARCCD detector: header ORGX, ORGY seem to be in mm ... converting to pixels
else
ORGXMM=`echo "$ORGX*$QX" | bc -l | awk '{printf "%3.2f", $1}'`
ORGYMM=`echo "$ORGY*$QY" | bc -l | awk '{printf "%3.2f", $1}'`
NXMM=`echo "$NX*$QX" | bc -l | awk '{printf "%3.2f", $1}'`
NYMM=`echo "$NY*$QY" | bc -l | awk '{printf "%3.2f", $1}'`
echo MARCCD detector: header ORGX, ORGY seem to be in pixel units
fi

elif [ "$DET" == "adsc" ]; then

DETECTOR="ADSC MINIMUM_VALID_PIXEL_VALUE= 1 OVERLOAD= 65000"
echo Data from ADSC detector. Obtaining ORGX, ORGY depends on beamline setup:
SENSOR_THICKNESS=0.01
sed s/\;// tmp2 > tmp1
mv tmp1 tmp2

# find X_RAY_WAVELENGTH:
X_RAY_WAVELENGTH=`grep WAVELENGTH tmp2 | head -1 | sed s/WAVELENGTH=//`

# find NX, QX, ORGX and ORGY:
X_RAY_ENERGY=`echo "12398.5/$X_RAY_WAVELENGTH" | bc -l | awk '{printf "%5.1f", $1}'`
NX=`grep SIZE1 tmp2 | tail -1 | sed s/SIZE1=//`
QX=`grep PIXEL_SIZE tmp2 | sed s/PIXEL_SIZE=//`
# FIXME - next 2 lines should be done properly, from header
NY=$NX
QY=$QX
NXMM=`echo "$NX*$QX" | bc -l | awk '{printf "%3.2f", $1}'`
NYMM=`echo "$NY*$QY" | bc -l | awk '{printf "%3.2f", $1}'`
BEAM_CENTER_X=`grep BEAM_CENTER_X tmp2 | sed s/BEAM_CENTER_X=//`
BEAM_CENTER_Y=`grep BEAM_CENTER_Y tmp2 | sed s/BEAM_CENTER_Y=//`
# fix 2010-04-26 - tell user about possible ORGX, ORGY alternatives -
# at ESRF and ... (pls fill in!) the following should be used:
ORGX=`echo "scale=1; $BEAM_CENTER_Y/$QX" | bc -l`
ORGY=`echo "scale=1; $BEAM_CENTER_X/$QX" | bc -l`
echo - at ESRF BLs use: ORGX=$ORGX ORGY=$ORGY
# this 2nd alternative convention should be used at the following beamlines (pls complete the list): ALS 5.0.3, ...
ORGX=`echo "scale=1; $NX-$BEAM_CENTER_X/$QX" | bc -l `
ORGY=`echo "scale=1; $BEAM_CENTER_Y/$QX" | bc -l `
echo - at e.g. ALS 5.0.3 use: ORGX=$ORGX ORGY=$ORGY
# this 3rd alternative convention should be used at the following beamlines (pls complete the list): ALS 8.2.2, ...
ORGX=`echo "scale=1; $BEAM_CENTER_X/$QX" | bc -l `
ORGY=`echo "scale=1; $NX-$BEAM_CENTER_Y/$QX" | bc -l `
ORGXMM=`echo "$ORGX*$QX" | bc -l | awk '{printf "%3.2f", $1}'`
ORGYMM=`echo "$ORGY*$QY" | bc -l | awk '{printf "%3.2f", $1}'`
echo - at e.g. ALS 8.2.2 use: ORGX=$ORGX ORGY=$ORGY - this is written to XDS.INP
# the latter alternative is written into the generated XDS.INP ! You have to correct this manually in XDS.INP, or adjust this script.
# find DETECTOR_DISTANCE and OSCILLATION_RANGE:
DETECTOR_DISTANCE=`grep DISTANCE tmp2 | sed s/DISTANCE=//`
OSCILLATION_RANGE=`grep OSC_RANGE tmp2 | sed s/OSC_RANGE=//`

elif [ "$DET" == "pilatus" ]; then
DETECTOR="PILATUS MINIMUM_VALID_PIXEL_VALUE=0 OVERLOAD= 1048576 !PILATUS"
QX=0.172 QY=0.172
echo Data from a Pilatus detector
sed s/#// tmp2 > tmp1
mv tmp1 tmp2

# find SENSOR_THICKNESS:
SENSOR_THICKNESS=`grep thickness tmp2 | sed -e s/'Silicon sensor, thickness'// | awk '{print $1*1000}'`
# find X_RAY_WAVELENGTH:
X_RAY_WAVELENGTH=`grep Wavelength tmp2 | sed -e s/Wavelength// -e s/A// | awk '{print $1}'`

X_RAY_ENERGY=`echo "12398.5/$X_RAY_WAVELENGTH" | bc -l | awk '{printf "%5.1f", $1}'`
# find NX and NY; 2463/2527 is 6M, 1475/1679 is 2M
NX=`grep X-Binary-Size-Fastest-Dimension tmp2 | awk '{print $2}'`
NY=`grep X-Binary-Size-Second-Dimension tmp2 | awk '{print $2}'`
NXMM=`echo "$NX*$QX" | bc -l | awk '{printf "%3.2f", $1}'`
NYMM=`echo "$NY*$QY" | bc -l | awk '{printf "%3.2f", $1}'`
# find ORGX and ORGY:
ORGX=`grep Beam_xy tmp2 | sed -e s/$// -e s/$// -e s/\,// | awk '{print $2}'`
ORGY=`grep Beam_xy tmp2 | sed -e s/$// -e s/$// -e s/\,// | awk '{print $3}'`
ORGXMM=`echo "$ORGX*$QX" | bc -l | awk '{printf "%3.2f", $1}'`
ORGYMM=`echo "$ORGY*$QY" | bc -l | awk '{printf "%3.2f", $1}'`
# find DETECTOR_DISTANCE and OSCILLATION_RANGE:
DETECTOR_DISTANCE=`awk '/distance/{print $2}' tmp2`
DETECTOR_DISTANCE=`echo "$DETECTOR_DISTANCE*1000" | bc -l`

OSCILLATION_RANGE=`awk '/Angle/{print $2}' tmp2`

else
echo should never come here
exit 1
fi

echo ORGX= $ORGX ORGY= $ORGY - check these values with adxv !
echo DETECTOR_DISTANCE= $DETECTOR_DISTANCE
echo OSCILLATION_RANGE= $OSCILLATION_RANGE
echo X-RAY_WAVELENGTH= $X_RAY_WAVELENGTH
# now we know everything that is required to generate XDS.INP

cat > XDS.INP << eof
!****************************************************************************************
! Written by generate_XDS.INP version $REVISION
!
! A few notes on usage and modification of this input file:
!
! If XDS fails after IDXREF with an error indicating that
! an insufficient percentage of reflections were indexed,
! but you are confident that the indexing solution is
! correct, rerun XDS with the JOB keyword changed to
! "DEFPIX INTEGRATE CORRECT".
!
! After the first round of integration (in a triclinic cell)
! it is often helpful to reintegrate using the globally
! refined cell parameters and the correct space group.
! You can do this by renaming GXPARM.XDS to XPARM.XDS
! - e.g. by the command "cp GXPARM.XDS XPARM.XDS" -
! and rerunning DEFPIX, INTEGRATE and CORRECT.
!
! If you are trying to determine the optimal starting point for
! data collection starting from a single image, first complete
! a run of XDS using default parameters. Then, rerun XDS using
! the correct cell and spacegroup as described above
! ("cp GXPARM.XDS XPARM.XDS"), but on the second run of XDS, give
! the JOB keyword as "JOB=DEFPIX XPLAN".
!
! Be warned that the starting spindle angles given by XPLAN are
! relative to the first image given - XDS assumes that the first image
! given has a phi of 0 unless explicitly told otherwise. If this is not
! the case (e.g. you are indexing from a snapshot taken at phi=90,
! you will need to add the actual phi (as reported in the image header)
! to the values given by XPLAN in order for these values to make sense.
!****************************************************************************************
!
!****************************************************************************************
!General parameters:
!****************************************************************************************

JOB= XYCORR INIT COLSPOT IDXREF DEFPIX INTEGRATE CORRECT

ORGX= $ORGX ORGY= $ORGY ! check these values with adxv!
!In mm, ORGX=$ORGXMM and ORGY=$ORGYMM

DETECTOR_DISTANCE= $DETECTOR_DISTANCE
OSCILLATION_RANGE= $OSCILLATION_RANGE

X-RAY_WAVELENGTH= $X_RAY_WAVELENGTH
!X-ray energy is $X_RAY_ENERGY eV (calculated from wavelength)

NAME_TEMPLATE_OF_DATA_FRAMES=$NAME_TEMPLATE_OF_DATA_FRAMES

!REFERENCE_DATA_SET=xxx/XDS_ASCII.HKL ! e.g. to ensure consistent indexing.
!XDS will *only* use the reference dataset if the space group and cell
!parameters are specified below.

DATA_RANGE=1 $DATA_RANGE
SPOT_RANGE=1 $SPOT_RANGE

!BACKGROUND_RANGE=1 10 !Default is first five degrees.

SPACE_GROUP_NUMBER=0 ! 0 if unknown
UNIT_CELL_CONSTANTS= 70 80 90 90 90 90 ! put correct values if known

INCLUDE_RESOLUTION_RANGE=50 0 ! After CORRECT, insert high resol limit; re-run CORRECT
! Ice rings can be excluded from scaling with the keyword
! EXCLUDE_RESOLUTION_RANGE, e.g. EXCLUDE_RESOLUTION_RANGE=2.28 2.22

!SECONDS=60 !Uncomment this line to tell XDS to wait 1min for the next image before aborting.

!****************************************************************************************
!Parameters important for processing anomalous data:
!****************************************************************************************

FRIEDEL'S_LAW=FALSE ! This acts only on the CORRECT step. Even if the anomalous signal
! in your dataset is very small, it still may be useful for
! calculating an anomalous difference map.

STRICT_ABSORPTION_CORRECTION=FALSE !Change this to TRUE if the anomalous signal
! is strong: in that case, in CORRECT.LP the three
! "CHI^2-VALUE OF FIT OF CORRECTION FACTORS" values
! will be significantly> 1, e.g. 1.5.

WFAC1=1.0 !This parameter controls rejection of misfits during scaling. Sometimes,
! strong anomalous pairs may be rejected as misfits, in which case increasing
! WFAC1 (e.g. to 1.5) may improve anomalous signal.

!****************************************************************************************
!Regions of the detector to be excluded during indexing and integration:
!****************************************************************************************
! exclude (mask) untrusted areas of detector, e.g. beamstop shadow :
!UNTRUSTED_RECTANGLE= !1800 1950 2100 2150 ! x-min x-max y-min y-max ! repeat
!UNTRUSTED_ELLIPSE= !2034 2070 1850 2240 ! x-min x-max y-min y-max ! if needed
!UNTRUSTED_QUADRILATERAL= !x1 y1 x2 y2 x3 y3 x4 y4 ! see documentation

TRUSTED_REGION=0.00 1.2 ! Partially use corners of detectors; 1.41421=full use, 1.0=edge
! It is important to note that TRUSTED_REGION is the only resolution
! cutoff obeyed during indexing and integration;
! Both INCLUDE_RESOLUTION_RANGE and EXCLUDE_RESOLUTION_RANGE are
! ignored during IDXREF and INTEGRATE. So this parameter can be
! important when indexing or integration steps fail.

!****************************************************************************************
!Other parameters:
!****************************************************************************************
VALUE_RANGE_FOR_TRUSTED_DETECTOR_PIXELS=7000. 30000. ! Often 8000 is ok.
STRONG_PIXEL=4 ! COLSPOT: only use strong reflections (default is 3; 6 may be better for strong data)
MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT=3 ! default of 6 is sometimes too high
MAXIMUM_ERROR_OF_SPOT_POSITION=3 ! Increasing may help for poor quality data

MINPK=75 !Increase (e.g. MINPK=98) to improve data accuracy at the expense of completeness.

DELPHI=5 !Size of integration wedge. Increasing to 10 or 20 may be helpful in some cases.

NUMBER_OF_PROFILE_GRID_POINTS_ALONG_ALPHA/BETA=13 ! Default is 9 - Increasing may improve data
NUMBER_OF_PROFILE_GRID_POINTS_ALONG_GAMMA=13 ! accuracy, particularly if finely-sliced on phi,
! and does not seem to have any downsides.

MINIMUM_ZETA=0.05 ! Controls how close to the blind region (about phi) reflections should be
! integrated. 0.05 is the default; increasing MINIMUM_ZETA to 0.15 *may*
! improve data quality by removing unreliable reflections near the phi
! axis, but will reduce completeness in low symmetry space groups.

SEPMIN=6 ! Reduce if spots are close together (due to a long axis). Consider
CLUSTER_RADIUS=3 ! increasing if crystal is split and unit cell dimensions are relatively short.

! For bad or low resolution data remove the "!" in the following line (default is ALL):
! REFINE(IDXREF)=CELL BEAM ORIENTATION AXIS ! DISTANCE
REFINE(INTEGRATE)=CELL BEAM ORIENTATION ! AXIS DISTANCE (If integration is unstable, comment out this line.
! REFINE(CORRECT)=CELL BEAM ORIENTATION AXIS DISTANCE ! Default is: refine everything

ROTATION_AXIS=1 0 0 !At e.g. Australian Synchrotron, SERCAT ID-22 (?), APS 19-ID (?) this needs to be -1 0 0

!****************************************************************************************
! Parameters specifically for this detector and beamline (shouldn't need changing):
!****************************************************************************************
DETECTOR= $DETECTOR
SENSOR_THICKNESS= $SENSOR_THICKNESS
! attention CCD detectors: for very high resolution (better than 1A) make sure to specify SILICON
! as about 32* what CORRECT.LP suggests (absorption of phosphor is much higher than that of silicon)

NX= $NX NY= $NY QX= $QX QY= $QY ! to make CORRECT happy if frames are unavailable
!In mm, NX=$NXMM and NY=$NYMM

DIRECTION_OF_DETECTOR_X-AXIS=1 0 0
DIRECTION_OF_DETECTOR_Y-AXIS=0 1 0
INCIDENT_BEAM_DIRECTION=0 0 1
FRACTION_OF_POLARIZATION=0.98 ! better value is provided by beamline staff!
POLARIZATION_PLANE_NORMAL=0 1 0
eof
if [ "$DET" == "pilatus" ]; then
if [ $NX == "1475" ]; then
if ! grep -q FF_p2m0109_E12398_T6199_vrf_m0p20.tif tmp2 ; then
cat >> XDS.INP << eof
!EXCLUSION OF VERTICAL DEAD AREAS OF THE PILATUS 2M DETECTOR
UNTRUSTED_RECTANGLE= 486 496 0 1680
UNTRUSTED_RECTANGLE= 980 990 0 1680
!EXCLUSION OF HORIZONTAL DEAD AREAS OF THE PILATUS 2M DETECTOR
UNTRUSTED_RECTANGLE= 0 1476 194 214
UNTRUSTED_RECTANGLE= 0 1476 406 426
UNTRUSTED_RECTANGLE= 0 1476 618 638
UNTRUSTED_RECTANGLE= 0 1476 830 850
UNTRUSTED_RECTANGLE= 0 1476 1042 1062
UNTRUSTED_RECTANGLE= 0 1476 1254 1274
UNTRUSTED_RECTANGLE= 0 1476 1466 1486
eof
fi
# elif [ $NX == "2463" ]; then
# cat >> XDS.INP << eof
#eof
fi
fi
touch xds_to_ccp4.sh
echo "#!/bin/bash
#Removes old log file if present
rm xds_to_ccp4.log
#Makes new log file
touch xds_to_ccp4.log
#Pipe stdout and stderr to both logfile and tty
tail -f -n1 xds_to_ccp4.log &
exec > xds_to_ccp4.log 2>&1
echo \"

If this message is all you see,
something has probably gone wrong.
Are you sure XDS_ASCII.HKL is present
in the current directory?

\"
#Make first MTZ file (with DANO/SIGDANO)
echo \"INPUT_FILE=XDS_ASCII.HKL
OUTPUT_FILE=temp.hkl CCP4
MERGE=TRUE
GENERATE_FRACTION_OF_TEST_REFLECTIONS=0.05
FRIEDEL'S_LAW=FALSE\" > XDSCONV.INP
xdsconv
f2mtz HKLOUT temp.mtz<F2MTZ.INP

#Make second MTZ file (with F(+)/F(-))
echo \"INPUT_FILE=XDS_ASCII.HKL
OUTPUT_FILE=temp2.hkl CCP4_F
MERGE=TRUE
GENERATE_FRACTION_OF_TEST_REFLECTIONS=0.05
FRIEDEL'S_LAW=FALSE\" > XDSCONV.INP
xdsconv
f2mtz HKLOUT temp2.mtz<F2MTZ.INP

#Make SHELX format unmerged hkl file
echo \"INPUT_FILE=XDS_ASCII.HKL
OUTPUT_FILE=xds_shelx.hkl SHELX
MERGE=FALSE
FRIEDEL'S_LAW=FALSE\" > XDSCONV.INP
xdsconv

#cad the two previously generated MTZ files together
cad HKLIN1 temp.mtz HKLIN2 temp2.mtz HKLOUT xds_ccp4_merged.mtz<<EOF
LABIN FILE 1 E1=FP E2=SIGFP E3=DANO E4=SIGDANO E5=FreeRflag
LABIN FILE 2 E1=F(+) E2=SIGF(+) E3=F(-) E4=SIGF(-)
END
EOF

#Remove temporary files
rm temp.mtz
rm temp2.mtz
rm temp.hkl
rm temp2.hkl
rm F2MTZ.INP

echo \"

********************************XDS_TO_CCP4**********************************

This script takes XDS_ASCII.HKL and uses XDSCONV and F2MTZ to
generate a merged CCP4-format MTZ file named
\\\"xds_ccp4_merged.mtz\\\" with a test (free) set
labeled \\\"FreeRflag\\\" constituting 5% of reflections.

The space group of the output MTZ file will be the same as
that present in XDS_ASCII.HKL. It is probably worth checking
that this is the correct space group - either manually inspect
the systematic absences in CORRECT.LP, or run XDS_ASCII.HKL
through the CCP4 program POINTLESS (\\\"pointless xdsin XDS_ASCII.HKL\\\")

Anomalous data is kept, if present, and output as both
DANO/SIGDANO and F(+)/F(-).

This script also generates an unmerged SHELX hkl file,
suitable for input into SHELXC/D/E, named \\\"xds_shelx.hkl\\\".

A log of all processes is present in xds_to_ccp4.log.

Parts of this script have been shamelessly copied from
the XDS Wiki, at http://www.strucbio.biologie.uni-konstanz.de/xdswiki/

******************************************************************************
\"" > xds_to_ccp4.sh
touch xds_graph.sh
echo "#!/bin/bash
#Makes new log file
touch INTEGRATE_STATS.LP
#Pipe stdout and stderr to both logfile and tty
tail -f -n1 INTEGRATE_STATS.LP &
exec > INTEGRATE_STATS.LP 2>&1
echo \"
\\\$TABLE: XDS Integration Statistics :
\\\$SCATTER:Scale factors by image:A:1,3: :Number of overloads by image:A:1,5: :Number of strong reflections by image:A:1,7: :Number of rejects by image:A:1,8: :Mosaicity by image:A:1,10: \\$\\$
IMAGE IER SCALE NBKG NOVL NEWALD NSTRONG NREJ SIGMAB SIGMAR
\\$\\$ \\$\\$
\"
egrep \"[0-9]\.[0-9][0-9][0-9][0-9] [0-9]\.[0-9][0-9][0-9][0-9]\" INTEGRATE.LP
echo \" \\$\\$ \"
echo \"
\\\$TABLE: XDS Scaling Statistics by Resolution (X-axis is resolution in Angstroems) :
\\\$GRAPHS:Chi-Squared by resolution:A:2,4: :Observed and expected R-factors by resolution (as %):A:2,5,6: :Rejects by resolution:A:2,9: :Rejects by resolution as a percentage of total reflections:A:2,10: \\$\\$
LOW_RES Resolution(Angstroems) I/Sigma Chi^2 R-FACTOR(OBSERVED) R-FACTOR(EXPECTED) NUMBER_OF_REFLECTIONS ACCEPTED REJECTS REJECT_PERCENT
\\$\\$ \\$\\$
\"
awk '/RESOLUTION RANGE I\/Sigma Chi\^2 R\-FACTOR R\-FACTOR NUMBER ACCEPTED REJECTED/, /\-\-\-\-\-\-\-\-\-/' CORRECT.LP | egrep -v \"[a-z,A-Z,] | \-\-|\-99\.9\" | egrep \"[0-9]\" | awk '{print \$1,\$2,\$3,\$4,\$5,\$6,\$7,\$8,\$9,(\$9/(\$8+\$9))*100}'
echo \" \\$\\$\"
echo \"
\\\$TABLE: Overall data quality by Resolution (X-axis is resolution in Angstroems) :
\\\$GRAPHS:Completeness vs resolution :A:1,6: :Redundancy vs resolution :A:1,4: :I/Sigma by resolution:A:1,10: :Rmeas by resolution:A:1,11: :Anomalous correlation by resolution:A:1,13: :Significance of the anomalous signal by resolution:A:1,14: \\$\\$
Resolution(Angstroems) OBSERVED_REFLECTIONS UNIQUE_REFLECTIONS REDUNDANCY POSSIBLE_REFLECTIONS COMPLETENESS R-FACTOR(OBSERVED) R-FACTOR(EXPECTED) COMPARED I/SIGMA R-meas CC(1/2) Anomalous_Correlation SigAno Nano
\\$\\$ \\$\\$
\"
egrep -B25 \" WILSON STATISTICS \" CORRECT.LP | egrep -v \"[a-z,A-Z]|\*\*\*\" | egrep \" \" | awk '{ gsub(/[%*]/,\" \"); print }' | awk '{print \$1,\$2,\$3,\$2/\$3,\$4,\$5,\$6,\$7,\$8,\$9,\$10,\$11,\$12,\$13,\$14}'
echo \" \\$\\$ \"

echo \"
\\\$TABLE: I/Sigma of h00 reflections (for identification of systematic absences):
\\\$GRAPHS:I/Sigma vs h :A:1,5: \\$\\$
h k l RES I/SIGMA NUMBER
\\$\\$ \\$\\$
\"
awk '/REFLECTIONS OF TYPE H/,/COMPLETENESS AND QUALITY/' CORRECT.LP | egrep \"[1-9]\" | awk '{print \$1,\$2,\$3,\$4,\$7,\$8}' | awk '(\$1 > 0)'
echo \"\\$\\$\"

echo \"
\\\$TABLE: I/Sigma of 0k0 reflections (for identification of systematic absences):
\\\$GRAPHS:I/Sigma vs k :A:2,5: \\$\\$
h k l RES I/SIGMA NUMBER
\\$\\$ \\$\\$
\"
awk '/REFLECTIONS OF TYPE H/,/COMPLETENESS AND QUALITY/' CORRECT.LP | egrep \"[1-9]\" | awk '{print \$1,\$2,\$3,\$4,\$7,\$8}' | awk '(\$2 > 0)'
echo \"\\$\\$\"

echo \"
\\\$TABLE: I/Sigma of 00l reflections (for identification of systematic absences):
\\\$GRAPHS:I/Sigma vs l :A:3,5: \\$\\$
h k l RES I/SIGMA NUMBER
\\$\\$ \\$\\$
\"
awk '/REFLECTIONS OF TYPE H/,/COMPLETENESS AND QUALITY/' CORRECT.LP | egrep \"[1-9]\" | awk '{print \$1,\$2,\$3,\$4,\$7,\$8}' | awk '(\$3 > 0)'
echo \"\\$\\$\"

echo \"
\\\$TABLE:Beam center (ORGX/ORGY) plotted vs wedge (DELPHI):
\\\$GRAPHS:ORGX and ORGY vs batch (DELPHI) :A:1,2,3: \\$\\$
N ORGX ORGY
\\$\\$ \\$\\$
\"
egrep \"DETECTOR ORIGIN\" INTEGRATE.LP | grep -n \"\" | awk '{ gsub(/[a-z,A-Z,),(,:]/,\"\"); print }'
echo \"\\$\\$\"

echo \"
\\\$TABLE:Unit cell parameters plotted vs wedge:
\\\$GRAPHS:Unit cell lengths a, b and c plotted vs wedge:A:1,2,3,4: :Unit cell angles alpha, beta and gamma plotted vs wedge:A:1,5,6,7: \\$\\$
N a b c ALPHA BETA GAMMA
\\$\\$ \\$\\$
\"
egrep \"UNIT CELL PARAMETERS\" INTEGRATE.LP | grep -n \"\" | awk '{ gsub(/[a-z,A-Z,),(,:]/,\"\"); print }'
echo \"\\$\\$\"

echo \"
\\\$TABLE:Standard deviation of spot position (pixels) plotted vs wedge:
\\\$GRAPHS:Standard deviation of spot position plotted vs wedge:A:1,2: \\$\\$
N SD
\\$\\$ \\$\\$
\"
egrep \"SPOT POSITION\" INTEGRATE.LP | grep -n \"\" | awk '{ gsub(/[a-z,A-Z,),(,:]/,\"\"); print }'
echo \"\\$\\$\"

echo \"
\\\$TABLE:Detector distance (mm) plotted vs wedge (DELPHI):
\\\$GRAPHS:Detector distance plotted vs batch:A:1,2: \\$\\$
N DIST
\\$\\$ \\$\\$
\"
egrep \"DETECTOR DISTANCE \$mm\$\" INTEGRATE.LP | grep -n \"\" | awk '{ gsub(/[a-z,A-Z,),(,:]/,\"\"); print }'

echo \"\\$\\$\"

loggraph INTEGRATE_STATS.LP">xds_graph.sh
chmod +x xds_to_ccp4.sh
chmod +x xds_graph.sh
echo "XDS.INP is ready for use. The file has only the most important keywords.
Full documentation, including complete detector templates, is at
http://www.mpimf-heidelberg.mpg.de/~kabsch/xds . More documentation in XDSwiki
After running xds, inspect, using XDS-Viewer, at least the beamstop mask in
BKGPIX.cbf, and the agreement of predicted and observed spots in FRAME.cbf!

Two shell scripts have been generated that may be of use.

The first, xds_graph.sh, will plot various statistics after an XDS run.
After both INTEGRATE and CORRECT have finished, run it by typing \"./xds_graph.sh\"
without the quotation marks. This script uses loggraph to plot data, so you will
need CCP4 to see the graphical output. All the raw data is piped out to a log file,
INTEGRATE_STATS.LP.

The second script, xds_to_ccp4.sh, will take XDS_ASCII.HKL and generate both a CCP4
format MTZ file with anomalous data retained (as F(+)/F(-) and DANO/SigDANO)
and an unmerged SHELX format hkl file for input to SHELXC/D/E. Run it after CORRECT
by typing \"./xds_to_ccp4.sh\" at the prompt."

rm -f tmp1 tmp2
</pre>

Use this script in the same manner as Generate_XDS.INP. The two shell scripts created should be executable; If they are not, then make them so (chmod +x ./xxx.sh).

Generate XDS.INP

2012-10-18T18:36:07Z

Olibclarke:

This script generates XDS.INP based on a list of frame names supplied on the commandline. It currently works for MarCCD, ADSC and Pilatus detectors; since this is just a bash script, extension to other detectors is very easy.

On Mac OS X, installation of the "Command Line Tools" (from http://developer.apple.com/downloads; requires Apple ID) is required (I think). These are a part of the (larger, but also free) [http://developer.apple.com/tools/xcode Xcode] package.

== Usage ==

Usage is just (don't forget the quotation marks!):
generate_XDS.INP "frms/mydata_1_???.img"
XDS [http://strucbio.biologie.uni-konstanz.de/~dikay/XDS_html_doc/html_doc/xds_parameters.html#NAME_TEMPLATE_OF_DATA_FRAMES= supports] bzip2-ed frames. Thus, when specifying the frame name parameter of the script, you should leave out any .bz2 extension.

== The script ==

<pre>
#!/bin/bash
# purpose: generate XDS.INP
#
# tested with some datasets from ALS, SSRL, SLS, ESRF and BESSY; only MARCCD, ADSC/SMV, PILATUS detectors;
# for other detectors, values marked with XXX must be manually filled in.
#
# revision 0.03 . Kay Diederichs 2/2010
# revision 0.04 . Kay Diederichs 4/2010 - include alternative ORGX, ORGY calculations for ADSC
# revision 0.05 . Kay Diederichs 5/2010 - grep for "Corrected" in addition to "marccd"; needed for BESSY
# revision 0.06 . KD 6/2010 - add UNTRUSTED_RECTANGLE and UNTRUSTED_ELLIPSE; use `whereis catmar` and so on
# revision 0.07 . KD 6/2010 - decide about ORGX/Y info in MAR header being pixels or mm; other fixes
# revision 0.08 . KD 6/2010 - fixes for Pilatus 6M
# revision 0.09 . KD 6/2010 - get rid of requirement for mccd_xdsparams.pl and/or catmar; rather use "od"
# revision 0.10 . Tim Gruene 7/2010 - set link 'images' to image directory if path exceeds 72 characters
# revision 0.11 . KD 7/2010 - for MarCCD: look for distance info at different byte position
# revision 0.12 . KD 7/2010 - fix for negative PHISTART
# revision 0.13 . KD 8/2010 - store correct NX NY QX QY in XDS.INP
# revision 0.14 . KD 1/2011 - SENSOR_THICKNESS for Pilatus; MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT=3
# revision 0.15 . KD 2/2011 - add comment for -ive sign of APS 19-ID and Australian Synchrotron rotation axis
# revision 0.16 . KD 3/2011 - SENSOR_THICKNESS=0.01 for ADSC and MarCCD. Add comment about SILICON=
# revision 0.17 . KD 3/2011 - make it work for .bz2 frames; improve screen output
# revision 0.18 . KD 4/2011 - faster by doing "strings" only once; revert "images/${1##/*/}" "correction"
# revision 0.19 . KD 6/2011 - bugfix for 0.18
# revision 0.20 . KD 7/2011 - redirect stderr of /bin/ls to /dev/null
# revision 0.21 . KD 11/2011 - SEPMIN, CLUSTER_RADIUS hints; read NX NY from header (for Pilatus 2M)
# revision 0.22 . KD 12/2011 - Pilatus 2M UNTRUSTED_RECTANGLE lines, SENSOR_THICKNESS from header
# revision 0.23 . KD 1/2012 - add UNTRUSTED_QUADRILATERAL, remove MINIMUM_ZETA (0.05 is default now)
# revision 0.24 . KD 3/2012 - remove revision 0.10 since XDS now takes much longer paths
# revision 0.25 . KD 3/2012 - remove revision 0.22 for PSI Pilatus 2M; see http://www.globalphasing.com/autoproc/wiki/index.cgi?TroubleShootingKnownIssues
# revision 0.26 . KD 7/2012 - Mac-compatibility: replace od flags --skip-bytes= and --read-bytes= with -j and -N (thanks to Oliver Clarke for working this out!)
REVISION="0.26 (02-Jul-2012)"
#
# usage: e.g. generate_XDS.INP "frms/mydata_1_???.img"
# make sure to have the two quotation marks !
# the ? are wildcards for the frame numbers.
#
# limitations:
# - frame numbers are assumed to start with 1 and run consecutively
#
# known problems:
# - for ADSC detectors, there are at least three ways to obtain ORGX and ORGY values from the header (see below);
# - the same might be a problem for MAR headers, too (not sure about this)
# - on Mac OS X, Xcode (from http://developer.apple.com/tools/xcode) might be needed - not sure about this
#
# notes for debugging of the script:
# - add the -v option to the first line, to see where an error occurs
# - comment out the removal of tmp1 and tmp2 in the last line
#
# ====== Start of script ======
echo generate_XDS.INP version $REVISION . Obtain the latest version from
echo http://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/generate_XDS.INP
if [ "$1" == "help" ] || [ "$1" == "-help" ] || [ "$1" == "-h" ]; then
echo usage: generate_XDS.INP \"frms/mydata_1_???.img\" $_with_ the quotation marks!$
echo if the frames are compressed with bzip2, leave out the .bz2 extension!
exit
fi
#
# defaults:
#
DETECTOR="XXX MINIMUM_VALID_PIXEL_VALUE=XXX OVERLOAD=XXX"
ORGX=XXX
ORGY=XXX
DETECTOR_DISTANCE=XXX
OSCILLATION_RANGE=XXX
X_RAY_WAVELENGTH=XXX
QX=XXX
QY=XXX
NX=XXX
NY=XXX
SENSOR_THICKNESS=0
# see how we are called:
NAME_TEMPLATE_OF_DATA_FRAMES="$1"
# list frames matching the wildcards in NAME_TEMPLATE_OF_DATA_FRAMES
# don't accept the "direct beam" shot at SLS/Pilatus PX-I and PX-II
/bin/ls -C1 $1 $1.bz2 2>/dev/null | egrep -v "_00000.cbf|_000.img" > tmp1 || exit 1

# we can continue - the frames are found

# set upper limit of DATA_RANGE to number of frames (see "limitations" above)
DATA_RANGE=`wc -l tmp1 | awk '{print $1}'`

# set upper limit of SPOT_RANGE to half of DATA_RANGE, but not less than 1
SPOT_RANGE=`echo "scale=0; $DATA_RANGE/2" | bc -l`
SPOT_RANGE=`echo "if ($SPOT_RANGE<1) 1;if ($SPOT_RANGE>1) $SPOT_RANGE" | bc -l`

echo DATA_RANGE=1 $DATA_RANGE

# find out detector type
DET=XXX
FIRSTFRAME=`head -1 tmp1`
echo $FIRSTFRAME | grep -q bz2 && bzcat $FIRSTFRAME > tmp1 && FIRSTFRAME=tmp1
strings $FIRSTFRAME > tmp2
egrep -q 'marccd|Corrected' tmp2 && DET=mccd
grep -q PILATUS tmp2 && DET=pilatus
grep -q BEAM_CENTER_X tmp2 && DET=adsc
# identify other detector types in the same way (MAR IP would be straightforward)

# parse ASCII header of first frame

if [ "$DET" == "XXX" ]; then
echo "this is not a MAR, ADSC/SMV or PILATUS detector - fill in XXX values manually!"
DETECTOR="XXX MINIMUM_VALID_PIXEL_VALUE=XXX OVERLOAD=XXX"

# find parameters of first frame
elif [ "$DET" == "mccd" ]; then
echo Data from a MarCCD detector

DETECTOR="CCDCHESS MINIMUM_VALID_PIXEL_VALUE= 1 OVERLOAD= 65500"
SENSOR_THICKNESS=0.01
# use first frame of dataset to obtain parameters

# offsets are documented; values can be found in mccd_xdsparams.pl script
let SKIP=1024+80
NX=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
let SKIP=$SKIP+4
NY=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')

let SKIP=1720
DETECTOR_DISTANCE=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
DETECTOR_DISTANCE=`echo "scale=3; $DETECTOR_DISTANCE/1000" | bc -l`

let SKIP=1024+256+128+256+4
ORGX=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
ORGX=`echo "scale=2; $ORGX/1000" | bc -l `
let SKIP=$SKIP+4
ORGY=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
ORGY=`echo "scale=2; $ORGY/1000" | bc -l `

let SKIP=1024+256+128+256+44
PHISTART=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
let SKIP=1024+256+128+256+76
PHIEND=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
OSCILLATION_RANGE=`echo "scale=3; ($PHIEND-($PHISTART))/1000" | bc -l`

let SKIP=1024+256+128+256+128+4
QX=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
QX=`echo "scale=10; $QX/1000000" |bc -l `
let SKIP=$SKIP+4
QY=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
QY=`echo "scale=10; $QY/1000000" |bc -l `

let SKIP=1024+256+128+256+128+128+12
X_RAY_WAVELENGTH=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
X_RAY_WAVELENGTH=`echo "scale=5; $X_RAY_WAVELENGTH/100000" | bc -l`

# at most BLs, ORGX and ORGY are in pixels, but sometimes in mm ... guess:
NXBYFOUR=`echo "scale=0; $NX/4" | bc -l `
ORGXINT=`echo "scale=0; $ORGX/1" | bc -l `
if [ $ORGXINT -lt $NXBYFOUR ]; then
ORGX=`echo "scale=1; $ORGX/$QX" | bc -l`
ORGY=`echo "scale=1; $ORGY/$QY" | bc -l`
echo MARCCD detector: header ORGX, ORGY seem to be in mm ... converting to pixels
else
echo MARCCD detector: header ORGX, ORGY seem to be in pixel units
fi

elif [ "$DET" == "adsc" ]; then

DETECTOR="ADSC MINIMUM_VALID_PIXEL_VALUE= 1 OVERLOAD= 65000"
echo Data from ADSC detector. Obtaining ORGX, ORGY depends on beamline setup:
SENSOR_THICKNESS=0.01
sed s/\;// tmp2 > tmp1
mv tmp1 tmp2

# find X_RAY_WAVELENGTH:
X_RAY_WAVELENGTH=`grep WAVELENGTH tmp2 | head -1 | sed s/WAVELENGTH=//`

# find NX, QX, ORGX and ORGY:
NX=`grep SIZE1 tmp2 | tail -1 | sed s/SIZE1=//`
QX=`grep PIXEL_SIZE tmp2 | sed s/PIXEL_SIZE=//`
# FIXME - next 2 lines should be done properly, from header
NY=$NX
QY=$QX
BEAM_CENTER_X=`grep BEAM_CENTER_X tmp2 | sed s/BEAM_CENTER_X=//`
BEAM_CENTER_Y=`grep BEAM_CENTER_Y tmp2 | sed s/BEAM_CENTER_Y=//`
# fix 2010-04-26 - tell user about possible ORGX, ORGY alternatives -
# at ESRF and ... (pls fill in!) the following should be used:
ORGX=`echo "scale=1; $BEAM_CENTER_Y/$QX" | bc -l`
ORGY=`echo "scale=1; $BEAM_CENTER_X/$QX" | bc -l`
echo - at ESRF BLs use: ORGX=$ORGX ORGY=$ORGY
# this 2nd alternative convention should be used at the following beamlines (pls complete the list): ALS 5.0.3, ...
ORGX=`echo "scale=1; $NX-$BEAM_CENTER_X/$QX" | bc -l `
ORGY=`echo "scale=1; $BEAM_CENTER_Y/$QX" | bc -l `
echo - at e.g. ALS 5.0.3 use: ORGX=$ORGX ORGY=$ORGY
# this 3rd alternative convention should be used at the following beamlines (pls complete the list): ALS 8.2.2, ...
ORGX=`echo "scale=1; $BEAM_CENTER_X/$QX" | bc -l `
ORGY=`echo "scale=1; $NX-$BEAM_CENTER_Y/$QX" | bc -l `
echo - at e.g. ALS 8.2.2 use: ORGX=$ORGX ORGY=$ORGY - this is written to XDS.INP
# the latter alternative is written into the generated XDS.INP ! You have to correct this manually in XDS.INP, or adjust this script.
# find DETECTOR_DISTANCE and OSCILLATION_RANGE:
DETECTOR_DISTANCE=`grep DISTANCE tmp2 | sed s/DISTANCE=//`
OSCILLATION_RANGE=`grep OSC_RANGE tmp2 | sed s/OSC_RANGE=//`

elif [ "$DET" == "pilatus" ]; then
DETECTOR="PILATUS MINIMUM_VALID_PIXEL_VALUE=0 OVERLOAD= 1048576 !PILATUS"
QX=0.172 QY=0.172
echo Data from a Pilatus detector
sed s/#// tmp2 > tmp1
mv tmp1 tmp2

# find SENSOR_THICKNESS:
SENSOR_THICKNESS=`grep thickness tmp2 | sed -e s/'Silicon sensor, thickness'// | awk '{print $1*1000}'`
# find X_RAY_WAVELENGTH:
X_RAY_WAVELENGTH=`grep Wavelength tmp2 | sed -e s/Wavelength// -e s/A// | awk '{print $1}'`

# find NX and NY; 2463/2527 is 6M, 1475/1679 is 2M
NX=`grep X-Binary-Size-Fastest-Dimension tmp2 | awk '{print $2}'`
NY=`grep X-Binary-Size-Second-Dimension tmp2 | awk '{print $2}'`

# find ORGX and ORGY:
ORGX=`grep Beam_xy tmp2 | sed -e s/$// -e s/$// -e s/\,// | awk '{print $2}'`
ORGY=`grep Beam_xy tmp2 | sed -e s/$// -e s/$// -e s/\,// | awk '{print $3}'`

# find DETECTOR_DISTANCE and OSCILLATION_RANGE:
DETECTOR_DISTANCE=`awk '/distance/{print $2}' tmp2`
DETECTOR_DISTANCE=`echo "$DETECTOR_DISTANCE*1000" | bc -l`

OSCILLATION_RANGE=`awk '/Angle/{print $2}' tmp2`

else
echo should never come here
exit 1
fi

echo ORGX= $ORGX ORGY= $ORGY - check these values with adxv !
echo DETECTOR_DISTANCE= $DETECTOR_DISTANCE
echo OSCILLATION_RANGE= $OSCILLATION_RANGE
echo X-RAY_WAVELENGTH= $X_RAY_WAVELENGTH

# now we know everything that is required to generate XDS.INP

cat > XDS.INP << eof
! written by generate_XDS.INP version $REVISION
JOB= XYCORR INIT COLSPOT IDXREF DEFPIX INTEGRATE CORRECT
ORGX= $ORGX ORGY= $ORGY ! check these values with adxv !
DETECTOR_DISTANCE= $DETECTOR_DISTANCE
OSCILLATION_RANGE= $OSCILLATION_RANGE
X-RAY_WAVELENGTH= $X_RAY_WAVELENGTH
NAME_TEMPLATE_OF_DATA_FRAMES=$NAME_TEMPLATE_OF_DATA_FRAMES
! REFERENCE_DATA_SET=xxx/XDS_ASCII.HKL ! e.g. to ensure consistent indexing
DATA_RANGE=1 $DATA_RANGE
SPOT_RANGE=1 $SPOT_RANGE
! BACKGROUND_RANGE=1 10 ! rather use defaults (first 5 degree of rotation)

SPACE_GROUP_NUMBER=0 ! 0 if unknown
UNIT_CELL_CONSTANTS= 70 80 90 90 90 90 ! put correct values if known
INCLUDE_RESOLUTION_RANGE=50 0 ! after CORRECT, insert high resol limit; re-run CORRECT

FRIEDEL'S_LAW=FALSE ! This acts only on the CORRECT step
! If the anom signal turns out to be, or is known to be, very low or absent,
! use FRIEDEL'S_LAW=TRUE instead (or comment out the line); re-run CORRECT

! remove the "!" in the following line:
! STRICT_ABSORPTION_CORRECTION=TRUE
! if the anomalous signal is strong: in that case, in CORRECT.LP the three
! "CHI^2-VALUE OF FIT OF CORRECTION FACTORS" values are significantly> 1, e.g. 1.5
!
! exclude (mask) untrusted areas of detector, e.g. beamstop shadow :
! UNTRUSTED_RECTANGLE= 1800 1950 2100 2150 ! x-min x-max y-min y-max ! repeat
! UNTRUSTED_ELLIPSE= 2034 2070 1850 2240 ! x-min x-max y-min y-max ! if needed
! UNTRUSTED_QUADRILATERAL= x1 y1 x2 y2 x3 y3 x4 y4 ! see documentation
!
! parameters with changes wrt default values:
TRUSTED_REGION=0.00 1.2 ! partially use corners of detectors; 1.41421=full use
VALUE_RANGE_FOR_TRUSTED_DETECTOR_PIXELS=7000. 30000. ! often 8000 is ok
STRONG_PIXEL=4 ! COLSPOT: only use strong reflections (default is 3)
MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT=3 ! default of 6 is sometimes too high
! close spots: reduce SEPMIN and CLUSTER_RADIUS from their defaults of 6 and 3, e.g. to 4 and 2
! for bad or low resolution data remove the "!" in the following line:
! REFINE(IDXREF)=CELL BEAM ORIENTATION AXIS ! DISTANCE
REFINE(INTEGRATE)=CELL BEAM ORIENTATION ! AXIS DISTANCE
! REFINE(CORRECT)=CELL BEAM ORIENTATION AXIS DISTANCE ! Default is: refine everything

! parameters specifically for this detector and beamline:
DETECTOR= $DETECTOR
SENSOR_THICKNESS= $SENSOR_THICKNESS
! attention CCD detectors: for very high resolution (better than 1A) make sure to specify SILICON
! as about 32* what CORRECT.LP suggests (absorption of phosphor is much higher than that of silicon)
NX= $NX NY= $NY QX= $QX QY= $QY ! to make CORRECT happy if frames are unavailable
DIRECTION_OF_DETECTOR_X-AXIS=1 0 0
DIRECTION_OF_DETECTOR_Y-AXIS=0 1 0
INCIDENT_BEAM_DIRECTION=0 0 1
ROTATION_AXIS=1 0 0 ! at e.g. Australian Synchrotron, SERCAT ID-22 (?), APS 19-ID (?) this needs to be -1 0 0
FRACTION_OF_POLARIZATION=0.98 ! better value is provided by beamline staff!
POLARIZATION_PLANE_NORMAL=0 1 0
eof
if [ "$DET" == "pilatus" ]; then
if [ $NX == "1475" ]; then
if ! grep -q FF_p2m0109_E12398_T6199_vrf_m0p20.tif tmp2 ; then
cat >> XDS.INP << eof
!EXCLUSION OF VERTICAL DEAD AREAS OF THE PILATUS 2M DETECTOR
UNTRUSTED_RECTANGLE= 486 496 0 1680
UNTRUSTED_RECTANGLE= 980 990 0 1680
!EXCLUSION OF HORIZONTAL DEAD AREAS OF THE PILATUS 2M DETECTOR
UNTRUSTED_RECTANGLE= 0 1476 194 214
UNTRUSTED_RECTANGLE= 0 1476 406 426
UNTRUSTED_RECTANGLE= 0 1476 618 638
UNTRUSTED_RECTANGLE= 0 1476 830 850
UNTRUSTED_RECTANGLE= 0 1476 1042 1062
UNTRUSTED_RECTANGLE= 0 1476 1254 1274
UNTRUSTED_RECTANGLE= 0 1476 1466 1486
eof
fi
# elif [ $NX == "2463" ]; then
# cat >> XDS.INP << eof
#eof
fi
fi
echo XDS.INP is ready for use. The file has only the most important keywords.
echo Full documentation, including complete detector templates, is at
echo http://www.mpimf-heidelberg.mpg.de/~kabsch/xds . More documentation in XDSwiki
echo After running xds, inspect, using XDS-Viewer, at least the beamstop mask in
echo BKGPIX.cbf, and the agreement of predicted and observed spots in FRAME.cbf!
rm -f tmp1 tmp2
</pre>

== System-wide or personal installation ==

Ask your system adminstrator to cut-and-paste the script into e.g. /usr/local/bin/generate_XDS.INP, and to make it "executable".

But you may also cut-and-paste the script from this webpage into a file in e.g. your home directory; the filename should be generate_XDS.INP. After creating the file, make it executable - e.g. if it's in your $HOME, use:
chmod +x ~/generate_XDS.INP
After that, you can just run it in a similar way as if it were installed in your $PATH:
~/generate_XDS.INP "frms/mydata_1_???.img"
By using your own file, you can easily update to the latest revision, or even change the script, without having to bother the system administrator.

== Generating generate_XDS.INP from this webpage ==

Instead of cutting-and-pasting the lines of the script, you (or the system administrator) could just cut-and-paste the following four lines
<pre>
wget http://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/generate_XDS.INP -O - | \
sed -e s/\&nbsp\;/\ /g -e s/\&gt\;/\>/g -e s/\&lt\;/\</g -e s/amp\;//g -e s/\&quot\;/\"/g -e s/\&\#\1\6\0\;/\ /g | \
awk '/^#/,/rm -f tmp1 tmp2/' > generate_XDS.INP
chmod +x generate_XDS.INP
</pre>
to copy the script from this website into an executable file generate_XDS.INP in your current directory.

==Variant script (xds_generate_all)==
This script includes some minor modifications of the Generate_XDS.INP script - for instance, it tells the user the energy, as well as the wavelength of the X-rays, and gives the user the beam center in mm as well as pixels - useful when switching between HKL2000, MOSFLM and XDS.

It also generates two executable shell scripts in the same directory as XDS.INP.

The first of these scripts, xds_graph.sh, extracts various statistics from INTEGRATE.LP and CORRECT.LP, pipes them out to a logfile, INTEGRATE_STATS.LP, and plots them using loggraph. This script does not have some of the more useful features of XDSSTAT, such as calculation of per frame Rmeas values, but it does give a quick overview of various data quality parameters by resolution and image number. This script will only work with recent versions of XDS (after calculation of CC(1/2) - not sure exactly which version).

The second of these scripts, xds_to_ccp4.sh, will take XDS_ASCII.HKL and generate a CCP4-format MTZ file, containing anomalous data (as F(+)/F(-) and DANO/SigDANO) and a test set of 5% of reflections for calculation of the free R-factor. It will also generate an unmerged SHELX format HKL file, for input into SHELXC/D/E (e.g. via the HKL2MAP GUI).

Here is the script:
<pre>
#!/bin/bash
# purpose: xds_generate_all
#
# tested with some datasets from ALS, SSRL, SLS, ESRF and BESSY; only MARCCD, ADSC/SMV, PILATUS detectors;
# for other detectors, values marked with XXX must be manually filled in.
#
# revision 0.03 . Kay Diederichs 2/2010
# revision 0.04 . Kay Diederichs 4/2010 - include alternative ORGX, ORGY calculations for ADSC
# revision 0.05 . Kay Diederichs 5/2010 - grep for "Corrected" in addition to "marccd"; needed for BESSY
# revision 0.06 . KD 6/2010 - add UNTRUSTED_RECTANGLE and UNTRUSTED_ELLIPSE; use `whereis catmar` and so on
# revision 0.07 . KD 6/2010 - decide about ORGX/Y info in MAR header being pixels or mm; other fixes
# revision 0.08 . KD 6/2010 - fixes for Pilatus 6M
# revision 0.09 . KD 6/2010 - get rid of requirement for mccd_xdsparams.pl and/or catmar; rather use "od"
# revision 0.10 . Tim Gruene 7/2010 - set link 'images' to image directory if path exceeds 72 characters
# revision 0.11 . KD 7/2010 - for MarCCD: look for distance info at different byte position
# revision 0.12 . KD 7/2010 - fix for negative PHISTART
# revision 0.13 . KD 8/2010 - store correct NX NY QX QY in XDS.INP
# revision 0.14 . KD 1/2011 - SENSOR_THICKNESS for Pilatus; MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT=3
# revision 0.15 . KD 2/2011 - add comment for -ive sign of APS 19-ID and Australian Synchrotron rotation axis
# revision 0.16 . KD 3/2011 - SENSOR_THICKNESS=0.01 for ADSC and MarCCD. Add comment about SILICON=
# revision 0.17 . KD 3/2011 - make it work for .bz2 frames; improve screen output
# revision 0.18 . KD 4/2011 - faster by doing "strings" only once; revert "images/${1##/*/}" "correction"
# revision 0.19 . KD 6/2011 - bugfix for 0.18
# revision 0.20 . KD 7/2011 - redirect stderr of /bin/ls to /dev/null
# revision 0.21 . KD 11/2011 - SEPMIN, CLUSTER_RADIUS hints; read NX NY from header (for Pilatus 2M)
# revision 0.22 . KD 12/2011 - Pilatus 2M UNTRUSTED_RECTANGLE lines, SENSOR_THICKNESS from header
# revision 0.23 . KD 1/2012 - add UNTRUSTED_QUADRILATERAL, remove MINIMUM_ZETA (0.05 is default now)
# revision 0.24 . KD 3/2012 - remove revision 0.10 since XDS now takes much longer paths
# revision 0.25 . KD 3/2012 - remove revision 0.22 for PSI Pilatus 2M; see http://www.globalphasing.com/autoproc/wiki/index.cgi?TroubleShootingKnownIssues
# revision 0.26 Oliver Clarke 10/12 - Added generation of shell scripts for conversion to MTZ and visualisation of statistics. Other minor alterations to XDS.INP.
REVISION="0.26 (18-Oct-2012)"
#
# usage: e.g. xds_generate_all "frms/mydata_1_???.img"
# make sure to have the two quotation marks !
# the ? are wildcards for the frame numbers.
#
# limitations:
# - frame numbers are assumed to start with 1 and run consecutively
#
# known problems:
# - for ADSC detectors, there are at least three ways to obtain ORGX and ORGY values from the header (see below);
# - the same might be a problem for MAR headers, too (not sure about this)
#
# notes for debugging of the script:
# - add the -v option to the first line, to see where an error occurs
# - comment out the removal of tmp1 and tmp2 in the last line
#
# ====== Start of script ======
echo generate_XDS.INP version $REVISION . Obtain the latest version from
echo http://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/generate_XDS.INP
if [ "$1" == "help" ] || [ "$1" == "-help" ] || [ "$1" == "-h" ]; then
echo usage: xds_generate_all \"frms/mydata_1_???.img\" $_with_ the quotation marks!$
echo if the frames are compressed with bzip2, leave out the .bz2 extension!
exit
fi
#
# defaults:
#
DETECTOR="XXX MINIMUM_VALID_PIXEL_VALUE=XXX OVERLOAD=XXX"
ORGX=XXX
ORGY=XXX
ORGXMM=XXX
ORGYMM=XXX
DETECTOR_DISTANCE=XXX
OSCILLATION_RANGE=XXX
X_RAY_WAVELENGTH=XXX
X_RAY_ENERGY=XXX
QX=XXX
QY=XXX
NX=XXX
NY=XXX
NXMM=XXX
NYMM=XXX
SENSOR_THICKNESS=0
# see how we are called:
NAME_TEMPLATE_OF_DATA_FRAMES="$1"
# list frames matching the wildcards in NAME_TEMPLATE_OF_DATA_FRAMES
# don't accept the "direct beam" shot at SLS/Pilatus PX-I and PX-II
/bin/ls -C1 $1 $1.bz2 2>/dev/null | egrep -v "_00000.cbf|_000.img" > tmp1 || exit 1

# we can continue - the frames are found

# set upper limit of DATA_RANGE to number of frames (see "limitations" above)
DATA_RANGE=`wc -l tmp1 | awk '{print $1}'`

# set upper limit of SPOT_RANGE to half of DATA_RANGE, but not less than 1
SPOT_RANGE=`echo "scale=0; $DATA_RANGE/2" | bc -l`
SPOT_RANGE=`echo "if ($SPOT_RANGE<1) 1;if ($SPOT_RANGE>1) $SPOT_RANGE" | bc -l`

echo DATA_RANGE=1 $DATA_RANGE

# find out detector type
DET=XXX
FIRSTFRAME=`head -1 tmp1`
echo $FIRSTFRAME | grep -q bz2 && bzcat $FIRSTFRAME > tmp1 && FIRSTFRAME=tmp1
strings $FIRSTFRAME > tmp2
egrep -q 'marccd|Corrected' tmp2 && DET=mccd
grep -q PILATUS tmp2 && DET=pilatus
grep -q BEAM_CENTER_X tmp2 && DET=adsc
# identify other detector types in the same way (MAR IP would be straightforward)

# parse ASCII header of first frame

if [ "$DET" == "XXX" ]; then
echo "this is not a MAR, ADSC/SMV or PILATUS detector - fill in XXX values manually!"
DETECTOR="XXX MINIMUM_VALID_PIXEL_VALUE=XXX OVERLOAD=XXX"

# find parameters of first frame
elif [ "$DET" == "mccd" ]; then
echo Data from a MarCCD detector

DETECTOR="CCDCHESS MINIMUM_VALID_PIXEL_VALUE= 1 OVERLOAD= 65500"
SENSOR_THICKNESS=0.01
# use first frame of dataset to obtain parameters

# offsets are documented; values can be find in mccd_xdsparams.pl script
let SKIP=1024+80
NX=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
let SKIP=$SKIP+4
NY=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')

let SKIP=1720
DETECTOR_DISTANCE=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
DETECTOR_DISTANCE=`echo "scale=3; $DETECTOR_DISTANCE/1000" | bc -l`

let SKIP=1024+256+128+256+4
ORGX=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
ORGX=`echo "scale=2; $ORGX/1000" | bc -l `
let SKIP=$SKIP+4
ORGY=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
ORGY=`echo "scale=2; $ORGY/1000" | bc -l `

let SKIP=1024+256+128+256+44
PHISTART=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
let SKIP=1024+256+128+256+76
PHIEND=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
OSCILLATION_RANGE=`echo "scale=3; ($PHIEND-($PHISTART))/1000" | bc -l`

let SKIP=1024+256+128+256+128+4
QX=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
QX=`echo "scale=10; $QX/1000000" |bc -l `
let SKIP=$SKIP+4
QY=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
QY=`echo "scale=10; $QY/1000000" |bc -l `

let SKIP=1024+256+128+256+128+128+12
X_RAY_WAVELENGTH=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
X_RAY_WAVELENGTH=`echo "scale=5; $X_RAY_WAVELENGTH/100000" | bc -l`
X_RAY_ENERGY=`echo "12398.5/$X_RAY_WAVELENGTH" | bc -l | awk '{printf "%5.1f", $1}'`
# at most BLs, ORGX and ORGY are in pixels, but sometimes in mm ... guess:
NXBYFOUR=`echo "scale=0; $NX/4" | bc -l `
ORGXINT=`echo "scale=0; $ORGX/1" | bc -l `
if [ $ORGXINT -lt $NXBYFOUR ]; then
ORGXMM=$ORGX
ORGYMM=$ORGY
NXMM=`echo "$NX*$QX" | bc -l | awk '{printf "%3.2f", $1}'`
NYMM=`echo "$NY*$QY" | bc -l | awk '{printf "%3.2f", $1}'`
ORGX=`echo "scale=1; $ORGX/$QX" | bc -l`
ORGY=`echo "scale=1; $ORGY/$QY" | bc -l`
echo MARCCD detector: header ORGX, ORGY seem to be in mm ... converting to pixels
else
ORGXMM=`echo "$ORGX*$QX" | bc -l | awk '{printf "%3.2f", $1}'`
ORGYMM=`echo "$ORGY*$QY" | bc -l | awk '{printf "%3.2f", $1}'`
NXMM=`echo "$NX*$QX" | bc -l | awk '{printf "%3.2f", $1}'`
NYMM=`echo "$NY*$QY" | bc -l | awk '{printf "%3.2f", $1}'`
echo MARCCD detector: header ORGX, ORGY seem to be in pixel units
fi

elif [ "$DET" == "adsc" ]; then

DETECTOR="ADSC MINIMUM_VALID_PIXEL_VALUE= 1 OVERLOAD= 65000"
echo Data from ADSC detector. Obtaining ORGX, ORGY depends on beamline setup:
SENSOR_THICKNESS=0.01
sed s/\;// tmp2 > tmp1
mv tmp1 tmp2

# find X_RAY_WAVELENGTH:
X_RAY_WAVELENGTH=`grep WAVELENGTH tmp2 | head -1 | sed s/WAVELENGTH=//`

# find NX, QX, ORGX and ORGY:
X_RAY_ENERGY=`echo "12398.5/$X_RAY_WAVELENGTH" | bc -l | awk '{printf "%5.1f", $1}'`
NX=`grep SIZE1 tmp2 | tail -1 | sed s/SIZE1=//`
QX=`grep PIXEL_SIZE tmp2 | sed s/PIXEL_SIZE=//`
# FIXME - next 2 lines should be done properly, from header
NY=$NX
QY=$QX
NXMM=`echo "$NX*$QX" | bc -l | awk '{printf "%3.2f", $1}'`
NYMM=`echo "$NY*$QY" | bc -l | awk '{printf "%3.2f", $1}'`
BEAM_CENTER_X=`grep BEAM_CENTER_X tmp2 | sed s/BEAM_CENTER_X=//`
BEAM_CENTER_Y=`grep BEAM_CENTER_Y tmp2 | sed s/BEAM_CENTER_Y=//`
# fix 2010-04-26 - tell user about possible ORGX, ORGY alternatives -
# at ESRF and ... (pls fill in!) the following should be used:
ORGX=`echo "scale=1; $BEAM_CENTER_Y/$QX" | bc -l`
ORGY=`echo "scale=1; $BEAM_CENTER_X/$QX" | bc -l`
echo - at ESRF BLs use: ORGX=$ORGX ORGY=$ORGY
# this 2nd alternative convention should be used at the following beamlines (pls complete the list): ALS 5.0.3, ...
ORGX=`echo "scale=1; $NX-$BEAM_CENTER_X/$QX" | bc -l `
ORGY=`echo "scale=1; $BEAM_CENTER_Y/$QX" | bc -l `
echo - at e.g. ALS 5.0.3 use: ORGX=$ORGX ORGY=$ORGY
# this 3rd alternative convention should be used at the following beamlines (pls complete the list): ALS 8.2.2, ...
ORGX=`echo "scale=1; $BEAM_CENTER_X/$QX" | bc -l `
ORGY=`echo "scale=1; $NX-$BEAM_CENTER_Y/$QX" | bc -l `
ORGXMM=`echo "$ORGX*$QX" | bc -l | awk '{printf "%3.2f", $1}'`
ORGYMM=`echo "$ORGY*$QY" | bc -l | awk '{printf "%3.2f", $1}'`
echo - at e.g. ALS 8.2.2 use: ORGX=$ORGX ORGY=$ORGY - this is written to XDS.INP
# the latter alternative is written into the generated XDS.INP ! You have to correct this manually in XDS.INP, or adjust this script.
# find DETECTOR_DISTANCE and OSCILLATION_RANGE:
DETECTOR_DISTANCE=`grep DISTANCE tmp2 | sed s/DISTANCE=//`
OSCILLATION_RANGE=`grep OSC_RANGE tmp2 | sed s/OSC_RANGE=//`

elif [ "$DET" == "pilatus" ]; then
DETECTOR="PILATUS MINIMUM_VALID_PIXEL_VALUE=0 OVERLOAD= 1048576 !PILATUS"
QX=0.172 QY=0.172
echo Data from a Pilatus detector
sed s/#// tmp2 > tmp1
mv tmp1 tmp2

# find SENSOR_THICKNESS:
SENSOR_THICKNESS=`grep thickness tmp2 | sed -e s/'Silicon sensor, thickness'// | awk '{print $1*1000}'`
# find X_RAY_WAVELENGTH:
X_RAY_WAVELENGTH=`grep Wavelength tmp2 | sed -e s/Wavelength// -e s/A// | awk '{print $1}'`

X_RAY_ENERGY=`echo "12398.5/$X_RAY_WAVELENGTH" | bc -l | awk '{printf "%5.1f", $1}'`
# find NX and NY; 2463/2527 is 6M, 1475/1679 is 2M
NX=`grep X-Binary-Size-Fastest-Dimension tmp2 | awk '{print $2}'`
NY=`grep X-Binary-Size-Second-Dimension tmp2 | awk '{print $2}'`
NXMM=`echo "$NX*$QX" | bc -l | awk '{printf "%3.2f", $1}'`
NYMM=`echo "$NY*$QY" | bc -l | awk '{printf "%3.2f", $1}'`
# find ORGX and ORGY:
ORGX=`grep Beam_xy tmp2 | sed -e s/$// -e s/$// -e s/\,// | awk '{print $2}'`
ORGY=`grep Beam_xy tmp2 | sed -e s/$// -e s/$// -e s/\,// | awk '{print $3}'`
ORGXMM=`echo "$ORGX*$QX" | bc -l | awk '{printf "%3.2f", $1}'`
ORGYMM=`echo "$ORGY*$QY" | bc -l | awk '{printf "%3.2f", $1}'`
# find DETECTOR_DISTANCE and OSCILLATION_RANGE:
DETECTOR_DISTANCE=`awk '/distance/{print $2}' tmp2`
DETECTOR_DISTANCE=`echo "$DETECTOR_DISTANCE*1000" | bc -l`

OSCILLATION_RANGE=`awk '/Angle/{print $2}' tmp2`

else
echo should never come here
exit 1
fi

echo ORGX= $ORGX ORGY= $ORGY - check these values with adxv !
echo DETECTOR_DISTANCE= $DETECTOR_DISTANCE
echo OSCILLATION_RANGE= $OSCILLATION_RANGE
echo X-RAY_WAVELENGTH= $X_RAY_WAVELENGTH
# now we know everything that is required to generate XDS.INP

cat > XDS.INP << eof
!****************************************************************************************
! Written by generate_XDS.INP version $REVISION
!
! A few notes on usage and modification of this input file:
!
! If XDS fails after IDXREF with an error indicating that
! an insufficient percentage of reflections were indexed,
! but you are confident that the indexing solution is
! correct, rerun XDS with the JOB keyword changed to
! "DEFPIX INTEGRATE CORRECT".
!
! After the first round of integration (in a triclinic cell)
! it is often helpful to reintegrate using the globally
! refined cell parameters and the correct space group.
! You can do this by renaming GXPARM.XDS to XPARM.XDS
! - e.g. by the command "cp GXPARM.XDS XPARM.XDS" -
! and rerunning DEFPIX, INTEGRATE and CORRECT.
!
! If you are trying to determine the optimal starting point for
! data collection starting from a single image, first complete
! a run of XDS using default parameters. Then, rerun XDS using
! the correct cell and spacegroup as described above
! ("cp GXPARM.XDS XPARM.XDS"), but on the second run of XDS, give
! the JOB keyword as "JOB=DEFPIX XPLAN".
!
! Be warned that the starting spindle angles given by XPLAN are
! relative to the first image given - XDS assumes that the first image
! given has a phi of 0 unless explicitly told otherwise. If this is not
! the case (e.g. you are indexing from a snapshot taken at phi=90,
! you will need to add the actual phi (as reported in the image header)
! to the values given by XPLAN in order for these values to make sense.
!****************************************************************************************
!
!****************************************************************************************
!General parameters:
!****************************************************************************************

JOB= XYCORR INIT COLSPOT IDXREF DEFPIX INTEGRATE CORRECT

ORGX= $ORGX ORGY= $ORGY ! check these values with adxv!
!In mm, ORGX=$ORGXMM and ORGY=$ORGYMM

DETECTOR_DISTANCE= $DETECTOR_DISTANCE
OSCILLATION_RANGE= $OSCILLATION_RANGE

X-RAY_WAVELENGTH= $X_RAY_WAVELENGTH
!X-ray energy is $X_RAY_ENERGY eV (calculated from wavelength)

NAME_TEMPLATE_OF_DATA_FRAMES=$NAME_TEMPLATE_OF_DATA_FRAMES

!REFERENCE_DATA_SET=xxx/XDS_ASCII.HKL ! e.g. to ensure consistent indexing.
!XDS will *only* use the reference dataset if the space group and cell
!parameters are specified below.

DATA_RANGE=1 $DATA_RANGE
SPOT_RANGE=1 $SPOT_RANGE

!BACKGROUND_RANGE=1 10 !Default is first five degrees.

SPACE_GROUP_NUMBER=0 ! 0 if unknown
UNIT_CELL_CONSTANTS= 70 80 90 90 90 90 ! put correct values if known

INCLUDE_RESOLUTION_RANGE=50 0 ! After CORRECT, insert high resol limit; re-run CORRECT
! Ice rings can be excluded from scaling with the keyword
! EXCLUDE_RESOLUTION_RANGE, e.g. EXCLUDE_RESOLUTION_RANGE=2.28 2.22

!SECONDS=60 !Uncomment this line to tell XDS to wait 1min for the next image before aborting.

!****************************************************************************************
!Parameters important for processing anomalous data:
!****************************************************************************************

FRIEDEL'S_LAW=FALSE ! This acts only on the CORRECT step. Even if the anomalous signal
! in your dataset is very small, it still may be useful for
! calculating an anomalous difference map.

STRICT_ABSORPTION_CORRECTION=FALSE !Change this to TRUE if the anomalous signal
! is strong: in that case, in CORRECT.LP the three
! "CHI^2-VALUE OF FIT OF CORRECTION FACTORS" values
! will be significantly> 1, e.g. 1.5.

WFAC1=1.0 !This parameter controls rejection of misfits during scaling. Sometimes,
! strong anomalous pairs may be rejected as misfits, in which case increasing
! WFAC1 (e.g. to 1.5) may improve anomalous signal.

!****************************************************************************************
!Regions of the detector to be excluded during indexing and integration:
!****************************************************************************************
! exclude (mask) untrusted areas of detector, e.g. beamstop shadow :
!UNTRUSTED_RECTANGLE= !1800 1950 2100 2150 ! x-min x-max y-min y-max ! repeat
!UNTRUSTED_ELLIPSE= !2034 2070 1850 2240 ! x-min x-max y-min y-max ! if needed
!UNTRUSTED_QUADRILATERAL= !x1 y1 x2 y2 x3 y3 x4 y4 ! see documentation

TRUSTED_REGION=0.00 1.2 ! Partially use corners of detectors; 1.41421=full use, 1.0=edge
! It is important to note that TRUSTED_REGION is the only resolution
! cutoff obeyed during indexing and integration;
! Both INCLUDE_RESOLUTION_RANGE and EXCLUDE_RESOLUTION_RANGE are
! ignored during IDXREF and INTEGRATE. So this parameter can be
! important when indexing or integration steps fail.

!****************************************************************************************
!Other parameters:
!****************************************************************************************
VALUE_RANGE_FOR_TRUSTED_DETECTOR_PIXELS=7000. 30000. ! Often 8000 is ok.
STRONG_PIXEL=4 ! COLSPOT: only use strong reflections (default is 3; 6 may be better for strong data)
MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT=3 ! default of 6 is sometimes too high
MAXIMUM_ERROR_OF_SPOT_POSITION=3 ! Increasing may help for poor quality data

MINPK=75 !Increase (e.g. MINPK=98) to improve data accuracy at the expense of completeness.

DELPHI=5 !Size of integration wedge. Increasing to 10 or 20 may be helpful in some cases.

NUMBER_OF_PROFILE_GRID_POINTS_ALONG_ALPHA/BETA=13 ! Default is 9 - Increasing may improve data
NUMBER_OF_PROFILE_GRID_POINTS_ALONG_GAMMA=13 ! accuracy, particularly if finely-sliced on phi,
! and does not seem to have any downsides.

MINIMUM_ZETA=0.05 ! Controls how close to the blind region (about phi) reflections should be
! integrated. 0.05 is the default; increasing MINIMUM_ZETA to 0.15 *may*
! improve data quality by removing unreliable reflections near the phi
! axis, but will reduce completeness in low symmetry space groups.

SEPMIN=6 ! Reduce if spots are close together (due to a long axis). Consider
CLUSTER_RADIUS=3 ! increasing if crystal is split and unit cell dimensions are relatively short.

! For bad or low resolution data remove the "!" in the following line (default is ALL):
! REFINE(IDXREF)=CELL BEAM ORIENTATION AXIS ! DISTANCE
REFINE(INTEGRATE)=CELL BEAM ORIENTATION ! AXIS DISTANCE (If integration is unstable, comment out this line.
! REFINE(CORRECT)=CELL BEAM ORIENTATION AXIS DISTANCE ! Default is: refine everything

ROTATION_AXIS=1 0 0 !At e.g. Australian Synchrotron, SERCAT ID-22 (?), APS 19-ID (?) this needs to be -1 0 0

!****************************************************************************************
! Parameters specifically for this detector and beamline (shouldn't need changing):
!****************************************************************************************
DETECTOR= $DETECTOR
SENSOR_THICKNESS= $SENSOR_THICKNESS
! attention CCD detectors: for very high resolution (better than 1A) make sure to specify SILICON
! as about 32* what CORRECT.LP suggests (absorption of phosphor is much higher than that of silicon)

NX= $NX NY= $NY QX= $QX QY= $QY ! to make CORRECT happy if frames are unavailable
!In mm, NX=$NXMM and NY=$NYMM

DIRECTION_OF_DETECTOR_X-AXIS=1 0 0
DIRECTION_OF_DETECTOR_Y-AXIS=0 1 0
INCIDENT_BEAM_DIRECTION=0 0 1
FRACTION_OF_POLARIZATION=0.98 ! better value is provided by beamline staff!
POLARIZATION_PLANE_NORMAL=0 1 0
eof
if [ "$DET" == "pilatus" ]; then
if [ $NX == "1475" ]; then
if ! grep -q FF_p2m0109_E12398_T6199_vrf_m0p20.tif tmp2 ; then
cat >> XDS.INP << eof
!EXCLUSION OF VERTICAL DEAD AREAS OF THE PILATUS 2M DETECTOR
UNTRUSTED_RECTANGLE= 486 496 0 1680
UNTRUSTED_RECTANGLE= 980 990 0 1680
!EXCLUSION OF HORIZONTAL DEAD AREAS OF THE PILATUS 2M DETECTOR
UNTRUSTED_RECTANGLE= 0 1476 194 214
UNTRUSTED_RECTANGLE= 0 1476 406 426
UNTRUSTED_RECTANGLE= 0 1476 618 638
UNTRUSTED_RECTANGLE= 0 1476 830 850
UNTRUSTED_RECTANGLE= 0 1476 1042 1062
UNTRUSTED_RECTANGLE= 0 1476 1254 1274
UNTRUSTED_RECTANGLE= 0 1476 1466 1486
eof
fi
# elif [ $NX == "2463" ]; then
# cat >> XDS.INP << eof
#eof
fi
fi
touch xds_to_ccp4.sh
echo "#!/bin/bash
#Removes old log file if present
rm xds_to_ccp4.log
#Makes new log file
touch xds_to_ccp4.log
#Pipe stdout and stderr to both logfile and tty
tail -f -n1 xds_to_ccp4.log &
exec > xds_to_ccp4.log 2>&1
echo \"

If this message is all you see,
something has probably gone wrong.
Are you sure XDS_ASCII.HKL is present
in the current directory?

\"
#Make first MTZ file (with DANO/SIGDANO)
echo \"INPUT_FILE=XDS_ASCII.HKL
OUTPUT_FILE=temp.hkl CCP4
MERGE=TRUE
GENERATE_FRACTION_OF_TEST_REFLECTIONS=0.05
FRIEDEL'S_LAW=FALSE\" > XDSCONV.INP
xdsconv
f2mtz HKLOUT temp.mtz<F2MTZ.INP

#Make second MTZ file (with F(+)/F(-))
echo \"INPUT_FILE=XDS_ASCII.HKL
OUTPUT_FILE=temp2.hkl CCP4_F
MERGE=TRUE
GENERATE_FRACTION_OF_TEST_REFLECTIONS=0.05
FRIEDEL'S_LAW=FALSE\" > XDSCONV.INP
xdsconv
f2mtz HKLOUT temp2.mtz<F2MTZ.INP

#Make SHELX format unmerged hkl file
echo \"INPUT_FILE=XDS_ASCII.HKL
OUTPUT_FILE=xds_shelx.hkl SHELX
MERGE=FALSE
FRIEDEL'S_LAW=FALSE\" > XDSCONV.INP
xdsconv

#cad the two previously generated MTZ files together
cad HKLIN1 temp.mtz HKLIN2 temp2.mtz HKLOUT xds_ccp4_merged.mtz<<EOF
LABIN FILE 1 E1=FP E2=SIGFP E3=DANO E4=SIGDANO E5=FreeRflag
LABIN FILE 2 E1=F(+) E2=SIGF(+) E3=F(-) E4=SIGF(-)
END
EOF

#Remove temporary files
rm temp.mtz
rm temp2.mtz
rm temp.hkl
rm temp2.hkl
rm F2MTZ.INP

echo \"

********************************XDS_TO_CCP4**********************************

This script takes XDS_ASCII.HKL and uses XDSCONV and F2MTZ to
generate a merged CCP4-format MTZ file named
\\\"xds_ccp4_merged.mtz\\\" with a test (free) set
labeled \\\"FreeRflag\\\" constituting 5% of reflections.

The space group of the output MTZ file will be the same as
that present in XDS_ASCII.HKL. It is probably worth checking
that this is the correct space group - either manually inspect
the systematic absences in CORRECT.LP, or run XDS_ASCII.HKL
through the CCP4 program POINTLESS (\\\"pointless xdsin XDS_ASCII.HKL\\\")

Anomalous data is kept, if present, and output as both
DANO/SIGDANO and F(+)/F(-).

This script also generates an unmerged SHELX hkl file,
suitable for input into SHELXC/D/E, named \\\"xds_shelx.hkl\\\".

A log of all processes is present in xds_to_ccp4.log.

Parts of this script have been shamelessly copied from
the XDS Wiki, at http://www.strucbio.biologie.uni-konstanz.de/xdswiki/

******************************************************************************
\"" > xds_to_ccp4.sh
touch xds_graph.sh
echo "#!/bin/bash
#Makes new log file
touch INTEGRATE_STATS.LP
#Pipe stdout and stderr to both logfile and tty
tail -f -n1 INTEGRATE_STATS.LP &
exec > INTEGRATE_STATS.LP 2>&1
echo \"
\\\$TABLE: XDS Integration Statistics :
\\\$SCATTER:Scale factors by image:A:1,3: :Number of overloads by image:A:1,5: :Number of strong reflections by image:A:1,7: :Number of rejects by image:A:1,8: :Mosaicity by image:A:1,10: \\$\\$
IMAGE IER SCALE NBKG NOVL NEWALD NSTRONG NREJ SIGMAB SIGMAR
\\$\\$ \\$\\$
\"
egrep \"[0-9]\.[0-9][0-9][0-9][0-9] [0-9]\.[0-9][0-9][0-9][0-9]\" INTEGRATE.LP
echo \" \\$\\$ \"
echo \"
\\\$TABLE: XDS Scaling Statistics by Resolution (X-axis is resolution in Angstroems) :
\\\$GRAPHS:Chi-Squared by resolution:A:2,4: :Observed and expected R-factors by resolution (as %):A:2,5,6: :Rejects by resolution:A:2,9: :Rejects by resolution as a percentage of total reflections:A:2,10: \\$\\$
LOW_RES Resolution(Angstroems) I/Sigma Chi^2 R-FACTOR(OBSERVED) R-FACTOR(EXPECTED) NUMBER_OF_REFLECTIONS ACCEPTED REJECTS REJECT_PERCENT
\\$\\$ \\$\\$
\"
awk '/RESOLUTION RANGE I\/Sigma Chi\^2 R\-FACTOR R\-FACTOR NUMBER ACCEPTED REJECTED/, /\-\-\-\-\-\-\-\-\-/' CORRECT.LP | egrep -v \"[a-z,A-Z,] | \-\-|\-99\.9\" | egrep \"[0-9]\" | awk '{print \$1,\$2,\$3,\$4,\$5,\$6,\$7,\$8,\$9,(\$9/(\$8+\$9))*100}'
echo \" \\$\\$\"
echo \"
\\\$TABLE: Overall data quality by Resolution (X-axis is resolution in Angstroems) :
\\\$GRAPHS:Completeness vs resolution :A:1,6: :Redundancy vs resolution :A:1,4: :I/Sigma by resolution:A:1,10: :Rmeas by resolution:A:1,11: :Anomalous correlation by resolution:A:1,13: :Significance of the anomalous signal by resolution:A:1,14: \\$\\$
Resolution(Angstroems) OBSERVED_REFLECTIONS UNIQUE_REFLECTIONS REDUNDANCY POSSIBLE_REFLECTIONS COMPLETENESS R-FACTOR(OBSERVED) R-FACTOR(EXPECTED) COMPARED I/SIGMA R-meas CC(1/2) Anomalous_Correlation SigAno Nano
\\$\\$ \\$\\$
\"
egrep -B25 \" WILSON STATISTICS \" CORRECT.LP | egrep -v \"[a-z,A-Z]|\*\*\*\" | egrep \" \" | awk '{ gsub(/[%*]/,\" \"); print }' | awk '{print \$1,\$2,\$3,\$2/\$3,\$4,\$5,\$6,\$7,\$8,\$9,\$10,\$11,\$12,\$13,\$14}'
echo \" \\$\\$ \"

echo \"
\\\$TABLE: I/Sigma of h00 reflections (for identification of systematic absences):
\\\$GRAPHS:I/Sigma vs h :A:1,5: \\$\\$
h k l RES I/SIGMA NUMBER
\\$\\$ \\$\\$
\"
awk '/REFLECTIONS OF TYPE H/,/COMPLETENESS AND QUALITY/' CORRECT.LP | egrep \"[1-9]\" | awk '{print \$1,\$2,\$3,\$4,\$7,\$8}' | awk '(\$1 > 0)'
echo \"\\$\\$\"

echo \"
\\\$TABLE: I/Sigma of 0k0 reflections (for identification of systematic absences):
\\\$GRAPHS:I/Sigma vs k :A:2,5: \\$\\$
h k l RES I/SIGMA NUMBER
\\$\\$ \\$\\$
\"
awk '/REFLECTIONS OF TYPE H/,/COMPLETENESS AND QUALITY/' CORRECT.LP | egrep \"[1-9]\" | awk '{print \$1,\$2,\$3,\$4,\$7,\$8}' | awk '(\$2 > 0)'
echo \"\\$\\$\"

echo \"
\\\$TABLE: I/Sigma of 00l reflections (for identification of systematic absences):
\\\$GRAPHS:I/Sigma vs l :A:3,5: \\$\\$
h k l RES I/SIGMA NUMBER
\\$\\$ \\$\\$
\"
awk '/REFLECTIONS OF TYPE H/,/COMPLETENESS AND QUALITY/' CORRECT.LP | egrep \"[1-9]\" | awk '{print \$1,\$2,\$3,\$4,\$7,\$8}' | awk '(\$3 > 0)'
echo \"\\$\\$\"

echo \"
\\\$TABLE:Beam center (ORGX/ORGY) plotted vs wedge (DELPHI):
\\\$GRAPHS:ORGX and ORGY vs batch (DELPHI) :A:1,2,3: \\$\\$
N ORGX ORGY
\\$\\$ \\$\\$
\"
egrep \"DETECTOR ORIGIN\" INTEGRATE.LP | grep -n \"\" | awk '{ gsub(/[a-z,A-Z,),(,:]/,\"\"); print }'
echo \"\\$\\$\"

echo \"
\\\$TABLE:Unit cell parameters plotted vs wedge:
\\\$GRAPHS:Unit cell lengths a, b and c plotted vs wedge:A:1,2,3,4: :Unit cell angles alpha, beta and gamma plotted vs wedge:A:1,5,6,7: \\$\\$
N a b c ALPHA BETA GAMMA
\\$\\$ \\$\\$
\"
egrep \"UNIT CELL PARAMETERS\" INTEGRATE.LP | grep -n \"\" | awk '{ gsub(/[a-z,A-Z,),(,:]/,\"\"); print }'
echo \"\\$\\$\"

echo \"
\\\$TABLE:Standard deviation of spot position (pixels) plotted vs wedge:
\\\$GRAPHS:Standard deviation of spot position plotted vs wedge:A:1,2: \\$\\$
N SD
\\$\\$ \\$\\$
\"
egrep \"SPOT POSITION\" INTEGRATE.LP | grep -n \"\" | awk '{ gsub(/[a-z,A-Z,),(,:]/,\"\"); print }'
echo \"\\$\\$\"

echo \"
\\\$TABLE:Detector distance (mm) plotted vs wedge (DELPHI):
\\\$GRAPHS:Detector distance plotted vs batch:A:1,2: \\$\\$
N DIST
\\$\\$ \\$\\$
\"
egrep \"DETECTOR DISTANCE \$mm\$\" INTEGRATE.LP | grep -n \"\" | awk '{ gsub(/[a-z,A-Z,),(,:]/,\"\"); print }'

echo \"\\$\\$\"

loggraph INTEGRATE_STATS.LP">xds_graph.sh
chmod +x xds_to_ccp4.sh
chmod +x xds_graph.sh
echo "XDS.INP is ready for use. The file has only the most important keywords.
Full documentation, including complete detector templates, is at
http://www.mpimf-heidelberg.mpg.de/~kabsch/xds . More documentation in XDSwiki
After running xds, inspect, using XDS-Viewer, at least the beamstop mask in
BKGPIX.cbf, and the agreement of predicted and observed spots in FRAME.cbf!

Two shell scripts have been generated that may be of use.

The first, xds_graph.sh, will plot various statistics after an XDS run.
After both INTEGRATE and CORRECT have finished, run it by typing \"./xds_graph.sh\"
without the quotation marks. This script uses loggraph to plot data, so you will
need CCP4 to see the graphical output. All the raw data is piped out to a log file,
INTEGRATE_STATS.LP.

The second script, xds_to_ccp4.sh, will take XDS_ASCII.HKL and generate both a CCP4
format MTZ file with anomalous data retained (as F(+)/F(-) and DANO/SigDANO)
and an unmerged SHELX format hkl file for input to SHELXC/D/E. Run it after CORRECT
by typing \"./xds_to_ccp4.sh\" at the prompt."

rm -f tmp1 tmp2
</pre>

Use this script in the same manner as Generate_XDS.INP. The two shell scripts created should be executable; If they are not, then make them so (chmod +x ./xxx.sh).

DonelleMarshal174

2012-10-18T17:05:35Z

Olibclarke: Blanked the page

Generate XDS.INP

2012-10-18T16:38:36Z

Olibclarke: /* xds_generate_all: variant of Generate_XDS.INP including scripts for visualisation of statistics and conversion to MTZ */

This script generates XDS.INP based on a list of frame names supplied on the commandline. It currently works for MarCCD, ADSC and Pilatus detectors; since this is just a bash script, extension to other detectors is very easy.

On Mac OS X, installation of the "Command Line Tools" (from http://developer.apple.com/downloads; requires Apple ID) is required (I think). These are a part of the (larger, but also free) [http://developer.apple.com/tools/xcode Xcode] package.

== Usage ==

Usage is just (don't forget the quotation marks!):
generate_XDS.INP "frms/mydata_1_???.img"
XDS [http://strucbio.biologie.uni-konstanz.de/~dikay/XDS_html_doc/html_doc/xds_parameters.html#NAME_TEMPLATE_OF_DATA_FRAMES= supports] bzip2-ed frames. Thus, when specifying the frame name parameter of the script, you should leave out any .bz2 extension.

== The script ==

<pre>
#!/bin/bash
# purpose: generate XDS.INP
#
# tested with some datasets from ALS, SSRL, SLS, ESRF and BESSY; only MARCCD, ADSC/SMV, PILATUS detectors;
# for other detectors, values marked with XXX must be manually filled in.
#
# revision 0.03 . Kay Diederichs 2/2010
# revision 0.04 . Kay Diederichs 4/2010 - include alternative ORGX, ORGY calculations for ADSC
# revision 0.05 . Kay Diederichs 5/2010 - grep for "Corrected" in addition to "marccd"; needed for BESSY
# revision 0.06 . KD 6/2010 - add UNTRUSTED_RECTANGLE and UNTRUSTED_ELLIPSE; use `whereis catmar` and so on
# revision 0.07 . KD 6/2010 - decide about ORGX/Y info in MAR header being pixels or mm; other fixes
# revision 0.08 . KD 6/2010 - fixes for Pilatus 6M
# revision 0.09 . KD 6/2010 - get rid of requirement for mccd_xdsparams.pl and/or catmar; rather use "od"
# revision 0.10 . Tim Gruene 7/2010 - set link 'images' to image directory if path exceeds 72 characters
# revision 0.11 . KD 7/2010 - for MarCCD: look for distance info at different byte position
# revision 0.12 . KD 7/2010 - fix for negative PHISTART
# revision 0.13 . KD 8/2010 - store correct NX NY QX QY in XDS.INP
# revision 0.14 . KD 1/2011 - SENSOR_THICKNESS for Pilatus; MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT=3
# revision 0.15 . KD 2/2011 - add comment for -ive sign of APS 19-ID and Australian Synchrotron rotation axis
# revision 0.16 . KD 3/2011 - SENSOR_THICKNESS=0.01 for ADSC and MarCCD. Add comment about SILICON=
# revision 0.17 . KD 3/2011 - make it work for .bz2 frames; improve screen output
# revision 0.18 . KD 4/2011 - faster by doing "strings" only once; revert "images/${1##/*/}" "correction"
# revision 0.19 . KD 6/2011 - bugfix for 0.18
# revision 0.20 . KD 7/2011 - redirect stderr of /bin/ls to /dev/null
# revision 0.21 . KD 11/2011 - SEPMIN, CLUSTER_RADIUS hints; read NX NY from header (for Pilatus 2M)
# revision 0.22 . KD 12/2011 - Pilatus 2M UNTRUSTED_RECTANGLE lines, SENSOR_THICKNESS from header
# revision 0.23 . KD 1/2012 - add UNTRUSTED_QUADRILATERAL, remove MINIMUM_ZETA (0.05 is default now)
# revision 0.24 . KD 3/2012 - remove revision 0.10 since XDS now takes much longer paths
# revision 0.25 . KD 3/2012 - remove revision 0.22 for PSI Pilatus 2M; see http://www.globalphasing.com/autoproc/wiki/index.cgi?TroubleShootingKnownIssues
# revision 0.26 . KD 7/2012 - Mac-compatibility: replace od flags --skip-bytes= and --read-bytes= with -j and -N (thanks to Oliver Clarke for working this out!)
REVISION="0.26 (02-Jul-2012)"
#
# usage: e.g. generate_XDS.INP "frms/mydata_1_???.img"
# make sure to have the two quotation marks !
# the ? are wildcards for the frame numbers.
#
# limitations:
# - frame numbers are assumed to start with 1 and run consecutively
#
# known problems:
# - for ADSC detectors, there are at least three ways to obtain ORGX and ORGY values from the header (see below);
# - the same might be a problem for MAR headers, too (not sure about this)
# - on Mac OS X, Xcode (from http://developer.apple.com/tools/xcode) might be needed - not sure about this
#
# notes for debugging of the script:
# - add the -v option to the first line, to see where an error occurs
# - comment out the removal of tmp1 and tmp2 in the last line
#
# ====== Start of script ======
echo generate_XDS.INP version $REVISION . Obtain the latest version from
echo http://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/generate_XDS.INP
if [ "$1" == "help" ] || [ "$1" == "-help" ] || [ "$1" == "-h" ]; then
echo usage: generate_XDS.INP \"frms/mydata_1_???.img\" $_with_ the quotation marks!$
echo if the frames are compressed with bzip2, leave out the .bz2 extension!
exit
fi
#
# defaults:
#
DETECTOR="XXX MINIMUM_VALID_PIXEL_VALUE=XXX OVERLOAD=XXX"
ORGX=XXX
ORGY=XXX
DETECTOR_DISTANCE=XXX
OSCILLATION_RANGE=XXX
X_RAY_WAVELENGTH=XXX
QX=XXX
QY=XXX
NX=XXX
NY=XXX
SENSOR_THICKNESS=0
# see how we are called:
NAME_TEMPLATE_OF_DATA_FRAMES="$1"
# list frames matching the wildcards in NAME_TEMPLATE_OF_DATA_FRAMES
# don't accept the "direct beam" shot at SLS/Pilatus PX-I and PX-II
/bin/ls -C1 $1 $1.bz2 2>/dev/null | egrep -v "_00000.cbf|_000.img" > tmp1 || exit 1

# we can continue - the frames are found

# set upper limit of DATA_RANGE to number of frames (see "limitations" above)
DATA_RANGE=`wc -l tmp1 | awk '{print $1}'`

# set upper limit of SPOT_RANGE to half of DATA_RANGE, but not less than 1
SPOT_RANGE=`echo "scale=0; $DATA_RANGE/2" | bc -l`
SPOT_RANGE=`echo "if ($SPOT_RANGE<1) 1;if ($SPOT_RANGE>1) $SPOT_RANGE" | bc -l`

echo DATA_RANGE=1 $DATA_RANGE

# find out detector type
DET=XXX
FIRSTFRAME=`head -1 tmp1`
echo $FIRSTFRAME | grep -q bz2 && bzcat $FIRSTFRAME > tmp1 && FIRSTFRAME=tmp1
strings $FIRSTFRAME > tmp2
egrep -q 'marccd|Corrected' tmp2 && DET=mccd
grep -q PILATUS tmp2 && DET=pilatus
grep -q BEAM_CENTER_X tmp2 && DET=adsc
# identify other detector types in the same way (MAR IP would be straightforward)

# parse ASCII header of first frame

if [ "$DET" == "XXX" ]; then
echo "this is not a MAR, ADSC/SMV or PILATUS detector - fill in XXX values manually!"
DETECTOR="XXX MINIMUM_VALID_PIXEL_VALUE=XXX OVERLOAD=XXX"

# find parameters of first frame
elif [ "$DET" == "mccd" ]; then
echo Data from a MarCCD detector

DETECTOR="CCDCHESS MINIMUM_VALID_PIXEL_VALUE= 1 OVERLOAD= 65500"
SENSOR_THICKNESS=0.01
# use first frame of dataset to obtain parameters

# offsets are documented; values can be found in mccd_xdsparams.pl script
let SKIP=1024+80
NX=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
let SKIP=$SKIP+4
NY=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')

let SKIP=1720
DETECTOR_DISTANCE=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
DETECTOR_DISTANCE=`echo "scale=3; $DETECTOR_DISTANCE/1000" | bc -l`

let SKIP=1024+256+128+256+4
ORGX=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
ORGX=`echo "scale=2; $ORGX/1000" | bc -l `
let SKIP=$SKIP+4
ORGY=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
ORGY=`echo "scale=2; $ORGY/1000" | bc -l `

let SKIP=1024+256+128+256+44
PHISTART=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
let SKIP=1024+256+128+256+76
PHIEND=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
OSCILLATION_RANGE=`echo "scale=3; ($PHIEND-($PHISTART))/1000" | bc -l`

let SKIP=1024+256+128+256+128+4
QX=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
QX=`echo "scale=10; $QX/1000000" |bc -l `
let SKIP=$SKIP+4
QY=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
QY=`echo "scale=10; $QY/1000000" |bc -l `

let SKIP=1024+256+128+256+128+128+12
X_RAY_WAVELENGTH=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
X_RAY_WAVELENGTH=`echo "scale=5; $X_RAY_WAVELENGTH/100000" | bc -l`

# at most BLs, ORGX and ORGY are in pixels, but sometimes in mm ... guess:
NXBYFOUR=`echo "scale=0; $NX/4" | bc -l `
ORGXINT=`echo "scale=0; $ORGX/1" | bc -l `
if [ $ORGXINT -lt $NXBYFOUR ]; then
ORGX=`echo "scale=1; $ORGX/$QX" | bc -l`
ORGY=`echo "scale=1; $ORGY/$QY" | bc -l`
echo MARCCD detector: header ORGX, ORGY seem to be in mm ... converting to pixels
else
echo MARCCD detector: header ORGX, ORGY seem to be in pixel units
fi

elif [ "$DET" == "adsc" ]; then

DETECTOR="ADSC MINIMUM_VALID_PIXEL_VALUE= 1 OVERLOAD= 65000"
echo Data from ADSC detector. Obtaining ORGX, ORGY depends on beamline setup:
SENSOR_THICKNESS=0.01
sed s/\;// tmp2 > tmp1
mv tmp1 tmp2

# find X_RAY_WAVELENGTH:
X_RAY_WAVELENGTH=`grep WAVELENGTH tmp2 | head -1 | sed s/WAVELENGTH=//`

# find NX, QX, ORGX and ORGY:
NX=`grep SIZE1 tmp2 | tail -1 | sed s/SIZE1=//`
QX=`grep PIXEL_SIZE tmp2 | sed s/PIXEL_SIZE=//`
# FIXME - next 2 lines should be done properly, from header
NY=$NX
QY=$QX
BEAM_CENTER_X=`grep BEAM_CENTER_X tmp2 | sed s/BEAM_CENTER_X=//`
BEAM_CENTER_Y=`grep BEAM_CENTER_Y tmp2 | sed s/BEAM_CENTER_Y=//`
# fix 2010-04-26 - tell user about possible ORGX, ORGY alternatives -
# at ESRF and ... (pls fill in!) the following should be used:
ORGX=`echo "scale=1; $BEAM_CENTER_Y/$QX" | bc -l`
ORGY=`echo "scale=1; $BEAM_CENTER_X/$QX" | bc -l`
echo - at ESRF BLs use: ORGX=$ORGX ORGY=$ORGY
# this 2nd alternative convention should be used at the following beamlines (pls complete the list): ALS 5.0.3, ...
ORGX=`echo "scale=1; $NX-$BEAM_CENTER_X/$QX" | bc -l `
ORGY=`echo "scale=1; $BEAM_CENTER_Y/$QX" | bc -l `
echo - at e.g. ALS 5.0.3 use: ORGX=$ORGX ORGY=$ORGY
# this 3rd alternative convention should be used at the following beamlines (pls complete the list): ALS 8.2.2, ...
ORGX=`echo "scale=1; $BEAM_CENTER_X/$QX" | bc -l `
ORGY=`echo "scale=1; $NX-$BEAM_CENTER_Y/$QX" | bc -l `
echo - at e.g. ALS 8.2.2 use: ORGX=$ORGX ORGY=$ORGY - this is written to XDS.INP
# the latter alternative is written into the generated XDS.INP ! You have to correct this manually in XDS.INP, or adjust this script.
# find DETECTOR_DISTANCE and OSCILLATION_RANGE:
DETECTOR_DISTANCE=`grep DISTANCE tmp2 | sed s/DISTANCE=//`
OSCILLATION_RANGE=`grep OSC_RANGE tmp2 | sed s/OSC_RANGE=//`

elif [ "$DET" == "pilatus" ]; then
DETECTOR="PILATUS MINIMUM_VALID_PIXEL_VALUE=0 OVERLOAD= 1048576 !PILATUS"
QX=0.172 QY=0.172
echo Data from a Pilatus detector
sed s/#// tmp2 > tmp1
mv tmp1 tmp2

# find SENSOR_THICKNESS:
SENSOR_THICKNESS=`grep thickness tmp2 | sed -e s/'Silicon sensor, thickness'// | awk '{print $1*1000}'`
# find X_RAY_WAVELENGTH:
X_RAY_WAVELENGTH=`grep Wavelength tmp2 | sed -e s/Wavelength// -e s/A// | awk '{print $1}'`

# find NX and NY; 2463/2527 is 6M, 1475/1679 is 2M
NX=`grep X-Binary-Size-Fastest-Dimension tmp2 | awk '{print $2}'`
NY=`grep X-Binary-Size-Second-Dimension tmp2 | awk '{print $2}'`

# find ORGX and ORGY:
ORGX=`grep Beam_xy tmp2 | sed -e s/$// -e s/$// -e s/\,// | awk '{print $2}'`
ORGY=`grep Beam_xy tmp2 | sed -e s/$// -e s/$// -e s/\,// | awk '{print $3}'`

# find DETECTOR_DISTANCE and OSCILLATION_RANGE:
DETECTOR_DISTANCE=`awk '/distance/{print $2}' tmp2`
DETECTOR_DISTANCE=`echo "$DETECTOR_DISTANCE*1000" | bc -l`

OSCILLATION_RANGE=`awk '/Angle/{print $2}' tmp2`

else
echo should never come here
exit 1
fi

echo ORGX= $ORGX ORGY= $ORGY - check these values with adxv !
echo DETECTOR_DISTANCE= $DETECTOR_DISTANCE
echo OSCILLATION_RANGE= $OSCILLATION_RANGE
echo X-RAY_WAVELENGTH= $X_RAY_WAVELENGTH

# now we know everything that is required to generate XDS.INP

cat > XDS.INP << eof
! written by generate_XDS.INP version $REVISION
JOB= XYCORR INIT COLSPOT IDXREF DEFPIX INTEGRATE CORRECT
ORGX= $ORGX ORGY= $ORGY ! check these values with adxv !
DETECTOR_DISTANCE= $DETECTOR_DISTANCE
OSCILLATION_RANGE= $OSCILLATION_RANGE
X-RAY_WAVELENGTH= $X_RAY_WAVELENGTH
NAME_TEMPLATE_OF_DATA_FRAMES=$NAME_TEMPLATE_OF_DATA_FRAMES
! REFERENCE_DATA_SET=xxx/XDS_ASCII.HKL ! e.g. to ensure consistent indexing
DATA_RANGE=1 $DATA_RANGE
SPOT_RANGE=1 $SPOT_RANGE
! BACKGROUND_RANGE=1 10 ! rather use defaults (first 5 degree of rotation)

SPACE_GROUP_NUMBER=0 ! 0 if unknown
UNIT_CELL_CONSTANTS= 70 80 90 90 90 90 ! put correct values if known
INCLUDE_RESOLUTION_RANGE=50 0 ! after CORRECT, insert high resol limit; re-run CORRECT

FRIEDEL'S_LAW=FALSE ! This acts only on the CORRECT step
! If the anom signal turns out to be, or is known to be, very low or absent,
! use FRIEDEL'S_LAW=TRUE instead (or comment out the line); re-run CORRECT

! remove the "!" in the following line:
! STRICT_ABSORPTION_CORRECTION=TRUE
! if the anomalous signal is strong: in that case, in CORRECT.LP the three
! "CHI^2-VALUE OF FIT OF CORRECTION FACTORS" values are significantly> 1, e.g. 1.5
!
! exclude (mask) untrusted areas of detector, e.g. beamstop shadow :
! UNTRUSTED_RECTANGLE= 1800 1950 2100 2150 ! x-min x-max y-min y-max ! repeat
! UNTRUSTED_ELLIPSE= 2034 2070 1850 2240 ! x-min x-max y-min y-max ! if needed
! UNTRUSTED_QUADRILATERAL= x1 y1 x2 y2 x3 y3 x4 y4 ! see documentation
!
! parameters with changes wrt default values:
TRUSTED_REGION=0.00 1.2 ! partially use corners of detectors; 1.41421=full use
VALUE_RANGE_FOR_TRUSTED_DETECTOR_PIXELS=7000. 30000. ! often 8000 is ok
STRONG_PIXEL=4 ! COLSPOT: only use strong reflections (default is 3)
MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT=3 ! default of 6 is sometimes too high
! close spots: reduce SEPMIN and CLUSTER_RADIUS from their defaults of 6 and 3, e.g. to 4 and 2
! for bad or low resolution data remove the "!" in the following line:
! REFINE(IDXREF)=CELL BEAM ORIENTATION AXIS ! DISTANCE
REFINE(INTEGRATE)=CELL BEAM ORIENTATION ! AXIS DISTANCE
! REFINE(CORRECT)=CELL BEAM ORIENTATION AXIS DISTANCE ! Default is: refine everything

! parameters specifically for this detector and beamline:
DETECTOR= $DETECTOR
SENSOR_THICKNESS= $SENSOR_THICKNESS
! attention CCD detectors: for very high resolution (better than 1A) make sure to specify SILICON
! as about 32* what CORRECT.LP suggests (absorption of phosphor is much higher than that of silicon)
NX= $NX NY= $NY QX= $QX QY= $QY ! to make CORRECT happy if frames are unavailable
DIRECTION_OF_DETECTOR_X-AXIS=1 0 0
DIRECTION_OF_DETECTOR_Y-AXIS=0 1 0
INCIDENT_BEAM_DIRECTION=0 0 1
ROTATION_AXIS=1 0 0 ! at e.g. Australian Synchrotron, SERCAT ID-22 (?), APS 19-ID (?) this needs to be -1 0 0
FRACTION_OF_POLARIZATION=0.98 ! better value is provided by beamline staff!
POLARIZATION_PLANE_NORMAL=0 1 0
eof
if [ "$DET" == "pilatus" ]; then
if [ $NX == "1475" ]; then
if ! grep -q FF_p2m0109_E12398_T6199_vrf_m0p20.tif tmp2 ; then
cat >> XDS.INP << eof
!EXCLUSION OF VERTICAL DEAD AREAS OF THE PILATUS 2M DETECTOR
UNTRUSTED_RECTANGLE= 486 496 0 1680
UNTRUSTED_RECTANGLE= 980 990 0 1680
!EXCLUSION OF HORIZONTAL DEAD AREAS OF THE PILATUS 2M DETECTOR
UNTRUSTED_RECTANGLE= 0 1476 194 214
UNTRUSTED_RECTANGLE= 0 1476 406 426
UNTRUSTED_RECTANGLE= 0 1476 618 638
UNTRUSTED_RECTANGLE= 0 1476 830 850
UNTRUSTED_RECTANGLE= 0 1476 1042 1062
UNTRUSTED_RECTANGLE= 0 1476 1254 1274
UNTRUSTED_RECTANGLE= 0 1476 1466 1486
eof
fi
# elif [ $NX == "2463" ]; then
# cat >> XDS.INP << eof
#eof
fi
fi
echo XDS.INP is ready for use. The file has only the most important keywords.
echo Full documentation, including complete detector templates, is at
echo http://www.mpimf-heidelberg.mpg.de/~kabsch/xds . More documentation in XDSwiki
echo After running xds, inspect, using XDS-Viewer, at least the beamstop mask in
echo BKGPIX.cbf, and the agreement of predicted and observed spots in FRAME.cbf!
rm -f tmp1 tmp2
</pre>

== System-wide or personal installation ==

Ask your system adminstrator to cut-and-paste the script into e.g. /usr/local/bin/generate_XDS.INP, and to make it "executable".

But you may also cut-and-paste the script from this webpage into a file in e.g. your home directory; the filename should be generate_XDS.INP. After creating the file, make it executable - e.g. if it's in your $HOME, use:
chmod +x ~/generate_XDS.INP
After that, you can just run it in a similar way as if it were installed in your $PATH:
~/generate_XDS.INP "frms/mydata_1_???.img"
By using your own file, you can easily update to the latest revision, or even change the script, without having to bother the system administrator.

== Generating generate_XDS.INP from this webpage ==

Instead of cutting-and-pasting the lines of the script, you (or the system administrator) could just cut-and-paste the following four lines
<pre>
wget http://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/generate_XDS.INP -O - | \
sed -e s/\&nbsp\;/\ /g -e s/\&gt\;/\>/g -e s/\&lt\;/\</g -e s/amp\;//g -e s/\&quot\;/\"/g -e s/\&\#\1\6\0\;/\ /g | \
awk '/^#/,/rm -f tmp1 tmp2/' > generate_XDS.INP
chmod +x generate_XDS.INP
</pre>
to copy the script from this website into an executable file generate_XDS.INP in your current directory.

==Variant script (xds_generate_all)==
This script includes some minor modifications of the Generate_XDS.INP script - for instance, it tells the user the energy, as well as the wavelength of the X-rays, and gives the user the beam center in mm as well as pixels - useful when switching between HKL2000, MOSFLM and XDS.

It also generates two executable shell scripts in the same directory as XDS.INP.

The first of these scripts, xds_graph.sh, extracts various statistics from INTEGRATE.LP and CORRECT.LP, pipes them out to a logfile, INTEGRATE_STATS.LP, and plots them using loggraph. This script does not have some of the more useful features of XDSSTAT, such as calculation of per frame Rmeas values, but it does give a quick overview of various data quality parameters by resolution and image number. This script will only work with recent versions of XDS (after calculation of CC(1/2) - not sure exactly which version).

The second of these scripts, xds_to_ccp4.sh, will take XDS_ASCII.HKL and generate a CCP4-format MTZ file, containing anomalous data (as F(+)/F(-) and DANO/SigDANO) and a test set of 5% of reflections for calculation of the free R-factor. It will also generate an unmerged SHELX format HKL file, for input into SHELXC/D/E (e.g. via the HKL2MAP GUI).

Here is the script:
<pre>
#!/bin/bash
# purpose: xds_generate_all
#
# tested with some datasets from ALS, SSRL, SLS, ESRF and BESSY; only MARCCD, ADSC/SMV, PILATUS detectors;
# for other detectors, values marked with XXX must be manually filled in.
#
# revision 0.03 . Kay Diederichs 2/2010
# revision 0.04 . Kay Diederichs 4/2010 - include alternative ORGX, ORGY calculations for ADSC
# revision 0.05 . Kay Diederichs 5/2010 - grep for "Corrected" in addition to "marccd"; needed for BESSY
# revision 0.06 . KD 6/2010 - add UNTRUSTED_RECTANGLE and UNTRUSTED_ELLIPSE; use `whereis catmar` and so on
# revision 0.07 . KD 6/2010 - decide about ORGX/Y info in MAR header being pixels or mm; other fixes
# revision 0.08 . KD 6/2010 - fixes for Pilatus 6M
# revision 0.09 . KD 6/2010 - get rid of requirement for mccd_xdsparams.pl and/or catmar; rather use "od"
# revision 0.10 . Tim Gruene 7/2010 - set link 'images' to image directory if path exceeds 72 characters
# revision 0.11 . KD 7/2010 - for MarCCD: look for distance info at different byte position
# revision 0.12 . KD 7/2010 - fix for negative PHISTART
# revision 0.13 . KD 8/2010 - store correct NX NY QX QY in XDS.INP
# revision 0.14 . KD 1/2011 - SENSOR_THICKNESS for Pilatus; MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT=3
# revision 0.15 . KD 2/2011 - add comment for -ive sign of APS 19-ID and Australian Synchrotron rotation axis
# revision 0.16 . KD 3/2011 - SENSOR_THICKNESS=0.01 for ADSC and MarCCD. Add comment about SILICON=
# revision 0.17 . KD 3/2011 - make it work for .bz2 frames; improve screen output
# revision 0.18 . KD 4/2011 - faster by doing "strings" only once; revert "images/${1##/*/}" "correction"
# revision 0.19 . KD 6/2011 - bugfix for 0.18
# revision 0.20 . KD 7/2011 - redirect stderr of /bin/ls to /dev/null
# revision 0.21 . KD 11/2011 - SEPMIN, CLUSTER_RADIUS hints; read NX NY from header (for Pilatus 2M)
# revision 0.22 . KD 12/2011 - Pilatus 2M UNTRUSTED_RECTANGLE lines, SENSOR_THICKNESS from header
# revision 0.23 . KD 1/2012 - add UNTRUSTED_QUADRILATERAL, remove MINIMUM_ZETA (0.05 is default now)
# revision 0.24 . KD 3/2012 - remove revision 0.10 since XDS now takes much longer paths
# revision 0.25 . KD 3/2012 - remove revision 0.22 for PSI Pilatus 2M; see http://www.globalphasing.com/autoproc/wiki/index.cgi?TroubleShootingKnownIssues
# revision 0.26 Oliver Clarke 10/12 - Added generation of shell scripts for conversion to MTZ and visualisation of statistics. Other minor alterations to XDS.INP.
REVISION="0.26 (18-Oct-2012)"
#
# usage: e.g. xds_generate_all "frms/mydata_1_???.img"
# make sure to have the two quotation marks !
# the ? are wildcards for the frame numbers.
#
# limitations:
# - frame numbers are assumed to start with 1 and run consecutively
#
# known problems:
# - for ADSC detectors, there are at least three ways to obtain ORGX and ORGY values from the header (see below);
# - the same might be a problem for MAR headers, too (not sure about this)
#
# notes for debugging of the script:
# - add the -v option to the first line, to see where an error occurs
# - comment out the removal of tmp1 and tmp2 in the last line
#
# ====== Start of script ======
echo generate_XDS.INP version $REVISION . Obtain the latest version from
echo http://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/generate_XDS.INP
if [ "$1" == "help" ] || [ "$1" == "-help" ] || [ "$1" == "-h" ]; then
echo usage: xds_generate_all \"frms/mydata_1_???.img\" $_with_ the quotation marks!$
echo if the frames are compressed with bzip2, leave out the .bz2 extension!
exit
fi
#
# defaults:
#
DETECTOR="XXX MINIMUM_VALID_PIXEL_VALUE=XXX OVERLOAD=XXX"
ORGX=XXX
ORGY=XXX
ORGXMM=XXX
ORGYMM=XXX
DETECTOR_DISTANCE=XXX
OSCILLATION_RANGE=XXX
X_RAY_WAVELENGTH=XXX
X_RAY_ENERGY=XXX
QX=XXX
QY=XXX
NX=XXX
NY=XXX
NXMM=XXX
NYMM=XXX
SENSOR_THICKNESS=0
# see how we are called:
NAME_TEMPLATE_OF_DATA_FRAMES="$1"
# list frames matching the wildcards in NAME_TEMPLATE_OF_DATA_FRAMES
# don't accept the "direct beam" shot at SLS/Pilatus PX-I and PX-II
/bin/ls -C1 $1 $1.bz2 2>/dev/null | egrep -v "_00000.cbf|_000.img" > tmp1 || exit 1

# we can continue - the frames are found

# set upper limit of DATA_RANGE to number of frames (see "limitations" above)
DATA_RANGE=`wc -l tmp1 | awk '{print $1}'`

# set upper limit of SPOT_RANGE to half of DATA_RANGE, but not less than 1
SPOT_RANGE=`echo "scale=0; $DATA_RANGE/2" | bc -l`
SPOT_RANGE=`echo "if ($SPOT_RANGE<1) 1;if ($SPOT_RANGE>1) $SPOT_RANGE" | bc -l`

echo DATA_RANGE=1 $DATA_RANGE

# find out detector type
DET=XXX
FIRSTFRAME=`head -1 tmp1`
echo $FIRSTFRAME | grep -q bz2 && bzcat $FIRSTFRAME > tmp1 && FIRSTFRAME=tmp1
strings $FIRSTFRAME > tmp2
egrep -q 'marccd|Corrected' tmp2 && DET=mccd
grep -q PILATUS tmp2 && DET=pilatus
grep -q BEAM_CENTER_X tmp2 && DET=adsc
# identify other detector types in the same way (MAR IP would be straightforward)

# parse ASCII header of first frame

if [ "$DET" == "XXX" ]; then
echo "this is not a MAR, ADSC/SMV or PILATUS detector - fill in XXX values manually!"
DETECTOR="XXX MINIMUM_VALID_PIXEL_VALUE=XXX OVERLOAD=XXX"

# find parameters of first frame
elif [ "$DET" == "mccd" ]; then
echo Data from a MarCCD detector

DETECTOR="CCDCHESS MINIMUM_VALID_PIXEL_VALUE= 1 OVERLOAD= 65500"
SENSOR_THICKNESS=0.01
# use first frame of dataset to obtain parameters

# offsets are documented; values can be find in mccd_xdsparams.pl script
let SKIP=1024+80
NX=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
let SKIP=$SKIP+4
NY=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')

let SKIP=1720
DETECTOR_DISTANCE=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
DETECTOR_DISTANCE=`echo "scale=3; $DETECTOR_DISTANCE/1000" | bc -l`

let SKIP=1024+256+128+256+4
ORGX=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
ORGX=`echo "scale=2; $ORGX/1000" | bc -l `
let SKIP=$SKIP+4
ORGY=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
ORGY=`echo "scale=2; $ORGY/1000" | bc -l `

let SKIP=1024+256+128+256+44
PHISTART=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
let SKIP=1024+256+128+256+76
PHIEND=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
OSCILLATION_RANGE=`echo "scale=3; ($PHIEND-($PHISTART))/1000" | bc -l`

let SKIP=1024+256+128+256+128+4
QX=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
QX=`echo "scale=10; $QX/1000000" |bc -l `
let SKIP=$SKIP+4
QY=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
QY=`echo "scale=10; $QY/1000000" |bc -l `

let SKIP=1024+256+128+256+128+128+12
X_RAY_WAVELENGTH=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
X_RAY_WAVELENGTH=`echo "scale=5; $X_RAY_WAVELENGTH/100000" | bc -l`
X_RAY_ENERGY=`echo "12398.5/$X_RAY_WAVELENGTH" | bc -l | awk '{printf "%5.1f", $1}'`
# at most BLs, ORGX and ORGY are in pixels, but sometimes in mm ... guess:
NXBYFOUR=`echo "scale=0; $NX/4" | bc -l `
ORGXINT=`echo "scale=0; $ORGX/1" | bc -l `
if [ $ORGXINT -lt $NXBYFOUR ]; then
ORGXMM=$ORGX
ORGYMM=$ORGY
NXMM=`echo "$NX*$QX" | bc -l | awk '{printf "%3.2f", $1}'`
NYMM=`echo "$NY*$QY" | bc -l | awk '{printf "%3.2f", $1}'`
ORGX=`echo "scale=1; $ORGX/$QX" | bc -l`
ORGY=`echo "scale=1; $ORGY/$QY" | bc -l`
echo MARCCD detector: header ORGX, ORGY seem to be in mm ... converting to pixels
else
ORGXMM=`echo "$ORGX*$QX" | bc -l | awk '{printf "%3.2f", $1}'`
ORGYMM=`echo "$ORGY*$QY" | bc -l | awk '{printf "%3.2f", $1}'`
NXMM=`echo "$NX*$QX" | bc -l | awk '{printf "%3.2f", $1}'`
NYMM=`echo "$NY*$QY" | bc -l | awk '{printf "%3.2f", $1}'`
echo MARCCD detector: header ORGX, ORGY seem to be in pixel units
fi

elif [ "$DET" == "adsc" ]; then

DETECTOR="ADSC MINIMUM_VALID_PIXEL_VALUE= 1 OVERLOAD= 65000"
echo Data from ADSC detector. Obtaining ORGX, ORGY depends on beamline setup:
SENSOR_THICKNESS=0.01
sed s/\;// tmp2 > tmp1
mv tmp1 tmp2

# find X_RAY_WAVELENGTH:
X_RAY_WAVELENGTH=`grep WAVELENGTH tmp2 | head -1 | sed s/WAVELENGTH=//`

# find NX, QX, ORGX and ORGY:
X_RAY_ENERGY=`echo "12398.5/$X_RAY_WAVELENGTH" | bc -l | awk '{printf "%5.1f", $1}'`
NX=`grep SIZE1 tmp2 | tail -1 | sed s/SIZE1=//`
QX=`grep PIXEL_SIZE tmp2 | sed s/PIXEL_SIZE=//`
# FIXME - next 2 lines should be done properly, from header
NY=$NX
QY=$QX
NXMM=`echo "$NX*$QX" | bc -l | awk '{printf "%3.2f", $1}'`
NYMM=`echo "$NY*$QY" | bc -l | awk '{printf "%3.2f", $1}'`
BEAM_CENTER_X=`grep BEAM_CENTER_X tmp2 | sed s/BEAM_CENTER_X=//`
BEAM_CENTER_Y=`grep BEAM_CENTER_Y tmp2 | sed s/BEAM_CENTER_Y=//`
# fix 2010-04-26 - tell user about possible ORGX, ORGY alternatives -
# at ESRF and ... (pls fill in!) the following should be used:
ORGX=`echo "scale=1; $BEAM_CENTER_Y/$QX" | bc -l`
ORGY=`echo "scale=1; $BEAM_CENTER_X/$QX" | bc -l`
echo - at ESRF BLs use: ORGX=$ORGX ORGY=$ORGY
# this 2nd alternative convention should be used at the following beamlines (pls complete the list): ALS 5.0.3, ...
ORGX=`echo "scale=1; $NX-$BEAM_CENTER_X/$QX" | bc -l `
ORGY=`echo "scale=1; $BEAM_CENTER_Y/$QX" | bc -l `
echo - at e.g. ALS 5.0.3 use: ORGX=$ORGX ORGY=$ORGY
# this 3rd alternative convention should be used at the following beamlines (pls complete the list): ALS 8.2.2, ...
ORGX=`echo "scale=1; $BEAM_CENTER_X/$QX" | bc -l `
ORGY=`echo "scale=1; $NX-$BEAM_CENTER_Y/$QX" | bc -l `
ORGXMM=`echo "$ORGX*$QX" | bc -l | awk '{printf "%3.2f", $1}'`
ORGYMM=`echo "$ORGY*$QY" | bc -l | awk '{printf "%3.2f", $1}'`
echo - at e.g. ALS 8.2.2 use: ORGX=$ORGX ORGY=$ORGY - this is written to XDS.INP
# the latter alternative is written into the generated XDS.INP ! You have to correct this manually in XDS.INP, or adjust this script.
# find DETECTOR_DISTANCE and OSCILLATION_RANGE:
DETECTOR_DISTANCE=`grep DISTANCE tmp2 | sed s/DISTANCE=//`
OSCILLATION_RANGE=`grep OSC_RANGE tmp2 | sed s/OSC_RANGE=//`

elif [ "$DET" == "pilatus" ]; then
DETECTOR="PILATUS MINIMUM_VALID_PIXEL_VALUE=0 OVERLOAD= 1048576 !PILATUS"
QX=0.172 QY=0.172
echo Data from a Pilatus detector
sed s/#// tmp2 > tmp1
mv tmp1 tmp2

# find SENSOR_THICKNESS:
SENSOR_THICKNESS=`grep thickness tmp2 | sed -e s/'Silicon sensor, thickness'// | awk '{print $1*1000}'`
# find X_RAY_WAVELENGTH:
X_RAY_WAVELENGTH=`grep Wavelength tmp2 | sed -e s/Wavelength// -e s/A// | awk '{print $1}'`

X_RAY_ENERGY=`echo "12398.5/$X_RAY_WAVELENGTH" | bc -l | awk '{printf "%5.1f", $1}'`
# find NX and NY; 2463/2527 is 6M, 1475/1679 is 2M
NX=`grep X-Binary-Size-Fastest-Dimension tmp2 | awk '{print $2}'`
NY=`grep X-Binary-Size-Second-Dimension tmp2 | awk '{print $2}'`
NXMM=`echo "$NX*$QX" | bc -l | awk '{printf "%3.2f", $1}'`
NYMM=`echo "$NY*$QY" | bc -l | awk '{printf "%3.2f", $1}'`
# find ORGX and ORGY:
ORGX=`grep Beam_xy tmp2 | sed -e s/$// -e s/$// -e s/\,// | awk '{print $2}'`
ORGY=`grep Beam_xy tmp2 | sed -e s/$// -e s/$// -e s/\,// | awk '{print $3}'`
ORGXMM=`echo "$ORGX*$QX" | bc -l | awk '{printf "%3.2f", $1}'`
ORGYMM=`echo "$ORGY*$QY" | bc -l | awk '{printf "%3.2f", $1}'`
# find DETECTOR_DISTANCE and OSCILLATION_RANGE:
DETECTOR_DISTANCE=`awk '/distance/{print $2}' tmp2`
DETECTOR_DISTANCE=`echo "$DETECTOR_DISTANCE*1000" | bc -l`

OSCILLATION_RANGE=`awk '/Angle/{print $2}' tmp2`

else
echo should never come here
exit 1
fi

echo ORGX= $ORGX ORGY= $ORGY - check these values with adxv !
echo DETECTOR_DISTANCE= $DETECTOR_DISTANCE
echo OSCILLATION_RANGE= $OSCILLATION_RANGE
echo X-RAY_WAVELENGTH= $X_RAY_WAVELENGTH
# now we know everything that is required to generate XDS.INP

cat > XDS.INP << eof
!****************************************************************************************
! Written by generate_XDS.INP version $REVISION
!
! A few notes on usage and modification of this input file:
!
! If XDS fails after IDXREF with an error indicating that
! an insufficient percentage of reflections were indexed,
! but you are confident that the indexing solution is
! correct, rerun XDS with the JOB keyword changed to
! "DEFPIX INTEGRATE CORRECT".
!
! After the first round of integration (in a triclinic cell)
! it is often helpful to reintegrate using the globally
! refined cell parameters and the correct space group.
! You can do this by renaming GXPARM.XDS to XPARM.XDS
! - e.g. by the command "cp GXPARM.XDS XPARM.XDS" -
! and rerunning DEFPIX, INTEGRATE and CORRECT.
!
! If you are trying to determine the optimal starting point for
! data collection starting from a single image, first complete
! a run of XDS using default parameters. Then, rerun XDS using
! the correct cell and spacegroup as described above
! ("cp GXPARM.XDS XPARM.XDS"), but on the second run of XDS, give
! the JOB keyword as "JOB=DEFPIX XPLAN".
!****************************************************************************************
!
!****************************************************************************************
!General parameters:
!****************************************************************************************

JOB= XYCORR INIT COLSPOT IDXREF DEFPIX INTEGRATE CORRECT

ORGX= $ORGX ORGY= $ORGY ! check these values with adxv!
!In mm, ORGX=$ORGXMM and ORGY=$ORGYMM

DETECTOR_DISTANCE= $DETECTOR_DISTANCE
OSCILLATION_RANGE= $OSCILLATION_RANGE

X-RAY_WAVELENGTH= $X_RAY_WAVELENGTH
!X-ray energy is $X_RAY_ENERGY eV (calculated from wavelength)

NAME_TEMPLATE_OF_DATA_FRAMES=$NAME_TEMPLATE_OF_DATA_FRAMES

!REFERENCE_DATA_SET=xxx/XDS_ASCII.HKL ! e.g. to ensure consistent indexing.
!XDS will *only* use the reference dataset if the space group and cell
!parameters are specified below.

DATA_RANGE=1 $DATA_RANGE
SPOT_RANGE=1 $SPOT_RANGE

!BACKGROUND_RANGE=1 10 !Default is first five degrees.

SPACE_GROUP_NUMBER=0 ! 0 if unknown
UNIT_CELL_CONSTANTS= 70 80 90 90 90 90 ! put correct values if known

INCLUDE_RESOLUTION_RANGE=50 0 ! After CORRECT, insert high resol limit; re-run CORRECT
! Ice rings can be excluded from scaling with the keyword
! EXCLUDE_RESOLUTION_RANGE, e.g. EXCLUDE_RESOLUTION_RANGE=2.28 2.22

!SECONDS=60 !Uncomment this line to tell XDS to wait 1min for the next image before aborting.

!****************************************************************************************
!Parameters important for processing anomalous data:
!****************************************************************************************

FRIEDEL'S_LAW=FALSE ! This acts only on the CORRECT step. Even if the anomalous signal
! in your dataset is very small, it still may be useful for
! calculating an anomalous difference map.

STRICT_ABSORPTION_CORRECTION=FALSE !Change this to TRUE if the anomalous signal
! is strong: in that case, in CORRECT.LP the three
! "CHI^2-VALUE OF FIT OF CORRECTION FACTORS" values
! will be significantly> 1, e.g. 1.5.

WFAC1=1.0 !This parameter controls rejection of misfits during scaling. Sometimes,
! strong anomalous pairs may be rejected as misfits, in which case increasing
! WFAC1 (e.g. to 1.5) may improve anomalous signal.

!****************************************************************************************
!Regions of the detector to be excluded during indexing and integration:
!****************************************************************************************
! exclude (mask) untrusted areas of detector, e.g. beamstop shadow :
!UNTRUSTED_RECTANGLE= !1800 1950 2100 2150 ! x-min x-max y-min y-max ! repeat
!UNTRUSTED_ELLIPSE= !2034 2070 1850 2240 ! x-min x-max y-min y-max ! if needed
!UNTRUSTED_QUADRILATERAL= !x1 y1 x2 y2 x3 y3 x4 y4 ! see documentation

TRUSTED_REGION=0.00 1.2 ! Partially use corners of detectors; 1.41421=full use, 1.0=edge
! It is important to note that TRUSTED_REGION is the only resolution
! cutoff obeyed during indexing and integration;
! Both INCLUDE_RESOLUTION_RANGE and EXCLUDE_RESOLUTION_RANGE are
! ignored during IDXREF and INTEGRATE. So this parameter can be
! important when indexing or integration steps fail.

!****************************************************************************************
!Other parameters:
!****************************************************************************************
VALUE_RANGE_FOR_TRUSTED_DETECTOR_PIXELS=7000. 30000. ! Often 8000 is ok.
STRONG_PIXEL=4 ! COLSPOT: only use strong reflections (default is 3; 6 may be better for strong data)
MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT=3 ! default of 6 is sometimes too high
MAXIMUM_ERROR_OF_SPOT_POSITION=3 ! Increasing may help for poor quality data

MINPK=75 !Increase (e.g. MINPK=98) to improve data accuracy at the expense of completeness.

DELPHI=5 !Size of integration wedge. Increasing to 10 or 20 may be helpful in some cases.

NUMBER_OF_PROFILE_GRID_POINTS_ALONG_ALPHA/BETA=13 ! Default is 9 - Increasing may improve data
NUMBER_OF_PROFILE_GRID_POINTS_ALONG_GAMMA=13 ! accuracy, particularly if finely-sliced on phi,
! and does not seem to have any downsides.

MINIMUM_ZETA=0.05 ! Controls how close to the blind region (about phi) reflections should be
! integrated. 0.05 is the default; increasing MINIMUM_ZETA to 0.15 *may*
! improve data quality by removing unreliable reflections near the phi
! axis, but will reduce completeness in low symmetry space groups.

SEPMIN=6 ! Reduce if spots are close together (due to a long axis). Consider
CLUSTER_RADIUS=3 ! increasing if crystal is split and unit cell dimensions are relatively short.

! For bad or low resolution data remove the "!" in the following line (default is ALL):
! REFINE(IDXREF)=CELL BEAM ORIENTATION AXIS ! DISTANCE
REFINE(INTEGRATE)=CELL BEAM ORIENTATION ! AXIS DISTANCE (If integration is unstable, comment out this line.
! REFINE(CORRECT)=CELL BEAM ORIENTATION AXIS DISTANCE ! Default is: refine everything

ROTATION_AXIS=1 0 0 !At e.g. Australian Synchrotron, SERCAT ID-22 (?), APS 19-ID (?) this needs to be -1 0 0

!****************************************************************************************
! Parameters specifically for this detector and beamline (shouldn't need changing):
!****************************************************************************************
DETECTOR= $DETECTOR
SENSOR_THICKNESS= $SENSOR_THICKNESS
! attention CCD detectors: for very high resolution (better than 1A) make sure to specify SILICON
! as about 32* what CORRECT.LP suggests (absorption of phosphor is much higher than that of silicon)

NX= $NX NY= $NY QX= $QX QY= $QY ! to make CORRECT happy if frames are unavailable
!In mm, NX=$NXMM and NY=$NYMM

DIRECTION_OF_DETECTOR_X-AXIS=1 0 0
DIRECTION_OF_DETECTOR_Y-AXIS=0 1 0
INCIDENT_BEAM_DIRECTION=0 0 1
FRACTION_OF_POLARIZATION=0.98 ! better value is provided by beamline staff!
POLARIZATION_PLANE_NORMAL=0 1 0
eof
if [ "$DET" == "pilatus" ]; then
if [ $NX == "1475" ]; then
if ! grep -q FF_p2m0109_E12398_T6199_vrf_m0p20.tif tmp2 ; then
cat >> XDS.INP << eof
!EXCLUSION OF VERTICAL DEAD AREAS OF THE PILATUS 2M DETECTOR
UNTRUSTED_RECTANGLE= 486 496 0 1680
UNTRUSTED_RECTANGLE= 980 990 0 1680
!EXCLUSION OF HORIZONTAL DEAD AREAS OF THE PILATUS 2M DETECTOR
UNTRUSTED_RECTANGLE= 0 1476 194 214
UNTRUSTED_RECTANGLE= 0 1476 406 426
UNTRUSTED_RECTANGLE= 0 1476 618 638
UNTRUSTED_RECTANGLE= 0 1476 830 850
UNTRUSTED_RECTANGLE= 0 1476 1042 1062
UNTRUSTED_RECTANGLE= 0 1476 1254 1274
UNTRUSTED_RECTANGLE= 0 1476 1466 1486
eof
fi
# elif [ $NX == "2463" ]; then
# cat >> XDS.INP << eof
#eof
fi
fi
touch xds_to_ccp4.sh
echo "#!/bin/bash
#Removes old log file if present
rm xds_to_ccp4.log
#Makes new log file
touch xds_to_ccp4.log
#Pipe stdout and stderr to both logfile and tty
tail -f -n1 xds_to_ccp4.log &
exec > xds_to_ccp4.log 2>&1
echo \"

If this message is all you see,
something has probably gone wrong.
Are you sure XDS_ASCII.HKL is present
in the current directory?

\"
#Make first MTZ file (with DANO/SIGDANO)
echo \"INPUT_FILE=XDS_ASCII.HKL
OUTPUT_FILE=temp.hkl CCP4
MERGE=TRUE
GENERATE_FRACTION_OF_TEST_REFLECTIONS=0.05
FRIEDEL'S_LAW=FALSE\" > XDSCONV.INP
xdsconv
f2mtz HKLOUT temp.mtz<F2MTZ.INP

#Make second MTZ file (with F(+)/F(-))
echo \"INPUT_FILE=XDS_ASCII.HKL
OUTPUT_FILE=temp2.hkl CCP4_F
MERGE=TRUE
GENERATE_FRACTION_OF_TEST_REFLECTIONS=0.05
FRIEDEL'S_LAW=FALSE\" > XDSCONV.INP
xdsconv
f2mtz HKLOUT temp2.mtz<F2MTZ.INP

#Make SHELX format unmerged hkl file
echo \"INPUT_FILE=XDS_ASCII.HKL
OUTPUT_FILE=xds_shelx.hkl SHELX
MERGE=FALSE
FRIEDEL'S_LAW=FALSE\" > XDSCONV.INP
xdsconv

#cad the two previously generated MTZ files together
cad HKLIN1 temp.mtz HKLIN2 temp2.mtz HKLOUT xds_ccp4_merged.mtz<<EOF
LABIN FILE 1 E1=FP E2=SIGFP E3=DANO E4=SIGDANO E5=FreeRflag
LABIN FILE 2 E1=F(+) E2=SIGF(+) E3=F(-) E4=SIGF(-)
END
EOF

#Remove temporary files
rm temp.mtz
rm temp2.mtz
rm temp.hkl
rm temp2.hkl
rm F2MTZ.INP

echo \"

********************************XDS_TO_CCP4**********************************

This script takes XDS_ASCII.HKL and uses XDSCONV and F2MTZ to
generate a merged CCP4-format MTZ file named
\\\"xds_ccp4_merged.mtz\\\" with a test (free) set
labeled \\\"FreeRflag\\\" constituting 5% of reflections.

The space group of the output MTZ file will be the same as
that present in XDS_ASCII.HKL. It is probably worth checking
that this is the correct space group - either manually inspect
the systematic absences in CORRECT.LP, or run XDS_ASCII.HKL
through the CCP4 program POINTLESS (\\\"pointless xdsin XDS_ASCII.HKL\\\")

Anomalous data is kept, if present, and output as both
DANO/SIGDANO and F(+)/F(-).

This script also generates an unmerged SHELX hkl file,
suitable for input into SHELXC/D/E, named \\\"xds_shelx.hkl\\\".

A log of all processes is present in xds_to_ccp4.log.

Parts of this script have been shamelessly copied from
the XDS Wiki, at http://www.strucbio.biologie.uni-konstanz.de/xdswiki/

******************************************************************************
\"" > xds_to_ccp4.sh
touch xds_graph.sh
echo "#!/bin/bash
#Makes new log file
touch INTEGRATE_STATS.LP
#Pipe stdout and stderr to both logfile and tty
tail -f -n1 INTEGRATE_STATS.LP &
exec > INTEGRATE_STATS.LP 2>&1
echo \"
\\\$TABLE: XDS Integration Statistics :
\\\$SCATTER:Scale factors by image:A:1,3: :Number of overloads by image:A:1,5: :Number of strong reflections by image:A:1,7: :Number of rejects by image:A:1,8: :Mosaicity by image:A:1,10: \\$\\$
IMAGE IER SCALE NBKG NOVL NEWALD NSTRONG NREJ SIGMAB SIGMAR
\\$\\$ \\$\\$
\"
egrep \"[0-9]\.[0-9][0-9][0-9][0-9] [0-9]\.[0-9][0-9][0-9][0-9]\" INTEGRATE.LP
echo \" \\$\\$ \"
echo \"
\\\$TABLE: XDS Scaling Statistics by Resolution (X-axis is resolution in Angstroems) :
\\\$GRAPHS:Chi-Squared by resolution:A:2,4: :Observed and expected R-factors by resolution (as %):A:2,5,6: :Rejects by resolution:A:2,9: :Rejects by resolution as a percentage of total reflections:A:2,10: \\$\\$
LOW_RES Resolution(Angstroems) I/Sigma Chi^2 R-FACTOR(OBSERVED) R-FACTOR(EXPECTED) NUMBER_OF_REFLECTIONS ACCEPTED REJECTS REJECT_PERCENT
\\$\\$ \\$\\$
\"
awk '/RESOLUTION RANGE I\/Sigma Chi\^2 R\-FACTOR R\-FACTOR NUMBER ACCEPTED REJECTED/, /\-\-\-\-\-\-\-\-\-/' CORRECT.LP | egrep -v \"[a-z,A-Z,] | \-\-|\-99\.9\" | egrep \"[0-9]\" | awk '{print \$1,\$2,\$3,\$4,\$5,\$6,\$7,\$8,\$9,(\$9/(\$8+\$9))*100}'
echo \" \\$\\$\"
echo \"
\\\$TABLE: Overall data quality by Resolution (X-axis is resolution in Angstroems) :
\\\$GRAPHS:Completeness vs resolution :A:1,6: :Redundancy vs resolution :A:1,4: :I/Sigma by resolution:A:1,10: :Rmeas by resolution:A:1,11: :Anomalous correlation by resolution:A:1,13: :Significance of the anomalous signal by resolution:A:1,14: \\$\\$
Resolution(Angstroems) OBSERVED_REFLECTIONS UNIQUE_REFLECTIONS REDUNDANCY POSSIBLE_REFLECTIONS COMPLETENESS R-FACTOR(OBSERVED) R-FACTOR(EXPECTED) COMPARED I/SIGMA R-meas CC(1/2) Anomalous_Correlation SigAno Nano
\\$\\$ \\$\\$
\"
egrep -B25 \" WILSON STATISTICS \" CORRECT.LP | egrep -v \"[a-z,A-Z]|\*\*\*\" | egrep \" \" | awk '{ gsub(/[%*]/,\" \"); print }' | awk '{print \$1,\$2,\$3,\$2/\$3,\$4,\$5,\$6,\$7,\$8,\$9,\$10,\$11,\$12,\$13,\$14}'
echo \" \\$\\$ \"

echo \"
\\\$TABLE: I/Sigma of h00 reflections (for identification of systematic absences):
\\\$GRAPHS:I/Sigma vs h :A:1,5: \\$\\$
h k l RES I/SIGMA NUMBER
\\$\\$ \\$\\$
\"
awk '/REFLECTIONS OF TYPE H/,/COMPLETENESS AND QUALITY/' CORRECT.LP | egrep \"[1-9]\" | awk '{print \$1,\$2,\$3,\$4,\$7,\$8}' | awk '(\$1 > 0)'
echo \"\\$\\$\"

echo \"
\\\$TABLE: I/Sigma of 0k0 reflections (for identification of systematic absences):
\\\$GRAPHS:I/Sigma vs k :A:2,5: \\$\\$
h k l RES I/SIGMA NUMBER
\\$\\$ \\$\\$
\"
awk '/REFLECTIONS OF TYPE H/,/COMPLETENESS AND QUALITY/' CORRECT.LP | egrep \"[1-9]\" | awk '{print \$1,\$2,\$3,\$4,\$7,\$8}' | awk '(\$2 > 0)'
echo \"\\$\\$\"

echo \"
\\\$TABLE: I/Sigma of 00l reflections (for identification of systematic absences):
\\\$GRAPHS:I/Sigma vs l :A:3,5: \\$\\$
h k l RES I/SIGMA NUMBER
\\$\\$ \\$\\$
\"
awk '/REFLECTIONS OF TYPE H/,/COMPLETENESS AND QUALITY/' CORRECT.LP | egrep \"[1-9]\" | awk '{print \$1,\$2,\$3,\$4,\$7,\$8}' | awk '(\$3 > 0)'
echo \"\\$\\$\"

echo \"
\\\$TABLE:Beam center (ORGX/ORGY) plotted vs wedge (DELPHI):
\\\$GRAPHS:ORGX and ORGY vs batch (DELPHI) :A:1,2,3: \\$\\$
N ORGX ORGY
\\$\\$ \\$\\$
\"
egrep \"DETECTOR ORIGIN\" INTEGRATE.LP | grep -n \"\" | awk '{ gsub(/[a-z,A-Z,),(,:]/,\"\"); print }'
echo \"\\$\\$\"

echo \"
\\\$TABLE:Unit cell parameters plotted vs wedge:
\\\$GRAPHS:Unit cell lengths a, b and c plotted vs wedge:A:1,2,3,4: :Unit cell angles alpha, beta and gamma plotted vs wedge:A:1,5,6,7: \\$\\$
N a b c ALPHA BETA GAMMA
\\$\\$ \\$\\$
\"
egrep \"UNIT CELL PARAMETERS\" INTEGRATE.LP | grep -n \"\" | awk '{ gsub(/[a-z,A-Z,),(,:]/,\"\"); print }'
echo \"\\$\\$\"

echo \"
\\\$TABLE:Standard deviation of spot position (pixels) plotted vs wedge:
\\\$GRAPHS:Standard deviation of spot position plotted vs wedge:A:1,2: \\$\\$
N SD
\\$\\$ \\$\\$
\"
egrep \"SPOT POSITION\" INTEGRATE.LP | grep -n \"\" | awk '{ gsub(/[a-z,A-Z,),(,:]/,\"\"); print }'
echo \"\\$\\$\"

echo \"
\\\$TABLE:Detector distance (mm) plotted vs wedge (DELPHI):
\\\$GRAPHS:Detector distance plotted vs batch:A:1,2: \\$\\$
N DIST
\\$\\$ \\$\\$
\"
egrep \"DETECTOR DISTANCE \$mm\$\" INTEGRATE.LP | grep -n \"\" | awk '{ gsub(/[a-z,A-Z,),(,:]/,\"\"); print }'

echo \"\\$\\$\"

loggraph INTEGRATE_STATS.LP">xds_graph.sh
chmod +x xds_to_ccp4.sh
chmod +x xds_graph.sh
echo "XDS.INP is ready for use. The file has only the most important keywords.
Full documentation, including complete detector templates, is at
http://www.mpimf-heidelberg.mpg.de/~kabsch/xds . More documentation in XDSwiki
After running xds, inspect, using XDS-Viewer, at least the beamstop mask in
BKGPIX.cbf, and the agreement of predicted and observed spots in FRAME.cbf!

Two shell scripts have been generated that may be of use.

The first, xds_graph.sh, will plot various statistics after an XDS run.
After both INTEGRATE and CORRECT have finished, run it by typing \"./xds_graph.sh\"
without the quotation marks. This script uses loggraph to plot data, so you will
need CCP4 to see the graphical output. All the raw data is piped out to a log file,
INTEGRATE_STATS.LP.

The second script, xds_to_ccp4.sh, will take XDS_ASCII.HKL and generate both a CCP4
format MTZ file with anomalous data retained (as F(+)/F(-) and DANO/SigDANO)
and an unmerged SHELX format hkl file for input to SHELXC/D/E. Run it after CORRECT
by typing \"./xds_to_ccp4.sh\" at the prompt."

rm -f tmp1 tmp2
</pre>

Use this script in the same manner as Generate_XDS.INP. The two shell scripts created should be executable; If they are not, then make them so (chmod +x ./xxx.sh).

Generate XDS.INP

2012-10-18T16:35:53Z

Olibclarke:

This script generates XDS.INP based on a list of frame names supplied on the commandline. It currently works for MarCCD, ADSC and Pilatus detectors; since this is just a bash script, extension to other detectors is very easy.

On Mac OS X, installation of the "Command Line Tools" (from http://developer.apple.com/downloads; requires Apple ID) is required (I think). These are a part of the (larger, but also free) [http://developer.apple.com/tools/xcode Xcode] package.

== Usage ==

Usage is just (don't forget the quotation marks!):
generate_XDS.INP "frms/mydata_1_???.img"
XDS [http://strucbio.biologie.uni-konstanz.de/~dikay/XDS_html_doc/html_doc/xds_parameters.html#NAME_TEMPLATE_OF_DATA_FRAMES= supports] bzip2-ed frames. Thus, when specifying the frame name parameter of the script, you should leave out any .bz2 extension.

== The script ==

<pre>
#!/bin/bash
# purpose: generate XDS.INP
#
# tested with some datasets from ALS, SSRL, SLS, ESRF and BESSY; only MARCCD, ADSC/SMV, PILATUS detectors;
# for other detectors, values marked with XXX must be manually filled in.
#
# revision 0.03 . Kay Diederichs 2/2010
# revision 0.04 . Kay Diederichs 4/2010 - include alternative ORGX, ORGY calculations for ADSC
# revision 0.05 . Kay Diederichs 5/2010 - grep for "Corrected" in addition to "marccd"; needed for BESSY
# revision 0.06 . KD 6/2010 - add UNTRUSTED_RECTANGLE and UNTRUSTED_ELLIPSE; use `whereis catmar` and so on
# revision 0.07 . KD 6/2010 - decide about ORGX/Y info in MAR header being pixels or mm; other fixes
# revision 0.08 . KD 6/2010 - fixes for Pilatus 6M
# revision 0.09 . KD 6/2010 - get rid of requirement for mccd_xdsparams.pl and/or catmar; rather use "od"
# revision 0.10 . Tim Gruene 7/2010 - set link 'images' to image directory if path exceeds 72 characters
# revision 0.11 . KD 7/2010 - for MarCCD: look for distance info at different byte position
# revision 0.12 . KD 7/2010 - fix for negative PHISTART
# revision 0.13 . KD 8/2010 - store correct NX NY QX QY in XDS.INP
# revision 0.14 . KD 1/2011 - SENSOR_THICKNESS for Pilatus; MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT=3
# revision 0.15 . KD 2/2011 - add comment for -ive sign of APS 19-ID and Australian Synchrotron rotation axis
# revision 0.16 . KD 3/2011 - SENSOR_THICKNESS=0.01 for ADSC and MarCCD. Add comment about SILICON=
# revision 0.17 . KD 3/2011 - make it work for .bz2 frames; improve screen output
# revision 0.18 . KD 4/2011 - faster by doing "strings" only once; revert "images/${1##/*/}" "correction"
# revision 0.19 . KD 6/2011 - bugfix for 0.18
# revision 0.20 . KD 7/2011 - redirect stderr of /bin/ls to /dev/null
# revision 0.21 . KD 11/2011 - SEPMIN, CLUSTER_RADIUS hints; read NX NY from header (for Pilatus 2M)
# revision 0.22 . KD 12/2011 - Pilatus 2M UNTRUSTED_RECTANGLE lines, SENSOR_THICKNESS from header
# revision 0.23 . KD 1/2012 - add UNTRUSTED_QUADRILATERAL, remove MINIMUM_ZETA (0.05 is default now)
# revision 0.24 . KD 3/2012 - remove revision 0.10 since XDS now takes much longer paths
# revision 0.25 . KD 3/2012 - remove revision 0.22 for PSI Pilatus 2M; see http://www.globalphasing.com/autoproc/wiki/index.cgi?TroubleShootingKnownIssues
# revision 0.26 . KD 7/2012 - Mac-compatibility: replace od flags --skip-bytes= and --read-bytes= with -j and -N (thanks to Oliver Clarke for working this out!)
REVISION="0.26 (02-Jul-2012)"
#
# usage: e.g. generate_XDS.INP "frms/mydata_1_???.img"
# make sure to have the two quotation marks !
# the ? are wildcards for the frame numbers.
#
# limitations:
# - frame numbers are assumed to start with 1 and run consecutively
#
# known problems:
# - for ADSC detectors, there are at least three ways to obtain ORGX and ORGY values from the header (see below);
# - the same might be a problem for MAR headers, too (not sure about this)
# - on Mac OS X, Xcode (from http://developer.apple.com/tools/xcode) might be needed - not sure about this
#
# notes for debugging of the script:
# - add the -v option to the first line, to see where an error occurs
# - comment out the removal of tmp1 and tmp2 in the last line
#
# ====== Start of script ======
echo generate_XDS.INP version $REVISION . Obtain the latest version from
echo http://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/generate_XDS.INP
if [ "$1" == "help" ] || [ "$1" == "-help" ] || [ "$1" == "-h" ]; then
echo usage: generate_XDS.INP \"frms/mydata_1_???.img\" $_with_ the quotation marks!$
echo if the frames are compressed with bzip2, leave out the .bz2 extension!
exit
fi
#
# defaults:
#
DETECTOR="XXX MINIMUM_VALID_PIXEL_VALUE=XXX OVERLOAD=XXX"
ORGX=XXX
ORGY=XXX
DETECTOR_DISTANCE=XXX
OSCILLATION_RANGE=XXX
X_RAY_WAVELENGTH=XXX
QX=XXX
QY=XXX
NX=XXX
NY=XXX
SENSOR_THICKNESS=0
# see how we are called:
NAME_TEMPLATE_OF_DATA_FRAMES="$1"
# list frames matching the wildcards in NAME_TEMPLATE_OF_DATA_FRAMES
# don't accept the "direct beam" shot at SLS/Pilatus PX-I and PX-II
/bin/ls -C1 $1 $1.bz2 2>/dev/null | egrep -v "_00000.cbf|_000.img" > tmp1 || exit 1

# we can continue - the frames are found

# set upper limit of DATA_RANGE to number of frames (see "limitations" above)
DATA_RANGE=`wc -l tmp1 | awk '{print $1}'`

# set upper limit of SPOT_RANGE to half of DATA_RANGE, but not less than 1
SPOT_RANGE=`echo "scale=0; $DATA_RANGE/2" | bc -l`
SPOT_RANGE=`echo "if ($SPOT_RANGE<1) 1;if ($SPOT_RANGE>1) $SPOT_RANGE" | bc -l`

echo DATA_RANGE=1 $DATA_RANGE

# find out detector type
DET=XXX
FIRSTFRAME=`head -1 tmp1`
echo $FIRSTFRAME | grep -q bz2 && bzcat $FIRSTFRAME > tmp1 && FIRSTFRAME=tmp1
strings $FIRSTFRAME > tmp2
egrep -q 'marccd|Corrected' tmp2 && DET=mccd
grep -q PILATUS tmp2 && DET=pilatus
grep -q BEAM_CENTER_X tmp2 && DET=adsc
# identify other detector types in the same way (MAR IP would be straightforward)

# parse ASCII header of first frame

if [ "$DET" == "XXX" ]; then
echo "this is not a MAR, ADSC/SMV or PILATUS detector - fill in XXX values manually!"
DETECTOR="XXX MINIMUM_VALID_PIXEL_VALUE=XXX OVERLOAD=XXX"

# find parameters of first frame
elif [ "$DET" == "mccd" ]; then
echo Data from a MarCCD detector

DETECTOR="CCDCHESS MINIMUM_VALID_PIXEL_VALUE= 1 OVERLOAD= 65500"
SENSOR_THICKNESS=0.01
# use first frame of dataset to obtain parameters

# offsets are documented; values can be found in mccd_xdsparams.pl script
let SKIP=1024+80
NX=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
let SKIP=$SKIP+4
NY=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')

let SKIP=1720
DETECTOR_DISTANCE=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
DETECTOR_DISTANCE=`echo "scale=3; $DETECTOR_DISTANCE/1000" | bc -l`

let SKIP=1024+256+128+256+4
ORGX=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
ORGX=`echo "scale=2; $ORGX/1000" | bc -l `
let SKIP=$SKIP+4
ORGY=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
ORGY=`echo "scale=2; $ORGY/1000" | bc -l `

let SKIP=1024+256+128+256+44
PHISTART=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
let SKIP=1024+256+128+256+76
PHIEND=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
OSCILLATION_RANGE=`echo "scale=3; ($PHIEND-($PHISTART))/1000" | bc -l`

let SKIP=1024+256+128+256+128+4
QX=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
QX=`echo "scale=10; $QX/1000000" |bc -l `
let SKIP=$SKIP+4
QY=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
QY=`echo "scale=10; $QY/1000000" |bc -l `

let SKIP=1024+256+128+256+128+128+12
X_RAY_WAVELENGTH=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
X_RAY_WAVELENGTH=`echo "scale=5; $X_RAY_WAVELENGTH/100000" | bc -l`

# at most BLs, ORGX and ORGY are in pixels, but sometimes in mm ... guess:
NXBYFOUR=`echo "scale=0; $NX/4" | bc -l `
ORGXINT=`echo "scale=0; $ORGX/1" | bc -l `
if [ $ORGXINT -lt $NXBYFOUR ]; then
ORGX=`echo "scale=1; $ORGX/$QX" | bc -l`
ORGY=`echo "scale=1; $ORGY/$QY" | bc -l`
echo MARCCD detector: header ORGX, ORGY seem to be in mm ... converting to pixels
else
echo MARCCD detector: header ORGX, ORGY seem to be in pixel units
fi

elif [ "$DET" == "adsc" ]; then

DETECTOR="ADSC MINIMUM_VALID_PIXEL_VALUE= 1 OVERLOAD= 65000"
echo Data from ADSC detector. Obtaining ORGX, ORGY depends on beamline setup:
SENSOR_THICKNESS=0.01
sed s/\;// tmp2 > tmp1
mv tmp1 tmp2

# find X_RAY_WAVELENGTH:
X_RAY_WAVELENGTH=`grep WAVELENGTH tmp2 | head -1 | sed s/WAVELENGTH=//`

# find NX, QX, ORGX and ORGY:
NX=`grep SIZE1 tmp2 | tail -1 | sed s/SIZE1=//`
QX=`grep PIXEL_SIZE tmp2 | sed s/PIXEL_SIZE=//`
# FIXME - next 2 lines should be done properly, from header
NY=$NX
QY=$QX
BEAM_CENTER_X=`grep BEAM_CENTER_X tmp2 | sed s/BEAM_CENTER_X=//`
BEAM_CENTER_Y=`grep BEAM_CENTER_Y tmp2 | sed s/BEAM_CENTER_Y=//`
# fix 2010-04-26 - tell user about possible ORGX, ORGY alternatives -
# at ESRF and ... (pls fill in!) the following should be used:
ORGX=`echo "scale=1; $BEAM_CENTER_Y/$QX" | bc -l`
ORGY=`echo "scale=1; $BEAM_CENTER_X/$QX" | bc -l`
echo - at ESRF BLs use: ORGX=$ORGX ORGY=$ORGY
# this 2nd alternative convention should be used at the following beamlines (pls complete the list): ALS 5.0.3, ...
ORGX=`echo "scale=1; $NX-$BEAM_CENTER_X/$QX" | bc -l `
ORGY=`echo "scale=1; $BEAM_CENTER_Y/$QX" | bc -l `
echo - at e.g. ALS 5.0.3 use: ORGX=$ORGX ORGY=$ORGY
# this 3rd alternative convention should be used at the following beamlines (pls complete the list): ALS 8.2.2, ...
ORGX=`echo "scale=1; $BEAM_CENTER_X/$QX" | bc -l `
ORGY=`echo "scale=1; $NX-$BEAM_CENTER_Y/$QX" | bc -l `
echo - at e.g. ALS 8.2.2 use: ORGX=$ORGX ORGY=$ORGY - this is written to XDS.INP
# the latter alternative is written into the generated XDS.INP ! You have to correct this manually in XDS.INP, or adjust this script.
# find DETECTOR_DISTANCE and OSCILLATION_RANGE:
DETECTOR_DISTANCE=`grep DISTANCE tmp2 | sed s/DISTANCE=//`
OSCILLATION_RANGE=`grep OSC_RANGE tmp2 | sed s/OSC_RANGE=//`

elif [ "$DET" == "pilatus" ]; then
DETECTOR="PILATUS MINIMUM_VALID_PIXEL_VALUE=0 OVERLOAD= 1048576 !PILATUS"
QX=0.172 QY=0.172
echo Data from a Pilatus detector
sed s/#// tmp2 > tmp1
mv tmp1 tmp2

# find SENSOR_THICKNESS:
SENSOR_THICKNESS=`grep thickness tmp2 | sed -e s/'Silicon sensor, thickness'// | awk '{print $1*1000}'`
# find X_RAY_WAVELENGTH:
X_RAY_WAVELENGTH=`grep Wavelength tmp2 | sed -e s/Wavelength// -e s/A// | awk '{print $1}'`

# find NX and NY; 2463/2527 is 6M, 1475/1679 is 2M
NX=`grep X-Binary-Size-Fastest-Dimension tmp2 | awk '{print $2}'`
NY=`grep X-Binary-Size-Second-Dimension tmp2 | awk '{print $2}'`

# find ORGX and ORGY:
ORGX=`grep Beam_xy tmp2 | sed -e s/$// -e s/$// -e s/\,// | awk '{print $2}'`
ORGY=`grep Beam_xy tmp2 | sed -e s/$// -e s/$// -e s/\,// | awk '{print $3}'`

# find DETECTOR_DISTANCE and OSCILLATION_RANGE:
DETECTOR_DISTANCE=`awk '/distance/{print $2}' tmp2`
DETECTOR_DISTANCE=`echo "$DETECTOR_DISTANCE*1000" | bc -l`

OSCILLATION_RANGE=`awk '/Angle/{print $2}' tmp2`

else
echo should never come here
exit 1
fi

echo ORGX= $ORGX ORGY= $ORGY - check these values with adxv !
echo DETECTOR_DISTANCE= $DETECTOR_DISTANCE
echo OSCILLATION_RANGE= $OSCILLATION_RANGE
echo X-RAY_WAVELENGTH= $X_RAY_WAVELENGTH

# now we know everything that is required to generate XDS.INP

cat > XDS.INP << eof
! written by generate_XDS.INP version $REVISION
JOB= XYCORR INIT COLSPOT IDXREF DEFPIX INTEGRATE CORRECT
ORGX= $ORGX ORGY= $ORGY ! check these values with adxv !
DETECTOR_DISTANCE= $DETECTOR_DISTANCE
OSCILLATION_RANGE= $OSCILLATION_RANGE
X-RAY_WAVELENGTH= $X_RAY_WAVELENGTH
NAME_TEMPLATE_OF_DATA_FRAMES=$NAME_TEMPLATE_OF_DATA_FRAMES
! REFERENCE_DATA_SET=xxx/XDS_ASCII.HKL ! e.g. to ensure consistent indexing
DATA_RANGE=1 $DATA_RANGE
SPOT_RANGE=1 $SPOT_RANGE
! BACKGROUND_RANGE=1 10 ! rather use defaults (first 5 degree of rotation)

SPACE_GROUP_NUMBER=0 ! 0 if unknown
UNIT_CELL_CONSTANTS= 70 80 90 90 90 90 ! put correct values if known
INCLUDE_RESOLUTION_RANGE=50 0 ! after CORRECT, insert high resol limit; re-run CORRECT

FRIEDEL'S_LAW=FALSE ! This acts only on the CORRECT step
! If the anom signal turns out to be, or is known to be, very low or absent,
! use FRIEDEL'S_LAW=TRUE instead (or comment out the line); re-run CORRECT

! remove the "!" in the following line:
! STRICT_ABSORPTION_CORRECTION=TRUE
! if the anomalous signal is strong: in that case, in CORRECT.LP the three
! "CHI^2-VALUE OF FIT OF CORRECTION FACTORS" values are significantly> 1, e.g. 1.5
!
! exclude (mask) untrusted areas of detector, e.g. beamstop shadow :
! UNTRUSTED_RECTANGLE= 1800 1950 2100 2150 ! x-min x-max y-min y-max ! repeat
! UNTRUSTED_ELLIPSE= 2034 2070 1850 2240 ! x-min x-max y-min y-max ! if needed
! UNTRUSTED_QUADRILATERAL= x1 y1 x2 y2 x3 y3 x4 y4 ! see documentation
!
! parameters with changes wrt default values:
TRUSTED_REGION=0.00 1.2 ! partially use corners of detectors; 1.41421=full use
VALUE_RANGE_FOR_TRUSTED_DETECTOR_PIXELS=7000. 30000. ! often 8000 is ok
STRONG_PIXEL=4 ! COLSPOT: only use strong reflections (default is 3)
MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT=3 ! default of 6 is sometimes too high
! close spots: reduce SEPMIN and CLUSTER_RADIUS from their defaults of 6 and 3, e.g. to 4 and 2
! for bad or low resolution data remove the "!" in the following line:
! REFINE(IDXREF)=CELL BEAM ORIENTATION AXIS ! DISTANCE
REFINE(INTEGRATE)=CELL BEAM ORIENTATION ! AXIS DISTANCE
! REFINE(CORRECT)=CELL BEAM ORIENTATION AXIS DISTANCE ! Default is: refine everything

! parameters specifically for this detector and beamline:
DETECTOR= $DETECTOR
SENSOR_THICKNESS= $SENSOR_THICKNESS
! attention CCD detectors: for very high resolution (better than 1A) make sure to specify SILICON
! as about 32* what CORRECT.LP suggests (absorption of phosphor is much higher than that of silicon)
NX= $NX NY= $NY QX= $QX QY= $QY ! to make CORRECT happy if frames are unavailable
DIRECTION_OF_DETECTOR_X-AXIS=1 0 0
DIRECTION_OF_DETECTOR_Y-AXIS=0 1 0
INCIDENT_BEAM_DIRECTION=0 0 1
ROTATION_AXIS=1 0 0 ! at e.g. Australian Synchrotron, SERCAT ID-22 (?), APS 19-ID (?) this needs to be -1 0 0
FRACTION_OF_POLARIZATION=0.98 ! better value is provided by beamline staff!
POLARIZATION_PLANE_NORMAL=0 1 0
eof
if [ "$DET" == "pilatus" ]; then
if [ $NX == "1475" ]; then
if ! grep -q FF_p2m0109_E12398_T6199_vrf_m0p20.tif tmp2 ; then
cat >> XDS.INP << eof
!EXCLUSION OF VERTICAL DEAD AREAS OF THE PILATUS 2M DETECTOR
UNTRUSTED_RECTANGLE= 486 496 0 1680
UNTRUSTED_RECTANGLE= 980 990 0 1680
!EXCLUSION OF HORIZONTAL DEAD AREAS OF THE PILATUS 2M DETECTOR
UNTRUSTED_RECTANGLE= 0 1476 194 214
UNTRUSTED_RECTANGLE= 0 1476 406 426
UNTRUSTED_RECTANGLE= 0 1476 618 638
UNTRUSTED_RECTANGLE= 0 1476 830 850
UNTRUSTED_RECTANGLE= 0 1476 1042 1062
UNTRUSTED_RECTANGLE= 0 1476 1254 1274
UNTRUSTED_RECTANGLE= 0 1476 1466 1486
eof
fi
# elif [ $NX == "2463" ]; then
# cat >> XDS.INP << eof
#eof
fi
fi
echo XDS.INP is ready for use. The file has only the most important keywords.
echo Full documentation, including complete detector templates, is at
echo http://www.mpimf-heidelberg.mpg.de/~kabsch/xds . More documentation in XDSwiki
echo After running xds, inspect, using XDS-Viewer, at least the beamstop mask in
echo BKGPIX.cbf, and the agreement of predicted and observed spots in FRAME.cbf!
rm -f tmp1 tmp2
</pre>

== System-wide or personal installation ==

Ask your system adminstrator to cut-and-paste the script into e.g. /usr/local/bin/generate_XDS.INP, and to make it "executable".

But you may also cut-and-paste the script from this webpage into a file in e.g. your home directory; the filename should be generate_XDS.INP. After creating the file, make it executable - e.g. if it's in your $HOME, use:
chmod +x ~/generate_XDS.INP
After that, you can just run it in a similar way as if it were installed in your $PATH:
~/generate_XDS.INP "frms/mydata_1_???.img"
By using your own file, you can easily update to the latest revision, or even change the script, without having to bother the system administrator.

== Generating generate_XDS.INP from this webpage ==

Instead of cutting-and-pasting the lines of the script, you (or the system administrator) could just cut-and-paste the following four lines
<pre>
wget http://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/generate_XDS.INP -O - | \
sed -e s/\&nbsp\;/\ /g -e s/\&gt\;/\>/g -e s/\&lt\;/\</g -e s/amp\;//g -e s/\&quot\;/\"/g -e s/\&\#\1\6\0\;/\ /g | \
awk '/^#/,/rm -f tmp1 tmp2/' > generate_XDS.INP
chmod +x generate_XDS.INP
</pre>
to copy the script from this website into an executable file generate_XDS.INP in your current directory.

==xds_generate_all: variant of Generate_XDS.INP including scripts for visualisation of statistics and conversion to MTZ==
This script includes some minor modifications of the Generate_XDS.INP script - for instance, it tells the user the energy, as well as the wavelength of the X-rays, and gives the user the beam center in mm as well as pixels - useful when switching between HKL2000, MOSFLM and XDS.

It also generates two executable shell scripts in the same directory as XDS.INP.

The first of these scripts, xds_graph.sh, extracts various statistics from INTEGRATE.LP and CORRECT.LP, pipes them out to a logfile, INTEGRATE_STATS.LP, and plots them using loggraph. This script does not have some of the more useful features of XDSSTAT, such as calculation of per frame Rmeas values, but it does give a quick overview of various data quality parameters by resolution and image number. This script will only work with recent versions of XDS (after calculation of CC(1/2) - not sure exactly which version).

The second of these scripts, xds_to_ccp4.sh, will take XDS_ASCII.HKL and generate a CCP4-format MTZ file, containing anomalous data (as F(+)/F(-) and DANO/SigDANO) and a test set of 5% of reflections for calculation of the free R-factor. It will also generate an unmerged SHELX format HKL file, for input into SHELXC/D/E (e.g. via the HKL2MAP GUI).

Here is the script:
<pre>
#!/bin/bash
# purpose: xds_generate_all
#
# tested with some datasets from ALS, SSRL, SLS, ESRF and BESSY; only MARCCD, ADSC/SMV, PILATUS detectors;
# for other detectors, values marked with XXX must be manually filled in.
#
# revision 0.03 . Kay Diederichs 2/2010
# revision 0.04 . Kay Diederichs 4/2010 - include alternative ORGX, ORGY calculations for ADSC
# revision 0.05 . Kay Diederichs 5/2010 - grep for "Corrected" in addition to "marccd"; needed for BESSY
# revision 0.06 . KD 6/2010 - add UNTRUSTED_RECTANGLE and UNTRUSTED_ELLIPSE; use `whereis catmar` and so on
# revision 0.07 . KD 6/2010 - decide about ORGX/Y info in MAR header being pixels or mm; other fixes
# revision 0.08 . KD 6/2010 - fixes for Pilatus 6M
# revision 0.09 . KD 6/2010 - get rid of requirement for mccd_xdsparams.pl and/or catmar; rather use "od"
# revision 0.10 . Tim Gruene 7/2010 - set link 'images' to image directory if path exceeds 72 characters
# revision 0.11 . KD 7/2010 - for MarCCD: look for distance info at different byte position
# revision 0.12 . KD 7/2010 - fix for negative PHISTART
# revision 0.13 . KD 8/2010 - store correct NX NY QX QY in XDS.INP
# revision 0.14 . KD 1/2011 - SENSOR_THICKNESS for Pilatus; MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT=3
# revision 0.15 . KD 2/2011 - add comment for -ive sign of APS 19-ID and Australian Synchrotron rotation axis
# revision 0.16 . KD 3/2011 - SENSOR_THICKNESS=0.01 for ADSC and MarCCD. Add comment about SILICON=
# revision 0.17 . KD 3/2011 - make it work for .bz2 frames; improve screen output
# revision 0.18 . KD 4/2011 - faster by doing "strings" only once; revert "images/${1##/*/}" "correction"
# revision 0.19 . KD 6/2011 - bugfix for 0.18
# revision 0.20 . KD 7/2011 - redirect stderr of /bin/ls to /dev/null
# revision 0.21 . KD 11/2011 - SEPMIN, CLUSTER_RADIUS hints; read NX NY from header (for Pilatus 2M)
# revision 0.22 . KD 12/2011 - Pilatus 2M UNTRUSTED_RECTANGLE lines, SENSOR_THICKNESS from header
# revision 0.23 . KD 1/2012 - add UNTRUSTED_QUADRILATERAL, remove MINIMUM_ZETA (0.05 is default now)
# revision 0.24 . KD 3/2012 - remove revision 0.10 since XDS now takes much longer paths
# revision 0.25 . KD 3/2012 - remove revision 0.22 for PSI Pilatus 2M; see http://www.globalphasing.com/autoproc/wiki/index.cgi?TroubleShootingKnownIssues
# revision 0.26 Oliver Clarke 10/12 - Added generation of shell scripts for conversion to MTZ and visualisation of statistics. Other minor alterations to XDS.INP.
REVISION="0.26 (18-Oct-2012)"
#
# usage: e.g. xds_generate_all "frms/mydata_1_???.img"
# make sure to have the two quotation marks !
# the ? are wildcards for the frame numbers.
#
# limitations:
# - frame numbers are assumed to start with 1 and run consecutively
#
# known problems:
# - for ADSC detectors, there are at least three ways to obtain ORGX and ORGY values from the header (see below);
# - the same might be a problem for MAR headers, too (not sure about this)
#
# notes for debugging of the script:
# - add the -v option to the first line, to see where an error occurs
# - comment out the removal of tmp1 and tmp2 in the last line
#
# ====== Start of script ======
echo generate_XDS.INP version $REVISION . Obtain the latest version from
echo http://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/generate_XDS.INP
if [ "$1" == "help" ] || [ "$1" == "-help" ] || [ "$1" == "-h" ]; then
echo usage: xds_generate_all \"frms/mydata_1_???.img\" $_with_ the quotation marks!$
echo if the frames are compressed with bzip2, leave out the .bz2 extension!
exit
fi
#
# defaults:
#
DETECTOR="XXX MINIMUM_VALID_PIXEL_VALUE=XXX OVERLOAD=XXX"
ORGX=XXX
ORGY=XXX
ORGXMM=XXX
ORGYMM=XXX
DETECTOR_DISTANCE=XXX
OSCILLATION_RANGE=XXX
X_RAY_WAVELENGTH=XXX
X_RAY_ENERGY=XXX
QX=XXX
QY=XXX
NX=XXX
NY=XXX
NXMM=XXX
NYMM=XXX
SENSOR_THICKNESS=0
# see how we are called:
NAME_TEMPLATE_OF_DATA_FRAMES="$1"
# list frames matching the wildcards in NAME_TEMPLATE_OF_DATA_FRAMES
# don't accept the "direct beam" shot at SLS/Pilatus PX-I and PX-II
/bin/ls -C1 $1 $1.bz2 2>/dev/null | egrep -v "_00000.cbf|_000.img" > tmp1 || exit 1

# we can continue - the frames are found

# set upper limit of DATA_RANGE to number of frames (see "limitations" above)
DATA_RANGE=`wc -l tmp1 | awk '{print $1}'`

# set upper limit of SPOT_RANGE to half of DATA_RANGE, but not less than 1
SPOT_RANGE=`echo "scale=0; $DATA_RANGE/2" | bc -l`
SPOT_RANGE=`echo "if ($SPOT_RANGE<1) 1;if ($SPOT_RANGE>1) $SPOT_RANGE" | bc -l`

echo DATA_RANGE=1 $DATA_RANGE

# find out detector type
DET=XXX
FIRSTFRAME=`head -1 tmp1`
echo $FIRSTFRAME | grep -q bz2 && bzcat $FIRSTFRAME > tmp1 && FIRSTFRAME=tmp1
strings $FIRSTFRAME > tmp2
egrep -q 'marccd|Corrected' tmp2 && DET=mccd
grep -q PILATUS tmp2 && DET=pilatus
grep -q BEAM_CENTER_X tmp2 && DET=adsc
# identify other detector types in the same way (MAR IP would be straightforward)

# parse ASCII header of first frame

if [ "$DET" == "XXX" ]; then
echo "this is not a MAR, ADSC/SMV or PILATUS detector - fill in XXX values manually!"
DETECTOR="XXX MINIMUM_VALID_PIXEL_VALUE=XXX OVERLOAD=XXX"

# find parameters of first frame
elif [ "$DET" == "mccd" ]; then
echo Data from a MarCCD detector

DETECTOR="CCDCHESS MINIMUM_VALID_PIXEL_VALUE= 1 OVERLOAD= 65500"
SENSOR_THICKNESS=0.01
# use first frame of dataset to obtain parameters

# offsets are documented; values can be find in mccd_xdsparams.pl script
let SKIP=1024+80
NX=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
let SKIP=$SKIP+4
NY=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')

let SKIP=1720
DETECTOR_DISTANCE=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
DETECTOR_DISTANCE=`echo "scale=3; $DETECTOR_DISTANCE/1000" | bc -l`

let SKIP=1024+256+128+256+4
ORGX=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
ORGX=`echo "scale=2; $ORGX/1000" | bc -l `
let SKIP=$SKIP+4
ORGY=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
ORGY=`echo "scale=2; $ORGY/1000" | bc -l `

let SKIP=1024+256+128+256+44
PHISTART=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
let SKIP=1024+256+128+256+76
PHIEND=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
OSCILLATION_RANGE=`echo "scale=3; ($PHIEND-($PHISTART))/1000" | bc -l`

let SKIP=1024+256+128+256+128+4
QX=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
QX=`echo "scale=10; $QX/1000000" |bc -l `
let SKIP=$SKIP+4
QY=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
QY=`echo "scale=10; $QY/1000000" |bc -l `

let SKIP=1024+256+128+256+128+128+12
X_RAY_WAVELENGTH=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
X_RAY_WAVELENGTH=`echo "scale=5; $X_RAY_WAVELENGTH/100000" | bc -l`
X_RAY_ENERGY=`echo "12398.5/$X_RAY_WAVELENGTH" | bc -l | awk '{printf "%5.1f", $1}'`
# at most BLs, ORGX and ORGY are in pixels, but sometimes in mm ... guess:
NXBYFOUR=`echo "scale=0; $NX/4" | bc -l `
ORGXINT=`echo "scale=0; $ORGX/1" | bc -l `
if [ $ORGXINT -lt $NXBYFOUR ]; then
ORGXMM=$ORGX
ORGYMM=$ORGY
NXMM=`echo "$NX*$QX" | bc -l | awk '{printf "%3.2f", $1}'`
NYMM=`echo "$NY*$QY" | bc -l | awk '{printf "%3.2f", $1}'`
ORGX=`echo "scale=1; $ORGX/$QX" | bc -l`
ORGY=`echo "scale=1; $ORGY/$QY" | bc -l`
echo MARCCD detector: header ORGX, ORGY seem to be in mm ... converting to pixels
else
ORGXMM=`echo "$ORGX*$QX" | bc -l | awk '{printf "%3.2f", $1}'`
ORGYMM=`echo "$ORGY*$QY" | bc -l | awk '{printf "%3.2f", $1}'`
NXMM=`echo "$NX*$QX" | bc -l | awk '{printf "%3.2f", $1}'`
NYMM=`echo "$NY*$QY" | bc -l | awk '{printf "%3.2f", $1}'`
echo MARCCD detector: header ORGX, ORGY seem to be in pixel units
fi

elif [ "$DET" == "adsc" ]; then

DETECTOR="ADSC MINIMUM_VALID_PIXEL_VALUE= 1 OVERLOAD= 65000"
echo Data from ADSC detector. Obtaining ORGX, ORGY depends on beamline setup:
SENSOR_THICKNESS=0.01
sed s/\;// tmp2 > tmp1
mv tmp1 tmp2

# find X_RAY_WAVELENGTH:
X_RAY_WAVELENGTH=`grep WAVELENGTH tmp2 | head -1 | sed s/WAVELENGTH=//`

# find NX, QX, ORGX and ORGY:
X_RAY_ENERGY=`echo "12398.5/$X_RAY_WAVELENGTH" | bc -l | awk '{printf "%5.1f", $1}'`
NX=`grep SIZE1 tmp2 | tail -1 | sed s/SIZE1=//`
QX=`grep PIXEL_SIZE tmp2 | sed s/PIXEL_SIZE=//`
# FIXME - next 2 lines should be done properly, from header
NY=$NX
QY=$QX
NXMM=`echo "$NX*$QX" | bc -l | awk '{printf "%3.2f", $1}'`
NYMM=`echo "$NY*$QY" | bc -l | awk '{printf "%3.2f", $1}'`
BEAM_CENTER_X=`grep BEAM_CENTER_X tmp2 | sed s/BEAM_CENTER_X=//`
BEAM_CENTER_Y=`grep BEAM_CENTER_Y tmp2 | sed s/BEAM_CENTER_Y=//`
# fix 2010-04-26 - tell user about possible ORGX, ORGY alternatives -
# at ESRF and ... (pls fill in!) the following should be used:
ORGX=`echo "scale=1; $BEAM_CENTER_Y/$QX" | bc -l`
ORGY=`echo "scale=1; $BEAM_CENTER_X/$QX" | bc -l`
echo - at ESRF BLs use: ORGX=$ORGX ORGY=$ORGY
# this 2nd alternative convention should be used at the following beamlines (pls complete the list): ALS 5.0.3, ...
ORGX=`echo "scale=1; $NX-$BEAM_CENTER_X/$QX" | bc -l `
ORGY=`echo "scale=1; $BEAM_CENTER_Y/$QX" | bc -l `
echo - at e.g. ALS 5.0.3 use: ORGX=$ORGX ORGY=$ORGY
# this 3rd alternative convention should be used at the following beamlines (pls complete the list): ALS 8.2.2, ...
ORGX=`echo "scale=1; $BEAM_CENTER_X/$QX" | bc -l `
ORGY=`echo "scale=1; $NX-$BEAM_CENTER_Y/$QX" | bc -l `
ORGXMM=`echo "$ORGX*$QX" | bc -l | awk '{printf "%3.2f", $1}'`
ORGYMM=`echo "$ORGY*$QY" | bc -l | awk '{printf "%3.2f", $1}'`
echo - at e.g. ALS 8.2.2 use: ORGX=$ORGX ORGY=$ORGY - this is written to XDS.INP
# the latter alternative is written into the generated XDS.INP ! You have to correct this manually in XDS.INP, or adjust this script.
# find DETECTOR_DISTANCE and OSCILLATION_RANGE:
DETECTOR_DISTANCE=`grep DISTANCE tmp2 | sed s/DISTANCE=//`
OSCILLATION_RANGE=`grep OSC_RANGE tmp2 | sed s/OSC_RANGE=//`

elif [ "$DET" == "pilatus" ]; then
DETECTOR="PILATUS MINIMUM_VALID_PIXEL_VALUE=0 OVERLOAD= 1048576 !PILATUS"
QX=0.172 QY=0.172
echo Data from a Pilatus detector
sed s/#// tmp2 > tmp1
mv tmp1 tmp2

# find SENSOR_THICKNESS:
SENSOR_THICKNESS=`grep thickness tmp2 | sed -e s/'Silicon sensor, thickness'// | awk '{print $1*1000}'`
# find X_RAY_WAVELENGTH:
X_RAY_WAVELENGTH=`grep Wavelength tmp2 | sed -e s/Wavelength// -e s/A// | awk '{print $1}'`

X_RAY_ENERGY=`echo "12398.5/$X_RAY_WAVELENGTH" | bc -l | awk '{printf "%5.1f", $1}'`
# find NX and NY; 2463/2527 is 6M, 1475/1679 is 2M
NX=`grep X-Binary-Size-Fastest-Dimension tmp2 | awk '{print $2}'`
NY=`grep X-Binary-Size-Second-Dimension tmp2 | awk '{print $2}'`
NXMM=`echo "$NX*$QX" | bc -l | awk '{printf "%3.2f", $1}'`
NYMM=`echo "$NY*$QY" | bc -l | awk '{printf "%3.2f", $1}'`
# find ORGX and ORGY:
ORGX=`grep Beam_xy tmp2 | sed -e s/$// -e s/$// -e s/\,// | awk '{print $2}'`
ORGY=`grep Beam_xy tmp2 | sed -e s/$// -e s/$// -e s/\,// | awk '{print $3}'`
ORGXMM=`echo "$ORGX*$QX" | bc -l | awk '{printf "%3.2f", $1}'`
ORGYMM=`echo "$ORGY*$QY" | bc -l | awk '{printf "%3.2f", $1}'`
# find DETECTOR_DISTANCE and OSCILLATION_RANGE:
DETECTOR_DISTANCE=`awk '/distance/{print $2}' tmp2`
DETECTOR_DISTANCE=`echo "$DETECTOR_DISTANCE*1000" | bc -l`

OSCILLATION_RANGE=`awk '/Angle/{print $2}' tmp2`

else
echo should never come here
exit 1
fi

echo ORGX= $ORGX ORGY= $ORGY - check these values with adxv !
echo DETECTOR_DISTANCE= $DETECTOR_DISTANCE
echo OSCILLATION_RANGE= $OSCILLATION_RANGE
echo X-RAY_WAVELENGTH= $X_RAY_WAVELENGTH
# now we know everything that is required to generate XDS.INP

cat > XDS.INP << eof
!****************************************************************************************
! Written by generate_XDS.INP version $REVISION
!
! A few notes on usage and modification of this input file:
!
! If XDS fails after IDXREF with an error indicating that
! an insufficient percentage of reflections were indexed,
! but you are confident that the indexing solution is
! correct, rerun XDS with the JOB keyword changed to
! "DEFPIX INTEGRATE CORRECT".
!
! After the first round of integration (in a triclinic cell)
! it is often helpful to reintegrate using the globally
! refined cell parameters and the correct space group.
! You can do this by renaming GXPARM.XDS to XPARM.XDS
! - e.g. by the command "cp GXPARM.XDS XPARM.XDS" -
! and rerunning DEFPIX, INTEGRATE and CORRECT.
!
! If you are trying to determine the optimal starting point for
! data collection starting from a single image, first complete
! a run of XDS using default parameters. Then, rerun XDS using
! the correct cell and spacegroup as described above
! ("cp GXPARM.XDS XPARM.XDS"), but on the second run of XDS, give
! the JOB keyword as "JOB=DEFPIX XPLAN".
!****************************************************************************************
!
!****************************************************************************************
!General parameters:
!****************************************************************************************

JOB= XYCORR INIT COLSPOT IDXREF DEFPIX INTEGRATE CORRECT

ORGX= $ORGX ORGY= $ORGY ! check these values with adxv!
!In mm, ORGX=$ORGXMM and ORGY=$ORGYMM

DETECTOR_DISTANCE= $DETECTOR_DISTANCE
OSCILLATION_RANGE= $OSCILLATION_RANGE

X-RAY_WAVELENGTH= $X_RAY_WAVELENGTH
!X-ray energy is $X_RAY_ENERGY eV (calculated from wavelength)

NAME_TEMPLATE_OF_DATA_FRAMES=$NAME_TEMPLATE_OF_DATA_FRAMES

!REFERENCE_DATA_SET=xxx/XDS_ASCII.HKL ! e.g. to ensure consistent indexing.
!XDS will *only* use the reference dataset if the space group and cell
!parameters are specified below.

DATA_RANGE=1 $DATA_RANGE
SPOT_RANGE=1 $SPOT_RANGE

!BACKGROUND_RANGE=1 10 !Default is first five degrees.

SPACE_GROUP_NUMBER=0 ! 0 if unknown
UNIT_CELL_CONSTANTS= 70 80 90 90 90 90 ! put correct values if known

INCLUDE_RESOLUTION_RANGE=50 0 ! After CORRECT, insert high resol limit; re-run CORRECT
! Ice rings can be excluded from scaling with the keyword
! EXCLUDE_RESOLUTION_RANGE, e.g. EXCLUDE_RESOLUTION_RANGE=2.28 2.22

!SECONDS=60 !Uncomment this line to tell XDS to wait 1min for the next image before aborting.

!****************************************************************************************
!Parameters important for processing anomalous data:
!****************************************************************************************

FRIEDEL'S_LAW=FALSE ! This acts only on the CORRECT step. Even if the anomalous signal
! in your dataset is very small, it still may be useful for
! calculating an anomalous difference map.

STRICT_ABSORPTION_CORRECTION=FALSE !Change this to TRUE if the anomalous signal
! is strong: in that case, in CORRECT.LP the three
! "CHI^2-VALUE OF FIT OF CORRECTION FACTORS" values
! will be significantly> 1, e.g. 1.5.

WFAC1=1.0 !This parameter controls rejection of misfits during scaling. Sometimes,
! strong anomalous pairs may be rejected as misfits, in which case increasing
! WFAC1 (e.g. to 1.5) may improve anomalous signal.

!****************************************************************************************
!Regions of the detector to be excluded during indexing and integration:
!****************************************************************************************
! exclude (mask) untrusted areas of detector, e.g. beamstop shadow :
!UNTRUSTED_RECTANGLE= !1800 1950 2100 2150 ! x-min x-max y-min y-max ! repeat
!UNTRUSTED_ELLIPSE= !2034 2070 1850 2240 ! x-min x-max y-min y-max ! if needed
!UNTRUSTED_QUADRILATERAL= !x1 y1 x2 y2 x3 y3 x4 y4 ! see documentation

TRUSTED_REGION=0.00 1.2 ! Partially use corners of detectors; 1.41421=full use, 1.0=edge
! It is important to note that TRUSTED_REGION is the only resolution
! cutoff obeyed during indexing and integration;
! Both INCLUDE_RESOLUTION_RANGE and EXCLUDE_RESOLUTION_RANGE are
! ignored during IDXREF and INTEGRATE. So this parameter can be
! important when indexing or integration steps fail.

!****************************************************************************************
!Other parameters:
!****************************************************************************************
VALUE_RANGE_FOR_TRUSTED_DETECTOR_PIXELS=7000. 30000. ! Often 8000 is ok.
STRONG_PIXEL=4 ! COLSPOT: only use strong reflections (default is 3; 6 may be better for strong data)
MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT=3 ! default of 6 is sometimes too high
MAXIMUM_ERROR_OF_SPOT_POSITION=3 ! Increasing may help for poor quality data

MINPK=75 !Increase (e.g. MINPK=98) to improve data accuracy at the expense of completeness.

DELPHI=5 !Size of integration wedge. Increasing to 10 or 20 may be helpful in some cases.

NUMBER_OF_PROFILE_GRID_POINTS_ALONG_ALPHA/BETA=13 ! Default is 9 - Increasing may improve data
NUMBER_OF_PROFILE_GRID_POINTS_ALONG_GAMMA=13 ! accuracy, particularly if finely-sliced on phi,
! and does not seem to have any downsides.

MINIMUM_ZETA=0.05 ! Controls how close to the blind region (about phi) reflections should be
! integrated. 0.05 is the default; increasing MINIMUM_ZETA to 0.15 *may*
! improve data quality by removing unreliable reflections near the phi
! axis, but will reduce completeness in low symmetry space groups.

SEPMIN=6 ! Reduce if spots are close together (due to a long axis). Consider
CLUSTER_RADIUS=3 ! increasing if crystal is split and unit cell dimensions are relatively short.

! For bad or low resolution data remove the "!" in the following line (default is ALL):
! REFINE(IDXREF)=CELL BEAM ORIENTATION AXIS ! DISTANCE
REFINE(INTEGRATE)=CELL BEAM ORIENTATION ! AXIS DISTANCE (If integration is unstable, comment out this line.
! REFINE(CORRECT)=CELL BEAM ORIENTATION AXIS DISTANCE ! Default is: refine everything

ROTATION_AXIS=1 0 0 !At e.g. Australian Synchrotron, SERCAT ID-22 (?), APS 19-ID (?) this needs to be -1 0 0

!****************************************************************************************
! Parameters specifically for this detector and beamline (shouldn't need changing):
!****************************************************************************************
DETECTOR= $DETECTOR
SENSOR_THICKNESS= $SENSOR_THICKNESS
! attention CCD detectors: for very high resolution (better than 1A) make sure to specify SILICON
! as about 32* what CORRECT.LP suggests (absorption of phosphor is much higher than that of silicon)

NX= $NX NY= $NY QX= $QX QY= $QY ! to make CORRECT happy if frames are unavailable
!In mm, NX=$NXMM and NY=$NYMM

DIRECTION_OF_DETECTOR_X-AXIS=1 0 0
DIRECTION_OF_DETECTOR_Y-AXIS=0 1 0
INCIDENT_BEAM_DIRECTION=0 0 1
FRACTION_OF_POLARIZATION=0.98 ! better value is provided by beamline staff!
POLARIZATION_PLANE_NORMAL=0 1 0
eof
if [ "$DET" == "pilatus" ]; then
if [ $NX == "1475" ]; then
if ! grep -q FF_p2m0109_E12398_T6199_vrf_m0p20.tif tmp2 ; then
cat >> XDS.INP << eof
!EXCLUSION OF VERTICAL DEAD AREAS OF THE PILATUS 2M DETECTOR
UNTRUSTED_RECTANGLE= 486 496 0 1680
UNTRUSTED_RECTANGLE= 980 990 0 1680
!EXCLUSION OF HORIZONTAL DEAD AREAS OF THE PILATUS 2M DETECTOR
UNTRUSTED_RECTANGLE= 0 1476 194 214
UNTRUSTED_RECTANGLE= 0 1476 406 426
UNTRUSTED_RECTANGLE= 0 1476 618 638
UNTRUSTED_RECTANGLE= 0 1476 830 850
UNTRUSTED_RECTANGLE= 0 1476 1042 1062
UNTRUSTED_RECTANGLE= 0 1476 1254 1274
UNTRUSTED_RECTANGLE= 0 1476 1466 1486
eof
fi
# elif [ $NX == "2463" ]; then
# cat >> XDS.INP << eof
#eof
fi
fi
touch xds_to_ccp4.sh
echo "#!/bin/bash
#Removes old log file if present
rm xds_to_ccp4.log
#Makes new log file
touch xds_to_ccp4.log
#Pipe stdout and stderr to both logfile and tty
tail -f -n1 xds_to_ccp4.log &
exec > xds_to_ccp4.log 2>&1
echo \"

If this message is all you see,
something has probably gone wrong.
Are you sure XDS_ASCII.HKL is present
in the current directory?

\"
#Make first MTZ file (with DANO/SIGDANO)
echo \"INPUT_FILE=XDS_ASCII.HKL
OUTPUT_FILE=temp.hkl CCP4
MERGE=TRUE
GENERATE_FRACTION_OF_TEST_REFLECTIONS=0.05
FRIEDEL'S_LAW=FALSE\" > XDSCONV.INP
xdsconv
f2mtz HKLOUT temp.mtz<F2MTZ.INP

#Make second MTZ file (with F(+)/F(-))
echo \"INPUT_FILE=XDS_ASCII.HKL
OUTPUT_FILE=temp2.hkl CCP4_F
MERGE=TRUE
GENERATE_FRACTION_OF_TEST_REFLECTIONS=0.05
FRIEDEL'S_LAW=FALSE\" > XDSCONV.INP
xdsconv
f2mtz HKLOUT temp2.mtz<F2MTZ.INP

#Make SHELX format unmerged hkl file
echo \"INPUT_FILE=XDS_ASCII.HKL
OUTPUT_FILE=xds_shelx.hkl SHELX
MERGE=FALSE
FRIEDEL'S_LAW=FALSE\" > XDSCONV.INP
xdsconv

#cad the two previously generated MTZ files together
cad HKLIN1 temp.mtz HKLIN2 temp2.mtz HKLOUT xds_ccp4_merged.mtz<<EOF
LABIN FILE 1 E1=FP E2=SIGFP E3=DANO E4=SIGDANO E5=FreeRflag
LABIN FILE 2 E1=F(+) E2=SIGF(+) E3=F(-) E4=SIGF(-)
END
EOF

#Remove temporary files
rm temp.mtz
rm temp2.mtz
rm temp.hkl
rm temp2.hkl
rm F2MTZ.INP

echo \"

********************************XDS_TO_CCP4**********************************

This script takes XDS_ASCII.HKL and uses XDSCONV and F2MTZ to
generate a merged CCP4-format MTZ file named
\\\"xds_ccp4_merged.mtz\\\" with a test (free) set
labeled \\\"FreeRflag\\\" constituting 5% of reflections.

The space group of the output MTZ file will be the same as
that present in XDS_ASCII.HKL. It is probably worth checking
that this is the correct space group - either manually inspect
the systematic absences in CORRECT.LP, or run XDS_ASCII.HKL
through the CCP4 program POINTLESS (\\\"pointless xdsin XDS_ASCII.HKL\\\")

Anomalous data is kept, if present, and output as both
DANO/SIGDANO and F(+)/F(-).

This script also generates an unmerged SHELX hkl file,
suitable for input into SHELXC/D/E, named \\\"xds_shelx.hkl\\\".

A log of all processes is present in xds_to_ccp4.log.

Parts of this script have been shamelessly copied from
the XDS Wiki, at http://www.strucbio.biologie.uni-konstanz.de/xdswiki/

******************************************************************************
\"" > xds_to_ccp4.sh
touch xds_graph.sh
echo "#!/bin/bash
#Makes new log file
touch INTEGRATE_STATS.LP
#Pipe stdout and stderr to both logfile and tty
tail -f -n1 INTEGRATE_STATS.LP &
exec > INTEGRATE_STATS.LP 2>&1
echo \"
\\\$TABLE: XDS Integration Statistics :
\\\$SCATTER:Scale factors by image:A:1,3: :Number of overloads by image:A:1,5: :Number of strong reflections by image:A:1,7: :Number of rejects by image:A:1,8: :Mosaicity by image:A:1,10: \\$\\$
IMAGE IER SCALE NBKG NOVL NEWALD NSTRONG NREJ SIGMAB SIGMAR
\\$\\$ \\$\\$
\"
egrep \"[0-9]\.[0-9][0-9][0-9][0-9] [0-9]\.[0-9][0-9][0-9][0-9]\" INTEGRATE.LP
echo \" \\$\\$ \"
echo \"
\\\$TABLE: XDS Scaling Statistics by Resolution (X-axis is resolution in Angstroems) :
\\\$GRAPHS:Chi-Squared by resolution:A:2,4: :Observed and expected R-factors by resolution (as %):A:2,5,6: :Rejects by resolution:A:2,9: :Rejects by resolution as a percentage of total reflections:A:2,10: \\$\\$
LOW_RES Resolution(Angstroems) I/Sigma Chi^2 R-FACTOR(OBSERVED) R-FACTOR(EXPECTED) NUMBER_OF_REFLECTIONS ACCEPTED REJECTS REJECT_PERCENT
\\$\\$ \\$\\$
\"
awk '/RESOLUTION RANGE I\/Sigma Chi\^2 R\-FACTOR R\-FACTOR NUMBER ACCEPTED REJECTED/, /\-\-\-\-\-\-\-\-\-/' CORRECT.LP | egrep -v \"[a-z,A-Z,] | \-\-|\-99\.9\" | egrep \"[0-9]\" | awk '{print \$1,\$2,\$3,\$4,\$5,\$6,\$7,\$8,\$9,(\$9/(\$8+\$9))*100}'
echo \" \\$\\$\"
echo \"
\\\$TABLE: Overall data quality by Resolution (X-axis is resolution in Angstroems) :
\\\$GRAPHS:Completeness vs resolution :A:1,6: :Redundancy vs resolution :A:1,4: :I/Sigma by resolution:A:1,10: :Rmeas by resolution:A:1,11: :Anomalous correlation by resolution:A:1,13: :Significance of the anomalous signal by resolution:A:1,14: \\$\\$
Resolution(Angstroems) OBSERVED_REFLECTIONS UNIQUE_REFLECTIONS REDUNDANCY POSSIBLE_REFLECTIONS COMPLETENESS R-FACTOR(OBSERVED) R-FACTOR(EXPECTED) COMPARED I/SIGMA R-meas CC(1/2) Anomalous_Correlation SigAno Nano
\\$\\$ \\$\\$
\"
egrep -B25 \" WILSON STATISTICS \" CORRECT.LP | egrep -v \"[a-z,A-Z]|\*\*\*\" | egrep \" \" | awk '{ gsub(/[%*]/,\" \"); print }' | awk '{print \$1,\$2,\$3,\$2/\$3,\$4,\$5,\$6,\$7,\$8,\$9,\$10,\$11,\$12,\$13,\$14}'
echo \" \\$\\$ \"

echo \"
\\\$TABLE: I/Sigma of h00 reflections (for identification of systematic absences):
\\\$GRAPHS:I/Sigma vs h :A:1,5: \\$\\$
h k l RES I/SIGMA NUMBER
\\$\\$ \\$\\$
\"
awk '/REFLECTIONS OF TYPE H/,/COMPLETENESS AND QUALITY/' CORRECT.LP | egrep \"[1-9]\" | awk '{print \$1,\$2,\$3,\$4,\$7,\$8}' | awk '(\$1 > 0)'
echo \"\\$\\$\"

echo \"
\\\$TABLE: I/Sigma of 0k0 reflections (for identification of systematic absences):
\\\$GRAPHS:I/Sigma vs k :A:2,5: \\$\\$
h k l RES I/SIGMA NUMBER
\\$\\$ \\$\\$
\"
awk '/REFLECTIONS OF TYPE H/,/COMPLETENESS AND QUALITY/' CORRECT.LP | egrep \"[1-9]\" | awk '{print \$1,\$2,\$3,\$4,\$7,\$8}' | awk '(\$2 > 0)'
echo \"\\$\\$\"

echo \"
\\\$TABLE: I/Sigma of 00l reflections (for identification of systematic absences):
\\\$GRAPHS:I/Sigma vs l :A:3,5: \\$\\$
h k l RES I/SIGMA NUMBER
\\$\\$ \\$\\$
\"
awk '/REFLECTIONS OF TYPE H/,/COMPLETENESS AND QUALITY/' CORRECT.LP | egrep \"[1-9]\" | awk '{print \$1,\$2,\$3,\$4,\$7,\$8}' | awk '(\$3 > 0)'
echo \"\\$\\$\"

echo \"
\\\$TABLE:Beam center (ORGX/ORGY) plotted vs wedge (DELPHI):
\\\$GRAPHS:ORGX and ORGY vs batch (DELPHI) :A:1,2,3: \\$\\$
N ORGX ORGY
\\$\\$ \\$\\$
\"
egrep \"DETECTOR ORIGIN\" INTEGRATE.LP | grep -n \"\" | awk '{ gsub(/[a-z,A-Z,),(,:]/,\"\"); print }'
echo \"\\$\\$\"

echo \"
\\\$TABLE:Unit cell parameters plotted vs wedge:
\\\$GRAPHS:Unit cell lengths a, b and c plotted vs wedge:A:1,2,3,4: :Unit cell angles alpha, beta and gamma plotted vs wedge:A:1,5,6,7: \\$\\$
N a b c ALPHA BETA GAMMA
\\$\\$ \\$\\$
\"
egrep \"UNIT CELL PARAMETERS\" INTEGRATE.LP | grep -n \"\" | awk '{ gsub(/[a-z,A-Z,),(,:]/,\"\"); print }'
echo \"\\$\\$\"

echo \"
\\\$TABLE:Standard deviation of spot position (pixels) plotted vs wedge:
\\\$GRAPHS:Standard deviation of spot position plotted vs wedge:A:1,2: \\$\\$
N SD
\\$\\$ \\$\\$
\"
egrep \"SPOT POSITION\" INTEGRATE.LP | grep -n \"\" | awk '{ gsub(/[a-z,A-Z,),(,:]/,\"\"); print }'
echo \"\\$\\$\"

echo \"
\\\$TABLE:Detector distance (mm) plotted vs wedge (DELPHI):
\\\$GRAPHS:Detector distance plotted vs batch:A:1,2: \\$\\$
N DIST
\\$\\$ \\$\\$
\"
egrep \"DETECTOR DISTANCE \$mm\$\" INTEGRATE.LP | grep -n \"\" | awk '{ gsub(/[a-z,A-Z,),(,:]/,\"\"); print }'

echo \"\\$\\$\"

loggraph INTEGRATE_STATS.LP">xds_graph.sh
chmod +x xds_to_ccp4.sh
chmod +x xds_graph.sh
echo "XDS.INP is ready for use. The file has only the most important keywords.
Full documentation, including complete detector templates, is at
http://www.mpimf-heidelberg.mpg.de/~kabsch/xds . More documentation in XDSwiki
After running xds, inspect, using XDS-Viewer, at least the beamstop mask in
BKGPIX.cbf, and the agreement of predicted and observed spots in FRAME.cbf!

Two shell scripts have been generated that may be of use.

The first, xds_graph.sh, will plot various statistics after an XDS run.
After both INTEGRATE and CORRECT have finished, run it by typing \"./xds_graph.sh\"
without the quotation marks. This script uses loggraph to plot data, so you will
need CCP4 to see the graphical output. All the raw data is piped out to a log file,
INTEGRATE_STATS.LP.

The second script, xds_to_ccp4.sh, will take XDS_ASCII.HKL and generate both a CCP4
format MTZ file with anomalous data retained (as F(+)/F(-) and DANO/SigDANO)
and an unmerged SHELX format hkl file for input to SHELXC/D/E. Run it after CORRECT
by typing \"./xds_to_ccp4.sh\" at the prompt."

rm -f tmp1 tmp2
</pre>

Use this script in the same manner as Generate_XDS.INP. The two shell scripts created should be executable; If they are not, then make them so (chmod +x ./xxx.sh).