Generate XDS.INP: Difference between revisions

(25 intermediate revisions by 4 users not shown)

Line 1:

This script generates XDS.INP based on a list of frame names supplied on the commandline. It currently works for MarCCD, ADSC, Pilatus, Eiger, some Rigaku and one Bruker detector(s); ~~since~~ this is just a bash script, extension to other detectors should be very easy.

This script generates XDS.INP based on a list of frame names supplied on the commandline. It currently works for MarCCD, ADSC, Pilatus, Eiger, some Rigaku and one Bruker detector(s); there is also some support for microED with SMV files which have a wavelength given as 0 (look for "experimental-ED" in the script). Since this is just a bash script, extension to other detectors should be very easy.

== Usage ==

Line 5:

Usage is just (don't forget the quotation marks!):

generate_XDS.INP "/home/myname/frms/mydata_1_???.img"

XDS [~~http~~://~~strucbio~~.~~biologie~~.~~uni-konstanz~~.de~~/~dikay/XDS_html_doc~~/html_doc/xds_parameters.html#NAME_TEMPLATE_OF_DATA_FRAMES= supports] bzip2~~-ed frames~~. ~~Thus, when~~ specifying the frame name parameter of the script, you should leave out any .bz2 extension.

For .h5 files you need to load the *master.h5 , and you don't need the quotation marks:

generate_XDS.INP /home/myname/frms/mydata_1_master.h5

XDS [https://xds.mr.mpg.de/html_doc/xds_parameters.html#NAME_TEMPLATE_OF_DATA_FRAMES= supports] compression with gzip or bzip2. When specifying the frame name parameter of the script, you should leave out any .gz or .bz2 extension. Note that the processing of *.cbf.gz files can be made faster by [[Installation|loading]] the xds-zcbf.so [[LIB|plugin]].

For ~~improved~~ interaction with [[XDSGUI]], it is advantageous to provide an ''absolute'' filename for the data files - one that starts with a slash ("/").

For good interaction with [[XDSGUI]], it is advantageous to provide an ''absolute'' filename for the data files - one that starts with a slash ("/").

== The script ==

Expand code section below (i.e. click on blue <code>[Expand]</code> at the end of this line if there is no code visible)

<pre>

Line 145:

Line 151:

# revision 1.18 . JMK 19/08/2023 Pilatus 12M support, give help msg if $5 defined (i.e. unquoted frame pattern)

# revision 1.19 . KD 14/10/2023 Pilatus 12M at DLS I23 has reverse phi. Fix download commands.

REVISION="1.19 (14-~~Oct~~-~~2023~~)"

# revision 1.20 . KD 13/7/2024 Comment out line STRONG_PIXEL=4 in preparation for mid-2024 version of XDS.

# revision 1.21 . Thomas Hauß 12/03/2025 define for Bruker sfrm format DET= Bruker-sfrm and corresponding section

# revision 1.22 . KD 28/5/2025 UNTRUSTED_RECTANGLEs for Pilatus3 1M, S/N 10-0159

# revision 1.23 . KD 28/2/2026 remove setting negative DETECTOR_DISTANCE for adsc-CMOS1 - found with SBGrid 383 (PDB 5HV4)

# revision 1.24 . Jan Gebauer 14/03/2026 Implementation for ID30B Pilatus CdTe detector. Introduction of calc_CdTe_Silicon () for 5-25kEV range.

# Revision 1.25 . KD 17/4/2026 Pilatus4 Si 2M at SLS. HDF5: subtract 1 from /entry/instrument/detector//saturation_value to obtain OFFSET

REVISION="1.25 (17-Apr-2026)"

#

Line 172:

Line 184:

# make sure the locale does not interfere with e.g. awk calculations:

export LC_ALL="C"

# calc_CdTe_silicon() - Compute XDS SILICON parameter for CdTe (1mm sensor)

# Valid range: 5-25 keV (0.496-2.480 A)

# Fit: SILICON = 36548 * E_keV^(-2.661) R2=0.9997 vs calcmu

#

# Usage:

# calc_CdTe_silicon 12.4 -> keV input

# calc_CdTe_silicon 1.0A -> Angstrom input (A suffix, case-insensitive)

#

# Returns: SILICON value on stdout, or "XXX" if out of range

# In script usage e.g. SILICON="SILICON= $(calc_CdTe_silicon ${X_RAY_WAVELENGTH}A)"

calc_CdTe_silicon() {

local input="$1"

local kev

# Parse input: Angstrom or keV

if [[ "${input}" =~ ^([0-9]*\.?[0-9]+)[Aa]$ ]]; then

kev=$(awk -v l="${BASH_REMATCH[1]}" 'BEGIN { printf "%.6f", 12.39842/l }')

elif [[ "${input}" =~ ^[0-9]*\.?[0-9]+$ ]]; then

kev="${input}"

else

echo "XXX"; return 1

fi

# Range check 5-25 keV, apply power law

awk -v kev="$kev" '

BEGIN {

if (kev < 5.0 || kev > 25.0) { print "XXX"; exit }

printf "%.4f\n", 36548.0 * (kev ^ -2.661)

}'

}

#

# defaults:

Line 206:

Line 252:

CLUSTER_RADIUS=3.5

REFINE_CORRECT="CELL BEAM ORIENTATION AXIS POSITION ! Default is: refine everything"

SILICON="" # normally we don't need to set SILICON, but this is no correct for CdTe detector surfaces

dname=$(dirname "$1")

Line 279:

Line 326:

grep -q MARCONTROL tmp2 && DET=OLDMAR

grep -q "WAVELENGTH=0.0" tmp2 && DET=experimental-ED

if head -c 240 tmp2 | grep -q "FORMAT" && \

head -c 240 tmp2 | grep -q "VERSION" && \

head -c 240 tmp2 | grep -q "HDRBLKS"; then

DET="Bruker-sfrm"

fi

else

h5dump -d "/entry/instrument/detector/description" $FIRSTFRAME | grep -i Eiger > /dev/null && DET=eiger

# h5dump -d "/entry/instrument/detector/description" $FIRSTFRAME | grep -i "PILATUS4 CdTe 4M" > /dev/null && DET=eiger

h5dump -d "/entry/instrument/detector/description" $FIRSTFRAME | grep -i PILATUS4 > /dev/null && DET=eiger

fi

Line 491:

Line 545:

ORGY=`grep BEAM_CENTER_Y tmp2 | sed s/BEAM_CENTER_Y=//`

REVERSE_PHI="yes"

DETECTOR_DISTANCE=`grep ^DISTANCE tmp2 | sed s/DISTANCE=/-/`

# DETECTOR_DISTANCE=`grep ^DISTANCE tmp2 | sed s/DISTANCE=/-/` found to be wrong with SBGrid-383 28.2.26 ->rev 1.23

DETECTOR_DISTANCE=`grep ^DISTANCE tmp2 | sed s/DISTANCE=//`

OSCILLATION_RANGE=`grep OMEGA_DELTA tmp2 | sed s/OMEGA_DELTA=//`

DETECTOR_X_AXIS="-1 0 0"

Line 618:

Line 673:

DLS=0

OVERLOAD=`h5dump -d "/entry/instrument/detector/detectorSpecific/countrate_correction_count_cutoff" $FIRSTFRAME 2>/dev/null` || DLS=1

#special case of PILATUS4 Si 2M at SLS does not have /entry/instrument/detector/detectorSpecific/countrate_correction_count_cutoff but should be treated as Dectris - seems to be SLS own '

if [ "$DET_SN" == "D029955" ]; then

DLS=0

fi

if [ "$DLS" == 1 ]; then

echo Eiger HDF5 from Diamond

# unfortunately h5dump 1.10 is required to get this right for the DLS .h5 files. This version is available at DLS but maybe not elsewhere

OVERLOAD=`h5dump -d "/entry/instrument/detector/saturation_value" $FIRSTFRAME | awk '/$0$:/{print $2}'`

OVERLOAD=`h5dump -d "/entry/instrument/detector/saturation_value" $FIRSTFRAME | awk '/$0$:/{print $2-1}'`

# v0.95: fix the next 2 lines by taking care of negative values with the \- , and stop after first "(0)"

OSCILLATION_RANGE=`h5dump -d "/entry/data/omega" $FIRSTFRAME | awk '/$0$: [\-0-9]/{print $3-$2;exit}'`

Line 632:

Line 691:

echo Eiger HDF5 from Dectris

OVERLOAD=`h5dump -d "/entry/instrument/detector/detectorSpecific/countrate_correction_count_cutoff" $FIRSTFRAME | awk '/$0$:/{print $2}'`

if [ "$DET_SN" == "D029955" ]; then

OVERLOAD=`h5dump -d "/entry/instrument/detector/saturation_value" $FIRSTFRAME | awk '/$0$:/{print $2-1}'`

fi

OSCILLATION_RANGE=`h5dump -d "/entry/sample/goniometer/omega_range_average" $FIRSTFRAME | awk '/$0$: [0-9]/{print $2}'`

# STARTING_ANGLE: the \- was introduced in version 0.91 to allow negative values :

Line 726:

Line 788:

X_RAY_WAVELENGTH=`h5dump -d "/entry/instrument/beam/incident_wavelength" $FIRSTFRAME | awk '/$0$: [0-9]/{print $2}'`

# revision 1.24: Settings for the Pilatus 4M CdTe ad ID30B / ESRF

if [ "$DET_SN" == "D033638" ]; then

echo "Pilatus 4M CdTe at ID30B / ESRF: setting SENSOR_THICKNESS and SILICON parameters for CdTe sensor"

SENSOR_THICKNESS=1.0

SILICON="SILICON= $(calc_CdTe_silicon ${X_RAY_WAVELENGTH}A)"

OVERLOAD=1048500

DETECTOR="PILATUS MINIMUM_VALID_PIXEL_VALUE=0 OVERLOAD= $OVERLOAD"

# manually overwriting the DIRECTION_OF_DETECTOR_X/Y-AXIS and ROTATION_AXIS since the NeXus header of this detector is wrong (DIRECTION_OF_DETECTOR_X/Y-AXIS are swapped, and ROTATION_AXIS is inverted)

DIRECTION_OF_DETECTOR_X_AXIS="1.0 0.0 0.0"

DIRECTION_OF_DETECTOR_Y_AXIS="0.0 1.0 0.0"

ROTATION_AXIS="1.0 0.0 0.0"

fi

NX=`h5dump -d "/entry/instrument/detector/detectorSpecific/x_pixels_in_detector" $FIRSTFRAME | awk '/$0$: [0-9]/{print $2}'`

NY=`h5dump -d "/entry/instrument/detector/detectorSpecific/y_pixels_in_detector" $FIRSTFRAME | awk '/$0$: [0-9]/{print $2}'`

Line 950:

Line 1,025:

DIRECTION_OF_DETECTOR_X_AXIS="$R1 0 $R3"

# end of Bruker-cbf section

elif [ "$DET" == "Bruker-sfrm" ]; then

# echo "Detector from Bruker in sfrm format."

# MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT:

MNOPIAS=6

# use complete detector including corners:

TRUSTED_REGION="0 1.42"

SEPMIN=4 # 4 for Pixel Array Detectors

CLUSTER_RADIUS=2 # 2 for Pixel Array Detectors

# to read different format versions read these parametes first:

FORMAT=$(awk -F'FORMAT :' '{print $2}' tmp2 | awk '{print $1}')

VERSION=$(awk -F'VERSION:' '{print $2}' tmp2 | awk '{print $1}')

DETTYPE=$(awk -F'DETTYPE:' '{print $2}' tmp2 | awk '{print $1}')

DETTYPE="BRUKER_$DETTYPE"

echo "Bruker-sfrm file Format: $FORMAT, Version: $VERSION"

# use last value from CCDPARM and convert to integer

OVERLOAD=$(awk -F'CCDPARM:' '{print $2}' tmp2 | awk '{print int($5)}' | head -n1)

DETECTOR="${DETTYPE} MINIMUM_VALID_PIXEL_VALUE=0 OVERLOAD=${OVERLOAD}"

echo "DETECTOR $DETECTOR"

NY=$(awk -F'NROWS :' '{print $2}' tmp2 | awk '{print $1}')

NX=$(awk -F'NCOLS :' '{print $2}' tmp2 | awk '{print $1}')

echo "NX" $NX

echo "NY" $NY

read ORGX ORGY < <(awk -F'CENTER :' '{print $2}' tmp2 | awk '{print $1, $2}')

# There seems to be an error in ORGY, the offset from the detector

# center is calculated wrong with a in inverted sign. Here an attempt

# to correct this, may be not valid in all cases. Found via comparison

# with the Bruker CBF format. There the value of V should be positive!

ORGY=$(echo "scale=2; $NY - $ORGY" | bc -l)

echo "ORGX: $ORGX"

echo "ORGY: $ORGY"

# pixelsize from detector width NX and pixel per cm

pixelpercm=$(awk -F'DETTYPE:' '{print $2}' tmp2 | awk '{print $2}')

if [ NX == 512 ]; then

QX=$(echo "scale=6; 10.0 / $pixelpercm" | bc -l)

QY=QX=$(echo "scale=6; 10.0 / $pixelpercm" | bc -l)

else

QX=$(echo "scale=6; 5.0 / $pixelpercm" | bc -l)

QY=$(echo "scale=6; 5.0 / $pixelpercm" | bc -l)

fi

# QX=0.135 # from Bruker detector specification

# QY=0.135

echo "QX: $QX"

echo "QY: $QY"

# With Version < 11, only one distance written to header, then 2, second value seems ok.

if [ "$VERSION" -lt 11 ]; then

DETECTOR_DISTANCE=$(awk -F'DISTANC:' '{print $2}' tmp2 | awk '{print $1}')

else

DETECTOR_DISTANCE=$(awk -F'DISTANC:' '{print $2}' tmp2 | awk '{print $2}')

fi

DETECTOR_DISTANCE=$(echo "$DETECTOR_DISTANCE * 10" | bc -l) # convert cm to mm

echo "DETECTOR_DISTANCE: $DETECTOR_DISTANCE"

X_RAY_WAVELENGTH=$(awk -F'WAVELEN:' '{print $2}' tmp2 | awk '{print $1}')

echo "X_RAY_WAVELENGTH: $X_RAY_WAVELENGTH"

read TWOTHETA OMEGA PHI KAPPA < <(awk -F'ANGLES :' '{print $2}' tmp2 | awk '{print $1, $2, $3, $4}')

echo "TWOTHETA OMEGA PHI KAPPA: $TWOTHETA $OMEGA $PHI $KAPPA"

# axis 1: 2-theta, 2: omega, 3: phi, 4: chi

axis=$(awk -F'AXIS :' '{print $2}' tmp2 | awk '{print $1}')

if [ "$axis" == 3 ]; then

# PHI scan

echo "PHI scan"

DELTAPHI=$(awk -F'INCREME:' '{print $2}' tmp2 | awk '{print $1}')

R3=$(bc -l <<< "scale=7; s($KAPPA/$DEGTOR) * s($OMEGA/$DEGTOR)")

R1=$(bc -l <<< "scale=7; s($KAPPA/$DEGTOR) * c($OMEGA/$DEGTOR)")

R2=$(bc -l <<< "scale=7; c($KAPPA/$DEGTOR)")

ROTATION_AXIS="$R1 $R2 $R3"

OSCILLATION_RANGE=${DELTAPHI}

STARTING_ANGLE=${PHI}

else

echo "OMEGA scan"

ROTATION_AXIS="0 -1 0"

DELTAOMEGA=$(awk -F'INCREME:' '{print $2}' tmp2 | awk '{print $1}')

OSCILLATION_RANGE=${DELTAOMEGA}

STARTING_ANGLE=${OMEGA}

fi

# detector X-axis

R1=$(bc -l <<< "scale=7; c($TWOTHETA/$DEGTOR)")

R3=$(bc -l <<< "scale=7; s($TWOTHETA/$DEGTOR)")

DIRECTION_OF_DETECTOR_X_AXIS="$R1 0 $R3"

echo "DIRECTION_OF_DETECTOR_X_AXIS: $DIRECTION_OF_DETECTOR_X_AXIS"

echo "STARTING_ANGLE $STARTING_ANGLE"

echo "OSCILLATION_RANGE $OSCILLATION_RANGE"

echo "ROTATION_AXIS $ROTATION_AXIS"

# end of Bruker-sfrm section

else

echo should never come here

Line 1,015:

Line 1,186:

! More documentation in wiki.uni-konstanz.de/xds/index.php .

! After running xds, inspect at least the agreement of predicted and observed

! spots in FRAME.cbf!

! spots in FRAME.cbf (or SHOW_HKL.cbf in versions since BUILT=20240723)!

!=============================================================================

JOB= XYCORR INIT COLSPOT IDXREF DEFPIX INTEGRATE CORRECT

Line 1,033:

Line 1,204:

UNIT_CELL_CONSTANTS= 70 80 90 90 90 90 ! put correct values if known

INCLUDE_RESOLUTION_RANGE=50 0 ! after CORRECT, insert high resol limit; re-run CORRECT

! IDXREF ~~now~~ obeys INCLUDE_RESOLUTION_RANGE and EXCLUDE_RESOLUTION_RANGE to exclude ice-rings

! IDXREF obeys INCLUDE_RESOLUTION_RANGE and EXCLUDE_RESOLUTION_RANGE to exclude ice-rings

FRIEDEL'S_LAW=FALSE ! This acts only on the CORRECT step

Line 1,052:

Line 1,223:

TRUSTED_REGION=$TRUSTED_REGION

VALUE_RANGE_FOR_TRUSTED_DETECTOR_PIXELS=6000. 30000. ! often 7000 or 8000 is ok

STRONG_PIXEL=4 ! COLSPOT: only use strong reflections (default is 3)

!STRONG_PIXEL=4 ! COLSPOT: only use strong reflections (default is 3). For XDS versions before mid-2024.

MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT=$MNOPIAS ! default of 6 is sometimes too high

! close spots/long cell axis: reduce SEPMIN and CLUSTER_RADIUS from their defaults of 7 and 3.5

Line 1,065:

Line 1,236:

DETECTOR= $DETECTOR

SENSOR_THICKNESS= $SENSOR_THICKNESS

$SILICON

! attention CCD detectors: for very high resolution (better than 1A) make sure to specify SILICON

! as about 32* what CORRECT.LP suggests (absorption of phosphor is much higher than that of silicon).

Line 1,098:

Line 1,270:

eof

# --------- Append additional detector-specific parameters ----------

if [ "$DET" == "Bruker-cbf" ]; then

if [ "$DET" == "Bruker-cbf" -o "$DET" == "Bruker-sfrm" ]; then

echo "DELPHI=15 ! refine less often than the default of 5" >> XDS.INP

elif [ "$DET" == "adsc-CMOS1" ]; then

Line 1,237:

Line 1,409:

UNTRUSTED_RECTANGLE= 0 4149 3812 3851

eof

# PILATUS 1M, S/N 10-0159

elif [ "$NX" == "981" -a "$NY" == "1043" ]; then

if ! grep -q Flat_field tmp2 ; then

echo Flat_field not found, smaller UNTRUSTED_RECTANGLEs than for bigger Pilatus

cat >> XDS.INP << eof

! the following specifications are for a detector _without_ proper

! flat_field correction; they mask one additional pixel adjacent

! to each UNTRUSTED_RECTANGLE

!EXCLUSION OF VERTICAL DEAD AREAS OF THE PILATUS3 1M, S/N 10-0159

UNTRUSTED_RECTANGLE= 487 495 0 1044

!EXCLUSION OF HORIZONTAL DEAD AREAS OF THE PILATUS3 1M, S/N 10-0159

UNTRUSTED_RECTANGLE= 0 982 195 213

UNTRUSTED_RECTANGLE= 0 982 407 425

UNTRUSTED_RECTANGLE= 0 982 619 637

UNTRUSTED_RECTANGLE= 0 982 831 849

eof

else

echo Flat_field found, smaller UNTRUSTED_RECTANGLEs than for bigger Pilatus

cat >> XDS.INP << eof

!EXCLUSION OF VERTICAL DEAD AREAS OF THE PILATUS3 1M, S/N 10-0159

UNTRUSTED_RECTANGLE= 488 494 0 1044

!EXCLUSION OF HORIZONTAL DEAD AREAS OF THE PILATUS3 1M, S/N 10-0159

UNTRUSTED_RECTANGLE= 0 982 196 212

UNTRUSTED_RECTANGLE= 0 982 408 424

UNTRUSTED_RECTANGLE= 0 982 620 636

UNTRUSTED_RECTANGLE= 0 982 832 848

eof

fi

elif [ "$NX" == "2073" -a "$NY" == "2180" ]; then

# Pilatus 4M CdTe ; v1.24 numbers from Jan Gebauer

cat >> XDS.INP << eof

!masking non sensitive area of Pilatus 4M CdTe

UNTRUSTED_RECTANGLE= 0 2074 255 276

UNTRUSTED_RECTANGLE= 0 2074 530 551

UNTRUSTED_RECTANGLE= 0 2074 805 826

UNTRUSTED_RECTANGLE= 0 2074 1080 1101

UNTRUSTED_RECTANGLE= 0 2074 1355 1376

UNTRUSTED_RECTANGLE= 0 2074 1630 1651

UNTRUSTED_RECTANGLE= 0 2074 1905 1926

UNTRUSTED_RECTANGLE= 513 521 0 2181

UNTRUSTED_RECTANGLE= 1033 1041 0 2181

UNTRUSTED_RECTANGLE= 1553 1561 0 2181

UNTRUSTED_RECTANGLE= 255 257 0 2181

UNTRUSTED_RECTANGLE= 775 777 0 2181

UNTRUSTED_RECTANGLE= 1295 1297 0 2181

UNTRUSTED_RECTANGLE= 1815 1817 0 2181

eof

elif [ "$NX" == "1553" -a "$NY" == "1630" ]; then

# Pilatus4 Si 2M ; v1.25

cat >> XDS.INP << eof

!masking non sensitive area of Pilatus4 Si 2M

! UNTRUSTED_RECTANGLE= lines not needed because pixel counts are -1

eof

# Pilatus 12M

elif [ "$NX" == "2463" -a "$NY" == "5071" ]; then

: # handled above, before writing XDS.INP, untrusted regions come from site file

Line 1,289:

Line 1,516:

echo More documentation in wiki.uni-konstanz.de/xds/index.php .

echo After running xds, inspect at least the agreement of predicted and observed

echo spots in FRAME.cbf!

echo "spots in FRAME.cbf (or SHOW_HKL.cbf in versions since BUILT=20240723)!"

rm -f tmp1 tmp2

# end of generate_XDS.INP

# end of generate_XDS.INP</pre>

</~~pre~~>

</div>

== System-wide or personal installation ==

Line 1,332:

Line 1,560:

chmod +x generate_XDS.INP

</pre>

If you do use cut-and-paste from ~~the~~ webpage, be aware of the following problem report: ~~On the Mac, after~~ loading frames, by clicking “generate XDS.INP”, the program ~~gives~~ some strange symbol “Ã” in XDS.INP. And the more you click “save” button, the more “Ã” appear. This looks like e.g. <br>SPACE_GROUP_NUMBER=0 Ã ! 0 if unknown <br>UNIT_CELL_CONSTANTS= 70 80 90 90 90 90 Ã <br> ''The problem is due to the “Rich text” format in TextEdit when saving "generate_XDS.INP". It is solved by re-downloading the script, and changing format to Plain - everything should work then.''

If you use TextEdit on a Mac to cut-and-paste from this webpage, be aware of the following problem report: After loading frames, by clicking “generate XDS.INP”, the program shows some strange symbol “Ã” in XDS.INP. And the more you click “save” button, the more “Ã” appear. This looks like e.g. <br>SPACE_GROUP_NUMBER=0 Ã ! 0 if unknown <br>UNIT_CELL_CONSTANTS= 70 80 90 90 90 90 Ã <br>''The problem is due to the “Rich text” format in TextEdit when saving "generate_XDS.INP". It is solved by re-downloading the script, and changing format to Plain - everything should work then.''

== Calling generate_XDS.INP from a Python script ==

Line 1,350:

Line 1,578:

The script makes use of many GNU commands, like <code>ls, grep, egrep, awk, cut, cat, echo, wc, bc, head, sed, tail, cp, od, python</code>. Some of them (like <code>od</code> and <code>python</code>) are only used in case of specific detectors (MarCCD and RAXIS, respectively).

The script will only work if all the required commands are available. They reside in either the <code>coreutils</code> package, or specific packages (<code>gawk, sed, bc, grep, python</code> ...). Please note that to get the <code>strings</code> command on some Linux distributions, you need to install the <code>binutils</code> package.

For Eiger data processing, the <code>h5dump</code> program must be installed. This is part of <code>hdf5-tools</code> (Ubuntu) or <code>hdf5</code> (RHEL). The .h5 files collected at Diamond Light Source require ~~a very new~~ version ~~of h5dump (namely h5dump~~ 1.10) to extract the OVERLOAD parameter from the .h5 file; this version is available ~~by default~~ in ~~Ubuntu 20.04 and~~ RHEL/CentOS 8.

For Eiger data processing, the <code>h5dump</code> program must be installed. This is part of <code>hdf5-tools</code> (Ubuntu) or <code>hdf5</code> (RHEL). The .h5 files collected at Diamond Light Source require at least version 1.10 of h5dump to extract the OVERLOAD parameter from the .h5 file; this version is available in newer Linux distributions but not in RHEL/CentOS 7.

On Mac OS X, installation of the "Command Line Tools" (from http://developer.apple.com/downloads; requires Apple ID) is required (open a terminal and type <code>xcode-select --install</code>). These are also part of the (larger, but also free) [http://developer.apple.com/tools/xcode Xcode] package. This package comes with a license that has to be accepted by the user when running a Command Line Tool (e.g. <code>strings</code>) for the first time.

Line 1,371:

Line 1,599:

== Site Files ==

Site files may be used to add or modify parameters for specific hardware configurations. The primary use is for SEGMENT ~~defintions~~ of multi-segment detectors, and may also override FRACTION_OF_POLARIZATION, ROTATION_AXIS ~~and~~ TRUSTED_REGION specific to a site configuration ~~that is not recognized by the script~~. The site file is ~~saved as:~~ ~/.xds-site/<NAME>_<SN>. The exact name will be printed by ~~the script~~.

Site files may be used to add or modify parameters for specific hardware configurations. The primary use is for SEGMENT definitions of multi-segment detectors, and may also override FRACTION_OF_POLARIZATION, ROTATION_AXIS, TRUSTED_REGION, or UNTRUSTED_* specific to a site configuration. The site file is named ~/.xds-site/<NAME>_<SN>. The exact name will be printed by generate_XDS.INP when it is executed.

Site file for PILATUS 12M-DLS detector for the I23 beamline at Diamond Light Source:

[https://wiki.uni-konstanz.de/pub/linux_bin/PILATUS_12M_SN_120-0100 Site file for PILATUS 12M-DLS detector for the I23 beamline at Diamond Light Source]:

Expand code section below (i.e. click on blue <code>[Expand]</code> at the end of this line if there is no code visible)

<pre>

!--------------------------------------------------------------

Line 2,225:

Line 2,456:

SEGMENT_DISTANCE= 694.72

</pre>

</div>

== Limitations ==

* The script tries to interpret the header of the frames, so is currently limited to Dectris (Pilatus, Eiger), ADSC (Quantum), Rigaku (several types), MAR (CCD and image plate) detectors, and one Bruker detector. Other detectors need some values to be manually filled into XDS.INP - the relevant places are marked with XXX. These are detector properties (type, pixel size and number, min and max counts in a pixel), and experimental parameters like ROTATION_AXIS, OSCILLATION_RANGE, X-RAY_WAVELENGTH, DETECTOR_DISTANCE, and XORG, YORG. For fine-tuning of detector parameters, see the [http://xds.mpimf-heidelberg.mpg.de/html_doc/xds_prepare.html detector-specific templates].

* The script tries to interpret the header of the frames, so is currently limited to Dectris (Pilatus, Eiger), ADSC (Quantum), Rigaku (several types), MAR (CCD and image plate) detectors, and one Bruker detector, and Bruker sfrm Version 18 . Other detectors need some values to be manually filled into XDS.INP - the relevant places are marked with XXX. These are detector properties (type, pixel size and number, min and max counts in a pixel), and experimental parameters like ROTATION_AXIS, OSCILLATION_RANGE, X-RAY_WAVELENGTH, DETECTOR_DISTANCE, and XORG, YORG. For fine-tuning of detector parameters, see the [http://xds.mpimf-heidelberg.mpg.de/html_doc/xds_prepare.html detector-specific templates].

* The authors have made a "best effort" to provide a XDS.INP that results in the correct sign of the anomalous signal. In the case of one detector type (internally called Rigaku SMV) this requires reversal of one detector axis, and a negative DETECTOR_DISTANCE, as is found in some of the [http://xds.mpimf-heidelberg.mpg.de/html_doc/xds_prepare.html detector-specific templates]. '''For an unusual or unknown detector setup, the correct sign of the anomalous signal needs to be established and verified e.g. with a good dataset from a test crystal that has an anomalous signal.''' The authors do not take any responsibility for problems arising from incorrect sign of the anomalous signal, nor - obviously! - for any other mischief arising in or from data processing.

* At some beamlines, the ROTATION_AXIS ~~should~~ be -1 0 0 ("backwards") instead of the usual 1 0 0 ("horizontal"), or even 0 1 0 ("vertical") like at one of the PETRA Hamburg BLs. The frame headers do not have this information, so the default chosen by [[generate_XDS.INP]] may be wrong and need manual correction. ~~This~~ can be ~~automated~~ with a site file. Pls also see the article [[Beamline notes]].

* At some beamlines, the ROTATION_AXIS needs to be -1 0 0 ("backwards") instead of the usual 1 0 0 ("horizontal"), or even 0 1 0 ("vertical") like at one of the PETRA Hamburg BLs. The frame headers do not have this information, but for some beamlines the correct value has been implemented in the script. For other beamlines, the default chosen by [[generate_XDS.INP]] may be wrong and need manual correction of XDS.INP. The correct choice can be enforced by you with a [[Generate_XDS.INP#Site_Files|site file]]. Pls also see the article [[Beamline notes]].

* Sometimes, the x- and y- values of the primary beam position recorded in the header should be used for ORGY and ORGX (i.e reversed) instead of as ORGX and ORGY. For ADSC, this has been implemented in the script for ~~a number of~~ beamlines.

* Sometimes, the x- and y- values of the primary beam position recorded in the header should be used for ORGY and ORGX (i.e reversed) instead of as ORGX and ORGY. For ADSC, this has been implemented in the script for certain beamlines.

* there are apparently several flavours of HDF5 files produced at Diamond Light Source. They differ e.g. in the naming of the header items. This means that items like NX, NY, DETECTOR_DISTANCE and number of images cannot always be determined by the [[generate_XDS.INP]] script. Example: The data at /dls/i04-1/data/2021/mx28114-9/processing/Lenye_Diamini/ThiL/ThiL found during the CCP4 2021 online Cape Town workshop. A workaround is to use e.g. xia2 pipeline=3dii to process these files, and - if needed - extract those items from its output files, e.g. from DEFAULT/NATIVE/SWEEP1/index/XDS.INP .

@@ Line 1: / Line 1: @@
-This script generates XDS.INP based on a list of frame names supplied on the commandline. It currently works for MarCCD, ADSC, Pilatus, Eiger, some Rigaku and one Bruker detector(s); since this is just a bash script, extension to other detectors should be very easy.
+This script generates XDS.INP based on a list of frame names supplied on the commandline. It currently works for MarCCD, ADSC, Pilatus, Eiger, some Rigaku and one Bruker detector(s); there is also some support for microED with SMV files which have a wavelength given as 0 (look for "experimental-ED" in the script). Since this is just a bash script, extension to other detectors should be very easy.
 == Usage ==
@@ Line 5: / Line 5: @@
 Usage is just (don't forget the quotation marks!):
   generate_XDS.INP "/home/myname/frms/mydata_1_???.img"
-XDS [http://strucbio.biologie.uni-konstanz.de/~dikay/XDS_html_doc/html_doc/xds_parameters.html#NAME_TEMPLATE_OF_DATA_FRAMES= supports] bzip2-ed frames. Thus, when specifying the frame name parameter of the script, you should leave out any .bz2 extension.
+For .h5 files you need to load the *master.h5 , and you don't need the quotation marks:
+ generate_XDS.INP /home/myname/frms/mydata_1_master.h5
+XDS [https://xds.mr.mpg.de/html_doc/xds_parameters.html#NAME_TEMPLATE_OF_DATA_FRAMES= supports] compression with gzip or bzip2. When specifying the frame name parameter of the script, you should leave out any .gz or .bz2 extension. Note that the processing of *.cbf.gz files can be made faster by [[Installation|loading]] the xds-zcbf.so [[LIB|plugin]].
-For improved interaction with [[XDSGUI]], it is advantageous to provide an ''absolute'' filename for the data files - one that starts with a slash ("/").
+For good interaction with [[XDSGUI]], it is advantageous to provide an ''absolute'' filename for the data files - one that starts with a slash ("/").
 == The script ==
 <!-- leave a blank line after <pre>! KD 10/2023 -->
+<div class="toccolours mw-collapsible mw-collapsed">
+Expand code section below (i.e. click on blue <code>[Expand]</code> at the end of this line if there is no code visible)
+<div class="mw-collapsible-content">
 <pre>
@@ Line 145: / Line 151: @@
 # revision 1.18 . JMK 19/08/2023 Pilatus 12M support, give help msg if $5 defined (i.e. unquoted frame pattern)
 # revision 1.19 . KD 14/10/2023 Pilatus 12M at DLS I23 has reverse phi. Fix download commands.
-REVISION="1.19 (14-Oct-2023)"
+# revision 1.20 . KD 13/7/2024 Comment out line STRONG_PIXEL=4 in preparation for mid-2024 version of XDS.
+# revision 1.21 . Thomas Hauß 12/03/2025 define for Bruker sfrm format DET= Bruker-sfrm and corresponding section
+# revision 1.22 . KD 28/5/2025 UNTRUSTED_RECTANGLEs for Pilatus3 1M, S/N 10-0159
+# revision 1.23 . KD 28/2/2026 remove setting negative DETECTOR_DISTANCE for adsc-CMOS1 - found with SBGrid 383 (PDB 5HV4)
+# revision 1.24 . Jan Gebauer 14/03/2026 Implementation for ID30B Pilatus CdTe detector. Introduction of calc_CdTe_Silicon () for 5-25kEV range.
+# Revision 1.25 . KD 17/4/2026 Pilatus4 Si 2M at SLS. HDF5: subtract 1 from /entry/instrument/detector//saturation_value to obtain OFFSET
+REVISION="1.25 (17-Apr-2026)"
 #
@@ Line 172: / Line 184: @@
 # make sure the locale does not interfere with e.g. awk calculations:
 export LC_ALL="C"
+# calc_CdTe_silicon() - Compute XDS SILICON parameter for CdTe (1mm sensor)
+# Valid range: 5-25 keV (0.496-2.480 A)
+# Fit: SILICON = 36548 * E_keV^(-2.661)  R2=0.9997 vs calcmu
+#
+# Usage:
+#   calc_CdTe_silicon 12.4        -> keV input
+#   calc_CdTe_silicon 1.0A        -> Angstrom input (A suffix, case-insensitive)
+#
+# Returns: SILICON value on stdout, or "XXX" if out of range
+# In script usage e.g. SILICON="SILICON= $(calc_CdTe_silicon ${X_RAY_WAVELENGTH}A)"
+calc_CdTe_silicon() {
+    local input="$1"
+    local kev
+    # Parse input: Angstrom or keV
+    if [[ "${input}" =~ ^([0-9]*\.?[0-9]+)[Aa]$ ]]; then
+        kev=$(awk -v l="${BASH_REMATCH[1]}" 'BEGIN { printf "%.6f", 12.39842/l }')
+    elif [[ "${input}" =~ ^[0-9]*\.?[0-9]+$ ]]; then
+        kev="${input}"
+    else
+        echo "XXX"; return 1
+    fi
+    # Range check 5-25 keV, apply power law
+    awk -v kev="$kev" '
+    BEGIN {
+        if (kev < 5.0 || kev > 25.0) { print "XXX"; exit }
+        printf "%.4f\n", 36548.0 * (kev ^ -2.661)
+    }'
+}
 #
 # defaults:
@@ Line 206: / Line 252: @@
 CLUSTER_RADIUS=3.5
 REFINE_CORRECT="CELL BEAM ORIENTATION AXIS POSITION ! Default is: refine everything"
+SILICON=""  # normally we don't need to set SILICON, but this is no correct for CdTe detector surfaces
 dname=$(dirname "$1")
@@ Line 279: / Line 326: @@
   grep -q MARCONTROL tmp2 && DET=OLDMAR
   grep -q "WAVELENGTH=0.0" tmp2 && DET=experimental-ED
+ if head -c 240 tmp2 | grep -q "FORMAT" && \
+   head -c 240 tmp2 | grep -q "VERSION" && \
+   head -c 240 tmp2 | grep -q "HDRBLKS"; then
+    DET="Bruker-sfrm"
+ fi
 else
   h5dump -d "/entry/instrument/detector/description" $FIRSTFRAME | grep -i Eiger > /dev/null && DET=eiger
+# h5dump -d "/entry/instrument/detector/description" $FIRSTFRAME | grep -i "PILATUS4 CdTe 4M" > /dev/null && DET=eiger
+ h5dump -d "/entry/instrument/detector/description" $FIRSTFRAME | grep -i PILATUS4 > /dev/null && DET=eiger
 fi
@@ Line 491: / Line 545: @@
    ORGY=`grep BEAM_CENTER_Y tmp2 | sed s/BEAM_CENTER_Y=//`
    REVERSE_PHI="yes"
-   DETECTOR_DISTANCE=`grep ^DISTANCE tmp2 | sed s/DISTANCE=/-/`
+# DETECTOR_DISTANCE=`grep ^DISTANCE tmp2 | sed s/DISTANCE=/-/` found to be wrong with SBGrid-383 28.2.26 ->rev 1.23
+   DETECTOR_DISTANCE=`grep ^DISTANCE tmp2 | sed s/DISTANCE=//`
    OSCILLATION_RANGE=`grep OMEGA_DELTA tmp2 | sed s/OMEGA_DELTA=//`
    DETECTOR_X_AXIS="-1 0 0"
@@ Line 618: / Line 673: @@
    DLS=0
    OVERLOAD=`h5dump -d "/entry/instrument/detector/detectorSpecific/countrate_correction_count_cutoff" $FIRSTFRAME 2>/dev/null` || DLS=1
+  #special case of PILATUS4 Si 2M at SLS does not have /entry/instrument/detector/detectorSpecific/countrate_correction_count_cutoff but should be treated as Dectris - seems to be SLS own '
+  if [ "$DET_SN" == "D029955" ]; then
+    DLS=0
+  fi
    if [ "$DLS" == 1 ]; then
      echo Eiger HDF5 from Diamond
 # unfortunately h5dump 1.10 is required to get this right for the DLS .h5 files. This version is available at DLS but maybe not elsewhere
-     OVERLOAD=`h5dump -d "/entry/instrument/detector/saturation_value" $FIRSTFRAME | awk '/\(0\):/{print $2}'`
+     OVERLOAD=`h5dump -d "/entry/instrument/detector/saturation_value" $FIRSTFRAME | awk '/\(0\):/{print $2-1}'`
    # v0.95: fix the next 2 lines by taking care of negative values with the \- , and stop after first "(0)"
      OSCILLATION_RANGE=`h5dump -d "/entry/data/omega" $FIRSTFRAME | awk '/\(0\): [\-0-9]/{print $3-$2;exit}'`
@@ Line 632: / Line 691: @@
      echo Eiger HDF5 from Dectris
      OVERLOAD=`h5dump -d "/entry/instrument/detector/detectorSpecific/countrate_correction_count_cutoff" $FIRSTFRAME | awk '/\(0\):/{print $2}'`
+    if [ "$DET_SN" == "D029955" ]; then
+      OVERLOAD=`h5dump -d "/entry/instrument/detector/saturation_value" $FIRSTFRAME | awk '/\(0\):/{print $2-1}'`
+    fi
      OSCILLATION_RANGE=`h5dump -d "/entry/sample/goniometer/omega_range_average" $FIRSTFRAME | awk '/\(0\): [0-9]/{print $2}'`
    # STARTING_ANGLE:  the \- was introduced in version 0.91 to allow negative values :
@@ Line 726: / Line 788: @@
    X_RAY_WAVELENGTH=`h5dump -d "/entry/instrument/beam/incident_wavelength" $FIRSTFRAME | awk '/\(0\): [0-9]/{print $2}'`
+# revision 1.24: Settings for the Pilatus 4M CdTe ad ID30B / ESRF
+  if [ "$DET_SN" == "D033638" ]; then
+    echo "Pilatus 4M CdTe at ID30B / ESRF: setting SENSOR_THICKNESS and SILICON parameters for CdTe sensor"
+    SENSOR_THICKNESS=1.0
+    SILICON="SILICON= $(calc_CdTe_silicon ${X_RAY_WAVELENGTH}A)"
+    OVERLOAD=1048500
+    DETECTOR="PILATUS MINIMUM_VALID_PIXEL_VALUE=0 OVERLOAD= $OVERLOAD"
+# manually overwriting the DIRECTION_OF_DETECTOR_X/Y-AXIS and ROTATION_AXIS since the NeXus header of this detector is wrong (DIRECTION_OF_DETECTOR_X/Y-AXIS are swapped, and ROTATION_AXIS is inverted)
+    DIRECTION_OF_DETECTOR_X_AXIS="1.0 0.0 0.0"
+    DIRECTION_OF_DETECTOR_Y_AXIS="0.0 1.0 0.0"
+    ROTATION_AXIS="1.0 0.0 0.0"
+  fi
    NX=`h5dump -d "/entry/instrument/detector/detectorSpecific/x_pixels_in_detector" $FIRSTFRAME | awk '/\(0\): [0-9]/{print $2}'`
    NY=`h5dump -d "/entry/instrument/detector/detectorSpecific/y_pixels_in_detector" $FIRSTFRAME | awk '/\(0\): [0-9]/{print $2}'`
@@ Line 950: / Line 1,025: @@
    DIRECTION_OF_DETECTOR_X_AXIS="$R1 0 $R3"
 # end of Bruker-cbf section
+elif [ "$DET" == "Bruker-sfrm" ]; then
+# echo "Detector from Bruker in sfrm format."
+  # MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT:
+  MNOPIAS=6
+  # use complete detector including corners:
+  TRUSTED_REGION="0 1.42"
+  SEPMIN=4            # 4 for Pixel Array Detectors
+  CLUSTER_RADIUS=2    # 2 for Pixel Array Detectors
+  # to read different format versions read these parametes first:
+  FORMAT=$(awk -F'FORMAT :' '{print $2}' tmp2 | awk '{print $1}')
+  VERSION=$(awk -F'VERSION:' '{print $2}' tmp2 | awk '{print $1}')
+  DETTYPE=$(awk -F'DETTYPE:' '{print $2}' tmp2 | awk '{print $1}')
+  DETTYPE="BRUKER_$DETTYPE"
+  echo "Bruker-sfrm file Format: $FORMAT, Version: $VERSION"
+  # use last value from CCDPARM and convert to integer
+  OVERLOAD=$(awk -F'CCDPARM:' '{print $2}' tmp2 | awk '{print int($5)}' | head -n1)
+  DETECTOR="${DETTYPE} MINIMUM_VALID_PIXEL_VALUE=0 OVERLOAD=${OVERLOAD}"
+  echo "DETECTOR $DETECTOR"
+  NY=$(awk -F'NROWS  :' '{print $2}' tmp2 | awk '{print $1}')
+  NX=$(awk -F'NCOLS  :' '{print $2}' tmp2 | awk '{print $1}')
+  echo "NX" $NX
+  echo "NY" $NY
+  read ORGX ORGY < <(awk -F'CENTER :' '{print $2}' tmp2 | awk '{print $1, $2}')
+  # There seems to be an error in ORGY, the offset from the detector
+  # center is calculated wrong with a in inverted sign. Here an attempt
+  # to correct this, may be not valid in all cases. Found via comparison
+  # with the Bruker CBF format. There the value of V should be positive!
+  ORGY=$(echo "scale=2; $NY - $ORGY" | bc -l)
+  echo "ORGX: $ORGX"
+  echo "ORGY: $ORGY"
+  # pixelsize from detector width NX and pixel per cm
+  pixelpercm=$(awk -F'DETTYPE:' '{print $2}' tmp2 | awk '{print $2}')
+  if [ NX == 512 ]; then
+    QX=$(echo "scale=6; 10.0 / $pixelpercm" | bc -l)
+    QY=QX=$(echo "scale=6; 10.0 / $pixelpercm" | bc -l)
+  else
+    QX=$(echo "scale=6; 5.0 / $pixelpercm" | bc -l)
+    QY=$(echo "scale=6; 5.0 / $pixelpercm" | bc -l)
+  fi
+  # QX=0.135  # from Bruker detector specification
+  # QY=0.135
+  echo "QX: $QX"
+  echo "QY: $QY"
+  # With Version < 11, only one distance written to header, then 2, second value seems ok.
+  if [ "$VERSION" -lt 11 ]; then
+    DETECTOR_DISTANCE=$(awk -F'DISTANC:' '{print $2}' tmp2 | awk '{print $1}')
+  else
+    DETECTOR_DISTANCE=$(awk -F'DISTANC:' '{print $2}' tmp2 | awk '{print $2}')
+  fi
+  DETECTOR_DISTANCE=$(echo "$DETECTOR_DISTANCE * 10" | bc -l)  # convert cm to mm
+  echo "DETECTOR_DISTANCE: $DETECTOR_DISTANCE"
+  X_RAY_WAVELENGTH=$(awk -F'WAVELEN:' '{print $2}' tmp2 | awk '{print $1}')
+  echo "X_RAY_WAVELENGTH: $X_RAY_WAVELENGTH"
+  read TWOTHETA OMEGA PHI KAPPA < <(awk -F'ANGLES :' '{print $2}' tmp2 | awk '{print $1, $2, $3, $4}')
+  echo "TWOTHETA OMEGA PHI KAPPA: $TWOTHETA $OMEGA $PHI $KAPPA"
+  # axis 1: 2-theta, 2: omega, 3: phi, 4: chi
+  axis=$(awk -F'AXIS   :' '{print $2}' tmp2 | awk '{print $1}')
+  if [ "$axis" == 3 ]; then
+    # PHI scan
+    echo "PHI scan"
+    DELTAPHI=$(awk -F'INCREME:' '{print $2}' tmp2 | awk '{print $1}')
+    R3=$(bc -l <<< "scale=7; s($KAPPA/$DEGTOR) * s($OMEGA/$DEGTOR)")
+    R1=$(bc -l <<< "scale=7; s($KAPPA/$DEGTOR) * c($OMEGA/$DEGTOR)")
+    R2=$(bc -l <<< "scale=7; c($KAPPA/$DEGTOR)")
+    ROTATION_AXIS="$R1 $R2 $R3"
+    OSCILLATION_RANGE=${DELTAPHI}
+    STARTING_ANGLE=${PHI}
+  else
+    echo "OMEGA scan"
+    ROTATION_AXIS="0 -1 0"
+    DELTAOMEGA=$(awk -F'INCREME:' '{print $2}' tmp2 | awk '{print $1}')
+    OSCILLATION_RANGE=${DELTAOMEGA}
+    STARTING_ANGLE=${OMEGA}
+  fi
+  # detector X-axis
+  R1=$(bc -l <<< "scale=7; c($TWOTHETA/$DEGTOR)")
+  R3=$(bc -l <<< "scale=7; s($TWOTHETA/$DEGTOR)")
+  DIRECTION_OF_DETECTOR_X_AXIS="$R1 0 $R3"
+  echo "DIRECTION_OF_DETECTOR_X_AXIS: $DIRECTION_OF_DETECTOR_X_AXIS"
+  echo "STARTING_ANGLE $STARTING_ANGLE"
+  echo "OSCILLATION_RANGE $OSCILLATION_RANGE"
+  echo "ROTATION_AXIS $ROTATION_AXIS"
+  # end of Bruker-sfrm section
 else
    echo should never come here
@@ Line 1,015: / Line 1,186: @@
 ! More documentation in wiki.uni-konstanz.de/xds/index.php .
 ! After running xds, inspect at least the agreement of predicted and observed
-! spots in FRAME.cbf!
+! spots in FRAME.cbf (or SHOW_HKL.cbf in versions since BUILT=20240723)!
 !=============================================================================
 JOB= XYCORR INIT COLSPOT IDXREF DEFPIX INTEGRATE CORRECT
@@ Line 1,033: / Line 1,204: @@
 UNIT_CELL_CONSTANTS= 70 80 90 90 90 90 ! put correct values if known
 INCLUDE_RESOLUTION_RANGE=50 0  ! after CORRECT, insert high resol limit; re-run CORRECT
-! IDXREF now obeys INCLUDE_RESOLUTION_RANGE and EXCLUDE_RESOLUTION_RANGE to exclude ice-rings
+! IDXREF obeys INCLUDE_RESOLUTION_RANGE and EXCLUDE_RESOLUTION_RANGE to exclude ice-rings
 FRIEDEL'S_LAW=FALSE     ! This acts only on the CORRECT step
@@ Line 1,052: / Line 1,223: @@
 TRUSTED_REGION=$TRUSTED_REGION
 VALUE_RANGE_FOR_TRUSTED_DETECTOR_PIXELS=6000. 30000. ! often 7000 or 8000 is ok
-STRONG_PIXEL=4           ! COLSPOT: only use strong reflections (default is 3)
+!STRONG_PIXEL=4  ! COLSPOT: only use strong reflections (default is 3). For XDS versions before mid-2024.
 MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT=$MNOPIAS ! default of 6 is sometimes too high
 ! close spots/long cell axis: reduce SEPMIN and CLUSTER_RADIUS from their defaults of 7 and 3.5
@@ Line 1,065: / Line 1,236: @@
 DETECTOR= $DETECTOR
 SENSOR_THICKNESS= $SENSOR_THICKNESS
+$SILICON
 ! attention CCD detectors: for very high resolution (better than 1A) make sure to specify SILICON
 ! as about 32* what CORRECT.LP suggests (absorption of phosphor is much higher than that of silicon).
@@ Line 1,098: / Line 1,270: @@
 eof
 # --------- Append additional detector-specific parameters ----------
-if [ "$DET" == "Bruker-cbf" ]; then
+if [ "$DET" == "Bruker-cbf" -o "$DET" == "Bruker-sfrm" ]; then
    echo "DELPHI=15 ! refine less often than the default of 5" >> XDS.INP
 elif [ "$DET" == "adsc-CMOS1" ]; then
@@ Line 1,237: / Line 1,409: @@
   UNTRUSTED_RECTANGLE=    0 4149   3812 3851
 eof
+# PILATUS 1M, S/N 10-0159
+  elif [ "$NX" == "981" -a "$NY" == "1043" ]; then
+    if ! grep -q Flat_field tmp2 ; then
+    echo Flat_field not found, smaller UNTRUSTED_RECTANGLEs than for bigger Pilatus
+    cat >> XDS.INP << eof
+! the following specifications are for a detector _without_ proper
+! flat_field correction; they mask one additional pixel adjacent
+! to each UNTRUSTED_RECTANGLE
+!EXCLUSION OF VERTICAL DEAD AREAS OF THE PILATUS3 1M, S/N 10-0159
+UNTRUSTED_RECTANGLE= 487  495     0 1044
+!EXCLUSION OF HORIZONTAL DEAD AREAS OF THE PILATUS3 1M, S/N 10-0159
+UNTRUSTED_RECTANGLE=   0 982   195  213
+UNTRUSTED_RECTANGLE=   0 982   407  425
+UNTRUSTED_RECTANGLE=   0 982   619  637
+UNTRUSTED_RECTANGLE=   0 982   831  849
+eof
+    else
+    echo Flat_field found, smaller UNTRUSTED_RECTANGLEs than for bigger Pilatus
+    cat >> XDS.INP << eof
+!EXCLUSION OF VERTICAL DEAD AREAS OF THE PILATUS3 1M, S/N 10-0159
+UNTRUSTED_RECTANGLE= 488  494     0 1044
+!EXCLUSION OF HORIZONTAL DEAD AREAS OF THE PILATUS3 1M, S/N 10-0159
+UNTRUSTED_RECTANGLE=   0 982   196  212
+UNTRUSTED_RECTANGLE=   0 982   408  424
+UNTRUSTED_RECTANGLE=   0 982   620  636
+UNTRUSTED_RECTANGLE=   0 982   832  848
+eof
+    fi
+  elif [ "$NX" == "2073" -a "$NY" == "2180" ]; then
+# Pilatus 4M CdTe ; v1.24 numbers from Jan Gebauer
+    cat >> XDS.INP << eof
+!masking non sensitive area of Pilatus 4M CdTe
+ UNTRUSTED_RECTANGLE= 0 2074 255 276
+ UNTRUSTED_RECTANGLE= 0 2074 530 551
+ UNTRUSTED_RECTANGLE= 0 2074 805 826
+ UNTRUSTED_RECTANGLE= 0 2074 1080 1101
+ UNTRUSTED_RECTANGLE= 0 2074 1355 1376
+ UNTRUSTED_RECTANGLE= 0 2074 1630 1651
+ UNTRUSTED_RECTANGLE= 0 2074 1905 1926
+ UNTRUSTED_RECTANGLE= 513 521 0 2181
+ UNTRUSTED_RECTANGLE= 1033 1041 0 2181
+ UNTRUSTED_RECTANGLE= 1553 1561 0 2181
+ UNTRUSTED_RECTANGLE= 255 257 0 2181
+ UNTRUSTED_RECTANGLE= 775 777 0 2181
+ UNTRUSTED_RECTANGLE= 1295 1297 0 2181
+ UNTRUSTED_RECTANGLE= 1815 1817 0 2181
+eof
+  elif [ "$NX" == "1553" -a "$NY" == "1630" ]; then
+# Pilatus4 Si 2M ; v1.25
+    cat >> XDS.INP << eof
+!masking non sensitive area of Pilatus4 Si 2M
+! UNTRUSTED_RECTANGLE= lines not needed because pixel counts are -1
+eof
+# Pilatus 12M
    elif [ "$NX" == "2463" -a "$NY" == "5071" ]; then
      : # handled above, before writing XDS.INP, untrusted regions come from site file
@@ Line 1,289: / Line 1,516: @@
 echo More documentation in wiki.uni-konstanz.de/xds/index.php .
 echo After running xds, inspect at least the agreement of predicted and observed
-echo spots in FRAME.cbf!
+echo "spots in FRAME.cbf (or SHOW_HKL.cbf in versions since BUILT=20240723)!"
 rm -f tmp1 tmp2
-# end of generate_XDS.INP
+# end of generate_XDS.INP</pre>
-</pre>
+</div>
+</div>
 == System-wide or personal installation ==
@@ Line 1,332: / Line 1,560: @@
   chmod +x generate_XDS.INP
 </pre>
-If you do use cut-and-paste from the webpage, be aware of the following problem report: On the Mac, after loading frames, by clicking “generate XDS.INP”, the program gives some strange symbol “Ã” in XDS.INP. And the more you click “save” button, the more “Ã” appear. This looks like e.g. <br>SPACE_GROUP_NUMBER=0 Ã ! 0 if unknown <br>UNIT_CELL_CONSTANTS= 70 80 90 90 90 90 Ã <br> ''The problem is due to the “Rich text” format in TextEdit when saving "generate_XDS.INP". It is solved by re-downloading the script, and changing format to Plain - everything should work then.''
+If you use TextEdit on a Mac to cut-and-paste from this webpage, be aware of the following problem report: After loading frames, by clicking “generate XDS.INP”, the program shows some strange symbol “Ã” in XDS.INP. And the more you click “save” button, the more “Ã” appear. This looks like e.g. <br>SPACE_GROUP_NUMBER=0 Ã ! 0 if unknown <br>UNIT_CELL_CONSTANTS= 70 80 90 90 90 90 Ã <br>''The problem is due to the “Rich text” format in TextEdit when saving "generate_XDS.INP". It is solved by re-downloading the script, and changing format to Plain - everything should work then.''
 == Calling generate_XDS.INP from a Python script ==
@@ Line 1,350: / Line 1,578: @@
 The script makes use of many GNU commands, like <code>ls, grep, egrep, awk, cut, cat, echo, wc, bc, head, sed, tail, cp, od, python</code>. Some of them (like <code>od</code> and <code>python</code>) are only used in case of specific detectors (MarCCD and RAXIS, respectively).
 The script will only work if all the required commands are available. They reside in either the <code>coreutils</code> package, or specific packages (<code>gawk, sed, bc, grep, python</code> ...). Please note that to get the <code>strings</code> command on some Linux distributions, you need to install the <code>binutils</code> package.
-For Eiger data processing, the <code>h5dump</code> program must be installed. This is part of <code>hdf5-tools</code> (Ubuntu) or <code>hdf5</code> (RHEL). The .h5 files collected at Diamond Light Source require a very new version of h5dump (namely h5dump 1.10) to extract the OVERLOAD parameter from the .h5 file; this version is available by default in Ubuntu 20.04 and RHEL/CentOS 8.
+For Eiger data processing, the <code>h5dump</code> program must be installed. This is part of <code>hdf5-tools</code> (Ubuntu) or <code>hdf5</code> (RHEL). The .h5 files collected at Diamond Light Source require at least version 1.10 of h5dump to extract the OVERLOAD parameter from the .h5 file; this version is available in newer Linux distributions but not in RHEL/CentOS 7.
 On Mac OS X, installation of the "Command Line Tools" (from http://developer.apple.com/downloads; requires Apple ID) is required (open a terminal and type <code>xcode-select --install</code>). These are also part of the (larger, but also free) [http://developer.apple.com/tools/xcode Xcode] package. This package comes with a license that has to be accepted by the user when running a Command Line Tool (e.g. <code>strings</code>) for the first time.
@@ Line 1,371: / Line 1,599: @@
 == Site Files ==
-Site files may be used to add or modify parameters for specific hardware configurations. The primary use is for SEGMENT defintions of multi-segment detectors, and may also override FRACTION_OF_POLARIZATION, ROTATION_AXIS and TRUSTED_REGION specific to a site configuration that is not recognized by the script. The site file is saved as: ~/.xds-site/<NAME>_<SN>. The exact name will be printed by the script.
+Site files may be used to add or modify parameters for specific hardware configurations. The primary use is for SEGMENT definitions of multi-segment detectors, and may also override FRACTION_OF_POLARIZATION, ROTATION_AXIS, TRUSTED_REGION, or UNTRUSTED_* specific to a site configuration. The site file is named ~/.xds-site/<NAME>_<SN>. The exact name will be printed by generate_XDS.INP when it is executed.
-Site file for PILATUS 12M-DLS detector for the I23 beamline at Diamond Light Source:
+[https://wiki.uni-konstanz.de/pub/linux_bin/PILATUS_12M_SN_120-0100 Site file for PILATUS 12M-DLS detector for the I23 beamline at Diamond Light Source]:
+<div class="toccolours mw-collapsible mw-collapsed">
+Expand code section below (i.e. click on blue <code>[Expand]</code> at the end of this line if there is no code visible)
+<div class="mw-collapsible-content">
 <pre>
 !--------------------------------------------------------------
@@ Line 2,225: / Line 2,456: @@
 SEGMENT_DISTANCE=   694.72
 </pre>
+</div>
+</div>
 == Limitations ==
-* The script tries to interpret the header of the frames, so is currently limited to Dectris (Pilatus, Eiger), ADSC (Quantum), Rigaku (several types), MAR (CCD and image plate) detectors, and one Bruker detector. Other detectors need some values to be manually filled into XDS.INP - the relevant places are marked with XXX. These are detector properties (type, pixel size and number, min and max counts in a pixel), and experimental parameters like ROTATION_AXIS, OSCILLATION_RANGE, X-RAY_WAVELENGTH, DETECTOR_DISTANCE, and XORG, YORG. For fine-tuning of detector parameters, see the [http://xds.mpimf-heidelberg.mpg.de/html_doc/xds_prepare.html detector-specific templates].
+* The script tries to interpret the header of the frames, so is currently limited to Dectris (Pilatus, Eiger), ADSC (Quantum), Rigaku (several types), MAR (CCD and image plate) detectors, and one Bruker detector, and Bruker sfrm Version 18 . Other detectors need some values to be manually filled into XDS.INP - the relevant places are marked with XXX. These are detector properties (type, pixel size and number, min and max counts in a pixel), and experimental parameters like ROTATION_AXIS, OSCILLATION_RANGE, X-RAY_WAVELENGTH, DETECTOR_DISTANCE, and XORG, YORG. For fine-tuning of detector parameters, see the [http://xds.mpimf-heidelberg.mpg.de/html_doc/xds_prepare.html detector-specific templates].
 * The authors have made a "best effort" to provide a XDS.INP that results in the correct sign of the anomalous signal. In the case of one detector type (internally called Rigaku SMV) this requires reversal of one detector axis, and a negative DETECTOR_DISTANCE, as is found in some of the [http://xds.mpimf-heidelberg.mpg.de/html_doc/xds_prepare.html detector-specific templates]. '''For an unusual or unknown detector setup, the correct sign of the anomalous signal needs to be established and verified e.g. with a good dataset from a test crystal that has an anomalous signal.''' The authors do not take any responsibility for problems arising from incorrect sign of the anomalous signal, nor - obviously! - for any other mischief arising in or from data processing.
-* At some beamlines, the ROTATION_AXIS should be -1 0 0 ("backwards") instead of the usual 1 0 0 ("horizontal"), or even 0 1 0 ("vertical") like at one of the PETRA Hamburg BLs. The frame headers do not have this information, so the default chosen by [[generate_XDS.INP]] may be wrong and need manual correction. This can be automated with a site file. Pls also see the article [[Beamline notes]].
+* At some beamlines, the ROTATION_AXIS needs to be -1 0 0 ("backwards") instead of the usual 1 0 0 ("horizontal"), or even 0 1 0 ("vertical") like at one of the PETRA Hamburg BLs. The frame headers do not have this information, but for some beamlines the correct value has been implemented in the script. For other beamlines, the default chosen by [[generate_XDS.INP]] may be wrong and need manual correction of XDS.INP. The correct choice can be enforced by you with a [[Generate_XDS.INP#Site_Files|site file]]. Pls also see the article [[Beamline notes]].
-* Sometimes, the x- and y- values of the primary beam position recorded in the header should be used for ORGY and ORGX (i.e reversed) instead of as ORGX and ORGY. For ADSC, this has been implemented in the script for a number of beamlines.
+* Sometimes, the x- and y- values of the primary beam position recorded in the header should be used for ORGY and ORGX (i.e reversed) instead of as ORGX and ORGY. For ADSC, this has been implemented in the script for certain beamlines.
 * there are apparently several flavours of HDF5 files produced at Diamond Light Source. They differ e.g. in the naming of the header items. This means that items like NX, NY, DETECTOR_DISTANCE and number of images cannot always be determined by the [[generate_XDS.INP]] script. Example: The data at /dls/i04-1/data/2021/mx28114-9/processing/Lenye_Diamini/ThiL/ThiL found during the CCP4 2021 online Cape Town workshop. A workaround is to use e.g. xia2 pipeline=3dii to process these files, and - if needed - extract those items from its output files, e.g. from DEFAULT/NATIVE/SWEEP1/index/XDS.INP .