3
edits
Generate XDS.INP: Difference between revisions
m
Bruker-sfrm mentioned. TH
(Added Site File description, and site file for Pilatus 12M) |
m (Bruker-sfrm mentioned. TH) Tags: Mobile edit Mobile web edit Visual edit |
||
| (18 intermediate revisions by 3 users not shown) | |||
| Line 1: | Line 1: | ||
This script generates XDS.INP based on a list of frame names supplied on the commandline. It currently works for MarCCD, ADSC, Pilatus, Eiger, some Rigaku and one Bruker detector(s); | This script generates XDS.INP based on a list of frame names supplied on the commandline. It currently works for MarCCD, ADSC, Pilatus, Eiger, some Rigaku and one Bruker detector(s); there is also some support for microED with SMV files which have a wavelength given as 0 (look for "experimental-ED" in the script). Since this is just a bash script, extension to other detectors should be very easy. | ||
== Usage == | == Usage == | ||
| Line 5: | Line 5: | ||
Usage is just (don't forget the quotation marks!): | Usage is just (don't forget the quotation marks!): | ||
generate_XDS.INP "/home/myname/frms/mydata_1_???.img" | generate_XDS.INP "/home/myname/frms/mydata_1_???.img" | ||
XDS [ | XDS [https://xds.mr.mpg.de/html_doc/xds_parameters.html#NAME_TEMPLATE_OF_DATA_FRAMES= supports] bzip2-ed frames. Thus, when specifying the frame name parameter of the script, you should leave out any .bz2 extension. | ||
For improved interaction with [[XDSGUI]], it is advantageous to provide an ''absolute'' filename for the data files - one that starts with a slash ("/"). | For improved interaction with [[XDSGUI]], it is advantageous to provide an ''absolute'' filename for the data files - one that starts with a slash ("/"). | ||
| Line 11: | Line 11: | ||
== The script == | == The script == | ||
<!-- leave a blank line after <pre>! KD 10/2023 --> | <!-- leave a blank line after <pre>! KD 10/2023 --> | ||
<div class="toccolours mw-collapsible mw-collapsed"> | |||
Expand code section below (i.e. click on blue <code>[Expand]</code> at the end of this line if there is no code visible) | |||
<div class="mw-collapsible-content"> | |||
<pre> | <pre> | ||
| Line 145: | Line 149: | ||
# revision 1.18 . JMK 19/08/2023 Pilatus 12M support, give help msg if $5 defined (i.e. unquoted frame pattern) | # revision 1.18 . JMK 19/08/2023 Pilatus 12M support, give help msg if $5 defined (i.e. unquoted frame pattern) | ||
# revision 1.19 . KD 14/10/2023 Pilatus 12M at DLS I23 has reverse phi. Fix download commands. | # revision 1.19 . KD 14/10/2023 Pilatus 12M at DLS I23 has reverse phi. Fix download commands. | ||
REVISION="1. | # revision 1.20 . KD 13/7/2024 Comment out line STRONG_PIXEL=4 in preparation for mid-2024 version of XDS. | ||
# revision 1.21 . Thomas Hauß 12/03/2025 define for Bruker sfrm format DET= Bruker-sfrm and corresponding section | |||
REVISION="1.21 (13-Mar-2025)" | |||
# | # | ||
| Line 279: | Line 285: | ||
grep -q MARCONTROL tmp2 && DET=OLDMAR | grep -q MARCONTROL tmp2 && DET=OLDMAR | ||
grep -q "WAVELENGTH=0.0" tmp2 && DET=experimental-ED | grep -q "WAVELENGTH=0.0" tmp2 && DET=experimental-ED | ||
if head -c 240 tmp2 | grep -q "FORMAT" && \ | |||
head -c 240 tmp2 | grep -q "VERSION" && \ | |||
head -c 240 tmp2 | grep -q "HDRBLKS"; then | |||
DET="Bruker-sfrm" | |||
fi | |||
else | else | ||
h5dump -d "/entry/instrument/detector/description" $FIRSTFRAME | grep -i Eiger > /dev/null && DET=eiger | h5dump -d "/entry/instrument/detector/description" $FIRSTFRAME | grep -i Eiger > /dev/null && DET=eiger | ||
| Line 950: | Line 961: | ||
DIRECTION_OF_DETECTOR_X_AXIS="$R1 0 $R3" | DIRECTION_OF_DETECTOR_X_AXIS="$R1 0 $R3" | ||
# end of Bruker-cbf section | # end of Bruker-cbf section | ||
elif [ "$DET" == "Bruker-sfrm" ]; then | |||
# echo "Detector from Bruker in sfrm format." | |||
# MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT: | |||
MNOPIAS=6 | |||
# use complete detector including corners: | |||
TRUSTED_REGION="0 1.42" | |||
SEPMIN=4 # 4 for Pixel Array Detectors | |||
CLUSTER_RADIUS=2 # 2 for Pixel Array Detectors | |||
# to read different format versions read these parametes first: | |||
FORMAT=$(awk -F'FORMAT :' '{print $2}' tmp2 | awk '{print $1}') | |||
VERSION=$(awk -F'VERSION:' '{print $2}' tmp2 | awk '{print $1}') | |||
DETTYPE=$(awk -F'DETTYPE:' '{print $2}' tmp2 | awk '{print $1}') | |||
DETTYPE="BRUKER_$DETTYPE" | |||
echo "Bruker-sfrm file Format: $FORMAT, Version: $VERSION" | |||
# use last value from CCDPARM and convert to integer | |||
OVERLOAD=$(awk -F'CCDPARM:' '{print $2}' tmp2 | awk '{print int($5)}' | head -n1) | |||
DETECTOR="${DETTYPE} MINIMUM_VALID_PIXEL_VALUE=0 OVERLOAD=${OVERLOAD}" | |||
echo "DETECTOR $DETECTOR" | |||
NY=$(awk -F'NROWS :' '{print $2}' tmp2 | awk '{print $1}') | |||
NX=$(awk -F'NCOLS :' '{print $2}' tmp2 | awk '{print $1}') | |||
echo "NX" $NX | |||
echo "NY" $NY | |||
read ORGX ORGY < <(awk -F'CENTER :' '{print $2}' tmp2 | awk '{print $1, $2}') | |||
# There seems to be an error in ORGY, the offset from the detector | |||
# center is calculated wrong with a in inverted sign. Here an attempt | |||
# to correct this, may be not valid in all cases. Found via comparison | |||
# with the Bruker CBF format. There the value of V should be positive! | |||
ORGY=$(echo "scale=2; $NY - $ORGY" | bc -l) | |||
echo "ORGX: $ORGX" | |||
echo "ORGY: $ORGY" | |||
# pixelsize from detector width NX and pixel per cm | |||
pixelpercm=$(awk -F'DETTYPE:' '{print $2}' tmp2 | awk '{print $2}') | |||
if [ NX == 512 ]; then | |||
QX=$(echo "scale=6; 10.0 / $pixelpercm" | bc -l) | |||
QY=QX=$(echo "scale=6; 10.0 / $pixelpercm" | bc -l) | |||
else | |||
QX=$(echo "scale=6; 5.0 / $pixelpercm" | bc -l) | |||
QY=$(echo "scale=6; 5.0 / $pixelpercm" | bc -l) | |||
fi | |||
# QX=0.135 # from Bruker detector specification | |||
# QY=0.135 | |||
echo "QX: $QX" | |||
echo "QY: $QY" | |||
# With Version < 11, only one distance written to header, then 2, second value seems ok. | |||
if [ "$VERSION" -lt 11 ]; then | |||
DETECTOR_DISTANCE=$(awk -F'DISTANC:' '{print $2}' tmp2 | awk '{print $1}') | |||
else | |||
DETECTOR_DISTANCE=$(awk -F'DISTANC:' '{print $2}' tmp2 | awk '{print $2}') | |||
fi | |||
DETECTOR_DISTANCE=$(echo "$DETECTOR_DISTANCE * 10" | bc -l) # convert cm to mm | |||
echo "DETECTOR_DISTANCE: $DETECTOR_DISTANCE" | |||
X_RAY_WAVELENGTH=$(awk -F'WAVELEN:' '{print $2}' tmp2 | awk '{print $1}') | |||
echo "X_RAY_WAVELENGTH: $X_RAY_WAVELENGTH" | |||
read TWOTHETA OMEGA PHI KAPPA < <(awk -F'ANGLES :' '{print $2}' tmp2 | awk '{print $1, $2, $3, $4}') | |||
echo "TWOTHETA OMEGA PHI KAPPA: $TWOTHETA $OMEGA $PHI $KAPPA" | |||
# axis 1: 2-theta, 2: omega, 3: phi, 4: chi | |||
axis=$(awk -F'AXIS :' '{print $2}' tmp2 | awk '{print $1}') | |||
if [ "$axis" == 3 ]; then | |||
# PHI scan | |||
echo "PHI scan" | |||
DELTAPHI=$(awk -F'INCREME:' '{print $2}' tmp2 | awk '{print $1}') | |||
R3=$(bc -l <<< "scale=7; s($KAPPA/$DEGTOR) * s($OMEGA/$DEGTOR)") | |||
R1=$(bc -l <<< "scale=7; s($KAPPA/$DEGTOR) * c($OMEGA/$DEGTOR)") | |||
R2=$(bc -l <<< "scale=7; c($KAPPA/$DEGTOR)") | |||
ROTATION_AXIS="$R1 $R2 $R3" | |||
OSCILLATION_RANGE=${DELTAPHI} | |||
STARTING_ANGLE=${PHI} | |||
else | |||
echo "OMEGA scan" | |||
ROTATION_AXIS="0 -1 0" | |||
DELTAOMEGA=$(awk -F'INCREME:' '{print $2}' tmp2 | awk '{print $1}') | |||
OSCILLATION_RANGE=${DELTAOMEGA} | |||
STARTING_ANGLE=${OMEGA} | |||
fi | |||
# detector X-axis | |||
R1=$(bc -l <<< "scale=7; c($TWOTHETA/$DEGTOR)") | |||
R3=$(bc -l <<< "scale=7; s($TWOTHETA/$DEGTOR)") | |||
DIRECTION_OF_DETECTOR_X_AXIS="$R1 0 $R3" | |||
echo "DIRECTION_OF_DETECTOR_X_AXIS: $DIRECTION_OF_DETECTOR_X_AXIS" | |||
echo "STARTING_ANGLE $STARTING_ANGLE" | |||
echo "OSCILLATION_RANGE $OSCILLATION_RANGE" | |||
echo "ROTATION_AXIS $ROTATION_AXIS" | |||
# end of Bruker-sfrm section | |||
else | else | ||
echo should never come here | echo should never come here | ||
| Line 1,015: | Line 1,122: | ||
! More documentation in wiki.uni-konstanz.de/xds/index.php . | ! More documentation in wiki.uni-konstanz.de/xds/index.php . | ||
! After running xds, inspect at least the agreement of predicted and observed | ! After running xds, inspect at least the agreement of predicted and observed | ||
! spots in FRAME.cbf! | ! spots in FRAME.cbf (or SHOW_HKL.cbf in versions since BUILT=20240723)! | ||
!============================================================================= | !============================================================================= | ||
JOB= XYCORR INIT COLSPOT IDXREF DEFPIX INTEGRATE CORRECT | JOB= XYCORR INIT COLSPOT IDXREF DEFPIX INTEGRATE CORRECT | ||
| Line 1,052: | Line 1,159: | ||
TRUSTED_REGION=$TRUSTED_REGION | TRUSTED_REGION=$TRUSTED_REGION | ||
VALUE_RANGE_FOR_TRUSTED_DETECTOR_PIXELS=6000. 30000. ! often 7000 or 8000 is ok | VALUE_RANGE_FOR_TRUSTED_DETECTOR_PIXELS=6000. 30000. ! often 7000 or 8000 is ok | ||
STRONG_PIXEL=4 | !STRONG_PIXEL=4 ! COLSPOT: only use strong reflections (default is 3). For XDS versions before mid-2024. | ||
MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT=$MNOPIAS ! default of 6 is sometimes too high | MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT=$MNOPIAS ! default of 6 is sometimes too high | ||
! close spots/long cell axis: reduce SEPMIN and CLUSTER_RADIUS from their defaults of 7 and 3.5 | ! close spots/long cell axis: reduce SEPMIN and CLUSTER_RADIUS from their defaults of 7 and 3.5 | ||
| Line 1,098: | Line 1,205: | ||
eof | eof | ||
# --------- Append additional detector-specific parameters ---------- | # --------- Append additional detector-specific parameters ---------- | ||
if [ "$DET" == "Bruker-cbf" ]; then | if [ "$DET" == "Bruker-cbf" -o "$DET" == "Bruker-sfrm" ]; then | ||
echo "DELPHI=15 ! refine less often than the default of 5" >> XDS.INP | echo "DELPHI=15 ! refine less often than the default of 5" >> XDS.INP | ||
elif [ "$DET" == "adsc-CMOS1" ]; then | elif [ "$DET" == "adsc-CMOS1" ]; then | ||
| Line 1,289: | Line 1,396: | ||
echo More documentation in wiki.uni-konstanz.de/xds/index.php . | echo More documentation in wiki.uni-konstanz.de/xds/index.php . | ||
echo After running xds, inspect at least the agreement of predicted and observed | echo After running xds, inspect at least the agreement of predicted and observed | ||
echo spots in FRAME.cbf! | echo "spots in FRAME.cbf (or SHOW_HKL.cbf in versions since BUILT=20240723)!" | ||
rm -f tmp1 tmp2 | rm -f tmp1 tmp2 | ||
# end of generate_XDS.INP | # end of generate_XDS.INP | ||
</pre> | </pre> | ||
</div> | |||
</div> | |||
== System-wide or personal installation == | == System-wide or personal installation == | ||
| Line 1,332: | Line 1,441: | ||
chmod +x generate_XDS.INP | chmod +x generate_XDS.INP | ||
</pre> | </pre> | ||
If you | If you use TextEdit on a Mac to cut-and-paste from this webpage, be aware of the following problem report: After loading frames, by clicking “generate XDS.INP”, the program shows some strange symbol “Ô in XDS.INP. And the more you click “save” button, the more “Ô appear. This looks like e.g. <br>SPACE_GROUP_NUMBER=0 à ! 0 if unknown <br>UNIT_CELL_CONSTANTS= 70 80 90 90 90 90 à <br>''The problem is due to the “Rich text” format in TextEdit when saving "generate_XDS.INP". It is solved by re-downloading the script, and changing format to Plain - everything should work then.'' | ||
== Calling generate_XDS.INP from a Python script == | == Calling generate_XDS.INP from a Python script == | ||
| Line 1,350: | Line 1,459: | ||
The script makes use of many GNU commands, like <code>ls, grep, egrep, awk, cut, cat, echo, wc, bc, head, sed, tail, cp, od, python</code>. Some of them (like <code>od</code> and <code>python</code>) are only used in case of specific detectors (MarCCD and RAXIS, respectively). | The script makes use of many GNU commands, like <code>ls, grep, egrep, awk, cut, cat, echo, wc, bc, head, sed, tail, cp, od, python</code>. Some of them (like <code>od</code> and <code>python</code>) are only used in case of specific detectors (MarCCD and RAXIS, respectively). | ||
The script will only work if all the required commands are available. They reside in either the <code>coreutils</code> package, or specific packages (<code>gawk, sed, bc, grep, python</code> ...). Please note that to get the <code>strings</code> command on some Linux distributions, you need to install the <code>binutils</code> package. | The script will only work if all the required commands are available. They reside in either the <code>coreutils</code> package, or specific packages (<code>gawk, sed, bc, grep, python</code> ...). Please note that to get the <code>strings</code> command on some Linux distributions, you need to install the <code>binutils</code> package. | ||
For Eiger data processing, the <code>h5dump</code> program must be installed. This is part of <code>hdf5-tools</code> (Ubuntu) or <code>hdf5</code> (RHEL). The .h5 files collected at Diamond Light Source require | For Eiger data processing, the <code>h5dump</code> program must be installed. This is part of <code>hdf5-tools</code> (Ubuntu) or <code>hdf5</code> (RHEL). The .h5 files collected at Diamond Light Source require at least version 1.10 of h5dump to extract the OVERLOAD parameter from the .h5 file; this version is available in newer Linux distributions but not in RHEL/CentOS 7. | ||
On Mac OS X, installation of the "Command Line Tools" (from http://developer.apple.com/downloads; requires Apple ID) is required (open a terminal and type <code>xcode-select --install</code>). These are also part of the (larger, but also free) [http://developer.apple.com/tools/xcode Xcode] package. This package comes with a license that has to be accepted by the user when running a Command Line Tool (e.g. <code>strings</code>) for the first time. | On Mac OS X, installation of the "Command Line Tools" (from http://developer.apple.com/downloads; requires Apple ID) is required (open a terminal and type <code>xcode-select --install</code>). These are also part of the (larger, but also free) [http://developer.apple.com/tools/xcode Xcode] package. This package comes with a license that has to be accepted by the user when running a Command Line Tool (e.g. <code>strings</code>) for the first time. | ||
| Line 1,371: | Line 1,480: | ||
== Site Files == | == Site Files == | ||
Site files may be used to add or modify parameters for specific hardware configurations. The primary use is for SEGMENT | Site files may be used to add or modify parameters for specific hardware configurations. The primary use is for SEGMENT definitions of multi-segment detectors, and may also override FRACTION_OF_POLARIZATION, ROTATION_AXIS, TRUSTED_REGION, or UNTRUSTED_* specific to a site configuration. The site file is named ~/.xds-site/<NAME>_<SN>. The exact name will be printed by generate_XDS.INP when it is executed. | ||
Site file for PILATUS 12M-DLS detector for the I23 beamline at Diamond Light Source: | [https://wiki.uni-konstanz.de/pub/linux_bin/PILATUS_12M_SN_120-0100 Site file for PILATUS 12M-DLS detector for the I23 beamline at Diamond Light Source]: | ||
<div class="toccolours mw-collapsible mw-collapsed"> | |||
Expand code section below (i.e. click on blue <code>[Expand]</code> at the end of this line if there is no code visible) | |||
<div class="mw-collapsible-content"> | |||
<pre> | <pre> | ||
!-------------------------------------------------------------- | !-------------------------------------------------------------- | ||
| Line 2,225: | Line 2,337: | ||
SEGMENT_DISTANCE= 694.72 | SEGMENT_DISTANCE= 694.72 | ||
</pre> | </pre> | ||
</div> | |||
</div> | |||
== Limitations == | == Limitations == | ||
* The script tries to interpret the header of the frames, so is currently limited to Dectris (Pilatus, Eiger), ADSC (Quantum), Rigaku (several types), MAR (CCD and image plate) detectors, and one Bruker detector. Other detectors need some values to be manually filled into XDS.INP - the relevant places are marked with XXX. These are detector properties (type, pixel size and number, min and max counts in a pixel), and experimental parameters like ROTATION_AXIS, OSCILLATION_RANGE, X-RAY_WAVELENGTH, DETECTOR_DISTANCE, and XORG, YORG. For fine-tuning of detector parameters, see the [http://xds.mpimf-heidelberg.mpg.de/html_doc/xds_prepare.html detector-specific templates]. | * The script tries to interpret the header of the frames, so is currently limited to Dectris (Pilatus, Eiger), ADSC (Quantum), Rigaku (several types), MAR (CCD and image plate) detectors, and one Bruker detector, and Bruker sfrm Version 18 . Other detectors need some values to be manually filled into XDS.INP - the relevant places are marked with XXX. These are detector properties (type, pixel size and number, min and max counts in a pixel), and experimental parameters like ROTATION_AXIS, OSCILLATION_RANGE, X-RAY_WAVELENGTH, DETECTOR_DISTANCE, and XORG, YORG. For fine-tuning of detector parameters, see the [http://xds.mpimf-heidelberg.mpg.de/html_doc/xds_prepare.html detector-specific templates]. | ||
* The authors have made a "best effort" to provide a XDS.INP that results in the correct sign of the anomalous signal. In the case of one detector type (internally called Rigaku SMV) this requires reversal of one detector axis, and a negative DETECTOR_DISTANCE, as is found in some of the [http://xds.mpimf-heidelberg.mpg.de/html_doc/xds_prepare.html detector-specific templates]. '''For an unusual or unknown detector setup, the correct sign of the anomalous signal needs to be established and verified e.g. with a good dataset from a test crystal that has an anomalous signal.''' The authors do not take any responsibility for problems arising from incorrect sign of the anomalous signal, nor - obviously! - for any other mischief arising in or from data processing. | * The authors have made a "best effort" to provide a XDS.INP that results in the correct sign of the anomalous signal. In the case of one detector type (internally called Rigaku SMV) this requires reversal of one detector axis, and a negative DETECTOR_DISTANCE, as is found in some of the [http://xds.mpimf-heidelberg.mpg.de/html_doc/xds_prepare.html detector-specific templates]. '''For an unusual or unknown detector setup, the correct sign of the anomalous signal needs to be established and verified e.g. with a good dataset from a test crystal that has an anomalous signal.''' The authors do not take any responsibility for problems arising from incorrect sign of the anomalous signal, nor - obviously! - for any other mischief arising in or from data processing. | ||
* At some beamlines, the ROTATION_AXIS | * At some beamlines, the ROTATION_AXIS needs to be -1 0 0 ("backwards") instead of the usual 1 0 0 ("horizontal"), or even 0 1 0 ("vertical") like at one of the PETRA Hamburg BLs. The frame headers do not have this information, but for some beamlines the correct value has been implemented in the script. For other beamlines, the default chosen by [[generate_XDS.INP]] may be wrong and need manual correction of XDS.INP. The correct choice can be enforced by you with a [[Generate_XDS.INP#Site_Files|site file]]. Pls also see the article [[Beamline notes]]. | ||
* Sometimes, the x- and y- values of the primary beam position recorded in the header should be used for ORGY and ORGX (i.e reversed) instead of as ORGX and ORGY. For ADSC, this has been implemented in the script for | * Sometimes, the x- and y- values of the primary beam position recorded in the header should be used for ORGY and ORGX (i.e reversed) instead of as ORGX and ORGY. For ADSC, this has been implemented in the script for certain beamlines. | ||
* there are apparently several flavours of HDF5 files produced at Diamond Light Source. They differ e.g. in the naming of the header items. This means that items like NX, NY, DETECTOR_DISTANCE and number of images cannot always be determined by the [[generate_XDS.INP]] script. Example: The data at /dls/i04-1/data/2021/mx28114-9/processing/Lenye_Diamini/ThiL/ThiL found during the CCP4 2021 online Cape Town workshop. A workaround is to use e.g. xia2 pipeline=3dii to process these files, and - if needed - extract those items from its output files, e.g. from DEFAULT/NATIVE/SWEEP1/index/XDS.INP . | * there are apparently several flavours of HDF5 files produced at Diamond Light Source. They differ e.g. in the naming of the header items. This means that items like NX, NY, DETECTOR_DISTANCE and number of images cannot always be determined by the [[generate_XDS.INP]] script. Example: The data at /dls/i04-1/data/2021/mx28114-9/processing/Lenye_Diamini/ThiL/ThiL found during the CCP4 2021 online Cape Town workshop. A workaround is to use e.g. xia2 pipeline=3dii to process these files, and - if needed - extract those items from its output files, e.g. from DEFAULT/NATIVE/SWEEP1/index/XDS.INP . | ||