XDSwiki - User contributions [en]

Electron diffraction

2026-02-18T21:25:18Z

Grunet: special aspects for XDS when processing ED data.

XDS can process electron diffraction data as long as the general preliminaries for XDS are met. In brief, this means the crystal is illuminated with a (quasi-)parallel, non-moving beam while the crystal rotates with a constant velocity about a fixed rotation axis, and the detector is composed of planar segments. These are quite general conditions, making XDS very suitable for the processing of electron diffraction data. XDS can even correct for optical distortions of the electromagnetic lens system of the electron source, especially when they are in-plane with the detector and do not change during data collection <ref>http://doi.org/10.1107/S160057672200276X</ref>. Compared with X-ray diffraction, the crystallographer should be far more flexible in the acceptance of quality indicators.

=== Processing (REFINE= keywords) ===
The very short wavelength results in a very small maximum scattering angle 2θ<sub>max</sub> of less than 5°. With such, sin(x)≈x, with the result of some strong correlations <ref>https://doi.org/10.1107/S2059798318007726</ref>between parameters. It is usually not possible to refine BEAM and POSITION simultaneously. A recommended setting for XDS.INP:
REFINE(IDXREF)=BEAM AXIS ORIENTATION CELL

REFINE(INTEGRATE)=BEAM AXIS ORIENTATION

REFINE(CORRECT)=BEAM AXIS ORIENTATION CELL
When resolution is sufficiently high, at about d<sub>min</sub> < 0.8Å, both CELL and POSITION can be refined (but not the BEAM):
REFINE(CORRECT)=AXIS ORIENTATION CELL POSITION
In other cases, the INCIDENT_BEAM_CORRECTION and UNIT_CELL_CONSTANTS can be copied from XDS_ASCII.HKL to XDS.INP, and only the POSITION is refined.
REFINE(CORRECT)=AXIS ORIENTATION POSITION
(AXIS and ORIENTATION are usually very robust to be refined).

=== Scaling ===
Scaling works best with high multiplicity. With electron diffraction data, this is not always granted, because the rotation range is at best 150° and can be limited to much less. With low symmetry spacegroups, this can result in low multiplicity. On the other hand, electron diffraction data are typically collected from a multitude of crystals, and a subset of well-fitting ones are merged to increase data quality and multiplicity. In such cases, the parameter SNRC=50 <ref>https://doi.org/10.1107/S2059798320006348</ref> should be added to XDS.INP to switch off scaling for individual data sets and only scale when merging data from multiple crystals.

Other parameters for XSCALE are usually not needed. It turns out that the scaling model in XDS and XSCALE is well suited for ED data, too.

Bruker data

2020-01-29T08:53:34Z

Grunet:

== SFRM files ==

Bruker instruments save frames in SFRM file format.
XDS correctly reads these files. Data from CCD detectors, however, need to be "unwarped"; unwarping can be done within the Bruker software called Proteum2. See also [[ccp4com:Bruker_software|Bruker software]]. Data from more recent detectors, in particular all Photon detectors, do not need to unwarped and can be processed directly with XDS.

For such files, [https://www.jiscmail.ac.uk/cgi-bin/webadmin?A2=CCP4BB;ce43e97e.1907 Tim Grüne] provides the program <code>sfrmtools</code> at [https://homepage.univie.ac.at/tim.gruene/research/programs/conv/sfrmtools/]. It extracts parameters relevant for data processing with XDS from the header. The output should be saved in a file - only three more keyword/parameter lines are required for a minimal XDS.INP, namely e.g.
NAME_TEMPLATE_OF_DATA_FRAMES=/my/data/name_????.sfrm
OVERLOAD=1000000
INCIDENT_BEAM_DIRECTION=0 0 1
Additional ones can of course be used to modify the operation of XDS, e.g.
JOB= XYCORR INIT COLSPOT IDXREF DEFPIX INTEGRATE CORRECT
INCLUDE_RESOLUTION_RANGE=50 1
TRUSTED_REGION=0 1.4142
! UNTRUSTED_QUADRILATERAL= <8 numbers, easily specified using XDSGUI>
...

== CBF files ==

New detector types (optionally?) write CBF files, which XDS natively reads. [[generate_XDS.INP]] generates XDS.INP for CBF data written by the PHOTON II detector.

Timepix2cbf

2018-01-09T15:37:31Z

Grunet: /* Availability */

== Program Description ==
The [http://www.amscins.com/ ASI] TIMEPIX detector is a direct electron detector used in electron crystallography. It has various output formats, e.g. TIFF, or a binary format.
In order to process the data with XDS, it is (currently) easiest to convert the files to [http://www.bernstein-plus-sons.com/software/CBF/doc/CBFlib.html CBF format].
The program timepix2cbf converts the ASI TIMEPIX binary format to cbf, while taking care of the special pixels at the rim of each 256x256 panel: These are 3 pixels long.
timepix2cbf splits these pixels into three. The intensities are distributed evenly, which is the simplest, but not the best, approach to maintaining the data. Instead of dividing the content of the rim-pixels by three, all other pixel values are multiplied by three. This maintains data precision of the integer values in the CBF files. The GAIN keyword in XDS or DIALS should thus be set to 3. In case XDS is allowed to determine the GAIN automatically, it should produce a similar value. Please read the INIT.LP log file, or take a look at GAIN.cbf.

== 1024x1024 vs. 512x512 panel ==
timepix2cbf works both for the TIMEPIX 1024x1024 detector and the 512x512 detector. The larger version, 1024x1024 contains gaps between the four 512x512 panels ("quads"). These gaps are detector specific and need to be described with the SEGMENT= keyword in XDS.

== Availability ==
timepix2cbf is available for download as source code ([[File:Timepix2cbf_src.zip]]), or as 64bit linux binary ([[File:Timepix2cbf_bin.zip]]). The source code contains a project description for [https://netbeans.org/ Netbeans], and the zip-archive can thus be imported into [https://netbeans.org/ Netbeans] directly. The linux binary is statically linked and should run on recent linux distributions.

Timepix2cbf

2018-01-09T14:42:11Z

Grunet: /* Availability */

== Program Description ==
The [http://www.amscins.com/ ASI] TIMEPIX detector is a direct electron detector used in electron crystallography. It has various output formats, e.g. TIFF, or a binary format.
In order to process the data with XDS, it is (currently) easiest to convert the files to [http://www.bernstein-plus-sons.com/software/CBF/doc/CBFlib.html CBF format].
The program timepix2cbf converts the ASI TIMEPIX binary format to cbf, while taking care of the special pixels at the rim of each 256x256 panel: These are 3 pixels long.
timepix2cbf splits these pixels into three. The intensities are distributed evenly, which is the simplest, but not the best, approach to maintaining the data. Instead of dividing the content of the rim-pixels by three, all other pixel values are multiplied by three. This maintains data precision of the integer values in the CBF files. The GAIN keyword in XDS or DIALS should thus be set to 3. In case XDS is allowed to determine the GAIN automatically, it should produce a similar value. Please read the INIT.LP log file, or take a look at GAIN.cbf.

== 1024x1024 vs. 512x512 panel ==
timepix2cbf works both for the TIMEPIX 1024x1024 detector and the 512x512 detector. The larger version, 1024x1024 contains gaps between the four 512x512 panels ("quads"). These gaps are detector specific and need to be described with the SEGMENT= keyword in XDS.

== Availability ==
timepix2cbf is available for download as source code ([[File:Timepix2cbf_src.zip]]), or as 64bit linux binary ([[File:Timepix2cbf_bin.zip)]]). The source code contains a project description for [https://netbeans.org/ Netbeans], and the zip-archive can thus be imported into [https://netbeans.org/ Netbeans] directly. The linux binary is statically linked and should run on recent linux distributions.

Timepix2cbf

2018-01-09T14:41:03Z

Grunet: /* Availability */

== Program Description ==
The [http://www.amscins.com/ ASI] TIMEPIX detector is a direct electron detector used in electron crystallography. It has various output formats, e.g. TIFF, or a binary format.
In order to process the data with XDS, it is (currently) easiest to convert the files to [http://www.bernstein-plus-sons.com/software/CBF/doc/CBFlib.html CBF format].
The program timepix2cbf converts the ASI TIMEPIX binary format to cbf, while taking care of the special pixels at the rim of each 256x256 panel: These are 3 pixels long.
timepix2cbf splits these pixels into three. The intensities are distributed evenly, which is the simplest, but not the best, approach to maintaining the data. Instead of dividing the content of the rim-pixels by three, all other pixel values are multiplied by three. This maintains data precision of the integer values in the CBF files. The GAIN keyword in XDS or DIALS should thus be set to 3. In case XDS is allowed to determine the GAIN automatically, it should produce a similar value. Please read the INIT.LP log file, or take a look at GAIN.cbf.

== 1024x1024 vs. 512x512 panel ==
timepix2cbf works both for the TIMEPIX 1024x1024 detector and the 512x512 detector. The larger version, 1024x1024 contains gaps between the four 512x512 panels ("quads"). These gaps are detector specific and need to be described with the SEGMENT= keyword in XDS.

== Availability ==
timepix2cbf is available for download as [[File:Timepix2cbf_src.zip|200px|thumb|left|source code]], or as [[File:Timepix2cbf_bin.zip | 64bit linux binary]]. The source code contains a project description for [https://netbeans.org/ Netbeans], and the zip-archive can thus be imported into [https://netbeans.org/ Netbeans] directly. The linux binary is statically linked and should run on recent linux distributions.

Timepix2cbf

2018-01-09T14:39:27Z

Grunet: /* Availability */

== Program Description ==
The [http://www.amscins.com/ ASI] TIMEPIX detector is a direct electron detector used in electron crystallography. It has various output formats, e.g. TIFF, or a binary format.
In order to process the data with XDS, it is (currently) easiest to convert the files to [http://www.bernstein-plus-sons.com/software/CBF/doc/CBFlib.html CBF format].
The program timepix2cbf converts the ASI TIMEPIX binary format to cbf, while taking care of the special pixels at the rim of each 256x256 panel: These are 3 pixels long.
timepix2cbf splits these pixels into three. The intensities are distributed evenly, which is the simplest, but not the best, approach to maintaining the data. Instead of dividing the content of the rim-pixels by three, all other pixel values are multiplied by three. This maintains data precision of the integer values in the CBF files. The GAIN keyword in XDS or DIALS should thus be set to 3. In case XDS is allowed to determine the GAIN automatically, it should produce a similar value. Please read the INIT.LP log file, or take a look at GAIN.cbf.

== 1024x1024 vs. 512x512 panel ==
timepix2cbf works both for the TIMEPIX 1024x1024 detector and the 512x512 detector. The larger version, 1024x1024 contains gaps between the four 512x512 panels ("quads"). These gaps are detector specific and need to be described with the SEGMENT= keyword in XDS.

== Availability ==
timepix2cbf is available for download as [[File:Timepix2cbf_src.zip]] source code, or as 64bit [[File:Timepix2cbf_bin.zip]] linux binary. The source code contains a project description for [https://netbeans.org/ Netbeans], and the zip-archive can thus be imported into [https://netbeans.org/ Netbeans] directly. The linux binary is statically linked and should run on recent linux distributions.

File:Timepix2cbf src.zip

2018-01-09T14:36:34Z

Grunet: Source code (Netbeans project) for timepix2cbf conversion program.

Source code (Netbeans project) for timepix2cbf conversion program.

File:Timepix2cbf bin.zip

2018-01-09T14:35:52Z

Grunet: Linux binary for the program timepix2cbf. 64bit executable, statically linked.

Linux binary for the program timepix2cbf. 64bit executable, statically linked.

Timepix2cbf

2018-01-08T19:54:50Z

Grunet: Timepix2cbf converts TIMEPIX binary format to CBF for processing electron diffraction data with XDS.

== Program Description ==
The [http://www.amscins.com/ ASI] TIMEPIX detector is a direct electron detector used in electron crystallography. It has various output formats, e.g. TIFF, or a binary format.
In order to process the data with XDS, it is (currently) easiest to convert the files to [http://www.bernstein-plus-sons.com/software/CBF/doc/CBFlib.html CBF format].
The program timepix2cbf converts the ASI TIMEPIX binary format to cbf, while taking care of the special pixels at the rim of each 256x256 panel: These are 3 pixels long.
timepix2cbf splits these pixels into three. The intensities are distributed evenly, which is the simplest, but not the best, approach to maintaining the data. Instead of dividing the content of the rim-pixels by three, all other pixel values are multiplied by three. This maintains data precision of the integer values in the CBF files. The GAIN keyword in XDS or DIALS should thus be set to 3. In case XDS is allowed to determine the GAIN automatically, it should produce a similar value. Please read the INIT.LP log file, or take a look at GAIN.cbf.

== 1024x1024 vs. 512x512 panel ==
timepix2cbf works both for the TIMEPIX 1024x1024 detector and the 512x512 detector. The larger version, 1024x1024 contains gaps between the four 512x512 panels ("quads"). These gaps are detector specific and need to be described with the SEGMENT= keyword in XDS.

== Availability ==
timepix2cbf is available for download as source code, or as 64bit linux binary. The source code contains a project description for [https://netbeans.org/ Netbeans], and the zip-archive can thus be imported into [https://netbeans.org/ Netbeans] directly. The linux binary is statically linked and should run on recent linux distributions.

Tif2cbf

2017-12-15T13:28:49Z

Grunet:

The program tif2cbf converts TIFF images to CBF format. The program uses the [http://libtiff.org TIFF library] and the [http://www.bernstein-plus-sons.com/software/CBF/ CBF library]. It reads 8-bit, 16-bit, and 32-bit TIFF files and converts them to 32-bit CBF files that can be processed by XDS.
The program does not write a sensible CBF-header. It is on the TODO list to provide the program with an XDS.INP style input file to transfer the available parameters to the CBF files.

The program is available as source code from(with a project file for [https://netbeans.org/ Netbeans]) and as 64-bit statically linked binary in ELF format (i.e. for Linux) [http://www.psi.ch/lbr/gruene-tim my PSI homepage].

I acknowledge support from Wolfgang Kabsch, [http://www.scripps.edu/tainer/arvai/adxv.html Andy Arvai], and the author of [https://github.com/biochem-fan/eiger2cbf/ eiger2cbf].

Tif2cbf

2017-12-15T13:03:41Z

Grunet: The program tif2cbf converts TIFF files to CBF files that can be processed by XDS.

The program tif2cbf converts TIFF images to CBF format. The program uses the [http://libtiff.org TIFF library] and the [http://www.bernstein-plus-sons.com/software/CBF/ CBF library]. It reads 8-bit, 16-bit, and 32-bit TIFF files and converts them to 32-bit CBF files that can be processed by XDS.
The program does not write a sensible CBF-header. It is on the TODO list to provide the program with an XDS.INP style input file to transfer the available parameters to the CBF files.

The program is available as source code (with a project file for [https://netbeans.org/ Netbeans]) and as 64-bit statically linked binary in ELF format (i.e. for Linux).

I acknowledge support from Wolfgang Kabsch, [http://www.scripps.edu/tainer/arvai/adxv.html Andy Arvai], and the author of [https://github.com/biochem-fan/eiger2cbf/ eiger2cbf].

Xdsconv

2013-03-04T19:08:31Z

Grunet: /* added 'freerflag' to bash-script for setting all Rfree-flags valid (otherwise problems with e.g. buster) */

[http://www.mpimf-heidelberg.mpg.de/~kabsch/xds/html_doc/xdsconv_parameters.html XDSCONV] is the conversion program of the [[XDS]] suite.

Possible output formats are SHELX, CNS, CCP4 (for F SIGF DANO SIGDANO), CCP4_F (for F+ F- SIGF+ SIGF-) and CCP4_I (for I+ I- SIGI+ SIGI-).
----
XDSCONV does outlier rejection in some modes.
== Typical use ==
A typical input file XDSCONV.INP might look like
INPUT_FILE=XDS_ASCII.HKL
INCLUDE_RESOLUTION_RANGE=50 1 ! optional
OUTPUT_FILE=temp.hkl CCP4 ! Warning: do _not_ name this file "temp.mtz" !
FRIEDEL'S_LAW=FALSE ! default is FRIEDEL'S_LAW=TRUE
This produces the file temp.hkl which is then converted to a MTZ file XDS_ASCII.mtz with (these lines are also printed out by XDSCONV):
f2mtz HKLOUT temp.mtz<F2MTZ.INP
cad HKLIN1 temp.mtz HKLOUT XDS_ASCII.mtz<<EOF
LABIN FILE 1 ALL
END
EOF
This latter step is not necessary for CNS and SHELX output formats, which are written directly by XDSCONV. For the CNS output format, one could use MERGE=FALSE to keep observations separate. For the SHELX output format, MERGE=FALSE is the default (I guess because George Sheldrick suggests that his programs, in particular [http://strucbio.biologie.uni-konstanz.de/ccp4wiki/index.php/XPREP XPREP], should be fed unmerged data. However I sometimes found that I obtain better SHELXD results with merging inside XDSCONV, using MERGE=TRUE).

N.B. It is good practice to always use FRIEDEL'S_LAW=FALSE - see [[Tips and Tricks]].

=== how to change column labels ===
To have control over the column labels, one might want to modify the simple example above as:

f2mtz HKLOUT temp.mtz<F2MTZ.INP
cad HKLIN1 temp.mtz HKLOUT junk_xdsconv.mtz<<EOF
LABIN FILE 1 E1=FP E2=SIGFP E3=DANO E4=SIGDANO E5=ISYM
LABOUT FILE 1 E1=FP E2=SIGFP E3=DANO_sulf E4=SIGDANO_sulf E5=ISYM_sulf
END
EOF

ISYM column is important if you want to run SHARP afterwards.

In the case of a MTZ file that should be used for molecular replacement and refinement, the CAD step could be used to transfer the R_free flag from a different dataset to this new dataset. Alternatively, change of labels and transfer of columns can be done in the ccp4i GUI.

== how to obtain a MTZ file with DANO SIGDANO F(+) SIGF(+) F(-) SIGF(-) ==
You have to run XDSCONV twice, and combine the output with cad. At the latter step you can also change the column labels:
#!/bin/csh -f
# produce xds_allFinfo.mtz with FP SIGFP DANO SIGDANO F(+) SIGF(+) F(-) SIGF(-)
# in the same way, the labels produced with CCP4_I could be included!
#
# first xdsconv run producing FP SIGFP DANO SIGDANO
echo "INPUT_FILE= XDS_ASCII.HKL" > XDSCONV.INP
echo "OUTPUT_FILE= temp.hkl CCP4" >> XDSCONV.INP
echo "FRIEDEL'S_LAW= FALSE" >> XDSCONV.INP
xdsconv
f2mtz HKLOUT temp1.mtz<F2MTZ.INP

# second xdsconv run producing F(+) SIGF(+) F(-) SIGF(-)
echo "INPUT_FILE= XDS_ASCII.HKL" > XDSCONV.INP
echo "OUTPUT_FILE= temp.hkl CCP4_F" >> XDSCONV.INP
echo "FRIEDEL'S_LAW= FALSE" >> XDSCONV.INP
xdsconv
f2mtz HKLOUT temp2.mtz<F2MTZ.INP

# for CAD, the 2 LABOUT cards are only required if the labels should be changed
cad HKLIN1 temp1.mtz HKLIN2 temp2.mtz HKLOUT xds_allFinfo.mtz<<EOF
LABIN FILE 1 E1=FP E2=SIGFP E3=DANO E4=SIGDANO
LABIN FILE 2 E1=F(+) E2=SIGF(+) E3=F(-) E4=SIGF(-)
LABOUT FILE 1 E1=FP_Hg E2=SIGFP_Hg E3=DANO_Hg E4=SIGDANO_Hg
LABOUT FILE 2 E1=F(+)_Hg E2=SIGF(+)_Hg E3=F(-)_Hg E4=SIGF(-)_Hg
END
EOF

The following script does the same for the input file (first parameter to the script), but also adds a SUFFIX (second parameter) to the columns to better identify the data, and optionally copies the Rfree-flag from a reference mtz-file (third parameter). If the Rfree-flag is NOT named "FreeR_flag" (the default from ccp4i), you can provide its name as fourth parameter. All steps are logged into log-files, temporary files are deleted. The input file should end with .HKL (rather than e.g. .hkl).
The script also sets the resolution to that of the observed data using sftools. Otherwise the resolution of the reference data set might be shown if that is higher. You can call this script 'xds2mtz.sh'. If it is executed without arguments, you get a short usage instruction.
<nowiki>#!/bin/bash

function usage {
echo "Usage: xds2mtz file.HKL SUFFIX [Rfree.mtz [RfreeFlag]]"
echo ""
echo " file.HKL: Output from XDS or XSCALE"
echo " SUFFIX: Columns suffix, e.g. FP_SUFFIX"
echo " Rfree.mtz: Reference mtz-file for Rfree transfer"
echo " RfreeFlag: Label for Rfree set, defaults to \"FreeR_flag\""
echo ""
}

if [ -z $1 ]; then
echo "*** Error: Missing input XDS file name"
usage
exit -1;
fi
if [ ! -f $1 ]; then
echo "*** Error: File $1 does not exist"
usage
exit -1;
fi

BASE=$(basename $1)
SUFFIX=$2
RFREE=$3
FLAG=$4

echo "Base = $BASE, Suffix = $SUFFIX"

echo "INPUT_FILE= $1" > XDSCONV.INP
echo "OUTPUT_FILE= temp1.hkl CCP4" >> XDSCONV.INP
xdsconv && f2mtz HKLOUT temp1.mtz <F2MTZ.INP | tee ${BASE%.HKL}_dano.log

echo "INPUT_FILE= $1" > XDSCONV.INP
echo "OUTPUT_FILE= temp2.hkl CCP4_F" >> XDSCONV.INP
xdsconv && f2mtz HKLOUT temp2.mtz <F2MTZ.INP |tee ${BASE%.HKL}_pm.log

if [ -z $3 ]; then
echo "Proceeding without Rfree reference file"
cad HKLIN1 temp1.mtz HKLIN2 temp2.mtz HKLOUT ${BASE%.HKL}.mtz << eof | tee ${BASE%.HKL}_cad.log
LABIN FILE 1 E1=FP E2=SIGFP E3=DANO E4=SIGDANO
LABIN FILE 2 E1=F(+) E2=SIGF(+) E3=F(-) E4=SIGF(-)
LABOUT FILE 1 E1=FP_$SUFFIX E2=SIGFP_$SUFFIX E3=DANO_$SUFFIX E4=SIGDANO_$SUFFIX
LABOUT FILE 2 E1=F(+)_$SUFFIX E2=SIGF(+)_$SUFFIX E3=F(-)_$SUFFIX E4=SIGF(-)_$SUFFIX
eof
else
echo "Copying Rfree from file $3"
if [ -z $4 ]; then
FREERFLAG="FreeR_flag" # ccp4 standard name
else
FREERFLAG=$4
fi
echo "Extracting flagged indices from ${FREERFLAG}"
cad HKLIN1 temp1.mtz \
HKLIN2 temp2.mtz \
HKLIN3 $3 \
HKLOUT ${BASE%.HKL}.mtz << eof | tee ${BASE%.HKL}_cad.log
LABIN FILE 1 E1=FP E2=SIGFP E3=DANO E4=SIGDANO
LABIN FILE 2 E1=F(+) E2=SIGF(+) E3=F(-) E4=SIGF(-)
LABIN FILE 3 E1=${FREERFLAG}
LABOUT FILE 1 E1=FP_$SUFFIX E2=SIGFP_$SUFFIX E3=DANO_$SUFFIX E4=SIGDANO_$SUFFIX
LABOUT FILE 2 E1=F(+)_$SUFFIX E2=SIGF(+)_$SUFFIX E3=F(-)_$SUFFIX E4=SIGF(-)_$SUFFIX
LABOUT FILE 3 E1=${FREERFLAG}
eof

rm temp1.mtz

# correct for FreeRflag (if new file has more reflections than reference file)
freerflag hklin ${BASE%.HKL}.mtz hklout temp1.mtz << eof | tee ${BASE%.HKL}_freerflag.log
COMPLETE FREE=${FREERFLAG}
end
eof

# correct for real data in case Rfree data set contains too many hkls
# thanks to Andrey Lebedev
sftools << eof | tee ${BASE%.HKL}_sftools.log
READ ${BASE%.HKL}.mtz
SELECT ONLY COLUMN FP_$SUFFIX PRESENT
WRITE temp1.mtz
END
eof

mv temp1.mtz ${BASE%.HKL}.mtz

fi

rm -f XDSCONV.INP temp1.hkl temp1.mtz temp2.hkl temp2.mtz F2MTZ.INP XDSCONV.LP</nowiki>

== Hint for long-time XDSCONV users ==

The latest versions of the program do not require
SPACE_GROUP_NUMBER=
UNIT_CELL_PARAMETERS=
because these are picked up from the header of the input reflection file. However, if you want to ''change'' the parameters of either keyword then you have to specify '''both'''! I.e. if you want to change the spacegroup then you also have to specify the unit cell parameters.

Xdsconv

2013-03-01T17:57:57Z

Grunet: /* second bash script for converting XDS format to (anomalous) mtz including Rfree */

[http://www.mpimf-heidelberg.mpg.de/~kabsch/xds/html_doc/xdsconv_parameters.html XDSCONV] is the conversion program of the [[XDS]] suite.

Possible output formats are SHELX, CNS, CCP4 (for F SIGF DANO SIGDANO), CCP4_F (for F+ F- SIGF+ SIGF-) and CCP4_I (for I+ I- SIGI+ SIGI-).
----
XDSCONV does outlier rejection in some modes.
== Typical use ==
A typical input file XDSCONV.INP might look like
INPUT_FILE=XDS_ASCII.HKL
INCLUDE_RESOLUTION_RANGE=50 1 ! optional
OUTPUT_FILE=temp.hkl CCP4 ! Warning: do _not_ name this file "temp.mtz" !
FRIEDEL'S_LAW=FALSE ! default is FRIEDEL'S_LAW=TRUE
This produces the file temp.hkl which is then converted to a MTZ file XDS_ASCII.mtz with (these lines are also printed out by XDSCONV):
f2mtz HKLOUT temp.mtz<F2MTZ.INP
cad HKLIN1 temp.mtz HKLOUT XDS_ASCII.mtz<<EOF
LABIN FILE 1 ALL
END
EOF
This latter step is not necessary for CNS and SHELX output formats, which are written directly by XDSCONV. For the CNS output format, one could use MERGE=FALSE to keep observations separate. For the SHELX output format, MERGE=FALSE is the default (I guess because George Sheldrick suggests that his programs, in particular [http://strucbio.biologie.uni-konstanz.de/ccp4wiki/index.php/XPREP XPREP], should be fed unmerged data. However I sometimes found that I obtain better SHELXD results with merging inside XDSCONV, using MERGE=TRUE).

N.B. It is good practice to always use FRIEDEL'S_LAW=FALSE - see [[Tips and Tricks]].

=== how to change column labels ===
To have control over the column labels, one might want to modify the simple example above as:

f2mtz HKLOUT temp.mtz<F2MTZ.INP
cad HKLIN1 temp.mtz HKLOUT junk_xdsconv.mtz<<EOF
LABIN FILE 1 E1=FP E2=SIGFP E3=DANO E4=SIGDANO E5=ISYM
LABOUT FILE 1 E1=FP E2=SIGFP E3=DANO_sulf E4=SIGDANO_sulf E5=ISYM_sulf
END
EOF

ISYM column is important if you want to run SHARP afterwards.

In the case of a MTZ file that should be used for molecular replacement and refinement, the CAD step could be used to transfer the R_free flag from a different dataset to this new dataset. Alternatively, change of labels and transfer of columns can be done in the ccp4i GUI.

== how to obtain a MTZ file with DANO SIGDANO F(+) SIGF(+) F(-) SIGF(-) ==
You have to run XDSCONV twice, and combine the output with cad. At the latter step you can also change the column labels:
#!/bin/csh -f
# produce xds_allFinfo.mtz with FP SIGFP DANO SIGDANO F(+) SIGF(+) F(-) SIGF(-)
# in the same way, the labels produced with CCP4_I could be included!
#
# first xdsconv run producing FP SIGFP DANO SIGDANO
echo "INPUT_FILE= XDS_ASCII.HKL" > XDSCONV.INP
echo "OUTPUT_FILE= temp.hkl CCP4" >> XDSCONV.INP
echo "FRIEDEL'S_LAW= FALSE" >> XDSCONV.INP
xdsconv
f2mtz HKLOUT temp1.mtz<F2MTZ.INP

# second xdsconv run producing F(+) SIGF(+) F(-) SIGF(-)
echo "INPUT_FILE= XDS_ASCII.HKL" > XDSCONV.INP
echo "OUTPUT_FILE= temp.hkl CCP4_F" >> XDSCONV.INP
echo "FRIEDEL'S_LAW= FALSE" >> XDSCONV.INP
xdsconv
f2mtz HKLOUT temp2.mtz<F2MTZ.INP

# for CAD, the 2 LABOUT cards are only required if the labels should be changed
cad HKLIN1 temp1.mtz HKLIN2 temp2.mtz HKLOUT xds_allFinfo.mtz<<EOF
LABIN FILE 1 E1=FP E2=SIGFP E3=DANO E4=SIGDANO
LABIN FILE 2 E1=F(+) E2=SIGF(+) E3=F(-) E4=SIGF(-)
LABOUT FILE 1 E1=FP_Hg E2=SIGFP_Hg E3=DANO_Hg E4=SIGDANO_Hg
LABOUT FILE 2 E1=F(+)_Hg E2=SIGF(+)_Hg E3=F(-)_Hg E4=SIGF(-)_Hg
END
EOF

The following script does the same for the input file (first parameter to the script), but also adds a SUFFIX (second parameter) to the columns to better identify the data, and optionally copies the Rfree-flag from a reference mtz-file (third parameter). If the Rfree-flag is NOT named "FreeR_flag" (the default from ccp4i), you can provide its name as fourth parameter. All steps are logged into log-files, temporary files are deleted. The input file should end with .HKL (rather than e.g. .hkl).
The script also sets the resolution to that of the observed data using sftools. Otherwise the resolution of the reference data set might be shown if that is higher. You can call this script 'xds2mtz.sh'. If it is executed without arguments, you get a short usage instruction.
<nowiki>#!/bin/bash

function usage {
echo "Usage: xds2mtz file.HKL SUFFIX [Rfree.mtz [RfreeFlag]]"
echo ""
echo " file.HKL: Output from XDS or XSCALE"
echo " SUFFIX: Columns suffix, e.g. FP_SUFFIX"
echo " Rfree.mtz: Reference mtz-file for Rfree transfer"
echo " RfreeFlag: Label for Rfree set, defaults to \"FreeR_flag\""
echo ""
}

if [ -z $1 ]; then
echo "*** Error: Missing input XDS file name"
usage
exit -1;
fi
if [ ! -f $1 ]; then
echo "*** Error: File $1 does not exist"
usage
exit -1;
fi

BASE=$(basename $1)
SUFFIX=$2
RFREE=$3
FLAG=$4

echo "Base = $BASE, Suffix = $SUFFIX"

echo "INPUT_FILE= $1" > XDSCONV.INP
echo "OUTPUT_FILE= temp1.hkl CCP4" >> XDSCONV.INP
xdsconv && f2mtz HKLOUT temp1.mtz <F2MTZ.INP | tee ${BASE%.HKL}_dano.log

echo "INPUT_FILE= $1" > XDSCONV.INP
echo "OUTPUT_FILE= temp2.hkl CCP4_F" >> XDSCONV.INP
xdsconv && f2mtz HKLOUT temp2.mtz <F2MTZ.INP |tee ${BASE%.HKL}_pm.log

if [ -z $3 ]; then
echo "Proceeding without Rfree reference file"
cad HKLIN1 temp1.mtz HKLIN2 temp2.mtz HKLOUT ${BASE%.HKL}.mtz << eof | tee ${BASE%.HKL}_cad.log
LABIN FILE 1 E1=FP E2=SIGFP E3=DANO E4=SIGDANO
LABIN FILE 2 E1=F(+) E2=SIGF(+) E3=F(-) E4=SIGF(-)
LABOUT FILE 1 E1=FP_$SUFFIX E2=SIGFP_$SUFFIX E3=DANO_$SUFFIX E4=SIGDANO_$SUFFIX
LABOUT FILE 2 E1=F(+)_$SUFFIX E2=SIGF(+)_$SUFFIX E3=F(-)_$SUFFIX E4=SIGF(-)_$SUFFIX
eof
else
echo "Copying Rfree from file $3"
if [ -z $4 ]; then
FREERFLAG="FreeR_flag" # ccp4 standard name
else
FREERFLAG=$4
fi
echo "Extracting flagged indices from ${FREERFLAG}"
cad HKLIN1 temp1.mtz \
HKLIN2 temp2.mtz \
HKLIN3 $3 \
HKLOUT ${BASE%.HKL}.mtz << eof | tee ${BASE%.HKL}_cad.log
LABIN FILE 1 E1=FP E2=SIGFP E3=DANO E4=SIGDANO
LABIN FILE 2 E1=F(+) E2=SIGF(+) E3=F(-) E4=SIGF(-)
LABIN FILE 3 E1=${FREERFLAG}
LABOUT FILE 1 E1=FP_$SUFFIX E2=SIGFP_$SUFFIX E3=DANO_$SUFFIX E4=SIGDANO_$SUFFIX
LABOUT FILE 2 E1=F(+)_$SUFFIX E2=SIGF(+)_$SUFFIX E3=F(-)_$SUFFIX E4=SIGF(-)_$SUFFIX
LABOUT FILE 3 E1=${FREERFLAG}
eof

rm temp1.mtz

# correct for real data in case Rfree data set contains too many hkls
# thanks to Andrey Lebedev
sftools << eof | tee ${BASE%.HKL}_sftools.log
READ ${BASE%.HKL}.mtz
SELECT ONLY COLUMN FP_$SUFFIX PRESENT
WRITE temp1.mtz
END
eof

mv temp1.mtz ${BASE%.HKL}.mtz

fi

rm -f XDSCONV.INP temp1.hkl temp1.mtz temp2.hkl temp2.mtz F2MTZ.INP XDSCONV.LP</nowiki>

== Hint for long-time XDSCONV users ==

The latest versions of the program do not require
SPACE_GROUP_NUMBER=
UNIT_CELL_PARAMETERS=
because these are picked up from the header of the input reflection file. However, if you want to ''change'' the parameters of either keyword then you have to specify '''both'''! I.e. if you want to change the spacegroup then you also have to specify the unit cell parameters.

User:Grunet

2010-07-09T12:48:30Z

Grunet: Created page with 'Tim Gruene, homepage at [http://shelx.uni-ac.gwdg.de/~tg shelx]'

Tim Gruene, homepage at [http://shelx.uni-ac.gwdg.de/~tg shelx]

Generate XDS.INP

2010-07-09T12:41:58Z

Grunet: replaced script with rev. 0.10: setting link to image directory if path is too long.

This script should be in your $PATH as "generate_XDS.INP" . As the name suggests, it generates XDS.INP based on a list of frame names supplied on the commandline.
Currently works for MarCCD, ADSC and Pilatus 6M detectors.

<pre>
#!/bin/bash

# purpose: generate XDS.INP
# revision 0.03 . Kay Diederichs 2/2010
# revision 0.04 . Kay Diederichs 4/2010 - include alternative ORGX, ORGY calculations for ADSC
# revision 0.05 . Kay Diederichs 5/2010 - grep for "Corrected" in addition to "marccd"; needed for BESSY
# revision 0.06 . KD 6/2010 - add UNTRUSTED_RECTANGLE and UNTRUSTED_ELLIPSE; use `whereis catmar` and so on
# revision 0.07 . KD 6/2010 - decide about ORGX/Y info in MAR header being pixels or mm; other fixes
# revision 0.08 . KD 6/2010 - fixes for Pilatus 6M
# revision 0.09 . KD 6/2010 - get rid of requirement for mccd_xdsparams.pl and/or catmar; rather use "od"
# tested with some datasets from ALS, SSRL, SLS and ESRF; only MARCCD, ADSC/SMV, PILATUS 6M detectors;
# for other detectors, values marked with XXX must be manually filled in.
# revision 0.10 . Tim Gruene 7/2010 - set link 'images' to image directory if path exceeds 72 characters
#
# usage: e.g. generate_XDS.INP "frms/mydata_1_???.img"
# make sure to have the two quotation marks !
# the ? are wildcards for the frame numbers.
#
# two external programs that can read MAR headers:
# - for MARCCD detectors the "catmar" binary (which can be downloaded from http://www.marresearch.com/download.html)
# - the [[mccd_xdsparams.pl]] script (see http://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/Mccd_xdsparams.pl )
#
# limitations:
# - frame numbers are assumed to start with 1 and run consecutively
#
# known problems:
# - for ADSC detectors, there are at least three ways to obtain ORGX and ORGY values from the header (see below);
# - the same might be a problem for MAR headers, too (not sure about this)
#
# notes for debugging of the script:
# - add the -v option to the first line, to see where an error occurs
# - comment out the removal of tmp1 and tmp2 in the last line
#
# ====== Start of script ======
if [ "$1" == "help" ] || [ "$1" == "-help" ] || [ "$1" == "-h" ]; then
echo usage: generate_XDS.INP \"frms/mydata_1_???.img\" $_with_ the quotation marks!$
exit
fi
#
# defaults:
#
DETECTOR="XXX MINIMUM_VALID_PIXEL_VALUE=XXX OVERLOAD=XXX"
ORGX=XXX
ORGY=XXX
DETECTOR_DISTANCE=XXX
OSCILLATION_RANGE=XXX
X_RAY_WAVELENGTH=XXX

# see how we are called:
NAME_TEMPLATE_OF_DATA_FRAMES="$1"

# check that the image template name does not exceed 72 characters
# and set a link if necessary
if [ ${#1} -gt 72 ]; then
TMP_PATH=$(/bin/ls -C1 $1 | head -1)
TMP_FILENAME=$(basename ${TMP_PATH})
if [ ${#TMP_FILENAME} -gt 65 ]; then
echo "---> Unable to proceed: image filename "
echo "---> \"${TMP_FILENAME}\""
echo "---> exceeds 72 characters."
echo "---> Please rename files or set links."
exit
else
echo "---> Warning: Template name exceeds 72 characters. Setting link \"image\""
echo " to image directory"
TMP_DIRNAME=$(dirname ${TMP_PATH})
if [ -e "images" ]; then
echo "*** Error: The file or directory \"images\" already exists. Please remove"
echo " and re-run."
exit
else
ln -s ${TMP_DIRNAME} images
fi
NAME_TEMPLATE_OF_DATA_FRAMES="images/${1##/*/}"
echo " Using template filename \"${NAME_TEMPLATE_OF_DATA_FRAMES}\""
fi
fi

# list frames matching the wildcards in NAME_TEMPLATE_OF_DATA_FRAMES
# don't accept the "direct beam" shot at SLS/Pilatus
/bin/ls -C1 $1 | grep -v "_00000.cbf" > tmp1 || exit 1

# we can continue - the frames are found
echo Full documentation, including complete detector templates and XDS binaries, can be found at
echo http://www.mpimf-heidelberg.mpg.de/~kabsch/xds . More documentation: see XDSwiki

# set upper limit of DATA_RANGE to number of frames (see "limitations" above)
DATA_RANGE=`wc -l tmp1 | awk '{print $1}'`

# set upper limit of SPOT_RANGE to half of DATA_RANGE, but not less than 1
SPOT_RANGE=`echo "scale=0; $DATA_RANGE/2" | bc -l`
SPOT_RANGE=`echo "if ($SPOT_RANGE<1) 1;if ($SPOT_RANGE>1) $SPOT_RANGE" | bc -l`

echo DATA_RANGE=1 $DATA_RANGE

# find out detector type
DET=XXX
strings `head -1 tmp1` | egrep -q 'marccd|Corrected' && DET=mccd
strings `head -1 tmp1` | grep -q PILATUS && DET=pilatus
strings `head -1 tmp1` | grep -q BEAM_CENTER_X && DET=adsc
# identify other detector types in the same way (MAR IP would be straightforward)

# parse ASCII header of first frame

if [ "$DET" == "XXX" ]; then
echo "this is not a MAR, ADSC/SMV or PILATUS detector - fill in XXX values manually!"
DETECTOR="XXX MINIMUM_VALID_PIXEL_VALUE=XXX OVERLOAD=XXX"

# find parameters of first frame
elif [ "$DET" == "mccd" ]; then

DETECTOR="CCDCHESS MINIMUM_VALID_PIXEL_VALUE= 1 OVERLOAD= 65500"

# use first frame of dataset to obtain parameters
MARFRAME=`head -1 tmp1`

# offsets are documented; values can be find in mccd_xdsparams.pl script
let SKIP=1024+80
NX=$(od -t dI --skip-bytes=$SKIP --read-bytes=4 $MARFRAME | head -1 | awk '{print $2}')
let SKIP=$SKIP+4
NY=$(od -t dI --skip-bytes=$SKIP --read-bytes=4 $MARFRAME | head -1 | awk '{print $2}')

let SKIP=1024+256+128+256
DETECTOR_DISTANCE=$(od -t dI --skip-bytes=$SKIP --read-bytes=4 $MARFRAME | head -1 | awk '{print $2}')
DETECTOR_DISTANCE=`echo "scale=3; $DETECTOR_DISTANCE/1000" | bc -l`

let SKIP=$SKIP+4
ORGX=$(od -t dI --skip-bytes=$SKIP --read-bytes=4 $MARFRAME | head -1 | awk '{print $2}')
ORGX=`echo "scale=2; $ORGX/1000" | bc -l `
let SKIP=$SKIP+4
ORGY=$(od -t dI --skip-bytes=$SKIP --read-bytes=4 $MARFRAME | head -1 | awk '{print $2}')
ORGY=`echo "scale=2; $ORGY/1000" | bc -l `

let SKIP=1024+256+128+256+44
PHISTART=$(od -t dI --skip-bytes=$SKIP --read-bytes=4 $MARFRAME | head -1 | awk '{print $2}')
let SKIP=1024+256+128+256+76
PHIEND=$(od -t dI --skip-bytes=$SKIP --read-bytes=4 $MARFRAME | head -1 | awk '{print $2}')
OSCILLATION_RANGE=`echo "scale=3; ($PHIEND-$PHISTART)/1000" | bc -l`

let SKIP=1024+256+128+256+128+4
QX=$(od -t dI --skip-bytes=$SKIP --read-bytes=4 $MARFRAME | head -1 | awk '{print $2}')
QX=`echo "scale=10; $QX/1000000" |bc -l `
let SKIP=$SKIP+4
QY=$(od -t dI --skip-bytes=$SKIP --read-bytes=4 $MARFRAME | head -1 | awk '{print $2}')
QY=`echo "scale=10; $QY/1000000" |bc -l `

let SKIP=1024+256+128+256+128+128+12
X_RAY_WAVELENGTH=$(od -t dI --skip-bytes=$SKIP --read-bytes=4 $MARFRAME | head -1 | awk '{print $2}')
X_RAY_WAVELENGTH=`echo "scale=5; $X_RAY_WAVELENGTH/100000" | bc -l`

# at most BLs, ORGX and ORGY are in pixels, but sometimes in mm ... guess:
NXBYFOUR=`echo "scale=0; $NX/4" | bc -l `
ORGXINT=`echo "scale=0; $ORGX/1" | bc -l `
if [ $ORGXINT -lt $NXBYFOUR ]; then
ORGX=`echo "scale=1; $ORGX/$QX" | bc -l`
ORGY=`echo "scale=1; $ORGY/$QY" | bc -l`
echo MARCCD detector: header ORGX, ORGY seem to be in mm ... converting to pixels
else
echo MARCCD detector: header ORGX, ORGY seem to be in pixel units
fi

elif [ "$DET" == "adsc" ]; then

DETECTOR="ADSC MINIMUM_VALID_PIXEL_VALUE= 1 OVERLOAD= 65000"
echo this is an ADSC detector. Obtaining ORGX, ORGY from the header depends on beamline setup.
strings `head -1 tmp1` | sed s/\;// > tmp2

# find X_RAY_WAVELENGTH:
X_RAY_WAVELENGTH=`grep WAVELENGTH tmp2 | sed s/WAVELENGTH=//`

# find NX, QX, ORGX and ORGY:
NX=`grep SIZE1 tmp2 | tail -1 | sed s/SIZE1=//`
QX=`grep PIXEL_SIZE tmp2 | sed s/PIXEL_SIZE=//`
BEAM_CENTER_X=`grep BEAM_CENTER_X tmp2 | sed s/BEAM_CENTER_X=//`
BEAM_CENTER_Y=`grep BEAM_CENTER_Y tmp2 | sed s/BEAM_CENTER_Y=//`
# fix 2010-04-26 - tell user about possible ORGX, ORGY alternatives -
# at ESRF and ... (pls fill in!) the following should be used:
ORGX=`echo "scale=1; $BEAM_CENTER_Y/$QX" | bc -l `
ORGY=`echo "scale=1; $BEAM_CENTER_X/$QX" | bc -l `
echo ATTENTION: at ESRF BLs use: ORGX=$ORGX ORGY=$ORGY
# this 2nd alternative convention should be used at the following beamlines (pls complete the list): ALS 5.0.3, ...
ORGX=`echo "scale=1; $NX-$BEAM_CENTER_X/$QX" | bc -l `
ORGY=`echo "scale=1; $BEAM_CENTER_Y/$QX" | bc -l `
echo ATTENTION: at e.g. ALS 5.0.3 use: ORGX=$ORGX ORGY=$ORGY
# this 3rd alternative convention should be used at the following beamlines (pls complete the list): ALS 8.2.2, ...
ORGX=`echo "scale=1; $BEAM_CENTER_X/$QX" | bc -l `
ORGY=`echo "scale=1; $NX-$BEAM_CENTER_Y/$QX" | bc -l `
echo ATTENTION: at e.g. ALS 8.2.2 use: ORGX=$ORGX ORGY=$ORGY - this is now written to XDS.INP
# the latter alternative is written into the generated XDS.INP ! You have to correct this manually in XDS.INP, or adjust this script.
# find DETECTOR_DISTANCE and OSCILLATION_RANGE:
DETECTOR_DISTANCE=`grep DISTANCE tmp2 | sed s/DISTANCE=//`
OSCILLATION_RANGE=`grep OSC_RANGE tmp2 | sed s/OSC_RANGE=//`

elif [ "$DET" == "pilatus" ]; then
DETECTOR="PILATUS MINIMUM_VALID_PIXEL_VALUE=0 OVERLOAD= 1048576 QX=0.172 QY=0.172 !PILATUS 6M"
echo this is a Pilatus detector
head -50 `head -1 tmp1` | sed s/#//> tmp2

# find X_RAY_WAVELENGTH:
X_RAY_WAVELENGTH=`grep Wavelength tmp2 | sed -e s/Wavelength// -e s/A// | awk '{print $1}'`

# find ORGX and ORGY:
ORGX=`grep Beam_xy tmp2 | sed -e s/$// -e s/$// -e s/\,// | awk '{print $2}'`
ORGY=`grep Beam_xy tmp2 | sed -e s/$// -e s/$// -e s/\,// | awk '{print $3}'`

# find DETECTOR_DISTANCE and OSCILLATION_RANGE:
DETECTOR_DISTANCE=`awk '/distance/{print $2}' tmp2`
DETECTOR_DISTANCE=`echo "$DETECTOR_DISTANCE*1000" | bc -l`

OSCILLATION_RANGE=`awk '/Angle/{print $2}' tmp2`

else
echo should never come here
exit 1
fi

echo ORGX= $ORGX ORGY= $ORGY - check these values with adxv !
echo DETECTOR_DISTANCE= $DETECTOR_DISTANCE
echo OSCILLATION_RANGE= $OSCILLATION_RANGE
echo X-RAY_WAVELENGTH= $X_RAY_WAVELENGTH

# now we know everything that is required to generate XDS.INP

cat > XDS.INP << eof
JOB= XYCORR INIT COLSPOT IDXREF DEFPIX INTEGRATE CORRECT
ORGX= $ORGX ORGY= $ORGY ! check these values with adxv !
DETECTOR_DISTANCE= $DETECTOR_DISTANCE
OSCILLATION_RANGE= $OSCILLATION_RANGE
X-RAY_WAVELENGTH= $X_RAY_WAVELENGTH
NAME_TEMPLATE_OF_DATA_FRAMES=$NAME_TEMPLATE_OF_DATA_FRAMES
! REFERENCE_DATA_SET=xxx/XDS_ASCII.HKL ! e.g. to ensure consistent indexing
DATA_RANGE=1 $DATA_RANGE
SPOT_RANGE=1 $SPOT_RANGE
! BACKGROUND_RANGE=1 10 ! rather use defaults (first 5 degree of rotation)

SPACE_GROUP_NUMBER=0 ! 0 if unknown
UNIT_CELL_CONSTANTS= 70 80 90 90 90 90 ! put correct values if known
INCLUDE_RESOLUTION_RANGE=50 0 ! after CORRECT, insert high resol limit; re-run CORRECT

FRIEDEL'S_LAW=FALSE ! This acts only on the CORRECT step
! If the anom signal turns out to be, or is known to be, very low or absent,
! use FRIEDEL'S_LAW=TRUE instead (or comment out the line); re-run CORRECT

! remove the "!" in the following line:
! STRICT_ABSORPTION_CORRECTION=TRUE
! if the anomalous signal is strong: in that case, in CORRECT.LP the three
! "CHI^2-VALUE OF FIT OF CORRECTION FACTORS" values are significantly> 1, e.g. 1.5
!
! exclude (mask) untrusted areas of detector, e.g. beamstop shadow :
! UNTRUSTED_RECTANGLE= 1800 1950 2100 2150 ! x-min x-max y-min y-max ! repeat
! UNTRUSTED_ELLIPSE= 2034 2070 1850 2240 ! x-min x-max y-min y-max ! if needed
!
! parameters with changes wrt default values:
TRUSTED_REGION=0.00 1.2 ! partially use corners of detectors; 1.41421=full use
VALUE_RANGE_FOR_TRUSTED_DETECTOR_PIXELS=7000. 30000. ! often 8000 is ok
MINIMUM_ZETA=0.05 ! integrate close to the Lorentz zone; 0.15 is default
STRONG_PIXEL=6 ! COLSPOT: only use strong reflections (default is 3)
REFINE(INTEGRATE)=CELL BEAM ORIENTATION ! AXIS DISTANCE

! parameters specifically for this detector and beamline:
DETECTOR= $DETECTOR
!NX= 3072 NY= 3072 QX= 0.10260 QY= 0.10260 ! XDS finds this out by itself (not for Pilatus)
DIRECTION_OF_DETECTOR_X-AXIS=1 0 0
DIRECTION_OF_DETECTOR_Y-AXIS=0 1 0
INCIDENT_BEAM_DIRECTION=0 0 1
ROTATION_AXIS=1 0 0 ! at e.g. SERCAT ID-22 this needs to be -1 0 0
FRACTION_OF_POLARIZATION=0.98 ! better value is provided by beamline staff!
POLARIZATION_PLANE_NORMAL=0 1 0
eof
echo XDS.INP is ready for use. The file has only the most important keywords.
echo After running xds, inspect at least BKGPIX.cbf and FRAME.cbf with XDS-Viewer!
rm -f tmp1 tmp2

</pre>

Tutorial(First Steps)

2009-07-02T23:22:06Z

Grunet:

= A beginner's walk-through to XDS =

''This page is under development. If you think it might be useful to you and worth completing faster, why not simply bother the author by sending him and email?''

This tutorial is directed at people who are beginners to XDS and do not want to start reading about all the technical details but rather want their images processed. Other processing programs are often more attractive to new users because using a GUI make starting easier than a [[XDS.INP|script file]]. If you notice a mistake or gap in this tutorial, please contact the authors.

== Setting up ==
=== Image Location ===

It is not recommended to process the images from within the same directory as the images are. At least create a subdirectory. In many cases the data images are on an external disk or remotely mounted directory with an awfully long path name.
If you images are "/some/where/with/a/very/long/path", [[XDS.INP]] may not find the images and one must create a link to the actual directory.
<pre> #> ln -s /some/where/with/a/very/long/path images </pre>.

=== Getting [[XDS.INP]] ===
You must know the type of detector your data were collected on. Then visit the [http://www.mpimf-heidelberg.mpg.de/~kabsch/xds/html_doc/xds_prepare.html|XDS Homepage], download the appropriate template and rename it to [[XDS.INP]].

== Editing [[XDS.INP]] ==
This tutorial uses the template for XDS-MARCCD.INP.

Open [[XDS.INP]] with your favourite text editor (e.g. gedit, kate, kwrite, vim, emacs...) and make the following adjustments (from top to bottom):

# '''Pixel Size''' Find out the number of pixels of your detector. A very good program to find aus is [http://www.scripps.edu/~arvai/adxv.html adxv]: It allows to view the image header where these numbers often can be found. [[Image:Adxv_menu.jpeg|100px|adxv image viewer, main menu]] [[Image:Adxv_pixelsize.jpeg|100px|adxv showing image header info. Pixel size is highlighted]]
# '''Adjusting JOB Card''' Make a copy of the '''JOB''' line in [[XDS.INP]] for reference. Comment out one line by putting a exclamation mark "!" at the beginning of the line and prepare the other one to run the steps from XYCORR to CORRECT.[[Image:XDS_INP_adjustingJOB.jpeg|200px|Adjusting the JOB card in XDS.INP]]
# '''ORGX and ORGY''' Unless you can guess the direct beam position on the images, ''e.g.'' from ice rings, set both these values to half the detector size. In the case of this example, where the MAR-CCD is 3072x3072 pixels, this means
ORGX=1536.00 ORGY=1536.00 !Detector origin (pixels).
#'''Experiment specific settings''' The following numbers are also usually found in the image header:
DETECTOR_DISTANCE= 150.0 !(mm)
OSCILLATION_RANGE=1.0 !degrees (>0)
X-RAY_WAVELENGTH=1.07176 !Angstroem
NAME_TEMPLATE_OF_DATA_FRAMES=./images/tutorial_????.img ! TIFF
DATA_RANGE=1 180 !Numbers of first and last data image collected
#'''Background range''' The default template suggests to use the first 5 images for detecting the average background per image. It does not harm to increase this number to 20 images. Especially with frame width of less than 1 degree you should do this.
BACKGROUND_RANGE=1 20 !Numbers of first and last data image for background

This is it to start. type <pre>#> xds_par </pre> to run xds and wait until it's finished.

== Indexing Fails ==
If the detector origin is too far from the image centre, IDXREF is going to stop with the error message
!!! ERROR !!! SOLUTION IS INACCURATE
Do the following:
# In IDXREF.LP, find the table '''INDEXING OF OBSERVED SPOTS IN SPACE GROUP'''
***** INDEXING OF OBSERVED SPOTS IN SPACE GROUP # 1 *****
16196 OUT OF 76452 SPOTS INDEXED.
0 REJECTED REFLECTIONS (REASON: OVERLAP)
60256 REJECTED REFLECTIONS (REASON: TOO FAR FROM IDEAL POSITION)
EXPECTED ERROR IN SPINDLE POSITION 2.398 DEGREES
EXPECTED ERROR IN DETECTOR POSITION 1.72 PIXELS
Unless really hardly any reflections could be indexed (second line in the table), scroll a few lines up and copy the refined values for the beam origin
DETECTOR ORIGIN (PIXELS) AT 1514.16 1537.27
as improved ORGX and ORGY to [[XDS.INP]] and try indexing again.
XDS is quite robust and in many cases this approach works
# Check the refined detector distance
DETECTOR ORIGIN (PIXELS) AT 1514.16 1537.27
CRYSTAL TO DETECTOR DISTANCE (mm) 422.92
If it deviates by more than 1-2mm from the input distance, check all you parameters again. Is the wavelength really correct, as well as the distance? If you are sure about them, try indexing with only a few images
SPOT_RANGE = 1 20

Tutorial(First Steps)

2009-07-02T23:13:22Z

Grunet:

= A beginner's walk-through to XDS =

''This page is under development. If you think it might be useful to you and worth completing faster, why not simply bother the author by sending him and email?''

This tutorial is directed at people who are beginners to XDS and do not want to start reading about all the technical details but rather want their images processed. Other processing programs are often more attractive to new users because using a GUI make starting easier than a [[XDS.INP|script file]]. If you notice a mistake or gap in this tutorial, please contact the authors.

== Setting up ==
=== Image Location ===

It is not recommended to process the images from within the same directory as the images are. At least create a subdirectory. In many cases the data images are on an external disk or remotely mounted directory with an awfully long path name.
If you images are "/some/where/with/a/very/long/path", [[XDS.INP]] may not find the images and one must create a link to the actual directory.
<pre> #> ln -s /some/where/with/a/very/long/path images </pre>.

=== Getting [[XDS.INP]] ===
You must know the type of detector your data were collected on. Then visit the [http://www.mpimf-heidelberg.mpg.de/~kabsch/xds/html_doc/xds_prepare.html|XDS Homepage], download the appropriate template and rename it to [[XDS.INP]].

== Editing [[XDS.INP]] ==
This tutorial uses the template for XDS-MARCCD.INP.

Open [[XDS.INP]] with your favourite text editor (e.g. gedit, kate, kwrite, vim, emacs...) and make the following adjustments (from top to bottom):

# '''Pixel Size''' Find out the number of pixels of your detector. A very good program to find aus is [http://www.scripps.edu/~arvai/adxv.html adxv]: It allows to view the image header where these numbers often can be found. [[Image:Adxv_menu.jpeg|100px|adxv image viewer, main menu]] [[Image:Adxv_pixelsize.jpeg|100px|adxv showing image header info. Pixel size is highlighted]]
# '''Adjusting JOB Card''' Make a copy of the '''JOB''' line in [[XDS.INP]] for reference. Comment out one line by putting a exclamation mark "!" at the beginning of the line and prepare the other one to run the steps from XYCORR to CORRECT.[[Image:XDS_INP_adjustingJOB.jpeg|200px|Adjusting the JOB card in XDS.INP]]
# '''ORGX and ORGY''' Unless you can guess the direct beam position on the images, ''e.g.'' from ice rings, set both these values to half the detector size. In the case of this example, where the MAR-CCD is 3072x3072 pixels, this means
ORGX=1536.00 ORGY=1536.00 !Detector origin (pixels).
#'''Experiment specific settings''' The following numbers are also usually found in the image header:
DETECTOR_DISTANCE= 150.0 !(mm)
OSCILLATION_RANGE=1.0 !degrees (>0)
X-RAY_WAVELENGTH=1.07176 !Angstroem
NAME_TEMPLATE_OF_DATA_FRAMES=./images/tutorial_????.img ! TIFF
DATA_RANGE=1 180 !Numbers of first and last data image collected
#'''Background range''' The default template suggests to use the first 5 images for detecting the average background per image. It does not harm to increase this number to 20 images. Especially with frame width of less than 1 degree you should do this.
BACKGROUND_RANGE=1 20 !Numbers of first and last data image for background

This is it to start. type <pre>#> xds_par </pre> to run xds and wait until it's finished.

== Indexing Fails ==
If the detector origin is too far from the image centre, IDXREF is going to stop with the error message
!!! ERROR !!! SOLUTION IS INACCURATE
Do the following:
<source lang=fortran>
# In IDXREF.LP, find the table '''INDEXING OF OBSERVED SPOTS IN SPACE GROUP'''
***** INDEXING OF OBSERVED SPOTS IN SPACE GROUP # 1 *****
16196 OUT OF 76452 SPOTS INDEXED.
0 REJECTED REFLECTIONS (REASON: OVERLAP)
60256 REJECTED REFLECTIONS (REASON: TOO FAR FROM IDEAL POSITION)
EXPECTED ERROR IN SPINDLE POSITION 2.398 DEGREES
EXPECTED ERROR IN DETECTOR POSITION 1.72 PIXELS
</source>
Unless really hardly any reflections could be indexed (second line in the table), scroll a few lines up and copy the refined values for the beam origin
DETECTOR ORIGIN (PIXELS) AT 1514.16 1537.27
as improved ORGX and ORGY to [[XDS.INP]] and try indexing again.
XDS is quite robust and in many cases this approach works
# Check the refined detector distance
DETECTOR ORIGIN (PIXELS) AT 1514.16 1537.27
CRYSTAL TO DETECTOR DISTANCE (mm) 422.92
If it deviates by more than 1-2mm from the input distance, check all you parameters again. Is the wavelength really correct, as well as the distance? If you are sure about them, try indexing with only a few images
SPOT_RANGE = 1 20

Tutorial(First Steps)

2009-06-28T16:04:33Z

Grunet: New page: = A beginner's walk-through to XDS = ''This page is under development. If you think it might be useful to you and worth completing faster, why not simply bother the author by sending him ...

File:XDS INP adjustingJOB.jpeg

2009-06-28T15:25:49Z

Grunet: XDS.INP file being edited with vim, looking at the 'JOB' Card.

XDS.INP file being edited with vim, looking at the 'JOB' Card.

File:Adxv pixelsize.jpeg

2009-06-28T15:19:27Z

Grunet: screenshot from adxv image header with pixel size highlighted.

screenshot from adxv image header with pixel size highlighted.

File:Adxv menu.jpeg

2009-06-28T15:17:55Z

Grunet: Screen shot from ADXV image viewer with menu to view image header.

Screen shot from ADXV image viewer with menu to view image header.

Topics

2009-06-28T14:49:44Z

Grunet: /* Using XDS */ added link to tutorial

== Programs ==
* [[XDS]]
* [[XSCALE]]
* [[XDSCONV]]
* [[VIEW]]
* [[XDSSTAT]] (this is not part of the XDS package)
* [[Jiffies]] (little helpful scripts)
* [[SIM_MX]] (this is not part of the XDS package)
* [[adxv]] (this is not part of the XDS package)

== Using XDS ==
* [[XDS.INP]]
* [[Tutorial(First_Steps)|Tutorial]] Quick beginner's walk-through if you just want to get going.
* [[Processing]]
* [[IDXREF]] with subtopics [[Obtaining ORGX ORGY]], [[Ice rings]], [[Indexing]], [[Reindexing]]
* [[CORRECT]], see also [[Reindexing]]

== Typical procedures ==
* [[Space group determination]], see also [[Reindexing]] and [[Indexing]]
* [[Optimization]], subtopics [[MINIMUM_ZETA]], [[Aliens]], see also [[Ice rings]]
* [[Determining VALUE_RANGE_FOR_TRUSTED_DETECTOR_PIXELS]]
* [[Tips and Tricks]]
* [[Scaling with SCALA]]

== XDS development ==
* [[INTEGRATE]] - how does INTEGRATE work?
* [[Quality Control]]
* [[Known Bugs]]
* [[Wishlist]]
* [[Features that are not officially documented]]

== external Links ==

* [http://www.ccp4.ac.uk/xia/ xia2] Automated Data Reduction