Generate XDS.INP: Difference between revisions

== Usage ==

Usage is just (don't forget the quotation marks!):

  generate_XDS.INP "/home/myname/frms/mydata_1_???.img"

For improved interaction with [[XDSGUI]], it is advantageous to provide an ''absolute'' filename for the data files - one that starts with a slash ("/").

== The script ==

<pre>

#                      Cleaned up section for creating XDS.INP

# revision 1.15 . KD 22/05/2023 fix bug with second argument leading to usage message. Updating links.

#

#

# ====== Start of script ======

echo http://wiki.uni-konstanz.de/xds/index.php/generate_XDS.INP

   echo usage: generate_XDS.INP \"/file/system/frms/mydata_1_???.img\"  \(_with_ the quotation marks!\)

   echo If the frames are compressed leave out the extension .bz2, .gz, or .xz!

if [ "$is_h5" == 0 ]; then

  # extract strings from binary file header, removing leading '#' (Pilatus) and trailing ';' (CBF)

  # TODO: whenever FIRSTFRAME is used below, it should be copied to tmp2 (using IFS as above), and tmp2 should be used instead

  # this was done for "mccd", but still has to be done for the "raxis" detector types

  grep -q MARCONTROL tmp2 && DET=OLDMAR

  grep -q "WAVELENGTH=0.0" tmp2 && DET=experimental-ED

else

  h5dump -d "/entry/instrument/detector/description" $FIRSTFRAME | grep -i Eiger > /dev/null && DET=eiger

       # Known detectors for reversed-phi at SSRF BL18U1 (S/N 60-0123) and BL19U1 (S/N XX-XXX) (!; 2019-10-19: staff will fix this)

       DET_SN=`grep "Detector:" tmp2 | sed "s/^.*Detector: *//"`

       REVERSEPHI_SNs="

PILATUS3 6M, S/N 60-0125

PILATUS3 6M, S/N 60-0127

PILATUS3 6M, S/N 60-0123

"

       if echo "${DET_SN}${REVERSEPHI_SNs}" | sort | uniq -d | grep [0-9] > /dev/null; then

elif [ "$DET" == "raxis" ]; then

   DETECTOR="RAXIS MINIMUM_VALID_PIXEL_VALUE=0  OVERLOAD=2000000"

   DIRECTION_OF_DETECTOR_X_AXIS="$R1 0 $R3"

# end of Bruker-cbf section

else

   echo should never come here

     if [ -e $durin_path ]; then

       LIB="LIB=$durin_path"

     else

       LIB="!LIB=/usr/local/lib64/durin-plugin.so"

     fi

   fi

fi

if [ "$ROTATION_AXIS" != "" ]; then

elif [ "$REVERSE_PHI" == "no" ]; then

else

fi

# Append 2-theta comment if defined

if (( $(echo "${TWOTHETA}0 != 0"|bc -l) )); then

fi

# now we know almost everything that is required to generate XDS.INP

cat > XDS.INP << eof

!=============================================================================

! More documentation in wiki.uni-konstanz.de/xds/index.php .

! After running xds, inspect at least the agreement of predicted and observed

!=============================================================================

JOB= XYCORR INIT COLSPOT IDXREF DEFPIX INTEGRATE CORRECT

TRUSTED_REGION=$TRUSTED_REGION

VALUE_RANGE_FOR_TRUSTED_DETECTOR_PIXELS=6000. 30000. ! often 7000 or 8000 is ok

MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT=$MNOPIAS ! default of 6 is sometimes too high

! close spots/long cell axis: reduce SEPMIN and CLUSTER_RADIUS from their defaults of 7 and 3.5

eof

   echo "DELPHI=15 ! refine less often than the default of 5" >> XDS.INP

elif [ "$DET" == "adsc-CMOS1" ]; then

!                                                  and does not seem to have any downsides.

eof

   if [ "$NX" == "1028"  -a "$NY" == "1062" ]; then

# Eiger2 1M ; v0.97 numbers from Andreas Förster

     cat >> XDS.INP << eof

! the following specifications are for a detector _without_ proper

! to each UNTRUSTED_RECTANGLE

!EXCLUSION OF VERTICAL DEAD AREAS OF THE PILATUS 2M DETECTOR

  UNTRUSTED_RECTANGLE=    0 2069  1612 1651

eof

# Pilatus 6M

# FIXME: here we could test if a Flat_field correction was applied like we do for 2M

  UNTRUSTED_RECTANGLE=    0 4149  3812 3851

eof

   fi

fi

# Mechanism to append detector-specific parameters without editing this file:

   echo "Site parameters:"

   echo

   # Only FRACTION_POLARIZATION and ROTATION_AXIS are commented out automatically because they

   # Also likely adjustments are UNTRUSTED regions and maybe SILICON, but those don't replace existing parameters

   # Create a pattern like "XX\|YY" to search overriden parameters.

       echo "Overridden parameters:"

       echo

   fi

else

   echo "! Detector specific parameters can be placed in:" >> XDS.INP

fi

echo XDS.INP is ready for use. The file has only the most important keywords.

echo More documentation in wiki.uni-konstanz.de/xds/index.php .

echo After running xds, inspect at least the agreement of predicted and observed

rm -f tmp1 tmp2

</pre>

== System-wide or personal installation ==

Ask your system adminstrator to cut-and-paste the script into e.g. /usr/local/bin/generate_XDS.INP, and to make it "executable".

  chmod +x ~/generate_XDS.INP

After that, you can just run it in a similar way as if it were installed in your $PATH:

Instead of cutting-and-pasting the lines of the script, you (or the system administrator) could just cut-and-paste the following four lines

<pre>

   sed -e s/\&nbsp\;/\ /g -e s/\&gt\;/\>/g -e s/\&lt\;/\</g -e s/amp\;//g -e s/\&quot\;/\"/g -e s/\&\#\1\6\0\;/\ /g | \

  chmod +x generate_XDS.INP

</pre>

to copy the script from this website into an executable file generate_XDS.INP in your current directory. On a Mac (which does not seem to have wget), one could try

<pre>

   sed -e s/\&nbsp\;/\ /g -e s/\&gt\;/\>/g -e s/\&lt\;/\</g -e s/amp\;//g -e s/\&quot\;/\"/g -e s/\&\#\1\6\0\;/\ /g | \

  chmod +x generate_XDS.INP

</pre>

== Calling generate_XDS.INP from a Python script ==

== Dependencies ==

The script makes use of many GNU commands, like <code>ls, grep, egrep, awk, cut, cat, echo, wc, bc, head, sed, tail, cp, od, python</code>. Some of them (like <code>od</code> and <code>python</code>) are only used in case of specific detectors (MarCCD and RAXIS, respectively).  

On Mac OS X, installation of the "Command Line Tools" (from http://developer.apple.com/downloads; requires Apple ID) is required (open a terminal and type <code>xcode-select --install</code>). These are also part of the (larger, but also free) [http://developer.apple.com/tools/xcode Xcode] package. This package comes with a license that has to be accepted by the user when running a Command Line Tool (e.g. <code>strings</code>) for the first time.

See also [[Installation]].

== Limitations ==

* The authors have made a "best effort" to provide a XDS.INP that results in the correct sign of the anomalous signal. In the case of one detector type (internally called Rigaku SMV) this requires reversal of one detector axis, and a negative DETECTOR_DISTANCE, as is found in some of the [http://xds.mpimf-heidelberg.mpg.de/html_doc/xds_prepare.html detector-specific templates]. '''For an unusual or unknown detector setup, the correct sign of the anomalous signal needs to be established and verified e.g. with a good dataset from a test crystal that has an anomalous signal.''' The authors do not take any responsibility for problems arising from incorrect sign of the anomalous signal, nor - obviously! - for any other mischief arising in or from data processing.

* there are apparently several flavours of HDF5 files produced at Diamond Light Source. They differ e.g. in the naming of the header items. This means that items like NX, NY, DETECTOR_DISTANCE and number of images cannot always be determined by the [[generate_XDS.INP]] script. Example: The data at /dls/i04-1/data/2021/mx28114-9/processing/Lenye_Diamini/ThiL/ThiL found during the CCP4 2021 online Cape Town workshop. A workaround is to use e.g. xia2 pipeline=3dii to process these files, and - if needed - extract those items from its output files, e.g. from DEFAULT/NATIVE/SWEEP1/index/XDS.INP .