Generate XDS.INP: Difference between revisions

== Usage ==

== The script ==

<pre>

# revision 0.15 . KD 2/2011 - add comment for -ive sign of APS 19-ID and Australian Synchrotron rotation axis

# revision 0.21 . KD 11/2011 - SEPMIN, CLUSTER_RADIUS hints; read NX NY from header (for Pilatus 2M)

# revision 0.22 . KD 12/2011 - Pilatus 2M UNTRUSTED_RECTANGLE lines, SENSOR_THICKNESS from header

# revision 0.46 . Keitaro 6/2014 automatically set ROTATION_AXIS=-1 0 0 for Mar225 at SPring-8 BL26B2.

# revision 0.47 . Keitaro 7/2014 more generic dTREK format support (Saturn and RAXIS)

# revision 0.49 . Nobuhisa 2/2015 add  detector serial number for AichiSR BL2S1 to beam center convention 1

# revision 0.50 . KD 03/2015 workaround for Mar-1 change of parameter name "DISTANCE" to "POSITION" in REFINE(*) keywords

# revision 0.82 . KD 01/03/2018 STARTING_ANGLE for MarCCD/Pilatus/PHOTON, enabling to use dials.rs_mapper with spot2pdb.

# revision 0.83 . KD 25/06/2018 for ADSC detector #458 at APS BM19, revert the definition of ROTATION_AXIS=-1 0 0. See "Beamline notes" in this wiki.

# revision 0.85 . Jie Nan 09/01/2019 STARTING_ANGLE for Eiger

# revision 0.86 . Keitaro 03/05/2019 Add PILATUS3 6M, S/N 60-0123 at SSRF BL18U-1 with ROTATION_AXIS=-1 0 0

# revision 0.92 . KD 27/02/2020 read *_master.h5 from Diamond Light Source

# revision 0.93 . KD 13/03/2020 print out 2theta for MarCCD (DETECTOR_*_AXIS can be derived from this)

# revision 0.95 . KD 29/07/2020 fix DLS Eiger HDF5 variant OSCILLATION_RANGE, STARTING_ANGLE. Attention: DLS Eiger variant needs h5dump 1.10 for OVERLOAD!

# revision 0.96 . KD 03/10/2020 make script echo detector serial number if inverted ROTATION_AXIS is detected.

# revision 0.97 . KD 19/10/2020 add UNTRUSTED_RECTANGLEs for Eiger2 (which has a few pixels less than Eiger (thanks to Andreas Förster)

# revision 0.98 . Thomas Hauß (HZB) 06/11/2020 switch Python print syntax to Python3

# - for ADSC detectors, there are at least three ways to obtain ORGX and ORGY values from the header (see below);

# conversion radian / degrees:

DEGTOR=57.2957795

REVERSE_PHI="no"

# default MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT

# default DIRECTION_OF_DETECTOR_X-AXIS

DIRECTION_OF_DETECTOR_X_AXIS="1 0 0"

# default FRACTION_OF_POLARIZATION

pol_frac=0.98

STARTING_ANGLE=0

dtrek_det=""

SEPMIN=7.0

CLUSTER_RADIUS=3.5

test "${dname}" == "" && dname="."

NAME_TEMPLATE_OF_DATA_FRAMES="${dname}/${bname}"

# list frames matching the wildcards in NAME_TEMPLATE_OF_DATA_FRAMES

# cope with blanks in directory / file name

IFS=$'\n'

  data_last=`cut -b $pos1-$pos2 tmp1 | tail -n1 | bc`

  DATA_RANGE="$data_first $data_last"

  # set SPOT_RANGE to first half of DATA_RANGE

  data_num=`wc -l tmp1 | awk '{print $1}'`

  data_half=`echo "if ($data_half<=1) 1;if ($data_half>1) $data_half" | bc -l`

  spot_last=`echo "scale=0; $data_first+$data_half-1" | bc -l`

FIRSTFRAME=`head -1 tmp1`

echo $FIRSTFRAME | grep "\.h5$" && is_h5=1 || is_h5=0

# find out detector type

echo $FIRSTFRAME | grep -q '\.bz2$' && bzcat $FIRSTFRAME > tmp1 && FIRSTFRAME=tmp1

# for mac/linux compatibility. zcat foo.gz doesn't work on mac.

unset IFS

if [ "$is_h5" == 0 ]; then

  # TODO: whenever FIRSTFRAME is used below, it should be copied to tmp2 (using IFS as above), and tmp2 should be used instead

  # this was done for "mccd", but still has to be done for the "raxis" detector types

  head -n1 tmp2 | grep -q "^RAXIS" && DET=raxis

  head -n1 tmp2 | grep -q "^R-AXIS" && DET=raxis

  grep -q "^SOURCE_WAVELENGTH= *1" tmp2 && DET=dtrek

else

  h5dump -d "/entry/instrument/detector/description" $FIRSTFRAME | grep -i Eiger > /dev/null && DET=eiger

fi

# parse ASCII header of first frame

if [ "$DET" == "XXX" ]; then

# find parameters of first frame

   echo Data from a MarCCD detector

# http://www.sb.fsu.edu/~xray/Manuals/marCCD165header.html has header information

   DETECTOR="CCDCHESS MINIMUM_VALID_PIXEL_VALUE= 1 OVERLOAD= 65500"

   # Check detector serial number and recognize beamline for reversed-phi setting.

   cp $FIRSTFRAME tmp2

   unset IFS

   NX=$(od -t dI -j $SKIP -N 4 tmp2 | head -1 | awk '{print $2}')

   NY=$(od -t dI -j $SKIP -N 4 tmp2 | head -1 | awk '{print $2}')

   let SKIP=1720

   DETECTOR_DISTANCE=$(od -t dI -j $SKIP -N 4 tmp2 | head -1 | awk '{print $2}')

# Mar 12, 2020 KD

   let SKIP=1024+256+128+256+44

   STARTING_ANGLE=`echo "scale=2; $STARTING_ANGLE/1000" | bc -l `

   if [ "$DET_SN" == "4" ]; then

     TEMP=$ORGY

   let SKIP=1024+736

   let SKIP=1024+256+128+256+128+128+12

   X_RAY_WAVELENGTH=$(od -t dI -j $SKIP -N 4 tmp2 | head -1 | awk '{print $2}')

# at most BLs, ORGX and ORGY are in pixels, but sometimes in mm ... guess:

     echo MARCCD detector: header ORGX, ORGY seem to be in mm ... converting to pixels

elif [ "$DET" == "adsc" ]; then

   DETECTOR="ADSC MINIMUM_VALID_PIXEL_VALUE= 1 OVERLOAD= 65000"

   echo Data from ADSC detector. Obtaining ORGX, ORGY depends on beamline setup:

       # find X_RAY_WAVELENGTH:

       # find NX, QX, ORGX and ORGY:

       NX=`grep SIZE1 tmp2 | tail -1 | sed s/SIZE1=//`

       QX=`grep PIXEL_SIZE tmp2 | sed s/PIXEL_SIZE=//`

       BEAM_CENTER_X=`grep BEAM_CENTER_X tmp2 | sed s/BEAM_CENTER_X=//`

       BEAM_CENTER_Y=`grep BEAM_CENTER_Y tmp2 | sed s/BEAM_CENTER_Y=//`

       COMMENT_ORGXY="

! Following are possible beam center interpretations for ADSC detectors"

       ORGX1=`echo "scale=1; $BEAM_CENTER_Y/$QX" | bc -l`

       ORGY1=`echo "scale=1; $BEAM_CENTER_X/$QX" | bc -l`

       COMMENT_ORGXY="${COMMENT_ORGXY}

! ORGX= $ORGX1 ORGY= $ORGY1 ! For ESRF, PF, APS, AS MX2 ..."

       ORGX2=`echo "scale=1; $NX-$BEAM_CENTER_X/$QX" | bc -l `

       ORGY2=`echo "scale=1; $BEAM_CENTER_Y/$QX" | bc -l `

       COMMENT_ORGXY="${COMMENT_ORGXY}

! ORGX= $ORGX2 ORGY= $ORGY2 ! For ALS 5.0.3,.."

# this alternative is written into the generated XDS.INP ! You have to correct this manually in XDS.INP, or adjust this script.

       ORGX3=`echo "scale=1; $BEAM_CENTER_X/$QX" | bc -l `

       ORGX4=`echo "scale=1; $BEAM_CENTER_X/$QX" | bc -l `

       ORGY4=`echo "scale=1; $BEAM_CENTER_Y/$QX" | bc -l `

       COMMENT_ORGXY="${COMMENT_ORGXY}

! ORGX= $ORGX4 ORGY= $ORGY4 ! For SPring-8,.."

       fi

elif [ "$DET" == "adsc-CMOS1" ]; then

   DETECTOR="ADSC MINIMUM_VALID_PIXEL_VALUE= 1 OVERLOAD= 65000"

   echo Data from CMOS1 MBC detector.

   X_RAY_WAVELENGTH=`grep WAVELENGTH tmp2 | head -1 | sed s/WAVELENGTH=//`

   NX=`grep SIZE1 tmp2 | tail -1 | sed s/SIZE1=//`

   QX=`grep PIXEL_SIZE tmp2 | sed s/PIXEL_SIZE=//`

   ORGX=`grep BEAM_CENTER_X tmp2 | sed s/BEAM_CENTER_X=//`

   ORGY=`grep BEAM_CENTER_Y tmp2 | sed s/BEAM_CENTER_Y=//`

   REVERSE_PHI="yes"

elif [ "$DET" == "pilatus" ]; then

   OVERLOAD=1048574

   SEPMIN=4

   CLUSTER_RADIUS=2

   DETECTOR="PILATUS MINIMUM_VALID_PIXEL_VALUE=0 OVERLOAD= $OVERLOAD  !PILATUS"

# the default above guards against missing Pixel_size line in CBF header

       # find SENSOR_THICKNESS:

       # find NX and NY; 2463/2527 is 6M, 1475/1679 is 2M

       NX=`grep X-Binary-Size-Fastest-Dimension tmp2 | awk '{print $2}'`

       # find ORGX and ORGY:

       # find DETECTOR_DISTANCE, OSCILLATION_RANGE, and STARTING_ANGLE:

       DETECTOR_DISTANCE=`echo "$DETECTOR_DISTANCE*1000" | bc -l`

       # get detector serial number and check if it is included in the list

       # Known detectors for reversed-phi at SSRF BL18U1 (S/N 60-0123) and BL19U1 (S/N XX-XXX) (!; 2019-10-19: staff will fix this)

       DET_SN=`grep "Detector:" tmp2 | sed "s/^.*Detector: *//"`

       REVERSEPHI_SNs="

PILATUS3 6M, S/N 60-0125

PILATUS3 6M, S/N 60-0127

PILATUS3 6M, S/N 60-0123

"

       if echo "${DET_SN}${REVERSEPHI_SNs}" | sort | uniq -d | grep [0-9] > /dev/null; then

         echo inverted rotation axis at SSRF BL19U1

       fi

# Diamond I24:

       if [ "$DET_SN" == "PILATUS3 6M, S/N 60-0119" ] ; then

         if grep -q "Oscillation_axis X.CW +SLOW" tmp2 ; then

           echo ROTATION_AXIS="0 -1 0" at Diamond I24

         fi

       fi

elif [ "$DET" == "eiger" ]; then

# find out if HDF5 from Diamond (DLS=1) or Dectris (DLS=0)

   DLS=0

   if [ "$DLS" == 1 ]; then

     echo Eiger HDF5 from Diamond

     OVERLOAD=`h5dump -d "/entry/instrument/detector/saturation_value" $FIRSTFRAME | awk '/\(0\):/{print $2}'`

   # v0.95: fix the next 2 lines by taking care of negative values with the \- , and stop after first "(0)"

     STARTING_ANGLE=`h5dump    -d "/entry/data/omega" $FIRSTFRAME | awk '/\(0\): [\-0-9]/{print $2;exit}' | sed -e "s/,//"`

     echo OSCILLATION_RANGE=$OSCILLATION_RANGE STARTING_ANGLE=$STARTING_ANGLE

   # the above gives -1 0 0 for DLS data instead of the correct 1 0 0, so commented out for now

   else

     echo Eiger HDF5 from Dectris

     OSCILLATION_RANGE=`h5dump -d "/entry/sample/goniometer/omega_range_average" $FIRSTFRAME | awk '/\(0\): [0-9]/{print $2}'`

   # STARTING_ANGLE:  the \- was introduced in version 0.91 to allow negative values :

   fi

   DETECTOR="EIGER MINIMUM_VALID_PIXEL_VALUE=0 OVERLOAD= $OVERLOAD"

   QX=`h5dump -d "/entry/instrument/detector/x_pixel_size" $FIRSTFRAME | awk '/\(0\): [0-9]/{print $2*1000}'`

   # find DETECTOR_DISTANCE :

   SEPMIN=4

elif [ "$DET" == "raxis" ]; then

   DETECTOR="RAXIS MINIMUM_VALID_PIXEL_VALUE=0  OVERLOAD=2000000"

   #let SKIP=768

   #NX=$(od -t x -j $SKIP -N 4 $FIRSTFRAME |awk 'NR==1{print toupper($2)}'|perl -nle '@array= $_ =~/.{2}/g; print "ibase=16;obase=A;".join("",reverse @array)'|bc)

   NX=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(768);print("%.4d"%struct.unpack(">i",f.read(4)))')

   NY=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(772);print("%.4d"%struct.unpack(">i",f.read(4)))')

   DETECTOR_DISTANCE=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(344);print("-%.4f"%struct.unpack(">f",f.read(4)))')

   ORGX=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(540);print("%.4f"%struct.unpack(">f",f.read(4)))')

   ORGY=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(544);print("%.4f"%struct.unpack(">f",f.read(4)))')

   OSCILLATION_RANGE=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(524);phis,phie=struct.unpack(">ff",f.read(8));print("%.4f"%(phie-phis))')

   QX=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(776);print("%.6f"%struct.unpack(">f",f.read(4)))')

   QY=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(780);print("%.6f"%struct.unpack(">f",f.read(4)))')

   X_RAY_WAVELENGTH=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(292);print("%.6f"%struct.unpack(">f",f.read(4)))')

elif [ "$DET" == "dtrek" ]; then

   dname=`grep "DETECTOR_NAMES=" tmp2 | sed -e "s/.*=//"`

   if [ "$dname" == "CCD_" ]; then

     DETECTOR="SATURN MINIMUM_VALID_PIXEL_VALUE=1"

     dtrek_det="saturn"

   elif [ "$dname" == "PILT_" ]; then

     DETECTOR="PILATUS MINIMUM_VALID_PIXEL_VALUE=0"

     SEPMIN=3

     CLUSTER_RADIUS=1.5

   elif [ "$dname" == "RX_" ]; then

     DETECTOR="RAXIS MINIMUM_VALID_PIXEL_VALUE=0"

     dtrek_det="raxis"

   else

     DETECTOR="XXX MINIMUM_VALID_PIXEL_VALUE=XXX"

   fi

   # find OVERLOAD

   SATURATED_VALUE=`grep SATURATED_VALUE tmp2 | head -1 | sed s/SATURATED_VALUE=//`

   ORGY=${SPATIAL_DISTORTION_INFO[1]}

   GONIO_NAMES=(`grep ${dname}GONIO_NAMES= tmp2 | tail -1 | sed s/.*_GONIO_NAMES=//`)

   GONIO_VALUES=(`grep ${dname}GONIO_VALUES= tmp2 | tail -1 | sed s/.*_GONIO_VALUES=//`)

   idx=$((i-1))

   if [ "${GONIO_NAMES[$idx]}" == "Distance" ]; then

   fi

   done

   ROTATION=(`grep "^ROTATION=" tmp2 | tail -1 | sed s/ROTATION=//`)

   OSCILLATION_RANGE=${ROTATION[2]}

elif [ "$DET" == "MAR345" ]; then

  echo  "Data from a Mar345 image plate detector"

  DETECTOR="MAR345  MINIMUM_VALID_PIXEL_VALUE=0  OVERLOAD=130000"

  if grep -q 'CENTER' tmp2; then

   echo Beam center found.

   ORGY=`echo $NY / 2 | bc`

  fi

elif [ "$DET" == "OLDMAR" ]; then

  echo  "Data from old type MAR image plate detector"

  DETECTOR="MAR  MINIMUM_VALID_PIXEL_VALUE=0  OVERLOAD=130000"

  NY=$NX

  QY=$QX

  echo unsure if sign of anomalous signal is correct - please verify or try both hands!

elif [ "$DET" == "Bruker-cbf" ]; then

  echo  "Data from a Bruker-cbf detector"

# MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT:

# use complete detector including corners:

# polarization

  pol_frac=`echo "${pol_frac}+0.5" | bc -l`

  DETECTOR="BRUKER MINIMUM_VALID_PIXEL_VALUE=0 OVERLOAD=${OVERLOAD}"

# ORGX/Y-offsets relative to center of detector:

  ORGY=`echo "scale=2; ${NY}/2-($ORGY/$QY)" | bc -l`

# fix 16.3.20: instead of print, use printf "%.5f", because bc does not accept e.g. 3.1e-005

# echo OMEGA DELTAOMEGA PHI DELTAPHI KAPPA= $OMEGA $DELTAOMEGA $PHI $DELTAPHI $KAPPA

     R1=`echo "scale=7; s($KAPPA/$DEGTOR)*c($OMEGA/$DEGTOR)" | bc -l`

     R2=`echo "scale=7; c($KAPPA/$DEGTOR)          " | bc -l`

     OSCILLATION_RANGE=${DELTAPHI}

     STARTING_ANGLE=${PHI}

  else

     echo OMEGA scan

     OSCILLATION_RANGE=${DELTAOMEGA}

     STARTING_ANGLE=${OMEGA}

  fi

# 2theta

   echo TWOTHETA=$TWOTHETA

   R1=`echo "scale=7; c($TWOTHETA/$DEGTOR)" | bc -l`

else

   echo should never come here

cat > XDS.INP << eof

! written by generate_XDS.INP version $REVISION

JOB= XYCORR INIT COLSPOT IDXREF DEFPIX INTEGRATE CORRECT

$COMMENT_ORGXY

NAME_TEMPLATE_OF_DATA_FRAMES=$NAME_TEMPLATE_OF_DATA_FRAMES

SPACE_GROUP_NUMBER=0                  ! 0 if unknown

! however, if the spots from COLSPOT extend to 2A then POSITION could, and if 1.5A POSITION should be refined

REFINE(IDXREF)=CELL BEAM ORIENTATION AXIS  ! add POSITION if high resolution, or DETECTOR_DISTANCE inaccurate

! parameters specifically for this detector and beamline:

DETECTOR= $DETECTOR

! attention CCD detectors: for very high resolution (better than 1A) make sure to specify SILICON

! as about 32* what CORRECT.LP suggests (absorption of phosphor is much higher than that of silicon).

NX= $NX NY= $NY  QX= $QX  QY= $QY ! to make CORRECT happy if frames are unavailable

eof

cat >> XDS.INP << eof

!used by DEFPIX and CORRECT to exclude ice-reflections / ice rings - uncomment if necessary

eof

if [ "$DET" == "Bruker-cbf" ]; then

   echo "DELPHI=15 ! refine less often than the default of 5" >> XDS.INP

elif [ "$DET" == "adsc-CMOS1" ]; then

   echo UNTRUSTED_RECTANGLE= 0 1468  2451 2631  >> XDS.INP

elif [ "$DET" == "pilatus" -o "$DET" == "eiger" ]; then

cat >> XDS.INP << eof

eof

# Eiger2 1M ; v0.97 numbers from Andreas Förster

     cat >> XDS.INP << eof

UNTRUSTED_RECTANGLE=    0 1029    512  551

eof

     if ! grep -q Flat_field tmp2 ; then

     cat >> XDS.INP << eof

! the following specifications are for a detector _without_ proper

! to each UNTRUSTED_RECTANGLE

!EXCLUSION OF VERTICAL DEAD AREAS OF THE PILATUS 2M DETECTOR

eof

     fi

# Eiger2 4M ; v0.97 numbers from Andreas Förster

     cat >> XDS.INP << eof

  UNTRUSTED_RECTANGLE=    0 2069  1612 1651

eof

# Pilatus 6M

# FIXME: here we could test if a Flat_field correction was applied like we do for 2M

UNTRUSTED_RECTANGLE=  0 2464  2315 2333

eof

# Eiger 9M

     cat >> XDS.INP << eof

UNTRUSTED_RECTANGLE= 0 3110 2717 2757

eof

# Eiger2 9M ; v0.97 numbers from Andreas Förster

     cat >> XDS.INP << eof

UNTRUSTED_RECTANGLE=    0 3109  2712 2751

eof

# Eiger 16M

     cat >> XDS.INP << eof

!EXCLUSION OF VERTICAL DEAD AREAS OF THE EIGER 16M DETECTOR + ONE PIXEL ON EACH SIDE

  UNTRUSTED_RECTANGLE= 1029 1042      0 4371

  UNTRUSTED_RECTANGLE= 3109 3122      0 4371

eof

# Eiger2 16M ; v0.97 numbers from Andreas Förster

     cat >> XDS.INP << eof

  UNTRUSTED_RECTANGLE= 2068 2081      0 4363

  UNTRUSTED_RECTANGLE= 3108 3121      0 4363

  UNTRUSTED_RECTANGLE=    0 4149    512  551

  UNTRUSTED_RECTANGLE=    0 4149  1062 1101

  UNTRUSTED_RECTANGLE=    0 4149  3812 3851

eof

   fi

fi

echo XDS.INP is ready for use. The file has only the most important keywords.

rm -f tmp1 tmp2

</pre>

== System-wide or personal installation ==

Ask your system adminstrator to cut-and-paste the script into e.g. /usr/local/bin/generate_XDS.INP, and to make it "executable".

  chmod +x ~/generate_XDS.INP

After that, you can just run it in a similar way as if it were installed in your $PATH:

== Copying generate_XDS.INP from XDSwiki webserver ==

On Linux:

  chmod a+x generate_XDS.INP

On Mac:

  chmod a+x generate_XDS.INP

See also [[Installation]].

Instead of cutting-and-pasting the lines of the script, you (or the system administrator) could just cut-and-paste the following four lines

<pre>

   sed -e s/\&nbsp\;/\ /g -e s/\&gt\;/\>/g -e s/\&lt\;/\</g -e s/amp\;//g -e s/\&quot\;/\"/g -e s/\&\#\1\6\0\;/\ /g | \

  chmod +x generate_XDS.INP

</pre>

to copy the script from this website into an executable file generate_XDS.INP in your current directory. On a Mac (which does not seem to have wget), one could try

<pre>

   sed -e s/\&nbsp\;/\ /g -e s/\&gt\;/\>/g -e s/\&lt\;/\</g -e s/amp\;//g -e s/\&quot\;/\"/g -e s/\&\#\1\6\0\;/\ /g | \

  chmod +x generate_XDS.INP

</pre>

== Dependencies ==

The script makes use of many GNU commands, like <code>ls, grep, egrep, awk, cut, cat, echo, wc, bc, head, sed, tail, cp, od, python</code>. Some of them (like <code>od</code> and <code>python</code>) are only used in case of specific detectors (MarCCD and RAXIS, respectively).  

For Eiger data processing, the <code>h5dump</code> program must be installed. This is part of <code>hdf5-tools</code> (Ubuntu) or <code>hdf5</code> (RHEL). The .h5 files collected at Diamond Light Source require a very new version of h5dump (namely h5dump 1.10) to extract the OVERLOAD parameter from the .h5 file; this version is available by default in Ubuntu 20.04 and RHEL/CentOS 8.

See also [[Installation]].