Performance

Single-machine considerations

In the order of effect:

XDS scales well (i.e. the wallclock time for data processing goes down when the number of available cores is increased) in the INIT, COLSPOT, IDXREF, INTEGRATE and CORRECT steps when using the MAXIMUM_NUMBER_OF_PROCESSORS keyword. This triggers program-level parallelization, using OpenMP threads.
the program scales very well in the COLSPOT and INTEGRATE steps when using the MAXIMUM_NUMBER_OF_JOBS keyword. This triggers a shell-level parallelization. There is a slight penalty associated with high values of MAXIMUM_NUMBER_OF_JOBS= :
- in INTEGRATE, geometry refinement results are not transferred between JOBs: see Pathologies;
- in COLSPOT, the phi values at the borders between JOBs are less accurate (in particular if the mosaicity is high), and the same reflection may be listed twice in SPOT.XDS if it extends over the border between JOBs. The latter effect may be mitigated by having as many SPOT_RANGEs as JOBs, and leaving gaps between the SPOT_RANGEs; see Problems#IDXREF_produces_too_long_axes.
combining these two keywords gives the highest performance in my experience (see 2VB1#XDS_processing for an example). As a rough guide, I'd choose them to be approximately equal; an even number for MAXIMUM_NUMBER_OF_PROCESSORS should be chosen because that fits better with usual hardware. If in doubt, use a lower number for MAXIMUM_NUMBER_OF_JOBS than for MAXIMUM_NUMBER_OF_PROCESSORS.
NUMBER_OF_IMAGES_IN_CACHE avoids repeated (3-fold) reading of data frames in the INTEGRATE task during processing of a batch of frames. This comes at the expense of memory (RAM) and is discussed in Eiger. The default is DELPHI/OSCILLATION_RANGE+1 and is usually adequate. Only on low-memory systems (e.g a 8GB RAM machine for processing Eiger 16M data collected with 0.1° oscillation range, at DELPHI=5 and MAXIMUM_NUMBER_OF_JOBS=1) should this be set to 0, to conserve memory and avoid slow processing due to thrashing, or even killed XDS processes. If the cache size of a process exceeds 8GB, XDS will print a warning, and in that case the user has to explicitly include a NUMBER_OF_IMAGES_IN_CACHE=<desired number> line in XDS.INP, to confirm that actually so much memory should be used.
some overcommitting of resources (i.e. MAXIMUM_NUMBER_OF_PROCESSORS * MAXIMUM_NUMBER_OF_JOBS > number of cores) is beneficial; you'll have to play with these two parameters. If MAXIMUM_NUMBER_OF_PROCESSORS * MAXIMUM_NUMBER_OF_JOBS is >4096 (the default in RHEL), you may have to adjust the maxproc limit of your shell; in bash: ulimit -u unlimited.
the next thing to consider is DELPHI together with OSCILLATION_RANGE: if DELPHI (the rotation range of a batch of frames) is an integer multiple of MAXIMUM_NUMBER_OF_PROCESSORS * OSCILLATION_RANGE that would be good because it nicely balances the usage of the threads. For this purpose, you may want to change (if possible, raise) the value of DELPHI (default is 5 degrees). If you are doing fine-slicing then mis-balancing of threads is not an issue - but for those users who want to collect 1° frames (which I think is not the best way nowadays ...) it should be a consideration. Additional consideration: the total number of frames should be an integer multiple of the intended number of frames in a batch. Example: 360 frames of 0.5° can be processed on a 8-core machine optimally by specifying DELPHI=4, since then there are 8 frames in a batch and the complete dataset has 45 batches. For weak data one should consider raising DELPHI to 12; that would give 15 batches. A trick: if you want to use DELPHI=8 in this situation then just specify DATA_RANGE=1 368 (pretending 23 batches of 8°) instead of DATA_RANGE=1 360 . XDS will complain about the missing 8 frames, but that has no adverse effects except that no FRAME.cbf will be produced. All of this doesn't matter for a single dataset, but for mass processing of datasets it does make a difference.
performance-wise, I/O also plays a role because as soon as you run 24 or so processes then a single GB ethernet connection may be limiting. OTOH shell-level parallelization smoothes the load.
REFINE(INTEGRATE)= ! (empty list) makes INTEGRATE go much faster through the frames, since frames are processed less often when analyzing a batch of frames, and no geometry refinement takes place.
XDS with the MAXIMUM_NUMBER_OF_JOBS keyword can use several machines. This requires some setup as explained at the bottom of [1].
Hyperthreading (SMT), if available, is often beneficial. A "virtual" core has only about 20% performance of a "physical" core but it comes at no cost - you just have to switch it on in the BIOS of the machine.

Cluster

If a cluster of computers is available that allow login - without asking for a password - by ssh and that have NFS-mounted the relevant directories under the same paths, one can use the CLUSTER_NODES= keyword in XDS.INP.

If the other computers are not reachable by ssh, but coupled with a batch queueing system, then the forkxds script of the XDS distribution has to be modified: the node names are not relevant, and the ssh invocation has to be replaced by a qsub invocation. An example script will be available soon.

Multi-socket machines

Multi-socket machines consist of several nodes each comprising several CPUs and some amount of memory. The nodes are connected by specialized hardware (sometimes called interconnect or bus) that transports data between the nodes. Typically, node-local memory is faster to read and write than memory on a different node. This NUMA (non-uniform memory architecture) setup has consequences for the performance when used for running XDS jobs.

In particular, good performance is obtained if MAXIMUM_NUMBER_OF_JOBS is chosen as the number of nodes, and MAXIMUM_NUMBER_OF_PROCESSORS is chosen as the number of CPU cores (physical + virtual) of each socket. One then has to take care that each job ends up on its own socket. The following scripts do this. Please note that numactl has to be installed.

#!/bin/bash
#                      forkcolspot
#
# enables  multi-tasking by splitting the COLSPOT step of
# xds into independent jobs. Each job is carried out by the
# Fortran program mcolspot or mcolspot_par started by this
# script as a background process with a different set of
# input parameters.
#
# 'forkcolspot' is called by xds or xds_par in the COLSPOT
# step using the Fortran instruction
# CALL SYSTEM('forkcolspot ntask maxcpu'),
#    ntask  ::total number of jobs
#   maxcpu  ::maximum number of processors used by each job
#
# Clearly, this can only work if forkcolspot, mcolspot, and
# mcolspot_par are correctly installed in the search path
# for executables.
#
# W.Kabsch and K.Rohm     Version Februar 2005
# NOTE: No blanks allowed adjacent to the = signs !!!

# K.Diederichs 3/2016 NUMA affinity added
#export KMP_AFFINITY="verbose"
maxnode=`numactl -H|awk '/available/{print $2-1}'`
#echo highest node is $maxnode

ntask=$1  #total number of jobs
maxcpu=$2 #maximum number of processors used by each job
	   #maxcpu=1: use 'mcolspot' (single processor)
	   #maxcpu>1: use 'mcolspot_par' (openmp version)

pids=""                    #list of background process ID's
itask=1
inode=0   # initialize inode
while test $itask -le $ntask
do
# KD modification: which node?
   let inode=$inode+1
   if [ $inode -gt $maxnode ] 
      then let inode=0
   fi
#end modification
   if [ $maxcpu -gt 1 ]
      then echo "$itask" | numactl --cpunodebind=$inode mcolspot_par &
      else echo "$itask" | mcolspot     &
   fi
   pids="$pids $!"  #append id of the background process just started

   itask=`expr $itask + 1`
done
trap "kill -15 $pids" 2 15  # 2:Control-C; 15:kill
wait  #wait for all background processes issued by this shell
rm -f mcolspot.tmp  #this temporary file was generated by ads

#!/bin/bash
#                      forkintegrate
#
# enables  multi-tasking by splitting the INTEGRATE step of
# xds into independent jobs. Each job is carried out by the
# Fortran program mintegrate or mintegrate_par started by
# this script as a background process with a different set
# of input parameters.
#
# 'forkintegrate' is called by xds (or xds_par) in the
# INTEGRATE step using the Fortran instruction
# CALL SYSTEM('forkintegrate fframe ni ntask niba0 maxcpu'),
#    fframe ::id number of the first data image
#    ni     ::number of images in the data set
#    ntask  ::total number of jobs
#    niba0  ::minimum number of images in a batch
#    maxcpu ::maximum number of processors used by each job
#
# Clearly, this can only work if forkintegrate, mintegrate,
# and mintegrate_par are correctly installed in the search
# path for executables.
#
# W.Kabsch and K.Rohm     Version Februar 2005
# NOTE: No blanks allowed adjacent to the = signs !!!

# K.Diederichs 3/2016 NUMA affinity added
#export KMP_AFFINITY="verbose"
maxnode=`numactl -H|awk '/available/{print $2-1}'`
#echo highest node is $maxnode


fframe=$1 #id number of the first image
ni=$2     #number of images in the data set
ntask=$3  #total number of jobs
niba0=$4  #minimum number of images in a batch
maxcpu=$5 #maximum number of processors used by each job
	   #maxcpu=1: use 'mintegrate' (single processor)
	   #maxcpu>1: use 'mintegrate_par' (openmp version)

minitask=$(($ni / $ntask)) #minimum number of images in a job
mtask=$(($ni % $ntask))    #number of jobs with minitask+1 images
pids=""                    #list of background process ID's
nba=0
litask=0
itask=1
inode=0   # initialize inode
while test $itask -le $ntask
do
# KD modification: which node?
   let inode=$inode+1
   if [ $inode -gt $maxnode ] 
      then let inode=0
   fi
#end modification
   if [ $itask -gt $mtask ]
      then nitask=$minitask
      else nitask=$(($minitask + 1))
   fi
   fitask=`expr $litask + 1`
   litask=`expr $litask + $nitask`
   if [ $nitask -lt $niba0 ]
      then n=$nitask
      else n=$niba0
   fi
   if [ $n -lt 1 ]
      then n=1
   fi
   nbatask=$(($nitask / $n))
   nba=`expr $nba + $nbatask`
   image1=$(($fframe + $fitask - 1)) #id number of the first image

   if [ $maxcpu -gt 1 ]
      then echo "$image1 $nitask $itask $nbatask" | numactl --cpunodebind=$inode mintegrate_par &
      else echo "$image1 $nitask $itask $nbatask" | mintegrate     &
   fi
   pids="$pids $!"  #append id of the background process just started

   itask=`expr $itask + 1`
done
trap "kill -15 $pids" 2 15  # 2:Control-C; 15:kill
wait  #wait for all background processes issued by this shell
rm -f mintegrate.tmp  #this temporary file was generated by xds

As an alternative to numactl, one may use taskset or KMP_AFFINITY.

If likwid would be used instead of numactl one could have much better control of affinity groups.

In my tests on a 4-socket machine, the difference between runs with the original scripts and the NUMA-aware ones was a reduction of wallclock time by about 8%. With a 2-socket machine, I saw a <1% effect. But this will depend very much on the specific hardware.

Multi-socket machines in a cluster

In that case, I'd suggest to modify e.g. forkcolspot_cluster to not run mcolspot_par directly on the remote machine, but rather to run a script on that machine that checks the number of nodes, and runs mcolspot_par on the right node.

processing compressed data

XDS can process data files that were previously compressed with compress (.Z), gzip (.gz), bzip2 (.bz2) or xz (.xz). It does this by on-the-fly decompression to temporary files with standard names (SCRATCH2XXYY.tmp) where XX (XX = 01..99) stands for the "JOB" and YY (YY = 01..99) for the thread number that produces the temporary file.

Compression saves a lot of disk space, but decompression is time-consuming in terms of CPU and I/O. The penalty associated with decompression can be mitigated by

(Linux only) using symlinks pointing to /dev/shm which results in SCRATCH2XXYY.tmp being written to RAM instead of (network) disk. A script (typically called mklinks) achieving this is

#!/bin/bash
# purpose: create symlinks for xds_par
# usage: mklinks [# of jobs]

maxjobs=$1
test -z $1 && maxjobs=1

maxprocs=$(grep processor /proc/cpuinfo | wc -l)
echo creating symlinks for $maxprocs threads and $maxjobs JOBs

# create unique directory for SCRATCH2 files:
tempdir="/dev/shm/xds${PWD//\//_}"
rm -rf $tempdir
mkdir $tempdir

for j in $(seq 1 $maxjobs); do
  for i in $(seq 1 $maxprocs); do
    ln -sfn $tempdir/SCRATCH_$(printf "%02d" "$j")$(printf "%02d" "$i").tmp
    ln -sfn $tempdir/SCRATCH2_$(printf "%02d" "$j")$(printf "%02d" "$i").tmp
  done
done

This has to be run in the XDS processing directory of the current dataset, before running xds_par. After finishing data processing, one may cleanup with this script (typically called rmlinks):

#!/bin/bash
tempdir="/dev/shm/xds${PWD//\//_}"
rm -rf $tempdir
rm -f SCRATCH*

if decompressing .bz2 files, one can use the faster lbunzip2 (if it is installed) simply by making a symlink to it (assuming $HOME/bin is in your $PATH):

ln -s `which lbunzip2` $HOME/bin/bunzip2

Both measures can be combined.

Linux kernel setting

On RHEL6

cat /sys/kernel/mm/redhat_transparent_hugepage/enabled

On RHEL7

cat /sys/kernel/mm/transparent_hugepage/enabled

should show always, not never to be active (the active setting is bracketed).

Performance

Contents

Single-machine considerations

Cluster

Multi-socket machines

Multi-socket machines in a cluster

processing compressed data

Linux kernel setting

Navigation menu

Performance

Single-machine considerations

Cluster

Multi-socket machines

Multi-socket machines in a cluster

processing compressed data

Linux kernel setting

Navigation menu

Search