Problems

This is a collection of problems and their solutions.

XDS does not start

XDS must be installed in the PATH, i.e. in one of those directories in the filesystem where the shell looks for binaries. On both Linux and macOS, there is a default set of such directories; these typically should have those binaries that the distribution provides. Additional software and its binaries can be installed by the user/admin in different (additional) directories. On macOS, these typically go into /Applications if they come with an installer, but /usr/local/bin could also be used e.g. for single binaries. On Linux, typically large packages have their own directory, often under /opt if they are for all users, or under $HOME for a single user.

The downloading and installation of the XDS package is documented at https://xds.mr.mpg.de/html_doc/downloading.html , and also in Installation.

The important point is that after unpacking the XDS distribution, the resulting directory should be put into your PATH variable, as explained in the URL above. Failing to do so results in an error message, and a non-functional XDS installation.

For example, if you get the error message (e.g. in the terminal where you started XDSGUI)

/bin/bash: xds_par: command not found

then this means that either the XDS package was not installed at all, or the PATH variable does not point to its directory.

XDS crashes

XDS should never crash (if it terminates with an error message, this does not count as crash). If it does, it is either a bug in the program which should be brought to the attention of Wolfgang Kabsch or Kay Diederichs, and will be fixed, or it is a problem with your computer, or (rarely) it is due to improper inputs (this is the only example I know of).

If it crashes for the second reason, these are the things to try/consider:

read error

... may result in e.g.

forrtl: severe (24): end-of-file during read, unit 2, file bin1_01.tmp

Any error message that XDS prints out afterwards is just a follow-up error. The most likely cause of the problem is that your disk is full, or your quota exceeded. You must fix the root problem before you can continue.

huge frames leading to overflow of the shell's stack limit

Increase the stack limit of your shell, e.g. (if using csh or tcsh) with

limit stacksize 102400 

or (in case of bash)

ulimit -s 102400 

The numbers above mean a 10-fold increase over the default, and should be enough. I've found this to be necessary for unusually large frames (32 MB).

xds_par in this case also might need an increase of the environment variable OMP_STACKSIZE (e.g. "setenv OMP_STACKSIZE 128M").

According to http://stackoverflow.com/questions/13264274/why-segmentation-fault-is-happening-in-this-openmp-code one should also check the "virtual address space limit" with ulimit (increase with ulimit -v) .

Problems with OpenMP

If the "xds_par" binary crashed, try "xds". xds_par uses OpenMP for parallelization, which adds complexity. If it works with xds, but not with xds_par, then there is a chance that some environment variable needs to be set/changed. In any case the XDS developers would like to learn about this.

As of March 2021, Thomas Hauß (HZB) reported an OpenMP-related problem that is due to a bug in a library that the ifort compiler links into xds_par. It is unknown to us which version of the compiler has or does not have this bug. The bug has been seen with xds_par (VERSION Jan 31, 2020 BUILT=20200417) and is a crash with error message containing:

OMP: Error #13: Assertion failure at z_Linux_util.cpp(2361).
OMP: Hint Please submit a bug report with this message, compile and run commands used, and machine configuration info including native compiler and operating system versions. Faster response will be obtained by including all program sources. For information on submitting this issue, please see http://www.intel.com/software/products/support/.
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
xds_par            00000000005620C4  Unknown               Unknown  Unknown
libpthread-2.17.s  00002ADC08AAA630  Unknown               Unknown  Unknown
libc-2.17.so       00002ADC08EF1387  gsignal               Unknown  Unknown
libc-2.17.so       00002ADC08EF2A78  abort                 Unknown  Unknown
xds_par            000000000063FB83  Unknown               Unknown  Unknown
xds_par            000000000062BDFF  Unknown               Unknown  Unknown
xds_par            000000000060748C  Unknown               Unknown  Unknown
xds_par            000000000067980E  Unknown               Unknown  Unknown
xds_par            000000000063D737  Unknown               Unknown  Unknown
xds_par            000000000063EC58  Unknown               Unknown  Unknown
xds_par            0000000000629A0E  Unknown               Unknown  Unknown
xds_par            0000000000419056  xds_                    21586  MAIN_XDS.f90
xds_par            0000000000418951  MAIN__                      1  MAIN_XDS.f90
xds_par            0000000000415862  Unknown               Unknown  Unknown
libc-2.17.so       00002ADC08EDD555  __libc_start_main     Unknown  Unknown
xds_par            0000000000415769  Unknown               Unknown  Unknown

The bug is not related to the xds_par source code, but also happens with other software. The workaround is to set the environment variable

   bash: export KMP_INIT_AT_FORK=FALSE
   tcsh: setenv KMP_INIT_AT_FORK FALSE

ASSERT VIOLATION

If you get e.g.

!!! ERROR !!! ASSERT VIOLATION IN "RESTHKL"

this indicates that an internal error has occurred, i.e. some situation was hit that the program is not prepared to handle. There must be something very special about your dataset - maybe a very small number of reflections, maybe only negative intensities, maybe something else - which triggers the bug. If this happens then get in touch with the XDS developers - they will most likely ask to make your XDS.INP and data frames available to them, to fix the program.

Indexing failures

In case of IDXREF problems, you should definitively inspect, using XDS-viewer or adxv, the FRAME.cbf file which is written by COLSPOT. This file shows the first frame of the SPOT_RANGE, together with strong pixels that COLSPOT found on this frame, marked with white pixels. IDXREF will use those spot positions which have at least as many strong pixels as MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT specifies.

You will only be able to look at this file if you prevent XDS from running INTEGRATE. This can be achieved e.g. by

JOB= XYCORR INIT COLSPOT IDXREF DEFPIX ! INTEGRATE CORRECT

since otherwise, FRAME.cbf is overwritten by INTEGRATE.

To look at all spot positions found by COLSPOT, you can use the COLSPOT tab of XDSGUI. This nicely shows ice rings, and may also help to find shaded regions on the detector.

ugly diffraction pattern

If the spots are split or unclean, you may get a seemingly terrible match between observed and calculated spot positions:

***** REFINED SOLUTION BASED ON INDEXED REFLECTIONS IN SUBTREE # 1 *****

REFINED VALUES OF DIFFRACTION PARAMETERS DERIVED FROM      2916 INDEXED SPOTS
REFINED PARAMETERS:   AXIS BEAM ORIENTATION CELL
STANDARD DEVIATION OF SPOT    POSITION (PIXELS)    11.74
STANDARD DEVIATION OF SPINDLE POSITION (DEGREES)    5.94

which leads to very many reflections being rejected from refinement:

 ***** INDEXING OF OBSERVED SPOTS IN SPACE GROUP #   1 *****
     1594 OUT OF     27897 SPOTS INDEXED.
       0 REJECTED REFLECTIONS (REASON: OVERLAP)
   26303 REJECTED REFLECTIONS (REASON: TOO FAR FROM IDEAL POSITION)

The fix is to e.g. double the parameters

MAXIMUM_ERROR_OF_SPOT_POSITION=   6.0
MAXIMUM_ERROR_OF_SPINDLE_POSITION=   4.0

which then leads to less rejections:

***** INDEXING OF OBSERVED SPOTS IN SPACE GROUP #   1 *****
    11669 OUT OF     27897 SPOTS INDEXED.
       2 REJECTED REFLECTIONS (REASON: OVERLAP)
   16226 REJECTED REFLECTIONS (REASON: TOO FAR FROM IDEAL POSITION)

and surprisingly good integration.

IDXREF ends with !!! ERROR !!! message

These messages should really be called !!! WARNING !!!. The two types occuring most often are:

 !!! ERROR !!! SOLUTION IS INACCURATE

This message is printed if the "STANDARD DEVIATION OF SPOT POSITION (PIXELS)" is larger than the MAXIMUM_ERROR_OF_SPOT_POSITION= (default of 3).

or

!!! ERROR !!! INSUFFICIENT PERCENTAGE (< 50%) OF INDEXED REFLECTIONS

This message is printed if less than the given percentage of reflections that COLSPOT found was actually indexed. If this occurs, it is clearly a warning sign.

The message continues in both cases:

AUTOMATIC DATA PROCESSING STOPPED. AS THE CRITERIA FOR A GOOD
SOLUTION ARE RATHER STRICT, YOU MAY CHOOSE TO CONTINUE DATA
PROCESSING AFTER CHANGING THE "JOB="-CARD IN "XDS.INP" TO
"JOB= DEFPIX INTEGRATE CORRECT".
IF THE BEST SOLUTION IS REALLY NONSENSE YOU SHOULD FIRST HAVE
A LOOK AT THE ASCII-FILE "SPOT.XDS". THIS FILE CONTAINS THE
INITIAL SPOT LIST SORTED IN DECREASING SPOT INTENSITY. SPOTS
NEAR THE END OF THE FILE MAY BE ARTEFACTS AND SHOULD BE ERASED.
ALTERNATIVELY YOU MAY TRY DIFFERENT VALUES FOR "INDEX_ORIGIN"
AS SUGGESTED IN THE ABOVE LISTING.
IF THE CRYSTAL HAS SLIPPED AT THE BEGINNING OF DATA COLLECTION
YOU MAY CHOOSE TO SKIP SOME OF THE FIRST FRAMES BY CHANGING
THE "DATA_RANGE=" IN FILE "XDS.INP" AND START ALL OVER AGAIN.

This is printed out for you to actually read, and take action accordingly: it may result from one or more additional crystals contributing to the diffraction patterns. Or it may just be the case that there are ice rings with very many isolated ice reflections (in that case you may want to use EXCLUDE_RESOLUTION_RANGE for the IDXREF task, or just to ignore the message and go on with JOB=DEFPIX INTEGRATE CORRECT). Or you have MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT of 3, but your reflections comprise many more pixels and COLSPOT may "see" several reflections when there's only one (in that case use MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT=6 or even 12). In most cases you just change the JOB= line in XDS.INP to read

JOB= DEFPIX INTEGRATE CORRECT

and then continue to run XDS. In other cases you may want to change SPOT.XDS, or other keywords in XDS.INP (see also below). But in any case this is an important alert that should make you check the correctness of the parameters that describe the data collection (X-RAY_WAVELENGTH, DETECTOR_DISTANCE, ORGX, ORGY, OSCILLATION_RANGE, NAME_TEMPLATE_OF_DATA_FRAMES).

See also Problems#frame_numbers_beyond_999999 below.

IDXREF produces too short cell parameter(s)

The first table "CLUSTER COORDINATES AND INDICES WITH RESPECT TO REC. LATTICE BASIS VECTORS" should show close-to-integer difference vectors. The following is an example how it should not look like:

  #  COORDINATES OF REC. BASIS VECTOR    LENGTH   1/LENGTH

   1  -0.0000476-0.0014008-0.0044654  0.0046802     213.67
   2  -0.0101960-0.0056194 0.0019466  0.0118037      84.72
   3  -0.0145331 0.0238225-0.0071754  0.0288134      34.71

CLUSTER COORDINATES AND INDICES WITH RESPECT TO REC. LATTICE BASIS VECTORS 
  #  COORDINATES OF VECTOR CLUSTER   FREQUENCY       CLUSTER INDICES   
   1  0.0004199-0.0012633-0.0043826     2442.      0.97     -0.04     -0.00
   2  0.0101757 0.0060701-0.0019630     2175.     -0.03     -1.02      0.01
   3 -0.0076118 0.0114167-0.0040603     1965.      0.13      0.04      0.50    <---- half-integer indices!?
   4  0.0100552 0.0071646 0.0025337     1944.     -1.01     -0.99      0.01
   5  0.0072840-0.0101455 0.0084405     1841.     -1.10     -0.01     -0.49    <---- half-integer indices!?
   6  0.0000976 0.0027828 0.0089584     1792.     -2.00      0.00     -0.00
   7  0.0103103 0.0043851-0.0063902     1790.      0.98     -1.02      0.00
   8  0.0025742 0.0163995-0.0098507     1760.      0.95     -0.99      0.51    <---- half-integer indices!?
   9  0.0068686-0.0089437 0.0128884     1724.     -2.08      0.03     -0.49    <---- half-integer indices!?
  10 -0.0174443 0.0123504 0.0161324     1694.     -4.01      0.99      0.52    <---- half-integer indices!?
  11  0.0272195-0.0050005-0.0136142     1678.      2.99     -1.97     -0.50    <---- half-integer indices!?
 ...

PARAMETERS OF THE REDUCED CELL (ANGSTROEM & DEGREES)
    34.71     84.72    213.67     90.35     90.27     90.06

Do you see the half-integer difference vectors in the last column? This clearly indicates that the third axis above the table (with 1/length=34.71) actually should be twice that size. If this happens, the histogram of indexed spots often has two equally-large subtrees:

SUBTREE    POPULATION

    1          976
    2          972
    3           10
    4            7
    5            6

Remedy: take the reduced cell (as found below the table), but double the 34.71, and stick it into the line

UNIT_CELL_CONSTANTS=69.42     84.72    213.67     90.35     90.27     90.06 

in XDS.INP. To make XDS actually use that line, specify

SPACE_GROUP_NUMBER=1

and leave the space group determination for later.

Alternatively, try several well-separated SPOT_RANGEs instead of 1. For example, instead of

SPOT_RANGE=1 900

try e.g.

SPOT_RANGE=1 100
SPOT_RANGE=801 900

In the cases where I've tried this, this splitting of SPOT_RANGE into several wedges always resulted in IDXREF picking up the correct cell.

The reason for the indexing failure in these cases seems to be the fact that the crystal changed its orientation within the SPOT_RANGE by more than ~ 0.1°.

The other reason for this failure mode - but in this case there is only one SUBTREE with high POPULATION - is when the default SEPMIN= 6.00 CLUSTER_RADIUS= 3 are used, but the spots are closer than 6 pixels. This often happens with Pilatus data; the recommendation for this detector is to use SEPMIN= 4.00 CLUSTER_RADIUS= 2 in XDS.INP, or even SEPMIN= 2.00 CLUSTER_RADIUS= 1.

Difference vectors are neither integers nor halfs

This can happen if SPACE_GROUP_NUMBER is wrong, i.e. the user "forces" a lattice (e.g. body-centered) that does not match the true one (which may be primitive). So the first thing to try is SPACE_GROUP_NUMBER=0.

Sometimes, IDXREF nevertheless finds no good lattice:

  #  COORDINATES OF VECTOR CLUSTER   FREQUENCY       CLUSTER INDICES
   1  0.0090336-0.0044767 0.0137041     1636.      0.99      0.99     -0.00
   2 -0.0239422 0.0085088 0.0078304     1556.     -2.00      1.00      0.00
   3  0.0088799 0.0092726 0.0077661     1555.      0.50      0.50     -0.56
   4 -0.0000164 0.0138382-0.0058215     1486.     -0.49     -0.49     -0.56     <---- 0.56 is neither close to 0.5 nor to 0 or 1
   5 -0.0148222 0.0040131 0.0216634     1468.     -1.00      2.00     -0.00
   6  0.0098829 0.0070246-0.0042191     1430.      0.50     -0.50     -0.44     <---- same here
   7 -0.0030325 0.0110238 0.0056577     1422.     -0.50      0.50     -0.44     <----  and here
...

If it is not a case of SEPMIN and CLUSTER_RADIUS being too large (see above), try to increase INDEX_ERROR - in this case, it indexes beautifully with INDEX_ERROR=0.14 .

IDXREF produces too long axes

This may (rarely) happen when MAXIMUM_NUMBER_OF_JOBS is > 1. In this case, different JOBS' COLSPOT runs may report some reflections twice in SPOT.XDS. Since their phi values are close, they correspond to long unit cell parameters. This happens more easily if the mosaicity is high and therefore reflections extend over the borders between JOBs.

This effect may be mitigated by having as many SPOT_RANGEs as JOBs, and leaving gaps between the SPOT_RANGEs.

IDXREF prints !!! WARNING !!! message

If you see

!!! WARNING !!! REFINEMENT DID NOT CONVERGE
                LAST CORRECTION SHIFT WAS   6.1E-02 (should be <  1.0E-03)

then try the following: find the line

REFINE(IDXREF)= ...

in XDS.INP. If it is commented out with a !, remove the !. Then, change it to have

REFINE(IDXREF)=CELL BEAM ORIENTATION AXIS ! POSITION

i.e. remove POSITION from the list of refinable parameters. Once that is done, save XDS.INP and re-run the IDXREF step.

This problem occurs if the POSITION value (called DISTANCE in former XDS versions) is large, and XDS cannot refine it meaningfully.

If this does not help, try to refine even less items, e.g. leave out AXIS.

IDXREF prints ERROR IN REFINE !!! RETURN CODE IS IER= 0

This is due to either wrong inputs in XDS.INP, or due to bad data, e.g. spots from many crystals whose diffraction patterns overlap. The first possibility applies if the diffraction pattern is clean. The second applies if the diffraction pattern is ugly, and unsuitable for indexing. In that case, maybe try a different SPOT_RANGE.

IDXREF.LP does not show the expected lattice

By specifying the spacegroup and unitcell, you tell XDS that it should index based on those reflections that match that spacegroup and unitcell. In a sense, you _force_ that spacegroup and cell. This will discard other (spurious?) reflections, and usually leads to a clean list of Bravais lattice possibilities.

If the data reduction fails nevertheless (in terms of bad R-factors and ISa in the CORRECT step), then chances are that you specified some parameter wrongly, or not accurate enough (ORGX and ORGY are the most likely candidates), or that the crystal does not match your idea about its spacegroup and unit cell. Actually the latter happens pretty frequently (which is why it is always the safest way to collect 180° of spindle rotation unless you know your crystals very well; this is also a good strategy in other respects).

If you tell XDS that you do not know the spacegroup (SPACE_GROUP_NUMBER= 0), then IDXREF takes all observed reflections into account. By design, the spacegroup decision is then postponed until the CORRECT step, or rather to a run of pointless after CORRECT, and it may be not vital to closely inspect IDXREF.LP, because CORRECT.LP has basically the same information, plus more.

none of the lattices in IDXREF.LP (except aP) has a good QUALITY OF FIT

It is a good idea to use many frames for the SPOT_RANGE, (e.g. the first half of the DATA_RANGE, as does the generate_XDS.INP script) for the COLSPOT and IDXREF steps.

It is entirely possible to run COLSPOT for the complete DATA_RANGE (in that case the parameters of SPOT_RANGE are the same as for the DATA_RANGE), and to try smaller SPOT_RANGEs in IDXREF - this means COLSPOT has to be run only once, and the JOBS= line has e.g.

JOBS= IDXREF 

only. Maybe some SPOT_RANGE sub-range gives a clear answer. In that case it may be useful to use

REFINE(INTEGRATE)=! AXIS BEAM ORIENTATION CELL DISTANCE

because otherwise the spurious reflections in the other ranges will probably disturb the on-the-fly refinement of parameters.

I've seen datasets where each reflection had a satellite associated with the main reflection, but separate from it. This is well visible in FRAME.cbf written by COLSPOT. In such a case it helps to use

MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT= 12 ! re-run COLSPOT after changing the parameter !

thus doubling (w.r.t. the default of 6) the required spotsize that makes a reflection be used for indexing. In the latest case this made it possible to index cleanly on a single frame (which is actually not uncommon).

A final possibility: your crystal may really be triclinic - hopefully you collected 180° of data, or even a bit more than that.

frame numbers beyond 999999

If the number of question marks "?" in NAME_TEMPLATE_OF_DATA_FRAMES is larger than 6, to allow for e.g.

DATA_RANGE= 1000001 10001000

then IDXREF cannot properly read SPOT.XDS written by COLSPOT, and stops with

 !!! ERROR !!! CANNOT READ SPOT.XDS

This happens when the filenames of the frames have digits before the actual number, as in myframe1000001.cbf . The workaround is to specify e.g.

NAME_TEMPLATE_OF_DATA_FRAMES=myframe100????.cbf 
DATA_RANGE=1 1000

(Problem reported by Pedro Dinis)

Integration failure

INTEGRATE stops with error message

If INTEGRATE stops after e.g.

******************************************************************************
                    PROCESSING OF IMAGES      136 ...     140
******************************************************************************

with

!!! ERROR !!! AUTOMATIC DETERMINATION OF SPOT SIZE PARAMETERS HAS FAILED.
              YOU MAY RESTART THIS STEP AFTER SPECIFYING VALUES IN XDS.INP FOR:
              REFLECTING_RANGE=    REFLECTING_RANGE_E.S.D.=
              BEAM_DIVERGENCE=     BEAM_DIVERGENCE_E.S.D.= 

then there are two ways to resolve this:

• you could insert a line DELPHI=10 (or DELPHI=20, for 10 or 20 degrees batches; the default is 5 degree batches) into XDS.INP and re-run INTEGRATE
• as the error message suggests, you should reduce the upper limit of the DATA_RANGE, to stop before the problematic frames, and re-run INTEGRATE. In this example, you would modify XDS.INP to have

DATA_RANGE=1 135
JOB=INTEGRATE CORRECT 

Save XDS.INP, run XDS and inspect INTEGRATE.LP, to find the lines (e.g.)

BEAM_DIVERGENCE=   0.478  BEAM_DIVERGENCE_E.S.D.=   0.048
REFLECTING_RANGE=  1.100  REFLECTING_RANGE_E.S.D.=  0.157

Copy them to XDS.INP. Restore the original DATA_RANGE and continue.

If this problem happens with multiple XDS jobs working on the same data set, you might also get a message like

.../bin1_02.tmp
Image              PC                Routine            Line        Source             
xds_par            0000000000592B91  Unknown               Unknown  Unknown
xds_par            00000000004BEF21  joinintegrate_           8995  MAIN_XDS.f90
xds_par            0000000000410A5B  xds_                    21690  MAIN_XDS.f90
xds_par            000000000040B3A4  MAIN__                      1  MAIN_XDS.f90
xds_par            00000000004083F6  Unknown               Unknown  Unknown
libc.so.6          00007FB40222E830  Unknown               Unknown  Unknown
xds_par            00000000004082E9  Unknown               Unknown  Unknown

which looks like a real crash of the program, but in this case with a known reason.

Another error mode of INTEGRATE (in processing of small-molecule data) is ...

IMAGE IER  SCALE     NBKG NOVL NEWALD NSTRONG  NREJ  SIGMAB  SIGMAR
    5   0  0.975  3968463    1      5       1     0  0.1028  1.3486
    6   0  0.967  3969949    2      2       2     0  0.1688  0.0000
    7   0  0.929  3975306    0      3       0     0  0.0000  0.0000
 !!! ERROR !!! CANNOT ALLOCATE MEMORY
              YOU COULD RERUN THIS STEP WITH SMALLER VALUES FOR THE PARAMETERS
              NUMBER_OF_PROFILE_GRID_POINTS_ALONG_ALPHA/BETA=
              NUMBER_OF_PROFILE_GRID_POINTS_ALONG_GAMMA=

The error message is misleading in this case: there are too few reflections to build the average profile. The fix is: restart INTEGRATE after inserting e.g.

DELPHI=20 ! default is 5, so try with e.g. 10, 20, 45, 90, 180 

in XDS.INP and re-run INTEGRATE.

INTEGRATE cell and/or distance run away; xds crashes or has to be killed

If during INTEGRATE the cell keeps increasing or the distance decreasing or both, then xds starts to consume large amounts of memory and becomes very slow. This may finally exhaust all available memory, and the job either crashes or has to be killed by the user.

The fix here is to use

REFINE(INTEGRATE)= BEAM POSITION ORIENTATION ! CELL 

instead of (what used to be the default until June 2017)

REFINE(INTEGRATE)= BEAM POSITION ORIENTATION CELL 

Quite generally, the more conservative setting without CELL refinement is adequate unless your crystals diffract to quite high resolution. A sure sign that you should not be refining CELL is that the refined cell and/or distance values in INTEGRATE.LP fluctuate without obvious physical reason. Moreover, distance ("POSITION") refinement nicely soaks up any cell change resulting from radiation damage.

A compromise (if you suspect that the cell parameters actually change differently) would be

REFINE(INTEGRATE)= BEAM ORIENTATION CELL ! POSITION

This should also avoid the run-away but gives more freedom to the refinement. Of course it requires a refined distance (from IDXREF, or rather from GXPARM.XDS->XPARM.XDS renaming).

See also

XDS.INP#What can go wrong with this file?

Low dose data

Indexing

IDXREF.LP