Problems
This is a collection of problems and their solutions.
XDS crashes
XDS should never crash (if it terminates with an error message, this does not count as crash). If it does, it is either a bug in the program which should be brought to the attention of Wolfgang Kabsch or Kay Diederichs, and will be fixed, or it is a problem with your computer.
If it crashes for the second reason, these are the things to try/consider:
read error
... may result in e.g.
forrtl: severe (24): end-of-file during read, unit 2, file bin1_01.tmp
Any error message that XDS prints out afterwards is just a follow-up error. The most likely cause of the problem is that your disk is full, or your quota exceeded. You must fix the root problem before you can continue.
huge frames leading to overflow of the shell's stack limit
Increase the stack limit of your shell, e.g. (if using csh or tcsh) with
limit stacksize 102400
or (in case of bash)
ulimit -s 102400
The numbers above mean a 10-fold increase over the default, and should be enough. I've found this to be necessary for unusually large frames (32 MB). xds_par in this case also might need an increase of the environment variable OMP_STACKSIZE (e.g. "setenv OMP_STACKSIZE 128M").
memory consumption exceeding the 4GB barrier when using the 32bit binary
As pointed out by James Holton, the memory consumed by huge frames in combination with many threads may be larger than 32 bits allow. Execution then stops with
forrtl: severe (41): insufficient virtual memory
The workaround is to limit the MAXIMUM_NUMBER_OF_PROCESSORS, or to use XDS instead of XDS_PAR. An even better solution is to switch to the 64bit binary, but that of course requires a 64bit operating system.
Problems with OpenMP
If the "xds_par" binary crashed, try "xds". xds_par uses OpenMP for parallelization, which adds complexity. If it works with xds, but not with xds_par, then there is a chance that some environment variable needs to be set/changed. In any case the XDS developers would like to learn about this.
limitation of the parallel 32bit binary
The 32bit OpenMP binary may not be able to allocate enough storage when using many threads (e.g. >10). The error message might be something like:
OMP: Error #34: System unable to allocate necessary resources for OMP thread: OMP: System error #11: Resource temporarily unavailable
64bit binary on a 32 bit operating system
If the error message is e.g.
xds: Exec format error. Wrong Architecture.
or
-bash: /usr/local/bin/xds: cannot execute binary file
make sure to use the 32bit version of XDS instead - you try to run the 64bit version on a 32bit operating system; this won't work.
CPU without SSE2 support
If the error message is
forrtl: severe (168): Program Exception - illegal instruction Image PC Routine Line Source xds 08055250 Unknown Unknown Unknown xds 0804B3F6 Unknown Unknown Unknown
this means that unfortunately the CPU of your machine is too old to be supported by XDS (which in this respect is compiled with the default options of the ifort compiler). This may happen e.g. with old AMD CPUs that don't support SSE2 (on Linux check the flags field of /proc/cpuinfo for the presence of the sse2 flag).
According to [1] the following Intel-compatible CPUs did not implement SSE2 after SSE2 was introduced (2001):
- AMD CPUs prior to Athlon 64, including all Socket A-based CPUs
- Intel CPUs prior to Pentium 4
- VIA C3
- Transmeta Crusoe
ASSERT VIOLATION
If you get e.g.
!!! ERROR !!! ASSERT VIOLATION IN "RESTHKL"
this indicates that an internal error has occurred, i.e. some situation was hit that the program is not prepared to handle. There must be something very special about your dataset - maybe a very small number of reflections, maybe only negative intensities, maybe something else - which triggers the bug. If this happens then get in touch with the XDS developers - they will most likely ask to make your XDS.INP and data frames available to them, to fix the program.
Indexing failures
In case of IDXREF problems, you should definitively inspect, using XDS-viewer or adxv, the FRAME.cbf file which is written by COLSPOT. This file shows the first frame of the SPOT_RANGE, together with strong pixels that COLSPOT found on this frame, marked with white pixels. IDXREF will use those spot positions which have at least as many strong pixels as MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT specifies.
You will only be able to look at this file if you prevent XDS from running INTEGRATE. This can be achieved e.g. by
JOB= XYCORR INIT COLSPOT IDXREF DEFPIX ! INTEGRATE CORRECT
since otherwise, FRAME.cbf is overwritten by INTEGRATE.
To look at all spot positions found by COLSPOT, you could try
echo "set yrange [] reverse ; plot 'SPOT.XDS' us 1:2 w dots" | gnuplot -persist
This may also help to find shaded regions on the detector.
IDXREF ends with !!! ERROR !!! message
These messages should really be called !!! WARNING !!!. The two types occuring most often are:
!!! ERROR !!! SOLUTION IS INACCURATE
This message is printed if the "STANDARD DEVIATION OF SPOT POSITION (PIXELS)" is larger than the MAXIMUM_ERROR_OF_SPOT_POSITION= (default of 3).
or
!!! ERROR !!! INSUFFICIENT PERCENTAGE (< 50%) OF INDEXED REFLECTIONS
This message is printed if less than the given percentage of reflections that COLSPOT found was actually indexed. If this occurs, it is clearly a warning sign.
The message continues in both cases:
AUTOMATIC DATA PROCESSING STOPPED. AS THE CRITERIA FOR A GOOD SOLUTION ARE RATHER STRICT, YOU MAY CHOOSE TO CONTINUE DATA PROCESSING AFTER CHANGING THE "JOB="-CARD IN "XDS.INP" TO "JOB= DEFPIX INTEGRATE CORRECT". IF THE BEST SOLUTION IS REALLY NONSENSE YOU SHOULD FIRST HAVE A LOOK AT THE ASCII-FILE "SPOT.XDS". THIS FILE CONTAINS THE INITIAL SPOT LIST SORTED IN DECREASING SPOT INTENSITY. SPOTS NEAR THE END OF THE FILE MAY BE ARTEFACTS AND SHOULD BE ERASED. ALTERNATIVELY YOU MAY TRY DIFFERENT VALUES FOR "INDEX_ORIGIN" AS SUGGESTED IN THE ABOVE LISTING. IF THE CRYSTAL HAS SLIPPED AT THE BEGINNING OF DATA COLLECTION YOU MAY CHOOSE TO SKIP SOME OF THE FIRST FRAMES BY CHANGING THE "DATA_RANGE=" IN FILE "XDS.INP" AND START ALL OVER AGAIN.
This is printed out for you to actually read, and take action accordingly. In most cases you just change the JOBS - line in XDS.INP to read
JOB= DEFPIX INTEGRATE CORRECT
and then continue to run XDS. In other cases you may want to change SPOT.XDS, or other keywords in XDS.INP (see also below). But in any case this is an important alert that should make you check the correctness of the parameters that describe the data collection (X-RAY_WAVELENGTH, DETECTOR_DISTANCE, ORGX, ORGY, OSCILLATION_RANGE, NAME_TEMPLATE_OF_DATA_FRAMES).
IDXREF produces too short cell parameter(s)
The first table "CLUSTER COORDINATES AND INDICES WITH RESPECT TO REC. LATTICE BASIS VECTORS" should show close-to-integer difference vectors. The following is an example how it should not look like:
# COORDINATES OF REC. BASIS VECTOR LENGTH 1/LENGTH
1 -0.0000476-0.0014008-0.0044654 0.0046802 213.67
2 -0.0101960-0.0056194 0.0019466 0.0118037 84.72
3 -0.0145331 0.0238225-0.0071754 0.0288134 34.71
CLUSTER COORDINATES AND INDICES WITH RESPECT TO REC. LATTICE BASIS VECTORS
# COORDINATES OF VECTOR CLUSTER FREQUENCY CLUSTER INDICES
1 0.0004199-0.0012633-0.0043826 2442. 0.97 -0.04 -0.00
2 0.0101757 0.0060701-0.0019630 2175. -0.03 -1.02 0.01
3 -0.0076118 0.0114167-0.0040603 1965. 0.13 0.04 0.50
4 0.0100552 0.0071646 0.0025337 1944. -1.01 -0.99 0.01
5 0.0072840-0.0101455 0.0084405 1841. -1.10 -0.01 -0.49
6 0.0000976 0.0027828 0.0089584 1792. -2.00 0.00 -0.00
7 0.0103103 0.0043851-0.0063902 1790. 0.98 -1.02 0.00
8 0.0025742 0.0163995-0.0098507 1760. 0.95 -0.99 0.51
9 0.0068686-0.0089437 0.0128884 1724. -2.08 0.03 -0.49
10 -0.0174443 0.0123504 0.0161324 1694. -4.01 0.99 0.52
11 0.0272195-0.0050005-0.0136142 1678. 2.99 -1.97 -0.50
...
PARAMETERS OF THE REDUCED CELL (ANGSTROEM & DEGREES)
34.71 84.72 213.67 90.35 90.27 90.06
Do you see the half-integer difference vectors in the last column? This clearly indicates that the third axis above the table (with 1/length=34.71) actually should be twice that size. If this happens, the histogram of indexed spots often has two equally-large subtrees:
SUBTREE POPULATION
1 976
2 972
3 10
4 7
5 6
Remedy: take the reduced cell (as found below the table), but double the 34.71, and stick it into the line
UNIT_CELL_CONSTANTS=69.42 84.72 213.67 90.35 90.27 90.06
in XDS.INP. To make XDS actually use that line, specify
SPACE_GROUP_NUMBER=1
and leave the space group determination for later.
Alternatively, try several well-separated SPOT_RANGEs instead of 1. For example, instead of
SPOT_RANGE=1 900
try e.g.
SPOT_RANGE=1 100 SPOT_RANGE=801 900
In the cases where I've tried this, this splitting of SPOT_RANGE into several wedges always resulted in IDXREF picking up the correct cell.
The reason for the indexing failure in these cases seems to be the fact that the crystal changed its orientation within the SPOT_RANGE by more than ~ 0.1°.
IDXREF produces too long axes
This may (rarely) happen when MAXIMUM_NUMBER_OF_JOBS is > 1. In this case, different JOBS' COLSPOT runs may report some reflections twice in SPOT.XDS. Since their phi values are close, they correspond to long unit cell parameters. This happens more easily if the mosaicity is high and therefore reflections extend over the borders between JOBs.
This effect may be mitigated by having as many SPOT_RANGEs as JOBs, and leaving gaps between the SPOT_RANGEs.
IDXREF prints !!! WARNING !!! message
If you see
!!! WARNING !!! REFINEMENT DID NOT CONVERGE
LAST CORRECTION SHIFT WAS 6.1E-02 (should be < 1.0E-03)
then try the following: find the line
REFINE(IDXREF)= ...
in XDS.INP. If it is commented out with a !, remove the !. Then, change it to have
REFINE(IDXREF)=CELL BEAM ORIENTATION AXIS ! DISTANCE
i.e. remove DISTANCE from the list of refinable parameters. Once that is done, save XDS.INP and re-run the IDXREF step.
This problem occurs if the DISTANCE is large, and XDS cannot refine it meaningfully.
If this does not help, try to refine even less items, e.g. leave out AXIS.
IDXREF.LP does not show the expected lattice
By specifying the spacegroup and unitcell, you tell XDS that it should index based on those reflections that match that spacegroup and unitcell. In a sense, you _force_ that spacegroup and cell. This will discard other (spurious?) reflections, and usually leads to a clean list of Bravais lattice possibilities.
If the data reduction fails nevertheless (in terms of bad R-factors and ISa in the CORRECT step), then chances are that you specified some parameter wrongly, or not accurate enough (ORGX and ORGY are the most likely candidates), or that the crystal does not match your idea about its spacegroup and unit cell. Actually the latter happens pretty frequently (which is why it is always the safest way to collect 180° of spindle rotation unless you know your crystals very well; this is also a good strategy in other respects).
If you tell XDS that you do not to know the spacegroup (SPACE_GROUP_NUMBER= 0), then IDXREF takes all observed reflections into account. By design, the spacegroup decision is then postponed until the CORRECT step, or rather to a run of pointless after CORRECT, and it may be not vital to closely inspect IDXREF.LP, because CORRECT.LP has basically the same information, plus more.
none of the lattices in IDXREF.LP (except aP) has a good QUALITY OF FIT
It is a good idea to use many frames for the SPOT_RANGE, (e.g. the first half of the DATA_RANGE, as does the generate_XDS.INP script) for the COLSPOT and IDXREF steps.
It is entirely possible to run COLSPOT for the complete DATA_RANGE (in that case the parameters of SPOT_RANGE are the same as for the DATA_RANGE), and to try smaller SPOT_RANGEs in IDXREF - this means COLSPOT has to be run only once, and the JOBS= line has e.g.
JOBS= IDXREF
only. Maybe some SPOT_RANGE sub-range gives a clear answer. In that case it may be useful to use
REFINE(INTEGRATE)=! AXIS BEAM ORIENTATION CELL DISTANCE
because otherwise the spurious reflections in the other ranges will probably disturb the on-the-fly refinement of parameters.
I've seen datasets where each reflection had a satellite associated with the main reflection, but separate from it. This is well visible in FRAME.cbf written by COLSPOT. In such a case it helps to use
MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT= 12 ! re-run COLSPOT after changing the parameter !
thus doubling (w.r.t. the default of 6) the required spotsize that makes a reflection be used for indexing. In the latest case this made it possible to index cleanly on a single frame (which is actually not uncommon).
A final possibility: your crystal may really be triclinic - hopefully you collected 180° of data, or even a bit more than that.
Integration failure
If INTEGRATE stops after e.g.
******************************************************************************
PROCESSING OF IMAGES 136 ... 140
******************************************************************************
with
!!! ERROR !!! AUTOMATIC DETERMINATION OF SPOT SIZE PARAMETERS HAS FAILED.
YOU MAY RESTART THIS STEP AFTER SPECIFYING VALUES IN XDS.INP FOR:
REFLECTING_RANGE= REFLECTING_RANGE_E.S.D.=
BEAM_DIVERGENCE= BEAM_DIVERGENCE_E.S.D.=
then you should reduce the upper limit of the DATA_RANGE, to stop before the problematic frames, and re-run INTEGRATE. In this example, you would modify XDS.INP to have
DATA_RANGE=1 135 JOB=INTEGRATE CORRECT
Save XDS.INP, run XDS and inspect INTEGRATE.LP, to find the lines (e.g.)
BEAM_DIVERGENCE= 0.478 BEAM_DIVERGENCE_E.S.D.= 0.048 REFLECTING_RANGE= 1.100 REFLECTING_RANGE_E.S.D.= 0.157
Copy them to XDS.INP. Restore the original DATA_RANGE and continue.
Another error mode of INTEGRATE (in processing of small-molecule data) is ...
IMAGE IER SCALE NBKG NOVL NEWALD NSTRONG NREJ SIGMAB SIGMAR
5 0 0.975 3968463 1 5 1 0 0.1028 1.3486
6 0 0.967 3969949 2 2 2 0 0.1688 0.0000
7 0 0.929 3975306 0 3 0 0 0.0000 0.0000
!!! ERROR !!! CANNOT ALLOCATE MEMORY
YOU COULD RERUN THIS STEP WITH SMALLER VALUES FOR THE PARAMETERS
NUMBER_OF_PROFILE_GRID_POINTS_ALONG_ALPHA/BETA=
NUMBER_OF_PROFILE_GRID_POINTS_ALONG_GAMMA=
The error message is misleading in this case: there are too few reflections to build the average profile. The fix is: restart INTEGRATE after inserting e.g.
DELPHI=20 ! default is 5, so try with e.g. 10, 20, 45, 90, 180
and re-run INTEGRATE.