Difficult datasets

The following suggestions apply not only to small-molecule datasets, but also to very weak or low-resolution, or otherwise difficult macromolecular datasets.

These datasets have few (or few strong, or few well-defined) reflections per frame. Therefore, the multitude of parameters describing the diffraction experiment needs to be reduced (in refinement one would say: to avoid overfitting). This means that some parameters may need adjustment.

optimizing the parameters for COLSPOT

IDXREF depends on the reflection positions stored in SPOT.XDS . This file is written by COLSPOT, and its contents is visualized in XDSGUI's IDXREF tab (or e.g. with GNUPLOT: plot "SPOT.XDS" using 1:2).

Obviously, only if the true reflection pattern is represented well by SPOT.XDS can IDXREF be successful. So, play with the following parameters, and use XDSGUI to find their influence:

STRONG_PIXEL=
MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT=

It is important to correctly mask any shadowed areas on the detector; if not done properly, the variation at the edges will lead to spurious "spots" being picked up by COLSPOT.

optimizing the parameters for IDXREF

For low-resolution (macromolecular) data, don't refine POSITION (i.e. detector distance) in REFINE(IDXREF) ! This is also what generate_XDS.INP defaults to, and produces very stable indexing - however, if the distance is not accurate, one needs a second (optimisation) pass.

For close reflections, you may want to reduce SEPMIN below its default of 6, and set CLUSTER_RADIUS to half of the new value.

For high-resolution (e.g. small molecule) data, do refine POSITION (i.e. detector distance) in REFINE(IDXREF). So you must change XDS.INP w.r.t. what generate_XDS.INP produces. For such datasets one also wants to set

TEST_RESOLUTION_RANGE=10. 1.

optimizing the parameters for DEFPIX

For high-resolution data, one probably wants to adjust

VALUE_RANGE_FOR_TRUSTED_DETECTOR_PIXELS= 3000 30000  ! this is used by DEFPIX

because the variation in background is usually large. Better to mask with

UNTRUSTED_RECTANGLE=
UNTRUSTED_QUADRILATERAL=
UNTRUSTED_ELLIPSE=

which is easily done using XDSGUI.

optimizing the parameters for INTEGRATE

DELPHI=45 ! (or some multiple of 15; the default is 5) to base reflection profiles and refinements on more reflections - try this first if you get error messages about incomplete profiles or failure to allocate memory in the INTEGRATE step, or if the geometric parameters during INTEGRATE refinements "run away"
REFINE(INTEGRATE)= ! do not refine anything in INTEGRATE. This will be very fast and very stable. Very good results are usually obtained with REFINE(INTEGRATE)=POSITION ! maybe add ORIENTATION BEAM; AXIS and CELL should probably not be refined.
be sure to recycle GXPARM.XDS to XPARM.XDS.

Furthermore, you may grab the lines e.g.

BEAM_DIVERGENCE=   2.067  BEAM_DIVERGENCE_E.S.D.=   0.207
REFLECTING_RANGE=  2.303  REFLECTING_RANGE_E.S.D.=  0.329

from INTEGRATE.LP, and insert them into XDS.INP . The latter is accomplished easily with

grep _E INTEGRATE.LP | tail -2 > x
grep -v _E.S.D XDS.INP >> x
mv x XDS.INP

or similar. These values then serve as a Bayesian prior for the values that will actually be used for integrating.

If you used larger OSCILLATION_RANGE than 1°, you should increase MAXIMUM_ERROR_OF_SPINDLE_POSITION= (default is 2°) because you expect larger deviations of experimentally determined and calculated phi positions of reflections. Same might apply to MAXIMUM_ERROR_OF_SPOT_POSITION= ; the default is 3 pixels which may be too low to account for broad reflections. If you don't do this, many reflections that violate these limits will not be used for geometry refinement, which may turn out to be unstable.

If your crystal diffracts to high angles (e.g. small molecules) then the absorption in the detector surface affects both position and intensity of the recorded x-rays. XDS has two parameters, SENSOR_THICKNESS and SILICON, to take care of that. If the data are measured with a Pilatus or Eiger detector, the defaults are ok. If however a CCD is used then both SENSOR_THICKNESS and SILICON should be adjusted manually - see SILICON.

It is important to inspect FRAME.cbf after the XDS run. This will reveal problems e.g. with masking of shadows and additional lattices.

optimizing the parameters for CORRECT

NBATCH= (total angular rotation range)/45 ! to reduce the number of scale factors; the value of NBATCH is shown as NXBIN in CORRECT.LP for the ABSORB and DECAY correction; default is (total angular rotation range)/5. Other values, like (total rotation range)/60 or (total rotation range)/90 might also be tried. Try to match NBATCH with DELPHI (see above).
maybe CORRECTIONS= ABSORB ! don't try to correct for MODULATION and DECAY in scaling because these may be poorly determined; however in general it helps to correct for them
FRIEDEL'S_LAW=TRUE unless there is a known anomalous scatterer. A clear sign that FRIEDEL'S_LAW=TRUE is required is that the CHI^2-VALUE OF FIT OF CORRECTION FACTORS is significantly below 1 (e.g. 0.5). If in doubt use the setting that maximizes ISa.

optimizing the parameters for XSCALE

If you use XSCALE to merge several XDS datasets, you should make sure that you specify in XSCALE.INP the same values for NBATCH and CORRECTIONS as you do, in XDS.INP, for the CORRECT step.