Xdscc12: Difference between revisions

From XDSwiki
Jump to navigation Jump to search
No edit summary
 
(17 intermediate revisions by 2 users not shown)
Line 1: Line 1:
XDSCC12 is a program for generating [[CC1/2|delta-CC<sub>1/2</sub>]] and delta-CC<sub>1/2</sub>ano values for XDS_ASCII.HKL (written by [[XDS]]), or for XSCALE.HKL containing several files of type XDS_ASCII.HKL after scaling in [[XSCALE]] (with MERGE=FALSE).


It implements the method described in Assmann, Brehm and Diederichs (2016) Identification of rogue datasets in serial crystallography. J. Appl. Cryst. 49, 1021 [http://journals.iucr.org/j/issues/2016/03/00/zw5005/zw5005.pdf], and it does this not only for the individual datasets in XSCALE.HKL, but also for individual frames, or groups of frames, of a single dataset collected with the rotation method and processed by [[XDS]].
XDSCC12 is a program for generating [[CC1/2|delta-CC<sub>1/2</sub>]] and delta-CC<sub>1/2-ano</sub> values for XDS_ASCII.HKL (written by [[XDS]]), or for XSCALE.HKL (written by [[XSCALE]]) containing data from several files of type XDS_ASCII.HKL after scaling (with MERGE=FALSE).  


The program can be downloaded for [ftp://turn5.biologie.uni-konstanz.de/pub/linux_bin/xdscc12 Linux] or [ftp://turn5.biologie.uni-konstanz.de/pub/mac_bin/xdscc12 Mac].
It implements the method described in Assmann, Brehm and Diederichs (2016) Identification of rogue datasets in serial crystallography. J. Appl. Cryst. 49, 1021 [http://journals.iucr.org/j/issues/2016/03/00/zw5005/zw5005.pdf], and it does this not only for the individual datasets in XSCALE.HKL, but also for individual frames, or groups (batches) of frames, of a single dataset collected with the rotation method and processed by [[XDS]].
 
The program can be downloaded for [https://{{SERVERNAME}}/pub/linux_bin/xdscc12 Linux] or [https://{{SERVERNAME}}/pub/mac_bin/xdscc12 Mac].


Usage (this text can be obtained with <code>xdscc12 -h</code>):
Usage (this text can be obtained with <code>xdscc12 -h</code>):
<pre>
<pre>
xdscc12 KD 2019-04-30. Academic use only; no redistribution. -h option shows options.
xdscc12 KD 2023-01-08. Academic use only; no redistribution. -h option shows options.
Please cite Assmann, G., Brehm, W., Diederichs, K. (2016) J.Appl.Cryst. 49, 1021-1028
Please cite Assmann, G., Brehm, W., Diederichs, K. (2016) J.Appl.Cryst. 49, 1021-1028
running 'xdscc12 -h' on 20190502 at 16:11:46 +0200
running 'xdscc12' on 20240403 at 16:49:03 +0200
usage: xdscc12 [-dmin <lowres>] [-dmax <highres>] [-nbin <nbin>] [-mode <1 or 2>] [-<abcdefstwz>] [-r <ref>] FILE_NAME
usage: xdscc12 [-dmin <lowres>] [-dmax <highres>] [-nbin <nbin>] [-mode <1 or 2>] [-<abcdefstwz>] [-r|-R <ref>] FILE_NAME
dmin (default 999A), dmax (default 1A) and nbin (default 10) have the usual meanings.
dmin (default 999A), dmax (default 1A) and nbin (default 10) have the usual meanings.
mode can be 1 (equal volumes of resolution shells) or 2 (increasing volumes; default).
mode can be 1 (equal volumes of resolution shells) or 2 (increasing volumes; default).
   -r: next parameter: ASCII reference file with lines: h,k,l,Fcalc or h,k,l,Fcalc+,Fcalc-
   -r: <ref> is ASCII reference file with lines: h,k,l,Fref or h,k,l,Fref+,Fref-
       this allows calculation of CC of isomorphous signal with reference
  -R: <ref> is ASCII reference file with lines: h,k,l,Iref or h,k,l,Iref+,Iref-
   -s: read two columns from reference: Fcalc(+), Fcalc(-).  
       -r and -R allow calculation of CC of isomorphous signal with reference
   -s: read two columns from reference: Fref+, Fref- or Iref+, Iref-.  
       this allows calculation of CC of anomalous signal with that of reference
       this allows calculation of CC of anomalous signal with that of reference
  -A: sort INPUT_FILEs in XSCALE.INP.rename_me by anomalous instead of isomorphous delta-CC1/2
   -t: total oscillation (degree) to batch fine-sliced frames into
   -t: total oscillation (degree) to batch fine-sliced frames into
  FILE_NAME can be XDS or XSCALE reflection file
  FILE_NAME can be XDS or XSCALE reflection file
  other options can be combined (e.g. -def), and switch the following off:
  other options can be combined (e.g. -def), and switch the following OFF:
   -a: individual isomorphous summary values
   -a: individual isomorphous summary values
   -b: individual (Fisher-transformed) delta-CC1/2 values
   -b: individual (Fisher-transformed) delta-CC1/2 values
Line 27: Line 30:
   -f: individual delta-CC1/2ano reflection numbers
   -f: individual delta-CC1/2ano reflection numbers
   -w: weighting of intensities with their sigmas
   -w: weighting of intensities with their sigmas
   -z: no Fisher transformation of delta-CC1/2 values
   -z: Fisher transformation of delta-CC1/2 values
The program writes a commented XSCALE.INP.rename_me that is sorted on delta-CC1/2
</pre>
</pre>


The program output is terse but supposed to be self-explanatory. The isomorphous delta-CC<sub>1/2</sub> of a batch of frames (width chosen with the -t option) relative to all data is most easily visualized via [[XDSGUI]] (Statistics tab); the anomalous delta-CC<sub>1/2</sub> may be plotted with e.g. gnuplot after grepping the relevant lines from the output.
== Example output ==
<pre>
xdscc12 KD 2020-12-9. Academic use only; no redistribution. -h option shows options.
Please cite Assmann, G., Brehm, W., Diederichs, K. (2016) J.Appl.Cryst. 49, 1021-1028
  running 'xdscc12 temp.ahkl' on 20220413 at 12:13:53 +0200
  no option -w found, therefore statistics are weighted by sigma values
  no option -z found, therefore delta-CC1/2 values are Fisher-transformed
  reflection file is temp.ahkl
!SPACE_GROUP_NUMBER=   19
!UNIT_CELL_CONSTANTS=     38.30     79.10     79.10  90.000  90.000  90.000
  # of datasets=          20
  # obs (w/o misfits), unique, misfits =       51918       20213           0
  max and min resolution of data in file =   39.55000       1.801329   
  data between   39.55000      and   1.801329     A will be used
  10 resolution shells (for lines starting with b,c,e,f,r,s):
  5.644  4.011  3.281  2.844  2.545  2.324  2.152  2.013  1.899  1.801
 
  overall CC1/2:    83.328 nref=   14986 (but the overall CC1/2 is meaningless!)
  <CC1/2>:    44.468 (frequency-weighted average of CC1/2 in resolution shells)
  CC1/2 in resolution shells:
   91.4   81.0   72.0   68.4   42.1   41.4   32.4   29.0   33.2   28.2
  CC* in resolution shells:
   97.7   94.6   91.5   90.1   77.0   76.5   70.0   67.0   70.6   66.3
  frequency, i.e. number of unique reflections in resolution shells:
    493    887   1134   1345   1501   1649   1808   1921   2061   2187
  headings for lines starting with a,b,c:
a:  <CC1/2> of each dataset:
a:   reflections of this dataset only      reflections of all datasets
a: set   nref    with without   delta     nref    with without   delta
b: delta-CC1/2 in resolution shells
c: # reflections for delta-CC1/2
a      1   1241  61.936  42.634  26.227    14470  44.075  42.523   1.910
b 61.432 27.680 8.094 2.942 41.231 44.296 26.090 10.996 30.482 30.456
c     47     73     85    110    135    137    146    170    164    174
a      2   1565  43.941  34.156  11.512    14350  44.978  44.104   1.091
b 36.910 -5.982 -1.113 58.592 46.243 -24.157 -.234 20.068 11.307 -3.709
c     49     90    126    134    158    168    196    203    216    225
a      3   1754  39.551  29.634  11.233    14478  43.683  42.374   1.606
b 13.559 11.752 17.587 17.093 16.730 19.779 3.638 20.348 6.182 4.113
c     43    107    131    160    175    198    218    230    246    246
a      4   1468  41.694  25.840  17.768    14382  44.801  43.256   1.917
b -9.445 36.760 36.335 15.954 2.951 20.264 21.375 19.554 2.513 35.809
c     30     93     99    115    164    174    202    220    188    183
a      5   1412  41.785  39.193   3.100    14292  45.561  45.291   0.340
b 21.421 1.775 7.645 1.361 6.005 22.513 11.859 -16.883 .973 -2.353
c     41     68     82    111    144    173    176    190    210    217
a      6   1363  49.626  42.827   8.634    14293  44.938  44.354   0.728
b 32.175 53.390 25.822 21.623 -42.441 27.299 7.214 12.080 6.573 15.115
c     43     84     99    120    147    161    173    183    166    187
a      7   1686  48.062  37.817  12.521    14407  43.991  42.971   1.258
b 11.830 -4.851 11.505 15.574 67.688 20.009 5.129 -7.014 8.766 1.024
c     59    117    133    153    142    179    183    243    252    225
a      8   1795  46.357  34.049  14.614    14433  44.757  43.282   1.829
b 14.234 6.729 13.626 19.758 7.065 16.373 21.204 11.431 18.082 15.739
c     57    103    137    183    168    202    205    240    237    263
a      9   1483  50.778  46.558   5.526    14363  44.923  44.479   0.554
b 40.845 .431 30.046 -7.049 -1.607 -2.724 -9.486 -.902 20.019 27.538
c     46     88    119    135    154    161    193    192    203    192
a     10   1332  38.220  34.823   3.918    14506  43.988  43.550   0.541
b 33.078 34.752 9.425 25.010 -2.922 -14.118 14.600 12.844 1.512 -8.735
c     51     77     93    118    123    154    166    172    181    197
a     11   1477  45.163  38.543   8.015    14361  45.415  44.577   1.051
b 9.914 8.880 2.467 18.150 5.381 11.874 4.743 29.698 14.834 -17.211
c     65     82    111    133    115    159    174    202    220    216
a     12   1654  45.251  30.457  17.159    14409  44.363  42.437   2.373
b 28.681 6.430 -40.284 36.914 30.008 47.293 29.607 9.512 -1.455 21.488
c     53    109    116    142    154    177    196    229    235    243
a     13   1422  51.281  36.632  18.036    14302  44.261  42.742   1.872
b 23.460 4.559 43.871 13.497 28.091 35.298 -1.493 17.764 17.226 19.655
c     38     73    101    123    153    153    169    192    224    196
a     14   1668  51.749  49.607   2.882    14441  44.642  44.527   0.145
b 5.096 37.325 5.515 16.257 21.437 -7.901 14.955 4.960 -15.485 -2.206
c     52    115    121    161    172    177    201    205    223    241
a     15   1369  46.667  38.742   9.674    14263  44.480  43.770   0.882
b -22.140 29.901 48.902 -7.002 26.620 .483 8.071 13.097 11.535 -5.181
c     56     92    105    115    116    138    151    177    196    223
a     16   1257  33.100  23.275  10.645    14291  44.849  43.826   1.273
b -14.293 -.418 44.139 -1.828 30.203 21.053 5.098 22.834 -4.129 -.948
c     34     60     57    120    133    127    159    188    199    180
a     17   1370  42.421  39.259   3.794    14264  44.877  44.550   0.409
b 30.851 -17.854 11.508 12.578 -18.416 2.277 20.934 12.146 -.761 .881
c     33     72    106    131    134    151    157    189    197    200
a     18   1269  45.248  26.342  21.465    14302  43.295  41.438   2.264
b 23.189 2.571 31.364 37.883 60.830 25.784 13.345 33.459 -3.441 5.251
c     45    101     97    111    127    143    144    164    168    169
a     19   1644  43.200  43.114   0.105    14365  45.169  45.113   0.071
b 23.714 13.389 29.752 38.784 -16.069 6.952 10.177 9.560 -19.196 -16.013
c     56     87    132    138    168    179    183    215    236    250
a     20   1615  54.027  42.202  15.317    14366  45.818  44.496   1.661
b 20.068 20.724 30.458 21.475 33.352 28.157 5.646 16.916 9.398 8.795
c     52     87    131    149    156    181    195    200    230    234
  -------------------------------------------------------------
 
overall CC1/2ano:   -25.548 nref=    2668 (but the overall value is meaningless!)
<CC1/2ano> :   -21.370 (frequency-weighted average of CC1/2ano in resolution shells)
CC1/2ano in resolution shells:
  -75.0   -1.7  -30.1  -41.3  -38.8  -20.1  -18.2   -8.6  -17.7   -9.6
frequency i.e. number of unique Friedel pairs:
     73    140    168    241    262    320    319    351    397    397
 
  headings for lines starting with d,e,f:
d: <CC1/2ano> of each dataset:
d: reflections of this dataset only      reflections of all datasets
d: set   nref    with without   delta     nref    with without   delta
e: delta-CC1/2ano in resolution shells
f: # reflections for delta-CC1/2ano
d     1    190 -30.590 -20.348 -10.922     2233 -17.969 -17.817  -0.157
e -42.962 -6.341 4.502 -41.515 -76.721 -36.825 43.947 19.535 21.803 -23.853
f     11     12      9     14     17     28     26     22     26     25
d     2    234 -16.480 -16.515   0.036     2336 -26.125 -26.132   0.008
e -3.116 -2.078 -3.249 -23.820 6.424 9.198 -3.791 1.123 6.966 -4.309
f      5     12     16     12     35     30     40     24     29     31
d     3    343 -40.435 -31.820  -9.887     2245 -22.659 -19.640  -3.160
e -1.713 12.234 -11.942 -63.179 5.125 -4.423 15.291 -30.082 .859 3.714
f      7     16     27     33     42     45     38     54     43     38
d     4    250 -32.340 -29.274  -3.387     2385 -19.104 -18.426  -0.702
e 22.408 37.540 12.328 -13.456 -2.729 1.911 -36.173 3.026 -6.757 12.916
f      3     13     12     25     39     42     41     30     23     22
d     5    179 -22.995 -24.947   2.070     2417 -26.768 -27.230   0.499
e 36.142 13.527 -10.456 -7.251 13.331 -.923 27.239 -4.208 -9.635 -10.625
f      5      4      7      6     19     22     29     30     23     34
d     6    188 -10.090 -13.603   3.562     2412 -21.564 -21.759   0.205
e -43.278 1.987 48.529 -2.536 35.294 -14.994 21.066 -6.607 -.446 -8.161
f      5      9      8     10     21     28     25     30     27     25
d     7    292 -34.206 -35.887   1.915     2316 -26.435 -26.723   0.309
e -19.814 34.972 .715 48.559 -11.683 -4.391 -14.145 5.337 -3.722 2.527
f     12     15     16     23     31     43     28     46     45     33
d     8    305  -6.575  -7.745   1.176     2353 -19.132 -19.241   0.113
e -7.804 -5.876 -10.001 4.878 -37.785 8.976 14.698 -4.716 9.080 11.786
f     11     17     17     26     28     45     34     38     42     47
d     9    283 -35.313 -32.348  -3.347     2398 -20.019 -19.463  -0.578
e -90.495 31.505 6.592 -45.999 -7.955 -11.621 6.013 -6.399 18.725 2.816
f      7     16     14     23     31     32     39     37     52     32
d    10    228 -31.230 -36.231   5.639     2184 -20.991 -21.921   0.975
e -10.975 26.119 -31.629 9.944 10.170 24.916 -19.125 25.874 -12.123 -.504
f      6     23     12     16     17     32     31     33     27     31
d    11    204 -25.646 -21.039  -4.870     2344 -26.292 -25.997  -0.317
e 8.152 1.052 -8.483 4.821 5.281 -.012 -33.636 6.064 -12.304 -7.890
f     13     11     16     14     19     25     24     28     26     28
d    12    314 -20.795 -27.734   7.363     2299 -21.005 -22.984   2.080
e -53.587 19.115 79.436 41.903 -3.288 4.497 22.135 12.845 -21.183 -1.646
f      4     15     16     27     34     37     34     46     54     47
d    13    233  -6.551  -6.622   0.071     2411 -21.747 -22.084   0.354
e 7.268 -17.245 31.471 30.207 .209 7.647 5.126 -61.071 16.081 14.792
f      4     11     15     20     20     34     38     34     27     30
d    14    321 -39.745 -43.950   5.095     2335 -21.004 -21.556   0.578
e 6.661 20.182 -13.578 -14.964 .663 20.951 -5.357 11.106 6.979 4.542
f     13     26     24     29     37     30     31     33     48     50
d    15    211 -25.574 -27.428   1.993     2451 -21.107 -22.106   1.049
e 9.034 24.046 51.709 10.903 25.126 -31.105 -39.654 -43.178 6.990 15.268
f      9     18     23     17     21     28     15     22     28     30
d    16    164 -28.372 -24.930  -3.704     2381 -18.839 -18.689  -0.156
e 20.336 23.610 -96.956 -10.020 7.356 1.404 14.821 -20.025 -4.886 -17.264
f      2      6      2      9     17     20     33     24     20     31
d    17    218 -13.441  -3.184 -10.302     2460 -20.794 -19.807  -1.029
e -13.966 -12.793 5.985 10.830 -5.428 -23.834 17.910 -9.981 -25.620 -22.550
f      2     13     11     19     16     22     26     38     34     37
d    18    136 -32.513 -31.586  -1.034     2379 -21.312 -21.014  -0.312
e 55.859 17.851 -8.273 -57.797 -68.191 27.216 15.219 15.258 -14.424 -10.376
f      4     14     12     16     14     19     18     13     16     10
d    19    295 -26.211 -37.786  12.848     2289 -19.913 -21.583   1.745
e -3.719 -15.248 39.777 -13.377 11.415 8.559 -15.482 8.791 25.058 49.701
f      8     13     17     33     30     35     40     38     38     43
d    20    309 -33.133 -33.400   0.300     2351 -21.683 -22.124   0.463
e -30.012 13.423 -2.376 15.330 -16.239 -2.850 -9.170 5.549 10.005 1.910
f      9      9     15     32     33     36     40     48     48     39
 
  best delta-CC1/2_only=   26.22747   
  median of delta-cc1/2 ("only" i.e. 6th col of "a" lines) =   10.93907   
  noise= (MAD, median absolute deviation) from this median =   5.816339   
  median of delta-cc1/2 ("all" i.e. 10th col of "a" lines) =   1.174427   
  noise= (MAD, median absolute deviation) from this median =  0.6438574   
  median of delta-cc1/2-ano ("only" i.e. 6th col of "d" lines) =  0.1854570   
  noise= (MAD, median absolute deviation) from this median =   3.552712   
  median of delta-cc1/2-ano ("all" i.e. 10th col of "d" lines) =  0.1588948   
  noise= (MAD, median absolute deviation) from this median =  0.4448406   
 
Wrote a commented XSCALE.INP.rename_me that is sorted on delta-CC1/2 "only"
You may edit that file, or e.g. add lines after each INPUT_FILE line with sed '/INPUT_FILE/a INCLUDE_RESOLUTION_RANGE=99 3'
normal termination
</pre>
and the resulting file XSCALE.INP.rename_me is:
<pre>
SPACE_GROUP_NUMBER=  19
UNIT_CELL_CONSTANTS=    38.30    79.10    79.10  90.000  90.000  90.000
OUTPUT_FILE= temp.ahkl
PRINT_CORRELATIONS= FALSE
SAVE_CORRECTION_IMAGES= FALSE
FRIEDEL'S_LAW= FALSE
! median of delta-cc1/2 "only" values=    10.939
! noise (MAD) of these values=    5.816
! median of delta-cc1/2 "all" values=    1.174
! noise (MAD) of these values=    0.644
! median of delta-cc1/2-ano "only" values=    0.185
! noise (MAD) of these values=    3.553
! median of delta-cc1/2-ano "all" values=    0.159
! noise (MAD) of these values=    0.445
! input files sorted by deltacc12_only (highest first):
! deltacc12 only / all:  26.2275    1.9103 deltacc12-ano only /all:  -10.9223  -0.1569
INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0001/xds/XDS_ASCII.HKL
! deltacc12 only / all:  21.4650    2.2639 deltacc12-ano only /all:  -1.0339  -0.3117
INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0018/xds/XDS_ASCII.HKL
! deltacc12 only / all:  18.0364    1.8724 deltacc12-ano only /all:    0.0708    0.3545
INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0013/xds/XDS_ASCII.HKL
! deltacc12 only / all:  17.7684    1.9166 deltacc12-ano only /all:  -3.3870  -0.7021
INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0004/xds/XDS_ASCII.HKL
! deltacc12 only / all:  17.1589    2.3726 deltacc12-ano only /all:    7.3634    2.0795
INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0012/xds/XDS_ASCII.HKL
! deltacc12 only / all:  15.3174    1.6608 deltacc12-ano only /all:    0.3001    0.4632
INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0020/xds/XDS_ASCII.HKL
! deltacc12 only / all:  14.6140    1.8288 deltacc12-ano only /all:    1.1762    0.1129
INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0008/xds/XDS_ASCII.HKL
! deltacc12 only / all:  12.5208    1.2582 deltacc12-ano only /all:    1.9155    0.3089
INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0007/xds/XDS_ASCII.HKL
! deltacc12 only / all:  11.5122    1.0907 deltacc12-ano only /all:    0.0360    0.0079
INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0002/xds/XDS_ASCII.HKL
! deltacc12 only / all:  11.2330    1.6064 deltacc12-ano only /all:  -9.8867  -3.1596
INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0003/xds/XDS_ASCII.HKL
! deltacc12 only / all:  10.6452    1.2727 deltacc12-ano only /all:  -3.7041  -0.1557
INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0016/xds/XDS_ASCII.HKL
! deltacc12 only / all:    9.6737    0.8816 deltacc12-ano only /all:    1.9932    1.0487
INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0015/xds/XDS_ASCII.HKL
! deltacc12 only / all:    8.6341    0.7283 deltacc12-ano only /all:    3.5620    0.2049
INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0006/xds/XDS_ASCII.HKL
! deltacc12 only / all:    8.0151    1.0507 deltacc12-ano only /all:  -4.8702  -0.3171
INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0011/xds/XDS_ASCII.HKL
! deltacc12 only / all:    5.5262    0.5542 deltacc12-ano only /all:  -3.3475  -0.5778
INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0009/xds/XDS_ASCII.HKL
! deltacc12 only / all:    3.9182    0.5411 deltacc12-ano only /all:    5.6393    0.9750
INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0010/xds/XDS_ASCII.HKL
! deltacc12 only / all:    3.7935    0.4089 deltacc12-ano only /all:  -10.3019  -1.0286
INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0017/xds/XDS_ASCII.HKL
! deltacc12 only / all:    3.0997    0.3399 deltacc12-ano only /all:    2.0699    0.4987
INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0005/xds/XDS_ASCII.HKL
! deltacc12 only / all:    2.8817    0.1446 deltacc12-ano only /all:    5.0954    0.5780
INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0014/xds/XDS_ASCII.HKL
! deltacc12 only / all:    0.1054    0.0707 deltacc12-ano only /all:  12.8477    1.7449
INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0019/xds/XDS_ASCII.HKL
</pre>


For multiple datasets, the output lines show the contribution of each dataset toward the total CC<sub>1/2</sub>. Negative numbers indicate a worsening of the overall signal.
== Explanation of output ==
The program output in the terminal window is terse but supposed to be self-explanatory; it can (and most often should) be saved or re-directed to a file.
xdscc12 FILE_FROM_XSCALE > xdscc12.log  #  or xdscc12 FILE_FROM_XSCALE | tee xdscc12.log
All statistics (tables) produced by XDSCC12 may be visualized with e.g. gnuplot, after grepping the relevant lines from the output.
If XDSCC12 is used with a XDS_ASCII.HKL reflection file (from XDS), the isomorphous delta-CC<sub>1/2</sub> of a batch of frames (width chosen with the <code>-t</code> option; typically <code>-t 1</code>) relative to all data is most easily visualized via [[XDSGUI]] (Statistics tab). Negative numbers indicate a worsening of the overall signal.


Statistics are given (in resolution shells) for the isomorphous and the anomalous signal.
If XDSCC12 is used with a XSCALE.HKL generated from multiple datasets, the output lines show the contribution of each dataset toward the total CC<sub>1/2</sub>. In this case, the program writes a file called XSCALE.INP.rename_me which shows statistics of delta-CC<sub>1/2</sub> and delta-CC<sub>1/2-ano</sub> values, and has a sorted enumeration of the INPUT_FILEs - the first of these provides the best data set, and the last one is the worst one. This XSCALE.INP.rename_me can then be edited (i.e. for deleting a few data sets with strongly negative delta-CC<sub>1/2</sub>), and renamed to XSCALE.INP . Only delete the clearly worst data sets, and not more than 10% of the existing ones! This procedure can be iterated, i.e. after another round of XSCALE, XDSCC12 could be run again.


To find outlier data sets in [[XSCALE]]d data, you may try the -w option. This can be helpful since the ''a'' and ''b'' parameters of the error model are adjusted by XSCALE such that the sigmas are very high, which may reduce the delta-CC<sub>1/2</sub> signal.
Overall statistics are reported in the lines starting with <code>a</code> and <code>d</code> for
* <b>only</b> those unique reflections that are actually present in the batch of frame / batch / dataset. These values are in columns 3-6.
* <b>all</b> unique reflections of the merged dataset (but a frame / batch / dataset may not have all unique reflections, so the "all" values report the mean influence). These values are in columns 7-10.
Typically, it is sensible to disregard the "all" values, and to base decisions on the "only" values, because the latter are not affected by the number of reflections of the particular frame / batch / dataset. The words "all" and "only" are used in this sense throughout the terminal and file output of XDSCC12.


A complete description of how to process serial crystallography data with XDS/XSCALE is given in [[SSX]]. A program that implements the method of [https://doi.org/10.1107/S1399004713025431 Brehm and Diederichs (2014)] and theory of [https://doi.org/10.1107/S2059798317000699 Diederichs (2017)] is [[xscale_isocluster]].
Statistics for "only" the unique reflections of a frame / batch/ dataset are given in resolution shells for the isomorphous (in lines starting with <code>b</code> and <code>c</code>) and the anomalous signal (in lines starting with <code>d</code> and <code>e</code>). In case of [[SSX]] data (which have few reflections per data set, compared to complete data sets), we typically use <code>-nbin 1</code> as option, to define only a single resolution shell.


To remove bad frames from a XDS_ASCII.HKL file, you can use the program [[exclude_data_range_from_XDS_ASCII.HKL]], or re-INTEGRATE with the keyword [http://xds.mpimf-heidelberg.mpg.de/html_doc/xds_parameters.html#EXCLUDE_DATA_RANGE= EXCLUDE_DATA_RANGE] in [[XDS.INP]].
To find out about the influence of the ''a'' and ''b'' parameters of the XDS/XSCALE-adjusted error model, you may try the <code>-w</code> option; this assigns the same sigma to all reflections. Likewise, the [https://en.wikipedia.org/wiki/Fisher_transformation Fisher transformation], which serves to make changes in CC<sub>1/2</sub> comparable across resolution ranges, may be switched off for testing purposes, with the <code>-z</code> option.


== Correlation against a reference data set (-r <reference> option) ==
== Correlation against a reference data set (-r <reference> option) ==
Line 46: Line 298:
To prepare a reference data set if the refinement was done with phenix.refine, one could use e.g.
To prepare a reference data set if the refinement was done with phenix.refine, one could use e.g.
<pre>
<pre>
mtz2various hklin 2bn3_refine_001.mtz hklout temp.hkl <<eof
mtz2various hklin 2bn3_refine_001.mtz hklout reference.hkl <<eof
OUTPUT USER *
OUTPUT USER *
LABIN FC=F-model PHIC=PHIF-model
LABIN FC=F-model PHIC=PHIF-model
Line 56: Line 308:
sftools
sftools
read mymodel_001.mtz
read mymodel_001.mtz
write temp.hkl format(3i5,f10.3) col F-model
write reference.hkl format(3i5,f10.3) col F-model
y
y
quit
quit
</pre>
</pre>
For a Refmac-written MTZ file, you would use "col FC_ALL" instead of "col F-model".


=== Reference data with anomalous signal (additional -s option) ===
=== Reference data with anomalous signal (additional -s option) ===
Line 75: Line 328:
</pre>
</pre>
in which case <code>sftools</code> outputs only the acentric reflections - only those have anomalous differences. <code>XDSCC12</code> then has to be run with the <code>-s -r anom-reference.hkl</code> option.
in which case <code>sftools</code> outputs only the acentric reflections - only those have anomalous differences. <code>XDSCC12</code> then has to be run with the <code>-s -r anom-reference.hkl</code> option.
== See also ==
A complete description of how to process serial crystallography data with XDS/XSCALE is given in [[SSX]].
[[xscale_isocluster]] is a program that implements the method of [https://doi.org/10.1107/S1399004713025431 Brehm and Diederichs (2014)] and theory of [https://doi.org/10.1107/S2059798317000699 Diederichs (2017)]. It serves to identify groups of related datasets in a reflection file produced by XSCALE, and should be used before XDSCC12.
To remove bad frames from a XDS_ASCII.HKL file, you can  re-INTEGRATE or just re-CORRECT with the keyword [http://xds.mpimf-heidelberg.mpg.de/html_doc/xds_parameters.html#EXCLUDE_DATA_RANGE= EXCLUDE_DATA_RANGE] in [[XDS.INP]].

Latest revision as of 17:46, 31 October 2024

XDSCC12 is a program for generating delta-CC1/2 and delta-CC1/2-ano values for XDS_ASCII.HKL (written by XDS), or for XSCALE.HKL (written by XSCALE) containing data from several files of type XDS_ASCII.HKL after scaling (with MERGE=FALSE).

It implements the method described in Assmann, Brehm and Diederichs (2016) Identification of rogue datasets in serial crystallography. J. Appl. Cryst. 49, 1021 [1], and it does this not only for the individual datasets in XSCALE.HKL, but also for individual frames, or groups (batches) of frames, of a single dataset collected with the rotation method and processed by XDS.

The program can be downloaded for Linux or Mac.

Usage (this text can be obtained with xdscc12 -h): 

xdscc12 KD 2023-01-08. Academic use only; no redistribution. -h option shows options.
Please cite Assmann, G., Brehm, W., Diederichs, K. (2016) J.Appl.Cryst. 49, 1021-1028
 running 'xdscc12' on 20240403 at 16:49:03 +0200
usage: xdscc12 [-dmin <lowres>] [-dmax <highres>] [-nbin <nbin>] [-mode <1 or 2>] [-<abcdefstwz>] [-r|-R <ref>] FILE_NAME
dmin (default 999A), dmax (default 1A) and nbin (default 10) have the usual meanings.
mode can be 1 (equal volumes of resolution shells) or 2 (increasing volumes; default).
  -r: <ref> is ASCII reference file with lines: h,k,l,Fref or h,k,l,Fref+,Fref-
  -R: <ref> is ASCII reference file with lines: h,k,l,Iref or h,k,l,Iref+,Iref-
      -r and -R allow calculation of CC of isomorphous signal with reference
  -s: read two columns from reference: Fref+, Fref- or Iref+, Iref-. 
      this allows calculation of CC of anomalous signal with that of reference
  -A: sort INPUT_FILEs in XSCALE.INP.rename_me by anomalous instead of isomorphous delta-CC1/2
  -t: total oscillation (degree) to batch fine-sliced frames into
 FILE_NAME can be XDS or XSCALE reflection file
 other options can be combined (e.g. -def), and switch the following OFF:
  -a: individual isomorphous summary values
  -b: individual (Fisher-transformed) delta-CC1/2 values
  -c: individual delta-CC1/2 reflection numbers
  -d: individual anomalous summary values
  -e: individual (Fisher-transformed) delta-CC1/2ano values
  -f: individual delta-CC1/2ano reflection numbers
  -w: weighting of intensities with their sigmas
  -z: Fisher transformation of delta-CC1/2 values
 
The program writes a commented XSCALE.INP.rename_me that is sorted on delta-CC1/2

Example output

 xdscc12 KD 2020-12-9. Academic use only; no redistribution. -h option shows options.
 Please cite Assmann, G., Brehm, W., Diederichs, K. (2016) J.Appl.Cryst. 49, 1021-1028
  running 'xdscc12 temp.ahkl' on 20220413 at 12:13:53 +0200
  no option -w found, therefore statistics are weighted by sigma values
  no option -z found, therefore delta-CC1/2 values are Fisher-transformed
 
  reflection file is temp.ahkl
 !SPACE_GROUP_NUMBER=   19
 !UNIT_CELL_CONSTANTS=     38.30     79.10     79.10  90.000  90.000  90.000
  # of datasets=          20
  # obs (w/o misfits), unique, misfits =       51918       20213           0
  max and min resolution of data in file =   39.55000       1.801329    
  data between   39.55000      and   1.801329     A will be used
  10 resolution shells (for lines starting with b,c,e,f,r,s):
   5.644  4.011  3.281  2.844  2.545  2.324  2.152  2.013  1.899  1.801
  
  overall CC1/2:    83.328 nref=   14986 (but the overall CC1/2 is meaningless!)
  <CC1/2>:    44.468 (frequency-weighted average of CC1/2 in resolution shells)
  CC1/2 in resolution shells:
    91.4   81.0   72.0   68.4   42.1   41.4   32.4   29.0   33.2   28.2
  CC* in resolution shells:
    97.7   94.6   91.5   90.1   77.0   76.5   70.0   67.0   70.6   66.3
  frequency, i.e. number of unique reflections in resolution shells:
     493    887   1134   1345   1501   1649   1808   1921   2061   2187
 
  headings for lines starting with a,b,c:
 a:  <CC1/2> of each dataset:
 a:   reflections of this dataset only      reflections of all datasets
 a: set   nref    with without   delta     nref    with without   delta
 b: delta-CC1/2 in resolution shells
 c: # reflections for delta-CC1/2
 a      1   1241  61.936  42.634  26.227    14470  44.075  42.523   1.910
 b 61.432 27.680 8.094 2.942 41.231 44.296 26.090 10.996 30.482 30.456
 c     47     73     85    110    135    137    146    170    164    174
 a      2   1565  43.941  34.156  11.512    14350  44.978  44.104   1.091
 b 36.910 -5.982 -1.113 58.592 46.243 -24.157 -.234 20.068 11.307 -3.709
 c     49     90    126    134    158    168    196    203    216    225
 a      3   1754  39.551  29.634  11.233    14478  43.683  42.374   1.606
 b 13.559 11.752 17.587 17.093 16.730 19.779 3.638 20.348 6.182 4.113
 c     43    107    131    160    175    198    218    230    246    246
 a      4   1468  41.694  25.840  17.768    14382  44.801  43.256   1.917
 b -9.445 36.760 36.335 15.954 2.951 20.264 21.375 19.554 2.513 35.809
 c     30     93     99    115    164    174    202    220    188    183
 a      5   1412  41.785  39.193   3.100    14292  45.561  45.291   0.340
 b 21.421 1.775 7.645 1.361 6.005 22.513 11.859 -16.883 .973 -2.353
 c     41     68     82    111    144    173    176    190    210    217
 a      6   1363  49.626  42.827   8.634    14293  44.938  44.354   0.728
 b 32.175 53.390 25.822 21.623 -42.441 27.299 7.214 12.080 6.573 15.115
 c     43     84     99    120    147    161    173    183    166    187
 a      7   1686  48.062  37.817  12.521    14407  43.991  42.971   1.258
 b 11.830 -4.851 11.505 15.574 67.688 20.009 5.129 -7.014 8.766 1.024
 c     59    117    133    153    142    179    183    243    252    225
 a      8   1795  46.357  34.049  14.614    14433  44.757  43.282   1.829
 b 14.234 6.729 13.626 19.758 7.065 16.373 21.204 11.431 18.082 15.739
 c     57    103    137    183    168    202    205    240    237    263
 a      9   1483  50.778  46.558   5.526    14363  44.923  44.479   0.554
 b 40.845 .431 30.046 -7.049 -1.607 -2.724 -9.486 -.902 20.019 27.538
 c     46     88    119    135    154    161    193    192    203    192
 a     10   1332  38.220  34.823   3.918    14506  43.988  43.550   0.541
 b 33.078 34.752 9.425 25.010 -2.922 -14.118 14.600 12.844 1.512 -8.735
 c     51     77     93    118    123    154    166    172    181    197
 a     11   1477  45.163  38.543   8.015    14361  45.415  44.577   1.051
 b 9.914 8.880 2.467 18.150 5.381 11.874 4.743 29.698 14.834 -17.211
 c     65     82    111    133    115    159    174    202    220    216
 a     12   1654  45.251  30.457  17.159    14409  44.363  42.437   2.373
 b 28.681 6.430 -40.284 36.914 30.008 47.293 29.607 9.512 -1.455 21.488
 c     53    109    116    142    154    177    196    229    235    243
 a     13   1422  51.281  36.632  18.036    14302  44.261  42.742   1.872
 b 23.460 4.559 43.871 13.497 28.091 35.298 -1.493 17.764 17.226 19.655
 c     38     73    101    123    153    153    169    192    224    196
 a     14   1668  51.749  49.607   2.882    14441  44.642  44.527   0.145
 b 5.096 37.325 5.515 16.257 21.437 -7.901 14.955 4.960 -15.485 -2.206
 c     52    115    121    161    172    177    201    205    223    241
 a     15   1369  46.667  38.742   9.674    14263  44.480  43.770   0.882
 b -22.140 29.901 48.902 -7.002 26.620 .483 8.071 13.097 11.535 -5.181
 c     56     92    105    115    116    138    151    177    196    223
 a     16   1257  33.100  23.275  10.645    14291  44.849  43.826   1.273
 b -14.293 -.418 44.139 -1.828 30.203 21.053 5.098 22.834 -4.129 -.948
 c     34     60     57    120    133    127    159    188    199    180
 a     17   1370  42.421  39.259   3.794    14264  44.877  44.550   0.409
 b 30.851 -17.854 11.508 12.578 -18.416 2.277 20.934 12.146 -.761 .881
 c     33     72    106    131    134    151    157    189    197    200
 a     18   1269  45.248  26.342  21.465    14302  43.295  41.438   2.264
 b 23.189 2.571 31.364 37.883 60.830 25.784 13.345 33.459 -3.441 5.251
 c     45    101     97    111    127    143    144    164    168    169
 a     19   1644  43.200  43.114   0.105    14365  45.169  45.113   0.071
 b 23.714 13.389 29.752 38.784 -16.069 6.952 10.177 9.560 -19.196 -16.013
 c     56     87    132    138    168    179    183    215    236    250
 a     20   1615  54.027  42.202  15.317    14366  45.818  44.496   1.661
 b 20.068 20.724 30.458 21.475 33.352 28.157 5.646 16.916 9.398 8.795
 c     52     87    131    149    156    181    195    200    230    234
  -------------------------------------------------------------
  
 overall CC1/2ano:   -25.548 nref=    2668 (but the overall value is meaningless!)
 <CC1/2ano> :   -21.370 (frequency-weighted average of CC1/2ano in resolution shells)
 CC1/2ano in resolution shells:
   -75.0   -1.7  -30.1  -41.3  -38.8  -20.1  -18.2   -8.6  -17.7   -9.6
 frequency i.e. number of unique Friedel pairs:
      73    140    168    241    262    320    319    351    397    397
  
  headings for lines starting with d,e,f:
 d: <CC1/2ano> of each dataset:
 d: reflections of this dataset only      reflections of all datasets
 d: set   nref    with without   delta     nref    with without   delta
 e: delta-CC1/2ano in resolution shells
 f: # reflections for delta-CC1/2ano
 d     1    190 -30.590 -20.348 -10.922     2233 -17.969 -17.817  -0.157
 e -42.962 -6.341 4.502 -41.515 -76.721 -36.825 43.947 19.535 21.803 -23.853
 f     11     12      9     14     17     28     26     22     26     25
 d     2    234 -16.480 -16.515   0.036     2336 -26.125 -26.132   0.008
 e -3.116 -2.078 -3.249 -23.820 6.424 9.198 -3.791 1.123 6.966 -4.309
 f      5     12     16     12     35     30     40     24     29     31
 d     3    343 -40.435 -31.820  -9.887     2245 -22.659 -19.640  -3.160
 e -1.713 12.234 -11.942 -63.179 5.125 -4.423 15.291 -30.082 .859 3.714
 f      7     16     27     33     42     45     38     54     43     38
 d     4    250 -32.340 -29.274  -3.387     2385 -19.104 -18.426  -0.702
 e 22.408 37.540 12.328 -13.456 -2.729 1.911 -36.173 3.026 -6.757 12.916
 f      3     13     12     25     39     42     41     30     23     22
 d     5    179 -22.995 -24.947   2.070     2417 -26.768 -27.230   0.499
 e 36.142 13.527 -10.456 -7.251 13.331 -.923 27.239 -4.208 -9.635 -10.625
 f      5      4      7      6     19     22     29     30     23     34
 d     6    188 -10.090 -13.603   3.562     2412 -21.564 -21.759   0.205
 e -43.278 1.987 48.529 -2.536 35.294 -14.994 21.066 -6.607 -.446 -8.161
 f      5      9      8     10     21     28     25     30     27     25
 d     7    292 -34.206 -35.887   1.915     2316 -26.435 -26.723   0.309
 e -19.814 34.972 .715 48.559 -11.683 -4.391 -14.145 5.337 -3.722 2.527
 f     12     15     16     23     31     43     28     46     45     33
 d     8    305  -6.575  -7.745   1.176     2353 -19.132 -19.241   0.113
 e -7.804 -5.876 -10.001 4.878 -37.785 8.976 14.698 -4.716 9.080 11.786
 f     11     17     17     26     28     45     34     38     42     47
 d     9    283 -35.313 -32.348  -3.347     2398 -20.019 -19.463  -0.578
 e -90.495 31.505 6.592 -45.999 -7.955 -11.621 6.013 -6.399 18.725 2.816
 f      7     16     14     23     31     32     39     37     52     32
 d    10    228 -31.230 -36.231   5.639     2184 -20.991 -21.921   0.975
 e -10.975 26.119 -31.629 9.944 10.170 24.916 -19.125 25.874 -12.123 -.504
 f      6     23     12     16     17     32     31     33     27     31
 d    11    204 -25.646 -21.039  -4.870     2344 -26.292 -25.997  -0.317
 e 8.152 1.052 -8.483 4.821 5.281 -.012 -33.636 6.064 -12.304 -7.890
 f     13     11     16     14     19     25     24     28     26     28
 d    12    314 -20.795 -27.734   7.363     2299 -21.005 -22.984   2.080
 e -53.587 19.115 79.436 41.903 -3.288 4.497 22.135 12.845 -21.183 -1.646
 f      4     15     16     27     34     37     34     46     54     47
 d    13    233  -6.551  -6.622   0.071     2411 -21.747 -22.084   0.354
 e 7.268 -17.245 31.471 30.207 .209 7.647 5.126 -61.071 16.081 14.792
 f      4     11     15     20     20     34     38     34     27     30
 d    14    321 -39.745 -43.950   5.095     2335 -21.004 -21.556   0.578
 e 6.661 20.182 -13.578 -14.964 .663 20.951 -5.357 11.106 6.979 4.542
 f     13     26     24     29     37     30     31     33     48     50
 d    15    211 -25.574 -27.428   1.993     2451 -21.107 -22.106   1.049
 e 9.034 24.046 51.709 10.903 25.126 -31.105 -39.654 -43.178 6.990 15.268
 f      9     18     23     17     21     28     15     22     28     30
 d    16    164 -28.372 -24.930  -3.704     2381 -18.839 -18.689  -0.156
 e 20.336 23.610 -96.956 -10.020 7.356 1.404 14.821 -20.025 -4.886 -17.264
 f      2      6      2      9     17     20     33     24     20     31
 d    17    218 -13.441  -3.184 -10.302     2460 -20.794 -19.807  -1.029
 e -13.966 -12.793 5.985 10.830 -5.428 -23.834 17.910 -9.981 -25.620 -22.550
 f      2     13     11     19     16     22     26     38     34     37
 d    18    136 -32.513 -31.586  -1.034     2379 -21.312 -21.014  -0.312
 e 55.859 17.851 -8.273 -57.797 -68.191 27.216 15.219 15.258 -14.424 -10.376
 f      4     14     12     16     14     19     18     13     16     10
 d    19    295 -26.211 -37.786  12.848     2289 -19.913 -21.583   1.745
 e -3.719 -15.248 39.777 -13.377 11.415 8.559 -15.482 8.791 25.058 49.701
 f      8     13     17     33     30     35     40     38     38     43
 d    20    309 -33.133 -33.400   0.300     2351 -21.683 -22.124   0.463
 e -30.012 13.423 -2.376 15.330 -16.239 -2.850 -9.170 5.549 10.005 1.910
 f      9      9     15     32     33     36     40     48     48     39
  
  best delta-CC1/2_only=   26.22747    
  median of delta-cc1/2 ("only" i.e. 6th col of "a" lines) =   10.93907    
  noise= (MAD, median absolute deviation) from this median =   5.816339    
  median of delta-cc1/2 ("all" i.e. 10th col of "a" lines) =   1.174427    
  noise= (MAD, median absolute deviation) from this median =  0.6438574    
  median of delta-cc1/2-ano ("only" i.e. 6th col of "d" lines) =  0.1854570    
  noise= (MAD, median absolute deviation) from this median =   3.552712    
  median of delta-cc1/2-ano ("all" i.e. 10th col of "d" lines) =  0.1588948    
  noise= (MAD, median absolute deviation) from this median =  0.4448406    
  
 Wrote a commented XSCALE.INP.rename_me that is sorted on delta-CC1/2 "only"
 You may edit that file, or e.g. add lines after each INPUT_FILE line with sed '/INPUT_FILE/a INCLUDE_RESOLUTION_RANGE=99 3'
 normal termination

and the resulting file XSCALE.INP.rename_me is: 

SPACE_GROUP_NUMBER=   19
UNIT_CELL_CONSTANTS=     38.30     79.10     79.10  90.000  90.000  90.000
OUTPUT_FILE= temp.ahkl
PRINT_CORRELATIONS= FALSE
SAVE_CORRECTION_IMAGES= FALSE
FRIEDEL'S_LAW= FALSE
! median of delta-cc1/2 "only" values=    10.939
! noise (MAD) of these values=     5.816
! median of delta-cc1/2 "all" values=     1.174
! noise (MAD) of these values=     0.644
! median of delta-cc1/2-ano "only" values=     0.185
! noise (MAD) of these values=     3.553
! median of delta-cc1/2-ano "all" values=     0.159
! noise (MAD) of these values=     0.445
! input files sorted by deltacc12_only (highest first):
! deltacc12 only / all:   26.2275    1.9103 deltacc12-ano only /all:  -10.9223   -0.1569
INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0001/xds/XDS_ASCII.HKL
! deltacc12 only / all:   21.4650    2.2639 deltacc12-ano only /all:   -1.0339   -0.3117
INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0018/xds/XDS_ASCII.HKL
! deltacc12 only / all:   18.0364    1.8724 deltacc12-ano only /all:    0.0708    0.3545
INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0013/xds/XDS_ASCII.HKL
! deltacc12 only / all:   17.7684    1.9166 deltacc12-ano only /all:   -3.3870   -0.7021
INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0004/xds/XDS_ASCII.HKL
! deltacc12 only / all:   17.1589    2.3726 deltacc12-ano only /all:    7.3634    2.0795
INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0012/xds/XDS_ASCII.HKL
! deltacc12 only / all:   15.3174    1.6608 deltacc12-ano only /all:    0.3001    0.4632
INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0020/xds/XDS_ASCII.HKL
! deltacc12 only / all:   14.6140    1.8288 deltacc12-ano only /all:    1.1762    0.1129
INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0008/xds/XDS_ASCII.HKL
! deltacc12 only / all:   12.5208    1.2582 deltacc12-ano only /all:    1.9155    0.3089
INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0007/xds/XDS_ASCII.HKL
! deltacc12 only / all:   11.5122    1.0907 deltacc12-ano only /all:    0.0360    0.0079
INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0002/xds/XDS_ASCII.HKL
! deltacc12 only / all:   11.2330    1.6064 deltacc12-ano only /all:   -9.8867   -3.1596
INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0003/xds/XDS_ASCII.HKL
! deltacc12 only / all:   10.6452    1.2727 deltacc12-ano only /all:   -3.7041   -0.1557
INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0016/xds/XDS_ASCII.HKL
! deltacc12 only / all:    9.6737    0.8816 deltacc12-ano only /all:    1.9932    1.0487
INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0015/xds/XDS_ASCII.HKL
! deltacc12 only / all:    8.6341    0.7283 deltacc12-ano only /all:    3.5620    0.2049
INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0006/xds/XDS_ASCII.HKL
! deltacc12 only / all:    8.0151    1.0507 deltacc12-ano only /all:   -4.8702   -0.3171
INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0011/xds/XDS_ASCII.HKL
! deltacc12 only / all:    5.5262    0.5542 deltacc12-ano only /all:   -3.3475   -0.5778
INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0009/xds/XDS_ASCII.HKL
! deltacc12 only / all:    3.9182    0.5411 deltacc12-ano only /all:    5.6393    0.9750
INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0010/xds/XDS_ASCII.HKL
! deltacc12 only / all:    3.7935    0.4089 deltacc12-ano only /all:  -10.3019   -1.0286
INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0017/xds/XDS_ASCII.HKL
! deltacc12 only / all:    3.0997    0.3399 deltacc12-ano only /all:    2.0699    0.4987
INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0005/xds/XDS_ASCII.HKL
! deltacc12 only / all:    2.8817    0.1446 deltacc12-ano only /all:    5.0954    0.5780
INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0014/xds/XDS_ASCII.HKL
! deltacc12 only / all:    0.1054    0.0707 deltacc12-ano only /all:   12.8477    1.7449
INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0019/xds/XDS_ASCII.HKL

Explanation of output

The program output in the terminal window is terse but supposed to be self-explanatory; it can (and most often should) be saved or re-directed to a file. 

xdscc12 FILE_FROM_XSCALE > xdscc12.log  #  or xdscc12 FILE_FROM_XSCALE | tee xdscc12.log

All statistics (tables) produced by XDSCC12 may be visualized with e.g. gnuplot, after grepping the relevant lines from the output. If XDSCC12 is used with a XDS_ASCII.HKL reflection file (from XDS), the isomorphous delta-CC1/2 of a batch of frames (width chosen with the -t option; typically -t 1) relative to all data is most easily visualized via XDSGUI (Statistics tab). Negative numbers indicate a worsening of the overall signal.

If XDSCC12 is used with a XSCALE.HKL generated from multiple datasets, the output lines show the contribution of each dataset toward the total CC1/2. In this case, the program writes a file called XSCALE.INP.rename_me which shows statistics of delta-CC1/2 and delta-CC1/2-ano values, and has a sorted enumeration of the INPUT_FILEs - the first of these provides the best data set, and the last one is the worst one. This XSCALE.INP.rename_me can then be edited (i.e. for deleting a few data sets with strongly negative delta-CC1/2), and renamed to XSCALE.INP . Only delete the clearly worst data sets, and not more than 10% of the existing ones! This procedure can be iterated, i.e. after another round of XSCALE, XDSCC12 could be run again.

Overall statistics are reported in the lines starting with a and d for

  • only those unique reflections that are actually present in the batch of frame / batch / dataset. These values are in columns 3-6.
  • all unique reflections of the merged dataset (but a frame / batch / dataset may not have all unique reflections, so the "all" values report the mean influence). These values are in columns 7-10.

Typically, it is sensible to disregard the "all" values, and to base decisions on the "only" values, because the latter are not affected by the number of reflections of the particular frame / batch / dataset. The words "all" and "only" are used in this sense throughout the terminal and file output of XDSCC12.

Statistics for "only" the unique reflections of a frame / batch/ dataset are given in resolution shells for the isomorphous (in lines starting with b and c) and the anomalous signal (in lines starting with d and e). In case of SSX data (which have few reflections per data set, compared to complete data sets), we typically use -nbin 1 as option, to define only a single resolution shell.

To find out about the influence of the a and b parameters of the XDS/XSCALE-adjusted error model, you may try the -w option; this assigns the same sigma to all reflections. Likewise, the Fisher transformation, which serves to make changes in CC1/2 comparable across resolution ranges, may be switched off for testing purposes, with the -z option.

Correlation against a reference data set (-r <reference> option)

The correlation of the experimental data set against the user-supplied reference data is shown in the lines starting with r. To prepare a reference data set if the refinement was done with phenix.refine, one could use e.g. 

mtz2various hklin 2bn3_refine_001.mtz hklout reference.hkl <<eof
OUTPUT USER *
LABIN FC=F-model PHIC=PHIF-model
END
eof

- the column corresponding to PHIC will not be used by xdscc12. Alternatively, 

sftools
read mymodel_001.mtz
write reference.hkl format(3i5,f10.3) col F-model
y
quit

For a Refmac-written MTZ file, you would use "col FC_ALL" instead of "col F-model".

Reference data with anomalous signal (additional -s option)

The correlation of the anomalous difference of the experimental data set against the anomalous signal of the user-supplied reference data is shown in the lines starting with s. A simple way to obtain Fcalc(+) and Fcalc(-) is to run phenix.refine with options (in case of S as anomalous scatterer) 

refinement.input.xray_data.labels="F(+),SIGF(+),F(-),SIGF(-),merged"  refinement.refine.anomalous_scatterers.group.selection="element S" strategy=individual_sites+individual_adp+group_anomalous+occupancies

and then 

sftools <<eof
read mymodel_001.mtz
write anom-reference.hkl format(3i5,2f10.3) col "F-model(+)" "F-model(-)"
y
quit
eof

in which case sftools outputs only the acentric reflections - only those have anomalous differences. XDSCC12 then has to be run with the -s -r anom-reference.hkl option.

See also

A complete description of how to process serial crystallography data with XDS/XSCALE is given in SSX.

xscale_isocluster is a program that implements the method of Brehm and Diederichs (2014) and theory of Diederichs (2017). It serves to identify groups of related datasets in a reflection file produced by XSCALE, and should be used before XDSCC12.

To remove bad frames from a XDS_ASCII.HKL file, you can re-INTEGRATE or just re-CORRECT with the keyword EXCLUDE_DATA_RANGE in XDS.INP.

Retrieved from "https://wiki.uni-konstanz.de/xds/index.php?title=Xdscc12&oldid=4653"