Xdscc12: Difference between revisions
Jump to navigation
Jump to search
| (4 intermediate revisions by the same user not shown) | |||
| Line 2: | Line 2: | ||
XDSCC12 is a program for generating [[CC1/2|delta-CC<sub>1/2</sub>]] and delta-CC<sub>1/2-ano</sub> values for XDS_ASCII.HKL (written by [[XDS]]), or for XSCALE.HKL (written by [[XSCALE]]) containing data from several files of type XDS_ASCII.HKL after scaling (with MERGE=FALSE). | XDSCC12 is a program for generating [[CC1/2|delta-CC<sub>1/2</sub>]] and delta-CC<sub>1/2-ano</sub> values for XDS_ASCII.HKL (written by [[XDS]]), or for XSCALE.HKL (written by [[XSCALE]]) containing data from several files of type XDS_ASCII.HKL after scaling (with MERGE=FALSE). | ||
It implements the method described in Assmann, Brehm and Diederichs (2016) Identification of rogue datasets in serial crystallography. J. Appl. Cryst. 49, 1021 [http://journals.iucr.org/j/issues/2016/03/00/zw5005/zw5005.pdf], and it does this not only for the individual datasets in XSCALE.HKL, but also for individual frames, or groups of frames, of a single dataset collected with the rotation method and processed by [[XDS]]. | It implements the method described in Assmann, Brehm and Diederichs (2016) Identification of rogue datasets in serial crystallography. J. Appl. Cryst. 49, 1021 [http://journals.iucr.org/j/issues/2016/03/00/zw5005/zw5005.pdf], and it does this not only for the individual datasets in XSCALE.HKL, but also for individual frames, or groups (batches) of frames, of a single dataset collected with the rotation method and processed by [[XDS]]. | ||
The program can be downloaded for [https://{{SERVERNAME}}/pub/linux_bin/xdscc12 Linux] or [https://{{SERVERNAME}}/pub/mac_bin/xdscc12 Mac]. | The program can be downloaded for [https://{{SERVERNAME}}/pub/linux_bin/xdscc12 Linux] or [https://{{SERVERNAME}}/pub/mac_bin/xdscc12 Mac]. | ||
| Line 8: | Line 8: | ||
Usage (this text can be obtained with <code>xdscc12 -h</code>): | Usage (this text can be obtained with <code>xdscc12 -h</code>): | ||
<pre> | <pre> | ||
xdscc12 KD | xdscc12 KD 2023-01-08. Academic use only; no redistribution. -h option shows options. | ||
Please cite Assmann, G., Brehm, W., Diederichs, K. (2016) J.Appl.Cryst. 49, 1021-1028 | Please cite Assmann, G., Brehm, W., Diederichs, K. (2016) J.Appl.Cryst. 49, 1021-1028 | ||
running 'xdscc12 | running 'xdscc12' on 20240403 at 16:49:03 +0200 | ||
usage: xdscc12 [-dmin <lowres>] [-dmax <highres>] [-nbin <nbin>] [-mode <1 or 2>] [-<abcdefstwz>] [-r <ref>] FILE_NAME | usage: xdscc12 [-dmin <lowres>] [-dmax <highres>] [-nbin <nbin>] [-mode <1 or 2>] [-<abcdefstwz>] [-r|-R <ref>] FILE_NAME | ||
dmin (default 999A), dmax (default 1A) and nbin (default 10) have the usual meanings. | dmin (default 999A), dmax (default 1A) and nbin (default 10) have the usual meanings. | ||
mode can be 1 (equal volumes of resolution shells) or 2 (increasing volumes; default). | mode can be 1 (equal volumes of resolution shells) or 2 (increasing volumes; default). | ||
-r: | -r: <ref> is ASCII reference file with lines: h,k,l,Fref or h,k,l,Fref+,Fref- | ||
-R: <ref> is ASCII reference file with lines: h,k,l,Iref or h,k,l,Iref+,Iref- | |||
-s: read two columns from reference: | -r and -R allow calculation of CC of isomorphous signal with reference | ||
-s: read two columns from reference: Fref+, Fref- or Iref+, Iref-. | |||
this allows calculation of CC of anomalous signal with that of reference | this allows calculation of CC of anomalous signal with that of reference | ||
-A: sort INPUT_FILEs in XSCALE.INP.rename_me by anomalous instead of isomorphous delta-CC1/2 | |||
-t: total oscillation (degree) to batch fine-sliced frames into | -t: total oscillation (degree) to batch fine-sliced frames into | ||
FILE_NAME can be XDS or XSCALE reflection file | FILE_NAME can be XDS or XSCALE reflection file | ||
| Line 29: | Line 31: | ||
-w: weighting of intensities with their sigmas | -w: weighting of intensities with their sigmas | ||
-z: Fisher transformation of delta-CC1/2 values | -z: Fisher transformation of delta-CC1/2 values | ||
The program writes a commented XSCALE.INP.rename_me that is sorted on delta-CC1/2 | |||
</pre> | </pre> | ||
== Example output == | == Example output == | ||
| Line 283: | Line 276: | ||
INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0019/xds/XDS_ASCII.HKL | INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0019/xds/XDS_ASCII.HKL | ||
</pre> | </pre> | ||
== Explanation of output == | |||
The program output in the terminal window is terse but supposed to be self-explanatory; it can (and most often should) be saved or re-directed to a file. | |||
xdscc12 FILE_FROM_XSCALE > xdscc12.log # or xdscc12 FILE_FROM_XSCALE | tee xdscc12.log | |||
All statistics (tables) produced by XDSCC12 may be visualized with e.g. gnuplot, after grepping the relevant lines from the output. | |||
If XDSCC12 is used with a XDS_ASCII.HKL reflection file (from XDS), the isomorphous delta-CC<sub>1/2</sub> of a batch of frames (width chosen with the <code>-t</code> option; typically <code>-t 1</code>) relative to all data is most easily visualized via [[XDSGUI]] (Statistics tab). Negative numbers indicate a worsening of the overall signal. | |||
If XDSCC12 is used with a XSCALE.HKL generated from multiple datasets, the output lines show the contribution of each dataset toward the total CC<sub>1/2</sub>. In this case, the program writes a file called XSCALE.INP.rename_me which shows statistics of delta-CC<sub>1/2</sub> and delta-CC<sub>1/2-ano</sub> values, and has a sorted enumeration of the INPUT_FILEs - the first of these provides the best data set, and the last one is the worst one. This XSCALE.INP.rename_me can then be edited (i.e. for deleting a few data sets with strongly negative delta-CC<sub>1/2</sub>), and renamed to XSCALE.INP . Only delete the clearly worst data sets, and not more than 10% of the existing ones! This procedure can be iterated, i.e. after another round of XSCALE, XDSCC12 could be run again. | |||
Overall statistics are reported in the lines starting with <code>a</code> and <code>d</code> for | |||
* <b>only</b> those unique reflections that are actually present in the batch of frame / batch / dataset. These values are in columns 3-6. | |||
* <b>all</b> unique reflections of the merged dataset (but a frame / batch / dataset may not have all unique reflections, so the "all" values report the mean influence). These values are in columns 7-10. | |||
Typically, it is sensible to disregard the "all" values, and to base decisions on the "only" values, because the latter are not affected by the number of reflections of the particular frame / batch / dataset. The words "all" and "only" are used in this sense throughout the terminal and file output of XDSCC12. | |||
Statistics for "only" the unique reflections of a frame / batch/ dataset are given in resolution shells for the isomorphous (in lines starting with <code>b</code> and <code>c</code>) and the anomalous signal (in lines starting with <code>d</code> and <code>e</code>). In case of [[SSX]] data (which have few reflections per data set, compared to complete data sets), we typically use <code>-nbin 1</code> as option, to define only a single resolution shell. | |||
To find out about the influence of the ''a'' and ''b'' parameters of the XDS/XSCALE-adjusted error model, you may try the <code>-w</code> option; this assigns the same sigma to all reflections. Likewise, the [https://en.wikipedia.org/wiki/Fisher_transformation Fisher transformation], which serves to make changes in CC<sub>1/2</sub> comparable across resolution ranges, may be switched off for testing purposes, with the <code>-z</code> option. | |||
== Correlation against a reference data set (-r <reference> option) == | == Correlation against a reference data set (-r <reference> option) == | ||
| Line 288: | Line 298: | ||
To prepare a reference data set if the refinement was done with phenix.refine, one could use e.g. | To prepare a reference data set if the refinement was done with phenix.refine, one could use e.g. | ||
<pre> | <pre> | ||
mtz2various hklin 2bn3_refine_001.mtz hklout | mtz2various hklin 2bn3_refine_001.mtz hklout reference.hkl <<eof | ||
OUTPUT USER * | OUTPUT USER * | ||
LABIN FC=F-model PHIC=PHIF-model | LABIN FC=F-model PHIC=PHIF-model | ||
| Line 298: | Line 308: | ||
sftools | sftools | ||
read mymodel_001.mtz | read mymodel_001.mtz | ||
write | write reference.hkl format(3i5,f10.3) col F-model | ||
y | y | ||
quit | quit | ||
</pre> | </pre> | ||
For a Refmac-written MTZ file, you would use "col FC_ALL" instead of "col F-model". | |||
=== Reference data with anomalous signal (additional -s option) === | === Reference data with anomalous signal (additional -s option) === | ||
Latest revision as of 17:46, 31 October 2024
XDSCC12 is a program for generating delta-CC1/2 and delta-CC1/2-ano values for XDS_ASCII.HKL (written by XDS), or for XSCALE.HKL (written by XSCALE) containing data from several files of type XDS_ASCII.HKL after scaling (with MERGE=FALSE).
It implements the method described in Assmann, Brehm and Diederichs (2016) Identification of rogue datasets in serial crystallography. J. Appl. Cryst. 49, 1021 [1], and it does this not only for the individual datasets in XSCALE.HKL, but also for individual frames, or groups (batches) of frames, of a single dataset collected with the rotation method and processed by XDS.
The program can be downloaded for Linux or Mac.
Usage (this text can be obtained with xdscc12 -h):
xdscc12 KD 2023-01-08. Academic use only; no redistribution. -h option shows options.
Please cite Assmann, G., Brehm, W., Diederichs, K. (2016) J.Appl.Cryst. 49, 1021-1028
running 'xdscc12' on 20240403 at 16:49:03 +0200
usage: xdscc12 [-dmin <lowres>] [-dmax <highres>] [-nbin <nbin>] [-mode <1 or 2>] [-<abcdefstwz>] [-r|-R <ref>] FILE_NAME
dmin (default 999A), dmax (default 1A) and nbin (default 10) have the usual meanings.
mode can be 1 (equal volumes of resolution shells) or 2 (increasing volumes; default).
-r: <ref> is ASCII reference file with lines: h,k,l,Fref or h,k,l,Fref+,Fref-
-R: <ref> is ASCII reference file with lines: h,k,l,Iref or h,k,l,Iref+,Iref-
-r and -R allow calculation of CC of isomorphous signal with reference
-s: read two columns from reference: Fref+, Fref- or Iref+, Iref-.
this allows calculation of CC of anomalous signal with that of reference
-A: sort INPUT_FILEs in XSCALE.INP.rename_me by anomalous instead of isomorphous delta-CC1/2
-t: total oscillation (degree) to batch fine-sliced frames into
FILE_NAME can be XDS or XSCALE reflection file
other options can be combined (e.g. -def), and switch the following OFF:
-a: individual isomorphous summary values
-b: individual (Fisher-transformed) delta-CC1/2 values
-c: individual delta-CC1/2 reflection numbers
-d: individual anomalous summary values
-e: individual (Fisher-transformed) delta-CC1/2ano values
-f: individual delta-CC1/2ano reflection numbers
-w: weighting of intensities with their sigmas
-z: Fisher transformation of delta-CC1/2 values
The program writes a commented XSCALE.INP.rename_me that is sorted on delta-CC1/2
Example output
xdscc12 KD 2020-12-9. Academic use only; no redistribution. -h option shows options.
Please cite Assmann, G., Brehm, W., Diederichs, K. (2016) J.Appl.Cryst. 49, 1021-1028
running 'xdscc12 temp.ahkl' on 20220413 at 12:13:53 +0200
no option -w found, therefore statistics are weighted by sigma values
no option -z found, therefore delta-CC1/2 values are Fisher-transformed
reflection file is temp.ahkl
!SPACE_GROUP_NUMBER= 19
!UNIT_CELL_CONSTANTS= 38.30 79.10 79.10 90.000 90.000 90.000
# of datasets= 20
# obs (w/o misfits), unique, misfits = 51918 20213 0
max and min resolution of data in file = 39.55000 1.801329
data between 39.55000 and 1.801329 A will be used
10 resolution shells (for lines starting with b,c,e,f,r,s):
5.644 4.011 3.281 2.844 2.545 2.324 2.152 2.013 1.899 1.801
overall CC1/2: 83.328 nref= 14986 (but the overall CC1/2 is meaningless!)
<CC1/2>: 44.468 (frequency-weighted average of CC1/2 in resolution shells)
CC1/2 in resolution shells:
91.4 81.0 72.0 68.4 42.1 41.4 32.4 29.0 33.2 28.2
CC* in resolution shells:
97.7 94.6 91.5 90.1 77.0 76.5 70.0 67.0 70.6 66.3
frequency, i.e. number of unique reflections in resolution shells:
493 887 1134 1345 1501 1649 1808 1921 2061 2187
headings for lines starting with a,b,c:
a: <CC1/2> of each dataset:
a: reflections of this dataset only reflections of all datasets
a: set nref with without delta nref with without delta
b: delta-CC1/2 in resolution shells
c: # reflections for delta-CC1/2
a 1 1241 61.936 42.634 26.227 14470 44.075 42.523 1.910
b 61.432 27.680 8.094 2.942 41.231 44.296 26.090 10.996 30.482 30.456
c 47 73 85 110 135 137 146 170 164 174
a 2 1565 43.941 34.156 11.512 14350 44.978 44.104 1.091
b 36.910 -5.982 -1.113 58.592 46.243 -24.157 -.234 20.068 11.307 -3.709
c 49 90 126 134 158 168 196 203 216 225
a 3 1754 39.551 29.634 11.233 14478 43.683 42.374 1.606
b 13.559 11.752 17.587 17.093 16.730 19.779 3.638 20.348 6.182 4.113
c 43 107 131 160 175 198 218 230 246 246
a 4 1468 41.694 25.840 17.768 14382 44.801 43.256 1.917
b -9.445 36.760 36.335 15.954 2.951 20.264 21.375 19.554 2.513 35.809
c 30 93 99 115 164 174 202 220 188 183
a 5 1412 41.785 39.193 3.100 14292 45.561 45.291 0.340
b 21.421 1.775 7.645 1.361 6.005 22.513 11.859 -16.883 .973 -2.353
c 41 68 82 111 144 173 176 190 210 217
a 6 1363 49.626 42.827 8.634 14293 44.938 44.354 0.728
b 32.175 53.390 25.822 21.623 -42.441 27.299 7.214 12.080 6.573 15.115
c 43 84 99 120 147 161 173 183 166 187
a 7 1686 48.062 37.817 12.521 14407 43.991 42.971 1.258
b 11.830 -4.851 11.505 15.574 67.688 20.009 5.129 -7.014 8.766 1.024
c 59 117 133 153 142 179 183 243 252 225
a 8 1795 46.357 34.049 14.614 14433 44.757 43.282 1.829
b 14.234 6.729 13.626 19.758 7.065 16.373 21.204 11.431 18.082 15.739
c 57 103 137 183 168 202 205 240 237 263
a 9 1483 50.778 46.558 5.526 14363 44.923 44.479 0.554
b 40.845 .431 30.046 -7.049 -1.607 -2.724 -9.486 -.902 20.019 27.538
c 46 88 119 135 154 161 193 192 203 192
a 10 1332 38.220 34.823 3.918 14506 43.988 43.550 0.541
b 33.078 34.752 9.425 25.010 -2.922 -14.118 14.600 12.844 1.512 -8.735
c 51 77 93 118 123 154 166 172 181 197
a 11 1477 45.163 38.543 8.015 14361 45.415 44.577 1.051
b 9.914 8.880 2.467 18.150 5.381 11.874 4.743 29.698 14.834 -17.211
c 65 82 111 133 115 159 174 202 220 216
a 12 1654 45.251 30.457 17.159 14409 44.363 42.437 2.373
b 28.681 6.430 -40.284 36.914 30.008 47.293 29.607 9.512 -1.455 21.488
c 53 109 116 142 154 177 196 229 235 243
a 13 1422 51.281 36.632 18.036 14302 44.261 42.742 1.872
b 23.460 4.559 43.871 13.497 28.091 35.298 -1.493 17.764 17.226 19.655
c 38 73 101 123 153 153 169 192 224 196
a 14 1668 51.749 49.607 2.882 14441 44.642 44.527 0.145
b 5.096 37.325 5.515 16.257 21.437 -7.901 14.955 4.960 -15.485 -2.206
c 52 115 121 161 172 177 201 205 223 241
a 15 1369 46.667 38.742 9.674 14263 44.480 43.770 0.882
b -22.140 29.901 48.902 -7.002 26.620 .483 8.071 13.097 11.535 -5.181
c 56 92 105 115 116 138 151 177 196 223
a 16 1257 33.100 23.275 10.645 14291 44.849 43.826 1.273
b -14.293 -.418 44.139 -1.828 30.203 21.053 5.098 22.834 -4.129 -.948
c 34 60 57 120 133 127 159 188 199 180
a 17 1370 42.421 39.259 3.794 14264 44.877 44.550 0.409
b 30.851 -17.854 11.508 12.578 -18.416 2.277 20.934 12.146 -.761 .881
c 33 72 106 131 134 151 157 189 197 200
a 18 1269 45.248 26.342 21.465 14302 43.295 41.438 2.264
b 23.189 2.571 31.364 37.883 60.830 25.784 13.345 33.459 -3.441 5.251
c 45 101 97 111 127 143 144 164 168 169
a 19 1644 43.200 43.114 0.105 14365 45.169 45.113 0.071
b 23.714 13.389 29.752 38.784 -16.069 6.952 10.177 9.560 -19.196 -16.013
c 56 87 132 138 168 179 183 215 236 250
a 20 1615 54.027 42.202 15.317 14366 45.818 44.496 1.661
b 20.068 20.724 30.458 21.475 33.352 28.157 5.646 16.916 9.398 8.795
c 52 87 131 149 156 181 195 200 230 234
-------------------------------------------------------------
overall CC1/2ano: -25.548 nref= 2668 (but the overall value is meaningless!)
<CC1/2ano> : -21.370 (frequency-weighted average of CC1/2ano in resolution shells)
CC1/2ano in resolution shells:
-75.0 -1.7 -30.1 -41.3 -38.8 -20.1 -18.2 -8.6 -17.7 -9.6
frequency i.e. number of unique Friedel pairs:
73 140 168 241 262 320 319 351 397 397
headings for lines starting with d,e,f:
d: <CC1/2ano> of each dataset:
d: reflections of this dataset only reflections of all datasets
d: set nref with without delta nref with without delta
e: delta-CC1/2ano in resolution shells
f: # reflections for delta-CC1/2ano
d 1 190 -30.590 -20.348 -10.922 2233 -17.969 -17.817 -0.157
e -42.962 -6.341 4.502 -41.515 -76.721 -36.825 43.947 19.535 21.803 -23.853
f 11 12 9 14 17 28 26 22 26 25
d 2 234 -16.480 -16.515 0.036 2336 -26.125 -26.132 0.008
e -3.116 -2.078 -3.249 -23.820 6.424 9.198 -3.791 1.123 6.966 -4.309
f 5 12 16 12 35 30 40 24 29 31
d 3 343 -40.435 -31.820 -9.887 2245 -22.659 -19.640 -3.160
e -1.713 12.234 -11.942 -63.179 5.125 -4.423 15.291 -30.082 .859 3.714
f 7 16 27 33 42 45 38 54 43 38
d 4 250 -32.340 -29.274 -3.387 2385 -19.104 -18.426 -0.702
e 22.408 37.540 12.328 -13.456 -2.729 1.911 -36.173 3.026 -6.757 12.916
f 3 13 12 25 39 42 41 30 23 22
d 5 179 -22.995 -24.947 2.070 2417 -26.768 -27.230 0.499
e 36.142 13.527 -10.456 -7.251 13.331 -.923 27.239 -4.208 -9.635 -10.625
f 5 4 7 6 19 22 29 30 23 34
d 6 188 -10.090 -13.603 3.562 2412 -21.564 -21.759 0.205
e -43.278 1.987 48.529 -2.536 35.294 -14.994 21.066 -6.607 -.446 -8.161
f 5 9 8 10 21 28 25 30 27 25
d 7 292 -34.206 -35.887 1.915 2316 -26.435 -26.723 0.309
e -19.814 34.972 .715 48.559 -11.683 -4.391 -14.145 5.337 -3.722 2.527
f 12 15 16 23 31 43 28 46 45 33
d 8 305 -6.575 -7.745 1.176 2353 -19.132 -19.241 0.113
e -7.804 -5.876 -10.001 4.878 -37.785 8.976 14.698 -4.716 9.080 11.786
f 11 17 17 26 28 45 34 38 42 47
d 9 283 -35.313 -32.348 -3.347 2398 -20.019 -19.463 -0.578
e -90.495 31.505 6.592 -45.999 -7.955 -11.621 6.013 -6.399 18.725 2.816
f 7 16 14 23 31 32 39 37 52 32
d 10 228 -31.230 -36.231 5.639 2184 -20.991 -21.921 0.975
e -10.975 26.119 -31.629 9.944 10.170 24.916 -19.125 25.874 -12.123 -.504
f 6 23 12 16 17 32 31 33 27 31
d 11 204 -25.646 -21.039 -4.870 2344 -26.292 -25.997 -0.317
e 8.152 1.052 -8.483 4.821 5.281 -.012 -33.636 6.064 -12.304 -7.890
f 13 11 16 14 19 25 24 28 26 28
d 12 314 -20.795 -27.734 7.363 2299 -21.005 -22.984 2.080
e -53.587 19.115 79.436 41.903 -3.288 4.497 22.135 12.845 -21.183 -1.646
f 4 15 16 27 34 37 34 46 54 47
d 13 233 -6.551 -6.622 0.071 2411 -21.747 -22.084 0.354
e 7.268 -17.245 31.471 30.207 .209 7.647 5.126 -61.071 16.081 14.792
f 4 11 15 20 20 34 38 34 27 30
d 14 321 -39.745 -43.950 5.095 2335 -21.004 -21.556 0.578
e 6.661 20.182 -13.578 -14.964 .663 20.951 -5.357 11.106 6.979 4.542
f 13 26 24 29 37 30 31 33 48 50
d 15 211 -25.574 -27.428 1.993 2451 -21.107 -22.106 1.049
e 9.034 24.046 51.709 10.903 25.126 -31.105 -39.654 -43.178 6.990 15.268
f 9 18 23 17 21 28 15 22 28 30
d 16 164 -28.372 -24.930 -3.704 2381 -18.839 -18.689 -0.156
e 20.336 23.610 -96.956 -10.020 7.356 1.404 14.821 -20.025 -4.886 -17.264
f 2 6 2 9 17 20 33 24 20 31
d 17 218 -13.441 -3.184 -10.302 2460 -20.794 -19.807 -1.029
e -13.966 -12.793 5.985 10.830 -5.428 -23.834 17.910 -9.981 -25.620 -22.550
f 2 13 11 19 16 22 26 38 34 37
d 18 136 -32.513 -31.586 -1.034 2379 -21.312 -21.014 -0.312
e 55.859 17.851 -8.273 -57.797 -68.191 27.216 15.219 15.258 -14.424 -10.376
f 4 14 12 16 14 19 18 13 16 10
d 19 295 -26.211 -37.786 12.848 2289 -19.913 -21.583 1.745
e -3.719 -15.248 39.777 -13.377 11.415 8.559 -15.482 8.791 25.058 49.701
f 8 13 17 33 30 35 40 38 38 43
d 20 309 -33.133 -33.400 0.300 2351 -21.683 -22.124 0.463
e -30.012 13.423 -2.376 15.330 -16.239 -2.850 -9.170 5.549 10.005 1.910
f 9 9 15 32 33 36 40 48 48 39
best delta-CC1/2_only= 26.22747
median of delta-cc1/2 ("only" i.e. 6th col of "a" lines) = 10.93907
noise= (MAD, median absolute deviation) from this median = 5.816339
median of delta-cc1/2 ("all" i.e. 10th col of "a" lines) = 1.174427
noise= (MAD, median absolute deviation) from this median = 0.6438574
median of delta-cc1/2-ano ("only" i.e. 6th col of "d" lines) = 0.1854570
noise= (MAD, median absolute deviation) from this median = 3.552712
median of delta-cc1/2-ano ("all" i.e. 10th col of "d" lines) = 0.1588948
noise= (MAD, median absolute deviation) from this median = 0.4448406
Wrote a commented XSCALE.INP.rename_me that is sorted on delta-CC1/2 "only"
You may edit that file, or e.g. add lines after each INPUT_FILE line with sed '/INPUT_FILE/a INCLUDE_RESOLUTION_RANGE=99 3'
normal termination
and the resulting file XSCALE.INP.rename_me is:
SPACE_GROUP_NUMBER= 19 UNIT_CELL_CONSTANTS= 38.30 79.10 79.10 90.000 90.000 90.000 OUTPUT_FILE= temp.ahkl PRINT_CORRELATIONS= FALSE SAVE_CORRECTION_IMAGES= FALSE FRIEDEL'S_LAW= FALSE ! median of delta-cc1/2 "only" values= 10.939 ! noise (MAD) of these values= 5.816 ! median of delta-cc1/2 "all" values= 1.174 ! noise (MAD) of these values= 0.644 ! median of delta-cc1/2-ano "only" values= 0.185 ! noise (MAD) of these values= 3.553 ! median of delta-cc1/2-ano "all" values= 0.159 ! noise (MAD) of these values= 0.445 ! input files sorted by deltacc12_only (highest first): ! deltacc12 only / all: 26.2275 1.9103 deltacc12-ano only /all: -10.9223 -0.1569 INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0001/xds/XDS_ASCII.HKL ! deltacc12 only / all: 21.4650 2.2639 deltacc12-ano only /all: -1.0339 -0.3117 INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0018/xds/XDS_ASCII.HKL ! deltacc12 only / all: 18.0364 1.8724 deltacc12-ano only /all: 0.0708 0.3545 INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0013/xds/XDS_ASCII.HKL ! deltacc12 only / all: 17.7684 1.9166 deltacc12-ano only /all: -3.3870 -0.7021 INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0004/xds/XDS_ASCII.HKL ! deltacc12 only / all: 17.1589 2.3726 deltacc12-ano only /all: 7.3634 2.0795 INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0012/xds/XDS_ASCII.HKL ! deltacc12 only / all: 15.3174 1.6608 deltacc12-ano only /all: 0.3001 0.4632 INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0020/xds/XDS_ASCII.HKL ! deltacc12 only / all: 14.6140 1.8288 deltacc12-ano only /all: 1.1762 0.1129 INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0008/xds/XDS_ASCII.HKL ! deltacc12 only / all: 12.5208 1.2582 deltacc12-ano only /all: 1.9155 0.3089 INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0007/xds/XDS_ASCII.HKL ! deltacc12 only / all: 11.5122 1.0907 deltacc12-ano only /all: 0.0360 0.0079 INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0002/xds/XDS_ASCII.HKL ! deltacc12 only / all: 11.2330 1.6064 deltacc12-ano only /all: -9.8867 -3.1596 INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0003/xds/XDS_ASCII.HKL ! deltacc12 only / all: 10.6452 1.2727 deltacc12-ano only /all: -3.7041 -0.1557 INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0016/xds/XDS_ASCII.HKL ! deltacc12 only / all: 9.6737 0.8816 deltacc12-ano only /all: 1.9932 1.0487 INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0015/xds/XDS_ASCII.HKL ! deltacc12 only / all: 8.6341 0.7283 deltacc12-ano only /all: 3.5620 0.2049 INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0006/xds/XDS_ASCII.HKL ! deltacc12 only / all: 8.0151 1.0507 deltacc12-ano only /all: -4.8702 -0.3171 INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0011/xds/XDS_ASCII.HKL ! deltacc12 only / all: 5.5262 0.5542 deltacc12-ano only /all: -3.3475 -0.5778 INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0009/xds/XDS_ASCII.HKL ! deltacc12 only / all: 3.9182 0.5411 deltacc12-ano only /all: 5.6393 0.9750 INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0010/xds/XDS_ASCII.HKL ! deltacc12 only / all: 3.7935 0.4089 deltacc12-ano only /all: -10.3019 -1.0286 INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0017/xds/XDS_ASCII.HKL ! deltacc12 only / all: 3.0997 0.3399 deltacc12-ano only /all: 2.0699 0.4987 INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0005/xds/XDS_ASCII.HKL ! deltacc12 only / all: 2.8817 0.1446 deltacc12-ano only /all: 5.0954 0.5780 INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0014/xds/XDS_ASCII.HKL ! deltacc12 only / all: 0.1054 0.0707 deltacc12-ano only /all: 12.8477 1.7449 INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0019/xds/XDS_ASCII.HKL
Explanation of output
The program output in the terminal window is terse but supposed to be self-explanatory; it can (and most often should) be saved or re-directed to a file.
xdscc12 FILE_FROM_XSCALE > xdscc12.log # or xdscc12 FILE_FROM_XSCALE | tee xdscc12.log
All statistics (tables) produced by XDSCC12 may be visualized with e.g. gnuplot, after grepping the relevant lines from the output.
If XDSCC12 is used with a XDS_ASCII.HKL reflection file (from XDS), the isomorphous delta-CC1/2 of a batch of frames (width chosen with the -t option; typically -t 1) relative to all data is most easily visualized via XDSGUI (Statistics tab). Negative numbers indicate a worsening of the overall signal.
If XDSCC12 is used with a XSCALE.HKL generated from multiple datasets, the output lines show the contribution of each dataset toward the total CC1/2. In this case, the program writes a file called XSCALE.INP.rename_me which shows statistics of delta-CC1/2 and delta-CC1/2-ano values, and has a sorted enumeration of the INPUT_FILEs - the first of these provides the best data set, and the last one is the worst one. This XSCALE.INP.rename_me can then be edited (i.e. for deleting a few data sets with strongly negative delta-CC1/2), and renamed to XSCALE.INP . Only delete the clearly worst data sets, and not more than 10% of the existing ones! This procedure can be iterated, i.e. after another round of XSCALE, XDSCC12 could be run again.
Overall statistics are reported in the lines starting with a and d for
- only those unique reflections that are actually present in the batch of frame / batch / dataset. These values are in columns 3-6.
- all unique reflections of the merged dataset (but a frame / batch / dataset may not have all unique reflections, so the "all" values report the mean influence). These values are in columns 7-10.
Typically, it is sensible to disregard the "all" values, and to base decisions on the "only" values, because the latter are not affected by the number of reflections of the particular frame / batch / dataset. The words "all" and "only" are used in this sense throughout the terminal and file output of XDSCC12.
Statistics for "only" the unique reflections of a frame / batch/ dataset are given in resolution shells for the isomorphous (in lines starting with b and c) and the anomalous signal (in lines starting with d and e). In case of SSX data (which have few reflections per data set, compared to complete data sets), we typically use -nbin 1 as option, to define only a single resolution shell.
To find out about the influence of the a and b parameters of the XDS/XSCALE-adjusted error model, you may try the -w option; this assigns the same sigma to all reflections. Likewise, the Fisher transformation, which serves to make changes in CC1/2 comparable across resolution ranges, may be switched off for testing purposes, with the -z option.
Correlation against a reference data set (-r <reference> option)
The correlation of the experimental data set against the user-supplied reference data is shown in the lines starting with r. To prepare a reference data set if the refinement was done with phenix.refine, one could use e.g.
mtz2various hklin 2bn3_refine_001.mtz hklout reference.hkl <<eof OUTPUT USER * LABIN FC=F-model PHIC=PHIF-model END eof
- the column corresponding to PHIC will not be used by xdscc12. Alternatively,
sftools read mymodel_001.mtz write reference.hkl format(3i5,f10.3) col F-model y quit
For a Refmac-written MTZ file, you would use "col FC_ALL" instead of "col F-model".
Reference data with anomalous signal (additional -s option)
The correlation of the anomalous difference of the experimental data set against the anomalous signal of the user-supplied reference data is shown in the lines starting with s.
A simple way to obtain Fcalc(+) and Fcalc(-) is to run phenix.refine with options (in case of S as anomalous scatterer)
refinement.input.xray_data.labels="F(+),SIGF(+),F(-),SIGF(-),merged" refinement.refine.anomalous_scatterers.group.selection="element S" strategy=individual_sites+individual_adp+group_anomalous+occupancies
and then
sftools <<eof read mymodel_001.mtz write anom-reference.hkl format(3i5,2f10.3) col "F-model(+)" "F-model(-)" y quit eof
in which case sftools outputs only the acentric reflections - only those have anomalous differences. XDSCC12 then has to be run with the -s -r anom-reference.hkl option.
See also
A complete description of how to process serial crystallography data with XDS/XSCALE is given in SSX.
xscale_isocluster is a program that implements the method of Brehm and Diederichs (2014) and theory of Diederichs (2017). It serves to identify groups of related datasets in a reflection file produced by XSCALE, and should be used before XDSCC12.
To remove bad frames from a XDS_ASCII.HKL file, you can re-INTEGRATE or just re-CORRECT with the keyword EXCLUDE_DATA_RANGE in XDS.INP.