Xdscc12: Difference between revisions

(2 intermediate revisions by the same user not shown)

Line 2:

XDSCC12 is a program for generating [[CC1/2|delta-CC1/2]] and delta-CC1/2-ano values for XDS_ASCII.HKL (written by [[XDS]]), or for XSCALE.HKL (written by [[XSCALE]]) containing data from several files of type XDS_ASCII.HKL after scaling (with MERGE=FALSE).

It implements the method described in Assmann, Brehm and Diederichs (2016) Identification of rogue datasets in serial crystallography. J. Appl. Cryst. 49, 1021 [http://journals.iucr.org/j/issues/2016/03/00/zw5005/zw5005.pdf], and it does this not only for the individual datasets in XSCALE.HKL, but also for individual frames, or groups of frames, of a single dataset collected with the rotation method and processed by [[XDS]].

It implements the method described in Assmann, Brehm and Diederichs (2016) Identification of rogue datasets in serial crystallography. J. Appl. Cryst. 49, 1021 [http://journals.iucr.org/j/issues/2016/03/00/zw5005/zw5005.pdf], and it does this not only for the individual datasets in XSCALE.HKL, but also for individual frames, or groups (batches) of frames, of a single dataset collected with the rotation method and processed by [[XDS]].

The program can be downloaded for [https://{{SERVERNAME}}/pub/linux_bin/xdscc12 Linux] or [https://{{SERVERNAME}}/pub/mac_bin/xdscc12 Mac].

Line 31:

</pre>

~~The program output in the terminal window is terse but supposed to be self-explanatory; it can (and most often should) be saved or re-directed to a file.~~

== Example output ==

~~xdscc12 ... > xdscc12.log # or xdscc12 ... | tee xdscc12.log~~

<pre>

~~All statistics (tables) produced by XDSCC12 may be visualized with e.g. gnuplot, after grepping the relevant lines from the output.~~

If XDSCC12 is used with a XDS_ASCII.HKL reflection file (from XDS), the isomorphous delta-CC1/2 of a batch of frames (width chosen with the <code>-t</code> option; typically <code>-t 1</code>) relative to all data is most easily visualized via [[XDSGUI]] (Statistics tab). Negative numbers indicate a worsening of the overall signal.

If XDSCC12 is used with a XSCALE.HKL generated from multiple datasets, the output lines show the contribution of each dataset toward the total CC1/2. In this case, the program writes a file called XSCALE.INP.rename_me which shows statistics of delta-CC1/2 and delta-CC1/2-ano values, and has a sorted enumeration of the INPUT_FILEs - the first of these provides the best data set, and the last one is the worst one. This XSCALE.INP.rename_me can then be edited (i.e. for deleting a few data sets with very negative delta-CC1/2), and renamed to XSCALE.INP.

Statistics are given (in resolution shells) for the isomorphous and the anomalous signal. In case of [[SSX]] data (which have few reflections per data set, compared to complete data sets), we typically use <code>-nbin 1</code> as option.

To find out about the influence of the ''a'' and ''b'' parameters of the XDS/XSCALE-adjusted error model, you may try the <code>-w</code> option; this assigns the same sigma to all reflections. Likewise, the [https://en.wikipedia.org/wiki/Fisher_transformation Fisher transformation], which serves to make changes in CC1/2 comparable across resolution ranges, may be switched off for testing purposes, with the -z option.

== Example output ~~with explanation~~ ==

xdscc12 KD 2020-12-9. Academic use only; no redistribution. -h option shows options.

Please cite Assmann, G., Brehm, W., Diederichs, K. (2016) J.Appl.Cryst. 49, 1021-1028

Line 48:

Line 38:

no option -w found, therefore statistics are weighted by sigma values

no option -z found, therefore delta-CC1/2 values are Fisher-transformed

~~<pre>~~

reflection file is temp.ahkl

!SPACE_GROUP_NUMBER= 19

Line 67:

Line 57:

frequency, i.e. number of unique reflections in resolution shells:

493 887 1134 1345 1501 1649 1808 1921 2061 2187

headings for lines starting with a,b,c:

a: <CC1/2> of each dataset:

Line 282:

Line 272:

INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0019/xds/XDS_ASCII.HKL

</pre>

== Explanation of output ==

The program output in the terminal window is terse but supposed to be self-explanatory; it can (and most often should) be saved or re-directed to a file.

xdscc12 ... > xdscc12.log # or xdscc12 ... | tee xdscc12.log

All statistics (tables) produced by XDSCC12 may be visualized with e.g. gnuplot, after grepping the relevant lines from the output.

If XDSCC12 is used with a XDS_ASCII.HKL reflection file (from XDS), the isomorphous delta-CC1/2 of a batch of frames (width chosen with the <code>-t</code> option; typically <code>-t 1</code>) relative to all data is most easily visualized via [[XDSGUI]] (Statistics tab). Negative numbers indicate a worsening of the overall signal.

If XDSCC12 is used with a XSCALE.HKL generated from multiple datasets, the output lines show the contribution of each dataset toward the total CC1/2. In this case, the program writes a file called XSCALE.INP.rename_me which shows statistics of delta-CC1/2 and delta-CC1/2-ano values, and has a sorted enumeration of the INPUT_FILEs - the first of these provides the best data set, and the last one is the worst one. This XSCALE.INP.rename_me can then be edited (i.e. for deleting a few data sets with strongly negative delta-CC1/2), and renamed to XSCALE.INP . Only delete the clearly worst data sets, and not more than 10% of the existing ones! This procedure can be iterated, i.e. after another round of XSCALE, XDSCC12 could be run again.

Overall statistics are reported in the lines starting with <code>a</code> and <code>d</code> for

* only those unique reflections that are actually present in the batch of frame / batch / dataset. These values are in columns 3-6.

* all unique reflections of the merged dataset (but a frame / batch / dataset may not have all unique reflections, so the "all" values report the mean influence). These values are in columns 7-10.

Typically, it is sensible to disregard the "all" values, and to base decisions on the "only" values, because the latter are not affected by the number of reflections of the particular frame / batch / dataset. The words "all" and "only" are used in this sense throughout the terminal and file output of XDSCC12.

Statistics for "only" the unique reflections of a frame / batch/ dataset are given in resolution shells for the isomorphous (in lines starting with <code>b</code> and <code>c</code>) and the anomalous signal (in lines starting with <code>d</code> and <code>e</code>). In case of [[SSX]] data (which have few reflections per data set, compared to complete data sets), we typically use <code>-nbin 1</code> as option, to define only a single resolution shell.

To find out about the influence of the ''a'' and ''b'' parameters of the XDS/XSCALE-adjusted error model, you may try the <code>-w</code> option; this assigns the same sigma to all reflections. Likewise, the [https://en.wikipedia.org/wiki/Fisher_transformation Fisher transformation], which serves to make changes in CC1/2 comparable across resolution ranges, may be switched off for testing purposes, with the <code>-z</code> option.

== Correlation against a reference data set (-r <reference> option) ==

@@ Line 2: / Line 2: @@
 XDSCC12 is a program for generating [[CC1/2|delta-CC<sub>1/2</sub>]] and delta-CC<sub>1/2-ano</sub> values for XDS_ASCII.HKL (written by [[XDS]]), or for XSCALE.HKL (written by [[XSCALE]]) containing data from several files of type XDS_ASCII.HKL after scaling (with MERGE=FALSE).
-It implements the method described in Assmann, Brehm and Diederichs (2016) Identification of rogue datasets in serial crystallography. J. Appl. Cryst. 49, 1021 [http://journals.iucr.org/j/issues/2016/03/00/zw5005/zw5005.pdf], and it does this not only for the individual datasets in XSCALE.HKL, but also for individual frames, or groups of frames, of a single dataset collected with the rotation method and processed by [[XDS]].
+It implements the method described in Assmann, Brehm and Diederichs (2016) Identification of rogue datasets in serial crystallography. J. Appl. Cryst. 49, 1021 [http://journals.iucr.org/j/issues/2016/03/00/zw5005/zw5005.pdf], and it does this not only for the individual datasets in XSCALE.HKL, but also for individual frames, or groups (batches) of frames, of a single dataset collected with the rotation method and processed by [[XDS]].
 The program can be downloaded for [https://{{SERVERNAME}}/pub/linux_bin/xdscc12 Linux] or [https://{{SERVERNAME}}/pub/mac_bin/xdscc12 Mac].
@@ Line 31: / Line 31: @@
 </pre>
-The program output in the terminal window is terse but supposed to be self-explanatory; it can (and most often should) be saved or re-directed to a file.
+== Example output ==
- xdscc12 ... > xdscc12.log  #  or xdscc12 ... | tee xdscc12.log
+<pre>
-All statistics (tables) produced by XDSCC12 may be visualized with e.g. gnuplot, after grepping the relevant lines from the output.
-If XDSCC12 is used with a XDS_ASCII.HKL reflection file (from XDS), the isomorphous delta-CC<sub>1/2</sub> of a batch of frames (width chosen with the <code>-t</code> option; typically <code>-t 1</code>) relative to all data is most easily visualized via [[XDSGUI]] (Statistics tab). Negative numbers indicate a worsening of the overall signal.
-If XDSCC12 is used with a XSCALE.HKL generated from multiple datasets, the output lines show the contribution of each dataset toward the total CC<sub>1/2</sub>. In this case, the program writes a file called XSCALE.INP.rename_me which shows statistics of delta-CC<sub>1/2</sub> and delta-CC<sub>1/2-ano</sub> values, and has a sorted enumeration of the INPUT_FILEs - the first of these provides the best data set, and the last one is the worst one. This XSCALE.INP.rename_me can then be edited (i.e. for deleting a few data sets with very negative delta-CC<sub>1/2</sub>), and renamed to XSCALE.INP.
-Statistics are given (in resolution shells) for the isomorphous and the anomalous signal. In case of [[SSX]] data (which have few reflections per data set, compared to complete data sets), we typically use <code>-nbin 1</code> as option.
-To find out about the influence of the ''a'' and ''b'' parameters of the XDS/XSCALE-adjusted error model, you may try the <code>-w</code> option; this assigns the same sigma to all reflections. Likewise, the [https://en.wikipedia.org/wiki/Fisher_transformation Fisher transformation], which serves to make changes in CC<sub>1/2</sub> comparable across resolution ranges, may be switched off for testing purposes, with the -z option.
-== Example output with explanation ==
   xdscc12 KD 2020-12-9. Academic use only; no redistribution. -h option shows options.
   Please cite Assmann, G., Brehm, W., Diederichs, K. (2016) J.Appl.Cryst. 49, 1021-1028
@@ Line 48: / Line 38: @@
    no option -w found, therefore statistics are weighted by sigma values
    no option -z found, therefore delta-CC1/2 values are Fisher-transformed
-<pre>
    reflection file is temp.ahkl
   !SPACE_GROUP_NUMBER=   19
@@ Line 67: / Line 57: @@
    frequency, i.e. number of unique reflections in resolution shells:
     887   1134   1345   1501   1649   1808   1921   2061   2187
    headings for lines starting with a,b,c:
   a:  <CC1/2> of each dataset:
@@ Line 282: / Line 272: @@
 INPUT_FILE=/scratch/data/JamesHolton_microfocus/2019/wedge0019/xds/XDS_ASCII.HKL
 </pre>
+== Explanation of output ==
+The program output in the terminal window is terse but supposed to be self-explanatory; it can (and most often should) be saved or re-directed to a file.
+ xdscc12 ... > xdscc12.log  #  or xdscc12 ... | tee xdscc12.log
+All statistics (tables) produced by XDSCC12 may be visualized with e.g. gnuplot, after grepping the relevant lines from the output.
+If XDSCC12 is used with a XDS_ASCII.HKL reflection file (from XDS), the isomorphous delta-CC<sub>1/2</sub> of a batch of frames (width chosen with the <code>-t</code> option; typically <code>-t 1</code>) relative to all data is most easily visualized via [[XDSGUI]] (Statistics tab). Negative numbers indicate a worsening of the overall signal.
+If XDSCC12 is used with a XSCALE.HKL generated from multiple datasets, the output lines show the contribution of each dataset toward the total CC<sub>1/2</sub>. In this case, the program writes a file called XSCALE.INP.rename_me which shows statistics of delta-CC<sub>1/2</sub> and delta-CC<sub>1/2-ano</sub> values, and has a sorted enumeration of the INPUT_FILEs - the first of these provides the best data set, and the last one is the worst one. This XSCALE.INP.rename_me can then be edited (i.e. for deleting a few data sets with strongly negative delta-CC<sub>1/2</sub>), and renamed to XSCALE.INP . Only delete the clearly worst data sets, and not more than 10% of the existing ones! This procedure can be iterated, i.e. after another round of XSCALE, XDSCC12 could be run again.
+Overall statistics are reported in the lines starting with <code>a</code> and <code>d</code> for
+* <b>only</b> those unique reflections that are actually present in the batch of frame / batch / dataset. These values are in columns 3-6.
+* <b>all</b> unique reflections of the merged dataset (but a frame / batch / dataset may not have all unique reflections, so the "all" values report the mean influence). These values are in columns 7-10.
+Typically, it is sensible to disregard the "all" values, and to base decisions on the "only" values, because the latter are not affected by the number of reflections of the particular frame / batch / dataset. The words "all" and "only" are used in this sense throughout the terminal and file output of XDSCC12.
+Statistics for "only" the unique reflections of a frame / batch/ dataset are given in resolution shells for the isomorphous (in lines starting with <code>b</code> and <code>c</code>) and the anomalous signal (in lines starting with <code>d</code> and <code>e</code>). In case of [[SSX]] data (which have few reflections per data set, compared to complete data sets), we typically use <code>-nbin 1</code> as option, to define only a single resolution shell.
+To find out about the influence of the ''a'' and ''b'' parameters of the XDS/XSCALE-adjusted error model, you may try the <code>-w</code> option; this assigns the same sigma to all reflections. Likewise, the [https://en.wikipedia.org/wiki/Fisher_transformation Fisher transformation], which serves to make changes in CC<sub>1/2</sub> comparable across resolution ranges, may be switched off for testing purposes, with the <code>-z</code> option.
 == Correlation against a reference data set (-r <reference> option) ==