XDSGUI: Difference between revisions

From XDSwiki
Jump to navigation Jump to search
(39 intermediate revisions by the same user not shown)
Line 36: Line 36:
Untrusted areas can be specified by the user, using two (UNTRUSTED_ELLIPSE; UNTRUSTED_RECTANGLE) or four (UNTRUSTED QUADRILATERAL) right mouse clicks. The resulting areas are shown with red outline, and the keyword/parameter pairs are shown in the XDS.INP tab. Step-by-step:
Untrusted areas can be specified by the user, using two (UNTRUSTED_ELLIPSE; UNTRUSTED_RECTANGLE) or four (UNTRUSTED QUADRILATERAL) right mouse clicks. The resulting areas are shown with red outline, and the keyword/parameter pairs are shown in the XDS.INP tab. Step-by-step:
* "Load" a raw frame or a CBF file (e.g. FRAME.cbf) to have it displayed. The pane can be "dragged" with the left mouse button; the mouse wheel zooms. The parameters in the XDS.INP tab are taken for resolution calculations (i.e. the frame header is not being interpreted in any way).
* "Load" a raw frame or a CBF file (e.g. FRAME.cbf) to have it displayed. The pane can be "dragged" with the left mouse button; the mouse wheel zooms. The parameters in the XDS.INP tab are taken for resolution calculations (i.e. the frame header is not being interpreted in any way).
* if XDS.INP does not yet exist, click "generate XDS.INP" (this ''will'' read the header). Check the XDS.INP tab afterwards but then go back to the Frame tab. Note that the current generate_XDS.INP works well for Pilatus, ADSC, Mar and some Rigaku detectors; for other kinds of detectors the values marked XXX in XDS.INP have to be filled in manually.
* if XDS.INP does not yet exist, click "generate XDS.INP" (this ''will'' read the header). Check the XDS.INP tab afterwards but then go back to the Frame tab. Note that the current generate_XDS.INP works well for Pilatus, ADSC, Mar and some Rigaku detectors; for other kinds of detectors the values marked XXX in XDS.INP have to be filled in manually. The [[Eiger]] detector is not yet implemented in the script, but this will eventually come.
* left-click on "Untrusted areas" -> a pulldown menu appears
* left-click on "Untrusted areas" -> a pulldown menu appears
* left-click on (say) "Untrusted Rectangle (2 clicks)"
* left-click on (say) "Untrusted Rectangle (2 clicks)"
Line 52: Line 52:


=== XYCORR ===
=== XYCORR ===
this just shows the output of the XYCORR step - normally not important to look at.
=== INIT ===
=== INIT ===
this just shows the output of the INIT step - you can find out e.g. what the average background counts are. Not crucial to look at, though.
=== COLSPOT ===
=== COLSPOT ===
=== IDXREF ===
this just shows the output of the COLSPOT step - normally not crucial to look at. But you could find out about the number of strong reflections on each frame of the SPOT_RANGE. This may be interesting to see radiation damage, or change of crystal volume in the beam.
 
=== [[IDXREF]] ===
detailed explanation of output: see [[IDXREF.LP]].
The right side of the tab shows the indexed (black) and not-indexed (red) reflections of [[SPOT.XDS]].
 
=== DEFPIX ===
=== DEFPIX ===
 
this just shows the output of the DEFPIX step - normally not important to look at.
each tab shows its .LP file, respectively.


=== INTEGRATE ===
=== INTEGRATE ===
Line 70: Line 78:
The border between the left and right side of the window may be adjusted with the mouse.<br>
The border between the left and right side of the window may be adjusted with the mouse.<br>


=== TOOLS ===
=== tools ===


offers possibilities for running short scripts.  
offers possibilities for running short scripts.  
Line 76: Line 84:
Several such scripts are pre-defined; the user may create her own scripts. If scripts are modified, they are saved to ~/.xds-gui . The names of the buttons (e.g. "User defined command 1") can be changed by editing ~/.xds-gui .
Several such scripts are pre-defined; the user may create her own scripts. If scripts are modified, they are saved to ~/.xds-gui . The names of the buttons (e.g. "User defined command 1") can be changed by editing ~/.xds-gui .


* The first item of the left panel ("Show frame with predicted spots") generates the predicted pattern of reflections for a user-specified frame, overlaid on the frame, for display with [[XDS-viewer]]. The file FRAME.cbf (produced by INTEGRATE) is renamed to FRAME_$X.cbf (where X is the user-specified frame number) and remains in the temp subdirectory. It may of course be opened in the FRAME tab, but starting XDS-viewer automatically has the advantage that several frames with predictions may be inspected on the screen, at the same time. Please note: if the XDS directory resides in a FAT32 filesystem (which is often the case on a USB stick or disk), then "ln -s" should be replaced by "cp -p" since FAT32 does not support symlinks. Also note: for the script to work correctly, NAME_TEMPLATE_OF_DATA_FRAMES in XDS.INP has to specify an absolute, not a relative path.
* The first item of the left panel ("Show frame with predicted spots") generates the predicted pattern of reflections for a user-specified frame, overlaid on the frame, for display with [[XDS-viewer]]. The file FRAME.cbf (produced by INTEGRATE) is renamed to FRAME_$X.cbf (where X is the user-specified frame number) and remains in the temp subdirectory. It may of course be opened in the FRAME tab, but starting XDS-viewer automatically has the advantage that several frames with predictions may be inspected on the screen, at the same time. Please note: if the XDS directory resides in a FAT32 filesystem (which is often the case on a USB stick or disk), then "ln -s" (of the script line) should be replaced by "cp -p" since FAT32 does not support symlinks. Also note: for the script to work correctly, NAME_TEMPLATE_OF_DATA_FRAMES in XDS.INP has to specify an absolute, not a relative path.


* The second item ("Optimizing data quality") offers commands that manipulate [[XDS.INP]] in several ways. Please note: the popup "XDS.INP has been changed externally" is emitted by the Qt system and cannot be switched off. It appears if one of the scripts changes XDS.INP while it is opened by XDSGUI (which is always the case) . Thus, one should simply press the "Reload" button. The user-definable commandline may be used e.g. for
* The second item ("Optimizing data quality") offers commands that manipulate [[XDS.INP]] in several ways. Please note: the popup "XDS.INP has been changed externally" is emitted by the Qt system and cannot be switched off. It appears if one of the scripts changes XDS.INP while it is opened by XDSGUI (which is always the case) . Thus, one should simply press the "Reload" button. The user-definable command may be used e.g. for
  grep -v "JOB=" XDS.INP > x; echo "JOB= DEFPIX INTEGRATE CORRECT" > XDS.INP; cat x >> XDS.INP
  awk '/BEAM_DIVERGENCE=/{print "",$1,2*$2,$3,$4}' INTEGRATE.LP>x; grep -v BEAM_DIVERGENCE XDS.INP>>x; mv x XDS.INP
This multiplies the estimate of BEAM_DIVERGENCE in INTEGRATE.LP by 2 and thus provides more pixels to the background estimation (for another INTEGRATE). This is useful for weakly exposed data sets which have mostly pixels with zero counts. Of course, any other factor (other than 2) may be tried.


* The third item ("Saving and comparing good results") offers commands to save/restore the current data processing files to/from a "save" directory. Make sure to replace "xdiff" with "xxdiff" or "tkdiff", if one of the latter is available.
* The third item ("Saving and comparing good results") offers commands to save/restore the current data processing files to/from a "save" directory. Make sure to replace "xdiff" with "xxdiff" or "tkdiff", if one of the latter is available.
Line 86: Line 95:
  phenix.xtriage XDS_ASCII.HKL
  phenix.xtriage XDS_ASCII.HKL


=== XDSSTAT ===
=== statistics ===


This tab allows to run [[XDSSTAT]] with a mouseclick. As soon as XDSSTAT.LP exists, the main plots derived from it are displayed. Another button allows to run [[XDS-viewer]] to visualize the control images that XDSSTAT generates, namely anom.pck, scales.pck, rf.pck, misfits.pck, corr.pck, rlps.pck, peaks.pck and nobs.pck .
This tab allows to run [[XDSSTAT]] and [[XDSCC12]] with a mouseclick.  
 
As soon as XDSSTAT.LP exists, the main plots derived from it are displayed (second and further panes). The "view" button allows to run [[XDS-viewer]] to visualize the control images that XDSSTAT generates, namely anom.pck, scales.pck, rf.pck, misfits.pck, corr.pck, rlps.pck, peaks.pck and nobs.pck .
 
Likewise, as soon as the output file XDSCC12.LP exists, a plot derived from those lines starting with "a:" and "b:" will be generated. In black, this shows the overall contribution of a frame, or of several frames pooled, to the CC<sub>1/2</sub> value of those reflections that this frame (or these frames) contributes to. The rainbow-coloured curves divide the total resolution range into a user-selectable number (chosen with the -nbin option of XDSCC12) of resolution ranges; blue is low resolution and red is high resolution.
Fine-sliced frames (0.1°) can be pooled into 1°-pseudoframes by using the -t option, to increase the number of reflections contributing to the plot and to make it less noisy.


=== XDSCONV ===
=== XDSCONV ===
Line 100: Line 114:
== Availability ==
== Availability ==


The program is under development and probably has bugs. If it crashes, it should simply be restarted. A crash of the program ''does not interfere'' with the operation of [[XDS]]; likewise, closing the program window does not influence any XDS run started from XDSGUI.  
The [http://sourceforge.net/projects/xdsgui/ program's source code] is released under the terms of the [https://gnu.org/licenses/old-licenses/gpl-2.0.txt GPLv2]; it uses the [http://www.qt.io/ Qt library] which was released under the terms of the [https://gnu.org/licenses/lgpl.txt LGPL]. The program is under development and probably has bugs. If it crashes, it should simply be restarted. A crash of the program ''does not interfere'' with the operation of [[XDS]]; likewise, closing the program window does not influence any XDS run started from XDSGUI.  


=== Dependencies ===
=== Dependencies ===


XDSGUI depends on [[generate_XDS.INP]], [[XDS-viewer]], and of course [[XDS]] (both the xds and xds_par binaries!), and [[XDSCONV]]. The XDSSTAT tab requires [[XDSSTAT]]. One of the items in the TOOLS tab calls a graphical file comparison program; if the default (xdiff) is not available, [http://furius.ca/xxdiff xxdiff] or [http://sourceforge.net/projects/tkdiff tkdiff] may be used - please adjust the commandline below the button accordingly.
XDSGUI depends on [[generate_XDS.INP]], [[XDS-viewer]], and of course [[XDS]] (both the xds and xds_par binaries!), [[XDSCONV]], and [[2cbf]]. The ''statistics'' tab requires [[XDSSTAT]] and [[XDSCC12]]. [[Eiger]] data processing requires <code>h5dump</code> and <code>H5ToXds</code> (or equivalent). One of the items in the <code>tools</code> tab calls a graphical file comparison program; if the default (xdiff) is not available, [http://furius.ca/xxdiff xxdiff] or [http://sourceforge.net/projects/tkdiff tkdiff] or [http://kdiff3.sourceforge.net/ kdiff3] or [http://meldmerge.org/ meld] may be used - please adjust the commandline below the button accordingly. See also [[Installation]].


Technically, the word "dependency" means that these scripts/programs should be in your $PATH - an 'alias' is often not good enough because the TOOLS commands use bash as a shell, and that shell does not inherit the aliases from e.g. (t)csh.
Technically, the word "dependency" means that these scripts/programs should be in your $PATH - an 'alias' is often not good enough because the ''TOOLS'' commands use <code>bash</code> as a shell, and that shell does not inherit the aliases from e.g. (t)csh.


=== Installation ===
=== Installation ===


The current version of the program can be downloaded, by academic users, for Linux [ftp://turn5.biologie.uni-konstanz.de/pub/xdsgui.rhel6.64 64bit] (compiled on RHEL6), [ftp://turn5.biologie.uni-konstanz.de/pub/xdsgui.rhel6.32 32bit] (compiled on RHEL6), [ftp://turn5.biologie.uni-konstanz.de/pub/xdsgui.rhel5.64 64bit] (compiled on RHEL5, statically linked with Qt), [ftp://turn5.biologie.uni-konstanz.de/pub/xdsgui.rhel5.32 32bit] (compiled on RHEL5, statically linked with Qt), or [ftp://turn5.biologie.uni-konstanz.de/pub/xdsgui.dmg Mac] (works on OSX 10.6 and up).  
The current version of the program can be downloaded, by academic users, for Linux [ftp://turn5.biologie.uni-konstanz.de/pub/xdsgui.rhel6.64 64bit] (compiled on RHEL6), [ftp://turn5.biologie.uni-konstanz.de/pub/xdsgui.rhel6.32 32bit] (compiled on RHEL6), or [ftp://turn5.biologie.uni-konstanz.de/pub/xdsgui.dmg Mac] (works on OSX 10.6 and up).  
Industrial users of XDS: pls contact me directly.
Industrial users of XDS: pls contact me directly.


Line 119: Line 133:
'''and to start the GUI from a terminal window, by typing xdsgui'''. In a similar way, XDS-viewer should be put into the $PATH under the name xds-viewer, e.g. using
'''and to start the GUI from a terminal window, by typing xdsgui'''. In a similar way, XDS-viewer should be put into the $PATH under the name xds-viewer, e.g. using
  sudo ln -s /Applications/XDS-Viewer.app/Contents/MacOS/xds-viewer-bin /usr/local/bin/xds-viewer
  sudo ln -s /Applications/XDS-Viewer.app/Contents/MacOS/xds-viewer-bin /usr/local/bin/xds-viewer
==== Libraries that the program depends on ====
In case of Fedora or Redhat Enterprise Linux distributions, you may have to
yum install qt qt-x11
if you get the error message
xdsgui: error while loading shared libraries: libQtGui.so.4: cannot open shared object file: No such file or directory
or similar for libQtCore.so.4.


'''Ubuntu 14.04''' has libmng2 instead of libmng. Therefore, on 32bit systems you should try
'''Ubuntu 14.04''' has libmng2 instead of libmng. Therefore, on 32bit systems you should try
Line 126: Line 147:
if you get the error message
if you get the error message
  xdsgui: error while loading shared libraries: libmng.so.1: cannot open shared object file: No such file or directory
  xdsgui: error while loading shared libraries: libmng.so.1: cannot open shared object file: No such file or directory
On the '''Mac''' (e.g. Yosemite) one needs [https://itunes.apple.com/us/app/xcode/id497799835?ls=1&mt=12 Xcode] installed, seemingly before XDSGUI is installed. Otherwise one may obtain a popup error “You can’t open the application because it is not supported on this type of Mac.”


== Known bugs, problems and workarounds ==
== Known bugs, problems and workarounds ==
# ''Question concerning TOOLS: "Show frame..."  on Mac: it creates everything shown in the commandline but it seem to us that everything runs in the background. Therefore xds-viewer does not open anywhere as it is not seen. Also, it would be great if this image together with the predictions would be re-directed to the FRAME tab.'' <br> Answer: On Linux, the xds-viewer window is brought to the foreground automatically, whereas on the Mac this does not seem to happen. However, an icon for xds-viewer appears in the dock (on the right) and I can double-click it to see the xds-viewer window. I have no idea how to bring xds-viewer to the foreground automatically, and I need input from people who know Macs and tell me how to do it. The reason why we do not open automatically in the FRAME tab is that one can have several xds-viewer windows open at the same time, and compare the patterns. As a '''workaround''', you can manually open the resulting temp/FRAME_$X.cbf in the FRAME tab.
# ''Question concerning TOOLS: "Show frame..."  on Mac: it creates everything shown in the commandline but it seem to us that everything runs in the background. Therefore xds-viewer does not open anywhere as it is not seen. Also, it would be great if this image together with the predictions would be re-directed to the FRAME tab.'' <br> Answer: On Linux, the xds-viewer window is brought to the foreground automatically, whereas on the Mac this does not seem to happen. However, an icon for xds-viewer appears in the dock (on the right) and I can double-click it to see the xds-viewer window. I have no idea how to bring xds-viewer to the foreground automatically, and I need input from people who know Macs and tell me how to do it. The reason why we do not open automatically in the FRAME tab is that one can have several xds-viewer windows open at the same time, and compare the patterns. As a '''workaround''', you can manually open the resulting temp/FRAME_$X.cbf in the FRAME tab.
# ''Another question concerning TOOLS: "Further analyses" on Mac: again pointless runs nicely but there is no output for the user.'' <br> Answer: Could it be that you started xdsgui by double-clicking its icon in the Finder? In that case, there is no output visible, because there is no console. Unfortunately, for all the Tools, the output is in the console window where xdsgui was started. We will try to find out how to open a window where the output is then shown. Thus, currently the '''workaround''' for this problem is to start xdsgui in a console window.
# ''Another question concerning TOOLS: "Further analyses" on Mac: again pointless runs nicely but there is no output for the user.'' <br> Answer: Could it be that you started xdsgui by double-clicking its icon in the Finder? In that case, there is no output visible, because there is no console. Unfortunately, for all the Tools, the output is in the console window where xdsgui was started. Thus, you should really start xdsgui in a console window.
# both generate_XDS.INP and [[XDS]] do not seem to like spaces ("blanks") in file/directory names. While frame names rarely have a space in them, directories might, e.g. if an external disk has the label "John's USB disk" then this would be mounted on Linux as /media/John's USB disk/ and that would result in a error message from generate_XDS.INP and/or XDS. The '''workaround''' is to use a symlink on a different hard disk. version 0.45 of generate_XDS.INP makes a start towards a fix for this problem, but XDS also will have to be adjusted.
# both generate_XDS.INP and [[XDS]] do not seem to like spaces ("blanks") in file/directory names. While frame names rarely have a space in them, directories might, e.g. if an external disk has the label "John's USB disk" then this would be mounted on Linux as /media/John's USB disk/ and that would result in a error message from generate_XDS.INP and/or XDS. The '''workaround''' is to use a symlink on a different hard disk.  
# ''Problem description: On the Mac, after loading frames, by clicking “generate XDS.INP”, the program gives some strange symbol “” in XDS.INP. And the more you click “save” button, the more “” appeared. This looks like <br>SPACE_GROUP_NUMBER=0  ! 0 if unknown <br>UNIT_CELL_CONSTANTS= 70 80 90 90 90 90  .''<br> The problem is due to the “Rich text” format in TextEdit when saving "generate_XDS.INP". It is solved by re-downloading the script, and changing format to Plain - everything is working now.
# ''Problem description: On the Mac, after loading frames, by clicking “generate XDS.INP”, the program gives some strange symbol “” in XDS.INP. And the more you click “save” button, the more “” appeared. This looks like <br>SPACE_GROUP_NUMBER=0  ! 0 if unknown <br>UNIT_CELL_CONSTANTS= 70 80 90 90 90 90  .''<br> The problem is due to the “Rich text” format in TextEdit when saving "generate_XDS.INP". It is solved by re-downloading the script, and changing format to Plain - everything is working now.
# on a Mac, you may find a stream of the following error messages in the console window: <code>Jan 31 14:35:17  xdsgui[20939] <Error>: Error: this application, or a library it uses, has passed an invalid numeric value (NaN, or not-a-number) to CoreGraphics API and this value is being ignored. Please fix this problem. Jan 31 14:35:17  xdsgui[20939] <Error>: If you want to see the backtrace, please set CG_NUMERICS_SHOW_BACKTRACE environmental variable.</code> This sounds terrible, but is actually harmless. Since it is a message from Qt (which XDSGUI uses for graphics), ''it does not compromise the XDS data processing in any way''. Just suppress the printout of these messages, by <code>xdsgui 2>&1 | egrep -v 'CoreGraphics|CG_NUMERICS'</code>


If you find a bug, please send email to Kay dot Diederichs at uni-konstanz dot de , ideally with enough information/data to reproduce the bug.
If you find a bug, please send email to Kay dot Diederichs at uni-konstanz dot de , with enough information/data to reproduce the bug.


== Limitations ==
== Limitations ==


* The program depends on [[generate_XDS.INP]] to interpret the header of the frames, so is currently limited to Dectris (Pilatus), ADSC (Quantum), Rigaku (several types) and MAR (CCD and image plate) detectors. Other detectors need some values to be manually filled into XDS.INP - the relevant places are marked with XXX. These are detector properties (type, pixel size and number, min and max counts in a pixel), and experimental parameters like oscillation range, wavelength, distance, and direct beam position (or rather: point of detector that is closest to the crystal). For fine-tuning of detector parameters, see the [http://xds.mpimf-heidelberg.mpg.de/html_doc/xds_prepare.html detector-specific templates].
* The program depends on [[generate_XDS.INP]] to interpret the header of the frames, so is currently limited to Dectris (Pilatus), ADSC (Quantum), Rigaku (several types), MAR (CCD and image plate) detectors, and the Bruker PHOTON II. Other detectors need some values to be manually filled into XDS.INP - the relevant places are marked with XXX. These are detector properties (type, pixel size and number, min and max counts in a pixel), and experimental parameters like oscillation range, wavelength, distance, and direct beam position (or rather: point of detector that is closest to the crystal). For fine-tuning of detector parameters, see the [http://xds.mpimf-heidelberg.mpg.de/html_doc/xds_prepare.html detector-specific templates].
* The authors of [[generate_XDS.INP]] have made a "best effort" to provide a XDS.INP that results in the correct sign of the anomalous signal. In the case of one detector type (internally called Rigaku SMV) this requires reversal of one detector axis, and a negative DETECTOR_DISTANCE, as is found in some of the [http://xds.mpimf-heidelberg.mpg.de/html_doc/xds_prepare.html detector-specific templates]. '''For an unusual or unknown detector setup, the correct sign of the anomalous signal needs to be established and verified e.g. with a good dataset from a test crystal that has a anomalous signal.''' The authors do not take any responsibility for problems arising from incorrect sign of the anomalous signal, nor - obviously! - for any other mischief arising in or from data processing.
* The authors of [[generate_XDS.INP]] have made a "best effort" to provide a XDS.INP that results in the correct sign of the anomalous signal. In the case of one detector type (internally called Rigaku SMV) this requires reversal of one detector axis, and a negative DETECTOR_DISTANCE, as is found in some of the [http://xds.mpimf-heidelberg.mpg.de/html_doc/xds_prepare.html detector-specific templates]. '''For an unusual or unknown detector setup, the correct sign of the anomalous signal needs to be established and verified e.g. with a good dataset from a test crystal that has a significant anomalous signal.''' The authors do not take any responsibility for problems arising from incorrect sign of the anomalous signal, nor - obviously! - for any other mischief arising in or from data processing.
* At some beamlines, the ROTATION_AXIS should be -1 0 0 ("backwards") instead of the usual 1 0 0 ("horizontal"), or even 0 1 0 ("vertical") like at one of the PETRA Hamburg BLs. We have no exhaustive list of these beamlines, and the frame headers do not have this information, so the default chosen by [[generate_XDS.INP]] may be wrong and need manual correction.
* At some beamlines, the ROTATION_AXIS should be -1 0 0 ("backwards") instead of the usual 1 0 0 ("horizontal"), or even 0 1 0 ("vertical") like at one of the PETRA Hamburg BLs, or even 0 -1 0 (vertical and backwards) like for some measurements at Diamond's ID24. We have no exhaustive list of these beamlines, and the frame headers ususally do not have this information, so the default chosen by [[generate_XDS.INP]] may be wrong and need manual correction.
* The display of large frames in the FRAME tab may be slow over the network.
* The display of large frames in the FRAME tab may be slow over the network.
* Eiger data may be processed like this: a) run <code>generate_XDS.INP /path/to/XXX_master.h5</code> in an empty directory, to produce XDS.INP. b) start XDSGUI in the same directory (or navigate there, from its ''Projects'' tab) c) you will not be able to use the ''Frame'' tab for looking at the raw data, but you could use the tab to look at FRAME.cbf or other XDS-produced images. d) you should use/check/modify XDS.INP as usual; all the other tabs work normally.
* The green INCLUDE_RESOLUTION and red semi-transparent EXCLUDE_RESOLUTION_RANGE circles in the Frame tab as well as all resolution values in the Frame and IDXREF tabs are calculated from values in XDS.INP (rather than from the refined values in XPARM.XDS, which XDS uses). For the (curved) Pilatus 12M, the calculations in XDSGUI do not give the correct results.


== News ==
== News ==
Line 159: Line 185:
update June 2, 2014: fix bugs
update June 2, 2014: fix bugs


update December 11, 2014: fix bugs; SETTING SYMMETRY-BASED for [[pointless]] (in the TOOLS tab); radiation damage (R_d) plot with recommended limit shown in the XDSSTAT tab
update Sep 17, 2015: belatedly updated xdsgui.rhel6.* - this version correctly visualizes the tables of INTEGRATE.LP which have a different format since XDS version June-2015
update May 20, 2016: many bug fixes, rename XDSSTAT tab to <code>statistics</code> tab, include delta-CC<sub>1/2</sub> plot.
update Jan 31, 2017: many bug fixes. Now works with Eiger HDF5 - select the *master.h5 file in the FRAME tab.


== See also ==
== See also ==


[[Installation]]
[[Installation]]

Revision as of 16:34, 6 February 2017

XDSGUI is a GUI (graphical user interface) for XDS that is supposed to help both novice and experienced users. It graphically displays the ASCII and cbf files that XDS writes, and can run useful shell commands with a simple mouse click. The design goal of the program is to enable XDS data processing without the commandline, and to supply additional graphical information, in a simple, user-modifiable and user-extensible way.

How to use XDSGUI

The program may be used to analyze graphically any existing directory with files earlier written by XDS, or may be used to create these files. It should be started in a terminal window, which will later show the output of the programs that are run (XDS, pointless, XDSCONV, ...).

The user is supposed to use the tabs on the top, from left to right. Thus, one starts with the choice of the processing directory (Projects tab), and then advances to the Frame tab to display a raw dataframe or a .cbf file written by XDS or a .pck file written by XDSSTAT. Within the Frame tab, XDS.INP may be generated; for this to work, a raw dataframe has to be displayed. While still in the Frame tab, untrusted (shaded) areas may be marked with the mouse which will write the appropriate lines to XDS.INP; ORGX ORGY may be corrected if necessary, and the INCLUDE_RESOLUTION_RANGE, EXCLUDE_RESOLUTION_RANGE and TRUSTED_REGION adjusted as required. The latter operations are interactive in the sense that the user modifies the values in the XDS.INP tab, and then checks the result in the Frame tab. In other words, the user moves back and forth between the Frame and the XDS.INP tabs.

After finally moving to the XDS.INP tab, other parameters can be adjusted by the user, and finally XDS may be run by clicking a button. The resulting output files from XDS will then be displayed in the next tabs.

After finishing this first round of processing, the TOOLS tab may be used, which - among other things - offers those three options that I found most useful to optimize the data processing. After choosing (by mouse click) one of these three options, the user should go back to the XDS.INP tab, specify JOBS=DEFPIX INTEGRATE CORRECT, and run XDS again. Ideally, each of the three options should be tried separately, and its effect should be compared with the previous processing to verify that it really improved the processed data. A significant increase in ISa (> 1%) indicates that the processing has improved; a slight decrease in ISa (<1%) often is accompanied by an increase of CC1/2 at high resolution, and thus should be tolerated. Those options that improve ISa can then be used in combination.

After CORRECT, the data may be analyzed using the XDSSTAT tab, and the resulting XDS_ASCII.HKL may be converted to e.g. MTZ, using the XDSCONV tab.

Technical aspects of XDSGUI usage

  • the simple editor operating in the XDS.INP, XDSCONV and XSCALE tabs understands ctrl-X (cut), ctrl-c (copy), ctrl-v (paste), ctrl-z (undo) and ctrl-Z (redo).
  • ctrl-+ (control-plus) and ctrl-- (control-minus) enlarge or decrease the font size of characters (available since April 22, 2014).
  • flexibility of operation: all commands in the TOOLS tab may be modified by the user, simply by modifying the (bash) code in the text field below the button. The user-changed command is automagically and permanently stored in ~/.xds-gui . The user may also change what is written on a button, by editing ~/.xds-gui.

Tabs

The main window of XDSgui has the following sub-windows organized as tabs:

Projects

stores names of directories which contain files from XDS processing. If none of these is selected, the current directory is used. A new directory may be created. The list of project directories is saved to ~/.xds-gui . A project may be removed from the list by right-clicking a project and selecting "hide", or by editing ~/.xds-gui .

Frame

to display existing 2-dimensional data frames, or to start XDS data processing by displaying a raw data frame. In the latter case, the "generate XDS.INP" button runs the generate_XDS.INP script to create a first XDS.INP from the header of the raw data frame that is being displayed. (Note: this button needs to be used only once per dataset; all later changes to XDS.INP are done by different means - manually or scripted) The blue circle(s) delineate(s) the area(s) of the detector within TRUSTED_REGION; the green circles correspond to INCLUDE_RESOLUTION_RANGE, and the red hatched regions correspond to EXCLUDE_RESOLUTION_RANGEs. The locations of the circles are not as accurate as those that XDS uses internally, because a simplified formula (and only values from XDS.INP, not the refined ones from XPARM.XDS) is used to calculate the resolution; this e.g. does not take care of detector swingout or otherwise skew geometry.

Untrusted areas can be specified by the user, using two (UNTRUSTED_ELLIPSE; UNTRUSTED_RECTANGLE) or four (UNTRUSTED QUADRILATERAL) right mouse clicks. The resulting areas are shown with red outline, and the keyword/parameter pairs are shown in the XDS.INP tab. Step-by-step:

  • "Load" a raw frame or a CBF file (e.g. FRAME.cbf) to have it displayed. The pane can be "dragged" with the left mouse button; the mouse wheel zooms. The parameters in the XDS.INP tab are taken for resolution calculations (i.e. the frame header is not being interpreted in any way).
  • if XDS.INP does not yet exist, click "generate XDS.INP" (this will read the header). Check the XDS.INP tab afterwards but then go back to the Frame tab. Note that the current generate_XDS.INP works well for Pilatus, ADSC, Mar and some Rigaku detectors; for other kinds of detectors the values marked XXX in XDS.INP have to be filled in manually. The Eiger detector is not yet implemented in the script, but this will eventually come.
  • left-click on "Untrusted areas" -> a pulldown menu appears
  • left-click on (say) "Untrusted Rectangle (2 clicks)"
  • move mouse pointer to one corner of desired rectangle and right-click.
  • move mouse pointer (... a changing rectangle appears ...) to opposite corner and right-click: the rectangle is now fixed, and the UNTRUSTED_RECTANGLE keyword together with its parameters appears in XDS.INP tab, in red letters (you do not have to activate the XDS.INP tab to check it, but of course you could).
  • with two more right-clicks you get another rectangle, and so on
  • you can choose "Untrusted Ellipse (2 clicks)" or "Untrusted Quadrilateral (4 clicks)" and these work in exactly the same way
  • if you remove a UNTRUSTED_* line from XDS.INP then its shape will be removed from the Frame tab.

Unfortunately this interactive mode of establishing untrusted areas does not currently work well over a ssh (or NXclient) connection, probably because the lines/circles are re-drawn at high frequency. So it is recommended to run the program locally.

XDS.INP

this shows the current XDS.INP file. Coloured items correspond to the circles of the FRAME tab. Changes can be made with a simple editor, and the new version can be saved. A run of xds_par can be started (and killed).
The XDS.INP tab is directly connected with the FRAME tab; changing a value in the XDS.INP tab results in immediate change in the FRAME tab. This means that e.g. an UNTRUSTED area that was positioned wrongly can be edited or simply removed in the XDS.INP tab.
The origin of XDS.INP does not matter - it may e.g. be derived from a detector template, or written by the beamline software, or obtained by clicking the "generate_XDS.INP" button in the FRAME tab.

XYCORR

this just shows the output of the XYCORR step - normally not important to look at.

INIT

this just shows the output of the INIT step - you can find out e.g. what the average background counts are. Not crucial to look at, though.

COLSPOT

this just shows the output of the COLSPOT step - normally not crucial to look at. But you could find out about the number of strong reflections on each frame of the SPOT_RANGE. This may be interesting to see radiation damage, or change of crystal volume in the beam.

IDXREF

detailed explanation of output: see IDXREF.LP. The right side of the tab shows the indexed (black) and not-indexed (red) reflections of SPOT.XDS.

DEFPIX

this just shows the output of the DEFPIX step - normally not important to look at.

INTEGRATE

shows INTEGRATE.LP on the left side. During the INTEGRATE step, the temporary files are displayed. On the right side, plots of the tabular quantities given for each frame, or for each batch of data are shown. For the refined quantities, only the changes are plotted. The beam divergence E.S.D. and mosaicity E.S.D. plots show these quantities estimated from each frame (blue), refined for each batch (green) and averaged over the whole dataset (red horizontal line).
The border between the left and right side of the window may be adjusted with the mouse.

CORRECT

shows CORRECT.LP on the left, and plots of tabular quantities on the right. The upper line in the I/sigma (unmerged data) plot gives [math]\displaystyle{ {I/Sigma(I)}^{asymptotic} }[/math] ([Diederichs, Acta Cryst. (2010). D66, 733-740 http://dx.doi.org/10.1107/S0907444910014836]). The three lines in the Chi^2 plot are the Chi^2 values obtained during scaling, for the ABSORP, MODPIX and DECAY corrections, respectively.
In the lower plots, the colored lines correspond to the nine resolution shells for which values are found in the tables.
The border between the left and right side of the window may be adjusted with the mouse.

tools

offers possibilities for running short scripts.

Several such scripts are pre-defined; the user may create her own scripts. If scripts are modified, they are saved to ~/.xds-gui . The names of the buttons (e.g. "User defined command 1") can be changed by editing ~/.xds-gui .

  • The first item of the left panel ("Show frame with predicted spots") generates the predicted pattern of reflections for a user-specified frame, overlaid on the frame, for display with XDS-viewer. The file FRAME.cbf (produced by INTEGRATE) is renamed to FRAME_$X.cbf (where X is the user-specified frame number) and remains in the temp subdirectory. It may of course be opened in the FRAME tab, but starting XDS-viewer automatically has the advantage that several frames with predictions may be inspected on the screen, at the same time. Please note: if the XDS directory resides in a FAT32 filesystem (which is often the case on a USB stick or disk), then "ln -s" (of the script line) should be replaced by "cp -p" since FAT32 does not support symlinks. Also note: for the script to work correctly, NAME_TEMPLATE_OF_DATA_FRAMES in XDS.INP has to specify an absolute, not a relative path.
  • The second item ("Optimizing data quality") offers commands that manipulate XDS.INP in several ways. Please note: the popup "XDS.INP has been changed externally" is emitted by the Qt system and cannot be switched off. It appears if one of the scripts changes XDS.INP while it is opened by XDSGUI (which is always the case) . Thus, one should simply press the "Reload" button. The user-definable command may be used e.g. for
awk '/BEAM_DIVERGENCE=/{print "",$1,2*$2,$3,$4}' INTEGRATE.LP>x; grep -v BEAM_DIVERGENCE XDS.INP>>x; mv x XDS.INP

This multiplies the estimate of BEAM_DIVERGENCE in INTEGRATE.LP by 2 and thus provides more pixels to the background estimation (for another INTEGRATE). This is useful for weakly exposed data sets which have mostly pixels with zero counts. Of course, any other factor (other than 2) may be tried.

  • The third item ("Saving and comparing good results") offers commands to save/restore the current data processing files to/from a "save" directory. Make sure to replace "xdiff" with "xxdiff" or "tkdiff", if one of the latter is available.
  • The last item ("Further analyses") may be used for commands, e.g. running pointless against the XDS_ASCII.HKL file. The user-definable commandline may be used e.g. for
phenix.xtriage XDS_ASCII.HKL

statistics

This tab allows to run XDSSTAT and XDSCC12 with a mouseclick.

As soon as XDSSTAT.LP exists, the main plots derived from it are displayed (second and further panes). The "view" button allows to run XDS-viewer to visualize the control images that XDSSTAT generates, namely anom.pck, scales.pck, rf.pck, misfits.pck, corr.pck, rlps.pck, peaks.pck and nobs.pck .

Likewise, as soon as the output file XDSCC12.LP exists, a plot derived from those lines starting with "a:" and "b:" will be generated. In black, this shows the overall contribution of a frame, or of several frames pooled, to the CC1/2 value of those reflections that this frame (or these frames) contributes to. The rainbow-coloured curves divide the total resolution range into a user-selectable number (chosen with the -nbin option of XDSCC12) of resolution ranges; blue is low resolution and red is high resolution. Fine-sliced frames (0.1°) can be pooled into 1°-pseudoframes by using the -t option, to increase the number of reflections contributing to the plot and to make it less noisy.

XDSCONV

a simple XDSCONV.INP is provided by default. It may be edited by the user, and saved. Upon clicking a button, xdsconv is run and if necessary a MTZ file is produced. XDSCONV.LP may be displayed.

XSCALE

a simple XSCALE.INP is provided by default. It may be edited by the user, and saved. Upon clicking a button, xscale is run. XSCALE.LP may be displayed.

Availability

The program's source code is released under the terms of the GPLv2; it uses the Qt library which was released under the terms of the LGPL. The program is under development and probably has bugs. If it crashes, it should simply be restarted. A crash of the program does not interfere with the operation of XDS; likewise, closing the program window does not influence any XDS run started from XDSGUI.

Dependencies

XDSGUI depends on generate_XDS.INP, XDS-viewer, and of course XDS (both the xds and xds_par binaries!), XDSCONV, and 2cbf. The statistics tab requires XDSSTAT and XDSCC12. Eiger data processing requires h5dump and H5ToXds (or equivalent). One of the items in the tools tab calls a graphical file comparison program; if the default (xdiff) is not available, xxdiff or tkdiff or kdiff3 or meld may be used - please adjust the commandline below the button accordingly. See also Installation.

Technically, the word "dependency" means that these scripts/programs should be in your $PATH - an 'alias' is often not good enough because the TOOLS commands use bash as a shell, and that shell does not inherit the aliases from e.g. (t)csh.

Installation

The current version of the program can be downloaded, by academic users, for Linux 64bit (compiled on RHEL6), 32bit (compiled on RHEL6), or Mac (works on OSX 10.6 and up). Industrial users of XDS: pls contact me directly.

In case of Linux, after downloading the binary file, its name should be changed to "xdsgui" and then be made executable ("chmod a+x xdsgui"). The binary may then be copied into e.g. the directory /usr/local/bin by the administrator, or to the $HOME/bin directory by the user. Any other directory should be suitable as long as it is in your $PATH.

In case of a Mac, the DMG file may be installed in the usual way, by double-clicking it and dragging the icon into the Applications folder. It is advantageous to place a symbolic link, like

sudo ln -s /Applications/xdsgui.app/Contents/MacOS/xdsgui /usr/local/bin/xdsgui

and to start the GUI from a terminal window, by typing xdsgui. In a similar way, XDS-viewer should be put into the $PATH under the name xds-viewer, e.g. using

sudo ln -s /Applications/XDS-Viewer.app/Contents/MacOS/xds-viewer-bin /usr/local/bin/xds-viewer

Libraries that the program depends on

In case of Fedora or Redhat Enterprise Linux distributions, you may have to

yum install qt qt-x11

if you get the error message

xdsgui: error while loading shared libraries: libQtGui.so.4: cannot open shared object file: No such file or directory

or similar for libQtCore.so.4.

Ubuntu 14.04 has libmng2 instead of libmng. Therefore, on 32bit systems you should try

sudo ln -s /usr/lib/i386-linux-gnu/libmng.so.2 /usr/lib/i386-linux-gnu/libmng.so.1 

and on 64bit systems

sudo ln -s /usr/lib/x86_64-linux-gnu/libmng.so.2 /usr/lib/x86_64-linux-gnu/libmng.so.1 

if you get the error message

xdsgui: error while loading shared libraries: libmng.so.1: cannot open shared object file: No such file or directory

On the Mac (e.g. Yosemite) one needs Xcode installed, seemingly before XDSGUI is installed. Otherwise one may obtain a popup error “You can’t open the application because it is not supported on this type of Mac.”

Known bugs, problems and workarounds

  1. Question concerning TOOLS: "Show frame..." on Mac: it creates everything shown in the commandline but it seem to us that everything runs in the background. Therefore xds-viewer does not open anywhere as it is not seen. Also, it would be great if this image together with the predictions would be re-directed to the FRAME tab.
    Answer: On Linux, the xds-viewer window is brought to the foreground automatically, whereas on the Mac this does not seem to happen. However, an icon for xds-viewer appears in the dock (on the right) and I can double-click it to see the xds-viewer window. I have no idea how to bring xds-viewer to the foreground automatically, and I need input from people who know Macs and tell me how to do it. The reason why we do not open automatically in the FRAME tab is that one can have several xds-viewer windows open at the same time, and compare the patterns. As a workaround, you can manually open the resulting temp/FRAME_$X.cbf in the FRAME tab.
  2. Another question concerning TOOLS: "Further analyses" on Mac: again pointless runs nicely but there is no output for the user.
    Answer: Could it be that you started xdsgui by double-clicking its icon in the Finder? In that case, there is no output visible, because there is no console. Unfortunately, for all the Tools, the output is in the console window where xdsgui was started. Thus, you should really start xdsgui in a console window.
  3. both generate_XDS.INP and XDS do not seem to like spaces ("blanks") in file/directory names. While frame names rarely have a space in them, directories might, e.g. if an external disk has the label "John's USB disk" then this would be mounted on Linux as /media/John's USB disk/ and that would result in a error message from generate_XDS.INP and/or XDS. The workaround is to use a symlink on a different hard disk.
  4. Problem description: On the Mac, after loading frames, by clicking “generate XDS.INP”, the program gives some strange symbol “” in XDS.INP. And the more you click “save” button, the more “” appeared. This looks like
    SPACE_GROUP_NUMBER=0  ! 0 if unknown
    UNIT_CELL_CONSTANTS= 70 80 90 90 90 90 Â .

    The problem is due to the “Rich text” format in TextEdit when saving "generate_XDS.INP". It is solved by re-downloading the script, and changing format to Plain - everything is working now.
  5. on a Mac, you may find a stream of the following error messages in the console window: Jan 31 14:35:17 xdsgui[20939] <Error>: Error: this application, or a library it uses, has passed an invalid numeric value (NaN, or not-a-number) to CoreGraphics API and this value is being ignored. Please fix this problem. Jan 31 14:35:17 xdsgui[20939] <Error>: If you want to see the backtrace, please set CG_NUMERICS_SHOW_BACKTRACE environmental variable. This sounds terrible, but is actually harmless. Since it is a message from Qt (which XDSGUI uses for graphics), it does not compromise the XDS data processing in any way. Just suppress the printout of these messages, by xdsgui 2>&1 | egrep -v 'CoreGraphics|CG_NUMERICS'

If you find a bug, please send email to Kay dot Diederichs at uni-konstanz dot de , with enough information/data to reproduce the bug.

Limitations

  • The program depends on generate_XDS.INP to interpret the header of the frames, so is currently limited to Dectris (Pilatus), ADSC (Quantum), Rigaku (several types), MAR (CCD and image plate) detectors, and the Bruker PHOTON II. Other detectors need some values to be manually filled into XDS.INP - the relevant places are marked with XXX. These are detector properties (type, pixel size and number, min and max counts in a pixel), and experimental parameters like oscillation range, wavelength, distance, and direct beam position (or rather: point of detector that is closest to the crystal). For fine-tuning of detector parameters, see the detector-specific templates.
  • The authors of generate_XDS.INP have made a "best effort" to provide a XDS.INP that results in the correct sign of the anomalous signal. In the case of one detector type (internally called Rigaku SMV) this requires reversal of one detector axis, and a negative DETECTOR_DISTANCE, as is found in some of the detector-specific templates. For an unusual or unknown detector setup, the correct sign of the anomalous signal needs to be established and verified e.g. with a good dataset from a test crystal that has a significant anomalous signal. The authors do not take any responsibility for problems arising from incorrect sign of the anomalous signal, nor - obviously! - for any other mischief arising in or from data processing.
  • At some beamlines, the ROTATION_AXIS should be -1 0 0 ("backwards") instead of the usual 1 0 0 ("horizontal"), or even 0 1 0 ("vertical") like at one of the PETRA Hamburg BLs, or even 0 -1 0 (vertical and backwards) like for some measurements at Diamond's ID24. We have no exhaustive list of these beamlines, and the frame headers ususally do not have this information, so the default chosen by generate_XDS.INP may be wrong and need manual correction.
  • The display of large frames in the FRAME tab may be slow over the network.
  • Eiger data may be processed like this: a) run generate_XDS.INP /path/to/XXX_master.h5 in an empty directory, to produce XDS.INP. b) start XDSGUI in the same directory (or navigate there, from its Projects tab) c) you will not be able to use the Frame tab for looking at the raw data, but you could use the tab to look at FRAME.cbf or other XDS-produced images. d) you should use/check/modify XDS.INP as usual; all the other tabs work normally.
  • The green INCLUDE_RESOLUTION and red semi-transparent EXCLUDE_RESOLUTION_RANGE circles in the Frame tab as well as all resolution values in the Frame and IDXREF tabs are calculated from values in XDS.INP (rather than from the refined values in XPARM.XDS, which XDS uses). For the (curved) Pilatus 12M, the calculations in XDSGUI do not give the correct results.

News

update June 19, 2013: Fixes for TRUSTED_REGION on Pilatus, and for XDSCONV tab (thanks Folmer!).

update July 5, 2013: INTEGRATE plot showing crystal rotation; several bugfixes. Commandline name is now xdsgui.

update November 26, 2013: new XSCALE tab (rudimentary at present). PDF and PNG versions of all plots can be produced by right mouse click (PDF file lets you choose the output file name with a filesystem browser; PNG file is simply written as "output.png" to the current directory. The latter "feature" will be changed to also use a filesystem browser soon). Many bugfixes.

update Jan 10, 2014: fix crashes on startup.

update Jan 23, 2014: fix crashes (most notably coredump on Mac in CORRECT tab), and several other bugs.

update Mar 19, 2014: fix bugs, and make somewhat faster over slow connection.

update May 12, 2014: fix bugs; brightness, contrast and zoom remember their values between frames; ctrl-+ (Control and plus sign pressed together) and ctrl-- (Control and minus sign pressed together) change font size in the tabs; the direct beam position is now indicated with a "+" in the Frame tab, whereas ORGX ORGY are shown with a "X" (these are different if the detector is swung out or the beam is slightly non-perpendicular to the detector). Individual projects may be removed from the "Projects" tab with right mouse click / hide.

update June 2, 2014: fix bugs

update December 11, 2014: fix bugs; SETTING SYMMETRY-BASED for pointless (in the TOOLS tab); radiation damage (R_d) plot with recommended limit shown in the XDSSTAT tab

update Sep 17, 2015: belatedly updated xdsgui.rhel6.* - this version correctly visualizes the tables of INTEGRATE.LP which have a different format since XDS version June-2015

update May 20, 2016: many bug fixes, rename XDSSTAT tab to statistics tab, include delta-CC1/2 plot.

update Jan 31, 2017: many bug fixes. Now works with Eiger HDF5 - select the *master.h5 file in the FRAME tab.

See also

Installation