Installation: Difference between revisions

From XDSwiki
Jump to navigation Jump to search
No edit summary
 
(166 intermediate revisions by 3 users not shown)
Line 1: Line 1:
This article has some little script snippets that should ease the installation of XDS and related programs ([[XDS-viewer]], [[xdsstat]], [[XDSGUI|xdsgui]]). It is assumed that binaries (or links to binaries) should go to /usr/local/bin .
This article has information that should ease the installation of XDS and related programs like ([[XDS-viewer]], [[xdsstat]], [[XDSGUI|xdsgui]]). It is assumed that binaries (or links to binaries) should go to /usr/local/bin . This means that you should do this as the administrator (root), or have sudo rights (Mac, and some Linux distros).




== Linux ==
== Linux ==
Log in as root - we need write permission for /usr/local/bin .
'''Log in as root''' - we need write permission for /usr/local/bin .


To get the [[XDS]] package if you are an academic user,
=== [[XDS]] package ===
If you are an academic user, as root (on Ubuntu, use "sudo -i" to become root)
<pre>
<pre>
  cd /usr/local/bin
  cd /usr/local/bin
  wget -O- ftp://ftp.mpimf-heidelberg.mpg.de/pub/kabsch/XDS-INTEL64_Linux_x86_64.tar.gz | tar xzvf -
  wget --no-check-certificate -O- https://xds.mr.mpg.de/XDS-INTEL64_Linux_x86_64.tar.gz | tar xzvf -
  ln -sf XDS-INTEL64_Linux_x86_64/* .
  ln -sf XDS-INTEL64_Linux_x86_64/* .
</pre>
</pre>
To get the script [[generate_XDS.INP]] from its XDSwiki article,
In case the above website is unavailable, there is a [https://{{SERVERNAME}}/pub/xds backup site].
<pre>
 
cd /usr/local/bin
If you want to process data from the [[Eiger]] detector: see below for plugins that allow XDS to read the .h5 files. Similarly for gzip-ed CBF files as mostly collected at EMBL Hamburg.
wget http://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/generate_XDS.INP -O - | \
 
  sed -e s/\&nbsp\;/\ /g -e s/\&gt\;/\>/g -e s/\&lt\;/\</g -e s/amp\;//g -e s/\&quot\;/\"/g -e s/\&\#\1\6\0\;/\ /g | \
=== other programs: [[XDS-Viewer]], [[XDSSTAT]], [[XDSCC12]], [[XDSGUI]], [[XSCALE_ISOCLUSTER]], [[spot2pdb]], [[generate_XDS.INP]]  ===
  sed '/# end of generate_XDS.INP/,$d' | awk '/^#/,/rm -f tmp1 tmp2/' > generate_XDS.INP
... can be downloaded from https://{{SERVERNAME}}/pub/linux_bin/ . A simple way to obtain them is to create an empty folder, "cd" to that folder and then
  chmod +x generate_XDS.INP
wget -N https://{{SERVERNAME}}/pub/linux_bin/get_folder.sh
</pre>
  chmod +x get_folder.sh
Please note that to get the <code>strings</code> command, which is used by [[generate_XDS.INP]], on some Linux distributions (e.g. FC23) you need to install the <code>binutils</code> RPM package.
./get_folder.sh
You may repeat this every now and then to update the programs.
 
As root, the script copies the programs into /usr/local/bin/linux_bin, and links them to /usr/local/bin. As a normal user, they are downloaded to $HOME/bin/linux_bin, and linked to your $HOME/bin. In the latter case, make sure that $HOME/bin is in your $PATH.


To get [[XDS-Viewer]],
=== tools and helper programs for [[generate_XDS.INP]] and [[XDSGUI]], and libraries ===
<pre>
* check and installation of Unix tools for [[generate_XDS.INP]] is shown at [[Generate_XDS.INP#Dependencies]]
cd /usr/local/bin
* helper programs for [[XDSGUI]] are listed and discussed at [[XDSGUI#Dependencies]]. Nota bene: XDSGUI needs [[generate_XDS.INP]] unless you already have a working XDS.INP.  
wget ftp://turn5.biologie.uni-konstanz.de/pub/xds-viewer-0.6.64bit -O xds-viewer
chmod a+x xds-viewer 
ln -sf xds-viewer xdsviewer
</pre>
To get [[XDSSTAT]],
<pre>
cd /usr/local/bin
wget ftp://turn5.biologie.uni-konstanz.de/pub/xdsstat-linux64.bz2 -O xdsstat.bz2
bunzip2 -f xdsstat.bz2
chmod a+x xdsstat
</pre>
To get [[XDSCC12]],
<pre>
cd /usr/local/bin
wget ftp://turn5.biologie.uni-konstanz.de/pub/xdscc12.rhel6.64 -O xdscc12
chmod a+x xdscc12
</pre>
To get [[XDSGUI]],
<pre>
cd /usr/local/bin
wget ftp://turn5.biologie.uni-konstanz.de/pub/xdsgui.rhel6.64 -O xdsgui
chmod a+x xdsgui
</pre>


Since XDSGUI depends on graphics packages that may need to be installed, you can check the xdsgui binary with
Since XDSGUI depends on graphics packages that may need to be installed, you can check the xdsgui binary with
  ldd `which xdsgui`
  ldd `which xdsgui` | grep found
and that should show you the libraries it found, and more importantly, those that it didn't find. Your friendly system administrator will then work out the specific commands to install those libraries; on RedHat-type distributions that would typically be e.g. <code>yum provides libXfixes.so.3</code> the output of which will tell you that this is in the libXfixes RPM.
and that should show you the libraries it didn't find. Your friendly system administrator knows the specific commands to install those libraries.
Ubuntu 14 users: please read http://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/XDSGUI#Installation if you have a problem with missing libmng2.
 
N.B. this may list libpng15.so for the latest XDSGUI (compiled May 2023), but you can ignore that!
 
For RedHat-type distributions, that would typically be e.g.  
yum provides libXfixes.so.3
the output of which will tell you that this is in the libXfixes RPM.  
 
If in the case of Ubuntu you don't know the name of the package that provides a certain library, the
  sudo apt-file search <name-of-library>
command should find it for you (where <code><name-of-library></code> could e.g. be <code>libGLU.so.1</code>). This needs a one-time installation by
  sudo apt install apt-file
  sudo apt-file update


It is useful to install xxdiff or one of its alternatives (see below). The package is likely available for your distribution - google for it, or try e.g.
Versions of XDSGUI compiled before 2021: Ubuntu 20.04: If the Qt5 libraries are missing, use <code>sudo apt install libqt5gui5 libqt5printsupport5 libqt5opengl5</code>. On CentOS7, this would be <code>yum -y install qt5-qtbase-gui qt5-qtbase</code> .
 
On Ubuntu, latest XDSGUI may require <code>sudo apt install libxcb-xinerama0</code>
 
<code>xxdiff</code> is likely available for your distribution - google for it, or try e.g.
  yum -y install xxdiff
  yum -y install xxdiff
on RHEL6/CentOS6/SL6 systems, or
on RHEL6/CentOS6/SL6 systems, or
  apt-get install xxdiff
  apt install xxdiff
on Ubuntu.
on Ubuntu. If Ubuntu 20.04 doesn't find it, use the latest from http://archive.ubuntu.com/ubuntu/pool/universe/x/xxdiff/


On Centos7 (RHEL7, SL7) I went to http://rpmfind.net/linux/rpm2html/search.php?query=xxdiff and found that the Fedora 20 RPM can be installed using
An <code>xxdiff</code> binary that runs on CentOS7 (RHEL7, SL7) is provided by the <code>get_folder.sh</code> script.
yum install ftp://rpmfind.net/linux/fedora/linux/releases/20/Everything/x86_64/os/Packages/x/xxdiff-3.2-20.fc20.x86_64.rpm
Perhaps xxdiff will be available on EPEL some day - maybe it is available on RPMforge; I didn't check.


Alternatively, tkdiff may be installed, or other graphical comparison programs, like meld or kdiff3. vimdiff is also nice, and probably already installed.
Alternatively, tkdiff may be installed, or other graphical comparison programs, like meld or kdiff3. vimdiff is also nice, and probably already installed.


For [[Eiger]] data processing, H5ToXDS should be installed as [[Eiger|explained]], and <code>h5dump</code> should be installed by e.g. <code>yum -y install hdf5</code>.
For [[Eiger]] data processing, a h5dump binary (version 1.10 req'd for HDF5 data from DLS) and <code>H5ToXDS</code> is installed by the <code>get_folder.sh</code> script into /usr/local/bin. If the script is not used, <code>H5ToXDS</code> should be installed as [[Eiger|explained]], and <code>h5dump</code> should be installed by e.g. <code>yum -y install hdf5</code> (Ubuntu: <code>apt install hdf5-tools</code>) - but make sure that h5dump is version 1.10 or higher when processing HDF5 data from DLS.


== Mac (Intel) ==
The LIB= feature for HDF5 files requires either the [https://www.dectris.com/company/news/newsroom/news-details/process-eiger-data-with-xds-fast Neggia] library (does not work for HDF5 data from DLS) or the [https://github.com/DiamondLightSource/durin Durin plugin]. These may be installed through the <code>get_folder.sh</code> script, in <code>/usr/local/lib64</code>. For using LIB= with .gz files, there is the xds-zcbf plugin. All plugins can also be found in [https://{{SERVERNAME}}/pub/linux_bin the Linux installation directory.]


Attention: [[generate_XDS.INP]] requires [https://developer.apple.com/xcode/ Xcode] to be installed. '''Xcode''' is not installed by default, but can be downloaded free of charge from the Mac App Store: after installing Xcode, open '''Preferences''', select the '''Downloads''' tab, and click the '''Install''' next to '''Command Line Tools'''. Or: open a Terminal window and type <code>xcode-select --install</code>. This will bring up a prompt to install the necessary command line tools.
== macOS ==


You also have to explicitly agree to the License terms when running a Command Line Tool for the first time.
Become familiar with the concept and ways to run commands as "root" (administrator) - google "mac osx become root". Installation tasks that are run in a Terminal window require root privileges if the programs and their links are written to /usr/local/bin.  


I'm not sure if Xcode can be installed without administrator privileges; you may also wish to look at http://guide.macports.org/#installing.xcode .
Therefore, if /usr/local/bin does not yet exist, prepare the following steps by opening the Terminal window, and then
sudo mkdir /usr/local/bin
 
As an alternative that does not require root for installation, one may create a directory $HOME/bin and use that for the programs and links. Also, this requires modification of the $PATH, by a one-time
mkdir $HOME/bin
echo 'export PATH=$PATH:$HOME/bin' >> ~/.zprofile


=== personal installation (experimental writeup) ===
=== [[XDS]] package for Intel Macs ===
If you are an academic user,


'''Preparation''':
  # obtain root privileges (it will ask for your password):
* make sure your $PATH includes your $HOME/bin - the programs will be put into that directory. To achieve this, open a Terminal window, and then
  sudo su
<pre>
  cd /usr/local/bin
  cd $HOME
  curl -L -o - https://xds.mr.mpg.de/XDS-OSX_64.tar.gz | tar xzvf -
echo 'export PATH=$PATH:$HOME/bin' >> ~/.profile
</pre>
(I'm not sure if it should be .profile or .bashrc; I have seen it work with .profile but not with .bashrc on a Leopard system)
* create the $HOME/bin and $HOME/Applications directories. Still in the Terminal window, type
<pre>
  mkdir $HOME/bin
mkdir $HOME/Applications
</pre>
'''To install [[generate_XDS.INP]]''':
<pre>
  cd $HOME/bin
curl -L -o - http://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/generate_XDS.INP | \
  sed -e s/\&nbsp\;/\ /g -e s/\&gt\;/\>/g -e s/\&lt\;/\</g -e s/amp\;//g -e s/\&quot\;/\"/g -e s/\&\#\1\6\0\;/\ /g | \
  sed '/# end of generate_XDS.INP/,$d' | awk '/^#/,/rm -f tmp1 tmp2/' > generate_XDS.INP
chmod +x generate_XDS.INP
</pre>
'''To install [[XDS]]''':
<pre>
cd $HOME/bin
  curl -L -o - ftp://ftp.mpimf-heidelberg.mpg.de/pub/kabsch/XDS-OSX_64.tar.gz | tar xvf -
  ln -sf XDS-OSX_64/* .
  ln -sf XDS-OSX_64/* .
</pre>
# release root permissions:
exit
(If you don't have root privileges and run in $HOME/bin, only the curl and ln commands have to be run)
Starting with macOS Catalina, you may also need
sudo xattr -dr com.apple.quarantine XDS-OSX_64
if you get a security error popup (google "xattr com.apple.quarantine catalina").
 
Then make sure that the binaries are in the search path for executables, as also explained in https://xds.mr.mpg.de/html_doc/downloading.html .


'''To install [[XDS-Viewer]]''', click https://sourceforge.net/project/platformdownload.php?group_id=239755 , open up the downloaded XDS-Viewer.app by clicking, and then (once you see it in the Finder), go to the Terminal and
If you want to process .h5 files written by Eiger detectors on a Mac with Intel CPU, use the [https://{{SERVERNAME}}/pub/mac_bin/durin-plugin.so Durin plugin for Intel Mac processors] (source code [https://github.com/DiamondLightSource/durin available]) or the [https://{{SERVERNAME}}/pub/mac_bin/dectris-neggia.so Dectris-Neggia plugin for Intel Mac processors] (source code [https://github.com/dectris/neggia available]). The latter may be faster, but does not work on data collected at Diamond Light Source. The former should in principle work for all HDF5 data. See [[Installation#other_programs:_XDS-viewer,_XDSSTAT,_XDSCC12,_XDSGUI,_XSCALE_ISOCLUSTER,_generate_XDS.INP|below]] for installation of these .so files.
cp -pr /Volumes/XDS-Viewer/XDS-Viewer.app/ $HOME/Applications
ln -s $HOME/Applications/XDS-Viewer.app/Contents/MacOS/xds-viewer-bin $HOME/bin/xds-viewer
'''To install [[XDSSTAT]]''',
<pre>
cd $HOME/bin
curl -L -o xdsstat.bz2 ftp://turn5.biologie.uni-konstanz.de/pub/xdsstat-i386-mac.bz2
bunzip2 -f xdsstat.bz2
chmod a+x xdsstat
</pre>
'''To install [[XDSCC12]]''',
<pre>
cd $HOME/bin
curl -L -o xdscc12 ftp://turn5.biologie.uni-konstanz.de/pub/xdscc12-mac
chmod a+x xdscc12


</pre>
If you want to process *.cbf.gz files on a Mac with Intel CPU, you may get faster processing with the [https://{{SERVERNAME}}/pub/mac_bin/xds-zcbf.so xds-zcbf plugin for Intel Mac processors] ([https://git.embl.de/nikolova/xds-zcbf source code]).
'''To install [[XDSGUI]]''', please download from ftp://turn5.biologie.uni-konstanz.de/pub/xdsgui.dmg, open up the downloaded xdsgui.app by clicking, and then (once you see it in the Finder), go to the Terminal and
cp -pr /Volumes/xdsgui/xdsgui.app/ $HOME/Applications
ln -s $HOME/Applications/xdsgui.app/Contents/MacOS/xdsgui $HOME/bin/xdsgui
Please also check [[XDSGUI#Installation]].


It is useful to install xxdiff from http://furius.ca/downloads/xxdiff/releases/macosx/xxdiff-4.0b1.osx.dmg (or newer). After downloading, open up the downloaded xxdiff.app by clicking, and then (once you see it in the Finder), go to the Terminal and
See [[Installation#other_programs:_XDS-viewer,_XDSSTAT,_XDSCC12,_XDSGUI,_XSCALE_ISOCLUSTER,_generate_XDS.INP|below]] for installation of these .so files.
cp -pr /Volumes/xxdiff/xxdiff.app/ $HOME/Applications
ln -s $HOME/Applications/xxdiff.app/Contents/MacOS/xxdiff $HOME/bin/xxdiff


(the names of the files may not be entirely correct in these two lines)
==== XDS package for Macs with Apple M processors ====


=== system-wide installation (requires root permission) ===
The Intel Mac XDS programs also work on the Apple Silicon machines, but the plugin libraries (see below) for e.g. reading data from Eiger detectors don't, and the native binaries are faster.


To get [[generate_XDS.INP]]:
Thus, change the above <code>curl</code> and <code>ln</code> commands to
<pre>
# become root (it will ask for your password):
sudo su
  cd /usr/local/bin
  cd /usr/local/bin
  sudo sh -c 'curl -L -o - http://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/generate_XDS.INP | \
  curl -L -o - https://xds.mr.mpg.de/XDS-Apple_M1.tar.gz | tar xzvf -
  sed -e s/\&nbsp\;/\ /g -e s/\&gt\;/\>/g -e s/\&lt\;/\</g -e s/amp\;//g -e s/\&quot\;/\"/g -e s/\&\#\1\6\0\;/\ /g | \
ln -sf XDS-Apple_M1/* .
  sed '/# end of generate_XDS.INP/,$d' | awk '/^#/,/rm -f tmp1 tmp2/' > generate_XDS.INP'
# release root permissions:
sudo chmod +x generate_XDS.INP
exit
</pre>
 
As explained above, you may need
sudo xattr -d com.apple.quarantine XDS-Apple_M1/*
if you get a security-related popup saying that Apple cannot verify the software.
 
If you want to process .h5 files written by Eiger detectors, use the [https://{{SERVERNAME}}/pub/mac_bin/durin-plugin-Apple-arm64.so Durin plugin for Apple ARM64 processors] (source code [https://github.com/DiamondLightSource/durin available]) or the [https://{{SERVERNAME}}/pub/mac_bin/dectris-neggia-Apple-arm64.so Dectris-Neggia plugin for Apple ARM64 processors] (source code [https://github.com/dectris/neggia available]). The latter may be faster, but does not work on data collected at Diamond Light Source. The former should in principle work for all HDF5 data. 
 
If you want to process *.cbf.gz files, you may get faster processing with the [https://{{SERVERNAME}}/pub/mac_bin/xds-zcbf-Apple-arm64.so xds-zcbf plugin for Apple ARM64 processors] ([https://git.embl.de/nikolova/xds-zcbf source code]).
 
See [[Installation#other_programs:_XDS-viewer,_XDSSTAT,_XDSCC12,_XDSGUI,_XSCALE_ISOCLUSTER,_generate_XDS.INP|below]] for installation of these .so files.
 
=== Prerequisites (Xcode, CCP4) for [[generate_XDS.INP]], [[XDSGUI]] and other programs ===
[[generate_XDS.INP]] requires the [https://developer.apple.com/download/more/ Xcode command line tools] to be installed (free of charge): open a Terminal window, and enter <code>sudo xcode-select --install</code> (the download has a size of about 750MB; the downloader is much too pessimistic about the time it takes).


To get [[XDS]]:
Although CCP4 is not required by XDS itself, some of the programs (XDSSTAT, and MTZ file creation in XDSCONV, POINTLESS and COOT) use CCP4 tools or libraries. Therefore, CCP4 should be installed. Open a Terminal and check the existence of the file .zprofile , and its content:<pre>
kay@Kays-MacBook-Air-2018 ~ % ls -l .zprofile
-rw-r--r--  1 kay  staff  107  7 Mär 09:41 .zprofile
kay@Kays-MacBook-Air-2018 ~ % cat .zprofile
source $HOME/.profile
# Added by CCP4 package manager:
. '/Applications/ccp4-7.1/bin/ccp4.setup-sh'
</pre>If the file does not exist, create it - the easiest way is:
<pre>
<pre>
cd /usr/local/bin
echo . /Applications/ccp4-7.1/bin/ccp4.setup-sh >> $HOME/.zprofile
sudo sh -c 'curl -L -o - ftp://ftp.mpimf-heidelberg.mpg.de/pub/kabsch/XDS-OSX_64.tar.gz | tar xvf -'
# or
sudo ln -sf XDS-OSX_64/* .
echo . /Applications/ccp4-8.0/bin/ccp4.setup-sh >> $HOME/.zprofile
</pre>
# depending on which CCP4 version you have installed
</pre>The <code>get_folder.sh</code> script (below) checks the CCP4 setup, and creates the necessary files if needed.
 
Only Terminal windows that are opened afterwards will have access to the CCP4 programs!


To get [[XDS-Viewer]], click https://sourceforge.net/project/platformdownload.php?group_id=239755 , install in the usual graphical way to your /Applications and then
=== other programs: [[XDS-viewer]], [[XDSSTAT]], [[XDSCC12]], [[XDSGUI]], [[XSCALE_ISOCLUSTER]], [[generate_XDS.INP]]  ===
  sudo ln -s /Applications/XDS-Viewer.app/Contents/MacOS/xds-viewer-bin /usr/local/bin/xds-viewer
... can be downloaded from https://{{SERVERNAME}}/pub/mac_bin/ . A simple way to obtain them is is to open a Terminal, and type:
   
  sudo su
To get [[XDSSTAT]],
which asks for the password and gives you administrator permissions. Then, for Intel Macs,
<pre>
# any folder would do; here we use /usr/local/bin/mac_bin
  cd /usr/local/bin
mkdir -p /usr/local/bin/mac_bin
  sudo curl -L -o xdsstat.bz2 ftp://turn5.biologie.uni-konstanz.de/pub/xdsstat-i386-mac.bz2
cd /usr/local/bin/mac_bin
sudo bunzip2 -f xdsstat.bz2
curl -O -R https://{{SERVERNAME}}/pub/mac_bin/get_folder.sh
  sudo chmod a+x xdsstat
  chmod +x get_folder.sh
  </pre>
./get_folder.sh
For Silicon Macs,
# any folder would do; here we use /usr/local/bin/mac_bin
mkdir -p /usr/local/bin/mac_bin
  cd /usr/local/bin/mac_bin
  curl -O -R https://{{SERVERNAME}}/pub/mac_bin/get_folder_silicon.sh
  chmod +x get_folder_silicon.sh
  ./get_folder_silicon.sh
After that, release root privileges:
exit
If you want to update the programs some time later, just run these commands again.


To get [[XDSCC12]],
In order to enable [[Eiger]] data processing, the <code>get_folder.sh</code> (or <code>get_folder_silicon.sh</code>) script
<pre>
# copies Dectris' Neggia library and DLS's Durin library into <code>/usr/local/lib64</code> . NOTE that the names of the native libraries for the Apple Silicon Macs ends with *-Apple-arm64.so . On Silicon Macs you can either use the Intel binaries for XDSGUI and XDS, or the Silicon binaries - the latter are recommended. You could in principle mix them but then you have to be careful with what you enter in the LIB= line of XDS.INP; and what you enter in Menu/Preferences/Paths/generic library of XDSGUI.
cd /usr/local/bin
# copies [https://www.globalphasing.com GlobalPhasing's] <code>hdf2mini-cbf</code> into <code>/usr/local/bin</code>, but under the name <code>H5ToXds</code> (consider getting a [https://www.globalphasing.com/autoproc/ autoPROC] license!)
sudo curl -L -o xdscc12 ftp://turn5.biologie.uni-konstanz.de/pub/xdscc12-mac
# copies <code>h5dump</code> [https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-1.12/hdf5-1.12.1/bin/unix/hdf5-1.12.1-Std-macos11_64-clang.tar.gz] into <code>/usr/local/bin</code>
sudo chmod a+x xdscc12
</pre>


To get [[XDSGUI]], please download from ftp://turn5.biologie.uni-konstanz.de/pub/xdsgui.dmg and then
A [https://{{SERVERNAME}}/pub/mac_bin/XDS-Viewer-macOS10.13.dmg 10.13] Qt5 version of XDS-viewer (compiled by T. Hauß) works on macOS Catalina, and is installed by <code>get_folder.sh</code>. Older Macs need [https://{{SERVERNAME}}/pub/mac_bin/XDS-Viewer-0.6.dmg this version]; newer macOS should install [https://{{SERVERNAME}}/pub/mac_bin/XDS-Viewer-macOS10.15.dmg 10.15].  
sudo ln -s /Applications/xdsgui.app/Contents/MacOS/xdsgui /usr/local/bin/xdsgui
Please also check [[XDSGUI#Installation]].


It is useful to install xxdiff from http://furius.ca/downloads/xxdiff/releases/macosx/
For XDSGUI, you have a choice between the version for Intel Macs and that for Silicon Macs. Both do work on Silicon Macs, except for the plugin libraries - for these, the correct version (ending with -Apple-arm64.so) must be used together with xdsgui_mac_silicon.  


For [[Eiger]] data processing, [[Eiger#General_aspects|eiger2cbf-osx]] should be sym-linked into a directory in your $PATH, but under the name H5ToXds, and <code>h5dump</code> should be installed from a .dmg file.
Lastly, it is useful (for e.g. XDSGUI) to install <code>xxdiff</code> . This used to be available from http://furius.ca/downloads/xxdiff/releases/macosx/ , but is also [https://{{SERVERNAME}}/pub/mac_bin/xxdiff-4.0b1.osx.dmg here]. I had to go to System Preferences > Security & Privacy > Privacy and add <code>xxdiff</code> to the programs with full disk access.


=== Troubleshooting ===
=== Troubleshooting ===
Line 190: Line 179:
ls -l /usr/local/bin/xdsgui
ls -l /usr/local/bin/xdsgui
ls -l /usr/local/bin/xds-viewer
ls -l /usr/local/bin/xds-viewer
# in the same way, check the other programs that should be used
</pre>  
</pre>  
returns a message like <code>file or directory not found</code> then this means that you do not have the symlink. But even if these commands do not return errors, you should also check the targets of the symlink:
returns a message like <code>file or directory not found</code> then this means that you do not have the symlink. But even if these commands do not return errors, you should also check the targets of the symlink, as shown in the output of the <code>ls -l</code> commands.
<pre>
Again, these commands should not return an error message. If they do, the programs are not installed in the location where the symlink points to. You'll have to install the programs properly (in /Applications) and make the symlink point to the correct location.
ls -l /Applications/xdsgui.app/Contents/MacOS/xdsgui
 
ls -l /Applications/XDS-Viewer.app/Contents/MacOS/xds-viewer-bin
A quick way is also
</pre>
which generate_XDS.INP xdsgui xds xds-viewer xdsstat xdscc12
Again, these commands should not return an error message. If they do, the programs are not installed in the location where the symlink points to. You'll have to either install the programs properly (in /Applications) or make the symlink point to the correct location.
and this should return a line for each of the programs asked for.


== Windows ==
== Windows ==


The programs of the XDS package as well as XDSGUI, XDSSTAT and so on may be run on 64bit Windows10 within the [http://www.pcworld.com/article/3050473/windows/heres-how-windows-10s-ubuntu-based-bash-shell-will-actually-work.html Linux Subsystem for Windows]. This installs a Ubuntu 14.04.5 LTS environment, which supports the apt package manager.
The programs of the [[XDS]] package as well as [[XDSGUI]], [[XDSSTAT]], [[XDSCC12]] and so on may be run on 64bit Windows 10 within the [https://docs.microsoft.com/windows/wsl/install-win10 Linux Subsystem for Windows] (WSL, or preferably WSL2). This (easily!) installs e.g. an [https://ubuntu.com/wsl Ubuntu] environment, which supports the apt package manager, so in principle any software available for Ubuntu may be installed (or other distros, see [https://blogs.msdn.microsoft.com/commandline/2017/05/11/new-distros-coming-to-bashwsl-via-windows-store/]).


Graphical Linux programs require installation of a X server like [https://sourceforge.net/projects/xming/files/latest/download Xming] on the Windows host. To use it, one has to say in the shell window
A lot of technical detail is at [https://docs.microsoft.com/en-us/windows/wsl/compare-versions] .
 
CCP4 documents use of WSL-1 ; see info at http://www.ccp4.ac.uk/download/#os=wsl .
 
Graphical Linux programs require installation of a X server like [https://sourceforge.net/projects/xming/files/latest/download Xming] or [https://sourceforge.net/projects/vcxsrv VcXsvr] or [https://mobaxterm.mobatek.net/ Mobaxterm] on the Windows host. There are helpful writeups (e.g. [https://wiki.ubuntu.com/WSL#Running_Graphical_Applications] and [http://wsl-guide.org/en/latest/]). VcXsrc seems to have a problem with coot for some people (see [https://www.jiscmail.ac.uk/cgi-bin/wa-jisc.exe?A2=COOT;6ef4a967.2011 WSL2 installation] and responses). To use the X server with WSL, one has to say in the shell window
  export DISPLAY=:0
  export DISPLAY=:0
XDSGUI requires
before running the program. Running [[XDSGUI]] under WSL's Ubuntu 20.04 may require installation of a few packages with e.g.
  sudo apt-get install libgomp1 libqtgui4
  sudo apt-get install libgomp1 libqtgui4 binutils
and [[generate_XDS.INP]] requires a number of packages (some of which are only for specific detectors!); I'd start with
(see [[XDSGUI#Libraries_and_software_that_the_program_depends_on|XDSGUI]] and above). If required, also install python and hdf5-tools.
  sudo apt-get install coreutils binutils gawk sed bc grep
 
and if required, also install python and hdf5-tools.
XDSSTAT and the conversion to MTZ files by XDSCONV require a CCP4 installation accessible by WSL. In principle, CCP4 may be installed within WSL (have not tested this), or on the Windows host.
 
[[File:Xds_win.PNG|1000px]]
 
(Screenshot provided by Gustavo Lima)
 
A detailed [https://{{SERVERNAME}}/pub/xds%20%20prog%20installation%20in%20WSL%20(Lata%20Panicker).pdf writeup for Ubuntu 20.04 LTS] was provided by Dr Lata Panicker, SO(G), BARC, India.
 
Update September 2021: With the current version of Windows 10 and WSL, the Qt5 binary of XDSGUI does not seem to allow editing in the XDS.INP tab. If this happens, either use an external editor or use xdsgui.qt4. In the latter case, you may have to install Qt4 libraries.
 
=== WSL2 ===
 
CCP4 7.1 including coot-0.9.5 works in a Ubuntu 20.04 LTS machine under WSL2 on Windows 10 version 20H2, and newer.
 
[https://docs.microsoft.com/windows/wsl/install-win10 After installation of WSL2], I installed [https://mobaxterm.mobatek.net/ MobaXterm] as the Xserver. When the Windows firewall asked whether it should allow MobaXterm communication, I answered "yes" for both private and public networks.
 
If you want to install CCP4 as a regular user, you need (as this is Linux! And replace MYWSLNAME with your WSL user name):
sudo mkdir /opt/xtal
sudo chown MYWSLNAME /opt/xtal
 
A few libraries are needed before CCP4 installation (tcsh is needed by CCP4; most of the others are graphics libraries; binutils provides <code>strings</code>):
sudo apt install tcsh libqt5opengl5 libqt5printsupport5 libqt5gui5 libxcb-render0 libxcb-shm0 libglu1-mesa libgomp1 binutils
When installing CCP4, let it "modify command line environment" for you (on one of the very first screens of the installation GUI). If you forgot this, insert a line into your ~/.bashrc :
source /opt/xtal/ccp4-7.1/bin/ccp4.setup-sh
afterwards (this is not WSL specific, but is just Linux). I created a file in the Windows filesystem (replace MYUSERNAME with your Windows user name):
cd /mnt/c/Users/MYUSERNAME/
echo "[wsl2]" > .wslconfig
  echo "kernelCommandLine = vsyscall=emulate" >> .wslconfig
to make the shelx* programs work (see https://github.com/microsoft/WSL/issues/4694 ), and to make "vsyscall=emulate" appear in /proc/cmdline. After creating the file, WSL2 must be restarted with "wsl.exe --shutdown" (or the machine booted).
 
Performance of coot is good enough for occasional work. XDSGUI and multi-threaded XDS work as expected. For reading .h5 files, the NEGGIA plugin works correctly, but the DURIN plugin currently crashes (a [https://github.com/DiamondLightSource/durin/issues/22 bug] has been reported).
 
Update September 2021: With the current version of Windows 10 and WSL, the Qt5 binary of XDSGUI does not seem to allow editing in the XDS.INP tab. If this happens, either use an external editor or use xdsgui.qt4. In the latter case, you may have to install Qt4 libraries.


Unfortunately, only single-processor xds runs smoothly, whereas the parallel xds_par stops with an error message. This is the workaround:  
Update 2022: Windows 11 supports WSL2 even better, and does not need installation of X-server. https://ubuntu.com/tutorials/install-ubuntu-on-wsl2-on-windows-11-with-gui-support#5-install-and-use-a-gui-package


# since xds_par does not work anyway, just copy the xds binary over the xds_par binary so that it is available under both names
== See also ==
# in XDS.INP, specify MAXIMUM_NUMBER_OF_JOBS= <number of cores>


This will then use process-level parallelization instead of thread-level parallelization, and works rather well.
[[Cluster Installation]]

Latest revision as of 20:55, 29 October 2024

This article has information that should ease the installation of XDS and related programs like (XDS-viewer, xdsstat, xdsgui). It is assumed that binaries (or links to binaries) should go to /usr/local/bin . This means that you should do this as the administrator (root), or have sudo rights (Mac, and some Linux distros).


Linux

Log in as root - we need write permission for /usr/local/bin .

XDS package

If you are an academic user, as root (on Ubuntu, use "sudo -i" to become root)

 cd /usr/local/bin
 wget --no-check-certificate -O- https://xds.mr.mpg.de/XDS-INTEL64_Linux_x86_64.tar.gz | tar xzvf -
 ln -sf XDS-INTEL64_Linux_x86_64/* .

In case the above website is unavailable, there is a backup site.

If you want to process data from the Eiger detector: see below for plugins that allow XDS to read the .h5 files. Similarly for gzip-ed CBF files as mostly collected at EMBL Hamburg.

other programs: XDS-Viewer, XDSSTAT, XDSCC12, XDSGUI, XSCALE_ISOCLUSTER, spot2pdb, generate_XDS.INP

... can be downloaded from https://wiki.uni-konstanz.de/pub/linux_bin/ . A simple way to obtain them is to create an empty folder, "cd" to that folder and then

wget -N https://wiki.uni-konstanz.de/pub/linux_bin/get_folder.sh
chmod +x get_folder.sh
./get_folder.sh

You may repeat this every now and then to update the programs.

As root, the script copies the programs into /usr/local/bin/linux_bin, and links them to /usr/local/bin. As a normal user, they are downloaded to $HOME/bin/linux_bin, and linked to your $HOME/bin. In the latter case, make sure that $HOME/bin is in your $PATH.

tools and helper programs for generate_XDS.INP and XDSGUI, and libraries

Since XDSGUI depends on graphics packages that may need to be installed, you can check the xdsgui binary with

ldd `which xdsgui` | grep found

and that should show you the libraries it didn't find. Your friendly system administrator knows the specific commands to install those libraries.

N.B. this may list libpng15.so for the latest XDSGUI (compiled May 2023), but you can ignore that!

For RedHat-type distributions, that would typically be e.g.

yum provides libXfixes.so.3

the output of which will tell you that this is in the libXfixes RPM.

If in the case of Ubuntu you don't know the name of the package that provides a certain library, the

 sudo apt-file search <name-of-library> 

command should find it for you (where <name-of-library> could e.g. be libGLU.so.1). This needs a one-time installation by

 sudo apt install apt-file
 sudo apt-file update

Versions of XDSGUI compiled before 2021: Ubuntu 20.04: If the Qt5 libraries are missing, use sudo apt install libqt5gui5 libqt5printsupport5 libqt5opengl5. On CentOS7, this would be yum -y install qt5-qtbase-gui qt5-qtbase .

On Ubuntu, latest XDSGUI may require sudo apt install libxcb-xinerama0

xxdiff is likely available for your distribution - google for it, or try e.g.

yum -y install xxdiff

on RHEL6/CentOS6/SL6 systems, or

apt install xxdiff

on Ubuntu. If Ubuntu 20.04 doesn't find it, use the latest from http://archive.ubuntu.com/ubuntu/pool/universe/x/xxdiff/

An xxdiff binary that runs on CentOS7 (RHEL7, SL7) is provided by the get_folder.sh script.

Alternatively, tkdiff may be installed, or other graphical comparison programs, like meld or kdiff3. vimdiff is also nice, and probably already installed.

For Eiger data processing, a h5dump binary (version 1.10 req'd for HDF5 data from DLS) and H5ToXDS is installed by the get_folder.sh script into /usr/local/bin. If the script is not used, H5ToXDS should be installed as explained, and h5dump should be installed by e.g. yum -y install hdf5 (Ubuntu: apt install hdf5-tools) - but make sure that h5dump is version 1.10 or higher when processing HDF5 data from DLS.

The LIB= feature for HDF5 files requires either the Neggia library (does not work for HDF5 data from DLS) or the Durin plugin. These may be installed through the get_folder.sh script, in /usr/local/lib64. For using LIB= with .gz files, there is the xds-zcbf plugin. All plugins can also be found in the Linux installation directory.

macOS

Become familiar with the concept and ways to run commands as "root" (administrator) - google "mac osx become root". Installation tasks that are run in a Terminal window require root privileges if the programs and their links are written to /usr/local/bin.

Therefore, if /usr/local/bin does not yet exist, prepare the following steps by opening the Terminal window, and then

sudo mkdir /usr/local/bin

As an alternative that does not require root for installation, one may create a directory $HOME/bin and use that for the programs and links. Also, this requires modification of the $PATH, by a one-time

mkdir $HOME/bin
echo 'export PATH=$PATH:$HOME/bin' >> ~/.zprofile

XDS package for Intel Macs

If you are an academic user,

# obtain root privileges (it will ask for your password):
sudo su
cd /usr/local/bin
curl -L -o - https://xds.mr.mpg.de/XDS-OSX_64.tar.gz | tar xzvf -
ln -sf XDS-OSX_64/* .
# release root permissions:
exit

(If you don't have root privileges and run in $HOME/bin, only the curl and ln commands have to be run) Starting with macOS Catalina, you may also need

sudo xattr -dr com.apple.quarantine XDS-OSX_64

if you get a security error popup (google "xattr com.apple.quarantine catalina").

Then make sure that the binaries are in the search path for executables, as also explained in https://xds.mr.mpg.de/html_doc/downloading.html .

If you want to process .h5 files written by Eiger detectors on a Mac with Intel CPU, use the Durin plugin for Intel Mac processors (source code available) or the Dectris-Neggia plugin for Intel Mac processors (source code available). The latter may be faster, but does not work on data collected at Diamond Light Source. The former should in principle work for all HDF5 data. See below for installation of these .so files.

If you want to process *.cbf.gz files on a Mac with Intel CPU, you may get faster processing with the xds-zcbf plugin for Intel Mac processors (source code).

See below for installation of these .so files.

XDS package for Macs with Apple M processors

The Intel Mac XDS programs also work on the Apple Silicon machines, but the plugin libraries (see below) for e.g. reading data from Eiger detectors don't, and the native binaries are faster.

Thus, change the above curl and ln commands to

# become root (it will ask for your password):
sudo su
cd /usr/local/bin
curl -L -o - https://xds.mr.mpg.de/XDS-Apple_M1.tar.gz | tar xzvf -
ln -sf XDS-Apple_M1/* .
# release root permissions:
exit

As explained above, you may need

sudo xattr -d com.apple.quarantine XDS-Apple_M1/* 

if you get a security-related popup saying that Apple cannot verify the software.

If you want to process .h5 files written by Eiger detectors, use the Durin plugin for Apple ARM64 processors (source code available) or the Dectris-Neggia plugin for Apple ARM64 processors (source code available). The latter may be faster, but does not work on data collected at Diamond Light Source. The former should in principle work for all HDF5 data.

If you want to process *.cbf.gz files, you may get faster processing with the xds-zcbf plugin for Apple ARM64 processors (source code).

See below for installation of these .so files.

Prerequisites (Xcode, CCP4) for generate_XDS.INP, XDSGUI and other programs

generate_XDS.INP requires the Xcode command line tools to be installed (free of charge): open a Terminal window, and enter sudo xcode-select --install (the download has a size of about 750MB; the downloader is much too pessimistic about the time it takes).

Although CCP4 is not required by XDS itself, some of the programs (XDSSTAT, and MTZ file creation in XDSCONV, POINTLESS and COOT) use CCP4 tools or libraries. Therefore, CCP4 should be installed. Open a Terminal and check the existence of the file .zprofile , and its content:

kay@Kays-MacBook-Air-2018 ~ % ls -l .zprofile
-rw-r--r--  1 kay   staff  107  7 Mär 09:41 .zprofile
kay@Kays-MacBook-Air-2018 ~ % cat .zprofile
source $HOME/.profile
# Added by CCP4 package manager:
. '/Applications/ccp4-7.1/bin/ccp4.setup-sh'

If the file does not exist, create it - the easiest way is:

echo . /Applications/ccp4-7.1/bin/ccp4.setup-sh >> $HOME/.zprofile
# or
echo . /Applications/ccp4-8.0/bin/ccp4.setup-sh >> $HOME/.zprofile
# depending on which CCP4 version you have installed

The get_folder.sh script (below) checks the CCP4 setup, and creates the necessary files if needed.

Only Terminal windows that are opened afterwards will have access to the CCP4 programs!

other programs: XDS-viewer, XDSSTAT, XDSCC12, XDSGUI, XSCALE_ISOCLUSTER, generate_XDS.INP

... can be downloaded from https://wiki.uni-konstanz.de/pub/mac_bin/ . A simple way to obtain them is is to open a Terminal, and type:

sudo su

which asks for the password and gives you administrator permissions. Then, for Intel Macs,

# any folder would do; here we use /usr/local/bin/mac_bin
mkdir -p /usr/local/bin/mac_bin
cd /usr/local/bin/mac_bin
curl -O -R https://wiki.uni-konstanz.de/pub/mac_bin/get_folder.sh
chmod +x get_folder.sh
./get_folder.sh

For Silicon Macs,

# any folder would do; here we use /usr/local/bin/mac_bin
mkdir -p /usr/local/bin/mac_bin
cd /usr/local/bin/mac_bin
curl -O -R https://wiki.uni-konstanz.de/pub/mac_bin/get_folder_silicon.sh
chmod +x get_folder_silicon.sh
./get_folder_silicon.sh

After that, release root privileges:

exit

If you want to update the programs some time later, just run these commands again.

In order to enable Eiger data processing, the get_folder.sh (or get_folder_silicon.sh) script

  1. copies Dectris' Neggia library and DLS's Durin library into /usr/local/lib64 . NOTE that the names of the native libraries for the Apple Silicon Macs ends with *-Apple-arm64.so . On Silicon Macs you can either use the Intel binaries for XDSGUI and XDS, or the Silicon binaries - the latter are recommended. You could in principle mix them but then you have to be careful with what you enter in the LIB= line of XDS.INP; and what you enter in Menu/Preferences/Paths/generic library of XDSGUI.
  2. copies GlobalPhasing's hdf2mini-cbf into /usr/local/bin, but under the name H5ToXds (consider getting a autoPROC license!)
  3. copies h5dump [1] into /usr/local/bin

A 10.13 Qt5 version of XDS-viewer (compiled by T. Hauß) works on macOS Catalina, and is installed by get_folder.sh. Older Macs need this version; newer macOS should install 10.15.

For XDSGUI, you have a choice between the version for Intel Macs and that for Silicon Macs. Both do work on Silicon Macs, except for the plugin libraries - for these, the correct version (ending with -Apple-arm64.so) must be used together with xdsgui_mac_silicon.

Lastly, it is useful (for e.g. XDSGUI) to install xxdiff . This used to be available from http://furius.ca/downloads/xxdiff/releases/macosx/ , but is also here. I had to go to System Preferences > Security & Privacy > Privacy and add xxdiff to the programs with full disk access.

Troubleshooting

If any of the commands

ls -l /usr/local/bin/generate_XDS.INP
ls -l /usr/local/bin/xdsgui
ls -l /usr/local/bin/xds-viewer
# in the same way, check the other programs that should be used

returns a message like file or directory not found then this means that you do not have the symlink. But even if these commands do not return errors, you should also check the targets of the symlink, as shown in the output of the ls -l commands. Again, these commands should not return an error message. If they do, the programs are not installed in the location where the symlink points to. You'll have to install the programs properly (in /Applications) and make the symlink point to the correct location.

A quick way is also

which generate_XDS.INP xdsgui xds xds-viewer xdsstat xdscc12

and this should return a line for each of the programs asked for.

Windows

The programs of the XDS package as well as XDSGUI, XDSSTAT, XDSCC12 and so on may be run on 64bit Windows 10 within the Linux Subsystem for Windows (WSL, or preferably WSL2). This (easily!) installs e.g. an Ubuntu environment, which supports the apt package manager, so in principle any software available for Ubuntu may be installed (or other distros, see [2]).

A lot of technical detail is at [3] .

CCP4 documents use of WSL-1 ; see info at http://www.ccp4.ac.uk/download/#os=wsl .

Graphical Linux programs require installation of a X server like Xming or VcXsvr or Mobaxterm on the Windows host. There are helpful writeups (e.g. [4] and [5]). VcXsrc seems to have a problem with coot for some people (see WSL2 installation and responses). To use the X server with WSL, one has to say in the shell window

export DISPLAY=:0

before running the program. Running XDSGUI under WSL's Ubuntu 20.04 may require installation of a few packages with e.g.

sudo apt-get install libgomp1 libqtgui4 binutils

(see XDSGUI and above). If required, also install python and hdf5-tools.

XDSSTAT and the conversion to MTZ files by XDSCONV require a CCP4 installation accessible by WSL. In principle, CCP4 may be installed within WSL (have not tested this), or on the Windows host.

Xds win.PNG

(Screenshot provided by Gustavo Lima)

A detailed writeup for Ubuntu 20.04 LTS was provided by Dr Lata Panicker, SO(G), BARC, India.

Update September 2021: With the current version of Windows 10 and WSL, the Qt5 binary of XDSGUI does not seem to allow editing in the XDS.INP tab. If this happens, either use an external editor or use xdsgui.qt4. In the latter case, you may have to install Qt4 libraries.

WSL2

CCP4 7.1 including coot-0.9.5 works in a Ubuntu 20.04 LTS machine under WSL2 on Windows 10 version 20H2, and newer.

After installation of WSL2, I installed MobaXterm as the Xserver. When the Windows firewall asked whether it should allow MobaXterm communication, I answered "yes" for both private and public networks.

If you want to install CCP4 as a regular user, you need (as this is Linux! And replace MYWSLNAME with your WSL user name):

sudo mkdir /opt/xtal
sudo chown MYWSLNAME /opt/xtal

A few libraries are needed before CCP4 installation (tcsh is needed by CCP4; most of the others are graphics libraries; binutils provides strings):

sudo apt install tcsh libqt5opengl5 libqt5printsupport5 libqt5gui5 libxcb-render0 libxcb-shm0 libglu1-mesa libgomp1 binutils

When installing CCP4, let it "modify command line environment" for you (on one of the very first screens of the installation GUI). If you forgot this, insert a line into your ~/.bashrc :

source /opt/xtal/ccp4-7.1/bin/ccp4.setup-sh 

afterwards (this is not WSL specific, but is just Linux). I created a file in the Windows filesystem (replace MYUSERNAME with your Windows user name):

cd /mnt/c/Users/MYUSERNAME/
echo "[wsl2]" > .wslconfig
echo "kernelCommandLine = vsyscall=emulate" >> .wslconfig

to make the shelx* programs work (see https://github.com/microsoft/WSL/issues/4694 ), and to make "vsyscall=emulate" appear in /proc/cmdline. After creating the file, WSL2 must be restarted with "wsl.exe --shutdown" (or the machine booted).

Performance of coot is good enough for occasional work. XDSGUI and multi-threaded XDS work as expected. For reading .h5 files, the NEGGIA plugin works correctly, but the DURIN plugin currently crashes (a bug has been reported).

Update September 2021: With the current version of Windows 10 and WSL, the Qt5 binary of XDSGUI does not seem to allow editing in the XDS.INP tab. If this happens, either use an external editor or use xdsgui.qt4. In the latter case, you may have to install Qt4 libraries.

Update 2022: Windows 11 supports WSL2 even better, and does not need installation of X-server. https://ubuntu.com/tutorials/install-ubuntu-on-wsl2-on-windows-11-with-gui-support#5-install-and-use-a-gui-package

See also

Cluster Installation