Eiger: Difference between revisions

Jump to navigation Jump to search

Eiger (view source)

Revision as of 17:24, 27 February 2021

2,020 bytes removed ,  27 February 2021
→‎Benchmark using H5ToXds: remove - too old
(→‎Benchmark using H5ToXds: remove - too old)
Line 630: Line 630:


For faster processing, the [[Eiger#A_script_for_faster_XDS_processing_of_CBF-converted Eiger data|shell script]] below should be copied to /usr/local/bin/H5ToXds and made executable (<code>chmod a+rx /usr/local/bin/H5ToXds*</code>). The binary H5ToXds then should be named e.g. /usr/local/bin/H5ToXds.bin - note the .bin filename extension! The script ''also'' uses RAM to speed up processing; it uses it for fast storage of the temporary CBF file that H5ToXds/eiger2cbf/hdf2mini-cbf writes, and that each parallel thread ("processor") of XDS reads. The amount of additional RAM this requires is modest (about (number of pixels)*(number of threads) bytes).
For faster processing, the [[Eiger#A_script_for_faster_XDS_processing_of_CBF-converted Eiger data|shell script]] below should be copied to /usr/local/bin/H5ToXds and made executable (<code>chmod a+rx /usr/local/bin/H5ToXds*</code>). The binary H5ToXds then should be named e.g. /usr/local/bin/H5ToXds.bin - note the .bin filename extension! The script ''also'' uses RAM to speed up processing; it uses it for fast storage of the temporary CBF file that H5ToXds/eiger2cbf/hdf2mini-cbf writes, and that each parallel thread ("processor") of XDS reads. The amount of additional RAM this requires is modest (about (number of pixels)*(number of threads) bytes).
== Benchmark using H5ToXds ==
This was run on a single unloaded CentOS7.2 64bit machine with dual Intel(R) Xeon(R) CPU E5-2667 v2 @ 3.30GHz , HT enabled (showing 32 processors in /proc/cpuinfo), on a local XFS filesystem (all defaults), with four JOBs and 12 PROCESSORS. The numbers below refer to the H5ToXds binary as used in the script below.
The timing, using the XDS (BUILT=20151231), is on the first run
INIT:  elapsed wall-clock time      12.0 sec
COLSPOT: elapsed wall-clock time      44.9 sec
INTEGRATE: total elapsed wall-clock time      65.1 sec
CORRECT: elapsed wall-clock time        2.9 sec
Total elapsed wall-clock time for XDS      133.6 sec
When I repeat this, I get
Total elapsed wall-clock time for XDS      128.3 sec
Repeat once again:
Total elapsed wall-clock time for XDS      129.3 sec
So a bit of cache-warming helps, but not much. This machine has 64GB RAM. From the output of "top", the highest memory usage occurs during INTEGRATE, when each of the mintegrate_par processes consumes up to 7.4% of the memory. In other words, in this way less than 20GB of total memory are used. "top" shows a CPU consumption around (on average) 4 times 650%.
The number of JOBs and PROCESSORs could be optimized. I tried 6 JOBs and get
Total elapsed wall-clock time for XDS      120.1 sec
so there's still some room for improvement.
With program versions as of 2016-03-10, eiger2cbf-linux is practically as fast as the H5ToXds binary; hdf2mini-cbf is somewhat slower.
When unpacking the .h5 files to .cbf files and processing those, I get on the same machine and with same processing parameters:
Total elapsed wall-clock time for XDS      96.3 sec
which indicates a 24% overhead due to the HDF5-to-CBF conversion. However, one has to add to this the time for the HDF5-to-CBF conversion, which is (with 18 parallel H5ToXds jobs each converting 50 frames) 34.2 sec, so overall the "on-the-fly" route using the script below is faster than the "pre-conversion" route, at least on this machine.


== A script for faster XDS processing of CBF-converted Eiger data (this is only shown out of historic interest) ==
== A script for faster XDS processing of CBF-converted Eiger data (this is only shown out of historic interest) ==
Kay
Bureaucrats
2,684

edits

Retrieved from "https://wiki.uni-konstanz.de/xds/index.php/Special:MobileDiff/4258"
Cookies help us deliver our services. By using our services, you agree to our use of cookies.
More information

Navigation menu