Eiger: Difference between revisions

Line 2:

== General aspects ==

# The framecache of XDS uses memory to save on I/O; it saves a frame in RAM after reading it for the first time. ~~By default, each~~ XDS (or mcolspot/mintegrate) job stores NUMBER_OF_IMAGES_IN_CACHE=DELPHI/OSCILLATION_RANGE+1 ~~images~~ in memory which corresponds to one DELPHI-sized batch of data. ~~This requires (number of pixels)*(number of jobs)*4 Bytes per frame which amounts to 72 MB in case of the Eiger 16M when running with~~ MAXIMUM_NUBER_OF_JOBS=1~~. (If~~ DELPHI=20 and OSCILLATION_RANGE=0.05 your computer thus has to have at least ~~400~~*72MB = ~~29GB~~ of ~~memory for each job!)~~. If memory allocation fails, the fallback is to ~~the old behaviour of~~ reading each frame three times (instead of once).

# The framecache of XDS uses memory to save on I/O; it saves a frame in RAM after reading it for the first time. Each frame needs (number of pixels)*4 bytes = 72 MB in case of the Eiger 16M. Each XDS (or mcolspot/mintegrate) job stores NUMBER_OF_IMAGES_IN_CACHE=DELPHI/OSCILLATION_RANGE+1 such frames in memory which corresponds to one DELPHI-sized batch of data. With MAXIMUM_NUBER_OF_JOBS=1, DELPHI=5 (the default) and OSCILLATION_RANGE=0.1, your computer thus has to have at least 50*72MB = 3.6GB of RAM. If memory allocation fails, the fallback is to reading each frame three times (instead of once with framecache).

# ~~Apart from the framecache, XDS~~ needs (number of jobs)*(number of processes)*NX*NY*4 ~~Bytes~~, ~~plus about one GB~~ for the ~~code~~.

# Parallel processing with OpenMP uses several processes (threads). That additionally needs (number of jobs)*(number of processes)*NX*NY*4 bytes .

# Dectris provides the ''Neggia'' library ([https://github.com/dectris/neggia source],[https://www.dectris.com/support/downloads/sign-in binary]) for native reading of HDF5 files, which can be loaded into XDS at runtime using the <code>[[LIB]]=</code> [http://xds.mpimf-heidelberg.mpg.de/html_doc/xds_parameters.html#LIB= keyword]. With this library (which can also be found at https://{{SERVERNAME}}/pub/linux_bin for Linux, and at https://{{SERVERNAME}}/pub/mac_bin for MacOS), no conversion to CBF or otherwise is necessary. It is therefore just as fast and efficient to read HDF5 files as any other file format. At Diamond Light Source, a different HDF5 format was developed, and this requires the [https://github.com/DiamondLightSource/durin/releases/latest ''Durin'' plugin]. The latter can ~~also~~ read the HDF5 files written by the Dectris software, but frames are not read in parallel, so it is slower.

# Thus, if f = size of frame = NX*NY*4 bytes, j = number of jobs, t = number of threads, and c = NUMBER_OF_IMAGES_IN_CACHE, the required RAM for processing the pixels is j * f * ( t + c ) . Storage of intermediate results additionally needs roughly 1GB per job.

# Dectris provides the ''Neggia'' library ([https://github.com/dectris/neggia source],[https://www.dectris.com/support/downloads/sign-in binary]) for native reading of HDF5 files, which can be loaded into XDS at runtime using the <code>[[LIB]]=</code> [http://xds.mpimf-heidelberg.mpg.de/html_doc/xds_parameters.html#LIB= keyword]. With this library (which can also be found at https://{{SERVERNAME}}/pub/linux_bin for Linux, and at https://{{SERVERNAME}}/pub/mac_bin for MacOS), no conversion to CBF or otherwise is necessary. It is therefore just as fast and efficient to read HDF5 files as any other file format. At Diamond Light Source, a different HDF5 format was developed, and this requires the [https://github.com/DiamondLightSource/durin/releases/latest ''Durin'' plugin]. The latter can often read the HDF5 files written by the Dectris software, but frames are not read in parallel, so it is slower.

A suitable [[XDS.INP]] may have been written by the data collection (beamline) software. Latest [[generate_XDS.INP]] (using <code>generate_XDS.INP xxx_master.h5</code>) can be used if XDS.INP is not available.

@@ Line 2: / Line 2: @@
 == General aspects ==
-# The framecache of XDS uses memory to save on I/O; it saves a frame in RAM after reading it for the first time. By default, each XDS (or mcolspot/mintegrate) job stores NUMBER_OF_IMAGES_IN_CACHE=DELPHI/OSCILLATION_RANGE+1 images in memory which corresponds to one DELPHI-sized batch of data. This requires (number of pixels)*(number of jobs)*4 Bytes per frame which amounts to 72 MB in case of the Eiger 16M when running with MAXIMUM_NUBER_OF_JOBS=1. (If DELPHI=20 and OSCILLATION_RANGE=0.05 your computer thus has to have at least 400*72MB = 29GB of memory for each job!). If memory allocation fails, the fallback is to the old behaviour of reading each frame three times (instead of once).
+# The framecache of XDS uses memory to save on I/O; it saves a frame in RAM after reading it for the first time. Each frame needs (number of pixels)*4 bytes = 72 MB in case of the Eiger 16M. Each XDS (or mcolspot/mintegrate) job stores NUMBER_OF_IMAGES_IN_CACHE=DELPHI/OSCILLATION_RANGE+1 such frames in memory which corresponds to one DELPHI-sized batch of data. With MAXIMUM_NUBER_OF_JOBS=1, DELPHI=5 (the default) and OSCILLATION_RANGE=0.1, your computer thus has to have at least 50*72MB = 3.6GB of RAM. If memory allocation fails, the fallback is to reading each frame three times (instead of once with framecache).
-# Apart from the framecache, XDS needs (number of jobs)*(number of processes)*NX*NY*4 Bytes, plus about one GB for the code.
+# Parallel processing with OpenMP uses several processes (threads). That additionally needs (number of jobs)*(number of processes)*NX*NY*4 bytes .
-# Dectris provides the ''Neggia'' library ([https://github.com/dectris/neggia source],[https://www.dectris.com/support/downloads/sign-in binary]) for native reading of HDF5 files, which can be loaded into XDS at runtime using the <code>[[LIB]]=</code> [http://xds.mpimf-heidelberg.mpg.de/html_doc/xds_parameters.html#LIB= keyword]. With this library (which can also be found at https://{{SERVERNAME}}/pub/linux_bin for Linux, and at https://{{SERVERNAME}}/pub/mac_bin for MacOS), no conversion to CBF or otherwise is necessary. It is therefore just as fast and efficient to read HDF5 files as any other file format. At Diamond Light Source, a different HDF5 format was developed, and this requires the [https://github.com/DiamondLightSource/durin/releases/latest ''Durin'' plugin]. The latter can also read the HDF5 files written by the Dectris software, but frames are not read in parallel, so it is slower.
+# Thus, if f = size of frame = NX*NY*4 bytes, j = number of jobs, t = number of threads, and c = NUMBER_OF_IMAGES_IN_CACHE, the required RAM for processing the pixels is j * f * ( t + c ) . Storage of intermediate results additionally needs roughly 1GB per job.
+# Dectris provides the ''Neggia'' library ([https://github.com/dectris/neggia source],[https://www.dectris.com/support/downloads/sign-in binary]) for native reading of HDF5 files, which can be loaded into XDS at runtime using the <code>[[LIB]]=</code> [http://xds.mpimf-heidelberg.mpg.de/html_doc/xds_parameters.html#LIB= keyword]. With this library (which can also be found at https://{{SERVERNAME}}/pub/linux_bin for Linux, and at https://{{SERVERNAME}}/pub/mac_bin for MacOS), no conversion to CBF or otherwise is necessary. It is therefore just as fast and efficient to read HDF5 files as any other file format. At Diamond Light Source, a different HDF5 format was developed, and this requires the [https://github.com/DiamondLightSource/durin/releases/latest ''Durin'' plugin]. The latter can often read the HDF5 files written by the Dectris software, but frames are not read in parallel, so it is slower.
 A suitable [[XDS.INP]] may have been written by the data collection (beamline) software. Latest [[generate_XDS.INP]] (using <code>generate_XDS.INP xxx_master.h5</code>) can be used if XDS.INP is not available.