2,652
edits
Performance: Difference between revisions
→processing compressed data
| Line 26: | Line 26: | ||
XDS can process data files that were previously compressed with compress (<code>.Z</code>), gzip (<code>.gz</code>), bzip2 (<code>.bz2</code>) or xz (<code>.xz</code>). It does this by on-the-fly decompression to temporary files with standard names (<code>SCRATCH2XXYY.tmp</code>) where XX (XX = 01..99) stands for the "JOB" and YY (YY = 01..99) for the thread number that produces the temporary file. | XDS can process data files that were previously compressed with compress (<code>.Z</code>), gzip (<code>.gz</code>), bzip2 (<code>.bz2</code>) or xz (<code>.xz</code>). It does this by on-the-fly decompression to temporary files with standard names (<code>SCRATCH2XXYY.tmp</code>) where XX (XX = 01..99) stands for the "JOB" and YY (YY = 01..99) for the thread number that produces the temporary file. | ||
Compression saves a lot of disk space, but decompression | Compression saves a lot of disk space, but decompression may be time-consuming in terms of CPU and I/O. The penalty associated with decompression can be mitigated by | ||
* using a [http://xds.mpimf-heidelberg.mpg.de/html_doc/xds_parameters.html#LIB= LIB=] plugin that saves the I/O, and overhead of running an external program. This exists for gzip-compressed CBF files; see [[LIB#Existing_implementations]]. | |||
* (Linux only) using symlinks pointing to /dev/shm which results in <code>SCRATCH2XXYY.tmp</code> being written to RAM instead of (network) disk. A script (typically called <code>mklinks</code>) achieving this is | * (Linux only) using symlinks pointing to /dev/shm which results in <code>SCRATCH2XXYY.tmp</code> being written to RAM instead of (network) disk. A script (typically called <code>mklinks</code>) achieving this is | ||
<pre> | <pre> | ||