XDSwiki

LIB: Difference between revisions

← Older edit
VisualWikitext
No edit summary
m →Glue code: change X*Y*DEPTH to X*Y*NBYTE
 
(29 intermediate revisions by 2 users not shown)
Line 1: Line 1:
The possibility of using external libraries (that are loaded at runtime) has been available in C/C++ from the beginning, but in Fortran became only available in Fortran2003.  
The possibility of using external libraries (that are loaded at runtime) has been available in C/C++ for a long time, but in Fortran became only available as of Fortran2003.  


In the case of XDS, frame-reading and computation can be separated starting with version November-2016. This allows users/companies to develop their own specialized frame-reading libraries, and relieves the XDS maintainers from implementing even more file formats.
In the case of XDS, frame-reading and computation can be separated starting with version November-2016. The [https://xds.mr.mpg.de/html_doc/xds_parameters.html#LIB= LIB=] keyword allows users/companies to develop their own specialized frame-reading libraries, and relieves the XDS maintainers from implementing even more file formats. The feature was developed in order to be able to natively (i.e. without temporary intermediates) read the HDF5 files written for data from the [[Eiger]] detector.


In the following, small examples are given for
In the following, small examples are given for
* how a program ("'''host'''") may use an existing external library, e.g. the dectris-neggia library ([https://github.com/dectris/neggia source]; [https://www.dectris.com/neggia.html#main_head_navigation pre-compiled])
* how a program ("'''host'''") may use an existing external library, e.g. the dectris-neggia library ([https://github.com/dectris/neggia source]; [https://www.dectris.com/company/news/newsroom/news-details/process-eiger-data-with-xds-fast pre-compiled])
* how an external library ("'''client'''") may be implemented that XDS can use
* how an external library ("'''client'''") may be implemented that XDS can use


The glue code between host and client is based on http://cims.nyu.edu/~donev/Fortran/DLL/DLL.Forum.txt . There should be no need to change this, unless the interface design changes.
The [[LIB#Glue_code|glue code]] between host and client is based on http://cims.nyu.edu/~donev/Fortran/DLL/DLL.Forum.txt . There should be no need to change this, unless the interface design changes.


The interface was designed by Markus Methis (Dectris), Vittorio Boccone (Dectris) and Kay Diederichs. It is supposed to be generic, i.e. useful beyond XDS. In particular, the 4096 bytes of the info_array can be utilized to obtain and use header information (e.g. wavelength, distance, axes specifications and other metadata).   
The interface was designed by Markus Mathes (Dectris), Vittorio Boccone (Dectris) and Kay Diederichs. It is supposed to be generic, i.e. useful beyond XDS. In particular, the 4096 bytes of the info_array can be utilized to obtain and use header information (e.g. wavelength, distance, axes specifications and other metadata).   
 
The interface has several [[LIB#Existing_implementations|implementations]].


== Host code example ==
== Host code example ==
<pre>
<pre>
! example test program for existing external library
! Example test program for existing external library
! This should be saved in a file called test_generic_host.f90
! Kay Diederichs 4/2017
! Kay Diederichs 4/2017
!
!
Line 19: Line 22:
! ifort -qopenmp generic_data_plugin.f90 test_generic_host.f90 -o test_generic_host
! ifort -qopenmp generic_data_plugin.f90 test_generic_host.f90 -o test_generic_host
! or
! or
! gfortran -O -fopenmp -ldl generic_data_plugin.f90 test_generic.f90 -o test_generic_host
! gfortran -O -fopenmp generic_data_plugin.f90 test_generic_host.f90 -ldl -o test_generic_host
! run with  
! run with  
! ./test_generic_host < test.in
! ./test_generic_host < test.in
Line 30: Line 33:




PROGRAM test_generic
PROGRAM test_generic_host
     USE generic_data_plugin, ONLY: library, firstqm, lastqm, nx, ny, is_open, &
     USE generic_data_plugin, ONLY: library, firstqm, lastqm, nx, ny, is_open, &
         generic_open, generic_get_header, generic_get_data, generic_close
         generic_open, generic_get_header, generic_get_data, generic_close
Line 51: Line 54:
      
      
! set some more module variables
! set some more module variables
     firstqm=INDEX(actnam,'?')
     firstqm=INDEX(actnam,'?')   ! qm means question mark
     lastqm =INDEX(actnam,'?',BACK=.TRUE.)
     lastqm =INDEX(actnam,'?',BACK=.TRUE.)
     len    =LEN_TRIM(actnam)
     len    =LEN_TRIM(actnam)
Line 115: Line 118:


== Client code example ==
== Client code example ==
The following code should be saved as file test_generic_client.f90 :
<pre>
<pre>
! This reads single data files which have a header of 7680 bytes
! This reads single data files which have a header of 7680 bytes
! Kay Diederichs 4/2017
! Kay Diederichs 4/2017
! Kay Diederichs 7/2021 add code for the case that fn_template has no '?', and simplify&comment gfortran command.
!
! compile with
! compile with
! ifort -fpic -shared -static-intel -qopenmp -traceback -sox test_generic_client.f90 -o libtest_generic_client.so
! ifort -fpic -shared -static-intel -qopenmp -qopenmp-link=static -traceback -sox test_generic_client.f90 -o libtest_generic_client.so
! (this includes all required compiler libraries into the libtest_generic_client.so library)
! or
! or
! gfortran -c -fpic test_generic_client.f90 && ld -shared test_generic_client.o -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/ -lgfortran -o libtest_generic_client.so
! gfortran -fpic test_generic_client.f90 -shared -o libtest_generic_client.so
! (adding -static-libgfortran and -static-libquadmath should put these into the library as well so that it can be distributed.
! This requires a gfortran version higher than 12. For older gfortran versions, libgfortran.so and libquadmath.so can be
! renamed to something that the linker does not search for; it will then include libgfortran.a and libquadmath.a)
! The resulting file can be used with a LIB=./libtest_generic_client.so line in XDS.INP, and enables
! reading of data files with a 7680 bytes header plus 1024*1024 pixels of integer data, without any record structure.


MODULE plugin_test_mod
MODULE plugin_test_mod
       CHARACTER :: fn_template*132=''
       CHARACTER :: fn_template*132='',cformat*6='(i4.4)'
       INTEGER  :: lenfn,firstqm,lastqm
       INTEGER  :: lenfn,firstqm,lastqm
END MODULE
END MODULE
Line 147: Line 159:
       firstqm=INDEX(fn_template,'?')
       firstqm=INDEX(fn_template,'?')
       lastqm =INDEX(fn_template,'?',BACK=.TRUE.)
       lastqm =INDEX(fn_template,'?',BACK=.TRUE.)
      IF (firstqm==0) THEN
        firstqm=lenfn-7
        lastqm =lenfn-4
      END IF
      WRITE(cformat(3:5),'(i1,a1,i1)')lastqm-firstqm+1,'.',lastqm-firstqm+1
END SUBROUTINE plugin_open
END SUBROUTINE plugin_open
!
!
Line 177: Line 194:
! local variables
! local variables
     INTEGER k,i,dummy
     INTEGER k,i,dummy
     CHARACTER :: fn*132,cformat*6='(i4.4)'
     CHARACTER :: fn*132
     fn=fn_template
     fn=fn_template
    WRITE(cformat(3:5),'(i1,a1,i1)')lastqm-firstqm+1,'.',lastqm-firstqm+1
     IF (frame_number>0) WRITE(fn(firstqm:lastqm),cformat) frame_number
     IF (frame_number>0) WRITE(fn(firstqm:lastqm),cformat) frame_number
! -qopenmp compile option needs to be used otherwise race in writing fn
! -qopenmp compile option needs to be used otherwise race in writing fn
Line 219: Line 235:
! THE USE OR OTHER DEALINGS IN THE SOFTWARE.
! THE USE OR OTHER DEALINGS IN THE SOFTWARE.
!
!
! For more information, please refer to For more information, please refer to <http://unlicense.org/>
! For more information, please refer to <http://unlicense.org/>
!
!
!
!
Line 245: Line 261:
        
        
       interface ! strlen is a standard C function from <string.h>
       interface ! strlen is a standard C function from <string.h>
         function strlen(string) result(len) bind(C,name="strlen")
         integer(int64) function strlen(string) result(len) bind(C,name="strlen")
            use iso_fortran_env, only : int64
             use iso_c_binding
             use iso_c_binding
             type(c_ptr), value :: string ! a C pointer
             type(c_ptr), value :: string ! a C pointer
Line 515: Line 532:
     !  'NX' (integer)                  output  Number of pixels along X  
     !  'NX' (integer)                  output  Number of pixels along X  
     !  'NY' (integer)                  output  Number of pixels along Y
     !  'NY' (integer)                  output  Number of pixels along Y
     !  'NBYTE' (integer)              output  Number of bytes in the image... X*Y*DEPTH
     !  'NBYTE' (integer)              output  Number of bytes in the image is X*Y*NBYTE
     !  'QX' (4*REAL)                  output  Pixel size
     !  'QX' (4*REAL)                  output  Pixel size
     !  'QY' (4*REAL)                  output  Pixel size
     !  'QY' (4*REAL)                  output  Pixel size
Line 639: Line 656:


     ! now close the dl:
     ! now close the dl:
     status=dlclose(handle)
     status=0  ! inserted Feb 3, 2021 KD
!    status=dlclose(handle) ! commented out Feb 3, 2021 KD
     if(status/=0) then
     if(status/=0) then
       write(6,*) "[generic_data_plugin] - ERROR - Cannot open Handle"
       write(6,*) "[generic_data_plugin] - ERROR - Cannot open Handle"
Line 664: Line 682:
#3  0x00000000409b873d in clone () from /lib64/libc.so.6
#3  0x00000000409b873d in clone () from /lib64/libc.so.6
</pre>
</pre>
Google returns https://github.com/apple/cups/issues/4410 and https://bugzilla.redhat.com/show_bug.cgi?id=1065695 when searching for similar problems. Overall, this appears to be harmless and in fact I don't know how to change the code to make the segfault disappear  - I'd appreciate a patch!
Google returns https://github.com/apple/cups/issues/4410 and https://bugzilla.redhat.com/show_bug.cgi?id=1065695 when searching for similar problems. Overall, this appears to be harmless and in fact I don't know how to change the code to make the segfault disappear  - I'd appreciate a patch!  
Feb 3, 2021: commented out the line 'status=dlclose(handle)' and replaced it with 'status=0'. According to Sebastian Thorarensen this has no negative consequences on XDS, and solves the segfault problem.
 
== Existing implementations ==
 
# [https://github.com/dectris/neggia Dectris Neggia-plugin] to read HDF5 written by Dectris-supplied software of Eiger detectors
# [https://github.com/DiamondLightSource/durin Diamond's Durin-plugin] to read HDF5 written by Eiger detectors at Diamond (and presumably elsewhere); latest binaries for MacOS and Linux (RHEL6) as well as example XDS.INP and source at https://github.com/DiamondLightSource/durin/releases/latest . A binary for M1 Mac is available - see [[Installation]]
# [https://git.embl.de/nikolova/xds-zcbf/ EMBL-Hamburg's zcbf-plugin] to read gzip-compressed CBF files without intermediate file . A binary for M1 Mac is available - see [[Installation]].
 
Plugins for Linux and Intel-Mac can also be obtained through [https://www.globalphasing.com/autoproc/ GPhL's autoPROC].
 
See also [[Installation]].
 
== See also ==
# https://rosettacode.org/wiki/Call_a_function_in_a_shared_library#GNU_Fortran_on_Linux
Retrieved from "https://wiki.uni-konstanz.de/xds/index.php/LIB"