Cluster Installation: Difference between revisions

(4 intermediate revisions by the same user not shown)

Line 83:

export PATH=$PATH

echo $SGE_TASK_ID | $JOB

</pre>

== setup of XDS for a containerized system ==

In order to setup XDS for a container system, the ''forkxds'' script needs to be changed. Emilio Centeno <ecenteno@cells.es> created a "container-friendly" forkxds. This is the piece of code affected, with just one changed line:

<pre>

do

if [ $nhosts -gt 1 ] #distribute jobs among the cluster nodes

then

j=$(( (itask -1) % nhosts )) #changed from % nhosts + 1 to nhosts

# Original line

# echo "$itask" | ssh -x ${rhosts[$j]} "cd $PWD && $amain && sync" &

# Image in file (.sif)

# echo "$itask" | ssh -o StrictHostKeyChecking=no -x ${rhosts[$j]} "cd $PWD && ml Apptainer && apptainer exec /container_path/my_xds_container.sif $amain && sync" &

# Image already loaded in an instance

echo "$itask" | ssh -o StrictHostKeyChecking=no -x ${rhosts[$j]} "cd $PWD && ml Apptainer && apptainer exec instance://my_xds_instance $amain && sync" &

else

echo "$itask" | $amain && sync & #submit all jobs to the peer node

fi

pids="$pids $!" #append id of the new background process

itask=`expr $itask + 1`

# NOTE: sync after $amain complete pending disk writes on each node

done

</pre>

== Performance ==

If the ~~cluster~~ has ~~nodes with more cores~~ than the node ~~where xds_par was started,~~ the ~~binary file mintegrate.tmp (written by xds_par,~~ and ~~read by~~ mintegrate_par) ~~must~~ be ~~patched,~~ by ~~adding these two lines to the forkxds script:~~

~~printf "0~~: ~~%02x000000" <desired_number_of_threads> | xxd -r -g0 > nn.dat~~

Cluster nodes may have different numbers of processors.

~~dd bs~~=~~4 count=1 seek=13 conv=notrunc~~ if=~~nn.dat~~ of~~=mintegrate.tmp~~

Please note that the output line

~~If the cluster~~ is ~~homogeneous~~ then ~~this~~ is ~~of course no concern~~.

number of OpenMP threads used NN

in COLSPOT.LP and INTEGRATE.LP may be incorrect if MAXIMUM_NUMBER_OF_JOBS > 1, and the submitting node (the node that runs xds_par) has a different number of processors than the processing node(s) (the nodes that run mcolspot_par and mintegrate_par). The actual numbers of threads on the processing nodes may be obtained by

grep PARALLEL COLSPOT.LP

grep USING INTEGRATE.LP | uniq

The algorithm that determines the number of threads used on a processing node is:

NB = DELPHI / OSCILLATION_RANGE # this may be slightly adjusted by XDS if DATA_RANGE / NB is not integer

NCORE = number of processors in the processing node, obtained by OMP_GET_NUM_PROCS()

if MAXIMUM_NUMBER_OF_PROCESSORS is not specified in XDS.INP then MAXIMUM_NUMBER_OF_PROCESSORS = NCORE

number_of_threads = MIN( NB, NCORE, MAXIMUM_NUMBER_OF_PROCESSORS, 99 )

This is implemented in BUILT=20191015 onwards.

@@ Line 83: / Line 83: @@
 export PATH=$PATH
 echo $SGE_TASK_ID | $JOB
+</pre>
+== setup of XDS for a containerized system ==
+In order to setup XDS for a container system, the ''forkxds'' script needs to be changed. Emilio Centeno <ecenteno@cells.es> created a "container-friendly" forkxds. This is the piece of code affected, with just one changed line:
+<pre>
+do
+   if [ $nhosts -gt 1 ]        #distribute jobs among the cluster nodes
+   then
+      j=$(( (itask -1) % nhosts )) #changed from % nhosts + 1 to nhosts
+     # Original line
+      # echo "$itask" | ssh -x ${rhosts[$j]} "cd $PWD && $amain && sync" &
+      # Image in file (.sif)
+      # echo "$itask" | ssh -o StrictHostKeyChecking=no -x ${rhosts[$j]} "cd $PWD && ml Apptainer && apptainer exec /container_path/my_xds_container.sif $amain && sync" &
+      # Image already loaded in an instance
+      echo "$itask" | ssh -o StrictHostKeyChecking=no -x ${rhosts[$j]} "cd $PWD && ml Apptainer && apptainer exec instance://my_xds_instance $amain && sync" &
+   else
+      echo "$itask" | $amain && sync &  #submit all jobs to the peer node
+   fi
+   pids="$pids $!"             #append id of the new background process
+   itask=`expr $itask + 1`
+   # NOTE: sync after $amain complete pending disk writes on each node
+done
 </pre>
 == Performance ==
-If the cluster has nodes with more cores than the node where xds_par was started, the binary file mintegrate.tmp (written by xds_par, and read by mintegrate_par) must be patched, by adding these two lines to the forkxds script:
-  printf "0: %02x000000" <desired_number_of_threads> | xxd -r -g0 > nn.dat
+Cluster nodes may have different numbers of processors.
-  dd bs=4 count=1 seek=13 conv=notrunc if=nn.dat of=mintegrate.tmp
+Please note that the output line
-If the cluster is homogeneous then this is of course no concern.
+ number of OpenMP threads used  NN
+in COLSPOT.LP and INTEGRATE.LP may be incorrect if MAXIMUM_NUMBER_OF_JOBS > 1, and the submitting node (the node that runs xds_par) has a different number of processors than the processing node(s) (the nodes that run mcolspot_par and mintegrate_par). The actual numbers of threads on the processing nodes may be obtained by
+  grep PARALLEL COLSPOT.LP
+ grep USING INTEGRATE.LP | uniq
+The algorithm that determines the number of threads used on a processing node is:
+  NB = DELPHI / OSCILLATION_RANGE   # this may be slightly adjusted by XDS if DATA_RANGE / NB is not integer
+ NCORE = number of processors in the processing node, obtained by OMP_GET_NUM_PROCS()
+ if MAXIMUM_NUMBER_OF_PROCESSORS is not specified in XDS.INP then MAXIMUM_NUMBER_OF_PROCESSORS = NCORE
+ number_of_threads = MIN( NB, NCORE, MAXIMUM_NUMBER_OF_PROCESSORS, 99 )
+This is implemented in BUILT=20191015 onwards.