Cluster Installation

XDS can be run in cluster mode using any command line job scheduling software such as Grid Engine, Condor, Torque/PBS, LSF, SLURM etc. We implemented Grid Engine. It is a distributed resource management system which monitors the CPU and memory usage of the available computing resources and schedules the job to the least used computer. Grid Engine was chosen due to its high scalability, cost effectiveness, ease of maintenance and high throughput. Grid Engine was developed by Sun Microsystems'^TM^' (Sun Grid Engine, SGE) and later acquired by Oracle'^TM^' and subsequently acquired by UNIVA'^TM^'. The latest versions became closed source, but the older ones are open source supplied with many Linux distributions including Redhat/CentOS 6.x. There is also open source Open Grid Scheduler [[1]], Son of Gridengine [[2]]

Grid Engine consists of a master node daemon named sgemaster which schedules jobs to execution nodes. On each execution node a daemon named sge_execd runs a job and sends a completion signal back to sgemaster. Jobs are submitted to sgemaster using command such as qsub or using DRMAA C, JAVA or IDL bindings from any applications want to run XDS.

Grid Engine Installation

Redhas/CentOS Linux distribution comes with rpms for installing Grid Engine. One need to have administrative privileges to install. Install gridengine rpms on all the nodes using following command, Default shell for Grid Engine is /bin/csh [@ root@sudhir:/home/spothineni 1> yum install gridengine gridengine-qmaster gridengine-execd gridengine-qmon

root@sudhir:/home/spothineni 2> rpm -qa | grep gridengine

gridengine-qmaster-6.2u5-10.el6.4.x86_64 gridengine-qmon-6.2u5-10.el6.4.x86_64 gridengine-execd-6.2u5-10.el6.4.x86_64 gridengine-6.2u5-10.el6.4.x86_64 @]

By default gridengine installation directory /usr/share/gridengine, contents shown below.

[@ root@sudhir:/home/spothineni 3> cd /usr/share/gridengine

root@sudhir:/home/spothineni 4> ls bin default hadoop install_execd lib my_configuration.conf qmon utilbin ckpt doc inst_sge install_qmaster mpi pvm util @]

On bl1upper which qmaster node install using install_qmaster

[@ root@bl1upper:/usr/share/gridengine 5>./install_qmaster @]

Most of the answers are yes/no or typing enter.

Follwoing important environment variables are written to /usr/share/gridengine/default/common/settings.csh which should be in the $PATH.

$SGE_ROOT=/usr/share/gridengine $SGE_QMASTER_PORT=6444 $SGE_EXECD_PORT=6445 $SGE_CELL=default

There is an option to give administrative email which is very useful, when ever there is any problem gridengine will send error messages to email. Ready with a file contains admin and submit hosts or you can manually enter all the hosts separated by space, use full DNS names of hosts. In this installation shadow host is not used. After the shadow host step make sure allhosts group and all.q are created otherwise installation sge_execd will have problems. Scheduler Tuning selected as 'Max', it has disadvantage, gridengine immediately schedules with out assuming the load, this will cause successive job submissions will go to same host until all the slots are filled for that machine. Selecting 'Normal' will assume the load but there is overhead of few sec. extra time for job scheduling.

After finishing the installation the configuration files are automatically written to the directory /usr/share/gridengine/default since the cell name selected is 'default'. This directory can be choosen as a shared directory over NFS. Otherwise copy this directory to every host used int the cluster.

On execution node install execution daemon using following command

[@ root@bl1ws1:/usr/share/gridengine 5>./install_execd @]

the input is almost typing return if you already copied the 'default' directory to this node.

Son of Gridengine

rpms available in this link

http://arc.liv.ac.uk/downloads/SGE/releases/8.1.8/

by defualt these rpms install in single directory /opt/sge instead of scattering (by default) files to /usr/bin, /usr/share/gridengine, /usr/spool/gridengine

Default shell for Son of Gridengine is /bin/sh which is /bin/bash