ManyJobs

Introduction

This tool submits jobs to many clusters remotely from one machine (maybe your desktop). When a job starts, the computing node contacts to the master-host (maybe your desktop) and the master-host sends parameters for a job and set in environmental variables. When a job ends, the computing node contacts to the master-host (maybe your desktop) and the master-host submit a new job. One can set job dependency; if job A depends on job B, job A won't be assigned until job B ends. Currently, all communication is done with 'ssh' command.

It is necessary to set up 'ssh' to work to / from he clusters without password. ManyJobs is a python based tool for managing ManyTasks on widely distributed supercomputing resources. To date it has been used in production mode on the following machines: Abe@NCSA, Lonestar@TACC, Ranger@TACC, QueenBee@LONI, Kraken@NICS, Steele@RCAC, (eric,louie,poseidon,painter,oliver)@LONI, and the CCS Cluster at Tulane's CCS. All you need is an account with a user allocation in one of the above machines and you too can run Many Jobs.

Requirements

• ManyJobs is written in Python (>=2.4) script only.
  
• You have to set ssh authentication to ssh/gsissh to/from remote cluster head nodes without password by use of password keys.

Getting Started

• Check python installation
  
• Setting SSH/GSISSH password keys for both directions
  
• Prepare DataBase File.
  
• Prepare Job Submission Scripts.
  
• Prepare User Tasks Scripts.
  
• Setting User executable files on each cluster.
  

User Commands

• StartManyJobs.py
• DeleteAll.py
• Flush.py
• ClearDataBase.py
• CopyScript.py

Acknowledgement

ManyJob is developed as part of a LONI Institute Project to Tom Bishop, Shantenu Jha and Hideki Fujioka. Fujioka is the main developer. ManyJob provides a simple self-contained implementation of the SAGA-based BigJob concept. ManyJob has received further support from NSF Cybertools and LaSIGMA Projects.
-----------------------------------------------------------------------