Queued Work Stream

en construction

under construction

Work Streams

What is a work stream

A work stream is a series of "jobs" having a similar resource profile. In order not to overtax the system job scheduler with a myriad of relatively "small" work items, said items are inserted into "pseudo queues" and processed by a "master job".

• A user's work stream(s) will be found in directory $HOME/.job_queues
  This directory in turn contains subdirectories, one for each "pseudo queue".
• More than one master job can go "fishing" into a "pseudo queue".
• Job monitoring will be started by the master job using u.job-monitor

The main characteristics of a work stream are

• a name (arbitrary)
• a set of pseudo queues (may be used to implement some sort of priority scheme)
• a computing surface (number of nodes)
• a duration (number of hours, days, weeks...)
• a maximum idle time (if a stream is using a large number of nodes, its maximum idle time should be very short)

How do i insert work into a work queue

The ord_soumet utility is used to insert work into a "pseudo queue". The syntax is almost the same as for submitting a job to the system's batch scheduler. The "-q pseudo_queue_name@" parameter to ord_soumet is used to indicate that instead of being submitted directly, the piece of work (job) should rather be inserted into the "pseudo_queue_name" work queue.

How do i start a master job for a work stream

By submitting a master job with the (to come) u.run_work_stream command

How do i abort and rerun a piece of work

a piece of work may abort and signal to the master job that it should be rerun (up to N times) with the following command

. exit_and_rerun_work.dot N

this command will also make sure that the post work cleanup code inserted by ord_soumet will not be performed