Queued Work Stream : Différence entre versions
m |
m |
||
| Ligne 15: | Ligne 15: | ||
A work stream is a series of "jobs" having a similar resource profile. In order not to overtax the system job scheduler with a myriad of relatively "small" work items, said items are inserted into "pseudo queues" and processed by a "master job". | A work stream is a series of "jobs" having a similar resource profile. In order not to overtax the system job scheduler with a myriad of relatively "small" work items, said items are inserted into "pseudo queues" and processed by a "master job". | ||
| − | A user's work stream(s) will be found in directory $HOME/.job_queues | + | A user's work stream(s) will be found in directory '''$HOME/.job_queues''' , this directory in turn contains subdirectories, one for each "pseudo queue". |
| − | More than one master job can go "fishing" into a "pseudo queue". | + | More than one master job can go "fishing" into a "pseudo queue". |
| − | The main characteristics of a work stream are | + | The main characteristics of a work stream are |
| − | *a name (arbitrary) | + | *a name (arbitrary) |
| − | *a set of pseudo queues (may be used to implement some sort of priority scheme) | + | *a set of pseudo queues (may be used to implement some sort of priority scheme) |
| − | *a computing surface (number of nodes) | + | *a computing surface (number of nodes) |
| − | *a duration (number of hours, days, weeks...) | + | *a duration (number of hours, days, weeks...) |
*a maximum idle time (if a stream is using a large number of nodes, its maximum idle time should be very short) | *a maximum idle time (if a stream is using a large number of nodes, its maximum idle time should be very short) | ||
=== How do i insert work into a work queue === | === How do i insert work into a work queue === | ||
| − | The [[Soumet : travaux par lots / batch jobs|ord_soumet]] utility is used to insert work into a "pseudo queue". The syntax is almost the same as for submitting a job to the system's batch scheduler. The "-q pseudo_queue_name@" parameter to ord_soumet is used to indicate that instead of being submitted directly, the piece of work (job) should rather be inserted into the "pseudo_queue_name" work queue.<br> | + | The [[Soumet : travaux par lots / batch jobs|ord_soumet]] utility is used to insert work into a "pseudo queue". The syntax is almost the same as for submitting a job to the system's batch scheduler. The "'''-q pseudo_queue_name@'''" parameter to ord_soumet is used to indicate that instead of being submitted directly, the piece of work (job) should rather be inserted into the "pseudo_queue_name" work queue.<br> |
=== How do i start a master job for a work stream === | === How do i start a master job for a work stream === | ||
| − | By submitting a master job with the (to come) u.run_work_stream command | + | By submitting a master job with the (to come) '''u.run_work_stream''' command |
=== How do i control a work stream === | === How do i control a work stream === | ||
| + | |||
| + | === How do i abort and rerun a piece of work === | ||
| + | |||
| + | a piece of work may abort and signal to the master job that it should be rerun (up to N times) with the following command | ||
| + | |||
| + | '''. exit_and_rerun_work.dot N''' | ||
| + | |||
| + | this command will also make sure that the post work cleanup code inserted by ord_soumet will not be performed<br> | ||
Version depuis le 28 de novembre 2011 à 13:21
|
en construction |
under construction |
Matières
Work Streams
What is a work stream
A work stream is a series of "jobs" having a similar resource profile. In order not to overtax the system job scheduler with a myriad of relatively "small" work items, said items are inserted into "pseudo queues" and processed by a "master job".
A user's work stream(s) will be found in directory $HOME/.job_queues , this directory in turn contains subdirectories, one for each "pseudo queue".
More than one master job can go "fishing" into a "pseudo queue".
The main characteristics of a work stream are
- a name (arbitrary)
- a set of pseudo queues (may be used to implement some sort of priority scheme)
- a computing surface (number of nodes)
- a duration (number of hours, days, weeks...)
- a maximum idle time (if a stream is using a large number of nodes, its maximum idle time should be very short)
How do i insert work into a work queue
The ord_soumet utility is used to insert work into a "pseudo queue". The syntax is almost the same as for submitting a job to the system's batch scheduler. The "-q pseudo_queue_name@" parameter to ord_soumet is used to indicate that instead of being submitted directly, the piece of work (job) should rather be inserted into the "pseudo_queue_name" work queue.
How do i start a master job for a work stream
By submitting a master job with the (to come) u.run_work_stream command
How do i control a work stream
How do i abort and rerun a piece of work
a piece of work may abort and signal to the master job that it should be rerun (up to N times) with the following command
. exit_and_rerun_work.dot N
this command will also make sure that the post work cleanup code inserted by ord_soumet will not be performed
