Chunk lance
Chunk_lance
Chunk_lance allows to run a sequence of monthly (or sub-monthly) model jobs in one big job.
A GEM/GEMCLIM/CRCM climate simulation usually consists of a sequence of monthly jobs. But even a weather forecast simulation can consist of a sequence of, for example, n-daily jobs (set using the parameter 'Fcst_rstrt_S').
Each job consists of 2 parts:
- scripts (copies restart files from previous month, prepare config files for new month)
- model (main model job)
Chunk_lance will run a series of these 2 part jobs, checking at the end of each job if there is still enough time to calculate another job. If yes another job will get executed, if not an new chunk_job will get submitted automatically(!).
As with Um_lance, Chunk_lance needs to get executed from the config file directory and the model environment needs to have been set. To start a simulation from the beginning add the key '-start';
Chunk_lance -start
To continue a simulation just execute the command without any key:
Chunk_lance
The wallclock time for which one chunk_job will be running can be set in the file 'configexp.dot.cfg' with the parameter 'BACKEND_time_mod'.
In case the model job fails it will automatically get reexecuted a second time.
In case the whole simulation crashes, it can easily get resubmitted by executing 'Chunk_lance' again.
To know where the simulation was, Chunk_lance is using a log file called 'chunk_job.log'. The chunk_job itself will check this file to determine which job to execute next. Therefore this log file should not be touched.
However, to rerun part of a simulation one can alter the log file by hand. Just make sure there is never a blank line at the end of the log file since the chunk_job only checks the very last line of the log file!
But sometimes it happens that part of the restart files gets overwritten before a month is finished or that the restart files are corrupted. In that case we cannot just relaunch the month with Chunk_lance but have to get the original last uncorrupted restart files back (from the previous month) etc. and restart the simulation from there.
When the model job stops for whatever reason, there is an automatic check in Chunk_lance if the restart files are still the original ones or not. If they got already modified the following message will appear in the "chunk_job listing" (!!!) not in the model listing:
At least one of the restart files got already rewritten
Therefore the model could not get restarted automatically
You have to restart your simulation starting from the previous restart files
----- ABORT -----
If you see this message please do not simply restart the simulation with Chunk_lance but as said, from the previous restart file - see below.
Since the running job is now always called "cjob_${exp}_..." one cannot see anymore from the job name how far the simulation has progressed. But one can always have a look at the listings directory and also a log file is kept in the config file directory called 'chunk_job.log'.
Restart using Chunk_lance
In case a simulation stops and you want to find out which job (scripts, entry or model) crashed, you have have look in the listings directory (~/listings/${TRUE_HOST}). Check which of the following jobs has crashed:
${exp}_S (Scripts)
${exp}_M (Model)
Or you can have a look at the log file "chunk_job.log".
In any case, you can restart your simulation by simply executing
Chunk_lance
again in the config file directory.
Of course AFTER you fixed the problem - unless it was a machine problem. In the latter case, just restart the simulation with 'Chunk_lance'.
Restart from previous restart file
Continue simulation from a restart file:
* Copy restart file of previous month (if there is more than 1 part, copy all parts!) from the archive (${CLIMAT_archdir}/Restarts) back into the execution directory "~/MODEL_EXEC_RUN/$TRUE_HOST"
* Gunzip and unarchive (cmcarc -x -f ...) the restart file(s) in ~/MODEL_EXEC_RUN/$TRUE_HOST
The cmcarc-command will create a new directory, but the *.ca file will remain in the directory. You can remove it again.
* Go into the config file directory
* Edit the log file 'chunk_job.log':
(First I suggest to make a backup copy of the log file.)
The remove all lines concerning the month you want to rerun and all following lines.
* If you are running the entry in parallel,
you will also have to remove all the ${exp}_entry_finished flags in your config file directory for all the months you want to rerun.
Otherwise the entries for these months will not get rerun!
* Execute "Chunk_lance" again
