GEM 5.0 running

Creating the executable

a) Create directory

First create yourself a directory under your home(!) in which you want to create the executable.
I usually call mine:

 ~/gem/${GEM_version}/Abs/...

For GEM4: GEM_version=v_4.8.12.u
For GEM5: GEM_version=v_5.0.0

b) Set up directory

Go into the executable directory you created above.
Whenever you want to create/modify an executable you need to set the model environment first:

 For GEM4 with:   . s.ssmuse.dot GEM/x/4.8.lts12 or the alias 4812
 For GEM5 with:   . r.load.dot GEM/x/5.0         or the alias 500

1) Open experiment
First you have to "open the experiment" to do the initial setup, create files to find original source code, create links to the building directory and Makefiles etc. with:

 ouv_exp_gem
 rdemklink -v	

You have to do this only once!

2) Duplicating another executable directory (optional!)

If you want to duplicate another executable directory (from yourself or someone else) you must copy ONLY the source code, nothing else!
To copy all relevant files copy all files matching:

 *.[Ffchi]*

3) Create dependency
Then you need to create the dependencies (which routines include which include files and, vice versa, which include files are included in which routines). You need to redo this whenever you include another include file or add a routine!!!:

 make dep

It never hurts to redo this. It just takes time.

4) Compile
Whenever you modified or added one or more source code files you need to compile them to create the object file(s). To compile all routines that need compiling or to compile a modified module or include file execute:

 make obj
To speed up this compilation on Beluga you can also use 3 cpus with :
 make -j3 obj

make obj will (re)compile all routines that need compiling. It never hurts to use make obj, it just takes time.
If you modified only one subroutine all you need to do is recompile it with

 make subroutine.o

You can also compile several routines in the same "make" command: make subroutine1.o subroutine2.o

5) Create the executable
To (re)create the executable you need to load and link it with:

 make gemdm

c) Modify the source code

Once you set the environment the variables $gemdyn and $rpnphy will get set, pointing to the source code in the environment:

 ${gemdyn}/src/*/*
 ${rpnphy}/src/*/*

You can check there which routines you want to modify. Once you know which routine(s) you want to modify go back into your "executable directory". Then "check out" all the routines you want to modify with:

   rde co  subroutine.F90 
or
   rde co  comdeck.cdk

You can also check out several routines at the same time.
Once you modified or copied the routines you want you need to compile them and then recreate the executable, following points b) 2) to 4) above.

Listings

While the model is running is produces several listings. They all start with the name of the experiment, exp, followed by the year and month of the current month:
      ${exp}_YYYYMM_*

When starting a simulation with 'Chunk_lance' the output of the main 'chunk_job' will go into a listing called:
      cjob_${exp}.n.*

While a month is running all listings will go into the directory:

   ~/listings/${TRUE_HOST}

At the end of a month all model listings (${exp}_YYYYMM_*) will get archived under:

   ${CLIMAT_archdir}/${exp}/Listings/listings_${exp}.zip

These are the different types of listings:

a) Model listings

• ${exp}_YYYYMM_S*
  

This is the output of the Scripts which prepare directories and update the config files. When starting a simulation with 'Um_lance' this output appears in the window in which you executed the command. When a simulation is started with 'Chunk_lance' or when a second month is launched automatically the output of the scripts get redirected into a listing with the name ${exp}_S*.

• ${exp}_YYYYMM_M*
  

This is the main model output.
When a model job is running, each MPI process will write is output in a separate file, so while a simulation is running you will only see the first part of the output in the listing. Close to the end you will find a line starting with:
      INFO: temporary listings for all members in ...
Under that path you will find all the temporary process listings. Once a job finished all process listings will get added to the main model listing. If you want to see the simulation advancing while it is running, you can execute the command 'u.tail' on the main model listing. It will do a 'tail -f' on the listing of the main MPI process:
      u.tail ${exp}_YYYYMM_M...
Whenever there are problems with the model look for the error in the last of these two listings, which ever it may be, '_S' or '_M'. I usually execute something like:

   ls -lrt ${exp}*_[MS][._]*

b) Post processing listings

Every now and then, but at least at the end of timestep 0 and at the end of each job, the model executes a little job, preparing and submitting a post processing job. While the post processing is running it will produce the following listings:

• ${exp}_YYYYMM_PP0* and ${exp}_YYYYMM_PP*
  

Listing of the job creating and submitting the Post Processing job. If need be you can find in these listings the command to resubmit a post processing job. Look for 'soumet' close to the end. The '_PP_' job is the last job, executed at the end of a job. If post processing jobs were submitted during a job their submitting job will be called '_PP0_'.

• ${exp}_YYYYMM_DX*
  

This is the listing of the main post processing job. You will see there the results of the combination of the files, the transfer (Xfer) in the archive and the calculation of the Diagnostics. If you are not using 'superjobs' you will see the post processing job running under this name

• ${exp}_YYYYMM_SR*
  

This is the listing of the Saveing of the Restart files, which will get saved at the end of each month.

• ${exp}_YYYYMM_SL*
  

This is the listing of the Saveing of the all Listings above.

Restart from a restart file

Retrieve previous restart file(s)

In the commands below ${CLIMAT_archdir}, ${GEM_exp} ,and ${GEM_exp_prev} need to get replaced with the values from your 'configexp.cfg'.
${GEM_exp} is the name of the month you want to rerun, ${GEM_exp_prev} is the name of the previous month.
The restart file you want to restart your simulation from needs to be in your execution directory:

 ~/MODEL_EXEC_RUN/${TRUE_HOST}

There are 3 possibilities where this restart can be:

a) Already in your execution directory. Then there is nothing else you need to do.

b) In the directory: ~/MODEL_EXEC_RUN/${TRUE_HOST}/Restarts
In that case you need to copy it back under ~/MODEL_EXEC_RUN/${TRUE_HOST}. Since there might be more than one restart file per month they are under sub-directories called "laststep_...". Copy the one you want to restart from back to your execution directory, for example with:

 rs ~/MODEL_EXEC_RUN/${TRUE_HOST}/Restarts/${GEM_exp_prev}/laststep_.../${GEM_exp_prev} ~/MODEL_EXEC_RUN/${TRUE_HOST}

Replace the 'laststep_...' by the laststep directory name.
Note: If you still have post processings running this restart file might get archived and "disappear" from this location any time!

c) Already archived
If your restart file(s) got already archived you need to gunzip and unarchive it/them back under ~/MODEL_EXEC_RUN/${TRUE_HOST}. For large domains there might be several restart files and you need to gunzip and unarchive all of them, hence the loops (they also work when there is only one file).
First gunzip the file(s), for example with:

 cd ${CLIMAT_archdir}/Restarts
 for file in ${GEM_exp_prev}_step*.ca.gz ; do gunzip -c ${file} > ~/MODEL_EXEC_RUN/${TRUE_HOST}/${file%.gz} ; done

Then you need to unarchive the file(s). To do that go into your execution directory and unarchive it/them with cmcarc, for example:

 cd ~/MODEL_EXEC_RUN/${TRUE_HOST}
 for arch in ${GEM_exp_prev}_step*.ca ; do cmcarc -x -f ${arch} ; done 

Restart simulation

To restart your simulation go into your config directory.

a) If you are running your simulation with "Um_lance"

b) If you are running your simulation with "Chunk_lance"

1) Edit the file "chunk_job.log"
Make sure ther very last line of this file contains something like:

 1 scripts ${GEM_exp}_S starting at Tue Jun 23 09:04:47 EDT 2020

The number at the beginning of the line may vary as well as the date. What is important is the part: "... scripts ${GEM_exp}_S starting ..."
You should have this line in the file already. Just remove all lines below it. Make sure this is the very last line and that there is no blank line at the end!!!

2) Make sure you still have the script you want to execute in your config directory. It got created by the simulation itself and is called:

 ${GEM_exp}_S

If you do not have this script get it back from the archive:     ${CLIMAT_archdir}/Listings/jobs_${exp}.zip
${exp} is the name of your experiment without any date at the end.

3) Now you can restart your simulation with:

 Chunk_lance