When you have a run that take longer than two hours in development queue, it can be a bit of a wait when putting it in normal queue. Luckily, ADCIRC has a checkpointing feature called hotstarting that allows you to split the queue into several runs on the development queue.
Step-by-step guide
- Turn on the NHSTAR option to 3 (fort netcdf) in the fort.15 file, and turn on your NOUT(...) options to 3. This will append the next run to the same output file.
- ./adcprep the modified fort.15 file.
- Run your simulation in dev queue. It should output a .67.nc or .68.nc file now. These are your hotstart files, of which there are two for redundancy. Pick the most recent file (linux will not update the timestamp if writing is interrupted).
- Turn on the IHOT option to 367 or 368, depending on what file was most recent
- ./adcprep the modified control file.
- Run new sim, rinse and repeat from 2.
There is also a deprecated NHSTAR = 1 option that generates binary hotstart files. This is pretty bad and slow so don't do it.
Also, don't do this for every run, else TACC might get mad.