Faster Queue Times with ADCIRC Hotstarting

When you have a run that take longer than two hours in development queue, it can be a bit of a wait when putting it in normal queue. Luckily, ADCIRC has a checkpointing feature called hotstarting that allows you to split the queue into several runs on the development queue.

Step-by-step guide

Turn on the NHSTAR option to 3 (for netcdf) in the fort.15 file, and turn on your NOUT(...) options to 3. This will append the next run to the same output file.
./adcprep the modified fort.15 file.
Run your simulation in dev queue. It should output a .67.nc or .68.nc file now. These are your hotstart files, of which there are two for redundancy. Pick the most recent file (linux will not update the timestamp if writing is interrupted).
Turn on the IHOT option to 367 or 368, depending on what file was most recent
./adcprep the modified control file.
Run new sim, rinse and repeat from 2.

There is also a deprecated NHSTAR = 1 option that generates binary hotstart files. This is pretty bad and slow so don't do it.

Also, don't do this for every run, else TACC might get mad.