\chapter{Final model setup and running the model\label{chp:run}}
This chapter describes how to run the model, assuming that it has been compiled, and 
that the data files necessary to run the model have already been generated. It
also assumes that the experiment subdirectory has been set up properly.

\textcolor{red}{NB: For a experienced user the checklist in
Section~\ref{cha:smr} may be used. It gives step-by-step instructions on how to
set up the model.}

\section{Overview of a model run}
Running a model is different from the procedure used in earlier versions of
NERSC-HYCOM, where every input file was put into the model run directory
manually. With the new procedure everything is done automatically but it relies on the region directory
hierarchy, and that all necessary input files are present. 
Everything is now happening for a given experiment in the region dir.
The experiment directory will typically look like that:
\begin{verbatim}
-rwxr-xr-x 1 fanf fanf  11876 2011-07-07 10:05 blkdat.input
drwxr-xr-x 3 fanf fanf   4096 2009-03-12 11:47 data
-rwxr-xr-x 1 fanf fanf   1216 2009-02-16 14:37 EXPT.src
-rwxr-xr-x 1 fanf fanf    667 2009-02-16 14:37 infile2.in
-rwxr-xr-x 1 fanf fanf    118 2009-02-16 14:37 infile.evp
-rwxr-xr-x 1 fanf fanf    633 2009-02-16 14:37 infile.icestate
-rwxr-xr-x 1 fanf fanf   1296 2009-02-16 14:37 infile.in
drwxr-xr-x 3 fanf fanf   4096 2009-03-12 11:47 log
-rwxr-xr-x 1 fanf fanf   1941 2009-02-16 14:37 pbsjob.sh
-rwxr-xr-x 1 fanf fanf    452 2009-02-16 14:37 ports.input
-rwxr-xr-x 1 fanf fanf   1309 2009-02-16 14:37 postprocess.sh
-rwxr-xr-x 1 fanf nersc 21164 2010-09-03 12:09 preprocess.sh§
-rwxr-xr-x 1 fanf fanf   2240 2009-02-16 14:37 README
drwxr-x--- 2 fanf nersc  4096 2011-07-29 15:15 SCRATCH
drwxr-xr-x 5 fanf fanf   4096 2009-03-12 11:47 subprogs
\end{verbatim}
The files \file{EXPT.src}, \file{blkdat.input}, \file{ports.input}, contains information about the model configurations. These have already been described set in Section~\ref{sec:setup_expt}, and only minor changes may be applied.
 The files \file{infile.in}, \file{infile.evp}, and \file{infile2.in} contains information about: when to start your model, if you are going to store nesting, use tides, dump weekly/daily average. The file \file{blkdat.input}  contains informations about the vertical structure and model parametrisation (viscosity, turbulence model, ...). There is also 4 folders,  \file{data}, \file{SCRATCH}, \file{log}, and \file{subprogs}. The latter contains useful programs in python and matlab script that are independent of the model. The data directory, contains the model outputs gathered during the runs. The log directory, contains the logfiles from the last run (possible to change the main script so that it dump a name that depends on the job number). The SCRATCH directory is a temporary directory where all necessary file are copied before the model is run. 
Finally, there are three shell scripts \file{pbsjob.sh}, \file{preprocess.sh}, and \file{postprocess.sh}. The script pbsjob.sh is the main job script that need to be submitted to the machine. It is machine dependent and should contains information about the time of the job, account number, number of cpu, memory usage, etc. You will only need to call this one as it calls the two others. \file{preprocess.sh} copies all necessary files to the SCRATCH directory and returns an error message if some files are missing. \file{postprocess.sh} is called once the job is completed, and move all model output files to the data dir. 


\subsection{blkdat.input\label{sec:blkdat}}
This is the standard configuration file used when running HYCOM, and is
identical to that of standard HYCOM.
Here we describe only the model parameter that are independent from the model vertical structure. Only the one that we have expertise in are described, otherwise you can refer to the standard HYCOM doc.
\begin{itemize}
\item iniflg and yrflag are useful when restarting model from climatology (See Section \ref{sec:model_init}). Otherwise use yrflag=3.
\item jerlv0 allows a spatially varying light penetration that depends on KPAR climatology (recommended by default; i.e.=1).
\item sshflg to allow getting the steric height contribution of the SSH in a an additional variable.
\item All dsurfq, diagfq, proffq, tilefq, meanfq, rstrfq,cplifq are not used in NERSC VERSION.
\item nestfq and bnstfq are used for inner nested model. Typical values used a NERSC is 0.25 for both.
\item baclin and batrop are the baroclinic and barotropic time step of your model, that MUST be tuned to satisfy the CFL at your model resolution.This is very important to do, because you may speed up considerably your model by tuning these parameter without affecting the accuracy of you results. The two values cannot be chosen independently and a matlab script \file{subprogs/micomstep.m}. The script uses two arguments (range of baroclinic and barotropic time steps) and returns all the possibilities. 
\item incflg incstp and incup are used for incremental update. These are use when data assimilation does not maintain dynamical balance, but introduce a time shift in the update. Not necessary when using data assimilation method used at NERSC.
\item wbaro and btrlfr are related to the time discretisation. Leapfrog is recommended, but is is unconditionally unstable and Asselin filter is used for stability. (wbaro=0.125 is recommended).
 \item hybmap represents the vertical interpolation within the mix layer. the WENO like PPM is highly recommended.
 \item hybflg and advflg decide what variable to advect, The two should be equal. If you use ice, use 0, otherwise the option 1 is better. 
\item advflg select wich advection scheme to be used. MPDATA is deprecated because it is positive definite but no monotonous (doesn't keep the initial range of the advected quantity). The FCT4 is best but often unstable, so standard is FCT2.

\item momtyp and facdf4 define which momentum advection type you are using. momtyp=4 is best, but is a little unstable in places of strong flows. If momtum=4 is chosen, facdf4 must be given a value (from 1/32 for coarse model to 1/200 to high resolution model), veldf4 and veldf2 must be 0 and visco4 must be set to for example 0.05 and visco2 to 0. If momtyp is set to 2, you first need to decide if you prefer to use laplacian viscosity or biharmonic viscosity. The former is more realistic and smoother, and the later gives usually better results. Once you are satisfy with your viscosity, you need to tune the velocity diffusion and the thickness diffusion to reduce the noise. Noise in the velocity is often appearing at the surface, while noise in the dp typically appears bellow the mix layer depths.  Here, biharmonic diffusion is recommended (i.e. thkdf4 and veldf4). By experience the velocity diffusion is not  problematic, but the thickness diffusion can be very destructive of the stratification and its usage should be limited. It is possible to set the values as negative, then HYCOM will use a spatially varying values that it will read from a file with similar name (Tools for producing these files are describe in MSCPROGS/THKDF4)

\item thkmls is used when sssflg is activated- when surface salinity is relaxed. The actual relaxation depends on the effective mixed layer depth. The thkml variable sets the reference mixed layer depth for a 30-day e-folding time. The actual e-folding time is 30*MLD/thkml (shorter means a stronger relax). If a negative value is given, HYCOM will look for a file sssrmx.[a,b] that correspond to a threshold above which the relaxation is not proceeded. The reason is that relaxation can have a detrimental impact on some regions particularly where strong fronts occur and/or they are misplaced (e.g., Gulf Stream). In such places the water mass distribution is bimodal, and the relaxation towards an
average estimate reduces the sharpness of fronts. Typical value for the threshold is 0.5 psu (Mats Bentsen).
\item mlflag defines which turbulence model you are using for the mixed layer depths. KPP is usually the most recommended as it is well tuned and tested. However it requires a lot of tuning and lot of different option ( e.g. bblkpp to include a bottom boundary layer). There is a document made by SHOM that describe KPP in detail. GISS is an improved version of Mellor-Yamadaand thus better suited for near coastal area. By experience, it is the one that provides the best result in TOPAZ, but as a different mixing is computed for each point it can be noisy. Another advantage of the scheme is that it requires very little tuning.

\item tsofrq, tofset, and sofset can be used if you observed a drift in you model in T and S. It is possible to adjust for that by adding a value to the mode (value corresponds to the drift/century).

\item lbflag indicates how you treat your open boundary condition. A value of 2 is used with nesting. The option 3 is used for flather boundary condition (not tested yet). Value 0 and 1 assume that the boundary is a solid wall. The value 1 is used when you want to add a barotropic port in your model (ports.input is necessary, see Section \ref{sec:port}). This version of HYCOM seems to have problems when it comes to total volume conservation. Therefore, even if you don't want a port, it is recommended to set lbflg=1 and set the nb of port to 0 in the ports.input, then you will benefit from the fix.

\item wndflg. Unfortunately there is still a mystery for me. We should use wndflg=3 because the atmospheric forcing are on a p-grid, but this makes more noise (checker boarding). Therefore a value of 1 seems to perform best.
\item difsmo is an engineering (efficient) fix to remove checker boarding from the model surface. The value should be equal to the number of layer you want to remove it.

\item flxflg must be set to 99 when using the NERSC version
\item relax to activate nudging from relaxation at the boundary.
\item priver and epmass. The former activate the river forcing and the latter provides an additional barotropic signal to it. 
\end{itemize}
The other parameter have not be tested and are based on the blkdat.input examples from Alan Wallcraft and Zulema Garaffo.

\subsection{EXPT.src\label{sec:run_EXPT}}
This file has also been covered in Section~\ref{sec:setup_expt}. For the purpose of running
the model, two definitions are important in this file; the definition of the
``scratch'' directory, where the model will run, and the definition of the
``data'' directory, where model results are stored. Both of these
should be set to appropriate directories by the user. The following is an
example of their definition.
\begin{verbatim}
export D=$P/data                                  # Where data ends up
export S=/work/$USER/hycom/$R/expt_$X/SCRATCH   # Scratch area 
\end{verbatim}
The directories will of course have to be accessible in order for the
preprocessing and postprocessing scripts to work. If the directories do not
exist, a new one will be created. 
\subsection{infile.in\label{sec:infile.in}}
This file is particular to NERSC-HYCOM. This file determines the start and end
times of a model integration, sets the diagnostic times and diagnostics type as well as
some flags needed to run NERSC-HYCOM. Note that this file assumes that the date are given in day in year (days since the beginning of the year). An example of the contents of this file with line
numbers added is given here(line numbers added):
\begin{verbatim}
1   3.1          # version of inputfile (limits.dat)            DO NOT CHANGE
2   IDB          # (rungen)   Version number of run 
3   1988         # Reference year for simulation  (365. for spinup)
4      5 00      # (nday1)    First day of integration NOTE FORMAT F9.2
5     90 00      # (nday2)    Last day of integration  NOTE FORMAT F9.2
6   era40 era40  # forcing option, month, ecmwf, ncepr, ecmo, ecnc
7    200.0       # temperature relaxation time scale (F in blkdat.input)
8    200.0       # salinity    relaxation time scale
9   F  2         # laverage n Accumulate monthly averages every n hours
10  F  T         # Switch on and accumulate (T) or overwrite (F) daily averages 
11  F  6         # lnesto, nestdto - saves nesting bnd cond at nestdto intervals
12  T  6         # lnesti, nestdti - read and apply nesting bnd cond at nestdto intervals
13  T FES F F    # Tides (true,  CSR/FES, apply currents)
14  F            # lgridp     Activate storage of gridpoint information
15  # Days     Assi  Diagno.  Restart            (1992 for first synoptic experiment)
16       5.00  F     T        T
17      60.00  F     T        T
18      90.00  F     T        T
\end{verbatim}
The above will set the model up to start at day 5 in 1988 and stop at day 90 in
1988. Restart and diagnostic files will also be created for day 60. In you want to extend your run beyond year 1988, just  set nday2 to a value larger than 365. The Final restart file will be dump with the correct year (here 1989) with the corresponding year. 
In HYCOM the start and end times of an integration are also set in a file called
``limits''. This file is used in NERSC-HYCOM v2.2 as well, but it is calculated
from the file \file{infile.in} by the preprocessing script \file{preprocess.sh}.

The version of the input file is given in the first line, NERSC-HYCOM will check
that this is correct before model integration starts. The second line is a
three-letter ID commonly referred to as ``rungen''. When the model creates
diagnostic and restart files, the rungen will be placed at the start of the file
name. 

The third, fourth and fifth line is used to set integration ranges. A reference
year is given, as well as start and end times of the model integration in days
and hours. Both the start and end days are relative to the first day of the
reference year.

The sixth line is used to choose the forcing data set to use. There are two
flags, the first is the synoptic forcing data set to use, the second is the
climatology data set to use. If the model is run with climatological forcing,
the synoptic forcing flag should be set to ``month''. If the forcing is
pre-generated, the synoptic forcing flag should be set to ``prep''. Forcing can
be pre-generated by the script \file{nersc\_synotpic.sh} in the
\file{force/bin/} directory.  Table~\ref{tab:forcing}
shows the possible options for the synoptic and climate forcing flags.
\begin{center}
\begin{table}[tb]
\begin{tabular}{| m{.15\textwidth} | p{0.6\textwidth} |}
\hline
{\bf Synoptic flag} & {\bf Description} \\
\hline
month & Used when running NERSC-HYCOM with climatology option \\
\hline
prep  & Used when running NERSC-HYCOM with a pre-generated data set (set up with script
        \file{nersc\_synoptic.sh}). \\
\hline
era40 & Use the ERA40 data set \\
\hline
era-i & Use the ERA interim data set \\
\hline
ncepr & Use the NCEP data set \\
\hline
ecmwf & Use the ECMWF T159 operational data set \\
\hline
metno & Use the ECMWF T399 operational data set \\
\hline
ecnc  & Use the ECMWF T799 operational data set \\
\hline
\end{tabular}
\\[1em]
\begin{tabular}{| m{.15\textwidth} | p{0.6\textwidth} |}
\hline
{\bf Climatology flag} & {\bf Description} \\
\hline
era40 & Use ERA40-derived climatology \\
\hline
ncepr & Use NCEP-derived climatology \\
\hline
old   & Use ``old'' climatology (deprecated)\\
\hline
\end{tabular}
\caption{\label{tab:forcing}Synoptic and climatology forcing flags used with
NERSC-HYCOM.}
\end{table}
\end{center}
The generation of forcing is described in more detail in Section~\ref{sec:setup_force}.
The 7$^{th}$ and 8$^{th}$ line of \file{infile.in} are no longer used as the relaxation time scale is set by 1/30 days by default in HYCOM (TODO remove these lines).
The 9th line of \file{infile.in} switches on (T) or off (F) the weekly average
option. The second item on line 9 denotes how often the (in hours) the model
should be sampled to produce the weekly average.
%
Remember that to use weekly averages they must be 
set up at compilation by setting the CPP flag ``WEEKLY\_AVERAGE'', as described
in Section~\ref{sec:code_cpp}.
%L
Weekly averages will be dumped to files with names of the type
\file{XXXAVE\_YYYY\_MM\_WW.[ab]}. The file name contains year, month and week
of the start of the model run. Not that ``month'' and ``week'' is not real, they should
be considered as a way of splitting the year into 48 segments.It implies that in some months some weeks will contains 8 days and some other 7.


Line 10$^{th}$ of \file{infile.in} is used to activate daily averages. The first value
switches them on and off, while the second line will switch on
accumulation of daily averages. This ``accumulation flag'' is mainly used for
running ensembles - if it is switched on, the daily averages of the different
ensemble members will be accumulated into one daily-ensemble average. For normal
single-member runs, the accumulation flag should be set to false. The daily
averages will be dumped to files with names of the type
\file{XXXDAILY\_YYYY\_DDD\_YYYY\_DDD.[ab]}. The file name contains year and
ordinal day of the start of the model run, as well as the year and ordinal day
of the daily average.
To use daily averages they must be set up at compilation by setting the CPP flag
``DAILY\_AVERAGE'', as described in Section~\ref{sec:code_cpp}.

Line 11 and 12$^{th}$ of \file{infile.in} us ised to activate ``outer'' and ``inner''
nesting. The logical value (T or  F) is used to activate/deactivate the nesting,
while the second value, an integer, denotes how often to save nesting (for the
outer model) or how often to read nesting conditions (for the inner model). This
nesting interval value is given in hours, and the model which saves nesting conditions must have
a nesting interval which matches the model which actually receives them. To use 
inner and/or outer nesting requires preprocessing, described in
Section~\ref{sec:setup_nest}.

The 13$^{th}$ line is used to switch on tidal forcing with the NERSC-HYCOM
approach that is based on tidal atlases. There are four values on this line - the first
logical value is to switch on\/off tidal forcing. The second value denotes the tidal
atlas to use (FES\/CSR). The third logical value denotes if tidal atlas currents should
be used (or if just tidal elevations are used). The last value is a logical
which indicates ``slowstart'', where the tidal forcing will be slowly ramped up
from zero to the full tidal elevation values. This is usually switched off, but
can be used when tidal forcing is switched on for the first time during a model
integration. Normally it should be set to false.
%
To use tidal forcing requires preprocessing, described in
Section~\ref{sec:setup_tide}.

The 14$^{th}$ line is used to diagnose grid point time series sampled every hour. To do
this you need to set the flag to T. You will also need to specify the grid points to process using the
file \file{infile\_gp.in}, described in Section~\ref{sec:infile_gridp}. 

From line 16-17$^{th}$ you need to specify the date to which there will be input/output of restart or archive files (weekly or daily are automatically dumped). Currently the archive file dump has been switched off because they are not used in NERSC. The first column corresponds to the day in year (i.e starting from 000 with no upper limit); the second and the last should be F, T, T. There are some rules to follow when making the list of dump. There must be a diagnostic
time equal to the first and last integration times specified on lines 4 and 5. The list of diagnostic time must be strictly increasing.

\subsection{infile2.in\label{sec:infile2.in}}
This file is used to set up the random forcing used when running a model
ensemble. It is typically used for assimilation purposes. Its contents are
\begin{verbatim}
 1.2 'fversn' = version of inputfile (infile2.in)
  0   'randf ' = 1=random forcing, 0= no random forcing
 11   'seed  ' = Random forcing seed
 10.0    'vslp  ' = Variance in slp
  9.e-4  'vtaux ' = Variance in tau_x
  9.e-4  'vtauy ' = Variance in tau_y
  2.5    'vwspd ' = Maximum of the perturbed windspeed.
  4.e-2  'vcloud' = Variance in clouds
  9.0    'vtair ' = Variance in air temperature
  2.e-17 'vprcp ' = Variance in precipitation
  0.0    'vrlhum' = Variance in relative humidity
  25.e4  'scorr ' = Horizontal radius of correlation (meters)
  2.0    'tcorr ' = Temporal radius of correlation (days)
  2      'prsflg' = 0 uncorr wind/slp, 1= wind from slp, 2=wind from slp limited by wndspd
\end{verbatim}
In this file ``randf'' is used to switch on and off random forcing. Set it to 0
to switch off random forcing. The variance values denote the variance of
different random forcing fields: Sea level pressure, air-drag coefficient in x, air-drag coefficient in y, maximum wind speed allowed of the perturbation, clouds, temperature of the air, precipitation, relative humidity. The correlation value ``scorr'' given in meters
is used to set up the decorrelation length scale of the red noise. The decorrelation time scale 
of the red noise is set by the variable ``tcorr'' and is given in days.
The final value indicates how to calculate wind and stress random forcing
values. If ``prsflg'' is 0 the wind vectors are uncorrelated, while they are
correlated using a geostrophic assumption when ``prsflg'' is set to 2.
``prsflg'' should not be set to 1, as it leads to too high wind/stress forcing.

\subsection{infile.evp\label{sec:infile.evp}}
This file is used to set up the EVP sea-ice dynamics model. It contains settings
for EVP time step, and some parameters used to configure the sea-ice model, such
as the ice strength and the ice concentration factor used in the original Hibler
ice strength formulation. These parameters are also known as $C^\star$ and
$P^\star$. The file looks like this:
\begin{verbatim}
1.1      # File version
 7200.   # EVP time step
27500.   # EVP ice strength
 20.     # EVP ice concentration factor
\end{verbatim}
To use the EVP model, it must be enabled at compile-time with the EVP flag, see
Section~\ref{sec:code_cpp}.


\subsection{infile\_gp.in\label{sec:infile_gridp}}
This is the configuration file for the generation of gridpoint data from HYCOM,
which must be switched on in the file \file{infile.in}, see
Section~\ref{sec:infile.in}.  This file makes it possible to select grid points
based on longitude, latitude and depth ranges. It is also possible to subsample
the grid, selecting only every $n$-th grid point along the grid dimensions, It
is also possible to directly specify grid positions. An example of
\file{infile\_gp.in} is given here:
\begin{verbatim}
 1)     0.0          # minimum depth
 2)  9000.0          # maximum depth
 3)    50.0          # minimum lat
 4)    80.0          # maximum lat
 5)   -40.0          # minimum lon
 6)    40.0          # maximum lon
 7)       4          # skip in i index
 8)       4          # skip in j index
 9)  105   200       # Direct point specification
10)  200   200       # Direct point specification
...
\end{verbatim}
In the example above, only points with depths between 0-9000\,m, between
$50^\circ$\,N  and $80^\circ$\,N latitude, between
$40^\circ$\,W  and $40^\circ$\,E longitude will be included in the grid points
processed. In addition, only every fourth grid point (in each direction) will be
processed. Finally, it is possible to specify an indefinite number of grid points 
directly by giving the $i$ and $j$ indexes (line 9 and 10 above). The grid
point files are dumped in files with the position encoded into the file names.
To avoid clutter, the files are placed in directory, one directory for a given month and
year. The names of these directories are of the type \file{RRRGP\_YYYY\_MM/}
where RRR is the rungen ID of the model, YYYY is the year and MM is the month.
TODO: Netcdf output and processing.



\subsection{Pre- and post-processing routines and job scripts\label{sec:jobscripts}}
The pre and postprocessing scripts are named \file{preprocessing.sh} and
\file{postprocessing.sh} - as mentioned before they are responsible for setting up
model runs in the ``scratch'' directories and retrieving data from that
directory and placing them in the ``data'' directory, see
Section~\ref{sec:run_EXPT}. 

A sample job script \file{pbsjob.sh}, needed to run the model in a queue system, is given. This job script should work for a Cray XT4 system, using the
PBS queue system. For other queue systems new job scripts need to be created,
but the main steps for running the model are given by the following script
segment(line numbers added):
\begin{verbatim}
 1 #Fetch OpenMP and MPI "sizes"
 2 NMPI=`qstat -f $PBS_JOBID | awk '/mppwidth/ {print $3}'`
 3 NOMP=`qstat -f $PBS_JOBID | awk '/mppdepth/ {print $3}'`
 4 [ -z "$NOMP" ] && NOMP=0
 5 export NOMP NMPI
 6 
 7 # Enter directory from where the job was submitted - should be the experiment dir
 8 cd $PBS_O_WORKDIR       ||  { echo "Could not go to dir $PBS_O_WORKDIR  "; exit 1; }
 9 
10 # Initialize environment (sets Scratch dir ($S), Data dir $D, experiment dir $P++ )
11 source ../REGION.src  || { echo "Could not source ../REGION.src "; exit 1; }
12 source ./EXPT.src  || { echo "Could not source EXPT.src"; exit 1; }
13 
14 # Transfer data files to scratch - must be in "expt_XXX" dir for this script
15 ./preprocess.sh         ||  { echo "Preprocess had fatal errors "; exit 1; }
16 
17 # Enter Scratch dir and  Run model
18 cd $S     ||  { echo "Could not go to dir $S "; exit 1; }
19 aprun -n $NMPI -m 1000M ./hycom  # Run hycom
20 
21 # Cleanup and move data files to data directory - must be in "expt_XXX" dir for this script
22 cd $P     ||  { echo "Could not go to dir $P  "; exit 1; }
23 ./postprocess.sh 
24 exit $?
\end{verbatim}

The job must be submitted from the experiment directory in the above script.
A description of the run process, in chronological order,  is as follows:
\begin{enumerate}
\item In the segment above, lines 2-5 fetch the number of MPI threads to use, using
information in the PBS queueing system on Cray XT4. The most important part is
the retrieval of the number of MPI tasks. On another queue system this will have
to be done differently. It is also possible to specify these numbers manually in
the script, by substituting lines 2-3 with ``NMPI=10'' and ``NOMP=1'' (for
instance).
\item Line 8 makes sure that we change directory to the experiment directory before
proceeding. If these files can not be found an error is returned and the job
script will exit\footnote{Note that the commands after the $||$ sign are done if the command before the it
fails.}.
\item Lines 11-12 source the region and experiment source files, in order to set up
the environment for this experiment\footnotemark[\value{footnote}].
\item The preprocessing is done on line 15, which moves all the data needed to run the
model into the ``scratch'' directory. If this fails the job script exits.
\item Lines 18-19 change directory to the scratch directory, and runs the model.
Note that the ``aprun'' command is particular to the Cray XT4 system and will
have to be changed for other systems (mpiexec and poe, for instance)\footnotemark[\value{footnote}].
\item Lines 18-19 change directory to the scratch directory, and runs the model.
\item When the model finishes we go back into the experiment directory and run
the postprocessing script, on line 22 and 23\footnotemark[\value{footnote}].
\item On line 24 the script exits with the exit status of the last command
(postprocess.sh).
\end{enumerate}

\section{Tidbits}
This section contains some additional information for running the model

\subsection{Activating outer nesting\label{sec:run_nestouter}}
\subsection{Activating inner nesting\label{sec:run_nestinner}}

\subsection{Initializing the model from climatology\label{sec:model_init}}
The procedure for initializing a model from climatology is now more in line with
standard HYCOM. In standard HYCOM the file \file{limits} is used to indicate
integration limits, and if a model should be initialized from climatology. This 
file is not specified directly in NERSC-HYCOM, instead it is pre-processed
using the contents of the file \file{infile.in}.

In order to run the model with climatology in NERSC-HYCOM, you should create an
empty file called ``INITIALIZE'' in the experiment directory, which can be done
like this\\[.7em]
\indent\indent touch INITIALIZE\\[.7em]
This file will be read by the preprocessing script, more specifically the script
\file{subprogs/setlimits.py}. What it does is create the file \file{limits} in
the scratch directory, with a negative start time, which indicates to HYCOM that
the model should be initialized from climatology. The file ``INITIALIZE'' will
be deleted after preprocessing, to avoid re-initializing the model at a later
stage.

Note that in order to initialize the model, you should use climatological
forcing in \file{infile.in} (set line 6 for example as: month era40  ), and you should set yearflag to 0 or 1 in
\file{blkdat.input}.

\textcolor{red}{Finally, you should not use a reference year of 0  in \file{infile.in} when
initializing the model. I recommend using a reference year of 2 (line 3 is: 0002). This may seem
strange but it is due to particularities of the reference dates used in standard
HYCOM.}


\subsection{Starting the model from a restart file with ``wrong date''}
Sometimes it is necessary to initialize a model run from a restart file which
has a different date. This is usually done by copying the restart file from, for
instance, year 2002 to 1958 by modifying its filename. In this case the date
information inside the restart file will not match the file name. This
basically causes the model stop and complain that the date of the restart file
is wrong.

In order to correct this, HYCOM needs to be told that it shouldnt consider the
time inside the HYCOM restart, but that it should use the date specified in the
file \file{infile.in}(and put into the \file{limits} file in the scratch
directory) as the correct date. This is handled in a similar way as for the
model initialization and therefore is only possible if yrflag=3 (using realistic synoptic forcing).  By creating an empty file called ``INITIALIZE'' in the
experiment directory, which can be done like this\\[.7em]
\indent\indent touch INITIALIZE\\[.7em]
HYCOM will ignore any date inconcistencies. \textcolor{red}{Remember that this
is done only when yrflg=3 in \file{blkdat.input}.}

\subsection{Starting the model from a curviint restart file}
The curviint routine will interpolate fields from another model onto the current
experiment model, as described in Section~\ref{sec:setup_div}. If you want to start
from a curviint-file created by the routines in Section~\ref{sec:setup_div}, you can
create an empty file called CURVIINT in the experiment directory, like
this\\[.7em]
\indent\indent touch CURVIINT\\[.7em]
This will make the preprocessing script look for a curviint file in the region\_basedir/curviint/\${expt\_number}/ in the experiment.

\subsection{Quick access to data and scratch directories}
The experiment environment can be sourced to easily move to the scratch and data
directories. By typing these lines (when standing in the experiment directory)
\begin{verbatim}
source ../REGION.src  || { echo "Could not source ../REGION.src "; exit 1; }
source ./EXPT.src  || { echo "Could not source EXPT.src"; exit 1; }
\end{verbatim}
the scratch and data directories are placed in environment variables. To go to
the scratch directory, type\\[.7em]
\indent\indent cd \$S\\[.7em]
and to go to the data directory, type\\[.7em]
\indent\indent cd \$D\\[.7em]
A similar approach is also used in the job scripts to set up the directories
properly, see Section~\ref{sec:jobscripts}.

\section{Running the model}
Assuming all data files are set up properly, Section~\ref{cha:setup}, 
The code is set up properly, Section~\ref{cha:code}, and the model run 
configuration files are set up properly, ~\ref{chp:run}, the model should be
ready for integration.

At this point you can check if all the files needed are present by running the
preprocessing script \file{preprocess.sh}. This script can be outside of the
queue system.  If this script fails it means
something is missing, and you need to modify your setup. The script should give
you some indication on what went wrong, and some additional info is present in
the \file{log/} directory.

When you are content with the setup, the model is ready to run. It can be
submitted using the job script. On hexagon, this will run the model:\\[.7em]
\indent\indent qsub pbsjob.sh \\[.7em]

