\chapter{Model Setup\label{cha:setup}}
This chapter deals with the setup of the HYCOM model, using the shell scripts
and FORTRAN programs available at NERSC. The chapter only deals with setup of 
data/forcing files, for compilation of the model code see Chapter~\ref{sec:modelcode}.
%

In the new setup the NERSC version of
HYCOM has an approach which is closer to the standard setup of
HYCOM\citep{walxx}. The
benefit of this is that it should be easier to use the tools present in the
standard HYCOM distribution, and to exchange our tools with other groups. 
%
The new setup collects all the different routines and data used to drive the
model in one central directory for each model region. Within this directory
there are separate subdirectories for the model topography, atmospheric forcing
data, relaxation data and so on. Within these subdirectories are also shell scripts
used to set up the data, using both tools (usually FORTRAN programs) from the standard 
HYCOM distribution, as well as tools developed at NERSC. Finally, there are
also subdirectories for different experiments - which will contain the model
output from different experiments.

This approach is different from earlier versions, where all the forcing and
relaxation data was dumped into the directory where the model was run. The old approach 
could lead to confusion, since there was a potential for mixing
data files between different model setups. 
%
Especially for model bathymetry the potential for mix-up was great. In the
new approach this should be less of a problem if the general
guidelines in this document are followed.

To describe the new setup, the different sub-directories will be described in
this chapter. First of all, a description of the top-level region directory
will be given.

\section{The top-level region directory\label{sec:region}}
A top-level region directory is tied to the geographical location of the model
grid. When the placement of a model grid is determined (typically using the
conformal mapping routines), a region directory  for this grid should
be created. 

The name of the region directory should be something descriptive of
the region - as an example, take the TOPAZ3 grid, a suggested name of the
top-level directory in this case is TP3a0.12 - where TP3 is a three-letter ID of
the grid(TOPAZ3). The letter a is a version number reflecting {\it minor} 
changes to the grid (a large change may warrant a new ID), and 0.12 is
descriptive of the resolution, here roughly 0.12 degrees. This naming follows
the naming scheme from standard HYCOM. 

The top-level directory will contain subdirectories for all the forcing needed
to run the model, as well as directories for the different experiments which
have been set up.  The directory will also contain a configuration file called
\file{REGION.src}.  The contents of the top-level directory will typically look 
something like this:
\begin{verbatim}
   TODO: FIXME FIXME
   drwxr-s---  4 knutali mohnsv     4096 2008-09-11 20:27 bin
   drwxr-s---  4 knutali mohnsv     4096 2008-09-11 20:27 force
   drwxr-s---  4 knutali mohnsv     4096 2008-09-11 20:27 relax
   drwxr-s---  4 knutali mohnsv     4096 2008-09-11 20:27 topo
   drwxr-s---  4 knutali mohnsv     4096 2008-09-11 20:27 expt_01.0
   drwxr-s---  4 knutali mohnsv     4096 2008-09-11 20:27 expt_01.1
   -rw-r-----  1 knutali mohnsv  5632008 2008-09-11 19:17 REGION.src
   ...
\end{verbatim}

In this listing, there are subdirectories for the data needed to run the model
(here topo, force and relax), as well as experiment directories (here the
directories \file{exp\_01.0} and \file{exp\_01.1}). The file \file{REGION.src} is a
text file containing some definitions for the setup of the hycom model.  First of all, a
variable "R" is defined in this file, which should be set to the same name as the
top-level directory, for the TOPAZ3 model, R should be set to the previously suggested 
name, ``TP3a0.12''. 

Secondly, many path variable definitions are set up in the file \file{REGION.src}. These 
paths will be used by the various utility programs used to set up topography, forcing, 
relaxation files etc. If these paths are not set, many programs will stop and ask you to 
set these paths to correct values. If this happens, the file \file{REGION.src}
is the place to correct these paths - since it is used by most setup scripts
under the top-level region directory.

Setting all these paths might seem awkward, but it makes it relatively easy to
move the setup from machine to machine, where the placement of the data files
may be different.



\subsection{Setting up a new region\label{sec:newregion}}
When a new region needs to be set up, the first thing to do is to decide on a
name, as described in Section~\ref{sec:region}. When a suitable name is decided
upon, a new region directory should be created. The easiest way of doing this is
to start with an already existing region directory. It is recommended to keep
one ``template'' region on which to base other regions on. This way,
updates/modifications to existing setup scripts can be done by just modifying
the main ``template'' directory.

On hexagon, such a ``template'' resides under the directory\\[1em]
\indent\indent\indent\file{\~{ }/knutali/Progs/HYCOM2.2/hycom/NATa1.00/}.\\[1em]  
One can either copy this
directory recursively to the new top-level region directory, and then modify the
configuration file \file{REGION.src}, where the variable R is set to the name of
the top-level region directory, using the naming scheme described in
Section~\ref{sec:region}.

Another, and probably a easier approach is to run the script\\[1em] 
\indent\indent\indent\file{\~{}/knutali/Progs/HYCOM2.2/hycom/NATa1.00/bin/newregion.sh}\\[1em] The script will
copy all files to the new top-level region directory, except the data files
which are not valid for the new region - these must be set up by you. In
addition the setup file \file{REGION.src} will be modified by changing the
variable ``R''.  Any 
experiment directories under the template directory will also be copied, but
these will have to be modified by hand, as described in
Section~\ref{sec:exptdir}.

Finally, you should copy the grid file \file{regional.grid.[ab]} into the
\file{topo/} subdirectory. This is the only file which will be common to all
experiments of this region. You should also copy the files \file{grid.info}
into the same directory.
%
The files \file{newpos.uf} and \file{latlon.dat}, which contains similar info as
\file{regional.grid.[ab]}, should not be needed anymore. If you have some old
software still uses these files, however, you can put them in the \file{topo/}
subdirectory for safekeeping. Finally, some of the scripts pertaining to
topography setup in Section~\ref{sec:topo} can also be used to import both grid
and topography files.

\section{The experiment directories\label{sec:exptdir}}
The experiment directories contain configuration files for the different
experiments, such as \file{blkdat.input}, \file{infile.in}, \file{infile.evp},
and so on. No data files will be contained in these directories, they reside in
the force, topo, relax and other data directories which will be described later
on. 

The naming of the experiment
subdirectories is based on experiment number \envvar{X} which is a
floating-point number of the form
AA.B, the full name of the experiment directories is \file{expt\_\$X}
\footnote{When referring to the value of a variable X, it will be given as \$X,
this is also how they are referenced in bash scripts.}.
Examples of experiment directories in the directory listing above are 
\file{expt\_01.0} and \file{expt\_01.1}, with experiment numbers 01.0 and 01.1
respectivelty.

All aspects of the model can be changed from experiment to
experiment, except the placement of the model grid. This means for instance 
that experiments can have a different number of vertical layers, different
parameterizations, different code versions, and even different topographies.
Only the grid placement (given through files \file{regional.grid.[ab]}) has to
stay the same across experiments.

\subsection{Setting up experiments}
Before starting to create the data files needed to run HYCOM, the files in the
experiment directories need to be properly set up. First of all there is the
file \file{EXPT.src}, where some key properties of the model are set. The
variables are listed in Table~\ref{tab:envvar}.
\begin{table}[htb]
\begin{center}
\begin{tabular}{|c|c|}
\hline
Variable & Description \\
\hline
R$^*$ & A descriptive name of the model region \\
X & Experiment number - a number of the type AA.B, ex 01.0 \\
E & Based on \$X - a number of the type AAB, ex 010 \\
T & Topography version - can be tied to an experiment. \\
\hline
\end{tabular}
\caption{\label{tab:envvar} Environment variables set by the file
\file{EXPT.src}. These are used by most scripts which set up HYCOM. *) Variable
R is set in file \file{REGION.src}.}
\end{center}
\end{table}
Of the variables in the table, the variable ``X'' should be set to the
experiment number, matching the directory name. For example, if the experiment directory is
named \file{expt\_01.0}, the variable ``X'' should be set to ``01.0''. The
variable ``E'' is derived from ``X'' by removing the period. One can loosely say that $E=X \times 10$.

In the file \file{EXPT.src} you must also set the the topography version to use.
This concept is new, and allows a less error-prone way of setting up nesting,
for instance. The topography files will be stored in the \file{topo/}
subdirectory, and there can be several topography files/versions in that
directory. By setting the topography version in the file \file{EXPT.src}, you
basically link this experiment to a topography version, which will be used by
all setup scripts, thereby avoiding inconcistencies. 

Note that if you decide to switch the topography after some model runs for a
given experiment, you should not do it by modifying
the file \file{EXPT.src} in the current experiment directory - in stead you
should set up a new experiment directory. The basic rule is that once you decide
on a topography version for an experiment, you should not change it for that experiment.


In addition to changing the file \file{EXPT.src}, you should also change any
hycom setup files, such as \file{blkdat.input}, \file{infile.in}, and
\file{infile.evp}. This will not be covered fully here. Some variables
in \file{blkdat.input} must be correctly set up for your region, however, how to
set these is given in Table~\ref{tab:blkdat_basic}.
\begin{table}[htb]
\begin{center}
\begin{tabular}{|c|c|}
\hline
Variable & Description \\
\hline
idm & 1st grid dimension, must match idm in \file{regional.grid.b} \\
jdm & 2nd grid dimension, must match jdm in \file{regional.grid.b} \\
iexpt & Must match experiment variable ``E'' in \file{EXPT.src} \\
iversn & Must match hycom version (e.g 2.2.12 means iversn = 22) \\
\hline
\end{tabular}
\caption{\label{tab:blkdat_basic} Environment variables set by the file
\file{EXPT.src}. These are used by most scripts which set up HYCOM. *) Variable
R is set in file \file{REGION.src}.}
\end{center}
\end{table}
NB: The shell script \file{bin/newExpt.sh} under the top-level directory
will assist in setting up a {\it new} experiment. It should set the bare minimum of 
variables in \file{blkdat.input} and \file{EXPT.src}. Any experiment-specific tuning (e.g changing
vertical levels, viscosities etc.) most be done by the user. An example of this
routine can be found here on the machine hexagon:\\[1em]
\indent\indent\indent\file{\~{ }/knutali/Progs/HYCOM2.2/hycom/NATa1.00/bin/newExpt.sh}.\\[1em]  
Note that this script only works for a properly set up region (through the file
\file{REGION.src}).


\subsection{When to set up new experiments}
This is entirely up to you, if you decide its smart to set up a new experiments
when you change the viscosity of the model - then do it. There are a few
situations where you always should set up a new experiment, however:
\begin{itemize}
\item Changing topography version. If you dont change experiment for this there
will be a inconsistency between model depth in restart files (pbot) and
topography files. There will also be inconsistencies between relaxation fields
and topography files.
\item Changing the vertical grid (target densities, number of layers, number of
sigma levels, etc). If you dont set up new experiment for this, there will be a
inconsistency between the relaxation fields and the model fields.
\item Changing hycom versions. If the HYCOM version is updated, there are
usually changes to parameters in \file{blkdat.input}. Creating a new experiment
directory you can make the necessary changes to \file{blkdat.input}, wwithout
rendering the old \file{blkdat.input} useless for running the old model...
\end{itemize}

In addition to setting up up new experiments, it is also possible to seperate
model runs performed  in the same experiment directory. This is done by modifying the
``rungen'' variable in \file{infile.in}. For minor changes to parameters this
can be a better approach than setting up a completely new experiment directory.

%

\section{Topography\label{sec:topo}}
The topography files are located in the directory \file{topo/} under the region
directory.
Unlike the location of the model grid, there can be several versions of the
model topography for a region. For instance, there can be a topography version
for an un-nested model, a topography for nesting the model into an outer model "A",
and a topography for nesting the model into an outer model "B".
Another example is if you want to modify an existing topography to improve model
performance, such as making sure the bathymetry of the Denmark Strait is
sufficiently deep.

Standard HYCOM has something
called a ``topography version'' (this is set for the different experiments
through the environment variable T), which is used to
keep track of the different topographies you create for a model grid. Whenever 
you modify a topography, you should make sure you dont overwrite an existing
topography, but instead create a new topography with a new topography version
number. An example of a "versioned" topography file can be found in the
following example directory listing of {\it topo/}
\begin{verbatim}
   drwxr-s---  4 knutali mohnsv     4096 2008-09-11 20:27 bin
   -rw-r-----  1 knutali mohnsv  5632008 2008-09-11 19:17 depths800x880.uf
   -rw-r-----  1 knutali mohnsv  2818048 2008-09-11 20:22 depth_TP3a0.12_01.a
   -rw-r-----  1 knutali mohnsv       85 2008-09-11 20:22 depth_TP3a0.12_01.b
   -rw-r-----  1 knutali mohnsv      569 2008-09-11 12:57 grid.info
   drwxr-s---  2 knutali mohnsv     4096 2008-09-11 20:24 partit
   -rw-r-----  1 knutali mohnsv 53542912 2008-09-11 12:57 regional.grid.a
   -rw-r-----  1 knutali mohnsv     1174 2008-09-11 12:57 regional.grid.b
\end{verbatim}
This directory shows a single pair of topography files as
depth\_TP3a0.12\_01.[ab]. The naming convention of topography files is 
depth\_\$R\_\$T.[ab], where \$R is the name of the region (example TP3a0.12) and
\$T is the topography version (\$T).

To use the topography in a given experiment, you need to set it up in the main
experiment file ``EXPT.src'' located in the experiment subdirectories.
An example of this is found in the following example from a EXPT.src file:
\begin{verbatim}
    X="01.0"                               # X based on dir name (expt_$X)
    E="010"                                # E is X without "."
    T="01"                                 # Topography version

\end{verbatim}
this basically states that in experiment 01.0, the topography file with version
number "01" will be used.
\subsection{Help routines for setting up topo-files\label{sec:toporoutines}}
In the directory {\it topo/bin/} are some helpful routines for importing
topography files from older versions of NERSC HYCOM. These routines are
\begin{description}
\item[toponame.sh] Short script for getting the topography name. Input is
topography version. Simply a tool to remember filenames...
\item[regioncopy.sh] Use this script to import existing topography files, and
also newly created topography files to the topo dir. This script will
copy regional.depth.* files depths*.uf files and ndepths*.uf files to a temporary
directory, set up new topography file with proper format and correct file name, 
and them copy them to the topo directory.  Input is the topography version and
the location of the topo files to import. {\bf NB: The script will not copy the
new files to the topo dir if there are files with the same topography version
present}. 
\item[nestbat.sh] Use this script to set up the bathymetry of a nested model
domain. Input is the topography version used as a starting point, the topography
version of the final modified bathymetry, and also the location and experiment
name of the outer model where nesting will be recceived from.
\item[addhuge.py] The older versions of NERSC HYCOM used a zero value to denote
land. Newer versions will use the value $2.0^{99}$ to denote land (this follows
standard HYCOM behaviour).  This script will replace zero values with
$2.0^{99}$ in the standard HYCOM topography files (files of the
regional.depth.[ab]-"type"). 
The script is run by the regioncopy.sh script and is rarely needed
to be run by hand. The python path may have to be changed as the file is
migrated between different machines.
\item[depthtoregional.py] This routine will convert a old depths*.uf (or
ndepths*.uf) file to a topography file. The script is run by the regioncopy.sh 
script and is rarely needed to be run by hand. The python path may have to be
changed as the file is migrated between different machines.
\end{description}

Note that the regioncopy.sh script is the one you will usually use to set up the
bathymetry file if you have an already existing setup from an older model run.
Make sure that you set use a new topo version when running it.
if you specify an already existing topography the routine will stop with an
error message and not copy anything into the {\it topo/} directory. This is a
safety precaution to avoid any unfortunate incidents...

The script will convert from old topography files located in the input
directory you give to regioncopy.sh. There can be
several topography files in the original location. The script will choose one
file and use this priority when converting from the old topography files:
\begin{enumerate}
\item regional.depth.[ab]
\item ndepths*.uf
\item depths*.uf
\end{enumerate}
This means you should be careful when running the script. If you have a
regional.depth.[ab]-file this is usually the safest to use and is first on the
priority list. If, however, you have a non-nested model but still have a ndepths
file in the original directory you may get a problem. To be safe, move the
original files you {\it know} are wrong to a different location than the one you
supply to regioncopy.sh.

The nestbat routine will use smooth an existing topography towards the
topography from a outer model, which will provide nesting conditions. This means
that to use nestbat.sh, you must already have set up a model topography for this
region. The final product of the nestbat utine will be given a new topography
version.

The python scripts (.py suffix) depend on the location of the locally installed
python. If the script stops with strange error messages, you should check that 
the first line in the python scripts point to an actually existing python
installation.

\subsection{Generating MPI partitions for a topography}
In order to run a model using the Message Passing Interface(MPI), it is necessary
to specify how the model domain will be split into subdomains. A separate
routine from the standard HYCOM distribution is designed to do this, and it is
possible to call it through the shell script \file{topo/bin/tile\_grid.sh}. This
routine will create a text file describing the partition and place it inside the
\file{topo/partit/} subdirectory.

The \file{tile\_grid.sh} routine takes as input the number of subdomains along
the two dimensions of the model grid, along with the topography version it will
do the partitioning on. This means that you need to create MPI tiles for all the
topography versions you use.

The  routine \file{tile\_grid.sh} can be called in two modes, with positive or
negative arguments. For example:
\begin{itemize}
\item tile\_grid.sh  8  7 01
\item tile\_grid.sh -8 -7 01
\end{itemize}
%
The result of the two different partitioning methods is illustrated in
Figure~\ref{fig:partit}.
\begin{figure}[bt]
  \subfloat[Uniform partitioning of domain\label{fig:partita}]
     {\includegraphics[width=.45\textwidth]{Pics/depth_0051}}
  \subfloat[Non-uniform partitioning of domain\label{fig:partitb}]
     {\includegraphics[width=.45\textwidth]{Pics/depth_0056}}
  \caption{Examples of two different MPI partitions of a model domain}
  \label{fig:partit}
\end{figure}


With negative arguments, figure~\ref{fig:partita}, the
partioning will be uniform - the size of each tile will be (approximately) the
same. In effect, this gives the lowest memory footprint when running the model.
On the other hand the work is not evenly balanced, since some tiles will have
more sea points than others.
%
With positive arguments, figure~\ref{fig:partitb}, the partitioning is
non-uniform, but  the number of sea points on each
tile is approximately the same. The memory footprint of the model will be
higher, however, since it depends on the {\it maximum} size of all the tiles. It
is, load balancing, which can be benefitial.

Which partitioning to choose depends on the model, but here are some general
guidelines
\begin{enumerate}
\item choose a uniform partitioning when memory is an issue
\item choose a non-uniform partitioning when the total number of MPI tasks is
low
\item choose a uniform partitioning when the total number of MPI tasks is high,
as the communication is more efficient in this case
\end{enumerate}
Remember that these are guidelines, and will probably depend on the model grid.
Some degree of trial and error is always needed. Also note that 
memory often becomes an issue when the number of MPI tasks is low, giving some
inconsistency between point 1 and 2 above.

\textcolor{red}{NB: Note that the choice of domain partitioning is not strictly
necessary before you start compiling the model, see
Section~\ref{sec:compilesetup}.}

\subsection{Procedure for setting up a new model topography}
To create/import topographies this is the basic procedure to follow
\begin{itemize}
\item Decide on a topography version number T.
\item Locate the directory containing the old topography files. Make sure they
are consistent with the regional.grid.[ab] files for your region. 
\item Run the regioncopy.sh script to import the topography to the new setup.
Run it with topography version and the directory containing the old topography
files as input. Also, be aware of the possible pitfalls of this routine, see
\autoref{sec:toporoutines} in case of trouble or disturbing warning messages.
\item Set up the experiment file "EXPT.src" to use the new topography.
end up with a mess of inconcistencies.
\end{itemize}
To set up a nested model topography, follow this recipee:
\begin{itemize}
\item Decide on a new topography version number T, and the topography version to
use as basis for the new topography.
\item Locate the directory containing the outer model region. 
\item Run the nestbat.sh script to run the routines which smooth the model
topography towards that of the outer model.
\item Set up the experiment file "EXPT.src" to use the new topography.
\end{itemize}
Whenever you create a new
topography it is safest to set up a new experiment. Dont replace the topography
version in a experiment you have already set up and run - you will probably just
end up with a mess of inconcistencies.
\section{Relaxation\label{sec:relax}}
Relaxation fields can be used for sidewall nudging, and for initializing the
model. Currently the climatologies can be based on Levitus or PHC climatologies.
A two-step procedure is needed to set up the relaxation fields. First, a
horizotal interpolation on the z-levels of the climatology, a step which doesnt
depend on the model setup, apart from the grid location.  The final step is a
vertical interpolation from z-levels to the isopycnal levels, which depends on
the setup of hybrid and isopycnal layers of the model.

The setup of relaxation fields is usually done after setting up 
\file{EXPT.src} and \file{blkdat.input} in the experiment subdirectories. 
The topography files to be used in the experiment will also have to be set up.
%
The relaxation fields are set up in the subdirectory \file{relax/}. A typical
listing of the contents of that directory may look like this
\begin{verbatim}
   drwxr-s---  3 knutali mohnsv 4096 2008-11-24 20:15 010
   drwxr-s---  3 knutali mohnsv 4096 2008-11-24 20:19 bin
   drwx--S---  4 knutali mohnsv 4096 2008-11-24 19:48 levitus
   drwx--S---  3 knutali mohnsv 4096 2008-11-24 19:36 phc
   -rw-r-----  1 knutali mohnsv 1808 2008-11-24 20:09 README.KAL
   -rw-------  1 knutali mohnsv 3580 2008-09-11 10:27 README.relax
\end{verbatim}
The most important subdirectories contain the following
\begin{description}
\item[levitus] This directory will hold Levitus climatologies at predefined
isopotential(z) surfaces, created during the climatology generation process. The
original data files for the Levitus climatology can be acquired from the HYCOM
supplemental data files.
\item[phc] Same as for the Levitus subdirectory, but contains PHC climatologies.
\item[bin] This directory contains routines for creating the climatologies.
\item[010] This is an example of a directory containing experiment-related
data, produced by scripts in the \file{relax/bin/} directory. This is the
location of the final processed hycom input data, used to initialize and force
the hycom model.
\end{description}
In addition there are directories for visualizing the climatologies using
standard HYCOM routines, but they require some hand-tuning to work. Some info is also 
available in README files.

\subsection{Routines for generating relaxation fields}
The routines in the {\it relax/bin/} directory can be used to set up the
relaxation fields. The routines are
\begin{description}
\item[z\_levitus.sh] Use this routine to create horizontally interpolated
fields on the original Levitus z-levels. Input is "ksigma" which is the same as
the thref flag in blkdat.input. The fields created by this routine are dumped in
the {\it levitus/} subdirectory. The Levitus fields used in this procedure can be
retrieved from the HYCOM ftp site.
\item[z\_phc.sh] Same as for z\_levitus.sh, but uses the PHC climatology The fields 
created by this routine are dumped in the {\it phc/} subdirectory. The PHC fields 
used in this procedure can be retrieved from the HYCOM ftp site.
\item[relax.sh] Creates a climatology interpolated to density surfaces.
The input for this routine is the experiment number and the climatology to use.
The actual density surfaces for this experiment will be retrieved from the 
blkdat.input file for this experiment This routine requires
\file{z\_levitus.sh} or \file{z\_phc.sh} to be run first. The climatology is dumped to 
subdirectory corresponding to the variable ``E'' in the experiment file
\file{EXPT.src} (example -  010).
\item[relax\_rmu.sh] Creates a relaxation mask for the climatology - requires
some user input.
\item[old\_lev\_nersc.sh] Creates some data files needed when running hycom with
the NERSC "old" climatology option - which is generally deprecated..
\end{description}

\subsection{Prerequisites for generating relaxation fields}
To generate the climatologies data files for PHC or Levitus are needed, which
can be acquired from the HYCOM ftp site. Also, the topography files, 
and the file \file{blkdat.input} must have been set up for the experiment which
relaxation fileds will be created for.

In addition the environment variables HYCOM\_ALL, LEVITUS\_PATH 
and PHC\_PATH needs to be set up, this is udually done in the file
\file{REGION.src}.
The variable HYCOM\_ALL points to the path of the hycom {\it ALL} direcory,
while LEVITUS\_PATH and PHC\_PATH point to the location of the Levitus and
PHC climatologies, respectively.

In addition, to use the "old" climatology option, the path to the Mohn-Sverdrup
programs should beset in the variable MSCPROGS. Thisis done in the file \file{REGION.src}.

\subsection{Procedure for generating relaxation fields}
To create climatologies this is the basic procedure to follow
\begin{enumerate}
\item Create the horizontally interpolated climatologies using the routines
\file{bin/z\_levitus.sh} or \file{bin/z\_phc.sh}. This step is experiment-independent just
needs to be run once. the PHC climatology is generally recommended, especially
if the Arctic is in the model domain.
\item Create the relaxation fields with the script \file{bin/relax.sh}
\item Create the relaxation masks with the script \file{bin/relax\_rmu.sh}
\item To create files for the NERSC "old" climatology, run the script
\file{bin/old\_lev\_nersc.sh}
\end{enumerate}

\section{Forcing}
The forcing directory contains various input fields which are used to run the
model. In this directory it is possible to create atmospheric forcing fields
(winds, air temperature, radiative fluxes etc), river forcing fields, and so on.
The contents of this directory will typically look something like this
\begin{verbatim}
   drwxr-s---  4 knutali mohnsv 4096 Jan  3 19:45 bin
   drwxr-s---  4 knutali mohnsv 4096 Jan  3 20:12 nersc_era40
   drwxr-s---  2 knutali mohnsv 4096 Jan  3 20:15 other
   drwxr-s---  4 knutali mohnsv 4096 Jan  3 20:14 rivers
   drwxr-s---  3 knutali mohnsv 4096 Jan  3 20:18 seawifs
\end{verbatim}

Inside the \file{force/bin} directory there are several scripts which can be
used to set up the forcing, all relying on Fortran programs supplied with
standard HYCOM or routines developed at NERSC. The routines in the
\file{force/bin} directory are
\begin{description}
\item[\file{nersc\_rivers.sh}] Use this routine to set up river forcing using point
data supplied in a \file{rivers.dat} file. This file must be present in the
experiment subdirectory.
\item[\file{nersc\_clim.sh}] This routine can be used to set up atmospheric climatology
forcing fields. Available climatologies are ``era40'', ``ncepr'', and ``old''.
The ``old'' option is deprecated.
\item[\file{nersc\_synoptic.sh}] This routine can be used to set up synoptic
atmospheric forcing. It uses input from the file \file{infile.in} in the
experiment subdirectory to determine the synoptic and climatology data sets to
use, as well as the time period to create the forcing for.
\item[\file{seawifs\_mon\_kpar.sh}] This routine can be used to set up the water type 
(Jerlov) based on observations from the SEAWIFS satellite. The use of this data
set in hycom runs depends on the setting of the parameter ``jerlv0'' in
\file{blkdat.input}.
\end{description}


\subsection{Prerequisites for generating forcing fields}
To generate the forcing files several envronment variables have to be set in the
file \file{REGION.src}. First of all, the variables HYCOM\_ALL and MSCPROGS need
to be set. They point to the location of the standard HYCOM utility routines, as
ell as utility routines developed at NERSC.
%
Additionally, several environment variables pointing to the location of data
sets are needed, they are:
\begin{table}[htb]
\begin{center}
\begin{tabular}{|l|l|}
\hline
Variable & Description \\
\hline
SEAWIFS & Location of SEAWIFS data set, supplied by standard HYCOM \\
NCEP\_CLIM\_PATH & Location of NCEP climatology, produced at NERSC (NetCDF) \\
ERA40\_CLIM\_PATH & Location of ERA40 climatology, produced at NERSC (NetCDF) \\
ERA40\_PATH & Location of ERA40 data set  (NetCDF) \\
WOA2005\_PATH & Location of World Ocean Atlas (2005) data set (NetCDF) \\
..TODO.. & \\
\hline
\end{tabular}
\caption{\label{tab:forcing_envvar} Some of the environment variables set by the file
\file{REGION.src}. These are used by most scripts which set up HYCOM.  }
\end{center}
\end{table}
The variables in Table~\ref{tab:forcing_envvar} need to be changed whenever the
location of the data changes or when the system is ported to a new machine.

\section{Nesting setup (subdirectory nest\_nersc)}
The \file{nest\_nersc/bin/} subdirectory contains two routines for setting up data files for 
model nesting. Note that the nestbat routine(\file{nestbat.sh}) is not included in this
directory, it is in stead placed in the \file{topo/bin}. The routines in 
\file{nest\_nersc/bin/} contains setup routines for the two situations where the
model provides nesting files to a model,\file{nest\_outer.sh}, and when the model
receives nesting conditions from a model, \file{nest\_inner.sh}. Both of these
must be run before starting the model. The need to set up a model before doing
outer nesting is new in this version. 

The routines are
\begin{description}
\item[\file{nest\_outer.sh}]Run this script before running a model which {\it
creates} nesting conditions for another model.  When producing nesting conditions for another
model, the model providing these files must know about the grid of the ``inner
model''. To set up this information is the purpose of this script. The grid
files of the inner model will be copied to a data directory under the
\file{nest\_nersc/} directory. In addition, a file describing the mapping from
the outer model onto the inner model points is created.
%
\item[\file{nest\_inner.sh}]Run this script before running a model which {\it
receives} nesting conditions. It will set up masks for where the model
will relax the baroclinic fields towards those specified by  the outer model,
It will also produce a \file{ports.input} file which is needed to set up the
location of the barotropic nesting points.
\end{description}

The nesting in version 2.2 follows closely the approach of standard HYCOM, which
has led to some changes. It should now be possible to use the nesting approach
of standard HYCOM, which uses snapshot files to generate the nesting conditions.
However, there  is also an option to generate nesting conditions similar to
previous versions of the NERSC version.
%
Compared to earlier NERSC HYCOM versions, version 2.2 has new nesting files, and
easier routines for the creation of these. To run the ``outer'' model, it is no
longer needed to start the inner model first to create data files. Simply run
the script \file{nest\_outer.sh} first, and the setup should be ready. 


\textcolor{red}{NB: As stated earlier the generation of topography which is
smoothed towards the outer model must be done during the generation of model
topography, Section~\ref{sec:topo}.}

\section{Tide setup (subdirectory tides\_nersc)}
In version 2.2 there are several ways of enabling tides in the model. In
standard HYCOM tides can be enabled as a body force (mainly relevant for global
models), and as a forcing input on the model boundaries. The inclusion of tides
is new in standard HYCOM.
%
In addition the tidal forcing used in earlier versions of NERSC HYCOM can be
used. This approach uses tidal Atlases, which is not used by standard HYCOM. 

The routines used to set up the NERSC tidal forcing are given in the directory
\file{tides\_nersc/bin/}. The routines inside that directory can be used to set
up tidal forcing from two different tidal databases. 
\begin{description}
\item[\file{tides\_fes.sh}]Sets up tidal data along model boundary using the
FES2004 database.
\item[\file{tides\_csr.sh}]Sets up tidal data along model boundary using the
CSR database.
\end{description}
Both routines also create diagnostics of phase and amplitude data from he
database, interpolated onto model grid points.


\section{Various other setup routines (subdirectory div)}
