\chapter{Model Setup\label{cha:setup}}
This chapter deals with the setup of the HYCOM model, using shell scripts
and FORTRAN programs developed at NERSC. This chapter only deals with setup of 
data/forcing files, for compilation of the model code see Chapter~\ref{sec:setup_code}.
%

In this new setup, the NERSC version of HYCOM use a closer framework and code to the standard setup of
HYCOM\citep{wal07}, than in previous version. The benefit of this is that it should be easier to use the tools present in the
standard HYCOM distribution, and to exchange our tools with other groups. The new framework is also safer, as it avoid mixing up topography, parameter etc ...
 %
The new setup collects all the different routines and data used to drive the model in one central directory for each model region. Within this directory there are separate subdirectories for the model topography, atmospheric forcing
data, relaxation data as commonly used in standard HYCOM. Within these subdirectories, shell scripts are used to set up the data calling programs from the standard  and NERSC HYCOM distribution. Finally, there are also subdirectories for handling the different experiments.

%This approach is different from earlier versions, where all the forcing and
%relaxation data was dumped into the directory where the model was run. The old approach 
%could lead to confusion, since there was a potential for mixing
%data files between different model setups. Especially for model bathymetry the potential for mix-up was great. In the new approach this should be less of a problem if the general guidelines in this document are followed.

The general guidelines of setting up a model and a region are given in this document.
\textcolor{red}{NB: experienced user can directly jump on the checklist in
Section~\ref{cha:smr}. It gives step-by-step instructions on how to
set up the model.}

\section{The top-level region
directory\label{sec:setup_region}}%\label{sec:setup_newregion}}
The top-level region directory is the directory where all the input data needed
to run HYCOM is kept and created.
%
A top-level region directory is tied to the geographical location of the model
grid and its horizontal resolution. When the placement of a model grid is determined (typically using the
conformal mapping routines), a region directory for the grid is created at a given horizontal resolution. 
%
The contents of the top-level directory will typically look 
something like this:
\label{verb:exdirlist}
\begin{verbatim}
   drwxr-s---   4 fanf mohnsv     4096 2011-07-12 15:34 bin
   drwxr-s---   4 fanf mohnsv     4096 2011-07-12 15:34 force
   drwxr-s---   4 fanf mohnsv     4096 2011-07-12 15:34 relax
   drwxr-s---   4 fanf mohnsv     4096 2011-07-12 15:34 topo
   drwxr-x---   3 fanf mohnsv     4096 2011-07-12 15:34 nest_nersc
   drwxr-x---   3 fanf mohnsv     4096 2011-07-12 15:34 other_nersc 
   drwxr-x---   3 fanf mohnsv     4096 2011-07-12 15:34 tides_nersc     
   drwxr-s---   4 fanf mohnsv     4096 2011-07-12 15:34 expt_01.0
   drwxr-s---   4 fanf mohnsv     4096 2011-07-12 15:34 expt_01.1
   -rw-r-----   1 fanf mohnsv     5632008 2008-09-11 19:17 REGION.src
   lrwxrwxrwx   1 fanf mohnsv     52 2011-07-12 15:34 config 
   lrwxrwxrwx   1 fanf mohnsv     56 2011-07-12 15:34 src_2.2.12 
 \end{verbatim}
In this listing, there are subdirectories for the data needed to run the model as in the standard HYCOM
(here \file{topo/}, \file{force/}, and \file{relax/}), specific for the NERSC implementation (here \file{nest\_nersc/}, \file{tides\_nersc/}, and \file{other\_nersc/}) as well as experiment directories (here the directories \file{exp\_01.0} and \file{exp\_01.1}). The file \file{REGION.src} is a text file containing some definitions for the setup of the hycom model, and it will be used by many of the scripts for setting up the forcing.


\subsection{Setting up a new region\label{sec:setup_newregion}}
When a new region needs to be set up, the first thing to do is to decide on a
suitable name. This name will be used both to name some of the data files used
by HYCOM, as well as naming the region directory itself.
%
The name of the region directory should be something descriptive of
the region - as an example, take the TOPAZ3 grid, a suggested name of the
top-level directory in this case is ``TP3a0.12'' - where TP3 is a three-letter ID of
the grid (here based on the name ``TOPAZ3''). The letter a is a version number reflecting {\it minor} 
changes to the grid (a large change may warrant a new ID), and 0.12 is
descriptive of the resolution, here roughly 0.12 degrees. The name will be referred to 
as the region name in the following.


When a suitable name is decided upon, a new region directory should be created. The easiest way of doing this is to start with an already existing region directory. It is recommended to keep one ``template'' region on which to base other regions on. This way, updates/modifications to existing setup scripts can be done by just modifying the main ``template'' directory.
%
On the svn, a ``template'' of a North Atlantic configuration with 1$^\circ$ resolution is given:\\[1em]
\file{./hycom/HYCOM\_2.2.12/NATa1.00}.\\[1em]  
A shell script exists for setting up the architecture of your new region:\\[1em] 
\file{./hycom/HYCOM\_2.2.12/NATa1.00/bin/newRegion.sh}\\[1em] 
 Two arguments are required. The first is the region name, the second is the path where you want to place the region on the file system. As for example, the
following command\\[1em]
\file{./hycom/HYCOM\_2.2.12/NATa1.00/bin/newregion.sh TP3a0.12 /work/fanf/}\\[1em] 
will place the new region directory \file{TP3a0.12/} in the work directory
\file{/work/fanf/}.
The script will copy all files to the new top-level region directory, except the data files
that are not valid for the new region - these must be set up by you and this is described later in this chapter.
%
In addition this script will attempt to modify the setup file \file{REGION.src} for the new region.
At first you must replace the path of MSCPROGS with your own compiled as explained in Chapter \ref{sec:MSCPROGS}. (by default it assumes that the path is $\sim$/hycom/MSCPROGS)
You must also verify that:
\begin{itemize}
\item The path given in \file{REGION.src} exists. Whenever the model is moved to another machine, these paths must be
replaced to location of the corresponding datasets.
\item The variable ``R'' should be set to the region name described earlier. (This should be done automatically by the script \file{newRegion.sh})
\item The environment variables HYCOM\_ALL  exists and is properly set. It corresponds to configuration script from the standard HYCOM. This need to be compiled only once by the first user on the machine, so the path may not be on your account. In hexagon, this code can be found in \file{/home/nersc/fanf/ALL}
\end{itemize}

The following step for setting up a new region is to copy the various grid-related files and the topography files into the topo directory. This is explained in the next Section.

\textcolor{red}{NB: Now that the region directory has been set up, it will also contain all the
routines which are used in the rest of this chapter. When a script is mentioned in the
setup procedures later on, you should be able to find it somewhere inside the region
directory.}


% 
%\file{regional.grid.[ab]}, \file{grid.info}, \file{newpos.uf}, and \file{latlon.dat} to the 
%\file{topo/} subdirectory. 
%%
%These files will be common to all experiments for this region.
%%
%%
%The files \file{newpos.uf} and \file{latlon.dat}, which contains similar info as
%\file{regional.grid.[ab]}, are usually not be needed anymore. They are still kept in the
%\file{topo/} directory though, in case you have some old
%software still using these files.

\section{Topography\label{sec:setup_topo}}
All file related to the topography will be placed/produced in the directory \file{topo/} under the region
directory. Unlike the model grid location (i.e. the regional.grid.[a,b]), there can be several versions of the
model topography for a region. For instance, there can be a topography version
for an un-nested model, a topography for nesting the model from an outer model
``A'', and a topography for nesting the model from an outer model ``B''.
Another example is if you want to modify an existing topography to improve model
performance, such as making sure the bathymetry of the Denmark Strait is
sufficiently deep.

Standard HYCOM has something called a ``topography version'', which is used to
keep track of the different topographies you create for a model grid. Whenever 
you modify a topography it should be saved as a different topography version. You should make sure you don't overwrite an existing topography. Here is an example of a "versioned" topography folder of {\it topo/}
\begin{verbatim}
   drwxr-s---  4 knutali mohnsv     4096 2011-07-12 15:34 bin
   -rw-r-----  1 knutali mohnsv  5632008 2008-09-11 19:17 depths800x880.uf
   -rw-r-----  1 knutali mohnsv  2818048 2008-09-11 20:22 depth_TP3a0.12_00.a
   -rw-r-----  1 knutali mohnsv       85 2008-09-11 20:22 depth_TP3a0.12_00.b
   -rw-r-----  1 knutali mohnsv  2818048 2008-09-11 20:22 depth_TP3a0.12_01.a
   -rw-r-----  1 knutali mohnsv       85 2008-09-11 20:22 depth_TP3a0.12_01.b
   -rw-r-----  1 knutali mohnsv      569 2008-09-11 12:57 grid.info
   drwxr-s---  2 knutali mohnsv     4096 2008-09-11 20:24 partit
   -rw-r-----  1 knutali mohnsv 53542912 2008-09-11 12:57 regional.grid.a
   -rw-r-----  1 knutali mohnsv     1174 2008-09-11 12:57 regional.grid.b
\end{verbatim}
This directory shows two topography ``ab'' file pairs:\\
\file{depth\_TP3a0.12\_00.[ab]} and \file{depth\_TP3a0.12\_01.[ab]}. \\
The naming convention of topography files is 
\file{depth\_\$R\_\$T.[ab]}, where \$R is the name of the region (example TP3a0.12) and
\$T is the topography version (\$T). When HYCOM will be run with a given topography version, the corresponding depth file will be copied to regional.depth.[a,b] in the running directory.

\subsection{Importing a model topography}
A script located in the topo/bin directory allows to import topographies created by tools described in Section~\ref{sec:setup_topo} into the region structure.
\begin{itemize}
\item Decide on a topography version number T.
The following convention is recommended for \$T:
\begin{itemize}
 \item The topo number for files directly output from ETOPO, GEBCO is 00
 \item The topo number for files directly output from the grid program after topofix is 01
 \item The topo number for topography files modified for nesting purpose is $>=$ 2. The motivation is that if you want to nest your model into another outer model you want to start from topography 01 to avoid multiple interpolation.
\end{itemize}

\item Locate the directory containing the topography files that you want to import in the region/topo directory.  
\item Run the \file{regioncopy.sh} script, as follows:
\indent\file{region/topo/bin/regioncopy.sh 01 /work/knutali/topofiles}\\
where 01 is the chosen topography number, and \file{/work/knutali/topofiles} is the place where the new topography files  that you want to import are located.
Beware of the possible pitfalls of this routine, see Section~\ref{sec:toporoutines} in case of trouble or disturbing warning messages.
This script copy the grid files (if they are not present):
\begin{enumerate}
\item regional.grid.[ab]
\item latlon.dat
\item newpos.uf
\end{enumerate}
\end{itemize}
The files \file{newpos.uf} and \file{latlon.dat} are deprecated in the newer version as they contain recurrent info than the regional.grid.[a,b]. However they are still copied as some ''old-routine" may still need them.

Depth files  are also copied and it is ensured that they are compatible with the format of HYCOM 2.2 (see addhuge.py bellow). The script choose one depth file and use this priority order when converting from the old topography files:
\begin{enumerate}
\item regional.depth.[ab]
\item ndepths*.uf
\item depths*.uf
\end{enumerate}
Similarly \file{depths???x???.uf} and \file{ndepths???x???.uf} correponds to the previous version of HYCOM (version 2.1). This script can support them but regional.depth.[ab]-file is safest and prioritised. You should be careful if you consider using them. Usually, the ndepths corresponds to the depths that have been changed by nestbat, meaning that the topography has been interpolated to match a coarser model. It is not recommended to use bathymetry that have been interpolated several times. Ideally, We recommend to restart the grid creation is you have any doubt. 

\subsection{Create a nested model topography \label{sec:topo_create}}
To set up a nested model topography, follow this recipe:
\begin{itemize}
\item Decide on a new topography version number T, and the topography version to
use as basis for the new topography.
\item Locate the directory containing the outer model region. 
\item Run the \file{nestbat.sh} script to run the routines which smooth the inner model
topography towards that of the outer model.
\end{itemize}
When nesting a model into coarser one, try to follow this basic recommendations:
\begin{itemize}
\item Put the boundary far enough of the circulation feature you are interested in (approximately 30 grid cells).
\item Nesting need to applied  on a buffer zone to handle topography difference, viscosity difference etc . (approximately grid cell). Tools used in nestbat will interpolate the inner model topography in a buffer zone. The first point the buffer zone is equal to the outer model and the last one is equal to the bathymetry of the inner. 
\item Ratio between the outer and the inner should be around 1:3 - 1:5. Larger is not recommended and may lead to model instabilities. However, stability can be reached by extending the relaxation buffer zone, and forcing artificially the relaxation strength.
\item In order to solve the hyperbolic equation, the 3 first grid cells are used. If the coastline is not uniform (e.g. an island in the second grid cell) within them, this may caused some instabilities and slow down considerably the model running speed.
\end{itemize}




\subsection{Help routines for setting up topo-files\label{sec:toporoutines}}
Following is a list of script located in topo/bin that are not necessary for setting up a model, but that may be useful for advanced users. Most of them are already called in regioncopy.sh and nestbat.sh
\begin{description}
\item[toponame.sh] Short script for getting the topography name for a given topography version. 
\item[addhuge.py] The older versions of NERSC HYCOM used a zero value to denote
land. Newer versions will use the value $2.0^{99}$ to denote land (following
standard HYCOM).  This script replaces zero values with
$2.0^{99}$ in the standard HYCOM topography files.
The script is run by the regioncopy.sh script and is rarely needed
to be run by hand. The python path may have to be changed as the file is
migrated between different machines.
\item[depthtoregional.py] This routine will convert a old depths*.uf (or
ndepths*.uf) file to a topography file. The script is run by the regioncopy.sh 
script and is rarely needed to be run by hand. The python path may have to be
changed as the file is migrated between different machines.
\end{description}
The python scripts (.py suffix) depend on the location of the locally installed
python. If the script stops with strange error messages, you should check that 
the first line in the python scripts point to an actually existing python
installation.



\section{The experiment directories\label{sec:setup_expt}}
Each experiment directory contains a configuration files related to the experiment \file{EXPT.src}, model parametrisation setting files  \file{blkdat.input}, \file{infile.in}, \file{infile.evp}, \file{infile.icestate} and,
\file{ports.input}, and job script for launching job on supercomputers  \file{preprocess.sh}, \file{postprocess.sh}, \file{pbsjob.sh}. The forcing, the topography and the relaxation files are located at the root of the region directory
the \file{force/}, \file{topo/}, \file{relax/}. The useful files are copied before a job is submitted depending on the setting of \file{EXPT.src}.

All aspects of the model can be changed from experiment to experiment, except the placement of the model grid and the horizontal resolution (given through files \file{topo/regional.grid.[ab]}). This means for instance that experiments can have a different number of vertical layers, different parametrizations, different code versions, and even different topographies.

The naming of the experiment subdirectories is based on experiment number \envvar{X} which is a
floating-point number of the form AA.B, for instance $01.0$, $01.1$, $01.2$ etc. The full name of the experiment directories is \file{expt\_X/} where X would be the experiment  number. Again, examples of experiment directories are
\file{expt\_01.0/}  corresponding to experiment $01.0$, \file{expt\_01.1/} corresponding to experiment $01.1$ and so on. 




\subsection{Setting up experiments\label{sec:setup_exp}}
When you have run the script regioncopy.sh in previous Section, a directory \file{expt\_01.0/} has been created. It contains all necessary files, but some files in the experiment directory need to be properly set up before creating the relaxation, the climatology and the forcing fields some . It consists of the model bathymetry and the vertical density structure (number of layer and target densities). Such info is defined in \file{EXPT.src} and \file{blkdat.input}. Following is a brief description of this two files and advices on how to configure them depending of your need.

\file{EXPT.src} is the main experiment setting file. It set the model topography version used together with some other environment variables (see Table~\ref{tab:envvar}). The first variable  represents the experiment number that matches the directory name. If the experiment directory is named \file{expt\_01.0/}, the variable \envvar{X} should be set to ``01.0''. The variable \envvar{E} is derived from \envvar{X} by removing the period. Roughly speaking, $E=X \times 10$. The variable \envvar{T} corresponds to the topography number as explained in Section \ref{sec:topo_create}.

Following is an example of these variables in EXPT.src file:
\begin{verbatim}
    X="01.0"                               # X based on dir name (expt_$X)
    E="010"                                # E is X without "."
    T="01"                                 # Topography version
\end{verbatim}
this basically states that we are in expt\_01.0 and that the topography version
is number "01". Note that by convention it is recommended that the two first digits of \envvar{X} is matching the topography number \envvar{T}.

\begin{table}[tb]
\begin{center}
\begin{tabular}{|c|c|}
\hline
Variable & Description \\
\hline
X & Experiment number - a number of the type AA.B, an example is 01.0 \\
E & Based on \$X - a number of the type AAB, an example is 010 \\
T & Topography version - can be tied to an experiment. \\
\hline
\end{tabular}
\caption{\label{tab:envvar} Environment variables set by the file
\file{EXPT.src}. These are used by most scripts which set up HYCOM. }
\end{center}
\end{table}
Finally, there is a variable \envvar{S} which corresponds to the location of the scratch directory where files will be moved to before running HYCOM. This place is just temporary and can be deleted if wanted once the hycom run is finished.
In the supercomputer hexagon, it should look like that:\\
\indent\file{export S=/work/\$USER/hycom/\$R/expt\_\$X/SCRATCH}


The other file that need to be edited is \file{blkdat.input}. It contains the parametrization of the model such as number of layer, target densities, viscosity, diffusion ...
 Therefore, this file is quite complex and contains many variable. A description of \file{blkdat.input} is
provided in the HYCOM Users Manual \citep{xxx} and some additional advice are provided in section \ref{sec:blkdat}. 
At this stage, only variables in Table~\ref{tab:blkdat_basic}
must be set properly before running any of the setup scripts later described in this sections.
Once you have specify the idm, jdm, iexpt and iversn, you need to specify the vertical structure of your HYCOM run.\\

First, you need to decide on a number of vertical layer, \envvar{kdm}. Due to its hybrid coordinate properties, HYCOM allows to have less vertical layer than common models (in z-layer or sigma). HYCOM will have high vertical resolution in the mixed layer as the layer will be consider as z-coordinate; and in the interior, the model will have a higher discretisation in place of large density gradient thanks to the isopycnal properties. Thus, it was long thought that just 22 layers would be sufficient, but recent tests show that large benefit are obtained by increasing this number. By default, we recommend 28 layers.\\

Second, you need chose how many layers will be hybrid (can transit from isopycnal to z or $\sigma$-layers). If the number of hybrid is 0 then the model will be fully isopycnal (as in MICOM). Using hybrid coordinate is very beneficial for realistic simulation as you will get a better representation of the mix layer depths. Using only isopycnal layers can be useful in idealised framework, as they allow conservation of T,S and potential vorticity anomaly and allows to study area of development/intensification of instabilities. This option is also chosen in climate simulation where conservation of T and S is mandatory. You can also chose how many layer you allow to transit to $\sigma$-layers in shallow water. This is beneficial if you are interested in coastal area. Remember that $\sigma$-layers are considered as hybrid so the number of \envvar{nsigma} $<=$\envvar{nhybrd}+  .Finally there is an option to have all layer as z. Set nsigma=0.

Third, you need to chose to define of a reference pressure flag. Sigma-0 means that you put the reference at the surface, and Sigma-2 at 2000\,m. Problem is that these two set up will become inaccurate the further away they are from reference pressure. Sigma-0 will provide better result near the surface and Sigma-2 in the deeper part of the model. Beware that none of these configurations can handle thermobaricity, which is useful to resolve some processes such as the intrusion of Antarctic Bottom Water in the North Atlantic, or the bottom water flowing out of the Mediterranean Sea. A solution is proposed in the standard HYCOM referred as Sigma-2* (set thflag=2 and kapref accordingly). This implementation is not yet tested in the NERSC-framework, as you need reference state that vary locally depending on the area you are in (Mediterranean, NA, ...). 

Fourth, you need to set the reference target densities (from 1--kdm) and thbase. One needs to add 1000 to obtain the volumic mass in kg.$m^{-3}$. These values should be set depending  on the water masses present in the area. You must cover the full range, and if possible target known water masses density. Depending on how far away you are from the reference density, the model will decide in which vertical coordinate you are (isopycnal or z-layer). Note that in TOPAZ we use unrealistically low reference density near the surface, to ensure that the top layers remain in z--coordinate. This was motivated to make sure that some z-layer are present for ecosystem model. 

Finally, you need to set the stratification within the mixed layer depth and in the bottom layer (if sigma layers are used). The depth of the mix layer is computed from mixing scheme, e.g. GISS, MY, KPP . Given a certain number of HYCOM  z-layer  available, HYCOM will organise them depending on dp00, dp00x, dp00f (resp. ds00, ds00x, ds00f for sigma layer) . The first layer will have the minimum thickness (dp00), and their size will increase (by dp00x) until it reaches the maximum thickness allowed (dp00f). If these parameters are badly set (for example dp00f too small), the last available layer may be artificially very large causing artificial spurious diapycnal mixing. This was observed in the Supbolar Gyre, where the mix layer depth reaches 3000 m deep, and dp00f was set too small. the last layer of the mix layer depth was $>$ 1500 m. 

\begin{table}[tb]
\begin{center}
\begin{tabular}{|c|c|}
\hline
Variable & Description \\
\hline
idm & 1st grid dimension, must match idm in \file{regional.grid.b} \\
jdm & 2nd grid dimension, must match jdm in \file{regional.grid.b} \\
iexpt & Must match experiment variable ``E'' in \file{EXPT.src} \\
iversn & Must match hycom version (e.g 2.2.12 means iversn = 22) \\
kdm   &  Number of layers in the vertical \\
nhybrd & Number of hybrid levels (0=all isopycnal) \\
nsigma & number of sigma  levels (nhybrd-nsigma z-levels)\\
dp00     &  z-level spacing minimum thickness (m) \\
dp00x    & z-level spacing maximum thickness (m) \\
dp00f     & z-level spacing stretching factor (1.0=const.space) \\
ds00      & shallow z-level spacing minimum thickness (m) \\
ds00x    & shallow z-level spacing maximum thickness (m) \\
ds00f     & shallow z-level spacing stretching factor (1.0=const.space) \\
locsig & locally-referenced pot. density for stability (0=F,1=T)\\
kapref & thermobaric ref. state (-1=input,0=none,1,2,3=constant)\\
thflag & reference pressure flag (0=Sigma-0, 2=Sigma-2)\\
thbase & reference density (sigma units)\\
vsigma & spacially varying isopycnal target densities (0)\\
sigma  layer  1 & target reference density of the first layer (sigma units)\\
sigma  layer  2 & target reference density of the first layer (sigma units)\\
\hline
\end{tabular}
\caption{\label{tab:blkdat_basic} The bare minimum of variables to set in the
file at this stage \file{blkdat.input}}
\end{center}
\end{table}


NB: The shell script \file{bin/newExpt.sh} under the region directory
will assist in setting up a {\it new} experiment. It should set the bare minimum of 
variables in \file{blkdat.input} and \file{EXPT.src}. 

Note that if you decide to change the topography or the vertical stratification you will need to reprocess all the forcings, clim and relaxation files. Therefore, it is safer to just create a new experiment as explained in the next section.

\subsection{When to set up new experiments}
This is entirely up to you when to set up a new experiment. Some may consider that a changing viscosity is sufficient although it is not strictly necessary. 
There are a few situations when you MUST set-up a new experiment:
\begin{itemize}
\item Changing topography version. If you don't change experiment for this there
will be a inconsistency between model depth in restart files (pbot) and
topography files. There will also be inconsistencies between relaxation fields
and topography files.
\item Changing the vertical grid , i.e. variable in Table \ref{tab:blkdat_basic}. If you don't set up new experiment for this, there will be a inconsistency between the relaxation fields or the climatology with model fields.
\item Changing hycom versions. If the HYCOM version is updated, there are
usually changes to parameters in \file{blkdat.input}. Creating a new experiment
directory you can make the necessary changes to \file{blkdat.input}, without
rendering the old \file{blkdat.input} useless for running the old model...
\end{itemize}

\subsection{How to create/copy and backup experiment}
After creating a new region, a first template of experiment  (named expt\_01.0 ) is placed the region directory.
A whole bunch of scripts are available to handle experiment easily in region/bin/.
\begin{description}
\item[newExp.sh] copy the basic structure and necessary files for the experiment from an existing one and changes the EXPT.src. You will still need to re-create the relaxation, climatology and forcing files. this script is used as follow:\\
  \file{newExp.sh} old\_experiment\_number new\_experiment\_number
 \item[cleanEXP.sh] allows you to delete all the file related to an experiment: model output, relaxation, climatology and forcing files. this script is used as follow:\\
 \file{bin/cleanEXP.sh}  experiment\_number
 \item[backup\_exp.sh] allows you to backup all useful files related to an experiment: model output, relaxation, climatology and forcing files, nesting files and the topo files. Temporary files are not saved (i.e. the SCRATCH directory). The script create a file name called  backup\_expt\envvar{X}.tar.gz where X is the experiment number and save it in the location \envvar{BACKUP\_PATH/REGION\_NAME} where BACKUP\_PATH and REGION\_NAME (\$R) are set in the REGION.src. The script uses as follows:\\
\file{backup\_exp.sh} \envvar{X} \\
To recover an experiment you then just need to copy the backup file to the root of your region and then gunzip and detar it.
\end{description}
%
\subsection{Generating MPI partitions \label{sec:setup_partit}}
In order to run a model using the Message Passing Interface (MPI), it is necessary
to specify how the model domain will be split into subdomains. A separate
routine from the standard HYCOM distribution is designed to do this, and it is
possible to call it through the shell script \file{topo/bin/tile\_grid.sh}. This
routine will create a text file describing the partition (example: depth\_TP31.2\_01.0016 for 16 tiles) and a picture of the partitioning (example: depth\_GH01.2\_01.0016.ppm) that are placed inside the \file{topo/partit/} subdirectory. If the display program is available, the script will open the picture (on hexagon, this can be done by the command 'module load imagemagick-login').

The \file{tile\_grid.sh} routine takes as input the number of subdomains along
the two dimensions of the model grid, along with the topography version it will
do the partitioning on. This means that you need to create MPI tiles for all the
topography versions you use.

The  routine \file{tile\_grid.sh} can be called in two modes, with positive or
negative arguments. For example:
\begin{itemize}
\item bin/tile\_grid.sh  8  7 01
\item bin/tile\_grid.sh -8 -7 01
\end{itemize}
%
The result of the two different partitioning methods is illustrated in
Figure~\ref{fig:partit}.
\begin{figure}[bt]
  \subfloat[Uniform partitioning of domain\label{fig:partita}]
     {\includegraphics[width=.45\textwidth]{Pics/depth_0051}}
  \subfloat[Non-uniform partitioning of domain\label{fig:partitb}]
     {\includegraphics[width=.45\textwidth]{Pics/depth_0056}}
  \caption{Examples of two different MPI partitions of a model domain}
  \label{fig:partit}
\end{figure}


With negative arguments, figure~\ref{fig:partita}, the
partitioning will be uniform - the size of each tile will be (approximately) the
same. This setup gives the lowest memory footprint when running the model.
On the other hand the workload between the MPI tiles is not evenly balanced,
since some tiles will have more sea points than others.
%
With positive arguments, figure~\ref{fig:partitb}, the partitioning is
non-uniform, but  the number of sea points on each
tile is approximately the same. The tiling balances well the work between MPI tiles meaning that the MPI waiting time will be reduced, but the memory footprint of the model will be higher since the total memory usage depends on the {\it maximum} size of all the tiles. 

The optimal partitioning depends on the model and the machine, but here are some general
guidelines
\begin{enumerate}
\item choose a uniform partitioning (negative values) when memory is an issue
\item choose a non-uniform partitioning (positive values) when the total number of MPI tasks is
low
\item choose a uniform partitioning when the total number of MPI tasks is high,
as the communication seems to be  more efficient in this case
\end{enumerate}
Remember that these are guidelines, and will probably depend on the model grid.
Some degree of trial and error is always needed to find the optimal setup. Also
note that  
memory per MPI tile  often becomes an issue when the number of MPI tasks is low, giving some
inconsistency between point 1 and 2 above.

\textcolor{red}{NB: Note that the choice of domain partitioning is not strictly
necessary before you start compiling the model, see
Section~\ref{sec:setup_code}.}

\subsection{Precribed barotropic inflow: ports.input \label{sec:port}}
In the standard HYCOM, the barotropic boundary condition are handle using a file called ports.input. 
There are to ways to force an inflow at a boundary: using input from an outer model (: i.e. file{lbflag} is equal to 2 in blkdat.input) or using a prescribed inflow based on theory (i.e. file{lbflag} is equal to 1 in blkdat.input). In this section we will focus on the later. For example in TOPAZ, we would like to consider the inflow from the Pacific through the Bering Straits, but no outer model is available. Note that here we only consider the barotropic boundary condition (Barotropic velocities and sea level), the baroclinic is handle using relaxation. This is addressed either in the next section (if you want to relax the water to climatology) or section~\ref{sec:setup_nest} if you want to relax the condition to an outer model.

Barotropic condition in HYCOM are treated with a flather technique. The 3 first points are used to solve the hyperbolic equation.The way this is handle in HYCOM is crude and the coastline must be uniform in the 3 first grid cells otherwise large instabilities will appear.  You should make sure that there is no island, or change in the coast line within these grid cell (try even to do that for a couple more: e.g. 5).

To activate a prescribed barotropic inflow in HYCOM, you need to first set \file{lbflag} is equal to 1 in blkdat.input. This mean that HYCOM will look for a file call ports.input where the specificities of the flow are detailed. This file should be place in the rots of the experiment directory. Second you need to edit this file:
A typical ports.input with two ports will look like that:
\begin{verbatim}
   2                                                'nports   ' = number of ports
   1                                                'pefold   ' = port transport e-folding time in days
   4                                                'kdport   ' = port orientation (1=N, 2=S, 3=E, 4=W)
 222                                                'ifport   ' = first i-index
 222                                                'ilport   ' = last  i-index (=ifport for east/west port)
 813                                                'jfport   ' = first j-index
 836                                                'jlport   ' = last  j-index (=jfport for north/south port)
   0.0                                              'svpnow   ' = existing port transport in Sv (+ve towards E or S)
   0.7                                              'svport   ' = target   port transport in Sv (+ve towards E or S)
   3                                                'kdport   ' = port orientation (1=N, 2=S, 3=E, 4=W)
 800                                                'ifport   ' = first i-index
 800                                                'ilport   ' = last  i-index (=ifport for east/west port)
   2                                                'jfport   ' = first j-index
  30                                                'jlport   ' = last  j-index (=jfport for north/south port)
   0.0                                              'svpnow   ' = existing port transport in Sv (+ve towards E or S)
   0.7                                              'svport   ' = target   port transport in Sv (+ve towards E or S)
\end{verbatim}
The first line will tell you how many ports (Barotropic inflow) you are going to impose. The second line is the e-folding time, and should not be changed. Then begin the description of each port:
\begin{itemize}
\item Need to specify a box area where the flow will be imposed (ifport, ilport, jfport,jlport)
\item Need to specify on which side of the box, the flow will be imposed (kdport)
\item Specify how much transport you want to have (svport)
\end{itemize}
The variable svpnow is not used.
There are some option to provide transport that varies seasonally or inter-annually, see Section \label{sec:code_cpp}.
If the code is compiles with a flag SEASONAL\_PORT or INTERANNUAL\_PORT. In the former case, the model will look for a ASCII file called clim\_tran.txt located in REGIONDIR/relax/exptnumber/, which contains daily estimate of transport. In the later case, the model will look for in the same directory a file corresponding to the running year (example: cm\_tran\_1980.txt if the model is run in 1980).


\section{Relaxation\label{sec:setup_relax}}
Relaxation fields can be used for sidewall/surface nudging and for initializing the
model. Currently the climatologies is based either on Levitus or on PHC climatologies. Levitus corresponds to the World of Atlas 2005 climatology while PHC is the version 3.0 of the Polar Science Center Hydrographic Climatology. The latter is recommended if the Arctic is within the model domain. Again the topography and the vertical structure of the model edited in Section \ref{sec:setup_exp} must now on never be changed.
A two-step procedure is needed to set up the relaxation fields. First, a horizontal interpolation on the z-levels of the climatology, (a step which depends on the model grid location).  The final step is a vertical interpolation from z-levels to the isopycnal levels, which depends on the setup of hybrid and isopycnal layers of the model. The routines which are
used to do these interpolation steps are from the \file{HYCOM\_ALL} directory which is supplied with the standard HYCOM distribution.

The relaxation fields are set up in the subdirectory \file{relax/}. A typical
listing of the contents of that directory may look like this
\begin{verbatim}
   drwxr-s---  3 knutali mohnsv 4096 2008-11-24 20:15 010
   drwxr-s---  3 knutali mohnsv 4096 2008-11-24 20:19 bin
   drwx--S---  4 knutali mohnsv 4096 2008-11-24 19:48 levitus
   drwx--S---  3 knutali mohnsv 4096 2008-11-24 19:36 phc
   -rw-r-----  1 knutali mohnsv 1808 2008-11-24 20:09 README.KAL
   -rw-------  1 knutali mohnsv 3580 2008-09-11 10:27 README.relax
\end{verbatim}
\begin{description}
\item[levitus] holds Levitus climatologies at predefined isopotential(z) surfaces interpolated horizontally on the model grid.
\item[phc] holds phc climatologies at predefined isopotential(z) surfaces interpolated horizontally on the model grid.
\item[bin] This directory contains routines for creating the climatologies.
\item[010] This contains the final relaxation files for experiment 01.0. Each final files contains monthly climatology interpolated into the model grid. It consists of the temperature (relax\_tem.[a,b]), salinity (relax\_sal.[a,b]), layer interface (relax\_int.[a,b]), and  relaxation coefficients (relax\_rmu.[a,b]).
\end{description}

\subsection{Routines for generating relaxation fields}
The routines in the {\it relax/bin/} directory can be used to set up the relaxation fields. This routines are called with the experiment number and retrieve the information about vertical structure and topography automatically. 
\begin{description}
\item[z\_levitus.sh] creates horizontally interpolated
fields on the original Levitus z-levels. Input is ``ksigma'' which is the same as
the ``thflag'' flag in the \file{blkdat.input} (i.e. sigma 0 or 2) used by the experiment you want to
create climatologies for. The fields created by this routine are dumped in
the {\it levitus/} subdirectory. The Levitus fields used in this procedure can be
retrieved from the HYCOM ftp site.
\item[z\_phc.sh] does the same as for z\_levitus.sh, but uses the PHC climatology. The fields 
created by this routine are dumped in the {\it phc/} subdirectory. The PHC fields 
used in this procedure can be retrieved from the HYCOM ftp site.
\item[relax.sh] creates a climatology interpolated to the density layers of a given experiment.
The input for this routine is the experiment number and the climatology to use(phc or levitus).
The actual density surfaces for this experiment will be retrieved automatically from the 
\file{blkdat.input} file. This routine requires
\file{z\_levitus.sh} or \file{z\_phc.sh} to be run first. The climatology is dumped to 
subdirectory corresponding to the variable ``E'' in the experiment file
\file{EXPT.src} (example -  010).
\item[relax\_rmu.sh] computes the relaxation coefficient for the lateral boundaries and/or for the ports (barotropic inflow, see subsection \ref{sec:port}). For the latter, you may want that the water flowing into the domain is equal to the climatology. 
The relaxation coefficient are stored in relax\_rmu.[a,b]. The script is user interactive in order to create in a first part
an input file called rmu.in (which the script will fetch in the {\it ../../expt\_XX.X} directory). You will be asked if you want to include a port in you model that is not on a model relaxed
boundary. If you don't have any port or if it is on the model grid boundary, just answer F, and keep answering the script
(Note that a relaxation width of 20 grid cell and a relaxation time scale of 20 days is recommended), That's it.

If the answer was true- i.e. that you have a port not at the model lateral boundary (as the Bering Strait in the Figure), then you will need to edit the file rmu.in by yourself. The script will stop and copy of a template at you REGION root directory. Once the file is edited and placed in the experiment subdirectory.  You can then relaunch the script answer .true. once more and the  corresponding binary file will be created. The rmu.in file will typically look like that:
\begin{verbatim}
F                     # Flag for relaxation on Eastern  boundary
T                     # Flag for relaxation on Western  boundary
T                     # Flag for relaxation on Southern boundary
T                     # Flag for relaxation on Northern boundary
20                    # Boundary relaxation length (grid cells)
20                    # Time relax
20                    # Point relaxation length (grid cells)
BER T -167.0 66.5     # Bering Sea point
END T    0.0  0.0     # !!!!! Ends point specification
700 800   1 1   F     # Specifies rectangle boundary flags (F: relaxation off in this rectangle)
\end{verbatim}
The first four lines indicates if you want to have relaxation along each model boundary (T/F).  Then the fifth line indicates on how many grid cell the relaxation should occurred for lateral boundary relaxation (strongest on the first grid cell and decreasing). The sixth line is the same but for given point relaxation (i.e. for ports defined). If your point is on a relaxed boundary, it would not change anything. For point relaxation, the  mask is circular. The next line corresponds to the relaxation time scale, which means the number of days it takes for having the grid cell matching the relaxation value. 
Then follow a list of point relaxation that consists of a tag name, a boolean (T/F), followed by geographical coordinate and \# sign to indicate the description of the file. Is it quite unpractical to have to provide coordinate and not pivot point (unlike in ports.input). There is a tool in the MSCPROGS/GridToLL/gridtoll see Section \ref{sec:MSCPROG} to extract the lon,lat from model grid point. It is usually common to place the relaxation centred in your port and make sure that the mask cover well the ports. Such list must be finished by the 10th lines. 
Finally, you can also remove  the relaxation in a given area as in the figure with the last line.
This example will provide the solution shown in Figure \ref{fig:rmu}. This is just and example, as usually you want the relaxation to cover the whole boundary.

\begin{figure}[bt]
  \subfloat[Example of relaxation mask with the given example of rmu.in. Red areas are equal to 1/20.]
     {\includegraphics[width=.45\textwidth]{Pics/relax_rmu}}
   \label{fig:rmu}
\end{figure}
\item[old\_lev\_nersc.sh] Creates some data files needed when running hycom with the NERSC "old" climatology option - which is generally deprecated..
\end{description}

\subsection{Prerequisites for generating relaxation fields}
To generate the climatologies data files for PHC or Levitus are needed, which
can be acquired from the HYCOM ftp site:\\[.5em]
\indent\indent\href{ftp://hycom.rsmas.miami.edu/awall/hycom/}{ftp://hycom.rsmas.miami.edu/awall/hycom/}
\\[.5em]
The topography files and the file \file{blkdat.input} must have been set up for the experiment which
relaxation fields will be created for.

In addition the environment variables HYCOM\_ALL, LEVITUS\_PATH 
and PHC\_PATH needs to be set up, this is done in the file
\file{REGION.src}.
The variable HYCOM\_ALL points to the path of the hycom {\it ALL} direcory,
while LEVITUS\_PATH and PHC\_PATH point to the location of the Levitus and
PHC climatologies, respectively. These variables are also set in \file{REGION.src}.

In addition, to use the "old" climatology option, the path to the Mohn-Sverdrup
programs should be set in the variable MSCPROGS. This variable should also
be set up in the file \file{REGION.src}.

\subsection{Procedure for generating relaxation fields}
To create climatologies this is the basic procedure to follow
\begin{enumerate}
\item Create the horizontally interpolated climatologies using the routines
\file{bin/z\_levitus.sh} or \file{bin/z\_phc.sh}. This step is experiment-independent just
needs to be run once. the PHC climatology is generally recommended, especially
if the Arctic is in the model domain.
\item Create the relaxation fields with the script \file{bin/relax.sh}
\item Create the relaxation masks with the script \file{bin/relax\_rmu.sh}
\end{enumerate}

\section{Forcing\label{sec:setup_force}}
The forcing directory contains various input fields which are used to run the
model. In this directory it is possible to create atmospheric forcing fields
(winds, air temperature, radiative fluxes etc), river forcing fields, and so on.
The contents of this directory will typically look something like this
\begin{verbatim}
   drwxr-s---  4 knutali mohnsv 4096 Jan  3 19:45 bin
   drwxr-s---  4 knutali mohnsv 4096 Jan  3 20:12 nersc_era40
   drwxr-s---  2 knutali mohnsv 4096 Jan  3 20:15 other
   drwxr-s---  4 knutali mohnsv 4096 Jan  3 20:14 rivers
   drwxr-s---  3 knutali mohnsv 4096 Jan  3 20:18 seawifs
\end{verbatim}
The \file{bin} directory contain script for generating the forcing, \file{nersc\_era40} contain climatological atmospherical forcing, \file{rivers} contains the rivers output forcing, and seawifs represent the climatology of water type used in HYCOM for computing the light penetration coefficient.

The routines in the \file{force/bin} directory are:
\begin{description}
\item[\file{nersc\_rivers.sh}] set up river forcing using point data supplied in a \file{rivers.dat} file. This file must be present in the force subdirectory (REGION/force). The main routine called is \file{rivers}, briefly described in Section~\ref{sec:river_forcing}. If the rivers are few and their flux well monitored, this is probably the best solution, otherwise, use the nersc\_trip.sh to generate the river forcing.
%
\item[\file{nersc\_trip.sh}] set up river forcing using runoff data from ERA-interim (or ER40 with a fix if wanted) + the TRIP05 database to direct the runoff data towards rivers. 
Section~\ref{sec:river_trip} has a brief description of the routines used.
%
\item[\file{nersc\_clim.sh}] This routine can be used to set up atmospheric climatology
forcing fields. Available climatologies are ``era40'', ``ncepr''. Climatology is sometimes used to complete synoptic forcing.
%
\item[\file{nersc\_synoptic.sh}] This routine can be used to set up synoptic
atmospheric forcing. It uses input from the file \file{infile.in} in the
experiment subdirectory to determine the synoptic and climatology data sets to
use, as well as the time period to create the forcing for. 
The main routine used is \file{forfun\_nersc-2.2}, described in Section~\ref{sec:synoptic-2.2}.
%
\item[\file{seawifs\_mon\_kpar.sh}] This routine can be used to set up the water type 
(Jerlov) based on observations from the SEAWIFS satellite. The use of this data
set in hycom runs depends on the setting of the parameter ``jerlv0'' in
\file{blkdat.input}. Uses routines in the HCYOM \file{ALL} directory.
\end{description}


\subsection{Prerequisites for generating forcing fields}
To generate the forcing files several environment variables have to be set in the
file \file{REGION.src}. First of all, the variables HYCOM\_ALL and MSCPROGS need
to be set. They point to the location of the standard HYCOM utility routines, as
well as utility routines developed at NERSC.
%
Additionally, several environment variables pointing to the location of data
sets are needed, they are:
\begin{table}[tb]
\begin{center}
\begin{tabular}{|clc|cl}
\hline
Variable & Description & Available period \\
\hline
SEAWIFS & SEAWIFS data set, supplied by standard HYCOM & monthly climatology \\
NCEP\_CLIM\_PATH &  NCEP clim produced at NERSC (1.915$^\circ$,NetCDF) & monthly climatology \\
ERA40\_CLIM\_PATH & ERA40 clim produced at NERSC (1.118$^\circ$,NetCDF)& monthly climatology\\
ERA40\_PATH & ERA40 data set  (1.118$^\circ$,NetCDF) &1957-2002 \\
ERAI\_PATH & ERA interim data set (.5$^\circ$,NetCDF) & 1989-2009 \\
ECNC\_PATH & ECMWF high-resolution data set (1/4$^\circ$,NetCDF) & 6/11/2006-- today \\
TRIP\_PATH& River pathway from Oki and Sund on a 5 km& climatology\\
WOA2005\_PATH & World Ocean Atlas (2005) data set (1$^\circ$NetCDF) & climatology \\
\hline
\end{tabular}
\caption{\label{tab:forcing_envvar} Some of the environment variables set by the file
\file{REGION.src} related to forcing datasets.}
\end{center}
\end{table}
The variables in Table~\ref{tab:forcing_envvar} need to be changed whenever the
location of the data changes or when the system is ported to a new machine.

\section{Nesting setup (subdirectory nest\_nersc)\label{sec:setup_nest}}
The nesting in version 2.2 follows closely the approach of standard HYCOM. The contents of the nesting directory will typically look something like this:
\begin{verbatim}
drwxr-x--- 3 fanf nersc 4096 2011-07-19 11:38 010
drwxr-x--- 3 fanf nersc 4096 2011-07-19 11:38 011
drwxr-x--- 2 fanf nersc 4096 2011-07-19 11:54 bin
\end{verbatim}
Where folder 010 contains files related to nesting for experiment 01.0, and folder 011 for experiment 01.1. The folder will contain a folder \file{outer} if it provides boundary condition, and a folder \file{inner} if it uses boundary condition. An experiment can be at both time inner and outer, and can provide boundary condition to multiple models.

The \file{nest\_nersc/bin/} subdirectory contains two routines for setting up data files for 
model nesting. Note that the nestbat routine (\file{topo/nestbat.sh}) that smooth the topography of the inner model to that of the inner one must already have been run (see Section \ref{sec:topo_create}). The routines in 
\file{nest\_nersc/bin/} contains setup routines for the two situations where the
model provides nesting files to a model, \file{nest\_outer.sh}, and when the model
receives nesting conditions from a model, \file{nest\_inner.sh}. Both of these
must be run before starting the model. \footnote{The need to set up a model before doing
outer nesting is new in this version of NERSC-HYCOM.} 
\begin{description}
\item[\file{nest\_outer.sh}] produces necessary files to allow the model to {\it
dump} nesting conditions for another model (referred as inner model). The outer model must know about the grid of the ``inner model''. The script use 3 arguments as input: the outer experiment number, the path to the region directory of the inner model and the experiment of the inner model. All necessary files for the nesting will be stored in the directory
\file{nest\_nersc/\$E/outer/} directory. A diagnostic netcdf file is also dump to make sure everything is configured correctly. Finally the script creates/modifies the file \file{nesting.in} in the experiment directory to indicate where the files will be dump. You must make sure that the targeted folder has writing permission.

%
\item[\file{nest\_inner.sh}] Run this script before running a model which {\it
receives} nesting conditions. It will set up a relaxation coefficients masks (calles relax\_rmu.[a,b]) that will be used for the the baroclinic fields from the outer model. As for lateral boundary, 20 grid cells is recommended. It will also produce a file \file{ports.input.nest} that is needed to handle the boundary condition for the barotropic fields. As explained in Section  \label{sec:port}, there are some rules that you need to double check here. Make sure that there is no island, or change in the coast line within these grid cells (try even to do that for a couple more: e.g. 5).
\end{description}


\section{Tide setup (subdirectory tides\_nersc)\label{sec:setup_tide}}
In version 2.2 there are several ways of enabling tides in the model. In
standard HYCOM tides can be enabled as a body force (mainly relevant for global
models), and as a forcing input on the model boundaries. This is done in the
\file{blkdat.input}  in the experiment directory. In addition the tidal forcing used in earlier versions of NERSC HYCOM can be used. This approach uses tidal Atlas, which is not used by standard HYCOM. 

The routines that are used to set up the NERSC tidal forcing are given in the directory
\file{tides\_nersc/bin/}. The routines inside that directory can be used to set
up tidal forcing from two different tidal databases. 
\begin{description}
\item[\file{tides\_fes.sh}] sets up tidal data along model boundary using the
FES2004 database.
\item[\file{tides\_csr.sh}] sets up tidal data along model boundary using the
CSR database.
\end{description}
Both routines also create diagnostics of phase and amplitude data from he
database, interpolated onto model grid points. To access the FES and CSR data
sets, the variables CSR\_PATH and FES\_PATH pointing to the dataset paths
need to be set up in the \file{REGION.src} file.


\section{Other setup routines (subdirectory other\_nersc)\label{sec:setup_div}}
This is the folder where remaining useful script are located. So far it only contains the script curviint.sh.

\begin{description}
\item[\file{curviint.sh}] interpolates the initial state from another model. When initialising a model from climatology, the necessary spin up time is usually of 5--10 years.  Starting from a model already in equilibrium allows to reduce the necessary spin up time (approximately 1 year). This approach is typically used in nested configuration where the initial state for the inner model is generated from the outer one.  The program assumes that the model that it interpolates from includes the local domain, that both grid are orthogonal curvilinear and that the vertical structure are the same (same number of layers and reference densities).
\end{description}
