\chapter{Model Code\label{cha:code}}
The NERSC-HYCOM code is based on the standard HYCOM code, but there are some
additional features. The new features are mainly in the domain of ice modelling (sea-ice dynamics and thermodynamics) and data assimilation (for running perturbed ensemble simulation). There are also changes due to diagnostics (weekly and daily averages)
All the new routines from NERSC are located in a separate subfolder (\file{nersc}) and when the original file from HYCOM is changed the change are easily identified using the CPP FLAG \file{NERSC\_VERSION}.Work is done to converge as much as possible with the standard way of
doing things in HYCOM. For example, the barotropic/baroclinic nesting routines now rely on the
standard HYCOM approach, with only input/output and addition of tides being
different from the standard version. Another example is the conformal mapping
routines which in this version is completely decoupled from the HYCOM model
code. The conformal mapping routines are only used by model setup routines in
Section~\ref{cha:setup}.




\textcolor{red}{NB: For a experienced user the checklist in
Section~\ref{cha:smr} may be used. It gives step-by-step instructions on how to
set up the model.}


%\begin{table}[hp]
%\centering
%\footnotesize
%\caption{Transport estimates over sections shown in Figure \ref{fig:sectionsmap} for TOPAZ4, NORDIC for the year 2008. Transport is computed from the Start point to the End point. The transport is positive towards the eastern side of the first point. Model monthly averaged fields were used for the transport computations except for transport calculations in the Barents Sea Opening and Svin\o y section where observed transport and NORDIC transport were computed on daily averaged time scale. }
%\vspace{0.5cm}
%%\centering
%%\begin{adjustwidth}{-1.0cm}{0.0cm}% adjust the L and R margins by ? inch
%\begin{threeparttable}[b]
%\begin{tabular}{l  c c c c c c c }
%\hline\hline
%         &           &           & Model        & Net        & Positive  & Negative \\
%Sections & Start     & End       & or if avail. & transport  & transport & transport \\ 
%         & (lon,lat) & (lon,lat) & Obs. source  & in Sv      & in Sv     & in Sv 
%\\[0.9ex]
%% [1ex] adds vertical space
%\hline
%%1st row
%                  &           &          & TOPAZ Reana& 2.3 & 5.5 & -3.2 \\[0.5ex]
%\raisebox{1.5ex}{Fram Strait} & \raisebox{1.5ex}{(-20$^{\circ}$W,79$^{\circ}$N)}&\raisebox{1.5ex}{(15$^{\circ}$W,79$^{\circ}$N)} & TOPAZ free & 2 & 4.4 & -2.4 \\[0.5ex]
%                 &           &          & Mauritzen 11 & 2.08 & 7.8 & -6.78 \\[0.5ex]
%\hline
%%2nd row
%                  &           &          & TOPAZ Reana & 2.3 & 5.8 & -3.79 \\[0.5ex]
%Fram Strait AWI\tnote{*} & (-7$^{\circ}$E,78.83$^{\circ}$N)&(9$^{\circ}$E,78.83$^{\circ}$N) & TOPAZ free & 1.86 & 5.5 & -3.7 \\[0.5ex]
%                  &           &          & AWI\tnote{*} estimates & 4.29 & 16.49 & -12.49 \\[0.5ex]
%\hline
%                  &           &          & TOPAZ Reana & 2.95 & 3.1 &-0.14 \\[0.5ex]
%Barents Sea Opening & (20$^{\circ}$E,70.5$^{\circ}$N)&(19.15$^{\circ}$E,74.25$^{\circ}$N) & TOPAZ free & 2.7 & 2.96 & -0.24 \\[0.5ex]
%                  &           &          & IMR\tnote{**}\quad Daily & \raisebox{-1.5ex}{2.2} & \raisebox{-1.5ex}{} &  \raisebox{-1.5ex}{} \\[0.5ex]
%\hline
%%5th row
%                  &           &          & TOPAZ Reana & 3.2 & 4.8 & -1.6\\[0.5ex]
%Denmark Strait & (-37$^{\circ}$E,66.1$^{\circ}$N)&(-22.68$^{\circ}$E,66$^{\circ}$N) & TOPAZ free& 3.1 & 4.7 & -1.6 \\[0.5ex]
%                  &           &          & \citet{hansen08}\tnote{***} &  & 6.5 & -0.8 \\[0.5ex]
%\hline
%%6th row
%                  &           &          & TOPAZ Reana & -1.18 & 1.8 & -2.9\\[0.5ex]
%Iceland-Faroe & (-15$^{\circ}$E,65$^{\circ}$N)&(-6.85$^{\circ}$E,62.2$^{\circ}$N) & TOPAZ free& -1.4 & 2.25 & -3.7 \\[0.5ex]
%                  &           &          & \citet{hansen10} &  & 1 & -3.8 \\[0.5ex]
%\hline
%%7th row
%                  &           &          & TOPAZ Reana & -2.1 & 1.49 & -3.6\\[0.5ex]
%Faroe-Shetland & (-6.38$^{\circ}$E,61.16.2$^{\circ}$N)&(-3.55$^{\circ}$E,60.1$^{\circ}$N) & TOPAZ free & -1.81 & 2.14 & -3.94 \\[0.5ex]
%                  &           &          & \citet{hansen08} &  &  & -3.8 \\[0.5ex]
%\hline
%%7th row
%                  &           &          & TOPAZ Reana & -0.37 & 0.49 & -0.86\\[0.5ex]
%Shetland-Scotland & (-1.35$^{\circ}$E,60.4$^{\circ}$N)&(-3$^{\circ}$E,56.5$^{\circ}$N) & TOPAZ free & -0.51 & 0.22 & -0.73 \\[0.5ex]
%                  &           &          & \citet{turrell92} &  & 0.2 & -0.72 \\[0.5ex]
%\hline
%%8th row
%                  &           &          & TOPAZ Reana & -6.1 & 1.6 & -7.7\\[0.5ex]
%Svin\o y & (-1.7$^{\circ}$E,65.25$^{\circ}$N)&(5.2$^{\circ}$E,62.37$^{\circ}$N) & TOPAZ free & -5.8 & 2.6 & -8.4 \\[0.5ex]
%                  &           &          & IMR\tnote{**}\quad Daily & \raisebox{-1.5ex}{-4.7} & \raisebox{-1.5ex}{} & \raisebox{-1.5ex}{} \\[0.5ex]
%\hline
%%9th row
% \hline\hline
%\end{tabular}
%\begin{tablenotes}
%    \item[*] {AWI: Alfred Wegener Institute, personal communication. For this section, mean are processed from January to June for models and observations. }
%    \item[**] {IMR: Institute of Marine Research, personal communication}, they kindly provided daily average time series for 2008.
%    \item[***] { Estimate to be clarified: 6.5\,Sv represent 3\,Sv for the overflow and 3.5\,Sv for the surface outflow including the EGC and the surface outflow from Canadian Archipelago. EGC transport estimated further north (79$^{\circ}$N) is about 3\,Sv \citep{foldvik88}.}
%\end{tablenotes}
%
%\end{threeparttable}
%%\end{adjustwidth}
%
%\label{tab:transport}
%%\end{sidewaystable}
%\end{table}



\section{Differences from standard HYCOM}
The discrepancies from the standard HYCOM are now detailed. Most of the option will be activated by CPP flags when you are compiling the code, also some are specified in input files: infile.in, infile2.in, infile.evp, infile.icestate.
 The changes are group in different classes: 
Input/Output (restart name convention and run limits), Diagnostics (daily, weekly); tides (Atlas choices, add currents); nesting; ice dynamics; thermodynamics, and atmospheric forcing. 

\subsection{Input/Output}
In NERSC-HYCOM, the  output times can be determined directly in the
input file called \file{infile.in}. This file also determines the start and end
times of a model integration, as well as some other parameters for the model
runs. The FORTRAN routine for setting up the integration times and diagnostic outputs is 
\file{src\_2.2.12/nersc/m\_limits.F90}. This file will set up the diagnostic
times using types defined in the module \file{src\_2.2.12/nersc/mod\_diagnostics.F90}. The file names convention in NERSC-HYCOM is also different from the standard HYCOM.  In standard HYCOM
restart files are simply called \file{restart.in} or \file{restart.out}. In
NERSC-HYCOM the time info is a part of the restart file name, as well  as a
three-letter ID called ``rungen'' - given on the second line of the file
\file{infile.in}.  A typical file name for a restart file in NERSC-HYCOM is\\[1em]
\indent\indent\file{IDBrestart1988\_050\_00.[ab]}\\[1em]
where ``IDB'' is the three-letter ID, and after ``restart'' follows the year, day  of year 
(a.k.a ordinal date) and hour.  
For the diagnostics files, or ``archv'' files, the name is the same as in
standard HYCOM, but the rungen is used as a file name prefix. Note that in NERSC archv are very seldom used because it exists other type of diagnostic files (daily average, weekly average).

The choice of different file names means that there are slight changes in the
standard HYCOM FORTRAN files \file{src\_2.2.12/restart.F} and
\file{src\_2.2.12/archv.F}. Also, when it comes to restart files, HYCOM has
traditionally had several restart files for different model components, such as
biological, ice and ice dynamics models. In version 2.2 many of these restart
files are removed, and the fields are in stead dumped to the HYCOM restart file,
leaving a single restart file. This is done by setting the CPP flag
``SINGLE\_RESTART''. Some restart files, such as the ice model files,
are still saved to separate restart files. This can be switched off, however,
by setting the CPP flag ``SINGLE\_RESTART\_ONLY''.


\subsection{Diagnostics}
In addition to the standard HYCOM diagnostics, it is possible to create extra
diagnostics from model runs. Two such extra diagnostics are daily and weekly
averages, which must be enabled during compilation 
with CPP flags,
and in addition they must be enabled
in the file \file{infile.in} when running the model. The reason for using CPP
flags for these two options is that they require 3D variables to store the
averaged data and will requires a little bit more memory (this can be switched off).

Daily averages are processed by the module
\file{src\_2.2.12/nersc/mod\_daily\_average.F90}. To use daily averages they must
be set up by setting the CPP flag ``DAILY\_AVERAGE'', and they must be activated
at run-time in \file{infile.in}. The model fields are average at every model time step.
%
Weekly averages are processed by the module
\file{src\_2.2.12/nersc/mod\_average.F90}. To use weekly averages they must
set up by setting the CPP flag ``WEEKLY\_AVERAGE'', and they must be activated
in \file{infile.in}. You need to provide in the infile.in how often you use the model output for averaging (typically 2 hours).


It is also possible to diagnose grid point time series sampled every hour. This
is done by the FORTRAN module \file{src\_2.2.12/nersc/mod\_gridp.F90}. This is
activated at run-time in \file{infile.in}.

\subsection{Nesting}
The nesting in NERSC-HYCOM is now very similar to that of the standard HYCOM.
Because the nesting in standard HYCOM was introduced at a later stage than in
NERSC-HYCOM there were several duplicate routines for doing the nesting. This
included the application of nesting to baroclinic and barotropic variables,
which had to be put into the standard code at various places.
In the new version of NERSC-HYCOM the procedure of standard 
HYCOM is followed, the exception being the input and output of nesting files.
The input and output of nesting fields still follows conventions particular to
NERSC-HYCOM. However, with version 2.2 it should be possible to use the nesting
input/output approach of standard HYCOM as well.

The FORTRAN module \file{src\_2.2.12/nersc/mod\_nesting.F90} is used to
initialize, write and read nesting conditions. The actual calling of 
NERSC nesting routines is done inside the main hycom FORTRAN files
\file{src\_2.2.12/mod\_hycom.F90}, where it is used when saving nesting
conditions, and inside the routine \file{src\_2.2.12/forfun.F}, where it is used
to read nesting conditions. The routine which reads nesting conditions,
\file{nestcondread}, is a drop-in replacement for the equivalent routines in
standard HYCOM \file{forfun.f}: \file{rdnest\_in} and \file{rdbaro\_in}.

To let a model save nesting conditions to be used in another model, it must be
activated in \file{infile.in}. This is done on line 11 where you specify the
outer nesting flag, and the file saving interval of nesting conditions (in
hours). For a model to read nesting conditions line 12 of \file{infile.in} must
be edited, where the inner nesting flag and the nesting read interval is
specified. In addition inner nesting must be enabled at compile time by setting
the CPP flag ``INNER\_NESTING''. If your model includes ICE, your need also to define the flag ICE\_NEST, to save ice properties as well.

\subsection{Tides}
In version 2.2, tidal forcing has been introduced in standard HYCOM. In 
NERSC-HYCOM the tidal forcing has been present for a much longer time. The tidal
forcing in NERSC-HYCOM is based on the use of tidal atlases, where the
sea-surface elevation is specified on the boundary using prescribed amplitudes,
periods and phases shifts. The approach in standard HYCOM is somewhat different,
where (AFAIK) tidal atlases are not used. In standard HYCOM it is the
equilibrium tide which is used, excluding the effects associated with bathymetry
and basin configuration. 

The tides are handled by the FORTRAN module
\file{src\_2.2.12/nersc/mod\_tides\_nersc.F90}. This module is fairly
self-contained. It is called from the main hycom routine \file{src\_2.2.12/mod\_hycom.F}
for initialization and for updating the tidal constituents at regular intervals.
It is also called from the  FORTRAN file \file{src\_2.2.12/latbdy.F}, where it
is used to supply the correct boundary conditions during nesting. 
\textcolor{red}{Presently inner nesting must be enabled in order to use the NERSC tidal module. Some
restrictions are set on the use of the NERSC tidal flag in NERSC-HYCOM. The most
obvious is that tidal forcing using the NERSC tidal module and the standard
HYCOM tidal module can not both be active at the same time.}

\subsection{Ice dynamics: EVP, MIZ and ICESTATE}
Ice modelling is one of the main topic of interest and area of development of the NERSC-HYCOM. The code is very different from the standard HYCOM in this regard. Recently, effort are made in COAPS to couple the standard HYCOM with CICE model which may allows us to converge to the same distribution in the future. Ice modelling at NERSC are fairly comparable to CICE model, but the ice model at nersc is coded on a Arakawa C-grid instead of B-grid as in CICE in order to reach a better agreements with HYCOM.
NERSC-HYCOM gives the possibility to use a one thickness ice model (CPP flag ICE) or multi category ice model (CPP flag ICESTATE). You also get the possibility to use two different ice-rheologies: EVP and MIZ. In Elasticous Visous Plastic (EVP) the ice rheology will depend on the shear in x and y, and you will transit from a viscous rehology (if the stresses are within the ellipse) to a plastic rehology (if the stress are out of the ellipse). Such behaviour is well suited in pack ice and allows for ridging, crack, ice bridge. However, when the scale of the ice gets smaller (at the ice edge), this behaviour is no longer suited, and the rheology becomes collisional (Marginal ice zone rheology). It is quite difficult to identify where the rheology should transit from EVP to MIZ. Ideologically, the difference represent a different type of ice. At the edge, the concentration is lower, and the ice is forming panckake. Two approaches are proposed in the NERSC-HYCOM code. A first one is based on ice concentration (CPP flag EVP+MIZ) while a second is based of the floe size that are break by the waves (CPP flag EVP+MIZ+WAVES). The latter is currently in a development stage.
By default now, the advection of ice properties is calculated using a 3rd order WENO scheme, with a 2nd order Runge-Kutta time discretisation.
All the code related to the ice modelling are located in \file{src\_2.2.12/nersc/EVP} for EVP and MIZ and in ICESTATE for the multi-category ice model.

\subsection{Thermodynamic forcing}
The thermodynamic in NERSC-HYCOM over open water, ice covered and snow covered ice is based on the formulation from Drange and Simonsen 1996 with a correction of heat fluxes for subgrid scale ice thickness heterogeneities following Fichefet and Morales Maqueda [1997]. The surface fluxes are forced with a bulk formula parametrisation [Kara, 2000]. The short wave radiation are estimated theoretically, based on the cloud coverage. The latter is by default updated every day, but you have the possibility to update the field every 3 hours (CPP flag \file{DIURNAL}). There is 2 ways to define the momentum flux: the one from Large and Pond [1981] that is used by default, or the one from Kara et al. 2002 that use a more advance formula with respect to the wind speed and that accounts for discrepancies between the model SST and air temperature (CPP flag \file{KARA2002}).

\subsection{Atmospheric forcing}
The atmospheric forcing in NERSC-HYCOM is based on that of the standard HYCOM
version. However, since the thermodynamical forcing is different, there are some
extra fields used in NERSC-HYCOM. These fields include winds (standard HYCOM
uses only atmospheric stress), relative humidity, clouds, and sea-level
pressure. To get these fields some changes (or rather, additions) have been
introduced in the file \file{src\_2.2.12/forfun.F}. This file is based on
standard HYCOM, but the reading of the new fields has been introduced. The
modified \file{forfun.F} uses parameters from the FORTRAN module
\file{src\_2.2.12/nersc/mod\_forcing\_nersc.F90}, and the added atmospheric fields are
also read into variables defined in that file.

It is possible to force the model in a manner similar to standard HYCOM, where
forcing input is read from the "ab"-format. This requires that forcing fields are preprocessed using NERSC utility routine described in Section~\ref{sec:synoptic-2.2}. Otherwise, you get the possibility to read
synoptic forcing directly from few selected data sets :ERA40 dataset (NetCDF files), the
NCEPR data set (NetCDF files, aka ``ncep''), ECMWF T799 (NetCDF, aka ``ecnc''), ECMWF T159
(binary files, aka ``ecmwf''), ECMWF T399 (binary files, aka ``metno''), and the ERA interim reanalysis (ECMWF T399 (netcdf files, aka ``era-i''). In order to use this option, the CPP flag ``FORCING\_INLINE'' must be set when
compiling the code. For further information on setting up the forcing are described in Section~\ref{sec:setup_topo}.

\subsection{Random forcing}
The random forcing module is used to modify the atmospheric forcing used in
NERSC-HYCOM. By using the routines of this module, fields with a
prescribed variance and  correlation in time and space will be added to the
original forcing fields. The random forcing features are mainly used when
running a model ensemble, and its purpose is to produce a spread in the
ensemble, which is important when doing data assimilation using the EnKF.

The random forcing is taken care of by the FORTRAN module
\file{src\_2.2.12/nersc/mod\_random\_forcing.F90}. the forcing fields are assumed to
be red noise simulated by a spectral method as described in \citep{eve03a}. The perturbations are computed in a Fourier space with a decorrelation time-scale and horizontal decorrelation length scale. Variables to be perturbed, time and space decorrelation time scale are provided in the input file (\file{infile2.in} as described in Section \ref{sec:infile2.in}. It is possible to derive wind field perturbations geostrophically from the SLP perturbations, their intensity being inversely proportional to the value of the Coriolis parameter. This ensure some kind of consistency in the perturbed field. The standard deviation set in the input file correspond to 40$^\circ$N. There is a smooth transition to the Equator, where winds get aligned with the gradients of SLP perturbations. In this module are all routines related to the random forcing. In addition the value of the random forcing are stored in the  restart files to avoid discontinuity in the perturbation. This is handled by the restart routines in \file{src\_2.2.??/nersc/mod\_restart.F90}. \
There is also a possibility to perturb the ice dynamics by perturbing the squared parameter e$^2$ in the EVP rheology. This parameter represents the ratio between the minor and the major axis of the elliptic yield curve, which partly controls the transition between the viscous and plastic flows for a given stress. In other words,
it represents the shear to compression strength ratio. If the CPP flag PERTURB\_ICE is activated, this parameter is chosen randomly from a Gamma distribution (k=1 and $\sigma$ =5). To use this flag you need to have the DRANDGAMMA library available present in the acml library on hexagon (module load acml). \file{src\_2.2.??/nersc/EVP/evp\_init.F}.


%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%
\section{Code setup\label{sec:setup_code}}
%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%
This section deals with the setup of the model code, meaning the setting of
CPP compilation flags, the code itself , configuration files and Build directory,
and the preprocessing of model dimensions.
\subsection{Setup of directories}
To compile the HYCOM model three directories are needed: \file{src\_2.2.12/}, \file{config/}, and a Build directory that corresponds to a given experiment, (the name will be \file{/Build\_V2.2.12\_X01.0/} for expt 01.0). Template are available in the svn copy:
\\[0.7em]
\indent\indent \file{\~{}/hycom/HYCOM\_2.2.12/CodeOnly/}\\[0.7em]
However, the creation of these files should be automatic if the script \file{newRegion.sh} and \file{newExp.sh} have been used (See Section~\ref{sec:setup_expt}).
These should be handle automatically by the script. Beware that (\file{src\_2.2.12/}) and config directory (\file{config/}) are linked to the svn copy of the code, so that the model makes always use of the most recent code. The Build directory is copied physically as it contains files that details the model parametrisation and are thus dependent on the experiment.
%
%
\subsection{Setup of model dimensions\label{sec:code_dims}}
When setting up the model code, the grid dimensions as well as MPI setup
needs to be specified. This is provided inside the file \file{dimensions\_nersc.h}
inside the Build directory. 
This file is created automatically by a shell script. To do so, go into the Build directory corresponding to your experiment, and run the script \file{setuppatch.sh}.
This script is called with the number of MPI domains as its only argument, for example: \\[0.7em]
\indent \indent \file{setuppatch.sh 51}\\[0.7em]
The command above will look for ``patch'' files which have a total of 51 MPI. The ``patch'' files created in Section~\ref{sec:setup_partit} and placed in the directory \file{topo/partit/} relative to the top-level region directory. The names of the patch files are related to the topography version of the experiment number (set in exptXX.X EXPT.src) and the name of the region.  A typical ``patch'' files names would be:\\[0.7em] \indent \indent \file{topo/partit/depth\_FRMa0.05\_01.0051 }\\[0.7em] where in this example ``FRMa0.05'' is the region name, ``01'' is the topography version,
and the last number, ``0051'', is the total number of MPI tiles for this patch file.
The script will return an error message if the file is not found.

The file created \file{dimensions\_nersc.h} looks like this:
\begin{verbatim}
      integer, parameter :: itdm=400
      integer, parameter :: jtdm=320
      integer, parameter :: kdm=28      
      integer, parameter :: idm=50
      integer, parameter :: jdm=46
      integer, parameter :: iqr=8
      integer, parameter :: jqr=7 
\end{verbatim}
where $itdm$, $jtdm$ and $kdm$ are the 3D  dimensions of the model domain retrieve automatically from the blkdat.input. $idm$ and $jdm$ are the maximum horizontal dimensions of a tile (MPI sub-domain), and
$iqr$ and $jqr$ are the maxim number  of tiles in the two horizontal grid
dimensions. 



\subsection{Setup of Makefile configuration files}
To compile the model, it is necessary to specify the correct config file to
include. This file specifies the compiler and  compiler flags to use, as well as
some of the CPP flags needed for different architectures. The directory
\file{config/} under the top-level region directory contains configuration files
for various architectures. Their names are typically linked to architecture and
parallelization types. As an example, the config file \file{config/xt4\_mpi}
contains makefile macros needed when compiling HYCOM for the Cray XT4, and using
MPI parallelization (hexagon,Bergen). The model is also tested on other machine: IBM AIX 5L (NJORD, Trondheim) \file{config/sp5\_mpi}; intelCHPC (Cape town) \file{config/intelCHPC\_mpi}.

The files in the \file{config/} directory have been obtained from the standard
HYCOM distribution, and can be used as starting points for creating new config
files. In most cases, however, it should be possible to use these without
modification when compiling the model.

To set up the Makefile to use a specific configuration file, the file
\file{flags} in the build directory needs to be modified. At the very top of
this file two variables are defined, which is used to include the correct
configuration file:
\begin{verbatim}
   ARCH=xt4
   TYPE=mpi
\end{verbatim}
The variable ARCH denotes the architecture to compile for and the variable TYPE
denotes the parallelization type to use. This will be used to read the correct
config file into the makefile named \file{Makefile}. With the variable
definitions used above, the makefile will look for the configuration file 
\file{config/xt4\_mpi}.

In addition it is sometimes necessary to include libraries when compiling the
model. This can be set in the configuration files, but it can be more convenient
to set them in the file \file{flags}, where the variables FCFFLAGSNERSC and LIBS
should be set according to the location of the libraries you need to include. As
an example, consider including the NetCDF libraries, which can usually be
included in a  manner similar to this
\begin{verbatim}
   LIBS := $(LIBS) -L/usr/local/lib/ -lnetcdf
   FCFFLAGSNERSC := $(FCFFLAGSNERSC) -I/usr/local/include/
\end{verbatim}
this assumes that the NetCDF include files are installed in
\file{/usr/local/include/}, and that the libraries are installed in
\file{/usr/local/lib/}. The location of libraries differ from machine to machine,
so this must usually be modified when porting the code.

\subsection{Setup of CPP flags\label{sec:code_cpp}}
The CPP FLAGS  are an easy way to chose some part of the code. These choice are made in the file named \file{flags} in the build directory.
Here various CPP options from the standard HYCOM distribution can be set, as
well as CPP options particular to NERSC-HYCOM. Table \ref{tab:CPPNERSC} lists
the various NERSC-related CPP options used when compiling the model.
\begin{center}
\begin{table}[htb]
\begin{tabular}{| m{.29\textwidth} | p{0.70\textwidth} |}
\hline
Variable & Description \\
\hline
NERSC\_VERSION$^\star$ & Use NERSC-HYCOM code and any of the flag below (Must be defined). \\
\hline
MODEL DYNAMICS\\
\hline
DIURNAL$^\star$ & Update the clouds fields every 3 hours instead of daily for shortwave flux \\
FORCING\_INLINE$^\star$  & Read forcing data directly from raw datasets instead of standard HYCOM preprocessed files \\
MATS\_RELAX$^\star$ & turn off relaxation of surface salinity when departure from climatology is larger than .5 psu. This is deprecated in the version 2.2.37 of HYCOM, and you should use negative sssflg in blkdat to activate it, See Section \ref{sec:blkdat} \\
SEASONAL\_PORT& Allow seasonal variability in the barotropic inflow (input file needed)\\
INTERANNUAL\_PORT & Allow inter-annual variability in the barotropic inflow (input files needed) \\
NEST\_INNER$^\star$  & Applying NERSC-HYCOM nesting conditions 
                 at the boundary of the model. \\
NEST\_OUTER$^\star$    & Saving NERSC-HYCOM nesting conditions \\
SINGLE\_RESTART$^\star$ &  Put all models variables in .(a,b) files still dump old ICE and EVP files\\
SINGLE\_RESTART\_ONLY$^\star$ &  Stop dumping ICE.uf EVP.uf files\\
\hline
ICE MODELLING \\
\hline
ICE$^\star$         &  NERSC-HYCOM standard heat flux  formulation. (req. EVP) \\
EVP$^\star$        & Standard EVP sea-ice model (req. ICE)\\
MIZ           & Transition to collisional rheology (Marginal ice zone) in low concentration ice (req. EVP, ICE)\\
WAVES$^{\star\star}$  & Transition to MIZ estimated from floe size (calculated from wave field, req. EVP, MIZ) \\
ICE\_NEST$^\star$      & Activates sea-ice nesting when running (req. NEST\_INNER). \\
ICESTATE    & Advanced multi-category sea ice model\\
TEST\_ICE\_AGE & Compute ice age as seen by satellite (funct(compression,age),req. EVP,ICE) \\
ICEAGE & Compute ice age  \\
ALBSNW\_EVOL    & Albedo evolution based on Douville et al (95),  decrease with ageing (req. EVP,ICE)\\
SSNOWD\_ICE$^{\star\star}$ & use a snow model (req. EVP,ICE)\\
SSNOWD$^{\star\star}$ & use a  snow model (req. ICESTATE)\\
\hline
ECOSYSTEM MODEL\\
\hline
NOR05     & Switch on NORWECOM ecosystem model \\
ZOOPL       & Add zooplankton to NORWECOM (req. NOR05)\\
DETP        & Add phosphorous detritus to NORWECOM (req. NOR05) \\
NEST\_BIO & Nesting for ecosystem model NORWECOM (req. NOR05)\\ 
CALANUS & Individual based model for C. finmarchicus\\

\hline
ASSIMILATION \\
\hline
EVP\_DRIFT\_ASSIMILATION & Makes HYCOM dump daily averages of sea ice drift from
an ensemble \\
TRACK\_ASSIM & Makes HYCOM dump daily averages of sea surface height from
an ensemble \\
PARAM\_EST & Add bias estimate values for SSH, SST and sea ice strength \\
PERTURB\_ICE & Perturb the ratio of the ellipse in the EVP to get more spread in the ice ensemble \\
\hline
DIAGNOSTICS \\
\hline
WEEKLY\_AVERAGE & Used to dump on weekly averages of the model \\
DAILY\_AVERAGE & Used to dump on daily averages of the model�\\
DAILY\_ENSVAR & Dump the squared of ssh, fice, hice to compute the full variance (diurnal/ensemble) \\
ICE\_DYN\_DIAG   & stored diagnostic variables from EVP in daily average (stress,pressure, tensor) \\
\hline
\end{tabular}
\caption{\label{tab:CPPNERSC}CPP Flags used for setting up NERSC-HYCOM. The term ``req." indicates that some conjugate flag are required. $^\star$ indicates that the flag is recommend by default for standard implementation.$^{\star\star}$ indicates that the flag is not fully tested yet.}
\end{table}
\end{center}
These flags need to be set in the file \file{flag} inside the build directory.
Look for the variable CPPFLAGSNERSC and edit it to include the features you want
to use in the model, the following example;\\[0.7em]
\indent CPPFLAGSNERSC := \$(CPPFLAGSNERSC) -DNERSC\_VERSION -DNEST\_INNER -DICE
-DEVP\\[.7em]
will switch on inner nesting, the standard ice model and the evp sea-ice
dynamics model.\textcolor{red}{NB: Do not switch off NERSC\_VERSION unless you
know what you are doing, the compilation will likely fail.}

\subsection{Compilation}
Assuming everything is set up, the model is now ready for compilation. To
compile, simply type \\[0.7em]
\indent\indent make\\[.7em]
Note that it is no longer necessary to do ``make new'', since the source files
and the dependencies needed to compile the model are set up in the the config
files and the include files \file{flags} and \file{dependencies}.
Note that you need to clean the Build everytime you are editing the flags file or the make file, but not when you edit the source code. To clean up the directory, type:\\
\indent\indent make clean\\[.7em]
If the compilation fails, there can be several reasons. Typical culprits are
\begin{itemize}
\item Missing \file{dimensions\_nersc.h}, which must be set up before compiling,
see Section~\ref{sec:code_dims}.
\item Missing config files in the \file{config/} directory, make sure that the
ARCH and TYPE variables are set up properly in the file \file{flags} in the
build directory.
\item Invalid config files in the \file{config/} directory. When porting the code to a new machine, it is
sometimes necessary to create a new config file. Use one of the existing config
files in the directory \file{config/} as a starting point.
\item Missing libraries and/or include files, make sure they are set up properly
as described in Section~\ref{sec:setup_code}.
\item Missing source files, make sure they are set in the file \file{flags}.
They should be, but you never know. If HYCOM is modified, and new FORTRAN files
are created, you will have to add them to the file listing in \file{flags}.
\item Missing dependencies. When the model is compiled and FORTRAN90 modules are
used, dependencies need to be set up. These are normally set in the file
\file{dependencies}. If new FORTRAN90 modules are used with the NERSC-HYCOM
code, they may need to be added in this file.
\end{itemize}



