text:
To configure the ERT, you need to create a configuration file that is setup
correctly for the machine you will be testing.  Several examples
are provided with the ERT distribution.  They are located at under ``Config''
in the installation directory and are named ``config.*''.

In addition to the example configuration files, there are corresponding batch
scripts under ``Batch'' (for systems with batch queues) and results under
``Results''.

The examples currently provided are:

\begin{itemize}

\vspace{-0.1in}
\item{``config.babbage.nersc.gov.01'' - An example configured for testing a
Knight's Corner (KNC) node using various combinations of MPI and OpenMP to
exercise the host and not the MIC .}

\vspace{-0.1in}
\item{``config.babbage.nersc.gov.02'' - An example configured for testing a
Knight's Corner (KNC) node using OpenMP to exercise the MIC and not the host.}

\vspace{-0.1in}
\item{``config.edison.nersc.gov.01'' - An example configured for a
Cray XC30 node (2 x 12-core Intel "Ivy Bridge" processors) using various
combinations of MPI and OpenMP.}

\vspace{-0.1in}
\item{``config.madonna.lbl.gov.01'' - An example configured for a
Linux workstation (8-core Intel Xeon CPU E5530, 2.4 GHz) using various
combinations of MPI and OpenMP.}

\vspace{-0.1in}
\item{``config.madonna.lbl.gov.02'' - An example configured for a
Linux workstation (8-core Intel Xeon CPU E5530, 2.4 GHz) using only OpenMP.}

\vspace{-0.1in}
\item{``config.mira.alcf.anl.gov.01'' - An example configured for a
IBM Blue Gene/Q node (16 1600 MHz PowerPC A2 cores) using various
combinations of MPI and OpenMP.}

\vspace{-0.1in}
\item{``config.titan.ccs.ornl.gov.01'' - An example configured for a
Cray XK7 node (host: 16-core 2.2 GHz AMD Opteron 6274 (Interlagos) processor)
using various combinations of MPI and OpenMP.}

\vspace{-0.1in}
\item{``config.titan.ccs.ornl.gov.02'' - An example configured for a
Cray XK7 node (GPU: NVIDIA Kepler accelerator) using various number of
thread-blocks and threads-per-block.}

\vspace{-0.1in}
\end{itemize}

Looking at these examples will help clarify all the descriptions and
definitions below.

In general, the configuration file contains lines that define the parameters
needed by the ERT to compile the specified drivers and kernels, run them over a
range of MPI, OpenMP, and Cuda/GPU settings, gather the results, and generate
the final roofline characterization of the machine.

Starting with an example configuration file (see above) and modifying it is
probably the best way to get started.  In the next sections, all the ERT
configuration directives will be explained so you can understand the examples
and make your own configuration files.

If you problems using the ERT with your configuration file, please read the
section on troubleshooting the ERT.  It contains suggestions that the
developers and testers have used when trying to correct problems running
the ERT with new configuraiton files.

\subsection{Terminology}

Anytime a ``set of integers'' is mentioned below it refers to a comma
separated list where each entry can be a single integer or a range of
integers specified by two integers separated by a hyphen.  For example:

\begin{itemize}

\vspace{-0.1in}
\item{``8'' - Just the integer 8.}

\vspace{-0.1in}
\item{``1,2,4'' - The integers 1, 2, and 4.}

\vspace{-0.1in}
\item{``1-8'' - The integers 1, 2, 3, 4, 5, 6, 7, and 8.}

\vspace{-0.1in}
\item{``1,2,6-8'' - The integers 1, 2, 6, 7, and 8.}

\end{itemize}

\subsection{Comment Lines}

Any line beginning with a ``\#'' character is treated as a comment and will not
be used by the ERT.  Currently, comments and the comment character are not
allowed mid-line.  If they appear mid-line, they will simply be considered
part of the line and used by the ERT.

\subsection{Storing Results}
To specify where results will be stored:

\begin{itemize}

\vspace{-0.1in}
\item{``ERT\_\+RESULTS'' - this specifies the directory where all results of
running the ERT will be kept.  If this it is a relative path, it will be
relative to the directory you are in when you run the ERT.  If this is an
absolute path, the same directory will be used every time this configuration
file is specified regardless of where you are when you run the ERT.}

\end{itemize}

\subsection{Specification Performance Numbers}
To specify roofline performance numbers based on the machine specification,
i.e., theoretical performance numbers:

\begin{itemize}

\vspace{-0.1in}
\item{``ERT\_\+SPEC\_\+GBYTES\_\+L\#'' - Give the theoretical maximum bandwidth in
GBytes/sec for the L\# cache on the machine being tested.}

\vspace{-0.1in}
\item{``ERT\_\+SPEC\_\+GBYTES\_\+DRAM'' - Give the theoretical maximum bandwidth in
GBytes/sec for the DRAM on the machine being tested.}

\vspace{-0.1in}
\item{``ERT\_\+SPEC\_\+GFLOPS'' - Give the theoretical maximum computation rate
in GFLOPs/sec for the machine being tested.}

\vspace{-0.1in}
\end{itemize}

These numbers are passed along as metadata and are available in the final JSON
roofline output generated by the ERT.

\subsection{Building the Code}
To specify which code to build and how to build it:

\begin{itemize}

\vspace{-0.1in}
\item{``ERT\_\+DRIVER'' - the driver code to use with the selected kernel code
(see below).  There is currently only one driver, ``driver1''.  The ERT looks
for ``ERT\_\+DRIVER.c'' in ``Drivers'' under the ERT installation directory.}

\vspace{-0.1in}
\item{``ERT\_\+KERNEL'' - the kernel code to use with the selected driver code
(see above).  There is currently only one kernel, ``kernel1''.  The ERT looks
for ``ERT\_\+KERNEL.c'' in ``Kernel'' under the ERT installation directory.}

\vspace{-0.1in}
\item{``ERT\_\+MPI'' - ``False'' to compile without MPI and ``True'' to compile
with MPI.}

\vspace{-0.1in}
\item{``ERT\_\+MPI\_\+CFLAGS'' - if ``ERT\_\+MPI'' is ``True'', compilation flags
specific to MPI that are needed or wanted.}

\vspace{-0.1in}
\item{``ERT\_\+MPI\_\+LDFLAGS'' - if ``ERT\_\+MPI'' is ``True'', linking/loading flags
specific to MPI that are needed or wanted.}

\vspace{-0.1in}
\item{``ERT\_\+OPENMP'' - ``False'' to compile without OpenMP and ``True'' to
compile with OpenMP.}

\vspace{-0.1in}
\item{``ERT\_\+OPENMP\_\+CFLAGS'' - if ``ERT\_\+OPENMP'' is ``True'',
compilation flags specific to OpenMP that are needed or wanted.}

\vspace{-0.1in}
\item{``ERT\_\+OPENMP\_\+LDFLAGS'' - if ``ERT\_\+OPENMP'' is ``True'',
linking/loading flags specific to OpenMP that are needed or wanted.}

\vspace{-0.1in}
\item{``ERT\_\+GPU'' - ``False'' to compile without the Cuda code and
``True'' to compile with Cuda code.}

\vspace{-0.1in}
\item{``ERT\_\+GPU\_\+CFLAGS'' - if ``ERT\_\+GPU'' is ``True'',
compilation flags specific to the Cuda code that are needed or wanted.}

\vspace{-0.1in}
\item{``ERT\_\+GPU\_\+LDFLAGS'' - if ``ERT\_\+GPU'' is ``True'',
linking/loading flags specific to the Cuda code that are needed or wanted.}

\vspace{-0.1in}
\item{``ERT\_\+FLOPS'' - A set of integers\footnote
{In this context, a ``set of integers'' is a comma separated list where each
entry can be a single integer or a range of integers specified by two integers
separated by a hyphen.  For example, ``8'' or ``1,2,4'' or ``1-8'' or
``1,2,6-8''}
specifying the FLOPS per computational element (in the current case an 8
byte, double precision floating point number).  This is specific to the
current driver/kernel pair and will probably be generalized in the future.}

\vspace{-0.1in}
\item{``ERT\_\+ALIGN'' - An integer specifies the alignment (in bits) of the
data being manipulated.}

\vspace{-0.1in}
\item{``ERT\_\+CC'' - The compiler to use to build the ERT test(s).}

\vspace{-0.1in}
\item{``ERT\_\+CFLAGS'' - The flags to use when compiling the code.}

\vspace{-0.1in}
\item{``ERT\_\+LD'' - The linker/loader to use to build the ERT code(s).}

\vspace{-0.1in}
\item{``ERT\_\+LDFLAGS'' - The flags to use when linking/loading the code.}

\vspace{-0.1in}
\item{``ERT\_\+LDLIBS'' - Explicit libraries to include when linking/loading
the code.}

\end{itemize}

\subsection{Running the Kernel}
To specify how to run the kernel once it is built:

\begin{itemize}

\vspace{-0.1in}
\item{``ERT\_\+RUN'' - This is a command that is used almost verbatim to run the
driver/kernel code that is built.  There are three strings that are replaced
wherever they appear in this command to customize the command:}
\begin{itemize}

\vspace{-0.1in}
\item{``ERT\_\+OPENMP\_\+THREADS'' - The number of OpenMP threads being run.}

\item{``ERT\_\+MPI\_\+PROCS'' - The number of MPI processes being run.}

\item{``ERT\_\+CODE'' - The current code being run.}

\end{itemize}

In addition, when ``ERT\_\+GPU'' is ``True'', two arguments are added to
``ERT\_\+RUN'' command:  The number of thread-blocks and the number of
threads-per-block.
\end{itemize}

\subsection{MPI, OpenMP, and Cuda/GPU}
To specify valid combinations of MPI processes, OpenMP threads, GPU
thread-blocks, and GPU thread-per-block:

\begin{itemize}

\vspace{-0.1in}
\item{``ERT\_\+PROCS\_\+THREADS'' - A set of integers that constrain valid
products of the number of MPI processes and the number of OpenMP threads.
For example, if this is `8' then only combinations of MPI processes and
OpenMP threads have to multiply to give 8, e.g., 2 and 4, 8 and 1, are run.}

\vspace{-0.1in}
\item{``ERT\_\+MPI\_\+PROCS'' - A set of integers specifying possible number of
MPI processes to run.  These are subject to constraints imposed above.}

\vspace{-0.1in}
\item{``ERT\_\+OPENMP\_\+THREADS'' - A set of integers specifying possible
number of OpenMP threads to run.  These are subject to constraints imposed
above.}

\vspace{-0.1in}
\item{``ERT\_\+BLOCKS\_\+THREADS'' - A set of integers that constrain valid
products of the number of thread-blocks and the number threads-per-block.
For example, if this is `28672' then only combinations of thread-blocks and
threads-per-block have to multiply to give 28672, e.g., 28 and 1024, 448 and
64, are run.}

\vspace{-0.1in}
\item{``ERT\_\+GPU\_\+BLOCKS'' - A set of integers specifying possible
number of thread-blocks to run.  These are subject to constraints imposed
above.}

\vspace{-0.1in}
\item{``ERT\_\+GPU\_\+THREADS'' - A set of integers specifying possible
number of threads-per-block to run.  These are subject to constraints imposed
above.}

\end{itemize}

\subsection{Miscellaneous}
Some miscellaneous parameters:

\begin{itemize}

\vspace{-0.1in}
\item{``ERT\_\+NUM\_\+EXPERIMENTS'' - The number of times to rerun the same code
with all the parameters set the same.  This is used to get a statistical
sample since performance can sometimes vary without changing the local
experiment.}

\vspace{-0.1in}
\item{``ERT\_\+MEMORY\_\+MAX'' - The maximum number of bytes to allocate/use for
the entire run.  These are divided up across MPI processes and OpenMP
threads.}

\vspace{-0.1in}
\item{``ERT\_\+WORKING\_\+SET\_\+MIN'' - The minimum size (in 8-byte, double
precision floating point numbers) for an individual process/thread working
set.}

\vspace{-0.1in}
\item{``ERT\_\+TRIALS\_\+MIN'' - The minimum number of times to repeatly run
the loop over the working set.  This will be the maximum number of trials when
the working set is the largest and it will scale up so the product of the
maximum number of trails and the working set size are approximately constant.}

\vspace{-0.1in}
\item{``ERT\_\+GNUPLOT'' - How GNUplot is invoked.}

\end{itemize}

max_stars_repo_name: uo-cdux/ert-mirror

max_issues_repo_name: uo-cdux/ert-mirror

max_forks_repo_name: uo-cdux/ert-mirror


text:
\chapter{Extending GDB for \uCPP}

\section{Introduction}
A sequential program has a single call stack. A debugger knows about this call
stack and provides commands to walk up/down the call frames to examine the
values of local variables, as well as global variables. A concurrent program
has multiple call stacks (for coroutines/tasks), so a debugger must be extended
to locate these stacks for examination, similar to a sequential stack.  For
example, when a concurrent program deadlocks, looking at the task's call stack
can locate the resource and the blocking cycle that resulted in the
deadlock. Hence, it is very useful to display the call stack of each task to
know where it is executing and what values it is currently computing. Because
each programming language's concurrency is different, GDB has to be
specifically extended for \uCPP.

\section{Design Constraints}
As mentioned in Chapter \ref{GDB}, there are several ways to extend GDB. However,
there are a few design constraints on the selected mechanism. All the functions
implemented should maintain similar functionality to existing GDB commands. In addition to functional
requirements, usability and flexibility are requirements for this
project. These final requirements enable developers to be productive quickly
and do more with the extensions. The extensions created for \uCPPS are simple
to use and versatile.

The following new GDB command are all implemented through the Python API for
GDB.  Python is a scripting language with built-in data structures and
functions that enables the development of more complex operations and saves
time on development.

\section{\uCPPS source-code example}
Listing \ref{uCPP-src-code} shows a \uCPPS program that implicitly creates two
clusters, system and user, which implicitly have a processor (kernel thread)
and processor task. The program explicitly creates three additional processors
and ten tasks on the user cluster.

\begin{figure}
\begin{lstlisting}[style=C++, caption={\uCPPS source code used for GDB commands},
label={uCPP-src-code}, basicstyle=\small]
_Task T {
    std::string name;

    void a(int param) {
        if ( param != 0 ) a( param - 1 );
        int x = 3;
        std::string y = "example";
    }
    void main() {
        a(3);
    }
  public:
    T(const int tid) {
        name = "T" + std::to_string(tid);
        setName(tid.c_str());
    }
};

int main() {
    uProcessor procs[3];	// extra processors
    const int numTasks = 10;
    T* tasks[numTasks];		// extra tasks

    // allocate tasks with different names
    for (int id = 0; id < numTasks; id += 1) {
        tasks[i] = new T(id);
    }
    // deallocate tasks
    for (int id = 0; id < numTasks; id += 1) {
        delete tasks[i];
    }
}
\end{lstlisting}
\end{figure}

\section{Existing User-defined GDB Commands}
Listing \ref{uCPP-callstack} shows the GDB output at the base case of the
recursion for one of the tasks created in the \uCPPS program in listing
\ref{uCPP-src-code}.  The task is stopped at line 8. The backtrace shows the
three calls to function \verb|a|, started in the task's \verb|main|. The top
two frames are administrative frames from \uCPP. The values of the argument and
local variables are printed.

\begin{figure}
\begin{lstlisting}[caption={Call stack of function \texttt{a} in the \uCPPS
program from listing \ref{uCPP-src-code}}, label={uCPP-callstack}, basicstyle=\small\tt]
(gdb) backtrace
#0  T::a (this=0x907880, param=0) at test.cc:8
#1  0x000000000041d207 in T::a(this=0x907880, param=1) at test.cc:5
#2  0x000000000041d207 in T::a(this=0x907880, param=2) at test.cc:5
#3  0x000000000041d207 in T::a(this=0x907880, param=3) at test.cc:5
#4  0x000000000041d2c0 in T::main(this=0x907880) at test.cc:10
#5  0x000000000042633f in ...::invokeTask(uBaseTask&) at ...
#6  0x0000000000000000 in ?? ()
(gdb) info args
this = 0x907880
param = 0
(gdb) info locals
x = 3
y = "example"
\end{lstlisting}
\end{figure}

\subsection{Listing all clusters in a \uCPPS program}
Listing \ref{clusters-command} shows the new command \verb|clusters| to list
all program clusters along with their associated address.  The output shows the
two \uCPPS implicitly created clusters.
\begin{figure}
\begin{lstlisting}[caption={clusters command}, label={clusters-command}, basicstyle=\small\tt]
(gdb) clusters
        Name            Address
    systemCluster       0x655280
    userCluster         0x7c4280
\end{lstlisting}
\end{figure}

\subsection{Listing all processors in a cluster}
Listing \ref{cluster-procs} shows the new command \verb|processors|, which
requires a cluster argument to show all the processors in that cluster. In
particular, this example shows that there are four processors in the
\verb|userCluster|, with their associated address, PID, preemption and spin.
\begin{figure}
\begin{lstlisting}[caption={processors command}, label={cluster-procs}, basicstyle=\small\tt]
(gdb) processors userCluster
    Address    PID         Preemption      Spin
    0x7c6bf0    8396928     10              1000
    0x8c3b60    9453696     10              1000
    0x8c3d20    9977984     10              1000
    0x8c3ee0    10506368    10              1000
\end{lstlisting}
\end{figure}

\subsection{Listing all tasks in all clusters}
Listing \ref{tasks} shows the new command \verb|task| without an argument to
list all the tasks in a \uCPPS program at this point in the execution snapshot.
\begin{figure}
\begin{lstlisting}[caption={task command for displaying all tasks for all
clusters}, label={tasks}, basicstyle=\small\tt]
(gdb) task
        Cluster Name        Address
        systemCluster       0x655280
    ID      Task Name            Address         State
    0       uProcessorTask      0x6c39b0        uBaseTask::Blocked
    1       uSystemTask         0x783ef0        uBaseTask::Blocked
        Cluster Name        Address
        userCluster         0x7c4280
    ID      Task Name            Address         State
    0       uProcessorTask      0x806e70        uBaseTask::Blocked
    ...
    6       T0                  0xa49830        uBaseTask::Ready
    7       T1                  0xa89bb0        uBaseTask::Running
    8       T2                  0xac9f30        uBaseTask::Ready
    ...
    15      T9                  0xc8b7b0        uBaseTask::Ready
\end{lstlisting}
\end{figure}

\subsection{Listing all tasks in a cluster}
Listing \ref{cluster-tasks} shows the new command \verb|task| with a cluster
argument to list only the names of the tasks on that cluster.  In this version
of the command \verb|task|, the associated address for each task and its state
is displayed.
\begin{figure}
\begin{lstlisting}[caption={task command for displaying all tasks in a cluster}, label={cluster-tasks}, basicstyle=\small\tt]
(gdb) task systemCluster
    ID          Name            Address         State
    0       uProcessorTask      0x654200        uBaseTask::Blocked
    1       uSystemTask         0x7c3280        uBaseTask::Blocked
\end{lstlisting}
\end{figure}

\section{Changing Stacks}
The next extension for displaying information is writing new commands that allow
stepping from one \uCPPS task to another. Two switching approaches are
provided: structured and free form. The structured form remembers the task tour in a
LIFO way, while the free form does not remember the task tour. This semantics means
at least two commands are needed for the structured form and the free form.
Taking the idea of overloading in programming languages, the structured
commands for switching from the current task to a different task overload the
command \verb|task| with two arguments, and
for switching back to the previous task is a new command \verb|poptask|.

\subsection{Task Switching}
Both overloaded \verb|task| or \verb|poptask| commands allow two ways to
designate a task: its absolute address or relative id within a cluster. For
instance, to switch from any task to task \verb|T1| seen in listing
\ref{tasks}, the first command in listing \ref{task-addr-option} uses an
absolute address (0xa49830) and the second command uses a relative id (6) with
respect to userCluster. Core functionality of these approaches is essentially the same.
\begin{lstlisting}[caption={task command option}, label={task-addr-option}, basicstyle=\small\tt]
task 0xa49830
task userCluster 6
\end{lstlisting}

\subsection{Switching Implementation}
To implement the task tour, it is necessary to store the context information
for every context switching. This requirement means the \verb|task| command
needs to store this information every time it is
invoked.

Figure \ref{machine-context} shows a task points to a structure containing a
\verb|uContext_t| data structure, storing the stack and frame pointer, and the
stack pointer. Listing \ref{pushtask-code} shows these pointers are copied into
an instance of the Python tuple \verb|StackInfo| for every level of task
switching. This tuple also stores information about the program counter that is
calculated from the address of the \verb|uSwitch| Assembly function because a
task always stops in \verb|uSwitch| when its state is blocked.  Similarly,
switching commands retrieve this context information but from the task that
a user wants to switch to, and sets the equivalent registers to the new values.

\begin{figure}[h]
    \centering
    \includegraphics[width=8cm]{uContext_stack}
    \caption{Machine context (uMachContext) for each task}
    \label{machine-context}
\end{figure}

For free-form touring using the \verb|task| command, the value of the stack
pointer is copied to/from the \verb|rsp| register, the value of the frame
pointer is copied to/from the \verb|rbp| register, and finally the value of the
program counter is copied to/from the \verb|pc| register.

\begin{figure}
\begin{lstlisting}[style=Python, caption={Abridged \texttt{push\_task} source
code}, label={pushtask-code}, basicstyle=\small\tt]
# get GDB type of uContext_t*
uContext_t_ptr_type =
   gdb.lookup_type('UPP::uMachContext::uContext_t').pointer

# retrieve the context object from a task
# and cast it to the type uContext_t*
task_context = task['context'].cast(uContext_t_ptr_type)

# the offset where sp would be starting from uSwitch function
sp_address_offset = 48
# lookup the value of stack pointer (sp), frame pointer (fp),
# program counter (pc)
xsp = task_context['SP'] + sp_address_offset
xfp = task_context['FP']
if not gdb.lookup_symbol('uSwitch'):
    print('uSwitch symbol is not available')
    return

# This value is calculated here because we always here when
# the task is in blocked state
xpc = get_addr(gdb.parse_and_eval('uSwitch').address + 28, 'PC')
# must switch back to frame-0 to set 'pc' register with
# the value of xpc
gdb.execute('select-frame 0')

# retrieve register values and push sp, fp, pc into
# a global stack
global STACK
sp = gdb.parse_and_eval('$sp')
fp = gdb.parse_and_eval('$fp')
pc = gdb.parse_and_eval('$pc')
stack_info = StackInfo(sp = sp, fp = fp, pc = pc)
STACK.append(stack_info)

# update registers for a new task
gdb.execute('set $rsp={}'.format(xsp))
gdb.execute('set $rbp={}'.format(xfp))
gdb.execute('set $pc={}'.format(xpc))
\end{lstlisting}
\end{figure}

For structured touring, the implementation manages a stack of toured tasks.
The \verb|pushtask| command sets the appropriate registers to move to the
specified task, and remembers the previous task registers.  The \verb|poptask|
restores the previous stack register. Popping an empty stack prints a warning.
In particular, listing \ref{poptask} shows the backtrace after touring to task
6 of \verb|userCluster|, whose address can be seen on line 23. However, after
calling the command \verb|poptask|, the new backtrace shows the address of the
previous task, which is \verb|0xb0a2b0| and seen on lines 11 and 30.

\begin{figure}
\begin{lstlisting}[numbers=left, caption={\texttt{poptask} command}, label={poptask}, basicstyle=\footnotesize\tt]
(gdb) info threads
  Id   Target Id         Frame
* 1    Thread 0x7ffff7fe9780 (LWP 704) "utils" 0x000000000042f2ef in
        uCluster::processorPause (this=0x73f7a8) at
        /usr/local/u++-7.0.0/src/kernel/uCluster.cc:120
  2    Thread 0x802080 (LWP 708) "utils" T::a (this=0xa49830) at utils.cpp:7
  3    Thread 0x904080 (LWP 709) "utils" T::a (this=0xbcad30) at utils.cpp:7
  4    Thread 0x984080 (LWP 710) "utils" T::a (this=0xb8a9b0) at utils.cpp:7
  5    Thread 0xa05080 (LWP 711) "utils" UPP::uSigHandlerModule::sigAlrmHandler
        (sig=10, sfp=0xb864b0, cxt=0xb86380) at
        /usr/local/u++-7.0.0/src/kernel/uSignal.cc:218
(gdb) thread 2
(gdb) backtrace
<@\textcolor{ForestGreen}{\#0  T::a (this=0xb0a2b0, param=5) at utils.cpp:8}@>
#1  0x000000000041cb4c in T::main (this=0xb0a2b0) at
utils.cpp:11
#2  0x0000000000425bbc in UPP::uMachContext::invokeTask (This=...)
    at /usr/local/u++-7.0.0/src/kernel/uMachContext.cc:140
#3  0x0000000000000000 in ?? ()
(gdb) task userCluster 6
(gdb) backtrace
#0  uSwitch () at /usr/local/u++-7.0.0/src/kernel/uSwitch-x86_64.S:64
#1  0x00000000004288ec in uBaseCoroutine::taskCxtSw (this=0x8c3b78)
    at /usr/local/u++-7.0.0/src/kernel/uBaseCoroutine.cc:146
...
<@\textcolor{ForestGreen}{\#7  T::a (this=0xa49830) at utils.cpp:8}@>
#8  0x000000000041cb43 in T::main (this=0xa49830) at utils.cpp:11
#9  0x0000000000425bbc in UPP::uMachContext::invokeTask (This=...)
    at /usr/local/u++-7.0.0/src/kernel/uMachContext.cc:140
#10 0x0000000000000000 in ?? ()
(gdb) poptask
(gdb) backtrace
<@\textcolor{ForestGreen}{\#0  T::a (this=0xb0a2b0, param=5) at utils.cpp:8}@>
#1  0x000000000041cb4c in T::main (this=0xb0a2b0) at utils.cpp:11
#2  0x0000000000425bbc in UPP::uMachContext::invokeTask (This=...)
    at /usr/local/u++-7.0.0/src/kernel/uMachContext.cc:140
#3  0x0000000000000000 in ?? ()
\end{lstlisting}
\end{figure}

\subsection{Continuing Implementation}
When a breakpoint or error is encountered, all concurrent execution stops.  The
state of the program can now be examined and changed; after which the program
can be continued. Continuation must always occur from the top of the stack
(current call) for each task, and at the specific task where GDB stopped
execution.

However, during a task tour, the new GDB commands change the notion of the task
where execution stopped to make it possible to walk other stacks.  Hence, it is
a requirement for continuation that the task walk always return to frame-0 of
the original stopped task before any program continuation \cite{Reference11}.

% For every new function call, a new stack frame is created and the values of all
% the registers are changed for that frame. Therefore, in order to see the true
% value of hardware registers, innermost frame that is frame-0 must be selected
% \cite{Reference11}. However, it is possible to not be in frame-0, so prior to
% setting these values, the command must switch back to the innermost (currently
% executing) frame first.

To provide for this requirement, the original stop task is implicitly
remembered, and there is a new \verb|reset| command that \emph{must} be
explicitly executed before any continuation to restore the locate state.  To
prevent errors from forgetting to call the \verb|reset| command, additional
hooks are added to the existing built-in GDB continuation commands (see
\ref{continue-cmds}) to implicitly call \verb|reset|. The list of these
commands results from GDB documentation \cite{Reference15} and similar work
done for KOS \cite{Reference14}.

% These hooks call a new command called \verb|reset| prior to executing the
% command to enable continuation of a program to ensure that the program's
% context is automatically switched back to the context of the task that
% initiates the first context switch. The \verb|reset| command behaves as same as
% the command \verb|poptask|, however, it goes back directly to where the task is
% when the program last stops, which is the first task in the task tour.

\begin{figure}
\begin{lstlisting}[caption={Built-in GDB commands that allow continuation of a
program}, label={continue-cmds}, basicstyle=\small\tt]
continue
next
nexti
step
stepi
finish
advance
jump
signal
until
reverse-next
reverse-step
reverse-stepi
reverse-continue
reverse-finish
\end{lstlisting}
\end{figure}

\section{Result}
The current implementation successfully allows users to display a snapshot of
\uCPPS execution with respect to clusters, processors, and tasks. With this
information it is possible to tour the call stacks of the tasks to see
execution locations and data values. Additionally, users are allowed to
continue the execution where the program last pauses assuming that the program
has not crashed. The continuation of execution is done by automatically
reversing the task walk from any existing GDB commands such as \verb|continue|,
or a user can manually reverse the task walk using the command \verb|poptask|
and then continue.

ext: tex


text:
\section{Plasticity}
\label{sec:plasticity}

Plasticity or more specific \emph{neuroplasticity}, when it comes to brains, is the ability of the brain to change its neuronal structure depending on experience. It can be separated in \emph{functional plasticity} and \emph{structural plasticity}.

\begin{itemize}
\item Functional plasticity: Changes in synaptic transmission characteristics due to changes in amount of neurotransmitter or receptors
\item Structural plasticity: Changes in structure of the neuron, like growing or shrinking of axons, dendrites, synapses or even whole neurons
\end{itemize}

Furthermore, \emph{synaptic plasticity} describes all processes related to synapses, like changes in transmission characteristics and growing or shrinking of synapses. Regarding synaptic plasticity one can distinguish between \emph{pre-synaptic plasticity} and \emph{post-synaptic plasticity}. Pre-synaptic plasticity means, that the amount of released neurotransmitters or the time for readmission of neurotransmitters changes. Post-synaptic plasticity changes the sensibility for a given amount of neurotransmitters. For example the amount of receptors (ion channels) can be adapted. Furthermore, plasticity can vary in duration. \emph{Short-term plasticity} changes the transmission characteristic of the synapses in a scale of milliseconds to minutes, while \emph{long-term plasticity} has an effect for hours, or even years. Both can be separated in so called depression --- \acf{std} and \acf{ltd} --- which means a decrease in connectivity, and potentiation --- \acf{stp} and \acf{ltp} --- which means an increase in connectivity between the synapses.

Second messengers (see above in section \ref{sec:synapse}) play an important role in synaptic plasticity \parencite{skrebitskiui1999synaptic}. Metabotropic receptors can not only start mechanisms to open ion channels, they can also start processes to change the behavior of receptors, channels or neurotransmitters.

In artificial neural networks the synaptic transmission characteristic can be modeled as a weight $w_{ij}$ between neuron $i$ and neuron $j$, as already used in equation \eqref{eq:activation}. If there is no connection between two neurons, it holds that $w_{ij} = 0$. Functional plasticity corresponds to a change in $w_{ij} \in (0,1]$, indicated as $\Delta w_{ij}$, whereby structural plasticity sets the weight to zero or non-zero.

\subsection{Hebb's rule}

A rule that simplifies a functional plasticity mechanism is the \emph{Hebb rule}. Biologically, synapses strengthen their connection if the firing of neuron $j$ and the firing of neuron $i$ is close together. In that case the weight $w_{ij}$ will increase.

\begin{figure}[!b]
	\centering
	\includegraphics[width=0.6\textwidth]{neurons_plasticity/stdp.png}
	\caption[Spike-timing-dependent plasticity]{Spike-timing-dependent plasticity \parencite[figure 1]{dan2006spike}. The closer the post-synaptic neuron spikes before the pre-synaptic neuron spikes, the higher will be the \emph{decrease} in weight and vice versa.}
	\label{fig:stdp}
\end{figure}

For artificial neural networks holds therefore

\begin{equation}
\Delta w_{ij} = \eta \cdot z_j \cdot a_i,
\end{equation}

where $\eta$ is a learning rate. A high learning rate results in fast but inaccurate convergence, while a low learning rate results in slow but accurate convergence.

\subsection{Spike-timing-dependent plasticity}

The \acfi{stdp} is a causal type of Hebb's rule, depending on temporal correlations between the spikes of pre-synaptic and post-synaptic neurons \parencite{sjostrom2010spike}. Spikes arriving repeatedly at the pre-synapses, some time before or after an action potential at the post-synaptic neuron, lead to long-term plasticity. In case that spikes arrive shortly before an action potential of the post-synaptic neuron, it leads to \acl{ltp}. In case that spikes arrive shortly after an action potential of the post-synaptic neuron, it leads to \acl{ltd}. The time scale where plasticity occurs, varies between different types of synapses. In most cases it is of the order of milliseconds \parencite{dan2006spike}. An illustration of the \acs{stdp} is shown in figure \ref{fig:stdp}.

\begin{wrapfigure}{R}{0.45\textwidth}
	\centering
	\includegraphics[width=0.45\textwidth]{neurons_plasticity/learning-window.png}
	\caption[Learning windows]{\ac{stdp} learning windows \parencite[figure 2]{abbott2000synaptic}.}
    \label{fig:learning-windows}
    \vspace{-10pt}
\end{wrapfigure}

A mathematically formulation of the \ac{stdp} was done by \textcite{sjostrom2010spike} and more specific by \textcite{kempter1999hebbian}. Consider a post-synaptic neuron with activity $x_i(t)$ and a pre-synaptic neuron with activity $x_j(t)$. Assume many spikes at the post-synaptic neuron, counted by $l \in \{ 1,...,n_l \}$, where $n_l$ is the total number of spikes. $t_i^l$ indicates the time for every spike at neuron $i$ with $x_i(t_i^l) = 1$. Furthermore, the number of spikes of the pre-synaptic neurons is counted by $k \in \{ 1,...,n_k \}$, where $n_k$ is the total number of spikes. $t_j^k$ indicates the points in time at which the pre-synaptic neuron is firing with $x_j(t_j^k) = 1$.

The \emph{learning window} is a function $f(\Delta t)$, where $\Delta t = t_j^k - t_i^l$. It can be defined in various ways, some are shown in figure \ref{fig:learning-windows}. Using the learning window, the change of the weight can be defined as

\begin{equation}
\label{eq:stdp-orig}
\Delta w_{ij} = \sum_{k=1}^{n_k} \sum_{l=1}^{n_l} f(\Delta t) = \sum_{k=1}^{n_k} \sum_{l=1}^{n_l} f(t_j^k - t_i^l).
\end{equation}

$f(\Delta t)$ is often chosen as an exponential function \parencite{sjostrom2010spike}. It could be shown, that an exponential learning window is able to model experimental findings \parencite{zhang1998critical}. It can be defined as

\begin{equation}
f(\Delta t) = \begin{cases}
\;A_+ \exp(-\Delta t / \tau_+), & x > 0\\
\;-A_- \exp(\Delta t / \tau_-), & x < 0
\end{cases},
\end{equation}

where $A_+$ and $A_-$ are parameters to choose. $\tau_+$ and $\tau_-$ are time constants and often chosen as $\tau_+ = \tau_- = 10\,ms$.

If, for the exponential learning window, $t_j^k$ and $t_i^l$ are close to each other most of the time, which means that pre- and post-synaptic neurons spiking in succession, there will be a change in the weight between those neurons, depending on the sign. The more often those neurons fire close to each other, the higher will be the sum and the change of the weight. The principle of the exponential learning window is shown in figure \ref{fig:learning-windows}a.

For simulations of neural networks which are focused on more high level network based research, \ac{stdp} is often modeled more simple. One simplified version, where the weight only depends on the spiking between two discrete time steps, will be introduced in section \ref{sec:sorn}.

\subsection{Homeostatic plasticity}

A network just processing Hebb's rule would soon end in a destabilized state. If \acs{ltp} becomes dominant, the synapses in the network will increase unbounded. If \acs{ltd} becomes dominant, the synapses will decrease to zero at some point. Hence, the synaptic activity in the network is diverging and therefore needs constraints \parencite{miller1994role, abbott2000synaptic}. In figure \ref{fig:homeo} the stability problem is shown for a simplified example. A small feed forward network with $5$ layers is only able to propagate an input signal if the gain is close to $1$. If it is higher than $1$, the activity `explodes' and the sinusoidal input signal cannot be represented in the last layer. Otherwise, if the gain is lower than $1$, the signal `dies out' and is not able to reach the last layer. The same holds for recurrent networks, where the problem of propagation happens in time, compared to propagation in space regarding the layers in a feed forward structure \parencite{turrigiano2004homeostatic}. In machine learning this problem is known as the \emph{exploding and vanishing gradient problem} \parencite{pascanu2013difficulty}.

\acs{stdp} can allow some more stability than Hebb's rule alone \parencite{abbott2000synaptic}, depending on the learning window, which leads to more precise \acs{ltp} and \acs{ltd}. The ability, how much \acs{stdp} is able to stabilize the network, depends on the parameters of $f(\Delta t)$.

However, biologically, one can observe a whole set of different constraints which appear in natural neural networks in order to maintain a specific target firing rate. They are reviewed for example by \textcite{turrigiano2004homeostatic}, \textcite{turrigiano2008self}, \textcite{pozo2010unraveling} and \textcite{turrigiano2012homeostatic} and summarized at Scholarpedia by \textcite{williams2013homeostatic}. The main goal of those processes is to keep a specific activity in the network. It acts as a set point or baseline for the overall activity. If some areas in the network are going up in activity for some time, the networks reacts with a re-normalization of the whole activity, where the relative connections between the neurons stay the same. It is important to note that those mechanisms operate in a decentralized way, where every neuron adjusts for itself. There cannot be any centralized power, controlling the overall activity. This kind of plasticity is called \emph{homeostatic plasticity}. There are several different complex mechanisms of homeostatic plasticity and some of them are still not fully understood. However, those mechanisms can be classified by:

\begin{figure}[t]
	\centering
	\includegraphics[width=0.6\textwidth]{neurons_plasticity/homeostatic-problem.png}
	\caption[Instability of networks]{Instability of networks for a simplified case \parencite[figure 1]{turrigiano2004homeostatic}. \textbf{a)} A small feed forward network with $5$ layers, the activity of the dark neuron is shown in b and c. \textbf{b)} If the gain is higher than $1$, the activity `explodes' and the sinusoidal input signal cannot be represented in the last layer. \textbf{c)} If the gain is lower than $1$, the signal `dies out' and is not able to reach the last layer.}
	\label{fig:homeo}
\end{figure}

\begin{itemize}
\item Type: Intrinsic (affecting the neuron) or synaptic (pre-synaptic and post-synaptic)
\item Spatial: More global (all synapses connected to a neuron) and more local or quasi-local (single synapse or small groups of synapses connected to a neuron)
\item Temporal: Fast (seconds to hours) and slow (hours to days)
\end{itemize}

\subsection{Synaptic normalization}
\label{sec:synaptic-scaling}

One important class of synaptic homeostatic plasticity is \emph{synaptic scaling}, where incoming synaptic connections to a neuron are normalized to a target weight. While adjusting the weights, their relative strengths are maintained. \textcite{turrigiano2008self} summarizes that neurons use calcium-dependent sensors to detect changes in their own firing rates. Depending on those changes the number of receptors at the synapses will be regulated. Note that the more local the mechanisms are and the fewer synapses are involved, the more the synaptic scaling mechanism will interact with \ac{ltd} and \ac{ltp} mechanisms, like \ac{stdp}.

While the normalizing process is highly complex in natural networks, in artificial systems it is relatively simple to perform. For example, in a simplified more global rule, every weight connected to a neuron $i$ is just normalized with all incoming weights from neurons $j$, using

\begin{equation}
\label{eq:sn}
w_{ij}(t) \leftarrow \frac{w_{ij}(t)}{\sum_{j=1}^N w_{ij}(t)},
\end{equation}

\nomenclature{$w_{ij}$}{Weight between neuron $i$ and neuron $j$}

where $w_{ij}(t)$ is the weight between neuron $i$ and $j$ at time $t$. Due to the mathematical process of normalization, it is also called \emph{synaptic normalization}.

\subsection{Intrinsic plasticity}
\label{sec:ip}

\ACFI{ip} is a process, which is local and intrinsic, since it only affects the conduction inside of a single neuron. In \acs{ip} no synapse and therefore, when modeling artificial networks, no weight is affected. There are several different types of \acl{ip} and many of them are not fully understood \parencite{cudmore2008intrinsic}. In principal, the neuron regulates biophysical properties of ion channels in the membrane. \textcite{awatramani2005modulation} showed that a small depolarization of about $10\,ms$ (resulting in a potential between $-60\,mV$ and $-65\,mV$), induced by a change of the extracellular concentration of calcium ions, enhances the probability of transmitter release up to $2$-fold.

\begin{wrapfigure}{R}{0.45\textwidth}
	\centering
	\includegraphics[width=0.45\textwidth]{neurons_plasticity/ip}
    \caption[Intrinsic plasticity and spontaneous activity]{Simple implementation of intrinsic plasticity. If the neuron is not spiking (blue dot), the threshold is decreased in the next step of time. If the neuron is spiking (red dot), the threshold is increased in the next step of time. If the threshold moves below zero, the neuron fires spontaneously.}
    \vspace{10pt}
    \label{fig:intrinsic-plasticity}
\end{wrapfigure}

A simple implementation in artificial neural networks affects the threshold for conducting the action potential inside of the neuron. If the neuron was active, the threshold will increase for the next step. If the neuron was not active, the threshold will decrease, to make the neuron more sensible. Note that if the threshold of a neuron becomes negative, the neuron fires spontaneously. The effect is shown in figure \ref{fig:intrinsic-plasticity}. The rule can be formalized with

\begin{equation}
\label{eq:ip}
T_i(t+1) = T_i(t) + \eta_\IP \left( x_i(t) - H_\IP \right),
\end{equation}

\nomenclature{$\eta_\IP$}{Learning rate for intrinsic plasticity}
\nomenclature{$T_i(t)$}{Threshold of neuron $i$ at time $t$}

where $\eta_\IP$ is the learning rate and $x_i(t) \in \{0,1\}$ the activity of neuron $i$ at time $t$. $H_\IP$ is the \emph{target rate}. All neurons will in average be active with a rate of $H_\IP$. The choice of $H_\IP$ is not trivial, since not all choices of the target rate will necessarily lead to a network with good performance \parencite{lazar2009sorn}.

max_stars_repo_name: sagacitysite/mastatistik

max_issues_repo_name: sagacitysite/mastatistik

max_forks_repo_name: sagacitysite/mastatistik


text:
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{\label{sec:Admin-Intro}Introduction}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

This is the HTCondor Administrator's Manual.
Its purpose is to aid in
the installation and administration of an HTCondor pool.  For help on
using HTCondor, see the HTCondor User's Manual.

An HTCondor pool
\index{HTCondor!pool}
\index{pool of machines}
is comprised of a single machine which serves as the
\Term{central manager},
\index{central manager}
and an arbitrary number of other machines that
have joined the pool.  Conceptually, the pool is a collection of
resources (machines) and resource requests (jobs).  The role of HTCondor
is to match waiting requests with available resources.  Every part of
HTCondor sends periodic updates to the central manager, the centralized
repository of information about the state of the pool.  Periodically,
the central manager assesses the current state of the pool and tries
to match pending requests with the appropriate resources.

Each resource has an owner,
\index{resource!owner}
\index{machine!owner}
the one who sets the policy for the use of the machine.  This
person has absolute power over the use of the machine,
and HTCondor goes out
of its way to minimize the impact on this owner caused by HTCondor.  It
is up to the resource owner to define a policy for when HTCondor
requests will
serviced and when they will be denied.

Each resource request has an owner as well: the
user who submitted the job.  These people want HTCondor to provide as
many CPU cycles as possible for their work.  Often the interests of
the resource owners are in conflict with the interests of the resource
requesters.
The job of the HTCondor administrator is to configure the HTCondor pool to
find the happy medium that keeps both resource owners and users of
resources satisfied.  The purpose of this manual is to relate
the mechanisms that HTCondor provides to enable the administrator to find
this happy medium.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\label{sec:Machine-Roles}The Different Roles a Machine Can Play}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Every machine in an HTCondor pool can serve a variety of roles.  Most
machines serve more than one role simultaneously.  Certain roles can
only be performed by a single machine in the pool.  The following list
describes what these roles are and what resources are required on the
machine that is providing that service:

\begin{description}

\index{central manager}
\index{machine!central manager}
\item[Central Manager] There can be only one central manager for the pool.
This machine is the collector of information, and the negotiator
between resources and resource requests.  These two halves of the
central manager's responsibility are performed by separate daemons, so
it would be possible to have different machines providing those two
services.  However, normally they both live on the same machine.  This
machine plays a very important part in the HTCondor pool and should be
reliable.  If this machine crashes, no further matchmaking can be
performed within the HTCondor system,
although all current matches remain in effect until they are broken
by either party involved in the match.
Therefore, choose for central manager
a machine that is likely to be up and running all the time,
or at least one that will be rebooted quickly if something goes wrong.
The central manager will
ideally have a good network connection to all the
machines in the pool, since these pool machines all send updates over
the network to the central manager.

% editted up to this point

\index{execute machine}
\index{machine!execute}
\item[Execute] Any machine in the pool,
including the central manager,
can be configured as to whether or not it should execute HTCondor jobs.
Obviously, some of the machines will have to serve this
function, or the pool will not be useful.
Being an execute machine does not require lots of resources.
About the only resource that might matter is disk space.
%, since if the remote job dumps core, that
%file is first dumped to the local disk of the execute machine before
%being sent back to the submit machine for the owner of the job.
%However, if there isn't much disk space, HTCondor will simply limit the
%size of the core file that a remote job will drop.
In general the
more resources a machine has in terms of swap space, memory, number of CPUs,
the larger variety of resource requests it can serve.
%However, if
%there are requests that don't require many resources, any machine
%in your pool could serve them.

\index{submit machine}
\index{machine!submit}
\item[Submit] Any machine in the pool,
including the central manager,
can be configured as to whether or not it should allow HTCondor
jobs to be submitted.
The resource requirements for a submit machine
are actually much greater than the resource requirements for an
execute machine.  First, every submitted job that is
currently running on a remote machine runs a process on
the submit machine.
As a result, lots of running jobs
will need a fair amount of swap space and/or real memory.
In addition,
the checkpoint files from standard universe jobs are stored on
the local disk of the submit machine.
If these jobs have a large
memory image and there are a lot of them,
the submit machine will need a lot of disk space to hold these files.
This disk space requirement can be
somewhat alleviated by using a checkpoint server,
however the binaries of the jobs are still stored on the submit machine.

\index{checkpoint server}
\index{machine!checkpoint server}
\item[Checkpoint Server]  Machines in the pool can be configured to act as
checkpoint servers.
This is optional, and is not part of the
standard HTCondor binary distribution.  A checkpoint server is a
machine that stores checkpoint files for sets of jobs.
A machine with this role should have lots of disk space
and a good network connection to the rest of the pool, as the traffic
can be quite heavy.

\end{description}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\label{sec:HTCondor-Daemons}The HTCondor Daemons}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\index{HTCondor daemon!descriptions}
\index{daemon!descriptions}

The following list describes all the daemons and programs that could
be started under HTCondor and what they do:

\begin{description}

\index{HTCondor daemon!condor\_master@\Condor{master}}
\index{daemon!condor\_master@\Condor{master}}
\index{condor\_master daemon}
\item[\Condor{master}] This daemon
is responsible for keeping all the
rest of the HTCondor daemons running on each machine in the pool.  It
spawns the other daemons, and it periodically checks to see if there are
new binaries installed for any of them.  If there are, the
\Condor{master} daemon will
restart the affected daemons.  In addition, if any daemon crashes, the
\Condor{master} will send e-mail to the HTCondor administrator of the pool and
restart the daemon.  The \Condor{master} also supports various
administrative commands that enable the administrator to start,
stop or reconfigure daemons remotely.
The \Condor{master} will run on every machine in
the pool, regardless of the functions that each machine is performing.

\index{HTCondor daemon!condor\_startd@\Condor{startd}}
\index{daemon!condor\_startd@\Condor{startd}}
\index{condor\_startd daemon}
\item[\Condor{startd}] This daemon
represents a given resource to the HTCondor pool,
as a machine capable of running jobs.
It advertises certain attributes about machine that are used to
match it with pending resource requests.
The \Condor{startd} will run on any
machine in the pool that is to be able to execute jobs.
It is responsible for enforcing the policy that the resource owner configures,
which determines under what conditions jobs will be started,
suspended, resumed, vacated, or killed.
When the \Condor{startd} is ready to
execute an HTCondor job, it spawns the \Condor{starter}.

\index{HTCondor daemon!condor\_starter@\Condor{starter}}
\index{daemon!condor\_starter@\Condor{starter}}
\index{condor\_starter daemon}
\item[\Condor{starter}] This daemon
is the entity that actually
spawns the HTCondor job on a given machine.  It sets up the
execution environment and monitors the job once it is running.  When a
job completes, the \Condor{starter} notices this, sends back any status
information to the submitting machine, and exits.

\index{HTCondor daemon!condor\_schedd@\Condor{schedd}}
\index{daemon!condor\_schedd@\Condor{schedd}}
\index{condor\_schedd daemon}
\item[\Condor{schedd}] This daemon
represents resource requests to
the HTCondor pool.  Any machine that is to be a submit machine
needs to have a \Condor{schedd} running.
When users submit jobs,
the jobs go to the \Condor{schedd}, where they are stored in the
\Term{job queue}.
The \Condor{schedd} manages the job queue.
Various tools to view and
manipulate the job queue,
such as \Condor{submit}, \Condor{q}, and \Condor{rm},
all must connect to the \Condor{schedd} to do their work.  If the
\Condor{schedd} is not running on a given machine,
none of these commands will work.

The \Condor{schedd} advertises the number of waiting jobs in its job queue and
is responsible for claiming available resources to serve those requests.
Once a job has been matched with a given resource, the
\Condor{schedd} spawns a \Condor{shadow} daemon to serve that
particular request.

\index{HTCondor daemon!condor\_shadow@\Condor{shadow}}
\index{daemon!condor\_shadow@\Condor{shadow}}
\index{condor\_shadow daemon}
\item[\Condor{shadow}] This daemon
runs on the machine where a given
request was submitted and acts as the resource manager for the
request.  Jobs that are linked for HTCondor's standard universe, which
perform remote system calls, do so via the \Condor{shadow}.  Any
system call performed on the remote execute machine is sent over the
network, back to the \Condor{shadow} which performs the
system call on the submit machine, and the result
is sent back over the network to the job on the execute machine.
In addition,
the \Condor{shadow} is responsible for making decisions about the request,
such as where checkpoint files should be stored,
and how certain files should be accessed.

\index{HTCondor daemon!condor\_collector@\Condor{collector}}
\index{daemon!condor\_collector@\Condor{collector}}
\index{condor\_collector daemon}
\item[\Condor{collector}] This daemon
is responsible for collecting
all the information about the status of an HTCondor pool.  All other
daemons periodically send ClassAd updates to
the \Condor{collector}.  These ClassAds contain all the information about the
state of the daemons, the resources they represent or resource
requests in the pool.
The \Condor{status} command can be used to query the
\Condor{collector} for specific information about various parts of HTCondor.
In addition, the HTCondor daemons themselves query the \Condor{collector} for
important information, such as what address to use for sending
commands to a remote machine.

\index{HTCondor daemon!condor\_negotiator@\Condor{negotiator}}
\index{daemon!condor\_negotiator@\Condor{negotiator}}
\index{condor\_negotiator daemon}
\item[\Condor{negotiator}] This daemon
is responsible for all the match making within the HTCondor system.
Periodically, the \Condor{negotiator}
begins a \Term{negotiation cycle}, where it queries the
\Condor{collector} for
the current state of all the resources in the pool.  It contacts each
\Condor{schedd} that has waiting resource requests in priority order,
and tries to match available resources with those requests.
The \Condor{negotiator} is
responsible for enforcing user priorities in the system, where the
more resources a given user has claimed, the less priority they have
to acquire more resources.  If a user with a better priority has jobs
that are waiting to run, and resources are claimed by a user with a
worse priority, the \Condor{negotiator} can preempt that resource and match it
with the user with better priority.

\Note A higher numerical value of the user priority in HTCondor
translate into worse priority for that user.  The best priority
is 0.5, the lowest numerical value, and this priority gets
worse as this number grows.

\index{HTCondor daemon!condor\_kbdd@\Condor{kbdd}}
\index{daemon!condor\_kbdd@\Condor{kbdd}}
\index{condor\_kbdd daemon}
\item[\Condor{kbdd}] This daemon
is used on both Linux and Windows platforms.
On those platforms, the \Condor{startd} frequently cannot determine
console (keyboard or mouse) activity directly from the system, and
requires a separate process to do so.
On Linux, the
\Condor{kbdd} connects to the X Server and periodically checks to see
if there has been any activity.
On Windows, the \Condor{kbdd} runs as the logged-in user and registers
with the system to receive keyboard and mouse events.
When it detects console activity, the \Condor{kbdd} sends a
command to the \Condor{startd}.
That way, the \Condor{startd} knows the machine owner
is using the machine again and can perform whatever actions are
necessary, given the policy it has been configured to enforce.

\index{HTCondor daemon!condor\_ckpt\_server@\Condor{ckpt\_server}}
\index{daemon!condor\_ckpt\_server@\Condor{ckpt\_server}}
\index{condor\_ckpt\_server daemon}
\item[\Condor{ckpt\_server}] The checkpoint server
services requests to store and retrieve checkpoint files.
If the pool is configured to use a checkpoint server,
but that machine or the server itself is down,
HTCondor will revert to sending the checkpoint
files for a given job back to the submit machine.

\index{HTCondor daemon!condor\_gridmanager@\Condor{gridmanager}}
\index{daemon!condor\_gridmanager@\Condor{gridmanager}}
\index{condor\_gridmanager daemon}
\item[\Condor{gridmanager}] This daemon
handles management and execution of all \SubmitCmdNI{grid}
universe jobs.
The \Condor{schedd} invokes the \Condor{gridmanager} when
there are \SubmitCmdNI{grid} universe jobs in the queue,
and the \Condor{gridmanager} exits when there are no more
\SubmitCmdNI{grid} universe jobs in the queue.

\index{HTCondor daemon!condor\_credd@\Condor{credd}}
\index{daemon!condor\_credd@\Condor{credd}}
\index{condor\_credd daemon}
\item[\Condor{credd}] This daemon
runs on Windows platforms to manage password storage in a secure manner.

\index{HTCondor daemon!condor\_had@\Condor{had}}
\index{daemon!condor\_had@\Condor{had}}
\index{condor\_had daemon}
\item[\Condor{had}] This daemon
implements the high availability of a pool's central manager
through monitoring the communication of necessary daemons.
If the current, functioning, central manager machine
stops working, then this daemon ensures that another
machine takes its place, and becomes the central manager of
the pool.

\index{HTCondor daemon!condor\_replication@\Condor{replication}}
\index{daemon!condor\_replication@\Condor{replication}}
\index{condor\_replication daemon}
\item[\Condor{replication}] This daemon
assists the \Condor{had} daemon by keeping an updated copy of the
pool's state. This state provides a better transition
from one machine to the next, in the event
that the central manager machine stops working.

\index{HTCondor daemon!condor\_transferer@\Condor{transferer}}
\index{daemon!condor\_transferer@\Condor{transferer}}
\index{condor\_transferer daemon}
\item[\Condor{transferer}] This short lived daemon is invoked by
the \Condor{replication} daemon to accomplish the task of transferring
a state file before exiting.

\index{HTCondor daemon!condor\_procd@\Condor{procd}}
\index{daemon!condor\_procd@\Condor{procd}}
\index{condor\_procd daemon}
\item[\Condor{procd}] This daemon
controls and monitors process families within HTCondor. Its use
is optional in general, but it must be used if
group-ID based tracking (see Section~\ref{sec:GroupTracking}) is enabled.

\index{HTCondor daemon!condor\_job\_router@\Condor{job\_router}}
\index{daemon!condor\_job\_router@\Condor{job\_router}}
\index{condor\_job\_router daemon}
\item[\Condor{job\_router}] This daemon
transforms \SubmitCmdNI{vanilla} universe jobs into \SubmitCmdNI{grid}
universe jobs, such that the transformed jobs are capable
of running elsewhere, as appropriate.

\index{HTCondor daemon!condor\_lease\_manager@\Condor{lease\_manager}}
\index{daemon!condor\_lease\_manager@\Condor{lease\_manager}}
\index{condor\_lease\_manager daemon}
\item[\Condor{lease\_manager}] This daemon
manages leases in a persistent manner.
Leases are represented by ClassAds.

\index{HTCondor daemon!condor\_rooster@\Condor{rooster}}
\index{daemon!condor\_rooster@\Condor{rooster}}
\index{condor\_rooster daemon}
\item[\Condor{rooster}] This daemon wakes hibernating machines based
upon configuration details.

\index{HTCondor daemon!condor\_defrag@\Condor{defrag}}
\index{daemon!condor\_defrag@\Condor{defrag}}
\index{condor\_defrag daemon}
\item[\Condor{defrag}] This daemon manages the draining of machines
with fragmented partitionable slots, so that they become available
for jobs requiring a whole machine or larger fraction of a machine.

\index{HTCondor daemon!condor\_shared\_port@\Condor{shared\_port}}
\index{daemon!condor\_shared\_port@\Condor{shared\_port}}
\index{condor\_shared\_port daemon}
\item[\Condor{shared\_port}] This daemon listens for incoming TCP packets
on behalf of HTCondor daemons, thereby reducing the number of required
ports that must be opened when HTCondor is accessible through a firewall.

\end{description}

When compiled from source code,
the following daemons may be compiled in to provide optional functionality.
\begin{description}

\index{HTCondor daemon!condor\_hdfs@\Condor{hdfs}}
\index{daemon!condor\_hdfs@\Condor{hdfs}}
\index{condor\_hdfs daemon}
\item[\Condor{hdfs}] This daemon manages the configuration of a
Hadoop file system as well as the invocation of a properly configured
Hadoop file system.

\end{description}

% Commented out by Karen in July 2012, as it is quite out of date.
%See figure~\ref{fig:pool-arch} for a graphical representation of the
%pool architecture.

%\begin{figure}[hbt]
%\centering
%\includegraphics{admin-man/pool-arch.eps}
%\caption{\label{fig:pool-arch}Pool Architecture}
%\end{figure}

max_stars_repo_name: neurodebian/htcondor

max_issues_repo_name: neurodebian/htcondor

max_forks_repo_name: neurodebian/htcondor


text:
\documentclass[11pt]{report}

% sensible specification of margins, etc.
\usepackage[paperwidth=8.5in,paperheight=11in]{geometry}
\geometry{inner=1.0in, outer=1.5in, top=1.0in, bottom=0.5in,
  includefoot}
\geometry{twoside}

% for displaying graphics
\usepackage[]{graphicx}
% where can I find the graphics that will be imported for this version
\graphicspath{{./figures/}}

% running headers
\usepackage{fancyhdr}
\fancyhead{}
\fancyfoot{}
%\fancyhead[CO,CE]{---Draft---}
\fancyhead[LO,RE]{\it The G2Cam Toolkit}
\fancyhead[LE,RO]{\slshape \rightmark}
\fancyfoot[C]{\thepage}
%\fancyfoot[LE,RO] {\slshape \rightmark}
%\fancyfoot[LO,RE] {\slshape \leftmark}

% use modern and attractive fonts
\usepackage{fontspec}
\setmainfont[Mapping=tex-text]{Times New Roman}
\setsansfont[Mapping=tex-text]{Arial}
%\setmonofont[Scale=0.85]{Courier New}
\setmonofont[]{Courier New}
%% \setmainfont[Mapping=tex-text]{Bitstream Vera Serif}
%% \setsansfont[Mapping=tex-text]{Bitstream Vera Sans}
%% \setmonofont[Scale=0.85]{Bitstream Vera Sans Mono}

% Save space in headings
%% \usepackage[small,compact]{titlesec}
%% \titlespacing{\section}{0pt}{*0}{*0}
%% \titlespacing{\subsection}{0pt}{*0}{*0}
%% \titlespacing{\subsubsection}{0pt}{*0}{*0}

% provides itemize* environment
\usepackage{mdwlist}
% for tables with paragraphs
\usepackage{tabularx}

% for URLs without links
\usepackage{url}

% for code listings
\usepackage{listings}
\usepackage{color}
\usepackage{textcomp}
\lstset{
	%backgroundcolor=\color{lbcolor},
	tabsize=4,
	%rulecolor=,
	language=Python,
        basicstyle=\scriptsize,
        upquote=true,
        aboveskip={1.5\baselineskip},
        columns=fixed,
        extendedchars=true,
        breaklines=true,
        prebreak = \raisebox{0ex}[0ex][0ex]{\ensuremath{\hookleftarrow}},
        %frame=single,
        showtabs=false,
        showspaces=false,
        showstringspaces=false,
        identifierstyle=\ttfamily,
        keywordstyle=\color[rgb]{0,0,1},
        commentstyle=\color[rgb]{0.9,0.0,0.133},
        stringstyle=\color[rgb]{0.127,0.5126,0.941},
}
\usepackage{verbatim}
\newenvironment{myverbatim}
  {\setlength{\parskip}{0pt}\verbatim}
  {\endverbatim}

%\raggedright

\setlength{\parindent}{0cm}
\setlength{\parskip}{1em plus0.5em minus0.5em}

\setlength{\marginparsep}{0.35in}
\setlength{\marginparwidth}{1.5in}

\title{The G2CAM Subaru Instrument Interface Framework}
\author{Eric Jeschke\\
{\tt eric@naoj.org}\\
\\
Observation Control Software Group\\
Subaru Telescope\\
National Astronomical Observatory of Japan}

\pagestyle{fancy}

\begin{document}

\maketitle

\begin{abstract}
This document addresses an audience of software developers interested in
writing software for interfacing an observation instrument to Subaru
Telescope's Observation Control System (OCS). Some basic understanding
of software development principles as well as a general familiarity with
observatory infrastructures is assumed.
\end{abstract}

\tableofcontents
\setcounter{tocdepth}{3}

\newpage

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\chapter{Introduction}
\label{sh:intro}

% fig: showing the default configuration
\begin{figure}
  \begin{center}
    \begin{tabular}{c}
      \includegraphics[width=5in]{OCS_schematic.png}
    \end{tabular}
  \end{center}
  \caption[example]
%% %>>>> use \label inside caption to get Fig. number with \ref{}
          { \label{fig:obssubsys}
            Subaru Telescope Subsystem Overview
             (figure courtesy Scot Kleinman). }
\end{figure}

Subaru Telescope's distributed control system is based on four
fundamental subsystems, illustrated by Figure \ref{fig:obssubsys}:
\begin{itemize}
\item There is the Telescope Control System which controls the basic
movements of the telescope and its mechanical components, as well as
those of the dome, etc.

\item There is the Instrument Control System, which encompasses all of
the computer control components of an instrument.

\item There is the data archive system, known as STARS (Subaru Telescope
ARchive System).

\item At the center of these systems is the Observation Control System,
which interfaces to the three systems referred to above, as well as
providing the human interfaces to the operators and observers. Subaru's
OCS is known as ``Gen2'' (so called because it is the second generation
OCS for Subaru).
\end{itemize}

At Subaru, the interface between the instrument control system and the
observation control system is handled by a top-level computer referred
to as the OBCP (OBservation Control Processor). The OBCP is typically a
Unix/Linux-based system that is the locus for commands, status, and data
files transferred between the two systems.

On the right-hand side of Figure \ref{fig:obssubsys}, you can see the
OBCP and its interfaces with the OCS. This interface encompasses four
basic ``channels'':
\begin{itemize}
\item Commands to the instrument from the OCS and the instrument's responses;

\item Requests for external status data from the instrument to the OCS
and the OCS's replies with that requested status.

\item Transfers of internal status data of interest from the instrument
to the OCS for distribution to other subsystems, and

\item Requests for transfers of data files created by the instrument to
the OCS, and the OCS's replies about the results of transferring the
files. All files transferred from the instrument are automatically
archived from the OCS via STARS.
\end{itemize}

\chapter{G2CAM}
G2CAM is a object-oriented software framework, written in Python, for
interfacing top-level instrument control systems to Subaru Telescope's
Observation Control System. As shown in the diagram in Figure
\ref{fig:obssubsys}, G2CAM implements and abstracts the low level
interfaces used for commands to, status transfer to and from, and file
transfer from, the instrument's top level computer, or ``OBCP''. Using
this framework, developers can write multi-threaded applications to
interface with the OCS with a quite high level of abstraction. The
top-level G2CAM module can be written to interface with instrument's
own internal subsystems in a variety of ways, including Python classes
and functions, embedded C code, FIFOs, sockets, files, shared memory,
etc.

\section{Obtaining and Installing G2CAM}

\subsection{Requirements}
G2CAM will run happily with either python 2.7 or 3.4+.  It doesn't have
any external requirements other than the Python standard library.
For some testing purposes, it may be helpful to have {\tt astropy}
installed.

\subsection{Getting G2CAM}
g2cam is publicly available in a {\tt git} repository from the
\url{GitHub.com} web site.  To fetch it:
\begin{verbatim}
  $ git clone http://github.com/naojsoft/g2cam
\end{verbatim}

\subsection{Installation}
Install like any other python package:
\begin{verbatim}
  $ cd g2cam
  $ python setup.py install
\end{verbatim}
Be sure to use the version of python that you want to run g2cam with
when you do the install (e.g. if you have both python 2 and python 3
installed).

If you want to install it to a custom directory, please see the
{\tt README} file at the top level of the source tree, which describes
installation for more complicated setups.

For running instructions, please see Chapter \ref{ch:running} or the
above mentioned {\tt README} for the most current instructions.

\section{The G2CAM API}
To interface with the G2CAM framework, a Python module is created which
defines at least one class inheriting from the BASECAM class, which is
defined in the g2cam package.
This inheritance provides the class with some abstract superclass
methods and attributes that will normally be overridden by the subclass.

\begin{verbatim}
#
# My new instrument
#
from g2cam.Instrument import BASECAM, CamError, CamCommandError
...

class MYCAM(BASECAM):

    ....

\end{verbatim}
There is an example cam (called ``SIMCAM'') included in the ``examples''
directory of the g2cam package. Examining this module you will see what
arguments are necessary for the class constructor (also described in the
section below ``Putting it all together: the G2CAM cam'').
For methods defined inside the class there are a number of useful
convenience methods for logging, running concurrent tasks and
interacting with the OCS.

\section{Command processing}
Command processing in the G2CAM framework is based on an event-driven
callback model that will be familiar to most developers. By simply
defining non-private methods in the class, those functions immediately
become available commands to the OCS. For example
\begin{verbatim}
class MYCAM(BASECAM):
    ...
    ...
    #######################################
    # INSTRUMENT COMMANDS
    #######################################

    def snap(self, frameid=None, itime=10.0):

        self.logger.debug("snap called.")
        ...
        ...

\end{verbatim}
Defines a new instrument MYCAM that contains an callable command called
snap (presumably some kind of exposure command), with parameters frameid
and itime. From the OCS side, commands are case-insensitive, so the
command might be invoked as snap, SNAP, Snap, sNAp, etc.
On the G2CAM side the method name should be defined in lower case.

Parameters should be defined as Python keyword arguments, with default
values that will be assigned if they are not supplied at the time of
invocation.

The command does whatever it needs to do (calling other methods,
etc). There are several possible scenarios for command completion:
\begin{itemize}
\item The method raises a generic exception. In this case it will be
caught and reported as a command completion with error to the OCS (using
the exception error string). A traceback is also logged.

\item The method raises a CamCommandError exception (or a subclass
thereof). This is the preferred way of signaling an error. An error is
reported to the OCS using the exception string and no traceback is
logged.

\item The method returns normally and the result is an int or a tuple of
(int, str). In this case, the integer is interpreted as a return code: 0
is success, and non-zero is an error. In the tuple form, the string
provides a diagnostic of success or failure message.

\item The method returns normally, but the result is not an int or a
tuple of (int, str). In this case, the command is assumed to have
completed successfully, and a success code and generic diagnostic is
returned to the OCS. If a Python method exits a method by ``falling off
the end'' (i.e. no return statement), the return code is None.
That covers this particular case.
\end{itemize}
Note that since the framework is multi-threaded, executing a command
does not prevent other commands from being received and processed.
Although the API methods of the G2CAM framework are thread-safe, it is
up to the developer to make sure that shared instance
state in the subclass is protected via standard thread-safety techniques
such as semaphores, locks, condition variables, etc.
The {\tt threading} and {\tt queue} packages in the standard Python
library provide many useful classes for this purpose.

Although commands are typically driven from the OCS side, an instrument
can set up it's own autonomous command invocations.
The G2CAM framework includes a thread pool and some common methods for
executing a method or function on it (see section below ``Creating
concurrent tasks'').
Alternatively you may choose to create and invoke threads using
the Python threading library directly. Either way, a method can be
spawned off into an independent self-timed loop for handling periodic
tasks.

Since Python is a dynamic, interpreted language, the turnaround time for
developing and debugging new commands is very fast. Simply add new
methods to the module in an editor and restart g2cam.

\section{Requesting external status}
Although commands are typically initiated from the OCS side, requests
for status values are initiated from the instrument side. A typical
example is when taking an exposure, the instrument needs the telescope
pointing values for RA and DEC before and after the exposure for the
purpose of creating a FITS file.

The G2CAM framework provides a method in a delegate object for fetching
external status values. Let's enhance our snap method to illustrates how
it is used:
\begin{verbatim}

    def snap(self, frameid=None, itime=10.0):
        """
        A command to open the shutter of the camera and begin exposure.
        """
        ...

        # Create a dictionary, whose keys are status aliases we want
        statusDict = {'STATS.RA': None, 'STATS.DEC': None}

        try:
            self.ocs.requestOCSstatus(statusDict)

        except CamError as e:
            # Some kind of error occurred--bail out
            raise CamCommandError("Failed to obtain status: %s" % str(e))

        # Successful call, get values
        ra, dec = (statusDict['STATS.RA'], statusDict['STATS.DEC'])
        self.logger.debug("ra=%s, dec=%s" % (ra, dec))

        # Do other snap stuff...
        ...
        ...

\end{verbatim}
There are a couple of things to note about this example. One is the use
of the logging facility. If the run-time logging level is set to include
debug, then the information about the ra and dec will be logged. The
second is the use of exceptions to catch and signal errors. Note that
the call to {\tt requestOCSstatus()} is wrapped in a try/except clause, and
that we re-raise the error as a {\tt CamCommandError} if we run into
trouble. We could also simply return an error code, or an error code and
message, as described in the section on command processing. Thirdly,
note how we create a dictionary with the keys of the OCS status items
for which we want values. These keys are called status aliases. There
are hundreds upon hundreds of them. Someday hopefully we will have a
manual to describe them all. Until then, your best bet is to examine the
code in the example cam modules or ask a member of the Subaru OCS
software team or a Subaru support scientist what the aliases are for the
status values you need, and what the type of those values are.

The call to {\tt self.ocs.requestOCSstatus()} fills in the dictionary with the
values fetched from the OCS. The values are given the appropriate type
for the kind of status that they are (float, int, string, etc.). In this
case, the STATS.RA and STATS.DEC values are defined as string values in
the OCS, which is why they are returned as such. Of course, convenience
methods could be defined for fetching different sets of commonly needed
status items and massaging them as appropriate.

We should point out that in many cases, an attempt to fetch a valid
status alias that has no currently defined value (or even an invalid
status alias) may not raise an exception. Instead, that particular
dictionary element will have an error value in it, and the whole status
call will succeed, because the OCS system returns these error values
without signaling an overall error to the low level interface. These
individual errors may be checked for by comparing the values with the
following method (compare to above):
\begin{verbatim}

    def snap(self, frameid=None, itime=10.0):
        """
        ...
        ...

        try:
            self.ocs.requestOCSstatus(statusDict)

        except CamError as e:
            # Some kind of error occurred--bail out
            raise CamCommandError("Failed to obtain status: %s" % str(e))

        results = self.ocs.validateStatus(statusDict)
        if len(results) > 0:
            self.logger.error("Values are invalid for: %s" % str(results))
            for alias in results:
                ...
                ...
        ...
        ...

\end{verbatim}
The code could then examine the particular aliases to see if it is
possible to proceed without them.

\section{Distributing instrument status}
Instruments may want to distribute internal status that may be of
interest to external parties. In particular, if the skeleton files or
observation plan files need to conditionally do something based on the
state of the instrument, this is usually handled by examining the most
recent status that was sent by the instrument. Like status requests,
status distribution is initiated by the instrument side, and not the
OCS. Status may be distributed at any time by the instrument, and the
OCS system will absorb it, process it and make it available through the
OCS status system.

\subsection{Setting up instrument status tables}
Status can be organized into separate groups (called tables) and you can
send a group of status at a time (of course you can just lump all status
into one big group if you prefer).  Each table has a name that has a
prefix based on the instrument name.  Setting up the tables requires a
little bit of boilerplate code to set up the linkage between the G2CAM
status values and what is expected on the OCS side.
\begin{verbatim}

    #######################################
    # INITIALIZATION
    #######################################

    def initialize(self, ocsint):
        super(G2CAM, self).initialize(ocsint)

        self.logger.info("initialize() called")

        # Grab my handle to the OCS interface.
        self.ocs = ocsint

        # Get instrument configuration info
        self.obcpnum = self.ocs.get_obcpnum()
        self.insconfig = self.ocs.get_INSconfig()

        ...
        ...

        # Get our 3 letter instrument code and full instrument name
        self.inscode = self.insconfig.getCodeByNumber(self.obcpnum)
        self.insname = self.insconfig.getNameByNumber(self.obcpnum)

        self.statusTblName = ('%3.3sS0001' % self.inscode)

        # Make a table with some status items.
        self.mystatus = self.ocs.addStatusTable(self.statusTblName,
                                                ['status', 'mode', 'count',
                                                 'last_frame'])

        ...
        ...

\end{verbatim}
This code example shows the initialize method, which is part of the
standard boilerplate code used in initializing the instrument module
(see section ``Putting it all together: the MYCAM cam''). When called, the
parameter ocsint is a handle to the OCS interface delegate object. After
calling the superclass constructor, the method stores this away in its
internal state in the variable ocs (you encountered this variable in
earlier examples, now you see where it came from).

Using that handle, the initialize method can now invoke delegate calls
to the interface to accomplish various things. First, it finds out it's
own OBCP number (all Subaru instruments are enumerated), followed by
it's code (official 3-letter acronym) and full name (canonical name).

It then constructs a table name (the example shows the canonical table
name for an instrument) and then makes a call to {\tt addStatusTable()}
to create a table containing the listed status elements.

\subsection{Getting and setting internal status}
Beyond the boilerplate code, status values can be set any time, anywhere
in the module by simply assigning the values to the status table object
(you can also create multiple table objects) in dictionary style, or
using any of several convenience methods:
\begin{verbatim}

        ...
        # dictionary style interface...
        self.mystatus['status'] = 'ALIVE'
        self.mystatus['last_frame'] = ''

        # Convenience method, with keyword arguments:
        self.mystatus.setvals(status='ALIVE', last_frame='')
        ...
        # Convenience method, with dictionary:
        statusDict = {'status': 'ALIVE', 'last_frame': ''}
        self.mystatus.store(statusDict)
        ...

\end{verbatim}
The advantage of using the convenience functions is that it ensures that
all the listed status items are updated together, and not intermixed if
another thread is also updating, or reading, status.

Similarly, to read your own (instrument) status, you can either access
the items dictionary style, or use convenience methods:
\begin{verbatim}

        ...
        # dictionary style interface...
        last_frame = self.mystatus['last_frame']

        # Multiple items can be fetched, atomically as a group
        (status, last_frame) = self.mystatus.fetchList(['status',
                                                        'last_frame'])

        # Or you can use a dict, to get them as a group
        statusDict = {'status': None, 'last_frame': None}
        self.mystatus.fetch(statusDict)
        ...

\end{verbatim}
Again, using the convenience methods insures that all the status items
are fetched as a group, atomically.

\subsection{Transmitting internal status}
Status is not transmitted (distributed) until an explicit call is made
to do so. You might define a method to export the status like so:
\begin{verbatim}

    def putstatus(self, tableName="ALL"):
        """Export of our status.
        """
        if tableName == 'ALL':
            res = self.ocs.exportStatus()

        else:
            res = self.ocs.exportStatusTable(tableName)

        return res

\end{verbatim}
The {\tt exportStatus} and {\tt exportStatusTable} delegate methods
cause the current status table objects (all objects or a specific one,
respectively) to be transmitted to the OCS. The putstatus method then
becomes a convenience function for the class, that can be called from
anywhere to push out status, as well as a command that can be invoked
from the OCS side to test status transfers.

We are now in a position to modify our snap method to send some status
that indicates the last frame saved:
\begin{verbatim}

    def snap(self, frameid=None, itime=10.0):

        ...
        ...

        # Do other snap stuff...
        ...

        # Assuming command was successful:
        self.mystatus.setvals(last_frame=frameid)
        self.putstatus()

\end{verbatim}

\section{Viewing data in the OCS FITS viewer}
An instrument can send data to be viewed by the OCS online viewer
without having to allocate a frame and/or archive a file. If the data is
already on disk, the following code illustrates the necessary call:
\begin{verbatim}

    self.logger.info("Viewing FITS file '%s' in the OCS" % name)
    self.ocs.view_file_as_buffer(fitspath, num_hdu=0,
                                      imname=name)
\end{verbatim}

If the image data is already in memory, there are some other options:
\begin{verbatim}

    # view astropy.io.fits HDU
    self.ocs.view_hdu(hdu, name)

    # view bare numpy data array
    # populate header dictionary with keyword/value pairs if desired
    header = {}
    self.ocs.view_data(data_np, name, header=header)

\end{verbatim}

\section{Archiving files}
When an instrument has a FITS file to archive, it uses the fourth
interface provided by the framework. This action is also initiated from
the instrument side by a file transfer request. The OCS responds by
trying to transfer the file (currently using either FTP or SCP as the
protocol).
The following convenience function illustrates the necessary call:
\begin{verbatim}

    def archive_data_file(self, frameid, fitspath):

        self.logger.info("Archiving FITS file '%s' to the OCS" % frameid)

        self.ocs.archive_frame(frameid, fitspath)

        # If no exceptions raised, then we end up here.  Update status
        # to indicate last frame saved
        self.mystatus.setvals(last_frame=frameid)
        self.putstatus()

\end{verbatim}
As you can see the {\tt self.ocs.archive\_frame()} call takes a frame id (like
'SUKA00048245') and the path to the FITS file. The FITS file does not
need to have the name be the frameid. The call will send a file transfer
request to the OCS to ask it to copy the file, renaming it on the other
end as the frame id.

Note that in order for the OCS to transfer the files, an FTP, FTPS or
SSH service should be configured and running on the OBCP.

We are now in a position to modify our snap method to archive the file:
\begin{verbatim}

    def snap(self, frameid=None, itime=10.0):
        ...
        ...

        # Do other snap stuff...
        fitspath = ...

        # Create fits file...
        ...
        self.archive_data_file(frameid, fitspath)

\end{verbatim}

\subsection{Frame Ids}
You may be wondering what the frame id (frameid, AKA ``frame number'') is,
and how they are generated. If you look back to the snap examples, you
can see that the frameid was passed in to that command. This is the
prototypical way that instruments operate at Subaru: FITS files to be
archived are only created at the behest of a OCS initiated command, and
such commands will be designed to take the frame id or an initial frame
id and a count (in the case of generating multiple files).

In the rare case that your instrument needs to autonomously allocate a
frame id and archive a file, you can use the {\tt getFrame()} method to
allocate a frame. The following example shows how this would be done:
\begin{verbatim}

        ...
        ...

        # get a frame id
        frameid = self.getFrame('A')

        # Create fits file...
        fitspath = ...
        ...

        self.archive_data_file(frameid, fitspath)

\end{verbatim}
'A' frames are raw science frames. 'Q' frames are quick look frames.
You should use either 'A' or 'Q' according to the kind of data
to be archived.

\section{Creating concurrent tasks}
For many commands, the control flow would work out something like this:
\begin{itemize}
\item Command is received from the OCS side, calls the developer's method
      (e.g. {\tt snap()}, with parameters frameid and itime).

\item Instrument requests and receives status from the OCS, if needed.

\item Instrument does the command, possibly generating FITS files or
    status to be sent in the process. It uses the respective APIs to
    send status and or files to be archived.

\item Command finishes, via one of the appropriate ways outlined earlier.
\end{itemize}
Sometimes, this is not the flow of control that you want. For example,
you may want to have an asyncronous command that validates its
parameters and then exits quickly, but which initiates continued
processing on the instrument side that entails further processing. This
is often done for long-running commands. For example, suppose we want a
version of {\tt snap()} that returns immediately, but initiates an exposure
operation on the instrument. Later, the instrument will finish the snap
as previously described and send the finished fits files and update the
status value to indicate completion of the command. Let's see how we
might do that.
\begin{verbatim}

from g2cam.util import common_task as common
    ...
    ...
    def snap_async(self, frameid=None, itime=30.0):
        """Snap command that returns immediately, then later finishes
        asynchronously.
        """

        self.logger.info("long_snap called...")

        if not frameid:
            raise CamCommandError("No frame number passed!")

        self.logger.info("Starting concurrent task")
        t = common.FuncTask(self.snap, (), frameid=frameid,
                                           itime=itime)
        t.init_and_start(self)

\end{verbatim}
This command creates a generic function-calling task that will call the
method {\tt snap()} with no positional arguments and the exact same keyword
arguments that it was called with. Then it initializes and starts the
task. The function will be called by another thread in the framework's
thread pool, leaving this thread to exit the current command.

The general problem with this sort of approach is the same as for any
asynchronous command: what to do if the command fails. The initial
command completed with success, but how do you track the result of the
asynchronous part? Typically this would be handled via status value
notification. Alternatively, another way that this can be handled is to
have a 2-phase command. For example, snap() may initiate an exposure and
return immediately. A second command (e.g. readout) is sent later by the
OCS to read out the CCD, create the FITS files and archive them.

There are a few common enough tasks that they have been added as a
module of common task code that can be used by any g2cam instrument.

\subsection{Example: Sending Periodic Status}
Our example is used to illustrate the sending of periodic status.
The Subaru OCS likes to receive periodic status updates from the
instrument as a kind of heartbeat, to indicate that the instrument is up
and running. The exact interval is not that important, and instruments
use anything from 5 seconds to 5 minutes. A reasonable interval might be
60 seconds. Assuming you have defined a {\tt putstatus} method as
described earlier, you can effect that simply by invoking the common task:
\begin{verbatim}

    def start(self, wait=True):
        ...
        ...
        # Start task to periodically send status
        t = common.IntervalTask(self.putstatus, 60.0)
        self.status_task = t
        t.init_and_start(self)

\end{verbatim}
The {\tt start()} method is called after initialize, and should contain
whatever code you need to ``start'' your task.

The {\tt IntervalTask} type simply runs the given function at the given
intervals.

\section{Putting it all together: the MYCAM cam}

Here then, is a ``shell'' of a new camera MYCAM, from top to bottom:
\begin{verbatim}
#
# MYCAM.py -- shell for a new instrument MYCAM on the G2CAM framework
#
#
"""
This file implements a simulator for a simulated instrument (G2CAM).
"""
import sys, os, time
import threading

from g2cam.Instrument import BASECAM, CamError, CamCommandError
from g2cam.util import common_task as common

class MYCAMError(CamCommandError):
    pass

class MYCAM(BASECAM):

    def __init__(self, logger, env, ev_quit=None):

        super(MYCAM, self).__init__()

        self.logger = logger

        # Convoluted but sure way of getting this module's directory.
        # Useful if we need to load some files.
        self.mydir = os.path.split(sys.modules[__name__].__file__)[0]

        self.ev_quit = ev_quit
        self.ocs = None
        self.mystatus = None

    #######################################
    # INITIALIZATION
    #######################################

    def initialize(self, ocsint):
        """
        Initialize this instrument for use, but do not start it.
        """
        super(MYCAM, self).initialize(ocsint)

        self.logger.info('***** INITIALIZE CALLED *****')

        # Grab my handle to the OCS interface.
        self.ocs = ocsint

        # Get instrument configuration info
        self.obcpnum = self.ocs.get_obcpnum()
        self.insconfig = self.ocs.get_INSconfig()

        # Thread pool for autonomous tasks
        self.threadPool = self.ocs.threadPool

        # Required instance variables for starting tasks:
        self.tag = 'mycam'
        self.shares = ['logger', 'ev_quit', 'threadPool']

        # Get our 3 letter instrument code and full instrument name
        self.inscode = self.insconfig.getCodeByNumber(self.obcpnum)
        self.insname = self.insconfig.getNameByNumber(self.obcpnum)

        self.statusTblName = ('%3.3sS0001' % self.inscode)

        # Register my status table.
        self.mystatus = self.ocs.addStatusTable(self.statusTblName,
                                                ['status', 'mode', 'count'])

        # Establish initial status values
        self.mystatus.setvals(status='ALIVE', mode='LOCAL', count=0)

        # Handles to periodic tasks
        self.status_task = None

        # Lock for handling mutual exclusion
        self.lock = threading.RLock()

    def start(self, wait=True):
        super(MYCAM, self).start()

        self.logger.info('***** MYCAM STARTED *****')

        # Start auto-generation of status task
        t = common.IntervalTask(self.putstatus, 60.0)
        self.status_task = t
        t.init_and_start(self)

    def stop(self, wait=True):
        super(MYCAM, self).stop()

        # Terminate status generation task
        if self.status_task != None:
            self.status_task.stop()

        self.status_task = None

        self.logger.info('***** MYCAM STOPPED *****')

    #######################################
    # INTERNAL METHODS
    #######################################

    # Whatever internal methods you want go here.  You can begin the
    # name with underscore (_) to make it private

    #######################################
    # INSTRUMENT COMMANDS
    #######################################

    def sleep(self, sleep_time=0):
        """One of the commands that are in the OBCPTEST.cd
        """

        self.logger.info("\nSleeping for %d sec..." % sleep_time)
        time.sleep(sleep_time)
        self.logger.info("Woke up refreshed!")

    def obcp_mode(self, motor='OFF', mode=None):
        """
        One of the commands that are in the OBCPTEST.cd
        """
        self.lock.acquire()
        try:
            mode = self.mystatus['mode']
            if mode == 'LOCAL':
                self.mystatus.setvals(mode='SLAVE')
            else:
                self.mystatus.setvals(mode='LOCAL')

        finally:
            self.lock.release()

        self.putstatus()

    def fits_file(self, motor='OFF', frame_no=None):
        """
        One of the commands that are in the OBCPTEST.cd.
        """

        # Create FITS file (pyfits module is really useful here)
        # fitspath = ...

        self.ocs.archive_frame(frame_no, fitspath)

    def putstatus(self, target="ALL"):
        """
        Forced export of our status.
        """
        self.ocs.exportStatus()

    def getstatus(self, target="ALL"):
        """
        Forced import of status.
        """
        # Create a dictionary, whose keys are status aliases we want
        statusDict = {'STATS.RA': None, 'STATS.DEC': None}

        try:
            self.ocs.requestOCSstatus(statusDict)

        except CamError as e:
            # Some kind of error occurred--bail out
            raise CamCommandError("Failed to obtain status: %s" % str(e))

        # Successful call, get values
        ra, dec = (statusDict['STATS.RA'], statusDict['STATS.DEC'])
        self.logger.debug("ra=%s, dec=%s" % (ra, dec))

#END MYCAM.py
\end{verbatim}

\chapter{Running G2CAM}
\label{ch:running}
There is a simple wrapper program called {\tt g2cam} that provides the
appropriate environment for running a G2CAM module. A typical invocation
of this program to start the instrument might be as follows:
\begin{verbatim}

$ cd examples
$ g2cam --cam=MYCAM --gen2host=localhost --loglevel=20 \
     --log=mycam.log --stderr

\end{verbatim}
This says to start up the instrument with the log directed to the file
``mycam.log'', an additional copy of the logging directed to stderr, and
the logging level set to 0 (log everything). The PARA files will be
loaded from the path ../SkPara/cmd/MYCAM.

Invoking the program with the --help option will list all options and
their usage.

The source code includes an example cam that you can run for testing or
reference.

It is very helpful to get a companion piece for testing, which is a
virtual machine containing a simulator of the summit Gen2 system.  Using
the simulator you can test your developing g2cam in the same way it
would actually be commanded at the telescope.

Consult the OCS team at Subaru to get the correct hostname to use for
connecting to a simulator or summit Gen2 system.

\chapter{Appendix A: Considerations for SOSS Compatibility Mode}
SOSS Compatibility Mode is a mode of operation of Gen2 that maintains a
level of backward compatibility for legacy observation scripts written
for SOSS, Subaru's first OCS. This section describes some considerations
for developing observations that are compatible with with this mode,
which is the primary method of observing at Subaru.

\section{Abstract commands}
Subaru operators and observers prepare observation plans as OPE
files. These are essentially command scripts with command line
invocations--it looks much like a batch script, but without any control
constructs other than a sequential flow of commands. Observation
proceeds by highlighting the commands to be executed in a GUI and
pressing an execute button. These commands are either abstract
(high-level) commands or device dependent (low level)
commands. Device-dependent commands are commands that are issued to one
of the subsystems (such as those described above): telescope,
instrument, etc. Abstract commands are essentially macros which undergo
an expansion process known as decoding, before being passed to an
execution environment and executed.

Abstract commands are written using a macro template language and the
templates are stored as ``skeleton'' files. The macro language includes
some primitive control structures that implement the equivalent of
iteration, subroutine calls, etc. Ultimately, the processing of skeleton
files expands into sequential or concurrent issuance of device dependent
commands to one of the distributed subsystems. In this manner,
potentially complex patterns of interdependent distributed command
execution on the four subsystems is realized.

\section{Device dependent commands and PARA files}
Throughout this example, we have shown G2CAM methods using keyword
arguments with default values for device-dependent commands invoked by
the OCS. In the case of SOSS, there is an additional interface
specification for device-dependent commands, which is implemented via
``PARA'' (parameter) files. Essentially, each device-dependent command
that can be issued from the OCS has an interface (PARA) file associated
with it. The file describes the command name, parameters with their
types, and default values. The format of PARA files is described
elsewhere. We mention them here to note only that there is another
location where default values to paramters can be set.

\section{SOSS status aliases}
It should be noted that the OCS status aliases referring to instrument
status can be completely different from the keys used on the G2CAM side
(or you can choose to keep them in sync, for clarity, if you wish). Such
aliases on the OCS side are typically the instrument 3-character code as
prefix, followed by a dot and then an alphanumeric suffix. For example,
our status value from earlier may be known on the OCS side as MNI.STATUS
(assuming that ``MYI'' is the code for ``My New Instrument'').

\end{document}