text:
%
% Chapter 3
%

\chapter{LHC and the CMS detector}
\label{experiment}

\section{Introduction}

The prime goals of the LHC and the CMS~\cite{Chatrchyan:2008aa} experiment are exploring the physics at the~\TeV scale and studying the mechanism of electroweak symmetry breaking, including studying the SM Higgs boson's properties along with searches for new particles predicted by beyond the SM theories. The physics program's other main interests include studying the SM top quark properties, electroweak physics, and physics of hadrons containing a charm or bottom quark. Heavy-ion collisions address the physics of strongly interacting matter and the quark-gluon plasma. Hadrons consist of quarks and gluons; therefore, two colliding partons' initial energy is not known. In contrast, the collision energy is known at lepton colliders, where each particle has the same energy. Thus, hadron colliders can explore a wide range of collision energies, while lepton colliders are well suited for precision measurements.

\section{The Large Hadron Collider}
\label{sec:LHC}

The LHC is a hadron accelerator located at CERN. The design was intended to collide proton beams with a beam energy of 7~\TeV leading to a center-of-mass energy of 14~\TeV and reach a luminosity of $10^{34}~\lumi$. Lead ions can be accelerated up to an energy of 2.8~\TeV per nucleon and reach a luminosity of $10^{27}~\lumi$. The LHC is the most powerful tool for particle physics research that is currently available. The 26.7 km tunnel constructed between 1984 and 1989 for the Large Electron-Positron collider~\cite{Electron-Positron:1997351} was reused to install the LHC. There are eight arcs and eight straight sections lying 45-170~\m below the surface. Unlike particle-antiparticle colliders that can use a single ring for both beams, the LHC uses two rings with counter-rotating beams. There is less synchrotron radiation owing to the heavier particles being collided at the LHC.

The accelerator complex acts as an injector of the protons and heavy ions. Protons are obtained from hydrogen gas after the electrons are stripped off, and they enter the Linear accelerator 2~\cite{accelerator:1997427}, where they are accelerated to 50~\MeV. They are further accelerated in the Proton Synchrotron Booster~\cite{Synchrotron:1997372} to 1.4~\GeV. They are then injected in the Proton Synchrotron~\cite{Synchrotron:1997189}, where they are accelerated to 25~\GeV. A bunch train is produced within the Proton Synchrotron before extraction. The protons are then accelerated in the Super Proton Synchrotron~\cite{Synchrotron:1997188} to 450~\GeV before injecting in the LHC.

The four interaction points are equipped with particle detectors: the CMS experiment, the ``A Toroidal LHC ApparatuS'' ATLAS experiment~\cite{Aad:2008zzm}, the ``A Large Ion Collider Experiment'' ALICE experiment~\cite{Aamodt:2008zz}, and the ``LHC beauty'' LHCb experiment~\cite{Alves:2008zz}. Two further smaller experiments, the ``Total Cross Section, Elastic Scattering and Diffraction Dissociation'' TOTEM~\cite{Anelli:2008zza} and the ``LHC forward'' LHCf~\cite{Adriani:2008zz} experiments, are located near the CMS and the ATLAS interaction points, respectively.

The ATLAS and CMS experiments have both multi-purpose detectors installed. The detectors were built to detect particles from \pp or heavy-ion collisions. One of the main tasks currently is to study the production and decay of the discovered Higgs boson~\cite{Chatrchyan:2013lba}, disentangle its properties, and check if it is the SM Higgs boson or a Higgs boson of an extension of the SM. The other tasks involve high precision tests of QCD, electroweak interactions, and heavy flavor physics. Precision measurements of production, the couplings, and the spin of the top quark are also pursued. Several searches for supersymmetric particles and dark matter are also ongoing.

During the LHC Run 2 data-taking, a bunch spacing of 25~\ns was used, and \pp collision data were collected at $\sqs = 13 \TeV$ in 2016, 2017, and 2018. The integrated luminosity delivered to CMS as a function of time is shown in Figure~\ref{fig:lumi} for each \pp collision data-taking period. CMS does not record the whole delivered data, and only part of the recorded data that is considered good is used for physics analysis.

\begin{table}[!hbpt]
  \centering
  \caption{Integrated luminosity considered for physics analysis at the CMS experiment during Run 2}
  \begin{tabular}{|c|c|}
    \hline Year & Integrated luminosity \\
    \hline 2016 & 35.9~\fb \\
    \hline 2017 & 41.5~\fb \\
    \hline 2018 & 59.3~\fb \\
    \hline
  \end{tabular}
  \label{tab:luminosity}
\end{table}

\begin{figure}[htbp]
  \centering
  \includegraphics[width=0.8\textwidth]{plots/chapter3/int_lumi.png}
  \caption{Cumulative luminosity as a function of time delivered to CMS during stable beams for \pp collisions. The luminosity is shown for 2015 (purple), 2016 (orange), 2017 (light blue), and 2018 (dark blue)~\cite{lumi}.}
  \label{fig:lumi}
\end{figure}

\section{The Compact Muon Solenoid experiment}
The layout of the CMS detector is shown in Figure~\ref{fig:cms}. As the name ``Compact Muon Solenoid'' indicates, a superconducting solenoid is at the heart of CMS. A considerable bending power for the momentum measurement of charged particles within a compact design is achieved using a high magnetic field of 3.8 T. The internal part of the solenoid is large enough to accommodate the inner tracking system and the calorimetry. The inner tracking system is composed of a pixel detector close to the interaction region and a silicon strip tracker. The pixel detector can resolve individual vertices and distinguish between vertices from the primary interaction and secondary vertices from the decay of the primary interaction particles. The trajectories are precisely measured with the high granular silicon strip tracker, which can deal with high charged particle multiplicities.

\begin{figure}[htbp]
  \centering
  \includegraphics[width=0.9\textwidth]{plots/chapter3/cms_layered.png}
  \caption{Schematic view of the CMS detector.}
  \label{fig:cms}
\end{figure}

The energy of the particles is measured in calorimeters. The calorimeter material initiates electromagnetic or hadronic showers. For electromagnetic interactions, the characteristic interaction length is the radiation length $X_0$, while the characteristic interaction length for hadronic showers is the nuclear interaction length $\lambda_I$. The entire volume is sensitive in homogeneous calorimeters, while sampling calorimeters consist of metallic absorber sandwiched or threaded with an active material that generates the signal. CMS has an electromagnetic calorimeter (ECAL) made of lead tungstate ($\text{PbWO}_{4}$) crystals in front of a brass/scintillator sampling hadron calorimeter (HCAL). An additional layer of scintillators is outside the coil. The magnet is used as an absorber material. This iron/quartz-fiber calorimeter is referred to as the hadron outer detector. The muon detectors are sandwiched between the layers of the steel return yoke. Their main task is to trigger on muons and to identify the muons with good momentum resolution.

\subsection{The coordinate system of CMS}
The CMS experiment uses a right-handed coordinate system, with the origin at the nominal collision point. The x-axis points towards the LHC center, while the y-axis points upward towards the surface (perpendicular to the LHC plane), and the z-axis points along the anti-clockwise beam direction. Two angles are defined, where the azimuthal angle $\phi$ is measured from the x-axis in the x-y plane, and the polar angle $\theta$ is measured from the z-axis. Pseudorapidity is defined as
%
\begin{equation}
  \eta=-\ln \left[\tan \left(\frac{\theta}{2}\right)\right]
\end{equation}
%
and describes the angle of a particle relative to the beam axis, illustrated in Figure~\ref{fig:coordinate}. Distances in $\phi$ and $\eta$ are denoted $\Delta\phi$ and $\Delta\eta$. These distance measures are used to define the spatial separation between physics objects by \dr with

\begin{equation}
  \dr=\sqrt{(\Delta \phi)^{2}+(\Delta \eta)^{2}}
\end{equation}

\begin{figure}[htbp]
  \centering
  \includegraphics[width=0.9\textwidth]{plots/chapter3/coordinate.png}
  \caption{Coordinate system convention of CMS (left) and the relation between pseudorapidity $\eta$ and polar angle $\theta$ (right)}
  \label{fig:coordinate}
\end{figure}

\subsection{Kinematic quantities}
At LHC, the protons carry half of the collision energy. The hard interactions occur between the colliding protons' partons, and each parton carries a fraction $x_i$ of the proton momentum. As the parton mass can be neglected with respect to the momentum $\vec{p}$, the parton energy is given by its momentum. In the laboratory frame, the four-momenta of the partons are

\begin{equation}
  \begin{aligned}
    \text{p}_{1}&=x_{1} \cdot \frac{\sqs}{2}(1,0,0,1) \\
    \text{p}_{2}&=x_{2} \cdot \frac{\sqs}{2}(1,0,0,-1)
  \end{aligned}
\end{equation}

Then, the invariant mass of the hard collision is given by
%
\begin{equation}
  \hat{s} \equiv \text{M}^{2}=\left(\text{p}_{1}+\text{p}_{2}\right)^{2}=\frac{s}{4} \cdot\left[\left(x_{1}+x_{2}\right)^{2}-\left(x_{1}-x_{2}\right)^{2}\right]=x_{1} x_{2} s
\end{equation}
%
where $\sqrt{\hat{s}}$ denotes the center-of-mass energy of the parton-parton collision. The hard collision products have a zero total momentum for the x- and y-components but in general non-zero for the z-component. Thus, the hard-collision products' momentum and energy are measured transverse to the beam direction in the x-y plane. The transverse momentum is denoted by \pt and the transverse energy is denoted by \ET with $\ET = \text{E} \cdot \text{sin} \theta$. The weakly interacting particles, like neutrinos, do not produce a signal in the CMS detector and lead to an imbalance in the observed total transverse momentum in the event, missing transverse energy, denoted as \met.

\subsection{Detector requirements}
Isolated leptons in the final states leave clean signatures in the CMS detector. The muon has to be well-identified, and its momentum resolution needs to be good over a wide range of momenta. The charge of the muon has to be determined unambiguously. In addition, electrons and charged hadrons need to be reconstructed and identified with good efficiency and momentum resolution. The momenta of charged particles are measured using the curvature of their trajectory. Hence, considerable bending power is needed to measure precisely the charge of particles with large momentum. Furthermore, a good electromagnetic energy resolution is needed over a wide geometrical coverage and correct localization of the primary interaction.

The CMS detector can detect the hadrons produced from the hadronization of the quarks. Tau leptons that decay hadronically need to be identified with good efficiency for the Higgs boson physics. A good measurement of the impact parameter of charged particle tracks and good position measurement of the secondary vertices can improve the triggering and identification efficiency of hadronically decaying taus. This requires having the pixel detector close to the interaction region. Hadronic calorimeters with a large hermetic coverage ($\aeta < 5$) and with a fine lateral segmentation $(\Delta \eta \times \Delta \phi < 0.1 \times 0.1)$ are required for measuring the hadrons energy with good resolution and for the estimation of \met.

The collisions are happening at a rate of 40~\mhz and not all of these events can be stored; therefore, only interesting events have to be selected. The online event selection process, trigger, must reduce the rate to $\sim 1~\khz$. The short time between two bunch crossings, 25~\ns, has a major implication on the read-out and trigger system design. Multiple \pp interactions (pileup) happen in one bunch crossing. The interactions' products overlap and can be wrongly linked, and long response times of detector elements and their electronic signal longer than 25~\ns increase this effect. High granularity detectors with good time resolution result in a low occupancy. This requires many detector channels and, therefore, a good synchronization of the electronic detector channels. The large flux of the particles and resulting high radiation levels require radiation-hard detectors and front-end electronics.

\subsection{Magnet}

The curvature of the particle trajectory in a magnetic field determines the momenta and sign of charged particles. The most important aspect of muon measurement is the choice of magnetic field configuration. A good momentum resolution of $\Delta \text{p} / \text{p} \approx 10 \%$ is required for muons with a momentum of 1~\TeV. CMS uses a large superconducting solenoid with a magnetic field of 3.8 T. The solenoid is 13 m long and has an inner diameter of 5.9 m. The superconductor material is Niobium-titanium~\cite{BUL-NA-2003-150}. The conductor is made of high purity and is aluminum stabilized with a four-layer winding and is composed of five coils and has to withstand outward pressure. The superconducting wire is cooled using an indirect cooling by thermosyphon.

\subsection{Inner tracking system}

The inner tracking system~\cite{Khachatryan:2010pw} measures the trajectories of particles up to $\aeta < 2.5$. The particle flux is the highest close to the interaction region. The silicon pixel detector is placed close to the interaction region and allows for reconstructing vertices from heavy flavor hadrons with a b or c quark. Each pixel has a size of $\approx 100 \times 150 \mum^{2}$.  The pixel detector consisted of three layers at radii of 4, 7, and 11~\cm in the barrel. However, this was upgraded after 2016 and currently consists of four concentric barrel layers at radii of 29, 68, 109, and 160~\mm. There are three disks on each end at distances of 291, 396, and 516~\mm from the center of the detector. The spatial resolution of the radius and azimuthal angle $r$--$\phi$ measurement is about 10~\mum and 20~\mum for the $z$-coordinate measurement.

The barrel's tracking volume has a cylindrical shape with a length of 5.8~\m and a diameter of 2.6~\m. In the intermediate region ($20 < r < 55 \cm$) and outermost region ($r > 55 \cm$) silicon microstrips are used with a minimum cell size of $10 \cm \times 80 \mum$ and a maximum cell size of $25 \cm \times 180 \mum$, which provide the required granularity. The strip tracker is divided into a tracker inner barrel (TIB) made out of four layers and a tracker outer barrel (TOB) made out of six layers. Each part has two layers which provide a single-point measurement in the $r$--$\phi$ and $r$--$z$ coordinates. In the TIB, the single-point resolution is 23--34~\mum in the $r$-direction and 23~\mum in $z$, while it is 35--52~\mum in the $r$--$\phi$ direction and 52~\mum in $z$ for the TOB. In the two end caps, there are nine microstrip layers. The strip tracker's endcaps are divided into the tracker endcap made of 9 disks and the tracker inner disks made of three small disks, which fill the gap between the TIB and the tracker endcap.

\subsection{Electromagnetic calorimeter}

The ECAL is a hermetic, homogeneous calorimeter with a coverage of $\aeta < 3$, and it measures the energy of electrons and photons. It has an excellent energy resolution and high granularity and can separate close clusters very well~\cite{CMS:2010bta, Khachatryan:2015hwa}. ECAL uses $\text{PbWO}_4$ crystals, which have a short radiation length of $X_0 = 0.89 \cm$. These crystals are radiation hard and emit blue-green scintillation light with a maximum at 420 nm. The crystals allow a compact calorimeter inside the solenoid that is fast, has a fine granularity and is radiation-resistant. The decay time for the scintillation is short and in the same order as the LHC bunch crossing time, and about 80\% of the light is emitted within 25~\ns. However, the light output depends on the temperature. Therefore, to preserve energy resolution, a cooling system is needed. Besides, as the light yield is low, this requires photodetectors with intrinsic gain that can also operate in a high magnetic field. Figure~\ref{fig:ecal} illustrates the layout of the ECAL.

\begin{figure}[htbp]
  \centering
  \includegraphics[width=0.9\textwidth]{plots/chapter3/ecal.png}
  \caption{Schematic view of the ECAL.}
  \label{fig:ecal}
\end{figure}

The calorimeter is divided into a ECAL barrel (EB) which covers the region $0 < \aeta < 1.479$ and two ECAL endcaps (EE) which cover the region $1.479 < \aeta < 3.0$. Preshower detectors are installed in front of each endcap and cover the region $1.653 < \aeta < 2.6$. In the EB, the scintillation light is detected with silicon avalanche photodiodes (APD). These photodiodes respond to temperature changes and require a stable temperature. The barrel section is structured in 36 identical supermodules, 18 in each half barrel, with the crystals being arranged in a grid and covering $\Delta \eta \times \Delta \phi = 0.0174 \times 0.0174$. In the EE, the scintillation light is detected with vacuum phototriodes. A $5 \times 5$ array of crystals is termed supercrystals. The crystals and the supercrystals are arranged in a rectangular x-y grid. The preshower system is installed in front of EE to identify and reject $\pi^0$ mesons, which dominantly decay into two photons. It has a higher granularity than the EE and improves the position determination of electrons and photons. It is a sampling calorimeter consisting of two layers of a lead absorber, which initiate the electromagnetic showers from incoming electrons or photons. The lead radiators are followed by silicon strip sensors measuring the energy deposits and the transverse shower profiles. The silicon strip sensors are oriented orthogonally and are placed after a radiation length of 1 $X_0$ and 2 $X_0$ of the lead absorber.

The ECAL calibration and performance is estimated from the supercluster energy deposited by the electrons and photons. The calibration is performed, which corrects the detector's response to the energy deposited to match expectations. The calibration is applied to the superclusters to take into account the $\eta$ and $\phi$-dependent geometry and material effects, as well as the fact that electrons and photons shower slightly differently. The calibrations were tuned and validated using electrons' energy from \PW boson decays, the reconstructed mass from $\eta$ decays to two photons, and the energy resolution measured with \Zee events. I implemented an event skimmer using the updated electron identification and isolation selections to select events with \Zee resulting from the \pp collisions. The primary motivation for implementing the event skimmer is to obtain datasets enriched with \Zee events that can be further used for performing calibration in ECAL, and because trigger selections are not applied in the event skimmer, it can be used for trigger efficiency studies. One primary difference from the old event skimmer is using the effective area method to calculate electrons' isolation instead of the delta beta method. The description of these methods is given in section~\ref{isolation}. The event skimmer's acceptance is tested on the 2017 data, and the obtained event rate is within expectations. It is currently integrated into the CMS software framework.

\subsection{Hadron calorimeter}

The HCAL is a sampling calorimeter, and it measures the energy of hadrons. The HCAL system is located surrounding the ECAL system, and most of it is located inside the solenoid. The HCAL design has an important requirement to minimize the non-Gaussian tails in the energy resolution and to provide a good hermeticity for the determination of \met. Another design requirement is to maximize the interaction length of the material within the solenoid. This corresponds to minimizing the space allocated to the active material. It is divided into a HCAL barrel (HB) detector, covering the region $\aeta < 1.4$ and two HCAL endcap (HE) detectors, covering $1.3 < \aeta < 3.0$. The absorber material is brass. Brass is chosen because it has a relatively short interaction length, is easy to machine, and is non-magnetic. The active material is a plastic scintillator. In order to provide structural support, stainless steel is used in the innermost and outermost layers. The 3.7~\mm thick scintillator plates are sandwiched between the absorber plates. Wavelength-shifting fibers are embedded in the scintillator tiles, which channel the light to photodetectors that can operate in high axial magnetic fields called hybrid photodiodes. In the HB, the segmentation is $\Delta \eta \times \Delta \phi = 0.087 \times 0.087 \approx 5^{\circ} \times 5^{\circ}$, which is termed tower. In the HE, the $\phi$ segmentation is $5^\circ - 10^\circ$ and the $\eta$ segmentation is 0.087--0.35 depending on $\eta$.

The energy from the hadron showers that leak through the calorimeters' rear are sampled with additional layers of scintillators that are installed outside the coil within the return yoke using the iron as an absorber. This helps in improving the central shower containment and the \met resolution of the calorimeter. This sampling calorimeter is referred to as hadron outer detector. It is located along with the muon barrel system and closely follows its segmentation. It is divided into five sections termed rings along $\eta$. The hadron outer detector follows the HB geometry in $\eta$ and $\phi$ and covers the region $\aeta < 1.26$. Two further detectors, which cover the region $2.9 < \aeta < 5.0$, are specialized in measuring energetic forward hadronic showers and ensuring full geometric coverage for the transverse energy measurement. The HCAL forward (HF) sampling calorimeters use steel as absorber material and quartz fibers as the active material. Cherenkov light is emitted by the shower particles in the quartz fibers and is channeled to photomultipliers. The hadron showers' neutral components are preferentially sampled in the HF, leading to narrower and shorter hadronic showers, and is ideally suited for the forward region.

\subsection{Muon system}

The muon system is installed in the magnet return yokes of the CMS detector. The muon system's main task is to identify muons with good efficiency and reconstruct its momentum with good resolution and assign the correct charge even for high-\pt muons. It is also tasked with triggering events with muons. It is divided into a barrel detector covering $\aeta < 1.2$ and two endcap detectors, covering the region $0.9 < \aeta < 2.4$, defining three regions: the barrel region ($\aeta < 0.9$), the overlap region ($0.9 < \aeta < 1.2$), and the endcap region ($1.2 < \aeta < 2.4$). The muon system has three different types of gaseous detectors in different radiation environments. As the muon rate, the neutron-induced background rate, and the residual magnetic field are low in the barrel region, the drift tube (DT) chambers are used. In the two endcaps, the muon rate, the neutron-induced background, and the magnetic field are high, and hence the cathode strip chambers (CSC) are used in this region. Resistive plate chambers (RPC) are used in both subdetectors covering the region $\aeta < 1.6$. They are operated in avalanche mode to ensure good operation at high rates. The RPCs have double gaps with a width of 2~\mm filled with gas. They have a fast response with a good time resolution but a coarser position resolution than the DTs and CSCs and are thus used to identify the correct bunch crossing.

Figure~\ref{fig:muon} gives an overview of the muon system's layout. The barrel is divided into four stations arranged in cylinders interleaved with the iron yoke. It is also divided into five wheels along the beam direction following the five wheels of the return yokes. There are 12 layers divided into 3 Super Layers (SL) made out of four DTs layers in each chamber. The two SL measure the $r$--$\phi$ coordinate, while a third SL sandwich between them measures the $z$ coordinate. In the last muon station, there are only two SL to measure the $r$--$\phi$ coordinate. There are one or two associated RPCs in each DT chamber. The DTs have a single point resolution of $\approx 200 \mum$. The CSCs and RPCs are arranged in four disks in each endcap. They are divided into three concentric rings in the innermost station or two concentric rings in the other stations. Up to six space coordinates $(r, \phi, z)$ are measured in each CSC, and the provided spatial resolution is $\approx 200 \mum$, except for the innermost ring in the first disk, where it is $\approx 100 \mum$. The angular resolution in $\phi$ is $\sim 10~\text{mrad}$. The DTs or CSCs and the RPCs provide two independent and complementary information sources.

\begin{figure}[htbp]
  \centering
  \includegraphics[width=0.9\textwidth]{plots/chapter3/muon.png}
  \caption{Schematic view of the muon detectors.}
  \label{fig:muon}
\end{figure}

\subsection{Trigger and data acquisition system}

The online event selection process, trigger, reduces the rate to about 1~\khz. First, the detector electronics detect signals and pass them to the Level-1 trigger processors. The Level-1 trigger logic is located in a service cavern and selects events of interest. Due to the size of the CMS detector and underground caverns, the transit and time needed for a decision is about 3.2~\mus, during which the data is held in pipelined memory buffers. However, the time needed for Level-1 trigger calculations is only 1~\mus. Custom hardware processors form the Level-1 decision to keep or discard data using the calorimetry, muon systems, and correlation information. Trigger primitive objects, such as electrons or muons above a set \ET or \pt thresholds are used for the decision making. Reduced granularity and resolution data are used to form the trigger objects. Sums of \ET and \met are also employed. If a decision is made to keep the event, the data is transferred to the front-end read-out buffers. The data is further processed and compressed and placed in dual-port memories.

A processor farm is responsible for filtering the events. Each processor runs the same High-Level Trigger (HLT) code, reducing the output rate from 100~\khz to 1~\khz. Using a processor farm for event filtering allows for computer technology evolution and maximizes the flexibility in selecting data and algorithms. Whenever possible, only necessary objects are reconstructed, and only needed information of some subdetectors are used. The idea of partial reconstruction and many virtual trigger levels is the basis for the HLT. First, calorimeter and muon information is used, followed by the use of pixel tracker data. Finally, the full event information is used, including full tracking. The HLT system uses identification and isolation criteria as well as minimal energy or transverse momentum thresholds.

The data acquisition path of the ECAL sub-detector will be described in the following paragraphs. The process starts when the global trigger distributes the Level-1 decision. The data concentrator card is the main ECAL read-out unit and is responsible for collecting the crystal data. It also aggregates the trigger primitive information and performs an extensive integrity check of the data received. About 100 kB per event have been allocated for ECAL data, but the full event payload, if all channels are read-out, exceeds this target by a factor of nearly 20. The selective read-out processor can perform the reduction of the data volume. The suppression applied to a channel takes account of energy deposits in the vicinity to achieve the best resolution for large energy deposits~\cite{Almeida:2008zz}.

The calorimeter regions are classified into three groups by comparing the transverse energy deposited in each trigger tower with two configurable thresholds. Upon receiving the Level-1 decision, the information is transmitted to the selective read-out processor, determining the read-out mode that the data concentrator card has to apply to each trigger tower. There are two read-out modes: full read-out and zero suppression. With full read-out, the crystal time samples are saved without any suppression. With zero suppression, the signal's amplitude for each crystal is calculated by applying the finite impulse response filter over the digitalized time samples and compared with a configurable threshold: if the amplitude is above the suppression level, the crystal data is saved; otherwise, it is discarded.

I upgraded the selective read-out processor monitor from the legacy monitoring system and the error database to quickly diagnose re-appearing errors for the ECAL. The online monitoring system checks the status of various online electronic components of ECAL. The ECAL experts regularly use the new monitoring system to identify errors so that there is minimal data collection loss~\cite{Siddireddy:2018gxt}. Figure~\ref{fig:SRP} shows the online monitoring system in operation during data-taking period.

\begin{figure}[htbp]
  \centering
  \includegraphics[width=0.9\textwidth]{plots/chapter3/SRP.png}
  \caption{The ECAL electronics online monitoring system in operation.}
  \label{fig:SRP}
\end{figure}

\subsection{Luminosity measurement}

Luminosity is defined as the ratio of the event rate to the cross-section of a given process. It is the effective area quantifying the likelihood of a scattering event. The cross-section is measured in units of area, thus the luminosity is given in units of events per time per area, $(b.s)^{-1} = 10^{24}~\lumi$. The integrated luminosity is the integral over the instantaneous luminosity $\text{L}=\int \mathcal{L}(t) dt$. The number of expected events for a given process is given by the product of the integrated luminosity and the production cross-section, $\mathrm{N_{exp}}=\text{L} \cdot \sigma_{exp}$.

The performance of LHC is monitored in real-time using luminosity measurements and they provide an overall normalization for physics analyses~\cite{Bayatian:2006nff}. The cross-section of a given process can be estimated using a reference process, $\sigma_{\exp} = \frac{N_{\exp}}{N_{\text{ref}}} \cdot \sigma_{\text{ref}}$, where $N_{\text{ref}}$ and $\sigma_{\text{ref}}$ are the number of reference events and the reference process's cross-section. The above equation assumes that the same integrated luminosity has been used for both processes. The modern colliders use bunched beams~\cite{Tanabashi:2018oca}. If two bunches with $n_1$ and $n_2$ particles collide head-on with frequency $f$, then the instantaneous luminosity is given by:
%
\begin{equation}
  \mathcal{L}=f \cdot \frac{n_{1} n_{2}}{\sqrt{\epsilon_{x} \beta_{x}^{*} \epsilon_{y} \beta_{y}^{*}}}
\end{equation}
%
where ${x,y}$ are the beams transverse coordinates, $\beta^*_{x,y}$ are the interaction points amplitude functions, where the beam optics produce a narrow focus. Emittance is a measure of the beam width defined as $\epsilon_{x} \equiv \frac{\sigma_{x}^{2}}{\beta_{x}}$, with the root mean square of the transverse beam sizes in the horizontal or vertical direction being $\sigma_{x}$ and $\sigma_{y}$, respectively. The instantaneous luminosity can be increased by colliding bunches with a high population of low emittance at high frequency at locations where the beam optics provide low values of the amplitude functions. As the beam parameters change, the instantaneous luminosity has to be measured correspondingly.

The event rate of a reference process can estimate the instantaneous luminosity if the cross-section is known. The visible cross-section is given by: $\sigma_{\text{vis}}=\sigma(\text{E})\cdot\text{A}(t,\mu,\ldots)$, where the cross-section depends on the collision energy and the detector acceptance depends on time, the mean number of interactions per bunch crossing, and other parameters. There are independent detectors or parts of the detector for measuring the instantaneous luminosity called luminometers. Once the visible cross-section has been determined for a luminometer, the luminosity can be estimated using $\mathcal{L}=\frac{\dot{\text{N}}}{\sigma_{\text{vis}}}$, where $\dot{\text{N}}$ is the event rate for this luminometer.

The luminometer's visible cross-section should not be time-dependent nor depend on experimental conditions. Whenever the beam parameters change, the visible cross-section has to be determined again. The same detector configuration that is used during calibration has to be used during data-taking period. Thus, there are two important parts of the luminosity measurement: the event rate and the luminometer's calibration. These two parts are described in detail in the following paragraphs.

\textbf{Luminometer:} The monitoring and measurement of luminosity at CMS uses five detectors based on rate measurements~\cite{CMS:2019jhq, CMS:2018elu, CMS:2017sdi}. The luminometers used are the pixel detector, the DTs in the muon system's barrel, the HF calorimeter, the fast beam conditions monitor (BCM1f), and the pixel luminosity telescope (PLT). The PLT, BCM1f, and HF luminometers have an independent, fast read-out system. The luminometer of the pixel detector and the DTs use the standard CMS trigger and data acquisition system because they have very low occupancy and very good stability over time.

The HF is most sensitive to the hadronic showers' electromagnetic component and has a high rate of acquisition. There are two methods that have been studied to estimate the luminosity. The first method, which is referred to as zero-counting, counts the hits above the single physical towers' threshold and averages each tower's result. In the second method, the linear relationship between the total transverse energy deposit in the HF and the number of interactions and the luminosity is exploited. These two methods require that the mean value of interactions is proportional to the luminosity.

Many pixels in the CMS detector are employed for the pixel cluster counting (PCC) method. Two different tracks from the same bunch crossing have an exceedingly small probability of hitting a given pixel. The fraction of occupied pixels is less than per mille under pileup environments of 25 interactions per bunch crossing. Thus, the number of hit pixel clusters is a linear function of the number of interactions per crossing, and the number of hit pixel clusters is a good measure of the instantaneous luminosity, which is given by:
%
\begin{equation}
  \mathcal{L}=\frac{\left\langle\text{N}_{\text {cluster }}\right\rangle \cdot f}{\sigma_{\text{vis}}^{\text{PCC}}},
\end{equation}
%
where $\left\langle\text{N}_{\text {cluster }}\right\rangle$ is the mean number of hit pixel clusters, $f$ is the LHC orbit frequency, and $\sigma_{vis}^{\text{PCC}}$ the visible cross-section of the PCC method.

In the offline luminosity measurement, the pixel modules that have not been fully operational during calibration are omitted. The precision offline luminosity measurement is based on the PCC method because it has a very small dependence on pileup and other experimental conditions. The smaller statistical uncertainty of the HF measurements makes it useful for cross-checks or studies of the luminosity measurements' systematic uncertainties.

\textbf{Van der Meer (VdM) scans:} The luminosity per colliding bunch pair can be measured from machine parameters using VdM scans. The counting rate proportional to the rate of the beam-beam interaction is measured by the counter system. The luminometer rate is measured as a function of the beam-beam separation after one of the two beams is displaced, resulting in a maximum at zero separation. It is used under the assumption that the two bunch densities factorize in x and y. A dedicated LHC machine set up is required for performing these scans. The two beams are scanned through one another in the transverse plane of the detector. The luminometer rate and the luminosity must be determined at the same time to measure the visible cross-section:
%
\begin{equation}
  \sigma_{\text{vis}}=2 \pi \Sigma_{x} \Sigma_{y}\langle n\rangle_{0}
\end{equation}
%
where $\Sigma_{x}$ and $\Sigma_{y}$ are the effective beam widths and $\langle n\rangle_{0}=\frac{1}{2}\left(\text{R}_{x}+\text{R}_{y}\right)$ with normalization rates $\text{R}_x$ and $\text{R}_y$, which are the fitted scan curves amplitudes. The beam shapes are studied using the beam imaging scans. The transverse and longitudinal interaction point centroids are also determined using the VdM scan.

\textbf{Luminosity integration:} The instantaneous luminosity can be measured after the VdM scan using the luminometer's visible cross-section. The integrated luminosity is obtained by summing the luminosities of the luminosity section (LS). The LS is a convenient minimal time interval to consider for the estimation of luminosity corresponding to $t_{\text{LS}}$ ($\sim 23~\text{s}$). The average number of clusters per event is computed for each LS, and the luminosity for the LS is derived and multiplied by $t_{\text{LS}}$. The overall uncertainty in the luminosity measurement was between $2.3\%$ to $2.5\%$ for the three years. This is better than the design goal of a systematic accuracy of $5\%$.

\subsection{CMS phase-2 upgrade}

During the LHC phase-2, the High Luminosity LHC (HL-LHC) program's main objective is to deliver to the experiments a much larger dataset than during the phase-1 for new physics searches, Higgs boson coupling measurements, and precision tests of the SM. After 2026, the aim is to operate the upgraded accelerator at an instantaneous luminosity as high as $7.5 \times 10^{34}~\lumi$. This corresponds to a factor of 3 increase with respect to the phase-1 peak luminosity. This will allow the delivery of up to 4500~\fb over about 12 years of operations. A major experimental requirement for the upgrades will be to distinguish the primary interactions among the hundreds of softer collisions that will pileup ($\sim 200$) in each beam crossing. Therefore, the new systems will need high resolution to separate the trajectories and energy deposits of particles produced in these different collisions and then associate them with their correct origin. In Figure~\ref{fig:HLLHC} we can see the schedule for HL-LHC operation.

\begin{figure}[htbp]
  \centering
  \includegraphics[width=0.9\textwidth]{plots/chapter3/HLLHC.png}
  \caption{Schedule of LHC and HL-LHC operation.}
  \label{fig:HLLHC}
\end{figure}

In order to improve the event reconstruction and enhance its resilience against pileup, the CMS experiment foresees the use of per-particle time information at the level of 30~\ps precision. Combining the time information for electrons and photons with that of charged hadrons widens the physics reach for precise measurements in the Higgs sector and searches of new physics beyond the SM from simulation studies~\cite{Butler:2019rpu}. As a part of the CMS phase-2 upgrade, several projects have been proposed, including a new CMS silicon tracker, a new endcap calorimeter, and a minimum ionizing particle timing detector.

The ECAL subdetector needs to be upgraded to maintain its current performance up to the luminosity goals of the LHC phase-2. As a part of these upgrades, the front-end and off-detector electronics of the EB need to be upgraded along with the complete replacement of the EE~\cite{CMSCollaboration:2015zni}. The primary technical motivation for the upgrade of EB is the phase-2 trigger requirements to have a latency of 12.5~\mus, compared to the current 4~\mus, and a Level-1 trigger rate of about 750~\khz, compared to the current 100~\khz. In order to meet these requirements, the EB front-end cards and off-detector electronics need to be replaced.

The effect of radiation on lead tungstate crystals has been measured, and it is observed that the scintillation process remains unchanged. However, the formation of scattering color centers affects crystal transparency. The main concern for operating the calorimeter at the HL-LHC is the permanent loss of light transmission due to hadron irradiation. The studies to parametrize and predict the evolution of the crystals under the aging conditions expected at the HL-LHC have been performed at test beam campaigns over the past years~\cite{Adams:2016viv}. By the end of the LHC phase-2, the energy resolution's constant term is expected to increase to approximately 1.5\% at large $\eta$ in the EB. Thus, the CMS phase-2 upgrade will retain the EB crystals and the APDs because the constraints measured are compatible with the physics requirements.

The APDs dark current will increase significantly due to the bulk damage of the silicon caused by hadron irradiation. In order to circumvent this and reduce the APD dark current by a factor of 2, the EB operating temperature will be lowered from the nominal phase-1 $18^\circ \text{C}$ to $9^\circ \text{C}$. With this lowered temperature, it is expected that by the end of the LHC phase-2, the dark current will reach about 50~\mua (250~\MeV energy-equivalent noise) for APDs at $\aeta = 0$ and about 100~\mua at $\aeta = 1.45$. Figure~\ref{fig:APD} shows the evolution of the dark current for the two operating temperatures, along with a combined scenario where a further lowering of the temperature to $6^\circ \text{C}$ is actuated after about 1600~\fb. The cooling system currently in place will provide sufficient power to stabilize the temperature at $9^\circ \text{C}$. Only minor interventions to the distribution pipes are required to provide chilled water from the surface to the underground cavern and are currently being undertaken during the long LHC shutdown. A more complex modification to the coolant supply system is required for a further temperature reduction step and is still under evaluation.

\begin{figure}[htbp]
  \centering
  \includegraphics[width=0.6\textwidth]{plots/chapter3/APD.png}
  \caption{APD dark current shown for APDs at $\aeta = 1.45$ operated at $18^\circ \text{C}$ (red curve), at $9^\circ \text{C}$ (blue curve), and in a mixed scenario including a further step to $6^\circ \text{C}$ after the long LHC shutdown 4.}
  \label{fig:APD}
\end{figure}

The CMS phase-2 upgrade will upgrade the very front-end and front-end cards. The upgrade will maintain the mechanical structure of the cooling system and the motherboards. The motherboards are placed below the cooling structure and distribute the low voltage to the electronics and the bias voltage to the photodetectors. They also channel the APD signals to the very front-end cards. A redesigning has been done for the very front-end card to keep low noise and have a shorter pulse shaping and higher sampling rate. A trans-impedance amplifier (TIA) is selected to accomplish this. The TIA generates a voltage image of the photocurrent in the output of the APD, and only the total bandwidth of the system limits the shaping time. The dynamic range of signals up to 2~\TeV is maintained using two gains that differ by a factor of 10. A lossless data compression algorithm is used to reduce the needed bandwidth for transmitting the TIA signals digitized at 160~\mhz by 12-bit ADCs to the front-end cards.

The front-end card will use new technologies like Low Power Gigabit Transceiver and Versatile Link plus to send single crystal data sampled at 160~\mhz (compared to the current 40~\mhz) to the back-end electronics system. This removes the current requirement of having on-detector data selection and buffering. The data transmitted to the back-end electronics will be processed by the trigger primitive generation algorithm with a crystal level granularity (compared to the current trigger tower level granularity) for usage at the Level-1 trigger. This will increase by a factor of 25 the Level-1 trigger granularity. This also has added benefits in the topological rejection of signals due to direct ionization in the APDs (spikes). Powerful commercially available FPGAs and high-speed optical links will be used in the off-detector electronics. As the electronics are being upgraded, extensive algorithm development is ongoing to test novel algorithms for signal processing, trigger, and data acquisition~\cite{Ferri:2020nox}.

\textbf{Firmware Development:} I took part in the CMS phase-2 upgrade algorithm development and implemented the trigger primitive generation algorithm for ECAL's proposed Level-1 trigger. ECAL's Level-1 trigger proposal is to perform the trigger primitive generation algorithm in the off-detector electronics and use the crystal-level instead of the tower-level information to calculate the trigger primitives. There are three main stages in which the raw data is processed: linearizer, amplitude filter, and peak finder. During phase-1 trigger primitive generation, the amplified and digitized signal from 5 crystals of a strip is first linearized. The linearization takes into account the gain and corrects the digitized signal by the calibration coefficients. The amplitude of the strip pulse is measured using the amplitude filter. The filter is based on linear weighted sums, and the weights take into account the expected shape of the signal and subtract the residual pedestal dynamically. This output is then filtered by a peak finder that keeps only the maximum as a measure of the transverse energy contained in the strip.

For phase-2 trigger primitive generation, the three steps involved are kept the same, except for implementing the algorithm on crystal level digitized signal instead of implementing it on the digitized signal from 5 crystals of a strip. The input to the linearization step is the digitized signal along with linearizer coefficients. The linearizer coefficient is 24 bits in length. The first 12 bits are the pedestal, subtracted from the digitized signal to get the corrected signal. The next 8 bits are the multiplicative factor applied to the corrected signal to calibrate it. The remaining 4 bits is the bit shift applied to the calibrated signal to get the linearized signal. The amplitude of the linearized signal is estimated using the amplitude filter, which is implemented using a shift register. A shift register is a type of digital circuit using a cascade of flip-flops where the output of one flip-flop is connected to the next's input. A 4-stage shift register is used for implementing the amplitude filter, which takes as input the linearized signal along with five weights. The amplitude filter's output is given as input to the peak finder, which is implemented using a 2-stage shift register that finds the peak by comparing the current amplitude output to the previous and next output.

The main challenge for implementing the algorithm lies in the latency constraints. High-Level Synthesis is a tool that converts programs written in high-level programming languages such as C++ into hardware description languages such as VHDL. The algorithm has been implemented using High-Level Synthesis, and the synthesized code is tested on custom hardware (Virtex-7 field-programmable gate array (FPGA)). A clock cycle of $\sim 8.33~\ns$ or 120~\mhz is used for testing the initial implementation of the algorithm. The initiation interval corresponds to the time interval after which the algorithm can read the next input. As the collisions are happening every 25~\ns, the initiation interval is required to be 25~\ns as well. From the synthesis report in Figure~\ref{fig:HLS}, the initiation interval obtained after the algorithm has been synthesized is three clock cycles ($\sim 25~\ns$). Latency corresponds to the time interval from processing the digitized signal to obtaining the peak finder's output. The latency obtained after the algorithm has been synthesized is six clock cycles ($\sim 50~\ns$). The overall resource utilization in the hardware is minimal. In Figure~\ref{fig:HLS}, BRAM corresponds to block random access memory, which is not used, DSP corresponds to digital signal processing, which is used for the multiplication steps by the algorithm, FF corresponds to flip-flops, and LUT corresponds to look-up tables which are used for storage. The DSP usage is 9\%, FF usage is 14\%, while LUT usage is 19\%. This suggests that other algorithms, like multi-fit, clustering, etc., can easily be implemented on the hardware as many resources are remaining.

The synthesized VHDL code is converted into a bit file. The bit file contains all of the information necessary to properly configure the device created by our design's implementation tools. The bit file is loaded onto the Virtex-7 FPGA and has been successfully tested. The hardware's output matches the simulation output, which suggests that the algorithm is working as expected. The algorithm implementation has shown the feasibility of using High-Level Synthesis to do the CMS phase-2 upgrade's firmware development. High-Level Synthesis allows physicists like me to get involved in firmware development using object-oriented languages like C++ without knowing the nitty-gritty details of using hardware description languages like VHDL, which remains in the realm of engineers.

\begin{figure}[htbp]
  \centering
  \includegraphics[width=0.6\textwidth]{plots/chapter3/HLS.png}
  \caption{Performance estimates from High-Level Synthesis.}
  \label{fig:HLS}
\end{figure}

max_stars_repo_name: psiddire/nddiss

max_issues_repo_name: psiddire/nddiss

max_forks_repo_name: psiddire/nddiss


text:
\documentclass[t,usenames,dvipsnames]{beamer}
\usetheme{Copenhagen}
\setbeamertemplate{headline}{} % remove toc from headers
\beamertemplatenavigationsymbolsempty

\usepackage{amsmath, tikz, xcolor, bm, pgfplots, graphicx}
\pgfplotsset{compat = newest}
\usetikzlibrary{arrows, calc, decorations.pathreplacing, patterns}

\title{Hyperbolas}
\author{}
\date{}

\AtBeginSection[]
{
  \begin{frame}
    \frametitle{Objectives}
    \tableofcontents[currentsection]
  \end{frame}
}

\begin{document}

\begin{frame}
    \titlepage
\end{frame}

\begin{frame}{Intro}
The definition of a hyperbola is the same as the definition of an ellipse. The only difference is we replace the word \emph{sum} in the definition of an ellipse with \emph{difference} for a hyperbola.  \newline\\  \pause

A \alert{hyperbola} is the set of points such that the \underline{difference} of their distances from 2 fixed points (called \alert{foci}) is constant.
\end{frame}

\begin{frame}{Intro}
Just like an ellipse, the midpoint joining the foci is the \textbf{center}.    \newline\\   \pause

Whereas ellipses could appear taller or wider, hyperbolas will open up and down, or left and right. \newline\\  \pause

A key difference, however, is that hyperbolas will open left/right if the sign in front of $x$ is positive, and will open up/down if the sign in front of $y$ is positive; regardless of the values of $a$ and $b$.
\end{frame}

\begin{frame}{Intro}
\begin{center}
\begin{tikzpicture}[scale=0.7]
\begin{axis}[
axis lines = middle,
%grid,
xticklabels = {},
yticklabels = {},
xmin = -6, xmax = 6,
ymin = -6, ymax = 6
]
\draw [dashed, line width=1.25] (-1,-1) rectangle (1,1);
\draw [dashed, line width=1.25] (-5,-5) -- (5,5);
\draw [dashed, line width=1.25] (-5,5) -- (5,-5);
\addplot [blue, line width = 1.5, domain=-2.5:2.5] ({cosh(x)}, {sinh(x)});
\addplot [blue, line width = 1.5, domain=-2.5:2.5] (-{cosh(x)}, {sinh(x)});
\draw [fill=blue, color=blue] (-1,0) circle (2pt);
\node at (-1,0) [anchor=south east, color=blue, yshift=-0.15cm] {$\bm{(-a,0)}$};
\draw [fill=blue, color=blue] (1,0) circle (2pt);
\node at (1,0) [anchor=south west, color=blue, yshift=-0.15cm] {$\bm{(a,0)}$};
\draw [fill=violet, color=violet] (3.75,0) circle (2pt);
\node at (3.5,0) [anchor=north west, color=violet, xshift=-0.25cm] {$\bm{(c,0)}$};
\draw [fill=violet, color=violet] (-3.75,0) circle (2pt);
\node at (-3.5,0) [anchor=north east, color=violet, xshift=0.25cm] {$\bm{(-c,0)}$};
\draw [fill=black] (0,1) circle (2pt);
\node at (0,1) [anchor=south] {$\bm{(0,b)}$};
\draw [fill=black] (0,-1) circle (2pt);
\node at (0,-1) [anchor=north] {$\bm{(0,-b)}$};
\end{axis}
\end{tikzpicture} \newline\\
\begin{tabular}{rl}
    \onslide<2->{\textbf{Equation}  &    $\dfrac{(x-h)^2}{a^2} - \dfrac{(y-k)^2}{b^2} = 1$\\[10pt]}
    \onslide<3->{\textbf{Vertices}   &   $(h\pm a,k)$\\[10pt]}
    \onslide<4->{\textbf{Foci}   &   $(h\pm c,k)$ \\}
\end{tabular}
\end{center}
\end{frame}

\begin{frame}{Intro}
\begin{center}
\begin{tikzpicture}[scale=0.7]
\begin{axis}[
axis lines = middle,
%grid,
xticklabels = {},
yticklabels = {},
xmin = -6, xmax = 6,
ymin = -6, ymax = 6
]
\draw [dashed, line width=1.25] (-1,-1) rectangle (1,1);
\draw [dashed, line width=1.25] (-5,-5) -- (5,5);
\draw [dashed, line width=1.25] (-5,5) -- (5,-5);
\addplot [blue, line width = 1.5, domain=-2.5:2.5] ({cosh(x)}, {sinh(x)});
\addplot [blue, line width = 1.5, domain=-2.5:2.5] (-{cosh(x)}, {sinh(x)});
\draw [fill=blue, color=blue] (-1,0) circle (2pt);
\node at (-1,0) [anchor=south east, color=blue, yshift=-0.15cm] {$\bm{(-a,0)}$};
\draw [fill=blue, color=blue] (1,0) circle (2pt);
\node at (1,0) [anchor=south west, color=blue, yshift=-0.15cm] {$\bm{(a,0)}$};
\draw [fill=violet, color=violet] (3.75,0) circle (2pt);
\node at (3.5,0) [anchor=north west, color=violet, xshift=-0.25cm] {$\bm{(c,0)}$};
\draw [fill=violet, color=violet] (-3.75,0) circle (2pt);
\node at (-3.5,0) [anchor=north east, color=violet, xshift=0.25cm] {$\bm{(-c,0)}$};
\draw [fill=black] (0,1) circle (2pt);
\node at (0,1) [anchor=south] {$\bm{(0,b)}$};
\draw [fill=black] (0,-1) circle (2pt);
\node at (0,-1) [anchor=north] {$\bm{(0,-b)}$};
\end{axis}
\end{tikzpicture}   \newline\\
\begin{tabular}{rl}
    \onslide<2->{\textbf{\textit{x}-Axis}    &   Traverse Axis\\[10pt]}
    \onslide<3->{\textbf{\textit{y}-Axis}    &   Conjugate Axis  \\}
\end{tabular}
\end{center}
\end{frame}

\begin{frame}{Intro}
\begin{center}
\begin{tikzpicture}[scale=0.7]
\begin{axis}[
axis lines = middle,
%grid,
xticklabels = {},
yticklabels = {},
xmin = -6, xmax = 6,
ymin = -6, ymax = 6
]
\draw [dashed, line width=1.25] (-1,-1) rectangle (1,1);
\draw [dashed, line width=1.25] (-5,-5) -- (5,5);
\draw [dashed, line width=1.25] (-5,5) -- (5,-5);
\addplot [blue, line width = 1.5, domain=-2.5:2.5] ({cosh(x)}, {sinh(x)});
\addplot [blue, line width = 1.5, domain=-2.5:2.5] (-{cosh(x)}, {sinh(x)});
\draw [fill=blue, color=blue] (-1,0) circle (2pt);
\node at (-1,0) [anchor=south east, color=blue, yshift=-0.15cm] {$\bm{(-a,0)}$};
\draw [fill=blue, color=blue] (1,0) circle (2pt);
\node at (1,0) [anchor=south west, color=blue, yshift=-0.15cm] {$\bm{(a,0)}$};
\draw [fill=violet, color=violet] (3.75,0) circle (2pt);
\node at (3.5,0) [anchor=north west, color=violet, xshift=-0.25cm] {$\bm{(c,0)}$};
\draw [fill=violet, color=violet] (-3.75,0) circle (2pt);
\node at (-3.5,0) [anchor=north east, color=violet, xshift=0.25cm] {$\bm{(-c,0)}$};
\draw [fill=black] (0,1) circle (2pt);
\node at (0,1) [anchor=south] {$\bm{(0,b)}$};
\draw [fill=black] (0,-1) circle (2pt);
\node at (0,-1) [anchor=north] {$\bm{(0,-b)}$};
\end{axis}
\end{tikzpicture}   \newline\\
\begin{tabular}{rl}
    \onslide<2->{$\bm{c^2}$  &   $a^2+b^2$\\[10pt]}
    \onslide<3->{\textbf{Asymptotes} &   $y = \pm \dfrac{b}{a}(x-h) + k$  \\}
\end{tabular}
\end{center}
\end{frame}

\begin{frame}{Intro}
\begin{center}
\begin{tikzpicture}[scale=0.7]
\begin{axis}[
axis lines = middle,
%grid,
xticklabels = {},
yticklabels = {},
xmin = -6, xmax = 6,
ymin = -6, ymax = 6
]
\draw [dashed, line width=1.25] (-1,-1) rectangle (1,1);
\draw [dashed, line width=1.25] (-5,-5) -- (5,5);
\draw [dashed, line width=1.25] (-5,5) -- (5,-5);
\addplot [red, line width = 1.5, domain=-2.5:2.5] ({sinh(x)}, {cosh(x)});
\addplot [red, line width = 1.5, domain=-2.5:2.5] ({sinh(x)}, {-cosh(x)});
\draw [fill=red, color=red] (0,1) circle (2pt);
\node at (0,1) [anchor=south, color=red, yshift=0.15cm] {$\bm{(0,a)}$};
\draw [fill=blue, color=red] (0,-1) circle (2pt);
\node at (0,-1) [anchor=north, color=red, yshift=-0.15cm] {$\bm{(0,-a)}$};
\draw [fill=violet, color=violet] (0,3.75) circle (2pt);
\node at (0,3.75) [anchor=south, color=violet] {$\bm{(0,c)}$};
\draw [fill=violet, color=violet] (0,-3.75) circle (2pt);
\node at (0,-3.75) [anchor=north, color=violet] {$\bm{(0,-c)}$};
\draw [fill=black] (1,0) circle (2pt);
\node at (1,0) [anchor=south west] {$\bm{(b,0)}$};
\draw [fill=black] (-1,0) circle (2pt);
\node at (-1,0) [anchor=south east] {$\bm{(-b,0)}$};
\end{axis}
\end{tikzpicture}   \newline\\
\begin{tabular}{rl}
    \onslide<2->{\textbf{Equation}  &     $\dfrac{(y-k)^2}{a^2} - \dfrac{(x-h)^2}{b^2} = 1$}   \\[0.2in]
    \onslide<3->{\textbf{Vertices}   &      $(h,k\pm a)$} \\[0.15in]
    \onslide<4->{\textbf{Foci}    &   $(h, k\pm c)$}    \\
\end{tabular}
\end{center}
\end{frame}

\begin{frame}{Intro}
\begin{center}
\begin{tikzpicture}[scale=0.7]
\begin{axis}[
axis lines = middle,
%grid,
xticklabels = {},
yticklabels = {},
xmin = -6, xmax = 6,
ymin = -6, ymax = 6
]
\draw [dashed, line width=1.25] (-1,-1) rectangle (1,1);
\draw [dashed, line width=1.25] (-5,-5) -- (5,5);
\draw [dashed, line width=1.25] (-5,5) -- (5,-5);
\addplot [red, line width = 1.5, domain=-2.5:2.5] ({sinh(x)}, {cosh(x)});
\addplot [red, line width = 1.5, domain=-2.5:2.5] ({sinh(x)}, {-cosh(x)});
\draw [fill=red, color=red] (0,1) circle (2pt);
\node at (0,1) [anchor=south, color=red, yshift=0.15cm] {$\bm{(0,a)}$};
\draw [fill=blue, color=red] (0,-1) circle (2pt);
\node at (0,-1) [anchor=north, color=red, yshift=-0.15cm] {$\bm{(0,-a)}$};
\draw [fill=violet, color=violet] (0,3.75) circle (2pt);
\node at (0,3.75) [anchor=south, color=violet] {$\bm{(0,c)}$};
\draw [fill=violet, color=violet] (0,-3.75) circle (2pt);
\node at (0,-3.75) [anchor=north, color=violet] {$\bm{(0,-c)}$};
\draw [fill=black] (1,0) circle (2pt);
\node at (1,0) [anchor=south west] {$\bm{(b,0)}$};
\draw [fill=black] (-1,0) circle (2pt);
\node at (-1,0) [anchor=south east] {$\bm{(-b,0)}$};
\end{axis}
\end{tikzpicture}   \newline\\
\begin{tabular}{rl}
    \onslide<2->{\textbf{\textit{x}-Axis}       &   Conjugate Axis}    \\[0.15in]
    \onslide<3->{\textbf{\textit{y}-Axis}       &   Traverse Axis}    \\
\end{tabular}
\end{center}
\end{frame}

\begin{frame}{Intro}
\begin{center}
\begin{tikzpicture}[scale=0.7]
\begin{axis}[
axis lines = middle,
%grid,
xticklabels = {},
yticklabels = {},
xmin = -6, xmax = 6,
ymin = -6, ymax = 6
]
\draw [dashed, line width=1.25] (-1,-1) rectangle (1,1);
\draw [dashed, line width=1.25] (-5,-5) -- (5,5);
\draw [dashed, line width=1.25] (-5,5) -- (5,-5);
\addplot [red, line width = 1.5, domain=-2.5:2.5] ({sinh(x)}, {cosh(x)});
\addplot [red, line width = 1.5, domain=-2.5:2.5] ({sinh(x)}, {-cosh(x)});
\draw [fill=red, color=red] (0,1) circle (2pt);
\node at (0,1) [anchor=south, color=red, yshift=0.15cm] {$\bm{(0,a)}$};
\draw [fill=blue, color=red] (0,-1) circle (2pt);
\node at (0,-1) [anchor=north, color=red, yshift=-0.15cm] {$\bm{(0,-a)}$};
\draw [fill=violet, color=violet] (0,3.75) circle (2pt);
\node at (0,3.75) [anchor=south, color=violet] {$\bm{(0,c)}$};
\draw [fill=violet, color=violet] (0,-3.75) circle (2pt);
\node at (0,-3.75) [anchor=north, color=violet] {$\bm{(0,-c)}$};
\draw [fill=black] (1,0) circle (2pt);
\node at (1,0) [anchor=south west] {$\bm{(b,0)}$};
\draw [fill=black] (-1,0) circle (2pt);
\node at (-1,0) [anchor=south east] {$\bm{(-b,0)}$};
\end{axis}
\end{tikzpicture}   \newline\\
\begin{tabular}{rl}
    \onslide<2->{$\bm{c^2}$     &   $a^2+b^2$ }  \\[0.15in]
    \onslide<3->{\textbf{Asymptotes}    &   $y = \pm\dfrac{a}{b}(x-h)+k$}  \\
\end{tabular}
\end{center}
\end{frame}

\section{Find the center, vertices, foci, and asymptote equations for a hyperbola.}

\begin{frame}{Example 1}
Find the center, vertices, foci, and equations of the asymptotes for the hyperbola \[\frac{(x-2)^2}{4} - \frac{y^2}{25} = 1\]
\begin{minipage}{0.5\textwidth}
\onslide<2->{
\begin{tikzpicture}[scale=0.6]
\begin{axis}[
axis lines = middle,
axis line style = thick,
tick style = ultra thick,
grid,
xtick = {-4,-3,...,7},
ytick = {-6,-5,...,6},
xticklabels = {},
yticklabels = {},
xmin = -4, xmax = 8,
ymin = -6, ymax = 6,
width = 10cm,
]
\addplot [blue, line width = 1.5, domain=-5:5, smooth] ({2*cosh(x)+2}, {5*sinh(x)});
\addplot [blue, line width = 1.5, domain=-5:5, smooth] ({-2*cosh(x)+2}, {5*sinh(x)});
\addplot [mark=*, color=red] coordinates { (2,0) };
\addplot [mark=*, color=blue] coordinates {(0,0) (4,0)};
\end{axis}
\end{tikzpicture}}
\end{minipage}
\hspace{0.25cm}
\begin{minipage}{0.45\textwidth}
\onslide<3->{{\color{red}Center: $(2, 0)$}   \\[8pt]}
\onslide<4->{$a^2 = 4 \longrightarrow a = \pm 2$} \\[8pt]
\onslide<5->{{\color{blue}Vertices: $(2 \pm 2, 0)$}} \\[8pt]
\onslide<6->{{\color{blue}Vertices: $(0,0)$ and $(4,0)$}} \\[8pt]
\end{minipage}
\end{frame}

\begin{frame}{Example 1 $\frac{(x-2)^2}{4} - \frac{y^2}{25} = 1$}
\begin{minipage}{0.5\textwidth}
\begin{tikzpicture}[scale=0.6]
\begin{axis}[
axis lines = middle,
axis line style = thick,
tick style = ultra thick,
grid,
xtick = {-4,-3,...,7},
ytick = {-6,-5,...,6},
xticklabels = {},
yticklabels = {},
xmin = -4, xmax = 8,
ymin = -6, ymax = 6,
width = 10cm,
]
\addplot [blue, line width = 1.5, domain=-5:5, smooth] ({2*cosh(x)+2}, {5*sinh(x)});
\addplot [blue, line width = 1.5, domain=-5:5, smooth] ({-2*cosh(x)+2}, {5*sinh(x)});
\addplot [mark=*, color=red] coordinates { (2,0) };
\addplot [mark=*, color=blue] coordinates {(0,0) (4,0)};
\end{axis}
\end{tikzpicture}
\end{minipage}
\hspace{0.25cm}
\begin{minipage}{0.45\textwidth}
\onslide<2->{{\color{violet}Foci: $c^2 = a^2 + b^2$}} \\[8pt]
\onslide<3->{{\color{violet}Foci: $c^2 = 4 + 25$}} \\[8pt]
\onslide<4->{{\color{violet}Foci: $c = \pm \sqrt{29}$}} \\[8pt]
\onslide<5->{{\color{violet}Foci: $(2 \pm \sqrt{29}, 0)$}} \\[8pt]
\end{minipage}
\end{frame}

\begin{frame}{Example 1}
\begin{center}
\begin{tikzpicture}[scale=0.6]
\begin{axis}[
axis lines = middle,
axis line style = thick,
tick style = ultra thick,
grid,
xtick = {-4,-3,...,7},
ytick = {-6,-5,...,6},
xticklabels = {},
yticklabels = {},
xmin = -4, xmax = 8,
ymin = -6, ymax = 6,
width = 10cm,
]
\addplot [blue, line width = 1.5, domain=-5:5, smooth] ({2*cosh(x)+2}, {5*sinh(x)});
\addplot [blue, line width = 1.5, domain=-5:5, smooth] ({-2*cosh(x)+2}, {5*sinh(x)});
\addplot [mark=*, color=red] coordinates { (2,0) };
\addplot [mark=*, color=blue] coordinates {(0,0) (4,0)};
\end{axis}
\end{tikzpicture}   \newline\\
\onslide<2->{Asymptotes: $y - 0 = \pm \frac{5}{2}(x-2)$}    \\[8pt]
\onslide<3->{Asymptotes: $y = \pm \frac{5}{2}(x-2)$} \\[8pt]
\onslide<4->{$y = \frac{5}{2}x-5$ and $y = -\frac{5}{2}x+5$}
\end{center}
\end{frame}

\section{Write the equation for a hyperbola in standard form.}

\begin{frame}{Writing Equations in Standard Form}
Writing the equation of a hyperbola in standard form uses a similar approach to that for ellipses. \emph{However, the negative term will actually be \underline{subtracted} from the right side.}    \newline\\
\end{frame}

\begin{frame}{Example 2a}
Find the center, vertices, foci, and equations of the asymptotes for each hyperbola. \newline\\
(a) \quad $9x^2+162x-16y^2+64y+89=0$
\begin{align*}
    \onslide<2->{{\color{red}9x^2+162x}{\color{blue}-16y^2+64y} &= -89} \\[8pt]
    \onslide<3->{{\color{red}\text{Vertex: }(-9,-729)} \quad & \quad {\color{blue}\text{Vertex: }(2, 64)}} \\[8pt]
    \onslide<4->{9(x+9)^2 - 16(y-2)^2 &= -89 + |{\color{red}-729}| - |{\color{blue}64}|} \\[8pt]
    \onslide<5->{9(x+9)^2 - 16(y-2)^2 &= 576} \\[8pt]
    \onslide<6->{\frac{(x+9)^2}{64} - \frac{(y-2)^2}{36} &= 1} \\
\end{align*}
\end{frame}

\begin{frame}{Example 2a}
\begin{center}
    \includegraphics[scale=0.5]{ex2a.png}
\end{center}
\end{frame}

\begin{frame}{Example 2a \quad $\frac{(x+9)^2}{64} - \frac{(y-2)^2}{36} = 1$}
    \begin{align*}
        \text{Center: } & (-9,2) \\[8pt]
        \onslide<2->{a^2 &= 64} \\[8pt]
        \onslide<3->{a &= \pm 8} \\[8pt]
        \onslide<4->{\text{Vertices: } & (-9 \pm 8, 2)} \\[8pt]
        \onslide<5->{\text{Vertices: } & (-17,2) \text{ and } (-1,2)} \\[8pt]
    \end{align*}
\end{frame}

\begin{frame}{Example 2a \quad $\frac{(x+9)^2}{64} - \frac{(y-2)^2}{36} = 1$}
    \begin{align*}
        c^2 &= 64 + 36 \\[8pt]
        \onslide<2->{c &= \sqrt{10} = 10} \\[8pt]
        \onslide<3->{\text{Foci: } & (-9 \pm 10, 2)}\\[8pt]
        \onslide<4->{\text{Foci: } & (-19,2) \text{ and } (1,2)} \\
    \end{align*}
\end{frame}

\begin{frame}{Example 2a \quad $\frac{(x+9)^2}{64} - \frac{(y-2)^2}{36} = 1$}
    $a = 8 \quad b = 6 \quad \text{Center: }(-9,2)$
    \begin{align*}
        y &= \pm \frac{6}{8}(x+9) + 2 \\[8pt]
        y &= \pm \frac{3}{4}(x+9) + 2
    \end{align*}
\end{frame}

\begin{frame}{Example 2b}
Find the center, vertices, foci, and equations of the asymptotes for each hyperbola. \newline\\
(a) \quad $9y^2 - x^2 - 6x = 10$
\begin{align*}
    \onslide<2->{{\color{blue}9y^2} {\color{red}-1x^2-6x} &= 10} \\[8pt]
    \onslide<3->{{\color{blue}\text{Vertex: }(0,0)} \quad & \quad {\color{red}\text{Vertex: }(-3,9)}} \\[8pt]
    \onslide<4->{9(y-0)^2 - (x+3)^2 &= 10 + |{\color{blue}0}| - |{\color{red}9}|} \\[8pt]
    \onslide<5->{9(y-0)^2 - (x-3)^2 &= 1} \\[10pt]
    \onslide<6->{\frac{(y-0)^2}{\frac{1}{9}} - \frac{(x-3)^2}{1} &= 1} \\
\end{align*}
\end{frame}

\begin{frame}{Example 2b}
    \begin{center}
        \includegraphics[scale=0.5]{ex2b.png}
    \end{center}
\end{frame}

\begin{frame}{Example 2b \quad $\frac{(y-0)^2}{\frac{1}{9}} - \frac{(x-3)^2}{1} = 1$}
    \begin{align*}
      \text{Center: } & (3,0) \\[8pt]
      \onslide<2->{a^2 &= \frac{1}{9}} \\[8pt]
      \onslide<3->{a &= \frac{1}{3}} \\[8pt]
      \onslide<4->{\text{Vertices: } & \left(3, 0\pm \frac{1}{3}\right)}  \\[8pt]
      \onslide<5->{\text{Vertices: } & \left(3, \pm \frac{1}{3}\right)} \\[8pt]
    \end{align*}
\end{frame}

\begin{frame}{Example 2b \quad $\frac{(y-0)^2}{\frac{1}{9}} - \frac{(x-3)^2}{1} = 1$}
    \begin{align*}
      c^2 &= \frac{1}{9} + 1 \\[8pt]
      \onslide<2->{c^2 &= \frac{10}{9}} \\[10pt]
      \onslide<3->{c &= \frac{\sqrt{10}}{3}} \\[10pt]
      \onslide<4->{\text{Foci: } & \left(3, \frac{1}{3} \pm \frac{\sqrt{10}}{3}\right)} \\[10pt]
      \onslide<5->{\text{Foci: } & \left(3, \frac{1 \pm \sqrt{10}}{3}\right)} \\
    \end{align*}
\end{frame}

\begin{frame}{Example 2b \quad $\frac{(y-0)^2}{\frac{1}{9}} - \frac{(x-3)^2}{1} = 1$}
\[
a = \frac{1}{3} \quad b = 1 \quad \text{Center: }(3,0)
\]
    \begin{align*}
    \onslide<2->{y &= \pm \frac{1/3}{1}(x-3) - 0} \\[8pt]
    \onslide<3->{y &= \pm \frac{1}{3}(x-3)} \\
    \end{align*}
\end{frame}

\begin{frame}{Example 3}
Find the equation of the hyperbola with asymptotes $y = \pm 2x$ and vertices $(\pm 5, 0)$.  \newline\\
\begin{minipage}{0.5\textwidth}
\onslide<2->{
\begin{tikzpicture}[scale=0.3]
    \draw [dashed,gray] (-8,-8) grid (8,8);
    \draw[<->,>=stealth] (-8.5,0) -- (8.5,0) node [right] {$x$};
    \draw[<->,>=stealth] (0,-8.5) -- (0,8.5) node [right] {$y$};
    \draw[dashed,<->,>=stealth, thick, red] (-4,8) -- (4,-8);
    \draw[dashed,<->,>=stealth, thick, red] (-4,-8) -- (4,8);
    \draw[color=blue,fill=blue] (-5,0) circle (4pt);
    \draw[color=blue,fill=blue] (5,0) circle (4pt);
\end{tikzpicture}}
\end{minipage}
\hspace{0.25cm}
\begin{minipage}{0.4\textwidth}
\begin{align*}
    \onslide<3->{\color{blue} a &= 5} \\[8pt]
    \onslide<4->{\frac{b}{5} &= 2} \\[8pt]
    \onslide<5->{b &= 10} \\[8pt]
    \onslide<6->{\frac{x^2}{25} - \frac{y^2}{100} &= 1} \\
\end{align*}
\end{minipage}
\end{frame}

\end{document}

ext: tex


text:
% SIAM Article Template
\documentclass[]{article}

\usepackage{arxiv}

\usepackage{xcolor}
\definecolor{darkgreen}{rgb}{0.0, 0.5, 0.0}
\definecolor{amaranth}{rgb}{0.9, 0.17, 0.31}
\definecolor{azure}{rgb}{0.0, 0.5, 1.0}

\usepackage{listings}

\lstset{language=C++,
 basicstyle=\ttfamily,
 keywordstyle=\color{azure}\ttfamily,
 stringstyle=\color{amaranth}\ttfamily,
 commentstyle=\color{darkgreen}\ttfamily,
 morecomment=[l][\color{magenta}]{\#},
 breaklines=true
}

\usepackage{amsfonts}
\usepackage{amsmath}
\usepackage{xfrac}

\usepackage{hyperref}

% Information that is shared between the article and the supplement
% (title and author information, macros, packages, etc.) goes into
% ex_shared.tex. If there is no supplement, this file can be included
% directly.

% Optional PDF information
%\ifpdf
%\hypersetup{
%  pdftitle={Advanced Parallel Programming in C++},
%  pdfauthor={P. Diehl}
%}
%\fi

% The next statement enables references to information in the
% supplement. See the xr-hyperref package for details.

%% Use \myexternaldocument on Overleaf

% FundRef data to be entered by SIAM
%<funding-group>
%<award-group>
%<funding-source>
%<named-content content-type="funder-name">
%</named-content>
%<named-content content-type="funder-identifier">
%</named-content>
%</funding-source>
%<award-id> </award-id>
%</award-group>
%</funding-group>
\newcommand{\todo}[1]{{~\textbf{\color{red}[TODO: #1]}}}

\begin{document}

\title{Advanced Parallel Programming in C++}

\author{Patrick Diehl, Steven R. Brandt, and Hartmut Kaiser \\
Center of Computation \& Technology \\
Louisiana State University}

\maketitle

% REQUIRED
\begin{abstract}
 Many applied mathematics codes are based on the C++ programming language. However, many of these codes do not utilize modern C++ features, especially the features from the recent C++ 17 standard for parallel computations. These features can significantly simplify parallel computations using multiple cores. This review briefly introduces asynchronous programming facilities introduced in the C++ 11 standard and the parallel algorithms introduced in the C++ 17 standard. With these two paradigms, C++ code can be accelerated using multiple cores using the C++ standard without any need for some external special purpose libraries, like \emph{e.g.} OpenMP. The C++20 standard made additional and valuable contributions to the C++ api for parallelism and concurrency. While these may still be unavailable in some compilers, they are already implemented in HPX~\cite{kaiser2020hpx}.
\end{abstract}

% REQUIRED
%\begin{keywords}
%  example, \LaTeX
%\end{keywords}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Asynchronous programming}
\label{sec:async}
The C++11 standard~\cite{cxx11_standard} has introduced asynchronous task programming. Here, the concept of futurization using \lstinline{std::future} was introduced. A future is a placeholder for the result of an asynchronous function (launched e.g. using \lstinline{std::async()}). The runtime promises that once the computation of the task passed to \lstinline{std::async()} has finished, the future will receive the result. One can use the function \lstinline{.get()} to access the result. If the computation has finished the result will be returned. If the computation is still ongoing, the function will suspend the current thread until the future has become ready, at which point the result is returned to the user. Let us look at the following Taylor expansion for the natural logarithm
\begin{align}
    \ln(x) = \sum\limits_{n=1}^N \frac{(-1)^{n+1}}{n}(x-1)^n \textbf{.}
\end{align}
Listing~\ref{lst:async:cpp} shows the asynchronous computation of the Taylor series. First, a function \lstinline{taylor_part} that computes a part of the sum is defined in Line~\ref{lst:async:cpp:1}. In Line~\ref{lst:async:cpp:2} the sum of length $n$ is separated into two partitions of $\sfrac{n}{2}$ and each of them is asynchronously launched, and a future is returned. Since these two function launches run concurrently on two cores, we need to synchronize their execution and collect the results. In Line~\ref{lst:async:cpp:3} the \lstinline{.get()} function is called to access the result of each of the futures. Note that the code will introduce a barrier and waits until the first and second future are ready.

\begin{lstlisting}[
 basicstyle=\small\ttfamily,
 caption=Asynchronous computation of the Taylor expansion for the natural logarithm.,
 label={lst:async:cpp},
 float=htbp,
 numbers=left,
 escapechar=|,
 tabsize=2,
 numberstyle=\tiny,
 numbersep=5pt,
 belowskip=-0.8 \baselineskip,
 escapechar=|
 ]
#include <cmath>
#include <future>
#include <iostream>

double taylor_part(double x, size_t start, size_t end){ |\label{lst:async:cpp:1}|

    double result = 0;
    for(size_t i = start; i < end; i++)
    {
        result += std::pow(-1,i+1)/i * std::pow(x-1,i);
    }
    return result;
}

int main(void){

    double x = 0.5;
    size_t n = 100;

    // Launch the function async |\label{lst:async:cpp:2}|
    std::future<double> f1 = std::async(taylor_part,x,1,n/2);
    std::future<double> f2 = std::async(taylor_part,x,n/2,n);

    // Gather the result
    std::cout << f1.get() + f2.get() << std::endl;  |\label{lst:async:cpp:3}|
    return EXIT_SUCCESS;
}
\end{lstlisting}

The C++ standard library for parallelism and concurrency (HPX)~\cite{kaiser2020hpx} implements the same functionality as mandated by the C++ standard. However, additional, more advanced synchronization facilities are provided by HPX. The function \lstinline{.then()} provides a function similar to \lstinline{.get()}, however it takes a function that is only to be run after the future's value arrives. While slightly less intuitive to program, this method avoids blocking. Also, it is possible to use \lstinline{when_all()}, which waits for one or more futures. Once all futures are ready, \lstinline{when_all()} returns a future that becomes ready only after all argument futures have become ready. Listing~\ref{lst:async:hpx} uses these features.

\begin{lstlisting}[
 basicstyle=\small\ttfamily,
 caption=Asynchronous computation of the Taylor expansion for the natural logarithm using HPX.,
 label={lst:async:hpx},
 float=htbp,
 numbers=left,
 escapechar=|,
 tabsize=2,
 numberstyle=\tiny,
 numbersep=5pt,
 belowskip=-0.8 \baselineskip,
 escapechar=|
 ]
#include <hpx/hpx_main.hpp>
#include <hpx/future.hpp>
#include <cmath>
#include <iostream>

double taylor_part(double x, size_t start, size_t end){

    double result = 0;
    for(size_t i = start; i < end; i++)
    {
        result += std::pow(-1,i+1)/i * std::pow(x-1,i);
    }
    return result;
}

int main(void){

    double x = 0.5;
    size_t n = 100;
    size_t partitions = 3;
    std::vector<hpx::future<double>> futures;
    futures.reserve(partitions);

    // Launch the function async
    futures.push_back(hpx::async(taylor_part,x,1,n/3));
    for(size_t i = 1; i < partitions; i++)
      futures.push_back(hpx::async(taylor_part,x,i*n/3,(i+1)*n/3));

    // Gather the result
    hpx::when_all(futures).then([](auto&& f){
        std::vector<hpx::future<double>> futures = f.get();
        double result = 0;
        for(size_t i = 0; i < futures.size(); i++)
            result += futures[i].get();
        std::cout << result << std::endl;
        });
    return EXIT_SUCCESS;
}
\end{lstlisting}

Since the C++20 standard for coroutines has emerged, HPX futures have become awaitable. That means that the less natural style of programming using \lstinline{.then()} can be replaced by calls to \lstinline{co_await}. Listing~\ref{lst:coawait:hpx} illustrates how to use this style of programming with HPX. The call to \lstinline{co_await} suspends execution of the current thread until the value in the future is ready. Therefore, \lstinline{co_await} can return the value directly.

\begin{lstlisting}[
 basicstyle=\small\ttfamily,
 caption=Asynchronous computation of the Taylor expansion for the natural logarithm using HPX.,
 label={lst:coawait:hpx},
 float=htbp,
 numbers=left,
 escapechar=|,
 tabsize=2,
 numberstyle=\tiny,
 numbersep=5pt,
 belowskip=-0.8 \baselineskip,
 escapechar=|
 ]
#include <hpx/hpx_main.hpp>
#include <hpx/future.hpp>
#include <cmath>
#include <iostream>

double taylor_part(double x, size_t start, size_t end){

    double result = 0;
    for(size_t i = start; i < end; i++)
    {
        result += std::pow(-1,i+1)/i * std::pow(x-1,i);
    }
    return result;
}

// Can only use co_await inside a function that returns a future
hpx::future<double> get_taylor_part(double x, size_t n, size_t partitions) {
    double result = 0;
    std::vector<hpx::future<double>> futures;
    futures.reserve(partitions);

    // Launch the function async
    futures.push_back(hpx::async(taylor_part,x,1,n/3));
    for(size_t i = 1; i < partitions; i++)
      futures.push_back(hpx::async(taylor_part,x,i*n/3,(i+1)*n/3));

    // Gather the results
    auto futures2 = co_await hpx::when_all(futures);

    // Sum the results
    for(size_t i=0; i < futures2.size(); i++)
        result += co_await futures2[i];

    co_return result;
}

int main(void){
    double x = 0.5;
    size_t n = 100;
    size_t partitions = 3;
    std::cout << get_taylor_part(x,n,partitions).get() << std::endl;
    return EXIT_SUCCESS;
}
\end{lstlisting}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Parallel algorithms}
\label{sec:parallel}
A common opportunity for shared-memory parallel programming in C++ is Open Multi-Processing (OpenMP)~\cite{dagum1998openmp}. Let us look at the implementation of computing the square root of all elements in a vector $v$ of size $n$
\begin{align}
    v_i = \sqrt{v_i}, \; \forall i \in {1,\ldots,n}
\end{align}
in a parallel fashion using OpenMP (see Listing~\ref{lst:openmp}). In Line~\ref{lst:openmp:1}, a \lstinline|std::vector| is generated with length $n=10000$ and in Line~\ref{lst:openmp:2} the vector is filled with random numbers using the algorithms provided by the C++ standard library. In Line~\ref{lst:openmp:5} a \lstinline|for| loop is used to iterate over the elements of the vector. Now, the \lstinline|#pragma|-based OpenMP API is used to parallelize the \lstinline|for| loop. In Line~\ref{lst:openmp:4} the \lstinline|#pragma| to execute the loop in parallel is added. However, an additional API is needed to achieve the shared-memory parallelism.

\begin{lstlisting}[
 basicstyle=\small\ttfamily,
 caption=Example to compute the element-wise square root of a vector using OpenMP.,
 label={lst:openmp},
 float=htbp,
 numbers=left,
 escapechar=|,
 tabsize=2,
 numberstyle=\tiny,
 numbersep=5pt,
 belowskip=-0.8 \baselineskip,
 escapechar=|
 ]
#include <cmath>
#include <vector>
#include <numeric>
#include <random>
#include <algorithm>

int main(void){

    size_t n = 10000;

    // Generate the vector with length n
    std::vector<double> v(n); |\label{lst:openmp:1}|

    // Initialize the vector with -1
    std::generate(v.begin(),v.end(),std::rand);|\label{lst:openmp:2}|

    #pragma omp parallel for |\label{lst:openmp:4}|
    for (size_t i = 0 ; i < v.size(); i++) |\label{lst:openmp:5}|
        v[i] = std::sqrt(i);

    return EXIT_SUCCESS;
}
\end{lstlisting}

In the C++ 17 standard~\cite{cxx17_standard}, parallel algorithms were introduced. The shared-memory parallelism they provide is based on the Thread Building Blocks library (TBB)~\cite{10.5555/1352079.1352134}. While it is possible to write parallel code directly using the C++ standard, and there is no need to use a second API, the C++ standard library contains $69$ algorithms which work on standard containers, e.g. \lstinline|std::vector| or \lstinline|std::list|. Some versions of C++ libraries may not implement all of them. Also in C++ 17, execution policies were introduced to these $69$ algorithms, allowing them to execute in parallel. Depending on the version of your C++ libraries, this feature may also be incomplete.

In this next listing, we will revise the previous example to use parallel algorithms. See Listing~\ref{lst:for:each}. Up to Line~\ref{lst:for:each:1}, the code is unchanged, except for the new header file \lstinline|#include <execution>|, which is needed for the execution policies. In Line~\ref{lst:for:each:2}, a vector containing the indices of each vector element is generated. In Line~\ref{lst:for:each:3} the OpenMP parallel \lstinline|for| loop is replaced by \lstinline|std::for_each| from the C++ standard library algorithms. Note that this was possible before. However, the first argument of this function, the execution policy, defines that this algorithm is to be executed in parallel. The C++ 17 standard introduced the following execution policies:
\begin{itemize}
    \item \textbf{Sequential execution}:\\
    By adding \lstinline|std::execution::seq| the algorithm is executed sequentially using one thread as in the previous C++ standards.
    \item \textbf{Parallel execution}:\\
    By adding \lstinline|std::execution::par| the algorithm is executed in parallel using all available threads.
    \item \textbf{Vectorized parallel execution}:\\
    By adding \lstinline|std::execution::par_unseq| the algorithm is executed in parallel but in addition vectorization is used.
\end{itemize}
By specifying the execution policy in Line~\ref{lst:for:each:4}, the algorithm is easily parallelized. Note that this was implemented using only the C++ standard and not an external tool like OpenMP.

\begin{lstlisting}[
 basicstyle=\small\ttfamily,
 caption=Example: Compute the element-wise square root of a vector using C++.,
 label={lst:for:each},
 float=htbp,
 numbers=left,
 escapechar=|,
 tabsize=2,
 numberstyle=\tiny,
 numbersep=5pt,
 belowskip=-0.8 \baselineskip,
 escapechar=|
 ]
#include <cmath>
#include <vector>
#include <numeric>
#include <random>
#include <algorithm>
#include <execution>

int main(void){

    size_t n = 10000;
    // Generate the vector with length n
    std::vector<double> v(n);
    // Initialize the vector with -1
    std::generate(v.begin(),v.end(),std::rand);|\label{lst:for:each:1}|

    // Fill a new vector containing the indices |\label{lst:for:each:2}|
    std::vector<size_t> i = std::vector<size_t>(n);
    std::iota(std::begin(i), std::end(i), 0);

    std::for_each(|\label{lst:for:each:3}|
        std::execution::par,|\label{lst:for:each:4}|
        i.begin(),
        i.end(),
        [&](auto&& item)
        {
            v[item] = std::sqrt(v[item]);
        });

    return EXIT_SUCCESS;
}
\end{lstlisting}

Next, we will look at an example that computes the sum $s$ of all elements in a vector $v$ with $n$ elements
\begin{align}
    s = \sum\limits_{i=0}^n v_i \text{,}
\end{align}
see Listing~\ref{lst:reduce}. In Line~\ref{lst:reduce} a \lstinline|std::vector| with length \lstinline|n| is generated. The first algorithm of the C++ Standard Library \lstinline|std::fill| is used to fill all elements with negative one, see Line~\ref{lst:reduce:2}. Finally, the second algorithm \lstinline|std::accumulate| is used to compute the sum of all elements. Note that the naive approach would be to use a \lstinline|for| loop to iterate over all the elements. In Line~\ref{lst:reduce:4} the result is printed for validation. Again, one can use the execution policies to easily parrallelize this code by just adding the execution policy in Line~\ref{lst:reduce:par:2} and Line~\ref{lst:reduce:par:3}, respectively. The corresponding code is shown in Listing~\ref{lst:reduce:par}.

\begin{lstlisting}[
 basicstyle=\small\ttfamily,
 caption=Example to compute the sum of all elements in the vector using the C++ standard library.,
 label={lst:reduce},
 float=tb,
 numbers=left,
 escapechar=|,
 tabsize=2,
 numberstyle=\tiny,
 numbersep=5pt,
 belowskip=-0.8 \baselineskip,
 escapechar=|
 ]
#include <iostream>
#include <vector>
#include <numeric>

int main(void){

    size_t n = 10000;
    // Generate the vector with length n
    std::vector<int> v(n); |\label{lst:reduce:1}|
    // Initialize the vector with -1
    std::fill(n2.begin(),n2.end(),-1); |\label{lst:reduce:2}|

    // Compute the sum of all elements
    int s = std::accumulate(v.begin(),v.end(),0.0); |\label{lst:reduce:3}|

    // Output the result
    std::cout << "Result= " << result << std::endl;|\label{lst:reduce:4}|

    return EXIT_SUCCESS;
}
\end{lstlisting}

\begin{lstlisting}[
 basicstyle=\small\ttfamily,
 caption=Example to compute the sum of all elements in the vector using the parallel algorithms provided by the C++ 17 standard.,
 label={lst:reduce:par},
 float=htbp,
 numbers=left,
 escapechar=|,
 tabsize=2,
 numberstyle=\tiny,
 numbersep=5pt,
 belowskip=-0.8 \baselineskip,
 escapechar=|
 ]
#include <iostream>
#include <vector>
#include <numeric> |\label{lst:reduce:par:1}|
#include <execution>

int main(void){

  constexpr size_t n = 10000;

  // Generate the vector with length n
  std::vector<int> v(n);

  // Initialize the vector with -1
  std::fill(std::execution::par,v.begin(),v.end(),-1); |\label{lst:reduce:par:2}|

  // Compute the sum of all elements
  int s = std::reduce(std::execution::par,v.begin(),v.end(),0.0); |\label{lst:reduce:par:3}|

  // Output the result
  std::cout << "Result= " << result << std::endl;

  return EXIT_SUCCESS;
}
\end{lstlisting}

These two small examples show only a few of the parallel algorithms of the C++ 17 standard, which are experimentally supported by the GNU compiler collection (gcc) $\geq$ 9 and the Microsoft Visual Studio compiler (MVSC). However, the C++ standard library for parallelism and concurrency (HPX) implements all the defined features in the C++ 17 standard. Note that you can replace the \lstinline|std::| name space by \lstinline|hpx::| name space, since HPX strictly enhances the C++ 17 standard. We will show two small examples with new features which are not yet implemented in the major implementations. First, HPX provides some easier way to iterate over a vector without generating a vector of indices using \lstinline|std::for_each|, see Listing~\ref{lst:for:each:hpx}. Listing~\ref{lst:for:each:hpx} shows the simplification for iterating over the vector. In the previous example, we had to generate the vector of indices, which increases the memory consumption of the program. In Line~\ref{lst:for:each:hpx:2}, we use a \lstinline|hpx::for_loop| which is more like the \lstinline|for| loop where the start index is provided as the second argument and the end index as the third argument. In the lambda function, we can access the index, $i$ which will go from zero up to \lstinline|v.size()|. In addition, the parallel algorithms can be combined with the concept of futurization to obtain an \lstinline{hpx::future}. Note that this is not provided by the C++ 17 standard.

\begin{lstlisting}[
 basicstyle=\small\ttfamily,
 caption=Example to compute the sum of all elements in the vector using the parallel algorithms provided by HPX.,
 label={lst:for:each:hpx},
 float=htbp,
 numbers=left,
 escapechar=|,
 tabsize=2,
 numberstyle=\tiny,
 numbersep=5pt,
 belowskip=-0.8 \baselineskip
 ]
#include <hpx/hpx_main.hpp>
#include <hpx/algorithm.hpp>
#include <hpx/future.hpp>

#include <iostream>
#include <vector>
#include <numeric>

int main(void) {
  constexpr size_t n = 10000;

  // Generate the vector with length n
  std::vector<int> v(n);

  // Initialize the vector with -1
  hpx::ranges::fill(hpx::execution::par, v, -1); |\label{lst:for:each:hpx:1}|

  // Compute the sum of all elements |\label{lst:for:each:hpx:2}|
  hpx::future<double> f = hpx::ranges::reduce(
      hpx::execution::par(hpx::execution::task), v, 0.0);

  // Output the result
  std::cout << "Result= " << f.get() << std::endl;

  return EXIT_SUCCESS;
}
\end{lstlisting}

\section{Conclusions}
\label{sec:conclusions}

In this article we have shown that it is possible to write parallel C++ programs that take advantage of features in the C++17 and C++20 standards, both to provide task parallelism and loop parallelism (with the parallel algorithms). In addition, we've shown how to use more advanced parallel features only available in the HPX programming framework.

Moreover, we have identified two science applications which base their parallelism on this functionality. First, PeriHPX~\cite{jha2021nlmech,diehl2020asynchronous} a non-local continuums mechanics code using all these features for its shared memory implementation. Second, Octo-Tiger~\cite{10.1093/mnras/stab937} a multi-physics astrophysics code for simulating stellar mergers.

\appendix

\section*{Supplementary materials}
The source code of all the examples is available on GitHub\footnote{\url{https://github.com/diehlpk/modern-cpp-examples}} or Zenodo~\cite{}, respectively. The build system CMake is used to compile all the examples.

\section*{Acknowledgments}
I would like to acknowledge the students of the course ``Parallel Computational Math'' offered at the Mathematics Department at Louisiana State University for their feedback while teaching these modern C++ features as part of the course.

\bibliographystyle{plain}
\bibliography{references}
\end{document}

max_stars_repo_name: diehlpk/-ModernC-Paper

max_issues_repo_name: diehlpk/-ModernC-Paper

max_forks_repo_name: diehlpk/-ModernC-Paper