ATLAS Level-1 Trigger Timing-In Strategies

Abstract

The ATLAS detector at CERN's LHC will be exposed to proton-proton collisions at a bunch-crossing rate of 40 MHz. In order to reduce the data rate, a three-level trigger system selects potentially interesting events. Its first level is implemented in electronics and firmware, and aims at reducing the output rate to under 100 kHz. The Central Trigger Processor (CTP) combines information from the calorimeter and muon trigger processors, and makes the final Level-1 Accept (L1A) decision, which is transferred to all sub-detector front-ends. The functioning of the Level-1 Trigger is based on the correct timing of the signals. In this paper we present various strategies for sub-detector timing-in, in particular how to arrive at a decent initial timing setup using test-pulses in standalone mode, and in global mode with the CTP. In addition we describe how the beam pick-up detectors are a powerful tool to further refine the timing with bunches in the LHC machine. In this context we describe new developments on a proposal for precision read-out of the ATLAS beam pick-up detectors with commercial oscilloscopes in order to monitor the phase of the clock with respect to the LHC bunches.

Full text

Page 1

ATLAS Level-1 Trigger Timing-In Strategies
P. Borrego Amaral∗, S. Ask∗, N. Ellis∗, P. Farthouat∗, P. G¨ alln¨ o∗, J. Haller∗, A. Krasznahorkay∗†,
T. Maeno∗, T. Pauly∗, H. Pessoa Lima Junior‡§, I. Resurreccion Arcas∗, G. Schuler∗,
J. M. de Seixas‡, R. Spiwoks∗, R. Torga Teixeira∗, T. Wengler∗
∗CERN, Geneva, Switzerland
†University of Debrecen, Hungary
‡Federal University of Rio de Janeiro, Brazil
§Brazilian Center for Research in Physics, Brazil
Abstract
The ATLAS detector at CERN’s LHC will be exposed to
proton-proton collisions at a bunch-crossing rate of 40 MHz.
In order to reduce the data rate, a three-level trigger system
selects potentially interesting events. Its first level is imple-
mented in electronics and firmware, and aims at reducing the
output rate to under 100 kHz. The Central Trigger Processor
(CTP) combines information from the calorimeter and muon
trigger processors, and makes the final Level-1 Accept (L1A)
decision, which is transferred to all sub-detector front-ends.
The functioning of the Level-1 Trigger is based on the
correct timing of the signals. In this paper we present various
strategies for sub-detector timing-in, in particular how to arrive
at a decent initial timing setup using test-pulses in stand-
alone mode, and in global mode with the CTP. In addition
we describe how the beam pick-up detectors are a powerful
tool to further refine the timing with bunches in the LHC
machine. In this context we describe new developments on a
proposal for precision read-out of the ATLAS beam pick-up
detectors with commercial oscilloscopes in order to monitor
the phase of the clock with respect to the LHC bunches.
I. INTRODUCTION
The ATLAS Level-1 trigger [1] is a system that syn-
chronously processes information from the calorimeter and
muon trigger detectors, with a frequency of 40 MHz, cor-
responding to the frequency of bunch-crossings. The Central
Trigger Processor (CTP) [2] forms the Level-1 Accept (L1A)
and fans it out to Timing, Trigger and Control (TTC) par-
titions. In the ATLAS experiment, there are about 40 TTC
partitions, each containing a Local Trigger Processor [3], a
TTC system [4], and a busy tree which is used to throttle the
generation of L1As in case the buffers of the read-out drivers
become full.
The data of all sub-detectors are stored during the pro-
cessing time of the Level-1 trigger, and the mechanism of
retrieving them upon a L1A is based on timing only. After
the Level-1, each event fragment gets assigned identifiers
(L1ID, BCID1), which allow for asynchronous, identifier-
based processing in the subsequent High-Level Trigger. In
order to consistently read out data belonging to the same
1The L1ID is a 32-bit word, which contains the 24-bit event counter value,
and the 8-bit event counter reset counter value.
bunch-crossing it is mandatory to have perfectly timed-in sub-
detectors.
Bunches collide in ATLAS roughly every 25 ns. Since
collision products travel through the defector at nearly the
velocity of light, they travel about 7.5 m before the next
collision takes place. For a given time the detector signals in
the outer muon chambers are from an earlier bunch-crossing
than the signals detected in the inner detectors. Different
detector response times and the big spread of cable lengths
to and from the electronics cavern worsen the situation. Well-
defined procedures for timing-in the trigger and sub-detectors
are needed.
This paper describes various timing-in strategies. Emphasis
is put on stand-alone sub-detector timing-in using test-pulses,
and how the beam pick-up detectors become an essential tool
for further timing-in with beam.
II. THE DISTRIBUTION OF TIMING SIGNALS
The CTP received the bunch clock (BC) and the ORBIT
signal from the LHC machine via an interface to the TTC
[4]. The BC is a 40 MHz clock that is synchronised with the
RF frequency at 7 TeV. The ORBIT signal is a 1 µs long
pulse after each turn of the LHC (89 µs) and is used as
bunch counter reset (BCR) to synchronise the BC counters
in the sub-detector front-ends. These counters provide the
BCID, the bunch-crossing identifier, which identifies each of
the 3564 bunches (empty or filled) of the LHC bunch train.
The structure of the bunch train, especially the 2.75 µs long
gap2can be used as a reference point: the last bunch-crossing
of the long gap can be assigned BCID 0, for instance, such
that the first filled bunch has BCID 1.
Together with the L1A the CTP distributes the BC and
the ORBIT to all TTC partitions, where the Local Trigger
Processor (LTP) serves as the interface between the CTP and
the TTC partition. Since the CTP has only 20 outputs, several
partitions are daisy-chained via their LTPs. The LTP receives
the timing signals from the CTP or an LTP earlier in the
daisy chain, and makes the signals available on local outputs,
where they can be transferred to the TTC system for encoding
and transmission to the sub-detector front-ends. The LTP is in
addition an important tool for sub-detector timing-in, because
it allows to replace the CTP when running in stand-alone
2Abort gap needed to ramp up the extraction kicker magnets in case of a
beam dump.

Page 2

mode. In this mode the timing signals can be generated with
an LTP-internal pattern generator or can be fed in through
local inputs. Each signal can be independently programmed
to be taken from either the CTP link, pattern generator, or
local input. When running in stand-alone mode, the first LTP
in the daisy chain acts as master and provides all other LTPs
with triggers.
III. TYPICAL TIMING TASKS
There are 4 typical timing tasks for each sub-detector:
• Data Forming: The data must be sampled optimally with
respect to the BC clock (adjust timing of the BC phase).
• Data Alignment: The data must be aligned in time
at every input stage of processing, compensating for
differences in propagation time etc., such that all data
belong to the same event (adjust relative arrival times in
steps of 25 ns).
• BC Identification: The correct BCID has to be assigned
to the data such that all event fragments are correctly
labelled (adjust timing of bunch-counter reset (BCR)
signal in steps of 25 ns).
• Triggered BC Identification: The data read out fol-
lowing a L1A signal must belong to the bunch-crossing
responsible for the trigger (adjust timing of L1A signal
in steps of 25 ns).
While the first two tasks are sub-detector specific, the latter
two are more general, global timing adjustments.
IV. ATLAS TIMING-IN STRATEGIES
There are various scenarios at ATLAS where certain timing-
in procedures can be applied. The first step will be to time in
sub-detectors with test-pulses, which can be done stand-alone
(local mode), or with the CTP (global mode). With the test-
pulses a decent initial timing set-up can be achieved, up to a
few bunch-crossings.
When single beams become available, the beam pick-up
detectors (see section IV-B) will be used to “see” the filled
bunches. Since the CTP can be programmed to trigger on
specific filled bunches, a filled-bunch trigger can be defined.
Global BC identification is now possible. In addition, the
filled-bunch trigger can be combined with further signals, such
as the scintillation counter hodoscopes [5] in order to trigger
on beam-gas collisions.
With colliding beams a bunch-crossing trigger can be de-
fined with the beam pick-up detectors, and with combination
of the scintillation counter hodoscopes a trigger for minimum-
bias events can be obtained. At this stage the phase of the
clock can be adjusted to the phase of the incoming sub-
detector signals (data forming).
It should be noted that also cosmic-ray muons and beam-
halo muons will be used to refine the timing setup. In these
scenarios the overall timing is quite different because of the
special trajectories of the particles: They do not come from
inside the detector. Also, cosmic-ray muons do not come in
phase with the BC.
Test-Signal
Generator
dgen
dL1A
LTP
Pre-pulse
L1A
?t
Sub-detector
Fig. 1.Timing-in with test-pulses in local mode.
A. Timing-In With Test-Pulses
Many sub-detectors have the capability to generate test
signals in order to test their front-end electronics and for
calibrations. Fig. 1 illustrates how these test signals can be
used for stand-alone timing-in. A pre-pulse from the LTP
initiates the generation of the test signal. This should be done
synchronously with the ORBIT signal, which is necessary for
the next step – running in global mode. A fixed time ∆t after
the pre-pulse signal the LTP issues a L1A for reading out the
data. It is important to notice that the same time ∆t will be
used later on when running in global mode, and therefore it
is the same for all sub-detectors. It will be chosen such that
even the slowest sub-detector can cope.
The final goal is to arrive at a decent timing (up to a few
bunch-crossings) corresponding to beam-beam collisions. In
order to achieve this goal, all the propagation delays need to be
estimated upfront: The beam-beam timing should be predicted
with a simulation of the time-of-flight, detector response and
the specifics of the calibration system. All these delays should
be accounted for through a delay in the test signal generator
(or alternatively by delaying the pre-pulse signal in the LTP).
Also, the expected trigger latency and propagation of the
L1A needs to be accounted for, through delays in the L1A
generation of the LTP, in addition to the fixed time ∆t. After
having done all the preparatory work, only a short scan will be
necessary to recover the test-pulse data in the Level-1 buffer.
After timing-in the sub-detectors stand-alone, the CTP can
be used for timing-in in global mode. The CTP will in this
mode generate a L1A for a specific BCID, which is transferred
to the sub-detector front-ends through the LTP (see Fig. 2).
The pre-pulse signal is still generated within the LTP, with
the same time difference relative to the ORBIT. Nothing
has changed of the timing in the sub-detectors, so the data
will still be captured correctly. New is in global mode, that
BC identification is possible. By comparison of the BCIDs
of all the event fragments in the read out events, all the
necessary BCR offsets for consistent BC identification can
be determined. This will give a good initial timing setup for
beam-beam collisions, up to a few bunch-crossings, and leaves

Page 3

Test-Signal
Generator
L
T
P
Test-Signal
Generator
L
T
P
C
T
P
Fig. 2.Timing-in with test-pulses in global mode.
only the global timing to be established later, with beam.
B. Timing-In With Beam
With single beams the global BCR delay can be adjusted
such that the BCID corresponds to the correct bunch number
in the LHC bunch train. With colliding beams the optimal
clock phase for the sub-detector electronics can be adjusted,
and the global L1A delay can be set once the Level-1 trigger
latency is known. Crucial for the timing-in with beams will
be the beam pick-up detectors, with which filled bunches of
the LHC machine can be directly “seen”.
V. READ-OUT OF THE BEAM PICK-UP DETECTORS
The beam pick-up detectors (also called BPTX) are beam
position monitors as part of the LHC instrumentation, which
can be used by the experiments as timing reference with
respect to the bunches. For each of the 4 big experiments
there is one BPTX per incoming beam, which in the case of
ATLAS are located 175 m away on either side of the nominal
interaction point. One BPTX station consists of 4 electro-static
button electrodes (see Fig. 3 (top)) arranged around the beam
as in Fig. 3 (bottom). The read-out of the ATLAS BPTX
detectors is currently under study, and a proposal is decribed
in this paper.
Fig. 4 shows how the BPTX signals will be used by ATLAS.
The signals from the 4 buttons at each station will be combined
to one signal per station and transmitted through 200 m of
cable into the underground electronics cavern (USA15). From
there they are split to serve two purposes:
1) CTP Input: The signals are discriminated by preserving
the time information at the level of a few nanoseconds,
and fed into the CTP as trigger input signals where a
filled-bunch trigger or a bunch-crossing trigger can be
defined. Note that the time-of-flight of the bunches to
the interaction point and the cable delay from the BPTX
to the CTP will be accurately known.
With the filled-bunch trigger, gaps in the LHC bunch
train can be detected with the help of the bunch-to-
bunch scalers of the CTP MON [6], and a global BC
identification becomes possible.
2) Precision Read-Out: The signals are received by a
dedicated read-out system, which also receives clock
and orbit signals from the LHC machine through the
TTC machine interface [4]. The read-out system will
Fig. 3.
Arrangement of 4 buttons around the beam.
Top: One electro-static button electrode for the BPTX. Botton:
provide measurements of the phase between the clock
and the LHC bunches, for each of the 3564 bunches,
with an accuracy much smaller than the intrinsic time
resolution of the sub-detectors.3The aim is to achieve
about 20 ps.
The measurements will be used to check the position
of each bunch with respect to the clock, and also to
monitor the clock phase stability and to detect clock
drifts. Such clock drifts could arise from problems in
the signal chain, or temperature drifts in the optical
fibres used for transmitting the clock. The monitoring
frequency can be as low as once per minute.
The high-precision read-out system can also be used
to check for satellite bunches in neighbouring “radio-
frequency buckets”, which are 2.5 ns apart.
Fig. 5 shows the expected signal of a single button electrode
in volts on 50 Ω from a calculation for various numbers of
protons, and a nominal bunch length RMS of 250 ps which is
expected at 7 TeV. No transmission line is taken into account,
but it is expected that about 20 % of the amplitude survives
after 200 m of transmission line, with small distortion of
the signal shape, which can be calculated from the measured
attenuation spectrum. The expected signal amplitude is big
3The time resolution of the Liquid-Argon calorimeters is on the order of
100 ps. Note that the longitudinal bunch lenght RMS of a nominal LHC
bunch will be 250 ps.

Page 4

“TTC
Machine
Interface”
BC-Ref
BC-RF1
BC-RF2
BPTX
Read-out
System
(Oscilloscopes
+ Computer)
BC-Ref
Orbit
BC-RF1
BC-RF2
Electrical
BPTX1/2
Optical
Apps
Database
Configuration/
Steering
Discr.
CTP
USA15
Electrical
Orbit
200m
(few ns)
(20 ps)
Filled-Bunch Trigger
Monitoring
Fig. 4.
bunch trigger, and are read out by a dedicated read-out system for monitoring
of the clocks from the LHC machine.
The beam pick-up detector signals are fed into the CTP for a filled-
Time in nsTime in ns
-2 -2-1 -10011223344
Ω
Volts on 50
-10
-5
0
5
10
15
20
10^11
ℜ×
10^11
ℜ×
10^9
ℜ×
N = 1.15
N = 0.4
N = 5
Fig. 5.
of protons.
Signal expectation per button in volts on 50 Ω for different numbers
(20 V for high bunch intensity, 1 V for pilot bunch intensity)
and background (noise, reflections, etc.) is expected to be
small.
Without taking into account the transmission line, the
calculation yields:
uR(t) = −
t ?
−∞
dt?ZTNe
√2πσ
·t?
σ2e−t?2
2σ2e−t+t?
RC
(1)
where ZR= 1.04 Ω is the transfer impedance, R = 50 Ω and
C = 16 pF. There are 3 free parameters:
t0:Time of closest approach of the bunch to the elec-
trode.
N: Number of protons in the bunch.
σ: Bunch length RMS (Gaussian σ)
This description can be used as a fit function to the measured
signal in order to extract the free parameters t0, N, and σ.
VI. PRECISION READ-OUT OF THE ATLAS BPTX
For the precision read-out we propose to use commercial
oscilloscopes, which have the advantage of relatively low cost
(several 10 kCHF), no need to develop hardware and low-level
software, and that there will be guaranteed support from the
vendor. In addition, the signal is fully visible, i.e. there is no
signal discrimination before the read-out, which is necessary
for debugging. As most oscilloscopes usually have a maximum
of 4 channels, 2 oscilloscopes will be required for the 6 signals
seen in Fig. 4 (BPTX beam 1, BPTX beam 2, BC-Ref, BC-
RF1, BC-RF2, ORBIT).
We have performed a system test with a Tektronix
TDS 3054B [7], in order to show that this proposal is
feasible, for an estimation of the possible resolutions, and
to identify strengths and weaknesses of the read-out system.
The TDS 3054B has 4 channels, and offers a 5 GS/s real-
time sampling rate when using 1 channel, which corresponds
to a voltage measurement every 200 ps. It has a memory
deep enough to accomodate 10000 of such measurements,
corresponding to a time interval of 2 µs. The maximum
voltage on 50 Ω is 5 V (RMS) with peaks smaller than ±30 V,
and the vertical resolution is 8 bits, i.e. 0.3 % relative to the
selected maximum vertical scale.
As clock the 40 MHz clock of a TTCvx module was taken,
and as BPTX test signal the shaped output from an LTP pattern
generator was used. The pattern generator allowed to imitate
the LHC pattern of filled bunches, including the abort gap.
The oscilloscope data acquisition was triggered by the BPTX
signal itself, using the trigger hold-off to find the abort gap4.
The so acquired clock and the BPTX test signal was read out
via the built-in ethernet port using the HTTP1.1 protocol (the
oscilloscope can also be configured this way). Subsequent data
analysis was done using ROOT [8] and MINUIT [9]. Fig. 6
shows part of the read-out signals: The BPTX test signal is
seen as the top curve, where the vertical amplitude is arbitrary,
and the clock signal is the lower curve. Each point corresponds
to a single voltage measurement, and two consecutive points
are 200 ps apart. For each signal a simple discrimination is
performed: at −400 mV for the clock signal, and at the first
zero-crossing after a +200 V threshold for the BPTX test
signal. In an appropriate region a polynomial fit is used to
determine the exact time position of the discrimination. The
result of the fit is overlaid by a thin line, and the fitted time
position indicated by a vertical arrow.
For an estimation of the resolution of the time discrimi-
nation of the clock signal, the difference between two con-
secutive clock ticks is calculated for every second clock tick,
and shown in Fig. 7. A Gaussian with the correct mean value
of 25 ns is obtained, with a width of 20 ps, corresponding
to the resolution of a single measurement of the phase dif-
ference between two clock ticks. Because the steepness of
the expected BPTX signal is similar to the steepness of the
clock signal, this resolution corresponds also to the resolution
of the measurement of the phase between the clock and the
BPTX signal. This proves that 20 ps is indeed possible. The
resolution of a single time measurement on the other hand is
20 ps/√2 = 14 ps.
A full fit to the BPTX signal will not only increase the
resolution of the time measurement, but will also yield esti-
mates on the bunch intensity and length. A toy simulation was
used to estimate the resolution of this method. 1000 samples
were generated, using the parameters of the TDS 3054B
4The trigger hold-off time needs to be set to 88.92 µs−x, with x <
2.75 µs.

Page 5

Time in ns
530535 540545550 555560 565
Signal in V
-1
-0.8
-0.6
-0.4
-0.2
-0
0.2
0.4
Fig. 6.
Each point corresponds to a voltage measurement on 50 Ω every 200 ps.
Displayed is the BPTX test signal, and the BC. Polynomial fits to the signals
are performed in dedicated regions, and the result of the obtained time
measurement displayed as vertical arrows.
Detailed view of the signals read in from the scope via ethernet.
/ ndf / ndf
22
χχ
23.31 / 33 23.31 / 33
Constant Constant 4.2
ℜ±
4.2
ℜ±
139.1 139.1
Mean Mean 0.0
ℜ±
0.0
ℜ±
25 25
Sigma Sigma 0.0004
ℜ±
0.0004
ℜ±
0.0204 0.0204
Time in ns
24.8 24.85 24.924.95 2525.0525.125.1525.2
0
20
40
60
80
100
120
140
Fig. 7.
measurements. No clock tick is double counted. The resolution on the clock
difference measurement is 20 ps.
Histogram of the difference between consecutive clock time
(vertical resolution 0.2 V) and the nominal LHC bunches, and
subsequently fit with the full signal description (Eq. 1). The
resulting fit parameters follow Gaussian distributions:
t0
σ
N
=
=
=
(0 ± 2.6) ps
(252 ± 3) ps
(1.150 ± 0.015) × 1011
(2)
(3)
(4)
where t0is uncorrelated, and with a strong correlation between
σ and N of 0.68. This shows that the time resolution of a full
fit to the BPTX signal is expected to be well below 10 ps,
and even σ and N can be determined at the percent level.
With the system tests described here it becomes clear that
the precision read-out requirements for the ATLAS beam pick-
ups can be fulfilled with 2 modern commercial sampling
oscilloscopes with:
• 4 channels each
• Sampling rate ≥ 5 GS/s
• Memory deep enough to accomodate 89 µs
• Communication via ethernet
VII. CONCLUSIONS
We have presented in this paper strategies for timing in
the sub-detectors, in particular how to achieve a decent initial
timing up to a few bunch-crossings by using test-pulses, in
stand-alone mode and in global mode. For timing-in with
beam the beam pick-up detectors are very powerful: They can
be used as input to the CTP to define filled-bunch and bunch-
crossing triggers, which allow global BC identification. In
addition they allow to measure and monitor the phase between
the LHC clock and each LHC bunch to a precision of 20 ps.
ACKNOWLEDGEMENTS
We would like to thank our colleagues from the calorimeter
trigger, barrel muon trigger, end-cap muon trigger, data acqui-
sition and AB/BDI groups for their friendly and competent
collaboration without which the work presented in this article
would not have been possible.
REFERENCES
[1] The ATLAS Collaboration, “First-level Trigger Technical Design Re-
port,” CERN/LHCC/98-14, June 1998.
[2] R. Spiwoks et al., “The ATLAS Level-1 Central Trigger Processor
(CTP),” these proceedings.
[3] P.G¨ alln¨ o,
“TheATLASLocal
http://edms.cern.ch/document/374560
[4] S. Baron et al., “TTC challenges and upgrade for the LHC,” these
proceedings.
[5] M.Nessi,
“Minimumbias
http://edms.cern.ch/document/509337/.
[6] H. Lima, “User manual for the CTP Monitoring Module (CTP MON),”
https://edms.cern.ch/document/603922/1.
[7] http://www.tektronix.com/
[8] R. Brun et al., “ROOT,” http://root.cern.ch/.
[9] F. James and M. Roos, Comput. Phys. Commun. 10 (1975) 343.
Trigger Processor(LTP),”
scintillatorprojectstatus,”