 |
LHCb Event Building System
|
The task of the LHCb Event Building system is the
collection of the data fragments from the Readout Units into an entry node in
the Event Filter Farm (the Sub Farm Controller). The transport of the fragments
is achieved by means of a large Gigabit Ethernet switching network. The
rate of incoming data fragments is the nominal Level 1 trigger accept rate of 40
kHz (upgradeable to 100 kHz). The
fragments (some 60 per event) have to be concatenated into one contiguous event
at the destination, to be presented to the higher level triggers (Level 2 and
3).
In the event building project we study transport of the fragment thought the
switching network, the behavior of the switches under load, the scaling and behavior
of the network as a whole and the various possibilities of the construction of
(sub-)events at the various stages during data flow.
Data Flow
The data flow of the LHCb event building system is unidirectional from the
sources (Readout Units) to the destinations (Sub-farm Controllers). There is no synchronization
between sources. Every source sends data as soon as it is ready to do so. The
destination assignment is static and based on the event number. Data are thus
pushed through the system at every stage. The only feed-back in the system is
some back-pressure in the network to cope with temporary, localized network
contention.
Protocol
The protocol is adapted to the two basic features of the event building
system: 1) pure push-through, 2) high frequency. It is a custom very light
protocol implemented on top of Ethernet. It consists of a header and eventually
of a trailing error block to collect errors (mostly time-outs), which occurred
during the transport.
Fragment Assembly
At the time 3 possibilities to concatenate the event fragments into one
contiguous event are under study:
- Smart NICs
- Network Processors
- The host CPU of the Subfarmcontroller
- Event Building using Smart NICs:
Powerful Network Interface Cards often have a general purpose CPU for higher
level protocol functions. Normally this used to ease the burden on the host
CPU. In our case we use as specific NIC (from Alteon) to implement the full
event building algorithm and "to DMA" only complete contiguous
events into the host, with a rate of some 400 to 1000 Hz
- Event Building using Network Processors:
Network processors are an exciting new development in the networking
hardware market. They are intended to serve as front-end for very high-speed
network connections mostly in switches. Their general purpose CPUs enable
them to perform complex protocol handling, check-summing, frame alterations
etc.. Since they are software programmable they can quickly adapt to new
trends or protocol extensions. We have used one of these processor (the IBM
NP4GS3) to implement event-building on it. One could for example
envisage to have the event building done in the final stage of a switching
network, which is based on NP4GS3s
- Measurements of Switch Parameters:
The behavior of the switch is under various circumstances and traffic
patterns, is one of the most important basic features of the system. Our
measurements are done on a particular switch (Foundry
Networks Fast
Iron 4000) and focus
on understanding latencies and flow-control behavior. An overview of
the software can be found here.
- Simulation of the Switching Network:
Since it is not possible to build a full scale prototype of the switching
network, simulation must tell us in particular about the scaling behavior of
the switching network, partially using the parameters obtained from the
measurements we have done on existing switches.
- Studies of Optimized Network Topologies:
Our main data-collection network will most likely consist of several
switches, since a single switch for so many ports (about 200) does either
not exist or is not cost-effective. Our asymmetric data-flow makes it very
tempting to try to make use of the total bandwidth installed. A Gigabit
Ethernet Network consists of full-duplex point-to-point connections. Only
one direction is really used on each link for data transfer, at least in a
"simple & stupid" way of connecting the switch modules. A
smarter way tries to share the load by mixing sources (RUs) and destinations
(SFCs) on the modules, to take advantage of the full bandwidth in one module
(=sub-switch). One such topology is shown here.
- "The LHCb Event Building
Strategy" - Presentation at the IEEE NPSS Real Time 2001, June 4 -
8 Valencia, Spain: N. Neufeld [ppt,
pdf]
Publications
- "The LHCb Event Building Strategy" - Paper presented at
the IEEE NPSS Real Time 2001, June 4 - 8 Valencia, Spain: J-P. Dufey, B.
Jost and N. Neufeld [pdf]
(this is mostly about optimized network topologies)
- "Event Building in an Intelligent Network Interface Card for the LHCb
DAQ System" - Paper presented at the DAQ 2000 Workshop in Lyon: N.
Neufeld [pdf]
- "Embedded Event Building on a Gigabit Ethernet Network for the
LHCb Experiment", Marianna Zuin, Tesi di Laurea, Università Ca'
Foscari - Venezia, 2000 [pdf] (1460 kB)
Net processor makers race toward 10-Gbit/s
goal (from EETimes), NPUs seek out software
(EETimes),
The
10 Commandments - essential reading ;-) ,
"Carrier Sense multiple access with collision detection (CSMA/CD) access
method and physical layer specifications" (aka Ethernet :-))
IEEE Standard 802.3 (2000 version)
"PCI Local Bus Specification Revision 2.2, Dec. 1998" [pdf]
"Tigon/PCI Ethernet Controller Revision 1.04", Databook [pdf],
"Tigon Firmware 12.3.11" [pdf]
"IA-32 Intel Architecture Software Developer's Manual", Volume
1 [pdf], Volume 2 [pdf],
Volume 3 [pdf]
