Johannes Lehrbach from Volker Lindenstruth's FIAS research group defended his PhD Thesis on July 5th, 2024. During his PhD, he was instrumental in building the computing infrastructure for the Event Processing Nodes (EPN) of the ALICE experiment at CERN.

The ALICE experiment at the CERN Large Hadron Collider (LHC) was upgraded for run 3 to record Lead-Lead collisions at higher interaction rates. For the upgrade the central detectors got new sensors, new readout electronics, as well as a new computing paradigm. Due to a continuous readout of the detectors during run 3 the resulting data rates increased by more than an order of magnitude, compared to run 2. This had a direct impact on the required network, the computing resources, as well as the overall infrastructure.

During Lehrbach's thesis, a new data center was built to facilitate higher rack power densities at fast-changing loads while reliably providing the required cooling. The new data center enabled more powerful hardware, particularly systems accelerated by multiple GPUs. The InfiniBand network for run 3 was tailored to the ALICE data rates to reliably provide the required throughput while keeping costs in check. The EPN cluster and server configuration were optimized to run the ALICE software and maximize performance. The cluster was expanded twice to facilitate additional compute requirements, which were unknown initially. The whole system, therefore, proved good scalability.

The computing infrastructure, including the network implemented during this PhD, is now in production during ALICE run 3. The work was essential for ALICE's success, enabling data taking of roughly 50 kHz Lead-Lead collision rate and fulfilling initial design specifications. The EPN cluster provided sufficient computing to enable complete online TPC reconstruction with almost 800 Gigabytes per second of incoming detector data. The resulting compressed data was reliably transferred from the InfiniBand network of the EPN cluster toward the Ethernet network of the disk storage with close to 200 Gigabytes per second.

The doctor's hat for Johannes Lehrbach, designed by the FIAS/CERN team, fascinated those present: It consisted of a small data centre with interesting features; for example, the mobile phone could display active status messages.

more information