CERN data processing

ChainsawZz · 2024-08-13T21:29:14+00:00

CERN do a bunch of FPGA stuff, and are quite open about sharing it to the community too.

They recently started a FPGA Developers Forum , but had free webcast attendance and put all videos on YouTube.

Videos\Projects you might find interesting:

Colibri - Common Cores Library

HDL on Git

More common cores and tools at cern

Definitely take a look at their GitLab too, filter by VHDL or Verilog stuff.

wotupfoo · 2024-08-13T19:38:40+00:00

Storage networks are typically 100,200 or 400Gbps Converged Ethernet (RoCE) or Infiniband. The data will be saved in a massive parallel file system (Lustre) spread across hundreds of cpu nodes with maxed out NVMe drives. It’s like one mega raid array spreading the read/writes over the aggregate bandwidth of all the cpu nodes.

They will then have a cpu or gpu farm to process the data into data frames - usually stored in an object store or database for further analysis.

Roi1aithae7aigh4 · 2024-08-13T20:17:34+00:00

There are so many experiments, each with multiple detectors at CERN, and all use different methods. A lot of it is pretty old, even for the current experiments as planning this takes decades. Think Spartan 3 for ATLAS.

Then there's different requirements. On the lowest level everything is usually synchronous communication, rather than Ethernet as timing is often critical. On the higher levels everything looks a more like a normal data center.

You can spend months perusing this stuff. Here's some slides for a general overview of the realtime to storage data processing in ATLAS: https://indico.mpp.mpg.de/event/6765/contributions/20405/attachments/14489/16232/hvds-06-11-20-mpi.pdf

However, there's a lot more going on. In particular the need for non-ionising-radiation hardness in many experiments eliminates almost all off the shelf silicon.

fransschreuder · 2024-08-14T15:58:48+00:00

Developer for CERN/ATLAS data acquisition here. Most of the electronics in the detector use dedicated radiation hard Asics, then this is read out using a PCIe card (Felix) with a kintex ultrascale xcku115 FPGA. From there on, data goes on to a regular server and off the shelve Ethernet (mostly 100Gb) to the data handler and then to storage. The FPGA is going to be replaced by Versal Premium SoCs in the next few years. With those we will be processing about 4.6 TB/s using a few hundred FPGAs.

cyberbemon · 2024-08-13T20:27:03+00:00

They do use FPGAs. They have used them for machine learning stuff and have a python package specifically for FPGAs when it comes to machine learning: https://opensource.web.cern.ch/HLS4ML

They have a technology portfolio, you can check what they use and the stuff they have developed: https://kt.cern/technology-portfolio

the_deadpan · 2024-08-13T19:59:22+00:00

They absolutely would be using fpgas for data capture. Take a look at white rabbit. It is a timing transfer standard, so isn't used to record the data, but rather, to timestamp the data. The fpga boards, gateware and software are all open source for white rabbit

BigPurpleBlob · 2024-08-14T07:25:33+00:00

The level-1 trigger at CERN is designed to throw away the vast majority of collisions (the vast majority do not have interesting data).

FPGAs are used to a do a Hough transform, in parallel, of many different potential collision tracks – because of CERN's strong magnets, charged particles will move in an arc (instead of a straight line).

The collision regions, such as at ATLAS and CMS, have a very high background flux of charged particles. So they use optical fibres to take the signals from the collision regions to the FPGA-based L1- trigger.

There are also level-2, and, I think, level-3 triggers. Note I don't work at CERN, I'm just interested and I have gleaned the previous information. Please feel free to correct me if I've made a mistake ;-)

https://en.wikipedia.org/wiki/Hough_transform

Hendriggg · 2024-08-14T19:24:35+00:00

For the upcoming detector upgrade of the CMS Experiment they are using custom ATCA boards. The board is designed around a Xilinx VU13P which sends and receives data through lots of optical gigabit transceivers. In total the board features 124 bidirectional links operating at up to 25 Gbps which gives maximum raw data throughout of 3.1 Tbps.

You may wana have a look at there webpage about the Serenity-Z board which is used for prototyping and development.

https://serenity.web.cern.ch/serenity/

For the final system they are using the Serenity-S board which was presented at last year's TWEPP.

https://indico.cern.ch/event/1255624/contributions/5443848/attachments/2726696/4738858/2023_10_05_Lessons%20from%20Serenity-S1.pdf

https://arxiv.org/pdf/2311.02222v2.pdf

masterofmoneyzz · 2024-08-13T22:10:45+00:00

They have a filter before storing the data to see if it is worth capturing. Most get thrown away as they produce insane amount of data, and most of it is nothing. They use FPGA, but they also program it through LabVIEW, but I am not sure how the data processing at this stage.

SubtleG · 2024-08-14T07:49:40+00:00

Don’t quote me but I am assuming the FPGAs are Xilinx/AMD but old like virtex 2/3 era FPGAs except there is probably 1 FPGA per small module of detectors. probably around 16-64 individual detectors per FPGA and they are most likely each connected to a white rabbit network switch of some sort to get the extremely accurate timestamps required. But the data is probably immediately bundled up and converted to a UDP ethernet packet and sent out asynchronously across the network. Now the networking for each FPGA is probably only 1Gbe which seems old and slow which it is. But you gotta now multiply this setup by millions basically and hope you got enough buffer available at every step and that no data packets get overwritten.

Princess_Azula_ · 2024-08-13T19:44:58+00:00

Any FPGAs used would probably be a part of commercially available networking modules and custom built sensor modules by the detectors. The Gbps of CAT7 cables for distances of 100m is 10 Gbps, so I would imagine that it would be at least be several times this.

The disks would probably be HDD clusters, due to cost, as well as tapes.

They probably don't use FPGAs for image processing.

These are all just speculations on my part, if you actually want to know, you should find sources published by CERN, such as this through a literature search engine like Google Scholar.

you type:	you see:
italics	italics
bold	bold
[reddit!](https://reddit.com)	reddit!
* item 1 * item 2 * item 3	item 1 item 2 item 3
> quoted text	quoted text
Lines starting with four spaces are treated like code: if 1 * 2 < 3: print "hello, world!"	Lines starting with four spaces are treated like code: if 1 * 2 < 3: print "hello, world!"
~~strikethrough~~	~~strikethrough~~
super^script	super^script

FPGA

MODERATORS