Speakers
Description
Modern particle physics detectors contain many readout channels that continuously produce large amounts of data. These data must be transferred from back-end electronics to readout servers with high bandwidth and low latency while keeping CPU processing and memory-copy overhead low. This work presents an FPGA-based RoCEv2 protocol stack designed for detector data readout. The protocol stack uses RDMA to write detector data directly into registered server memory. Versions for 10, 40, and 100 GbE have been developed. The main functions include RoCEv2 packet generation, RDMA Send and Write operations, packet sequence management, iCRC calculation, and data integrity checking. The 10/40 GbE version also supports multiple queue pairs, fault isolation, retransmission, and error recovery. These functions are being added to the 100 GbE version.
The protocol stack was tested on several FPGA platforms with commercial RNICs. The 100 GbE implementation on a Xilinx VCU118 achieved approximately 93.5 Gbit/s in single-QP RDMA Write tests. Preliminary tests also measured a point-to-point latency below 3 μs. The 40 GbE implementation reached 37.4 Gbit/s, corresponding to 93.5% link utilization, and was ported to the ZCU102 platform with similar performance. Its use in a detector readout system was tested during BESIII Main Drift Chamber beam and cosmic-ray runs, where a single readout node sustained a data rate of 13.4 Gbit/s while transmitting full detector data without zero suppression.
The results show that the FPGA-based RoCEv2 protocol stack can work with commercial RNICs and provide near-line-rate transmission for particle physics detector readout. Possible future applications include the CEPC data acquisition system and other high-rate detector systems. Future work will add multi-QP and reliability support to the 100 GbE version, test multiple readout nodes, and develop a remote circular buffer for continuous and reliable data transfer.
