Highest Acceleration in the CERN Data Network Using Fibre-optic Components from tde

The European Organization for Nuclear Research, CERN, produces a huge amount of data every day. Billions of secondary particles per second sweep through the detectors of the largest particle accelerator LHC (Large Hadron Collider) alone. Archiving these vast quantities of data is an essential function at CERN. This flood of data could not be managed without a highly available high-performance data network and a high-end computer centre. To this end, the CERN engineers joined with their partner and networking specialist tde to develop a customised and highly compact cabling system with multi-mode fibre optics. This will guarantee reliable high-speed data transfer for future requirements.

The CERN laboratory

CERN, the European Organization for Nuclear Research, sits astride the Franco-Swiss border near Geneva. The Organization has about 2,500 staff members and is the largest and most advanced research centre for particle physics. More than 12,000 visiting scientists from 105 nationalities perform experiments and conduct fundamental physics research there. The scientists mainly focus on exploring the composition of matter using large particle accelerators.

More Speed in the Data Network

After CERN had decided to expand its existing fibre-optics infrastructure, computer centre engineers had to find a suitable solution. CERN wanted to upgrade the existing infrastructure to increase the data transfer rate between the installed components in the computer centre in order to be ready for current and future requirements. “Fibre-optic connections are one optimal candidate to satisfy the requirements for higher and higher data transfer rates,” explains Stefano Meroli, engineer responsible for the installation of the new optical fibre infrastructure. “We already had installed multi-mode connections with trunk cables, MPO connectors and optical modules in the past. However, the current needs have required a better structured cabling solution,” adds Stefano Meroli.

The Bar Was Set Very High

In order to guarantee those high data transfer rates, the new fibre-optic components had to be high-quality and fail-safe. A maximum reliability over a long life cycle can only be provided by using high-quality network components. Substandard components may end up impacting the reliability of the whole Data Centre. In a first step, the new multi-mode cabling solution was planned in a modular way for a data rate of 10 Gbit/s. However, the goal was to be able to quickly and simply migrate it to higher transfer technologies using 40 or 100 Gbit/s without the need to perform radical changes in the infrastructure. Finally, the targeted solution needed to be customisable to CERN’s technical requirements.

tde Made the Race

As a first step, the CERN engineers defined the technical requirements for the new cabling solution. In a comprehensive study, they then determined the network topology as well as, the type and scope of the fibre-optic components. Once CERN had defined its wishes and requirements for the new fibre-optic cabling, the last thing the research institute had to do was to find a suitable partner for its implementation. CERN signed a support frame contract, of which tde – trans data elektronik – was the sub-supplier in charge of providing fibre-optic components such as optical fibre trunk cables and terminal equipment. tde is a networking specialist based in Dortmund, Germany. The company had already successfully supplied pre-fitted optical fibre cables and terminal hardware (optical modules and subracks) during previous support frame contracts for other CERN projects.

Step by Step Towards a High-Performance Data Network

As a first step, CERN established a detailed analysis of the needs and delivered the list of all the requested fiber-optic components to be produced to tde. The consortium partners under the supervision of the CERN engineers then successfully installed the pre-assembled trunk cables with multi-mode OM4-class fibres plus optical modules and rack mount enclosures provided by tde. The bending-insensitive trunk cables with OM4 fibres feature a magenta-coloured jacket and six MPO connectors on either end. Each connector consists of 12 fibres. The selected MPO connectors are MTP-type multi-fibre connectors equipped with Elite ferrules. The optical fibre cables had to meet the Ethernet standard IEEE 802.3ba adopted in June of 2010. The following table summarises the requirements for an entire link with 40 Gbit/s and 100 Gbit/s data transfer rates and at 850 nm wavelength according to this standard.

At the same time, tde had to consider that a complete link consists of at least two modules and one trunk cable, which makes four plug connections per link. As network technicians patch several links together in today’s computer centres like at CERN, up to eight or even more plug connections may be present in a full link. Consequently, the loss of each plug connection has to be low. tde offers MPO components with very good performance values beyond the common standards.

Customised Modules

“CERN wanted some modifications regarding the distribution technology,” says André Engel, tde’s managing director. “They wanted drawer-type rack mount enclosures and modules that can be fixed without using tools.” In order to provide such drawer-type enclosures, tde had to reduce the packing density from eight down to six modules. As CERN was planning to use the modules and rack mount enclosures also out in the field, close to the particle accelerator, the research institution needed a nonmagnetic version. Therefore, tde had to manufacture the modules from stainless steel. “Basically, we developed a special cabling-system platform just for CERN. We even developed the configuration from scratch and in close co-operation with CERN,” explains André Engel. This was a clear and lasting success. From this customer-specific solution, the networking expert subsequently developed the configuration type of a new product, which is now part of its portfolio: “tML Xtended.” CERN installed the modules in 19" tBG2 subrack enclosures on 3U with a 1U guide channel. The front panel is made of aluminium.

The following illustration (figure 1) shows how the MPO connectors are attached to the modules in the CERN computer centre. For transfer rates of 10 Gbit/s, the connections are implemented via OTM (Optical Terminal Modules). These modules divide the 24 fibres of the two MPO connectors into 24 single-fibre LC/PC connectors on the front panel. For data rates of 40 Gbit/s or 100 Gbit/s, connections may be made via partial front panels using MPO “Key-up-to-Key-up” terminal plates. In this case, the trunk cables are connected directly to the active components via MPO multi-fibre patch cords.

The Infrastructure: Under a Lucky Star

CERN chose a star topology for this project, which means that all transfer stations are directly connected to a central node making the layout look like a star. At the star point side, the connections are set up as 19" standard subrack enclosures (3U) with a 1U guide channel. The star point is connected to each target rack within the server room via one cord. The destination racks are connected via a 19" rack mount enclosure with 1U. Each 19" rack mount enclosure (1U) can house up to three modules featuring 24 LC/PC connectors on its front panel (see figure 2) or MPO terminal plates for the attachment of the multi-fibre connectors.

Figure 3 shows the installation within a star point rack. Two of the six single cords of a trunk cable are connected to one OTM. The remaining four single cords are spare cords that are stored in the guiding channel for future upgrades. Blind plates are used to reserve the space to install OTM or terminal plates. The trunk cables were installed in available cable trays in the raised floor (see figure 4). Before handover to CERN, the losses were tested making sure that they were compliant with the CERN specification.

Planned For Years Ahead

“tde implemented the solution within the agreed timescale and achieved the required optical performance, mechanical design and high quality,” concludes Stefano Meroli. “Quality” was a very important aspect. CERN and tde were able to guarantee the highest quality standards by performing precise quality-assurance testing – at first during manufacturing on the tde sites, and then during and after installation on the CERN premises.

CERN is planning for the years ahead: using the new modular cabling system, the research institution currently reaches data transfer rates of 10 or even 40 Gbit/s and is perfectly prepared for future upgrades to 100 Gbit/s. In addition, CERN will benefit from the high-quality and fail-safe components assuring a long life cycle for this infrastructure.

Picture Selection: