Internet to power particle physics grid research

Scientists are on the verge of opening the Large Hadron Collider, sited on the French-Swiss border, which will use ultra-powerful magnets to race proton beams around a 27km circular underground tunnel and smash them into each other 40 million times a second.

Besides being awesome, these collisions will produce tiny particles not seen since just after the Big Bang and perhaps will enable scientists to find the elusive Higgs boson, which — if theories are correct — endows all objects with mass. The Large Hadron Collider may also help scientists figure out why all the matter in the universe wasn't destroyed by anti-matter.

Perhaps just as complicated as answering these questions of origin, however, is setting up a worldwide network capable of distributing the mountains of data produced by the seemingly infinite number of particle collisions. The Worldwide LHC Computing Grid was set up to perform this task. Data will be gathered from the European Organization for Nuclear Research (CERN), which hosts the collider, and distributed to thousands of scientists throughout the world.

One writer described the grid as a "parallel internet". Ruth Pordes, executive director of the Open Science Grid, which oversees the US infrastructure for the LHC network, describes it as an "evolution of the internet". New fibre-optic cables with special protocols will be used to move data from CERN to 11 Tier-1 sites around the globe, which in turn use standard internet technologies to transfer the data to more than 150 Tier-2 centres.

"It's using some advanced features and new technologies within the internet to distribute the data," Pordes says. "It's advancing the technologies, it's advancing the [data transfer] rates, and it's advancing the usability and reliability of the infrastructure."

The data is first produced in the collisions, which occur in caverns 100m underground. If all goes according to plan, the first proton beams will be injected into the LHC around mid-June, and will start smashing into each other about two months later.

When proton beams collide and produce new particles, data will be read from 150 million sensors and sent to a counting room where signals are filtered. The interesting data, or "raw data", is what remains, according to CERN.

Raw data is sent over dedicated 10Gbit/s optical fibre connections to the CERN Computer Centre, which is known as "Tier-0" in the LHC Computing Grid. Here, raw data is sent to tape storage and also to a CPU farm which processes information and generates "event summary data". Subsets of both the raw data and summaries are sent to the 11 Tier-1 sites around the world.

Each of the 11 Tier-1 centres are connected to CERN via a dedicated 10Gbit/s link, and the Tier-1 centres are connected to each other by a general purpose research network. Each Tier-1 centre receives only certain subsets of information.

The Tier-1 centres are responsible for reprocessing raw data, which is then kept on local disk and tape storage and distributed to Tier-2 centers, which are located in most parts of the world.

Tier-2 centers are connected to Tier-1 sites and each other by general purpose research networks, such as the US Department of Energy's Energy Sciences Network. Tier-2s are located mainly in universities, where physicists will analyse LHC data. Ultimately, about 7,000 physicists will scrutinise Large Hadron Collider data for information about the origins and makeup of our Universe, according to CERN.

The LHC collisions will produce 10 to 15 petabytes of data a year, says Michael Ernst of Brookhaven National Laboratory on Long Island, a Tier-1 site. Ernst directs the programme that will distribute data from the ATLAS detector. Brookhaven will be responsible for filtering data so it can be easily readable by scientists located at the more numerous Tier-2 facilities, Ernst says.

Brookhaven has about 1,200 multicore x86 servers dedicated to the LHC, along with disk and tape storage that holds seven petabytes of data. Ernst says Brookhaven will have to scale that storage up significantly by 2012, when he expects to be storing 13 petabytes of Large Hadron Collider data.

Worldwide, the LHC computing grid will be comprised of about 20,000 servers, primarily running Linux. Scientists at Tier-2 sites can access these servers remotely when running complex experiments based on LHC data, Pordes says. If scientists need a million CPU hours to run an experiment overnight, the distributed nature of the grid allows them to access that computing power from any part of the worldwide network, she says. With the help of Tier-1 sites such as Brookhaven, the goal is to make using the grid just as easy for universities as using their own internal networks, according to Pordes.

Asked if the LHC project is the most complicated thing he's ever worked on, Ernst gave a quick laugh and said, "Yeah, I would say so".