40 and 100 Gigabit Ethernet set to take off

IT managers who are getting started with — or even pushing the limits of — 10 Gigabit Ethernet in their LANs and datacentres won't have to wait long for higher-speed connectivity.

Pre-standard 40 Gigabit and 100 Gigabit Ethernet products, including server network interface cards, switch uplinks and switches, are expected to hit the market later this year. And standards-compliant products are expected to ship in the second half of next year, not long after the expected June 2010 ratification of the 802.3ba standard.

The IEEE, which began work on the standard in late 2006, is expected to define two different speeds of Ethernet for two different applications: 40G for server connectivity and 100G for core switching.

Despite the global economic slowdown, global revenue for 10G fixed Ethernet switches doubled in 2008, according to analyst firm Infonetics. And there is pent-up demand for 40 Gigabit and 100 Gigabit Ethernet, says John D'Ambrosia, chair of the 802.3ba task force in the IEEE and a senior research scientist at Force10 Networks.

"There are a number of people already who are using link aggregation to try and create pipes of that capacity," he says. "It's not the cleanest way to do things...(but) people already need that capacity."

D'Ambrosia says even though 40/100G Ethernet products haven't arrived yet, he's already thinking ahead to Terabit Ethernet standards and products by 2015. "We are going to see a call for a higher speed much sooner than we saw the call for this generation" of 10/40/100G Ethernet, he says.

According to the 802.3ba task force, bandwidth requirements for computing and core networking applications are growing at different rates, which necessitates the definition of two distinct data rates for the next generation of Ethernet. Servers, high performance computing clusters, blade servers, storage-area networks and network-attached storage all currently make use of 1G and 10G Ethernet, and use of 10G grew significantly in 2007 and 2008.

I/O bandwidth projections for server and computing applications, including server traffic aggregation, indicate that there will be a significant market potential for a 40G Ethernet interface, according to the task force. Ethernet at 40G will provide approximately the same cost balance between the LAN and the attached stations as 10G Ethernet, the task force believes.

Core networking applications have demonstrated the need for bandwidth beyond existing capabilities and beyond the projected bandwidth requirements for computing applications.

Switching, routing, and aggregation in datacentres, internet exchanges and service provider peering points, and high bandwidth applications such as video on demand and high performance computing, need a 100 Gigabit Ethernet interface, according to the task force.

"Initial applications (of 40/100G Ethernet) are already showing up, in stacking and highly aggregated LAN links, but the port counts are low," says George Zimmerman, CTO of SolarFlare, a maker of Ethernet physical layer devices.

Zimmerman says 10G is just now taking off in the access layer of large networks and will eventually move to the client side, creating the need for 40/100G in the distribution layer and the network core.

He says the application of 100 Gigabit Ethernet in the core is imminent, and is about two years away in the distribution layer. "Both will be driven by and drive 10G adoption in the access and client end of the network, where today the numbers are still much smaller than the potential," he says.

The 802.3ba specification will conform to the full-duplex operating mode of the IEEE 802.3 Media Access Control (MAC) layer, according to the task force.

By employing the existing 802.3 MAC protocol, 802.3ba is intended to maintain full compatibility with the installed base of Ethernet nodes, the task force says. The spec is also expected to use "proven and familiar media", including optical fibre, backplanes and copper cabling, and preserve existing network architecture, management and software, in an effort to keep design, installation and maintenance costs at a minimum.

Even though the 802.3ba standard is not expected to be ratified until June 2010, initial interoperability testing will commence later this year, says Brad Booth, chair of the Ethernet Alliance.

The specification and formation of the 40/100G task force did not come without some controversy, however. Participants in the Higher Speed Study Group (HSSG) within the IEEE were divided on whether to include 40G Ethernet as part of their charter or stay the course with 100 Gigabit Ethernet.

The HSSG agreed to work on a single standard that encompassed both 40G and 100G.

Driving demand for 40/100G Ethernet are the same drivers currently stoking 10G: datacentre virtualization and storage, and high-definition videoconferencing and medical imaging. Some vendors are building 40/100G Ethernet capabilities into their products now.

Cisco's Nexus 7000 datacentre switch, which debuted early last year, is designed for future delivery of 40/100G Ethernet.

"We have a little more headroom, which isn't bad to have when you look at future Ethernet speed transitions coming in the market," says Doug Gourlay, senior director of datacentre marketing and product management at Cisco. "We're pretty early advocates of the 100G effort in the IEEE.

"[But] the earliest you'll see products from any company that are credible deliveries and reasonably priced: second half of 2010 onward for 40/100G," he adds.

Spirent Communications, a maker of Ethernet testing gear, plans to release a 40G Ethernet testing module in the second half of this year, and 100 Gigabit Ethernet modules in early 2010, says Tim Jefferson, the company's general manager of the converged core solutions group.

Jefferson says one of the caveats that users should be aware of as they migrate from 10G to 40/100G Ethernet is the need to ensure precise clocking synchronisation between systems — especially between equipment from different vendors.

Imprecise clocking between systems at 40/100G — even at 10G — can increase latency and packet loss, Jefferson says.

"This latency issue is a bigger issue than most people anticipate," he says. "At 10G, especially at high densities, the specs allow for a little variance for clocks. As you aggregate traffic into 10G ports, just the smallest difference in the clocks between ports can cause high latency and packet loss. At 40G, it's an order of magnitude more important than it is for 10G and Gig.

"This is a critical requirement in datacentres today because a lot of the innovations going on with Ethernet and a lot of the demand for all these changes in data centers are meant to address lower latencies," Jefferson adds.

Another challenge is readying the cabling infrastructure for 40/100G, experts say. Ensuring the appropriate grade and length of fibre is essential to smooth, seamless operation, they say.

"The big consideration is, what's a customer's cabling installation going to look like and what they're looking for to be able to handle that," Booth says. "They are probably going to need to have a parallel fiber capability."

"The recommendations we're making to customers on their physical plant today are designed to take them from 1G to 10G; 10G to a unified fabric; and then address future 40G," Cisco's Gourlay says.

The proposed physical interfaces (PHY) for 40G Ethernet include a range to cover distances inside the data center up to 100m, to accommodate a range of server form factors, including blade, rack and pedestal, according to the Ethernet Alliance. The 100 Gigabit Ethernet rate will include distances and media appropriate for datacentre, as well as service provider interconnection for intra-office and inter-office applications, according to the organisation.

The proposed PHYs for 40G Ethernet are 1 metre backplane, 10 metre copper and 100 meter multimode fibre; and 10 metre copper, 100 meter multimode, and 10 kilometre and 40 kilometre single-mode fibre for 100 Gigabit Ethernet.