How 10GbE can Reduce Power and Cost Requirements in Data Centers

The Eco-Friendly Data Center

All IT professionals are aware of the issues they are facing regarding the power and real estate requirements associated with where to locate and how to power and cool their data centers.  While many existing data centers must try to optimize and live within the boundaries of their current facilities and locations, new data centers are being located based upon the decisions of the cost of energy, or said another way, the cost of cooling.

However, although cost is certainly an issue, it is not enough to think only of the corporate “pocket book”.  All power sources have some type of detrimental effect on the environment, and therefore, it is incumbent upon everyone, but especially upon the largest consumers, businesses, to minimize their impact on the environment while continuing to maintain alignment with corporate objectives such as maximizing value to company share holders.

The Savings and Costs of Virtualization

Virtualization is one of the key tools being used to help maximize the utility of a data center.  Server virtualization increases the utilization of servers by permitting one physical server to operate as many virtual servers.  The reduction in the number of physical servers saves on real estate costs, lowers electrical bills, and saves on cooling expenses.  There is minimal power differential when operating a CPU at 10% utilization versus operating one at 70% utilization.  Reducing the number of physical servers to power, and cool, can result in significant cost savings.

Because the number of configurations is also reduced, software patches and upgrades become much easier to manage – again requiring less energy and time to manage the servers.  Server load balancing can also be automated to optimize for the enterprise needs from a single point rather than by monitoring each server independently.  This allows better utilization of the corporate assets. 

It is also commonly accepted that virtualization is the killer application for 10 Gigabit Ethernet (10GbE).  While in a traditional deployment, each application server would not likely utilize 10GbE bandwidth, but rather 1 Gigabit Ethernet (1GbE) or maybe even 100Mbps.  The aggregation of the virtualized servers requires the ability of the network to seamlessly and transparent move applications from one virtual server to another whether or not in the same physical server.  In the case that the application needs to move between virtual servers in different physical locations, the virtualized server can fully utilize the available network speed while leaving ample room for other applications to operate unimpeded by the sudden bandwidth demand. 

The Development of 10GbE for Volume Applications

Although 10GbE is a much discussed technology, its deployment to date as been largely restricted to wide area networks (WANs), metropolitan area networks (MANs), high-performance computing (HPC) and selected enterprise deployments.  The reasons for such limited 10GbE deployments have been primarily related to the cost, power, performance, and “user friendliness” issues. 

Almost all of the deployments of 10GbE to date have had to use either optics modules and fiber or short reach copper with a CX4 cable.  These technologies lack the ability to drop down to lower speeds; therefore, addition of 10GbE requires a forklift upgrade of the equipment involved.  The 10GbE network also lacks the ability to run at lower speed which corresponds to a lower power.  

Another issue associated with 10GbE has been the limited performance of the 10GbE networks.  This problem is not related as much to the 10GbE technology as it has been to the microprocessors, host bus, chipsets and memory interfaces that have been used in server platforms. Often, the microprocessors and chipsets themselves consume and dissipate significant amounts of power yet lack the efficiency needed to utilize the network bandwidth.  New, state-of-the-art microprocessors and chipsets are more efficient in handling the network bandwidth, but still consume most of the power in a server platform.

Lastly, the cost of these solutions and the ease of use of the cabling have not made 10GbE an attractive solution unless the increased data traffic demands cannot be met otherwise.  For example, the CX4 is constrained by the lack of being able to field terminate cables, having a poor bend radius, and limited to a reach of 15 meters has made it difficult to use in many applications.  However, through the ratification of new standards and developments in technology, the ability to use 10GbE in server platforms has made great strides during the course of 2006 and into 2007, and 10GbE is now proving it can be a viable, cost effective and energy efficient technology.

Deploying 10GbE over Copper Cabling

The significance of operating 10GbE over copper cabling is more than the fact that it offers the opportunity to potentially use 10GbE over already-pulled Category 6 cabling, in addition to Category 6A and 7; but rather it offers the ability to drop down to 1GbE and eventually 100Mbps speeds.  The auto-negotiation feature will permit implementers to design PHY devices that operate over a broader range of speeds and media.  While auto-negotiation will provide the capability to be backwards compatible to lower speeds, it is unlikely that a 10GBASE-T system will be able to support operation at 10 Mb/s due to the current requirements of the transformer.  To date, fiber solutions have not been able to offer the ability to drop down to lower speeds and provide a power savings.

The very first 10GBASE-T physical layer components (PHYs) sampled in the market in the first half of 2007 showed power numbers of 10 to 12 Watts per solution.  This compares with a typical 10GbE optics module of about 3 Watts.  However, these PHYs will quickly be optimized and within two years 10GBASE-T PHYs are estimated to operate with the same power specifications as today’s optical modules, but with the added benefit of operating at lower speeds and costing less..

When purchasing new servers, there are many new options becoming available on the market.  At Interop Las Vegas 2007, there was the world’s first public demonstration of 10GbE over copper (10GBASE-T).  It brought together multiple PHY, switch and server vendors proving that interoperability of this technology is already happening.  However, although the maximum power budget of a server NIC is typically 25 Watts, it is commonly desired that a maximum power budget of 15 Watts is not exceeded.  As server adapters move increasingly towards dual port configurations, the ability to meet this desired power budget will be further stressed.  Therefore, an optimal combination of low power, minimized chip count and real estate requirements is obviously preferred.  For many volume server platforms, it will be a cost, performance and power optimized solution that will best meet the market demands.  Processors, chipsets, and NICs should work together to achieve this goal.  Therefore, when IT professionals are looking to incorporate a 10GbE NIC into their servers, they will want to be sure to chose the lowest power, backward compatibility, user friendly, and lowest cost combination possible.

In addition, although the distances of copper cabling is specified in the standard as 100 meters over Category 6A cabling, which is less than the distance of 300 meters capabilities of 10GBASE-SR multimode fiber interconnect, copper cabling is much more robust in terms of ability to bend and therefore can be configured in ways that are often not possible with fiber optic cabling; thereby, making copper cabling a much more suitable choice for horizontal deployments.  Furthermore, 10GBASE-T cabling is able to be field terminated allowing installation technicians the ability to quickly install cables to their appropriate distance.  The ability to optimize the configuration of a data center, as well as operate at lower speeds, will make 10GBASE-T an extremely attractive alternative in the next 18 months.

Real Estate Requirements

From a server network interface card (NIC) point of view, 10GbE produced additional challenges.  In order to achieve near line rate performance, there needed to be an elaborate TCP/IP offload engine (TOE) incorporated.  This TOE ability was required because the host processors would get overwhelmed by the network bandwidth and had 100 percent of their utilization focused on processing TCP/IP and had no bandwidth remaining to process the application.  To assist the overworked microprocessors, the TOEs were introduced.  These TOE solutions were placed onto NICs and included the controller that performed the TCP/IP offload function as well as a large number of external memory chips used for buffering and copying data.  This resulted in server NIC controllers that would reach close to 19 Watts by themselves.  When adding in the real estate and power requirements of other components such as the memory and, if so desired, an optics modules, other PHYs, etc, these NICs were not compliant with the maximum PCI host bus power budget of 25 Watts. 

Processor manufacturers have placed a heavy emphasis on lowering the power requirements of their processors.  When processors were used in a 10GbE network even only 18 months ago, they easily utilized 60 plus percent of their processing power for TCP/IP processing.  Today, the newest processors are seeing utilization numbers more along the lines of 25 percent and continuing to reduce.  In addition, chipset manufacturers have significantly improved the performance of their products.  Whereas it was not unusual two years ago to see platforms shipping with products where the 10GbE network would be bottlenecked at 4Gbps, it is now very common to see platforms running easily at 7 to 9 Gbps.

The end result of the improvements in processors and chipsets indicates that the TOE vendors were largely incorrect.  Although they produced many studies showing how these products maximized line rate performance, 8Gbps or more, and processor utilization in the single digits for the majority of platforms, the trade-off of power requirements and cost is simply not necessary.  As processors and chipsets continue to improve, the need for additional “offloading” will continue to decrease.  Although some assistance is preferred for applications like storage, there is often enough headroom for the processor to process the protocols if the controller can provide a small bit of TCP/IP assistance. 

Efficient architectures also eliminate the need for additional components, such as the memory chips, which further reduces the cost, power, and real estate requirements.  The smaller the footprint of the chassis with the lowest possible power budgets will further increase the flexibility of the cost savings offered to the data center in which it is deployed.

Reducing Routing Complexity

The introduction of 10GBASE-KR, the 10GbE serial backplane link developed to replace 10GBASE-KX4, will significantly reduce the routing complexity associated with designing backplanes.  This reduced complexity will offer the system designer more flexibility in the design of their backplanes and/or mid-planes to be put into a system.  The reduced complexity comes from the fact that previous 10GbE links required the use of an add-in card that used a four-lane interconnect known as 10GBASE-KX4.  The 10GBASE-KR interconnect was designed to make use of the single lane that operates at 1GbE, also known as 1000BASE-KX.  This eliminates the need for an add-in card to support 10GbE.

The ability to support 10GbE without an add-in card or without a multiple lane interface reduces the number of lanes and PCB traces; thereby, reducing the cost of materials needed to build a backplane or mid-plane chassis.  The ability to design and build smaller backplanes or mid-planes to support 10GbE will provide the opportunity to create new airflow designs to assist in cooling which can result in power savings.

Summary:  Choosing Data Center Server Solutions

10GbE shares many similarities with previous generations of Ethernet, but also has some key differences.  In the 10GbE standards, many of the differences are related to the transmission media.  In order to achieve true 10GbE performance, the system must be configured to support 10GbE throughout the network.  This includes everything from improving TCP/IP processing via hardware assist so the CPU has more cycles for application processing, to making sure the right interfaces (e.g. PCI Express) are in place, to having chipsets with memory subsystems that won’t limit the system performance.  Many of these issues are being addressed today, and the market can look forward to enjoying true 10GbE performance and all its benefits in the very near future.

About Tehuti Networks

Tehuti Networks is a leading provider of single-chip and board-level solutions that provide 10 Gigabit Ethernet line rate combined with low CPU utilization and low-power for networking end point applications.   These inexpensive, high performance solutions support all major operating systems as well as both hardware and software virtualization schemes.    The solutions improve server processing performance and provide significant cost benefits to original equipment manufacturers and IT users.. More information is available at www.tehutinetworks.net.