As power densities in each data centre cabinet grow, so have the challenges of keeping IT equipment cool. While liquid cooling is often talked about as the only solution for the future, others are working on pushing the capabilities of air cooling.
Data centres have seen an ongoing increase in average power density, especially over the past decade. Moreover, the power density per IT cabinet has seen an even greater increase as IT Equipment (ITE) power density continues to rise to levels that seemed extreme only a few years ago but are now the new normal.
There are major ITE manufacturers that offer standard “commodity” 1U servers with 800 to 1800 watt redundant power supplies which can require more than 1000 watts of cooling.
This is being driven by processor and IT equipment manufacturers which in turn are being held back because most mainstream data centres are hard pressed or unable to support more than 10-15 kW per rack.
Processor power levels are escalating from 100 to 250 watts and chip manufacturers have product roadmaps for CPUs and GPUs that are expected to exceed 500 watts per processor in the next few years.
This trend has left many wondering if air-cooled ITE has reached its power limits and if liquid cooling is really the only long-term solution. Currently, some HPC IT equipment is available as liquid or air-cooled versions, however the bulk of standard off-the-shelf ITE is still air cooled. And the majority of new mainstream data centres continue to be designed for air cooled ITE.
In reality, there are multiple issues that influence how high the power density in air-cooled IT equipment in standard cabinets can be pushed. This has led to renewed efforts by ITE manufacturers and data centre operators to push the boundaries of air cooling.
The basic principles of data centre cooling are well known; remove the heat load generated by IT equipment and transfer it via one or more physical mediums (air, liquids, or solids) and one of more forms of thermo-mechanical systems to reject it out of the facility. However, as power densities of IT processors and other computing related components have increased, this process has become more difficult to accomplish effectively and efficiently. For purposes of this report, we will discuss air and liquid cooling, with a primary focus on air-cooled IT equipment.
Power density – watts per what?
One of the more common issues that get discussed is the question of the power density limit for air cooled ITE. Data centre power density is typically expressed as average watt per SF. This is calculated based on the critical power available to the IT Load (watts or kilowatts), divided by the cooled area occupied by the ITE (whitespace). However, this does not really reflect the conditions of the cooled space occupied by the air-cooled IT equipment.
Air vs liquid cooling
The two greatest sources of heat in IT equipment are the processors (CPU, GPU, TPU, etc.) and memory. As mentioned, processor power levels are escalating from 100 to 250 watts and chip manufacturers have product roadmaps for CPUs and GPUs that are expected to exceed 500 watts per processor in the next few years (see AHSRAE Processor roadmap) and represent 50% or more of the ITE heat load.
While the power level of working memory chips (DIMM, etc.) is relatively low in comparison, the total amount of memory has increased significantly and continues to rise, and can represent 20% or more of the heat load.
As mentioned, liquid cooling is being primarily used to address the higher power processors. While there are many variations of liquid cooling technologies and methodologies, they generally fall into three categories:
- Liquid Cooling for Air-Cooled ITE
- Liquid Cooled ITE
- Immersion Liquid Cooling
Liquid Cooling for Air-Cooled ITE
The ITE is standard unmodified equipment. The process starts at the processor chip and the heat is typically transferred to an attached air-cooled heat sink. All the heat is transferred to room-based cooling units, or to close coupled cooling: In-row cooling, rear door heat exchanger (RDHX) or other types such as overhead aisle-based cooling units.
Liquid Cooled ITE
The process starts with internal components; the processor chip and memory the heat is typically transferred to an attached liquid cooled heat sink (cold plate), also sometimes referred to as direct liquid cooling. Typically, a portion (50-75%) of the heat is transferred to a liquid loop, the remainder of the heat goes to room-based cooling units, or to close coupled cooling: In row, RDHX, or other similar systems.
Immersion Liquid Cooling
The entire ITE is submerged in a dielectric fluid, which engulfs the chassis and all the components. Virtually all the heat is effectively transferred to the fluid. These are generalised summaries of three categories. Each have advantages, limitations and trade-offs, however the more granular aspects of liquid cooled ITE and immersion cooling raises the complexity, (due to the varies methodologies, fluid types and operating temperatures), which makes further discussion of these factors beyond the scope of this article series.