Yet independent of increases in power density, advantages to improved cooling can be demonstrated. Leakage current has become an increasing fraction of processor power with each technology node. As leakage current is strongly temperature dependent, processor power dissipation can be reduced through improved cooling. Achievable processor frequency is strongly dependent on temperature and voltage. The voltage dependence is approximately proportional, while temperature dependence of performance has reduced with each technology node. In the near future, the temperature dependence of performance may almost disappear.

Figure 1. Package and heatsink.

Through improved cooling, temperature can be reduced while voltage and frequency are increased, resulting in higher system performance. For a given cooling configuration, a combination of voltage and temperature exists which maximizes system performance per watt. At higher powers, additional performance is achieved at the expense of energy. Such increases may be limited by electromigration and other failure mechanisms, which are functions of both temperature and voltage. The temperature dependence of the leakage current has been increasing through recent technology nodes. Significant decreases in energy consumption and/or increases in processor frequency can be achieved through improved cooling.

Figure 2. Thermal resistance components.

Leakage Current Effects

Leakage current is current which bypasses the transistor gate, which is always present regardless of whether the gate is active, and which increases with both voltage and temperature. It can be considered as wasted energy, unlike energy used to perform computation.

Leakage current effects have increased as semiconductor lithography has progressed well into the subcontinuum range [2]. In some ASICs, leakage current can be over half of the total power. As the industry moves from 45nm to 32nm to 22nm and beyond, designs are intended to limit leakage current to about one-third of total power dissipation. Some of this will be achieved by lower junction temperatures. The dependence of leakage current on junction temperature is also becoming stronger. At 65nm, leakage current would change by a factor of two over a temperature range of about 45^oC; at 32nm, the range may shrink to 22^oC [3]. So the energy savings due to a given temperature

Figure 3. Component temperatures at nominal frequency.

reduction will increase with technology improvements. From the viewpoint of effectiveness of improved cooling, a relevant metric is the percent reduction in total power versus temperature. Recent observations of processors on the market show this to range from about
0.5%/^oC up to almost 2%/^oC [4]

Improved Cooling

Ellsworth and Simons [5] discussed a variety of package cooling improvements which could be used to cool up to several hundred W/cm². These began with traditional air cooling of a lidded package as shown in Figure 1. As the thermal path through the package substrate offers a high resistance, nearly all the heat flows upwards and outwards while spreading through the package lid and heatsink base, then finally to the heatsink fins and into the air stream. The heat flow path is through the following components:

• Silicon die

• Thermal interface material 1 (TIM1)

• Package lid

• Thermal interface material 2 (TIM2)

• Heatsink base

• Heatsink fins

• Coolant

Figure 4. Processor frequency.

Nonuniformity of power dissipation within the die increases package thermal resistance above that of a package with a uniformly powered by a significant factor, typically two or more [6, 7]. For a given nonuniform power distribution, this factor will increase as the thermal path is improved, moving asymptotically toward the ratio of the highest local power density to the average power density.

Recent developments in packaging and cooling largely consist of replacing traditional materials with those having higher thermal conductivity. In the package, organic TIM1 with metallic or ceramic fillers can be replaced by a completely metallic material, typically indium or indium alloy. Copper package lids can be replaced by composite materials, such as aluminum-graphite, copper-graphite, aluminum-diamond or silicon carbide-diamond. External to the package, heatsinks feature embedded heatpipes or vapor chambers, and in some cases the vapor chamber shares a continuous vapor/liquid space with heatpipes extending into the fins. A cold plate can replace the heatsink or become an integral part of the package, acting as the lid. Ultimately, coolant can contact the silicon with no interfaces between. At the system level, liquid can be used to transfer heat to air through a liquid-to-air heat exchanger, to another liquid through an interface or liquid-to-liquid heat exchanger, or the liquid can flow across the system boundary to transfer heat to the data center and beyond. Such technologies are described in detail by Kang and Miller [8].

Figure 5. System performance.

Performance Modeling

The processor has a nominal power dissipation of 240W, operating at nominal conditions of 0.85V and 95^oC.

Thermal resistance components of various combinations of packaging and cooling are shown in Figure 2. Values shown are typical for a die approximately one square inch (645mm2) in modern flip-chip packaging. Design variations consist of the following:

Package thermal interface material (TIM1) – The TIM1 can be organic or metallic in a traditional lidded package.

Coolant on silicon removes the TIM1 and lid or cold plate, simplifying package construction.

Package types can be lidded, which requires another thermal interface material (TIM2) or cold plate lid, which eliminates TIM2.

The system cooling configuration can be one of three types:

Air, in which all heat is dissipated to the air stream without liquid as an intermediary. In this case the air stream is fixed at 35^oC, the allowable limit for a typical data center.

Internal liquid, in which liquid is used to transfer heat to the air stream.

External liquid, in which heat is transferred to liquid without air as an intermediary. In this case, the external liquid temperature is fixed at 25^oC, slightly above the allowable maximum dewpoint of a typical data center.

Electrical performance is based on a generic technology representative of recent and upcoming process nodes, and is modeled by a few simple assumptions:

In the region of interest on gate-dominated paths, achievable frequency is assumed to be proportional to voltage:

Freq αV                                                         [1]

Dynamic power to be proportional to frequency times voltage squared:

P_d αFreqV²                                                      [2]

So dynamic power becomes proportional to voltage cubed:

P_d αV³                                                                        [3]

Static power (leakage current) is also proportional to voltage cubed.

At nominal conditions, static power is fixed at 35% of dynamic power:

P_s / P_t = P_s / P_t = P_s / (P_s + P_d) = 0.35                           [4]

Dynamic power is assumed to change by a factor of two every 22^oC, expressed by:

P_d α exp[0.0315(T_j - 95)]                                            [5]

At nominal frequency (and voltage) the temperatures of various packaging components are shown in Figure 3. While most of the temperature reduction is due to improved cooling, some is due to the reduction in dynamic power. This positive feedback converges toward a lower operating temperature.

Figure 6. System efficiency.

Reliability Concerns

One practical limit to increasing power is reliability, which is typically dominated by gate oxide  failure. This is assumed to change by a factor of two every 10^oC (typical of modern technology nodes, and happens to be equal to an often misapplied rule of thumb which has been around for decades) and/or every 15mV in the present study, resulting in:

Fail α exp{[0.0693(T_j - 95)] +

[46.4(V - 0.85)]}                         [6]

As the cooling performance is increased and thermal resistance is reduced, the reliability limit is reached at decreasing temperatures, higher voltages and slightly higher power than nominal. Ultimately leakage current is reduced to only one-seventh of its nominal value.

Frequency Improvements

Permitting the processor to operate at the highest combination of temperature and voltage, while maintaining gate oxide reliability equal or better than that of the nominal case, results in frequency improvements as high as 19%, as shown in Figure 4. The slopes of the lines from the origin to the operating point are proportional to computational efficiency, defined as frequency divided by power. In this case, as voltage is lowered, efficiency improves monotonically, with packaging and cooling technology making very little difference at the lowest values.

System Performance

If the processor provided all or nearly all of the power dissipation in a system, very low voltage (and frequency) operation would be most efficient. But processors consume, in all but a few cases, less power than memory. Volume systems typically feature fewer dual inline memory modules (DIMMs) per socket than high-end systems. High-end systems have been described as large boxes of memory with other hardware to drive it, and this is a reasonable description in terms of power dissipation, volume occupied and component cost.

Adding a constant memory power load changes power per socket and system performance to that of Figure 5. In this case memory power is equal to twice that of nominal processor power, typical of a modern high-end system. System performance is assumed to scale with the square root of processor frequency. This exponent relating processor frequency and system performance is at the low end of the range seen, which approaches a value of one at the high end and is strongly dependent on the program being exercised.

Unlike the example of a processor only, an optimum operating point at which the slope of power versus performance is maximized falls close to the midpoint of the range evaluated. This is slightly different for each packaging and cooling configuration. Dividing performance by power provides computational efficiency, which is shown compared to the nominal configuration in Figure 6. Here the optimum operating condition shifts to lower power as cooling is improved.

Figure 7. System performance versus TCO.

Total Cost of Ownership

While maximization of computational efficiency in terms of energy has been demonstrated, the total cost of ownership should be considered. Until just a few years ago, capital expenditure on computing equipment dominated energy cost, and the two expenses were the concern of different organizations with little communication. Recently higher energy costs, much higher energy usage and environmental concerns have resulted in a more holistic view of data center cost and efficiency.

In the present study, computing equipment cost is fixed to be equal to that of energy cost. At 11.4 cents per kilowatt-hour, a continuous watt of electricity costs exactly one dollar per year. This is close to an average data center’s electricity cost, so a rough approximation can be made that lifetime energy cost of a server in dollars is equal to its average power consumption in watts times its operational lifetime in years. Equal capital energy costs could thus be a server with a five year lifetime and capital cost equal to five times power consumption, for example.

Figure 7 plots normalized total cost of ownership (TCO) versus economic computational efficiency, a term invented for this study and defined as system performance divided by TCO. All optimum values feature TCO below the nominal value, and improvement in economic computational efficiency can be as high as 7%. As this is a simple model, some factors are neglected, which will alter the results if included. The cost of floorspace may be an additional factor, tax incentives and deductions will alter TCO calculations. Additional equipment external to the computing must be added or removed.

Conclusions

A case study showing energy reduction and/or performance maximization through improved cooling of a typical modern high-end server has been performed. In both energy efficiency and economic computational efficiency or TCO minimization, an optimum frequency (and voltage) exists for each packaging and cooling configuration. As cooling is improved, the advantage becomes more favorable. Energy computational efficiency can be increased to 15% above nominal, and economic computational efficiency can be increased to 7% above nominal. More detailed models will shift the values but should follow the same general trend. Lower temperature operation of processors is an available design option with immediate rewards.

Acknowledgments

This article is based on the author’s keynote presentation of the same title at “The Heat Is On: Performance and Cost Improvements through Thermal Management Design,” 21st March, 2011 [9]. The author thanks his colleagues Vadim Gektin (now at Huawei), who provided Figure 1, and Bruce Guenin, who invited him to write this article.

References

[1] International Technology Roadmap for Semiconductors, 2010, http://www.itrs.net/Links/2010ITRS/Home2010.htm

[2] N. S. Kim, T. Austin, D. Blaauw, T. Mudge, K. Flautner, J. S. Hu, M. J. Irwin, M. Kandemir and V. Narayanan, 2003, “Leakage Current: Moore’s Law Meets Static Power,” IEEE Computer, Vol. 36, No. 12, pp. 68-75

[3] S. Mukhopadhyay, A. Raychowdhury and K. Roy,  2003, “Accurate Estimation of Total Leakage Current in Scaled CMOS Logic Circuits Based on Compact Current Modeling,” Proceedings of the Design Automation Conference, pp. 169-174

[4] J. Wei, 2007, “Challenges in Package Cooling of High Performance Servers,” Proceedings of the 2007 International Electronic Packaging Technical Conference and Exhibition, IPACK2007-33637, ASME

[5] M. J. Ellsworth and R. E. Simons, 2005, “High Powered Chip Cooling – Air and Beyond,” ElectronicsCooling, Vol. 11, No. 3, http://www.electronics-cooling.com/2005/08/high-powered-chip-cooling-air-and-beyond/

[6] J. Deeney, 2002, “Thermal Modeling and Measurement of Large High Power Silicon Devices with Asymmetric Power Distribution,” Proceedings of the 35th International Symposium on Microelectronics, IMAPS

[7] D. Copeland, 2005, “64-bit Server Cooling Requirements,” Proceedings of the IEEE Twenty-First Annual IEEE Semiconductor Thermal Measurement and Management Symposium, pp. 94-98

[8] S. Kang and D. Miller, 2011, “Thermal Management Architectures for Rack Based Electronics Systems,” Proceedings of The Heat Is On: Performance and Cost Improvements through Thermal Management Design, MicroElectronics Packaging and Test Engineering Council (MEPTEC)

[9] D. Copeland, 2011, “Energy Reduction and Performance Maximization through Improved Cooling,” Proceedings of The Heat Is On: Performance and Cost Improvements through Thermal Management Design, MicroElectronics Packaging and Test Engineering Council (MEPTEC) l

Water cooling of computers was introduced more than twenty years ago, but had disappeared from the mainstream by the mid 1990s. The conversion of chip technology from bipolar to CMOS (complementary metal oxide semiconductor) was the main reason. As chip powers continue to increase, water cooling appears likely to become mainstream again. In the current market, the major benefits of changing from air to water cooling is increased packaging density and lower power noise. Among the systems most likely to benefit are rackmount servers, both 1 unit (1U = 44.45 mm) and blade configurations with even more severe space requirements.

In the near future, water cooling of processors and air cooling of other components may be introduced. The water (or refrigerant) used to cool the processors would transfer heat to the outside environment, probably through the building water supply, avoiding the CRAC (computer room air conditioner) units. The capital cost, energy consumption and floor space required by the CRAC units would be significantly reduced. In a case study by Prechtl and Kurtz [1], a total (server plus cooling) power reduction of 23% and CRAC power reduction of 50% were found.

Nomenclature

A	wetted surface area	m2
C	thermal conductance	W/K
Cp	thermal capacity	J/kgK
ε	effectiveness	-
h	heat transfer coefficient	W/m2K
	mass flow rate	kg/s
NTU	number of transfer units	-

One significant difference with air cooling is the volumetric thermal capacity of water, close to 4000 times that of air. The volume of water required per hour will be slightly lower than the volume of air required per second to provide the same thermal capacity. The convective resistance of a cold plate or heatsink and the thermal capacity of the coolant are related by:

NTU = hA/( Cp) (1)The effectiveness of a cold plate or heatsink is the ratio of heat transferred to the ideal case in which all fluid achieves the surface temperature:

ε = 1 – exp(-NTU) (2)

The resulting thermal conductance is given by:

C = ε m Cp (3)

For a fixed pressure drop, thermal conductance is maximized at NTU = 1, resulting in ε = 0.63. For a fixed pumping power (the product of volume flow rate and pressure drop), thermal conductance is maximized at NTU = 1.9, resulting in ε� = 0.85 [2]. Cold plates with effectiveness lower than optimum require excessive flow; those with effectiveness higher than optimum require excessive pressure drop.

Achieving such high values of effectiveness in a small form factor has historically been a manufacturing challenge. Even large cold plates for multichip modules, over 100 mm square, required only a few flow channels making several passes back and forth across the cold plate. CNC (computer numerically controlled) machining has practical limits of about 0.5 mm channel width and 0.5 mm fin thickness. Inserted folded fins also have limits of about 0.5 mm in channel width, but can use finstock to about 0.2 mm thickness. Even 0.5 mm channels result in small values of effectiveness for typical single-chip size cold plates, and smaller dimensions are required to achieve optimum effectiveness.

For the purposes of this article, a conventional cold plate is defined as one in which coolant channels are parallel and run from one side of the cold plate to the other. This can also include cases in which coolant channels turn back and forth across the face of the heat source(s). Maldistribution of flow can be significant. Lu, Yang and Wang [3] simulated 60 x 64 mm cold plates with channels 1 to 3 mm wide. Figure 1 shows velocity and temperature distributions at 1 m/s. With 20 channels 3 mm wide, variations in velocity and temperature are much greater than with 60 channels 1 mm wide.

Figure 1a. Velocity distribution at 1.0 m/s [3].

Figure 1b. Temperature distribution at 1.0 m/s [3].

Manifold Flow Distribution

Improvement of the conductance of cold plates has been studied but new designs have seldom been implemented. Harpole and Eninger [4] proposed a manifold microchannel in which alternate inlet and outlet channels guided flow in and out of parallel channels. In their case, the optimum dimensions were quite small. Manifold channel spacing was 333 �m, fin height 167 �m and channel width 7 to 14 �m, with the ratio of fin thickness to channel width from 0.5 to 1.0, significantly smaller than dimensions considered practical for fabrication in copper.

Manifolding has appeared in recent commercial products. Valenzuela and Jasinski [5] describe a normal flow cold plate (NCP) with alternating inlet and outlet manifold channels approximately 1 mm wide. Figure 2 shows a cutaway view of the NCP. The heat transfer matrix is quite thin but not described in detail. Effectiveness values of 0.8 at low flow rates and 0.6 at high flow rates were demonstrated. North and Cho [6] tested an assembly with a multistage manifold, as shown in Figure 3. Coolant enters the inlet port (1), is distributed to a manifold inlet channel (2), then flows the short path through the heat transfer matrix (3) into the adjacent outlet channel (4) and finally through the outlet port (5).

Figure 2. Normal flow cold plate (NCP) and manifold structure [5].
Figure 3a. Powdered metal cold plate (PCMP) assembly [6].
Figure 3b. Coolant flow path [6].Patterson et al. [7] considered four possible arrangements for multilevel flow: single level flow (1F), parallel flow (PF), counter flow (CF) and series flow (SF), shown in Figure 4. Their goals were reduction of both average wall temperature and wall temperature variation. Results with silicon and water at an inlet velocity of 1 m/s are shown in Figure 5. While series flow achieves good uniformity, the wall temperature rise is high. The counter flow arrangement provides low and uniform values of wall temperature rise.

Figure 4. Flow arrangements [7].

Figure 5. Wall temperature distribution [7].

Nontraditional Heat Transfer Surfaces

Heat transfer surfaces other than parallel plate fins have also emerged. North and Cho [6] described a porous metal heatsink in which spheroidal particles are bonded together, shown in Figure 6. Nominal particle diameters were 274, 325 and 537 �m. Prechtl and Kurtz [1] presented a microstructured fabrication process in which etched layers were joined together to form a multilayer heatsink, as shown in Figure 7. The resulting channels were 400 x 600, 200 x 300 and 100 x 200 �m.

Figure 6. Powdered metal cold plate (PCMP) heat transfer matrix [6].

Figure 7. Microstructured cold plate fabrication process [1].Muller and Frechette [8] presented a comprehensive numerical study of manifold microchannel heatsinks. With copper and water, using 1 cm² as the reference area, a thermal conductance of 15.5 W/K could be achieved with 0.005 W pumping power, significantly higher performance than achieved to date. A zigzag fin configuration was introduced, which could lead to even further improvement in performance.

Conclusions

A complete evaluation of cold plate performance requires measurement of flow rate, thermal resistance and pressure drop. Flow rate and thermal resistance for cold plates of different size must be normalized to a unit surface area, traditionally 1 cm². Figure 8 shows normalized thermal resistance and pressure drop as functions of volume flow rate per unit area for a normal flow cold plate [5]. At the lowest flow rate, effectiveness is about 63%, corresponding to NTU = 1, the optimum value for fixed pressure drop [2]. Small differences in pressure drop at different values of heat flux reflect changes in fluid properties with temperature.

Figure 8a. Ultra high flux NCP thermal performance [5].

Figure 8b. Ultra high flux NCP hydraulic performance [5].In air cooling, nonoptimized heatsinks were adequate until quite recently. The current need for minimization of noise, fan power, heatsink volume and weight have forced optimized designs into production. As a result of volume production, manufacturing techniques such as soldered stacked fins have seen significant reductions in cost. Manufacturing techniques, which allow variation of both the manifold and coolant channel dimensions will permit the effectiveness of the cold plate to be optimized. Once water cooling becomes mainstream, the dual expectations of designs moving closer to optimized values and reduction of manufacturing cost through volume are reasonable.

Acknowledgments

The author wishes to thank Olaf Kurtz of Atotech, George Meyer of Thermacore and Javier Valenzuela of Mikros for information about their products.

References

1. Prechtl, P., and Kurtz, O., “Efficient Liquid Cooling Technologies for Computer Systems”, Proceedings of the IMAPS Advanced Technology Workshop on Thermal Management, Palo Alto CA, October 2004.
2. Copeland, D. W., “Fundamental Performance of Heatsinks”, ASME Journal of Electronic Packaging, Vol. 125, No. 2, June 2003, pp. 221-225.
3. Lu, M.-C., Yang, B.-C., and Wang, C.-C., “Numerical Study of Flow Maldistribution on the Flow and Heat Transfer from Multi-Channel Cold Plates”, Proceedings of the 20th IEEE Semi-Therm Symposium, San Jose CA, March 2004, pp. 205-212.
4. Harpole, G. M., and Eninger, J. E., “Micro-Channel Heat Exchanger Optimization”, Proceedings of the 7th IEEE Semi-Therm Symposium, Phoenix AZ, February 1991, pp. 59-63.
5. Valenzuela, J., and Jasinski, T., “High Heat Flux Cooling with a Normal Flow Cold Plate (NCP), Thermal Management of Electronics Summit 2004, Natick, MA, August 2004.
6. North, M. T., and Cho, W.-L., “High Heat Flux Liquid-Cooled Porous Metal Heat Sink”, Proceedings of Interpack ’03, ASME paper IPACK2003-35320, Kaanapali HI, July 2003.
7. Patterson, M. K., Wei, X., Joshi, Y., and Prasher, R., “Numerical Study of Conjugate Heat Transfer in Stacked Microchannels”, Proceedings of ITherm 2004, Las Vegas NV, June 2004, pp. 372-380.
8. Muller, N., and Frechette, L. G., “Optimization and Design Guidelines for High Flux Micro-Channel Heat Sinks for Liquid and Gaseous Single-Phase Flow, Proceedings of ITherm 2002, San Diego CA, May-June 2002, pp. 449-456.