Reed's Ruminations: A Blog by Dan Reed

Serial Dismay, Parallel Excitement

Multicolored lights are one of the iconic and ubiquitous manifestations of the end of year holidays. They also conjure festive memories of my childhood, albeit with one major exception. That rather less than pleasant memory involved much consternation, veiled (and not so veiled) mutterings and ritual questioning of all things derived from Maxwell and Edison.

I speak, without hesitation or ambiguity, regarding the curse (sometimes literal) of serially wired holiday lights. My father and mother kept a healthy supply of spare bulbs for laborious and sequential replacement and testing whenever a single bulb failed, and a strand of thirty bulbs went dark. I still remember my excitement and delight when we first purchased strands with parallel circuit wiring. What a difference it made, when a bulb failure dimmed only one light, rather than an entire strand. As a child, I quickly gained a visceral appreciation for Kirchhoff's laws, long before I ever knew the name Gustav Kirchhoff.

Systemic Resilience

If there is an Aesop-like moral in this tale from my childhood, it relates to systemic design for resilience rather than component resilience alone. Parallel circuit resilience trumps serial circuit resilience, and the extra cost is repaid in greater systemic reliability. Alas, I fear we have not learned this lesson in parallel computer system design and parallel programming models and applications.

The standard domain decomposition models commonly used to solve the partial differential equations that underlie so much of computational science and that are embodied in parallel MPI implementations of the solvers, implicitly presume that all of the parallel processes (tasks) and the supporting hardware operate without error or failure. Periodic checkpointing is our primary concession to failure and the vagaries of job scheduling.

This is analogous to trusting that the series circuit holiday lights will not fail until we turn them off each evening. It can be effective for small numbers of lights (processes) but increasingly problematic for larger systems. All this suggests we need to rethink our models of hardware and software resilience, and embrace redundancy as the necessary cost of systemic reliability, particularly in the trans-petascale and exascale regimes. It may also mean that we can never use all of the hardware for non-redundant computation. After all, would one rather fail quickly at exascale rates or compute reliably at sustained trans-petascale rates?

Changing the Game

Cloud services now operate on the largest computing systems we have ever built on this planet, with service reliability expectations far higher than what we demand from scientific applications. Thus, I also believe there are lessons from cloud computing that are potentially applicable to computational science applications. Arguably the most important is exploring how to exploit eventual consistency rather than sequential consistency.

To understand this potential shift in perspective, I heartily recommend Werner Vogels' analysis of the power of eventual consistency for large-scale web services at Amazon. Eric Brewer's thoughts on the CAP theorem, drawn from his Inktomi experiences, have also shaped theoretical and empirical assessments of large-scale system reliability. For those not familiar with the CAP theorem, it postulates that one can choose any two of Consistency, Availability or Partition tolerance. More generally, it offers a framework to reason about conflicting objectives.

Lest everyone shudder in numerical and scientific horror at this prospect, remember that parallel, numerical applications are themselves samples drawn from the space of "possible answers." They are not "the answer," and they may not even be an unbiased sample. (See On Getting the "Right" Answer.) Are these weak consistency approaches transferrable to scientific applications? I do not know. I am only sure that we need to find out – quickly. If successful, this would be a profound change, with deep implications for computational science algorithms, software and application reliability.

Egg Baskets, Lambdas and Geo-resilience

If you have ever been sent to the henhouse to fetch eggs, you know the practical truth of the old aphorism, "Don't put all your eggs in one basket." With just one stumble – wait for it – you can have egg on your face. (Sorry, I just could not avoid the pun.) All of which is to say, distributing one's assets is normally a successful risk mitigation strategy. At least that is what we all thought about our retirement mutual funds until the world financial markets collectively declined.

I hear rustling among my faithful readers, collectively wondering how these colorful, though meandering, introductory paragraphs will segue to more salient topics, such as clouds or high-performance computing. Fear not, the answer is here, manifest as a few observations on massive data centers, wide area networking, and techniques for geo-distribution and resilience.

Protecting the Big Asset

Every operator of massive cloud data centers chooses the locations of those facilities after careful, qualitative and quantitative analysis of several factors, including the tax climate, availability and cost of electricity, the natural stability of the site (i.e., probability of earthquakes, tsunamis, hurricanes and other natural disasters), the region's political stability and physical security, access to optical fiber and broadband networks and a host of other issues. Given an objective scoring of these criteria, it is not surprising that multiple operators and competitors often site their facilities in the same region.

Regardless of location, a substantial fraction of the infrastructure associated with a data center has historically been devoted to electrical power and resilience. This has typically included multiple diesel generators, collectively capable of generating many megawatts of electricity, and large, lead acid battery banks to ensure an uninterruptable power supply (UPS). Not only are these generators and battery banks expensive, their deployment and acceptance are time consuming and often difficult. Indeed, before the economic downturn, demand for generators was so high that sites competed to secure a place in line for delivery, further delaying data center construction.

The reason we strive to protect data centers is obvious, their failure often has catastrophic business implications. Without doubt, though, some workloads are less affected by failure than others. If your web search for basket weaving supplies times out, you are probably content to retry the query a few seconds later. If your attempt to file a tax return electronically fails, that is more problematic.

This focus on resilience raises one rather obvious question, is there a better way? Returning to my initial metaphor, rather than protecting the one big basket of eggs, might one distribute the eggs in several, smaller baskets?

Geo-dispersion: The Other Alternative

If it were possible to replicate data and computation across multiple, geographically distributed data centers, one could reduce or eliminate UPS costs, and the failure of a single data center would not disrupt the cloud service or unduly affect its customers. Rather, requests to the service would simply be handled by one of the service replicas at another data center, perhaps with slightly greater latency due to time of flight delays. This is, of course, more easily imagined than implemented, but its viability is assessable on both economic and technical grounds.

In this spirit, let me begin by suggesting that we may need to rethink our definition of broadband WANs. Today, we happily talk of deploying 10 Gb/s lambdas, and some of our fastest transcontinental and international networks provision a small number of lambdas (i.e., 10, 40 or 100 Gb/s). However, a single mode optical fiber has much higher total capacity with current dense wave division multiplexing (DWDM) technology, and typical multistrand cables contain many fibers. Thus, the cable has an aggregate bandwidth of many terabits, even with current DWDM.

Despite the aggregate potential bandwidth of the cables, we are really provisioning many narrowband WANs across a single fiber. Rarely, if ever, do we consider bonding all of those lambdas to provision a single logical network. What might one do with terabits of bandwidth between data centers? If one has the indefeasible right to use (IRU) or owns the dark fiber, one need only provision the equipment to exploit multiple fibers for a single purpose.

Of course, exploiting this WAN bandwidth would necessitate dramatic change in the bipartite separation of local area networks (LANs) and WANs in cloud data centers. Melding these would also expose the full bisection bandwidth of the cloud data center to the WAN and its interfaces, simplifying data and workload replication and moving us closer to true geo-dispersion and geo-resilience. There are deep technical issues related to on-chip photonics, VCSELs and ROADMs, among others, to make this a reality.

In the end, these technical questions devolve to risk assessment and economics. First, the cost of replicated, smaller data centers without UPS must be less than that of a larger, non-replicated data center with UPS. Second, the wide area network (WAN) bandwidth, its fusion with data center LANs and their cost must be included in the economic assessment.

These are interesting technical and economic questions, and I invite economic analyses and risk assessments. I suspect, though, that it is time we embraced the true meeting of high-speed networking and put our eggs in multiple baskets.

Matryoshka Dolls and Data Centers

Over the past year, there has been much debate on the energy efficiency of large data centers and how best to measure that efficiency. Power Usage Effectiveness (PUE) has emerged as the de facto standard, where PUE is defined as the ratio of power entering the facility to power drawn by the computing equipment. Intuitively, the ideal PUE is unity (i.e., all incoming power is used by the computing equipment, with no overhead for cooling or energy distribution). For details on PUE and data center efficiency, I encourage you to read the documents from the The Green Grid, which has promulgated and evangelized these measurement standards. (Truth in advertizing: Microsoft is a leading member of the The Green Grid.)

Legacy and Modern

Many legacy data centers -- those built more than a few years ago – have PUEs in excess of two, or even three. This is largely due to inefficient computer room air-conditioning (CRAC) units, lack of hot and cold aisles, energy losses due to multiple (unnecessary) voltage conversions and aging or inappropriate building designs. If your "data center" is located in the nearest available space to your laboratory -- a retrofitted janitor's closet – and cooled by two box fans from Walmart, odds are you are not a paragon of PUE virtue, even if your aggregate computing power is small.

Today, state of the art data centers have PUEs below 1.5, and there are new designs that could approach a PUE of one by reducing UPS support where appropriate, operating at substantially higher temperatures and exploiting ambient cooling. Many people do not realize that computing hardware is much more resilient to high temperature than history and practice would suggest. It need not be chilled to temperatures suitable for polar bears.

Last year, Microsoft's Christian Belady illustrated this point by operating a group of servers in a tent for over six months, with zero failures. Although not a statistically valid sample, the experiment did illustrate why ASHRAE and hardware vendors have broadened the temperature and relative humidity envelope of acceptable operation.

More generally, efficient data center design and analysis are much like Matryoshka dolls, because the PUE one obtains depends on where and what one measures. Ideally, the bounding box for measurement should be the entire data center ecosystem, from power distribution and cooling for servers to data center networks to operations and support. Equally importantly, a PUE near one does not mean the computing system itself is efficient and well-matched to the offered load or that the computations are either necessary are useful. Choose wisely and measure thoughtfully and carefully.

Appearance and Utility

Finally, I would be remiss if I did not opine on the most obvious, visual difference between cloud data centers and high-performance computing (HPC) facilities. The former are designed for function, not appearance. They are usually nondescript facilities optimized for efficient hardware operation at large scale, not for human accessibility or for comfort. Indeed, container-based data centers look more like a warehouse and distribution center with parking and utility connections than Hollywood's idea of a computing center. Conversely, HPC facilities are usually showpieces with signs, elegant packaging and lighted spaces suitable for tours by visiting dignitaries.

At large scale, efficient trumps pretty. It's all about what one measures.

Language Shapes Behavior: Our Poor Cousin, Data

The Sapir-Whorf Hypothesis holds that language and culture shape behavior and thought. The idea has a long, rich and sometimes controversial history, though recent cognitive linguistic research, together with brain imaging, has lent it new credence. Lately, I have been reflecting on the nature of linguistic relativity in scientific research, particularly for HPC and cloud computing. In particular, how has our focus on floating point operations per second (FLOPS) shaped discourse, science politics and outcomes?

Look, It's Terascale!

If I were to remark, "That is a terascale system," you would likely interpret my comment as an assessment of the system's computing performance, more specifically its floating point capabilities. For the cognoscenti, the sentence likely engenders visions of multicore systems with out of order instruction issue and deep pipelining, small clusters with fast interconnects and (perhaps) some accelerators such as GPUs. You might even reflect on the historical evolution of high-performance computing, when a terascale system was ranked first on the Top500 list about a decade ago.

Given my remark, you would be unlikely to eye your desktop computer with its terascale (terabyte) commodity disk drive. Yet, based on its storage capacity, that desktop is just as surely a terascale platform as is the compute cluster. Ah, you say, I have mixed capacity (bytes) with capability (operations). Terabyte per second storage systems are rare, you say. Absolutely true, yet as this bit of third rate linguistic legerdemain illustrates, language and, perhaps more importantly, connotation really does trump denotation in our discourse.

Whether Petascale and Exascale?

Today, the world's fastest computing platforms have broken the one petaflop mark on the high-performance Linpack (HPL) benchmark, and planning groups across the United States, Europe and Asia are debating political and technical approaches for the construction of exascale computing systems. I cannot help but wonder, though, why we in technical computing talk so little about petascale and exascale data platforms? Why is there no international initiative to build a low latency, high bandwidth (many terabytes/second to even a petabyte/second) data analysis engine with multiple exabytes of storage capacity?

I suspect the reason lies in our background and linguistic heritage. Most of us in high-performance computing came from mathematical backgrounds, where success was defined by a proof and an equation and their embodiment in a computational model, rather than by insight gleaned from large-scale data analysis. I believe it is time we found a lingua franca that bridges FLOPS and bytes and reclaim the full connotation of exascale.

I Want My Exabytes

It is worth remembering that the explosive growth of observational data is itself largely a product of inexpensive CMOS sensors, based on the same semiconductor technologies that begot microprocessors. The examples of high resolution instruments are legion, from the Large Hadron Collider (LHC) and its international hierarchy of data archives to the proposed Square Kilometer Array (SKA) radio telescope, which may produce as much as an exabyte of data every few days. This data tsunami is not limited to the physical sciences; the biological and social sciences are being inundated as well.

One of the major lessons from web search and cloud data centers is the power of truly massive scale, near real-time data analysis. When anyone with a cheap cell phone and a web browser can extract data and insights from a non-trivial fraction of the human knowledge base, behavior and culture are transformed. I would like to believe that we can bring the same data-driven analytics to scientific research as are routinely exploited to find a good lasagna recipe. We need a balanced exascale initiative, in every sense of the word, lest we be a technical confirmation of the Sapir-Whorf hypothesis.

IT’s Not Easy Being Green (Actually, It Is Easy If You Try)

I can hear the groan's already at the bad pun embodied in the title of this essay. (Here's the cross-cultural deconstruction. First, there's the capitalized "IT" for information technology. Then, there's the reference to energy efficient computing ("green"). However, the cultural reference is to the Muppet Show and Kermit the Frog, whose refrain often was, "It's not easy being green.")

Contrary to the pun, actually, it is easy to be green, if one wants to do so. This is a point I tried to make in an interview that is part of a series InsideHPC has begun on green computing. In the interview, I discussed the challenges and opportunities associated with energy efficient computing at scale, whether operating large-scale data centers or petascale high-performance computing systems.

In the interview, I pointed out that the most obvious way to reduce computing-related energy consumption is simply to power down and turn off those systems not being used -- QED. However, that is insufficient alone. After all, one presumably wants to do some computing. Thus, systems and infrastructure must be designed for energy and operational efficiency and must be managed appropriately during operation.

As a practical matter, one really wants to maximize a ratio

(Effective operations)/(Cost times Watts)

Simply put, the goal is to maximize the number of effective operations relative to cost and energy consumption. This convolves many ideas, including the match of the application to the system (application execution efficiency), the system design and architecture, energy and power supply efficiency, packaging and cooling overhead, market costs for power and hardware and the costs of people and money.

Microsoft is absolutely committed to green computing, across its entire range of products and infrastructure, and a big portion of my team's research is related to developing more energy efficient computing systems at scale.