“Blue Waters” Supercomputer Lucky to be Alive

One could argue that the University of Illinois’ “Blue Waters” supercomputer, scheduled to officially open for business March 28, is lucky to be alive.

The 11.6 petaflop supercomputer, commissioned by the University and the National Science Foundation (NSF), will rank in the upper echelon of the world’s fastest machines—its compute power would place it third on the current list, just above Japan’s K Computer. However, the system will not be submitted to the TOP500 list because of concerns with the way the list is calculated, officials said.

University officials and the NSF are lucky to have a machine at all.

That’s due in part to IBM, which reportedly backed out of the contract when the company determined that it couldn’t make a profit. Fox News reported that the university then turned to Cray, which would have had to replace what was presumably a POWER or Xeon installation with the current mix of AMD CPUs and Nvidia GPU coprocessors.

Allen Blatecky, director of NSF’s Division of Advanced Cyberinfrastructure, told Fox that pulling the plug was a “real possibility.” And Cray itself had to work to find the parts necessary for the supercomputer to begin at least trial operations in the fall of 2012.

“I could name probably 10 things we had to juggle because they were late,” Barry Bolding, Cray’s vice president of corporate marketing, told Fox. “We ran into those over and over and over again, and each time we had to fix it and move on… We took huge risks that individual pieces might not be delivered on time.”

Blue Waters is just one of the investments the NSF has made in supercomputers. Last October, it inaugurated “Yellowstone,” one of the world’s most powerful computers, based at the NCAR-Wyoming Supercomputing Center (NWSC). In addition to Blue Waters, the NSF has funded “Stampede” at the University of Texas in Austin. In total, the NSF has said it spent $200 million, with an additional $200 million more to be paid out in maintenance costs over the life of the systems. Blue Waters’ estimated cost is $350 million, the Chicago Tribune reported, with the remainder picked up by the university.

Blue Waters is a Cray XE/XK hybrid machine, with 276 cabinets housing 22,640 compute nodes made up of AMD 6276 “Interlagos” processors running at 2.3 GHz and above, for a total of 362,240 compute cores attached to 1.382 petabytes of system memory. Each compute node consists of two Interlagos processors (with a total of 16 cores) attached to 64 GB of memory, with a total bandwidth of 102.4 Gbytes/s.

Thirty-two of the cabinets consist of a joint CPU-GPU configuration, with 3.072 compute nodes attached to an equal amount of Nvidia “Kepler” GPU accelerators. The additional “XK” cabinets, as they’re known, contribute 4.51 petaflops of the 11.61-petaflop total. A total of 26.4 petabytes of total storage is included, with 1.2 petabytes of that used as near-line cache.

Fox reported that one of the early uses for the supercomputer is to simulate earthquake propagation on several Southern California faults, with the eventual hope that all of California’s fault lines can be included.


Image: U. of Illinois