Blue Waters.[/caption] A small revolution of sorts is brewing. Some people are quietly discussing alternatives to the popular Top500 list of the world's fastest supercomputers. Bill Kramer, deputy project director for Blue Waters, the supercomputer belonging to the National Center for Supercomputing Applications (NCSA) at the University of Illinois at Urbana-Champaign, has said previously that he will not be submitting scores to Top500.org, the organization that compiles the list. Kramer said that work is being done by people such as Michael Heroux, a distinguished member of the technical staff at Sandia National Laboratories, to develop an alternative benchmark suite for evaluating supercomputer performance; Heroux acknowledged this. If such an alternative is agreed upon, the Top500 list might diminish in significance. Kramer's reasons for seeking alternatives are simple: Linpack, the performance benchmark used to compile the list, doesn't adequately measure the performance of today's modern supercomputers. "It's not only inadequate, it's counter-productive," he said in a recent interview. "First, to be clear, there's nothing wrong with the Linpack benchmark, when used appropriately... but it is no longer the dominant indicator for how well a system will do measuring real problems. "So it's a fine benchmark, used for its own purposes,” he added. “However, the way it's being used, in conjunction with a list that implies how useful a system is, is actually misleading and counter-productive.” Erich Strohmaier, a senior scientist at Lawrence Berkeley National Laboratory and a co-founder of the Top500 list, did not respond to requests for comment via email. Kramer said that he had no issue with compiling a list of the highest-performing supercomputers; indeed, he felt that Blue Waters (which he said generates 1.37 petaflops of sustained activity) would rank quite high on another benchmark. The Top500 list uses both maximum and peak figures generated by Linpack, an algorithm performing linear algebra, originally designed to benchmark supercomputers in the 1970s and 1980s. It's the tool used by Hans Meuer and Strohmaier in compiling the list, which began in 1993; since then, it has become the benchmark of benchmarks, the list by which supercomputers are evaluated and funded, which is where Kramer's problems begin. Systems are being funded and designed to rank on the list, he said, instead of being designed for real-world computing problems. Instead, Kramer favors benchmarks such as the Sustained Petascale Performance Test, a collection of 12 application benchmarks that are complete applications drawn from Blue Waters research teams. The SPP can be packaged up and distributed to serve as a more generic performance evaluation suite, after licensing issues are worked out. Other tests Kramer said he prefers include the NERSC5 SSP benchmark, representing scientific applications run at the National Energy Research Scientific Computing Center. When measured via these benchmarks, top systems on the Top500 list such as Oak Ridge's National Laboratories' Titan show only about half the performance of Blue Waters, Kramer said: "The main point is sustained performance.” Sandia's Heroux also is working on an alternative. Heroux initially said he was willing to provide more details, but after reconsideration said that he would rather not. " We are just getting started and I don't want to put out details that may be irrelevant in a few months," he said in an email. Benchmarking has often proven a controversial measure, with early PCs living and dying by what seemed to be small increments in the frame rate of the most popular games. The "0-60" metric is still used to evaluate the performance of an automobile. The Green500 list of supercomputers prioritizes energy efficiency. But Kramer hopes that his actions are seen as a way to help the HPC market think of performance in a different way. "Nobody up until now has been able to stand up and say that I'm not participating because it's going in the wrong direction," he said. Image: http://www.ncsa.illinois.edu/ (Blue Waters) Alternative to Top500 List In the Works
[caption id="attachment_9779" align="aligncenter" width="618"] Blue Waters.[/caption] A small revolution of sorts is brewing. Some people are quietly discussing alternatives to the popular Top500 list of the world's fastest supercomputers. Bill Kramer, deputy project director for Blue Waters, the supercomputer belonging to the National Center for Supercomputing Applications (NCSA) at the University of Illinois at Urbana-Champaign, has said previously that he will not be submitting scores to Top500.org, the organization that compiles the list. Kramer said that work is being done by people such as Michael Heroux, a distinguished member of the technical staff at Sandia National Laboratories, to develop an alternative benchmark suite for evaluating supercomputer performance; Heroux acknowledged this. If such an alternative is agreed upon, the Top500 list might diminish in significance. Kramer's reasons for seeking alternatives are simple: Linpack, the performance benchmark used to compile the list, doesn't adequately measure the performance of today's modern supercomputers. "It's not only inadequate, it's counter-productive," he said in a recent interview. "First, to be clear, there's nothing wrong with the Linpack benchmark, when used appropriately... but it is no longer the dominant indicator for how well a system will do measuring real problems. "So it's a fine benchmark, used for its own purposes,” he added. “However, the way it's being used, in conjunction with a list that implies how useful a system is, is actually misleading and counter-productive.” Erich Strohmaier, a senior scientist at Lawrence Berkeley National Laboratory and a co-founder of the Top500 list, did not respond to requests for comment via email. Kramer said that he had no issue with compiling a list of the highest-performing supercomputers; indeed, he felt that Blue Waters (which he said generates 1.37 petaflops of sustained activity) would rank quite high on another benchmark. The Top500 list uses both maximum and peak figures generated by Linpack, an algorithm performing linear algebra, originally designed to benchmark supercomputers in the 1970s and 1980s. It's the tool used by Hans Meuer and Strohmaier in compiling the list, which began in 1993; since then, it has become the benchmark of benchmarks, the list by which supercomputers are evaluated and funded, which is where Kramer's problems begin. Systems are being funded and designed to rank on the list, he said, instead of being designed for real-world computing problems. Instead, Kramer favors benchmarks such as the Sustained Petascale Performance Test, a collection of 12 application benchmarks that are complete applications drawn from Blue Waters research teams. The SPP can be packaged up and distributed to serve as a more generic performance evaluation suite, after licensing issues are worked out. Other tests Kramer said he prefers include the NERSC5 SSP benchmark, representing scientific applications run at the National Energy Research Scientific Computing Center. When measured via these benchmarks, top systems on the Top500 list such as Oak Ridge's National Laboratories' Titan show only about half the performance of Blue Waters, Kramer said: "The main point is sustained performance.” Sandia's Heroux also is working on an alternative. Heroux initially said he was willing to provide more details, but after reconsideration said that he would rather not. " We are just getting started and I don't want to put out details that may be irrelevant in a few months," he said in an email. Benchmarking has often proven a controversial measure, with early PCs living and dying by what seemed to be small increments in the frame rate of the most popular games. The "0-60" metric is still used to evaluate the performance of an automobile. The Green500 list of supercomputers prioritizes energy efficiency. But Kramer hopes that his actions are seen as a way to help the HPC market think of performance in a different way. "Nobody up until now has been able to stand up and say that I'm not participating because it's going in the wrong direction," he said. Image: http://www.ncsa.illinois.edu/ (Blue Waters)
Blue Waters.[/caption] A small revolution of sorts is brewing. Some people are quietly discussing alternatives to the popular Top500 list of the world's fastest supercomputers. Bill Kramer, deputy project director for Blue Waters, the supercomputer belonging to the National Center for Supercomputing Applications (NCSA) at the University of Illinois at Urbana-Champaign, has said previously that he will not be submitting scores to Top500.org, the organization that compiles the list. Kramer said that work is being done by people such as Michael Heroux, a distinguished member of the technical staff at Sandia National Laboratories, to develop an alternative benchmark suite for evaluating supercomputer performance; Heroux acknowledged this. If such an alternative is agreed upon, the Top500 list might diminish in significance. Kramer's reasons for seeking alternatives are simple: Linpack, the performance benchmark used to compile the list, doesn't adequately measure the performance of today's modern supercomputers. "It's not only inadequate, it's counter-productive," he said in a recent interview. "First, to be clear, there's nothing wrong with the Linpack benchmark, when used appropriately... but it is no longer the dominant indicator for how well a system will do measuring real problems. "So it's a fine benchmark, used for its own purposes,” he added. “However, the way it's being used, in conjunction with a list that implies how useful a system is, is actually misleading and counter-productive.” Erich Strohmaier, a senior scientist at Lawrence Berkeley National Laboratory and a co-founder of the Top500 list, did not respond to requests for comment via email. Kramer said that he had no issue with compiling a list of the highest-performing supercomputers; indeed, he felt that Blue Waters (which he said generates 1.37 petaflops of sustained activity) would rank quite high on another benchmark. The Top500 list uses both maximum and peak figures generated by Linpack, an algorithm performing linear algebra, originally designed to benchmark supercomputers in the 1970s and 1980s. It's the tool used by Hans Meuer and Strohmaier in compiling the list, which began in 1993; since then, it has become the benchmark of benchmarks, the list by which supercomputers are evaluated and funded, which is where Kramer's problems begin. Systems are being funded and designed to rank on the list, he said, instead of being designed for real-world computing problems. Instead, Kramer favors benchmarks such as the Sustained Petascale Performance Test, a collection of 12 application benchmarks that are complete applications drawn from Blue Waters research teams. The SPP can be packaged up and distributed to serve as a more generic performance evaluation suite, after licensing issues are worked out. Other tests Kramer said he prefers include the NERSC5 SSP benchmark, representing scientific applications run at the National Energy Research Scientific Computing Center. When measured via these benchmarks, top systems on the Top500 list such as Oak Ridge's National Laboratories' Titan show only about half the performance of Blue Waters, Kramer said: "The main point is sustained performance.” Sandia's Heroux also is working on an alternative. Heroux initially said he was willing to provide more details, but after reconsideration said that he would rather not. " We are just getting started and I don't want to put out details that may be irrelevant in a few months," he said in an email. Benchmarking has often proven a controversial measure, with early PCs living and dying by what seemed to be small increments in the frame rate of the most popular games. The "0-60" metric is still used to evaluate the performance of an automobile. The Green500 list of supercomputers prioritizes energy efficiency. But Kramer hopes that his actions are seen as a way to help the HPC market think of performance in a different way. "Nobody up until now has been able to stand up and say that I'm not participating because it's going in the wrong direction," he said. Image: http://www.ncsa.illinois.edu/ (Blue Waters) 
