Benchmarking and BenchMARKETing: The Light Side and the Dark Side of Performance Evaluation
The presentation describes two projects in the area of performance evaluation. The first project, Parallelism Evaluation Methodology, demonstrates the relationship of parallelism to performance, and shows where various agents change the parallelism on the path to performance. A simulation tool is described which allows the study of executed parallelism within a single CPU of a CRAY Y-MP. A methodology is presented and applied to the measurement and comparison of executed parallelism within several of the Perfect Benchmarks.
In the second project, Benchmark Set Construction Methodology, results are presented on the profile of usage, by populations of academic users in different application areas, on the CRAY Y-MP/464 at the National Center for Supercomputing Applications. Hardware Performance Monitor data collected over a period of one year is analyzed. The collected data contains HPM group 0 performance information for approximately 300,000 jobs representing about 18,000 CPU-hours. This data serves as a specification to be approximated by a benchmark set which represents the workload at this site.
If time allows, a short presentation will be given on the practice of BenchMARKETing: The Art of Selling Inferior Goods.
John Larson received a Ph.D. in Computer Science from the University of Illinois at Urbana-Champaign, and has worked for Burroughs Corporation on the 17-processor BSP, at Cray Research on the CRAY X-MP and Y-MP multiprocessors, and at the Center for Supercomputing Research and Development where he conducted research in performance evaluation. Today's talk highlights two of the research projects John conducted at CSRD.