CE-11.13

Title: 

CE-11.13

Author(s): 

Wilson Kwok

Oral Defence Date: 

12/02/2011

Location: 

TH 425

Committee: 

Professors Jozo Dujmovic and Dragutin Petkovic

Abstract: 

Benchmark suites are used to measure and evaluate the relative performance of competitive computer systems, software applications, and networks. Quantitative methods and criteria are used in measuring the quality of a benchmark suite in providing accurate performance measurements. The cost of industry standard benchmarking is directly proportional to the number of component benchmarks in benchmark suites. The goal of this project is to evaluate all component-level SPEC benchmark suites and determine whether the cost of developing these benchmark suites and performing corresponding measurements corresponds to the level of quality that they are able to achieve. Performance measurement results obtained using SPEC benchmark suites and of almost all commercially available computer systems are used in analyzing the quality of these benchmark suites. Our quantitative analysis is based on cluster analysis where each cluster contains similar workloads in itself but different from workloads in other clusters. This clustering method allows for quality assessments of the SPEC benchmark suites based on the level of similarity and difference between individual workloads where the quality of the benchmark suite is considered low if there is a high level of workload similarity or redundancy within the suite. Cluster analysis is also performed on filtered performance measurement data in a way that allows us to dive deeper into evaluating the quality of these benchmark suites from a different perspective. The presented analysis of the SPEC benchmark suites shows that SPEC component-level benchmark suites include highly redundant workloads where some groups of workloads are overrepresented and some are underrepresented.

Keywords: 

SPEC, CPU, CINT95 CFP95 CINT2000, CFP2000, CINT2006, CFP2006, CPU95, CPU2000, CPU2006, JVM98, OMP2001, OMPM2001, OMPL2001, MPI2007, MPIM2007, MPIL2007, benchmark suite, workload; quantitative criteria, performance measurement, agglomerative hierarchical clustering, white-box, black-box, redundancy, cluster analysis, utilization, density, filtering

Copyright: 

Wilson Kwok