Reliability of LSP Criteria
This presentation has two parts. In the first part we investigate the accuracy of parameters of LSP criterion functions for system evaluation. Main parameters are weights and conjunction/disjunction degrees (andness/orness). Weights reflect the level of relative importance of various decision variables. Andness/orness describes desired levels of simultaneity/replaceability in satisfying component criteria. These parameters are assessed by one or more professional evaluators and their values differ from (usually unknown) optimum values. We identify all potential sources of errors in LSP criterion functions. Our goal is to investigate the distribution of errors, their average values, and the quality of individual evaluators and their teams.
In the second part we analyze the reliability of evaluation results obtained using the LSP criteria. For each pair of competitive systems our goal is to compute the level of confidence in system ranking. The confidence is defined as the probability that the system ranking remains unchanged regardless of the criterion function parameter errors. We propose a simulation technique for the analysis of the reliability of ranking. The simulator is based on specific models for selection of random weights and random degrees of andness/orness. The proposed method is illustrated by a real life case study that investigates the reliability of evaluation and selection of a mainframe computer system.
Jozo Dujmovic' was born in Dubrovnik and received his BSEE, MS, and doctorate from the University of Belgrade. He is Professor of Computer Science at San Francisco State University, where he served for four years as Chairman of Computer Science Department. His teaching and research activities are in the areas of software metrics, decision analysis, and computer performance evaluation. He is the author of more than 100 refereed publications, recipient of three best paper awards, and a Senior Member of IEEE. He served as General Chair of IEEE MASCOTS 2000 and as a General Chair of ACM WOSP 2004.
Wen Yuan Fang received his BS and MS degrees from the Department of Computer Science at San Francisco State University. His professional and research interests are in the areas of decision analysis, software engineering, and system and security management. The results of his MS research are presented in two papers that are submitted for publication.