Evaluation and Comparison of Search Engines Using the LSP Method
Oral Defence Date:
Thursday, May 10, 2007 - 14:00
Professor Jozo Dujmovic, and Professor Hui Yang
Search engines were introduced in 1994 and the first attempts to develop techniques for their evaluation were published in 1996. During the period of ten years the search technology made a dramatic progress and currently search engines are the most influential web tools. General search services are offered by Internet service providers (e.g. AOL) and are accessible through all web browsers (e.g. Netscape). According to a study conducted by Nielsen NetRatings in November 2005 out of the 5.1 billion searches placed in that month, Google, Yahoo! Search, MSN Search, AOL Search, My Way, and Ask served 92.8% of the requests, with Google taking a leading 46.3%. Regardless a visible industrial interest in search engine companies and intensive research in the area of information retrieval, the existing search engine evaluation efforts are restricted to evaluating only selected aspects of search technology. Search engines are not quantitatively evaluated as complex industrial products that are designed to satisfy user requirements. This project focuses on building a comprehensive model for evaluation and comparison of general search engines that use proprietary search technology. Our objective is to evaluate the ability of search engines to satisfy a set of user requirements. We developed a quantitative model for search engine evaluation using the LSP method for evaluation and comparison of complex systems. The experimental part of the project includes the evaluation of Yahoo! Search, Ask, Google, and MSN. The criterion is based on needs of both nonprofessional and professional searches. To evaluate the performance of search process we developed a specialized search engine benchmarking tool, and used it measure a spectrum of performance indicators. The results of performance measurement are then aggregated with all attributes that reflect the functionality and usability of search engines to generate a compound indicator of the overall search engine quality.
evaluation, LSP method, general purpose search engines, quantitative comparison, benchmarking