Pharos: A scalable distributed architecture for locating heterogeneous information sources

Information retrieval over the Internet increasingly requires the filtering of thousands of information sources. As the number of sources increases, new ways of automatically summarizing, discovering, and selecting sources relevant to a user's query are needed. We introduce Pharos, a highly scalable distributed architecture for locating heterogeneous information sources. Its design is hierarchical, thus allowing it to scale as well as the number of information sources increases. We demonstrate the feasibility of the Pharos architecture using 2500 USENET newsgroups as separate collections. Each newsgroup is summarized via automated Library of Congress classification. We show that using Pharos as an intermediate retrieval mechanism provides acceptable accuracy of source selection compared to selecting sources using complete classification information, while maintaining good scalability.