Query processing for large-scale XML message brokering

Emerging distributed information systems such as Web services, personalized content delivery, and event monitoring require increasingly flexible and adaptive infrastructures. Recently, the publish/subscribe model has gained acceptance as a solution for the loose coupling of systems in terms of communication. Meanwhile, with respect to content, XML (Extensible Markup Language) is becoming a de facto standard for online data exchange. I propose an approach that integrates publish/subscribe and XML and, in particular, exploits declarative XML queries to offer flexibility and adaptivity in distributed systems. This approach is based on building XML message brokers, which I define as middleware components that perform three main functions: filtering, transformation, and routing of XML messages based on user-specified queries.;In this dissertation, I present YFilter/ONYX, an XML brokering system that provides the brokering functions for large numbers of queries over high volume message flows. I describe the architectural design of this system and its underlying technologies for providing efficiency and scalability. A key innovation is the exploitation of commonalities among queries; specifically, I present a series of novel sharing techniques that YFilter employs for filtering and more sophisticated transformation. A second innovation is the leveraging of relational techniques in the new context of XML message brokering; YFilter uses an effective mapping from XML transformation to relational query processing, which allows known relational techniques to be applied to achieve simplicity and performance of XML transformation. A third innovation is the design of a distributed system, called ONYX, that pushes declarative queries into the network for content-based routing and incremental processing of messages. I report on the results of extensive performance studies, demonstrating the efficiency and scalability of YFilter/ONYX under a wide variety of XML document types and query workloads.;In conclusion, YFilter/ONYX provides three key components, namely, filtering, transformation, and routing, for high-volume XML message brokering. As the adoption of XML-based distributed infrastructures gains momentum, the techniques developed in YFilter/ONYX and the results reported herein provide a foundation for building large-scale, high-function distributed information systems.