Switching linear dynamic systems with higher-order temporal structure

Automated analysis of temporal data is a task of utmost importance for intelligent machines. For example, ubiquitous computing systems need to understand the intention of humans from the stream of sensory information, and health-care monitoring systems can assist patients and doctors by providing automatically annotated daily health reports. In addition, a huge amount of multimedia data such as videos await to be analyzed and indexed for search purposes, while scientific data such as recordings of animal behavior and evolving brain signals are being collected in the hope to deliver a new scientific discovery about life.;In this dissertation, I present a set of extensions of switching linear dynamic systems (SLDSs) which provide the ability to capture the higher-order temporal structures within data and to produce more accurate results for the tasks such as labeling and estimation of global variations within data. The presented models are formulated within a dynamic Bayesian network formulation along with the inference and learning methods thereof.;The previous state-of-the-art standard SLDSs model the nature of continuous multivariate temporal data under the assumption that the characteristics of complex non-linear temporal sequences can be captured by Markov switching between a set of simpler primitives which are linear dynamic systems (LDSs). Accordingly, the SLDS model provides us with the ability to learn the temporal models from training data and to label novel sequences according to regimes that exhibit different dynamics.;However, the standard SLDS model is lacking in several aspects, which leads to its shortcomings in the scope of the data and the tasks it can handle, and in the quality of the labeling results. First, in terms of the quality of the continuous labeling tasks without known segment boundaries, it produces inaccurate and often over-segmented labeling results because it blindly adopts the geometric duration models implied by the Markov assumption. Second, the standard SLDS model does not provide principled mechanisms to capture or to infer the amount of global variations within data, which we refer to as the quantification task. Accordingly, they tend to produce both inaccurate labeling and quantification results. Third, it can not effectively model the data with grammar-like hierarchical temporal structure. Accordingly, the standard SLDSs do not provide means to interpret data at multiple temporal or semantic granularities and often produce less than impressive labeling results.;In this dissertation, we address all of the above limitations of standard SLDSs by enhancing the model to incorporate higher-order temporal structures.;First, segmental SLDSs (S-SLDSs) produce superior labeling results by capturing the descriptive duration patterns within each LDS segment. The encoded duration models describe data more descriptively and allow us to avoid the severe problem of over-segmented labels, which leads to superior accuracy.;Second, parametric SLDSs (P-SLDSs) allows us to encode the temporal data with global variations. In particular, we have identified two types of global systematic variations: temporal and spatial variations. The P-SLDS model assumes that there is an underlying canonical model which is globally transformed in time and space by the two associated global parameters respectively. Accordingly, P-SLDSs can solve the quantification problem of estimating the global variations within data and simultaneously produce the labeling results with superior accuracy.;Third, we present hierarchical SLDSs (H-SLDSs), a generalization of standard SLDSs with hierarchic Markov chains. H-SLDSs are able to encode temporal data which exhibits hierarchic structure where the underlying low-level temporal patterns repeatedly appear among different higher-level contexts. Accordingly, H-SLDSs can be used to analyze temporal data at multiple temporal granularities, and provide the additional ability to learn a more complex H-SLDS model easily by combining underlying models.;The developed SLDS extensions have been applied to two real-world problems. The first problem is to automatically decode the dance messages of honey bee dances where the goal is to correctly segment the dance sequences into different regimes and parse the messages about the location of food sources embedded in the data. We show that a combination of the P-SLDS and S-SLDS models has demonstrated improved labeling accuracy and message parsing (an instance of a quantification task) results. The second problem is to analyze wearable exercise data where we aim to provide an automatically generated exercise record at multiple temporal and semantic resolutions. It is demonstrated that the H-SLDS model with multiple layers can be learned from data, and can be successfully applied to interpret the exercise data at multiple granularities. It is also shown that the H-SLDS model produces superior labeling results than the standard SLDSs for low-level semantic patterns, due to the use of higher-level temporal structure.