Discovering and Characterizing Hidden Variables in Streaming Multivariate Time Series

Hidden variables in time series data arise naturally in many domains, such as robotics, industrial process control, finance, medicine, and climatology. In many cases, however, variables that are known to be causally relevant to the observed behavior of the true model cannot be measured directly, or the existence of such variables is unknown. This thesis first presents a neural network architecture, called the LO-net, for inferring both the existence and the values of hidden variables in streaming multivariate time series, leading to a deeper understanding of the domain and to more accurate predictions. The core idea is to first make predictions with one network (the observable or O-net) based on a time delay embedding, following this with a gradual reduction in the temporal scope of the embedding, thus forcing a second network (the latent or L-net) to learn to approximate the value of a single hidden variable. This estimate is then input to the O-net using the original time delay embedding. Experiments show the utility of this proposed approach using discrete time dynamical systems in which some of the state variables are hidden, and sensor data obtained from the camera of a mobile robot in which the sizes and locations of objects in the visual field are observed but their sizes and locations (distances) in the three-dimensional world are not. In some situations, the L-net is seen to approximate the output from the O-net instead of learning the hidden variable. To avoid this undesirable behavior, the LO-net is regularized. Experimental results of this approach have been shown using different regularization terms.;A second method for discovering hidden variables in multivariate dynamical systems starts with a Linear Dynamical System (LDS) where a variable is hidden. A new LDS is formed with random values for the initial value and weights on the hidden variable. These parameters are found by gradient descent on the prediction errors with respect to the observable variables. Closed-form iterative solutions for the gradients with respect to the initial values and weights of the hidden variable in the LDS framework are presented. This method is then extended to non-linear dynamical systems such as the robot data. Experimental results show that this method provides good predictions.;The contributions of this thesis are two novel methods for discovering and characterizing hidden variables in multivariate time series data and an evaluation of these methods on various dynamical systems and real-world data. In most of the previous work the aim is to design better models that work with hidden variables. Our aim is to actually discover and quantify the hidden variables in order to improve the understanding of the over all model.