| Recent years have witnessed the success of two broad categories of machine learning algorithms: (i) Online Learning; and (ii) Learning with nonlinear models. Typical machine learning algorithms assume that the entire data is available prior to the training task. This is often not the case in the real world, where data often arrives sequentially in a stream, or is too large to be stored in memory. To address these challenges, Online Learning techniques evolved as a promising solution to having highly scalable and efficient learning methodologies which could learn from data arriving sequentially. Next, as the real world data exhibited complex nonlinear patterns, it warranted the need for development of learning techniques that could search complex hypotheses space. Among the most notable successful methods for learning nonlinear models are kernel methods and deep neural networks. While these models enable searching complex hypothesis to learn models with a better performance, they are mostly designed for the batch setting which affects their scalability, and they also suffer from the difficulty in selecting the right hypothesis search space (e.g. which kernel to use, what architecture of neural network to use, etc.). In this dissertation we study the intersection of both these fields, and design novel algorithms that combine the merits of both online learning and nonlinear models by proposing methods that can learn nonlinear models in an online setting. Specifically, we investigate Online Learning Algorithms for Multiple Kernel Learning and Deep Neural Networks.;Multiple Kernel Models represent a class of high capacity models which are designed for learning highly nonlinear patterns, and also designed to handle multimodal data. Despite the promising ability, Multiple Kernel Learning is computationally very expensive, and it is a significantly challenging task to use such models in the online setting. In this dissertation we propose novel Online Multiple Kernel Algorithms, and make the following contributions:;We propose Online Multiple Kernel Regression Algorithms, which learn a kernel-based regressor in an online fashion, and dynamically explore a pool of diverse kernels to enhance the model performanceWe propose Temporal Kernel Descriptors, i.e., we design new kernels to effectively capture temporal properties of the data, and demonstrate the application of Online Multiple Kernel learning to applications which are sensitive to time.We propose Cost-Sensitive Online Multiple Kernel Classification, to address the challenges of learning online nonlinear models from imbalanced data streams, and also demonstrate the application of the proposed methods to online anomaly detection.;Learning with Deep Neural Networks (DNNs) has received increasing interest in recent years due to the overwhelming success demonstrated in several applications. However, using DNNs in the online setting remains an open problem, as most solutions are designed for the batch setting. In particular, choosing a right model architecture for online learning is a challenging task (in addition to convergence challenges such a vanishing gradient and diminishing feature reuse). To address these limitations, we develop algorithms for Online Deep Learning:We develop a novel Hedge Backpropagation algorithm which evolves theDNN from shallow to deep, thereby making DNNs online compatible. This way they are able to enjoy the fast convergence of Online Learning, and thepower of representation of Deep Learning. |
