| The objectivity of statistical analysis hinges on the assumptions made about the form and complexity of the model used to fit the data. These usually take the guise of "nuisance parameters" which must be set based on some meta-level knowledge of the problem to be solved. This dissertation seeks to contribute statistical methods which require as little meta-level knowledge as possible, and yet are computationally and analytically tractable enough to operate on real-world datasets.; This goal is partially achieved within the framework of Bayesian statistics, which allows the specification of prior knowledge, and lets the data correctly constrain model complexity. However, for all but the simplest of statistical models, a full Bayesian treatment is often analytically and computationally intractable. We therefore explore the usefulness of approximation techniques; in particular, those stemming from variational calculus, to gain analytical tractability when performing statistical inference in complex graphical models.; We provide a novel, analytically closed-form solution to estimating the cardinality of mixture models, by locally approximating the evidence for splitting existing models, and thus growing complexity as needed. We contribute a solution to the problem of estimating forgetting rates for online learning by modeling the non-stationarity of the model as a set of drifting parameters, thus allowing a variational Kalman smoother to estimate the time scale of the process drift. We also address the estimation of Bayesian distance metrics for locally weighted regression---a problem commonly known as supersmoothing---by probabilistically modeling the kernel weights assigned to the data.; Another contribution of this dissertation is the development of statistical inference methods which are computationally scalable. We derive a probabilistic version of back-fitting---a highly robust and scalable class of supervised non-parametric algorithms---and demonstrate that, among others, the framework of sparse Bayesian learning arises from this class as a special case.; We conclude that in several difficult statistical learning problems, principled approximation techniques, and careful model construction can create scalable and robust algorithms which eliminate the most difficult model complexity parameters, while retaining their applicability to large, complex and underconstrained data sets. |
