Adaptive Gaussian Processes For Multi-task Pattern Learning And Extrapolation

Adaptive multi-task patterns learning and extrapolation are applied in many applications,such as climate change forecasting,market fluctuation warning,environment monitoring,and river flow estimation.Particularly for Internet of Things(IoT)applications,we can see extensive specific scenarios with potential demand of adaptive multi-task patterns learning and extrapolation.Multi-Task Gaussian processes(MTGPs)have shown a significant progress both in expressiveness and interpretation of the relatedness between different tasks: from linear combinations of independent singleoutput Gaussian processes(GPs),through the direct modeling of the cross covariances.Note that all these advanced and representative MTPGs have to encode intrinsic correlation of input space and tasks interaction.In order to correct the deficiency of intrinsic correlation describing and tasks interaction modeling with increasing representation of MTGPs,we proposed a structured and expressive generalized convolution spectral mixture kernel(GCSM)for single task GPs and then extended it to account for multi-task learning scenario with cross convolution of channels and cross coregionalization.First,Spectral Mixture(SM)kernels form a powerful class of kernels for Gaussian processes,capable to discover patterns,extrapolate,and model negative covariances.Being a linear superposition of quasi-periodical Gaussian components,an SM kernel does not explicitly model dependencies between components.In this paper we investigated the benefits of modeling explicitly time and phase delay dependencies between components.We presented a framework to analyze the presence of such dependencies by using posterior covariance and covariance of components,and used it to provide motivating examples.We proposed an extension of the SM kernel,called GCSM,that explicitly models dependencies.GCSM is constructed in two steps: first,time and phase delay are incorporated into each base component(the square root of an SM component),next cross-convolution between a base component and the reversed complex conjugate of another base component is applied to obtain a complex-valued and positive definite kernel representing correlations between base components.This approach allows also to model SM kernels by considering only auto-convolution of base components.The number of hyper-parameters(for zero time and phase delay)remains equal to that of the SM kernel.We performed a thorough comparative experimental analysis of GCSM on synthetic and real-life data sets.Internal cross-validation is used to assess whether to use time and/or phase delay.Results indicated the beneficial effect of modeling time and phase delay dependencies between components,notably for natural phenomena involving little or no influence from human activity,like the monthly river flow extrapolation task,where moon and sun are primarily responsible for the rising and falling of river tidal flows which are delayed and augmented under the influence of gravity and resonances.Second,inspired from the proposed GCSM,we used the convolution theorem to design a new kernel for MOGPs,by modeling cross channel dependencies through cross convolution of time and phase delayed components in the spectral domain.Multi-output Gaussian processes(MOGPs)are an extension of Gaussian Processes for predicting multiple output variables(also called channels,tasks)simultaneously.The resulting kernel is called Multi-Output Convolution Spectral Mixture(MOCSM)kernel.Results of extensive experiments on synthetic and real-life sensor network datasets demonstrated the advantages of the proposed kernel and its state of the art performance.MOCSM enjoys the desirable property to reduce to the well known Spectral Mixture(SM)kernel when a single-channel is considered.A comparison with the recently introduced MultiOutput Spectral Mixture kernel reveals that this is not the case for the latter kernel,which contains quadratic terms that generate undesirable scale effects when the spectral densities of different channels are either very close or very far from each other in the frequency domain.Third we further analyzed Gaussian process regression networks(GPRN)framework for MTGPs and provided a parametric interpretation of the relatedness across tasks.While,all these advanced and representative GPRN based MTPGs used a sum of multiple independent channels to describe intrinsic correlation of input space and tasks interaction.Therefore here we encoded and enriched the channels dependencies of GPRN.In this paper we further extended expressiveness and interpretability of MTGPs models and introduced a new family of kernels capable to model two level channel dependency encoding intrinsic correlation and tasks interaction,including time and phase delay.Specifically,we used cross convolution of independent channels modeled by SM kernel to account for intrinsic correlation of channels dependency,proposed coupling coregionalization of LMC from different channels to represent tasks interaction of channels dependency.Both two level channel dependencies allowing information exchange between channels and explicitly dependent behaviors modeling give a generalized convolution of SM channel with coupling coregionalization(GCSM-CC),which as a new MTGPs framework for more extensive and generalized multi-task learning.Such mechanism enhances dependent patterns learning.The proposed kernels for MTGPs are validated on artificial data and compared with existing MTGPs on three real-world sensor network experiments.Results indicated the benefits of our more expressive representation with respect to performance and interpretability.