Learning aided system performance modeling in support of self-optimized resource scheduling in distributed environments

With the goal of autonomic computing, it is desirable to have a resource scheduler that is capable of self-optimization, which means that with a given high-level objective the scheduler can automatically adapt its scheduling decisions to the changing workload. This self-optimization capacity imposes challenges to system performance modeling because of increasing size and complexity of computing systems.;Our goals were twofold: to design performance models that can derive applications' resource consumption patterns in a systematic way, and to develop performance prediction models that can adapt to changing workloads. A novelty in the system performance model design is the use of various machine learning techniques to efficiently deal with the complexity of dynamic workloads based on monitoring and mining of historical performance data. In the environments considered in this thesis, virtual machines (VMs) are used as resource containers to host application executions because of their flexibility in supporting resource provisioning and load balancing.;Our study introduced three performance models to support self-optimized scheduling and decision-making. First, a novel approach is introduced for application classification based on the Principal Component Analysis (PCA) and the k-Nearest Neighbor (k-NN) classifier. It helps to reduce the dimensionality of the performance feature space and classify applications based on extracted features. In addition, a feature selection model is designed based on Bayesian Network (BN) to systematically identify the feature subset, which can provide optimal classification accuracy and adapt to changing workloads.;Second, an adaptive system performance prediction model is investigated based on a learning-aided predictor integration technique. Supervised learning techniques are used to learn the correlations between the statistical properties of the workload and the best-suited predictors.;In addition to a one-step ahead prediction model, a phase characterization model is studied to explore the large-scale behavior of application's resource consumption patterns.;Our study provides novel methodologies to model system and application performance. The performance models can self-optimize over time based on learning of historical runs, therefore better adapt to the changing workload and achieve better prediction accuracy than traditional methods with static parameters.