High-Dimensional Data Clustering and Statistical Analysis of Clustering-based Data Summarization Products

With the advancement of modern technology, we have seen the expansion of data in two dimensions: number of variables and number of observations. Such high-dimensionality and large data volume have posed new challenges to statistical analysis. This thesis considers two problems related to cluster analysis: high-dimensional data clustering and statistical analysis of clustering-based data summarization products.;High-dimensionality often makes traditional clustering methods ineffective. Variable selection is a common approach to reduce the dimensionality of data for better cluster analysis. Most of recently developed methods either explicitly or implicitly perform variable selection based on variable importance ( VI) measure. In this thesis, an algorithmic framework is introduced which iterates between constructing VI and performing variable selection conditioning on each other. Within this framework, we develop an ensemble VI which is constructed by averaging a set of VI's. Both theoretical and simulation studies show that the proposed ensemble VI has better variable selection performance than unensemble VI and is robust to the choice of the number of groups in cluster analysis. In addition to the development in VI, we propose a new VI-based variable selection method which selects a set of variables through sequentially testing the existence of group structure in data. Its effectiveness is demonstrated through simulation study and a real data application.;In the second problem, we consider a histogram type of data summarization product from massive climate data. Unlike traditional data reduction method which summarizes the observations in each spatial grid-box during a period of time by their average, a NASA science team recently uses a multivariate histogram, constructed by cluster analysis, to represent those observations. This method has been applied to the observations collected by Atmospheric Infrared Sounder (AIRS) to produce AIRS L3Q products. In this thesis, we study potential statistical tools using pairwise dissimilarities that are suitable for analyzing this histogram type of data. Through theoretical analysis and simulations, we investigate several different dissimilarity measures and find Mallows distance is preferable over others when the locations of the representative vectors are important for the analysis. We apply MultiDimensional Scaling and clustering method to analyze the AIRS data collected in December 2002. The results from these studies show the effectiveness of statistical methods based on Mallows distance in extracting information from this histogram type of data.