Research Of Clustering For LAMOST Early M Type Spectra Based On SPARK Platform

Astronomy is a science highly depending on the observations. Observation data is constantly accumulated over time. Especially in recent years, Sky Survey Plan makes the observation data reached an unprecedented magnitude. LAMOST sky survey, an important scientific sky survey of China has released three version spectrum data. Research of clustering algorithm application in LAMOST early M dwarfs based on Spark is done in this paper.K-Means, Bisecting K-Means, DNSCAN and OPTICS based on Spark are used in this paper for clustering of LAMOST DR3 M early dwarfs. Experiments show that Spark platform is appropriate for clustering algorithm parallelization. K-Means and Bisecting K-Means are more appropriate for sub classification early M dwarfs which DBSCAN and OPTICS are not. DBSCAN and OPTICS has more advantages in rare data finding. The dimensionality reduction method for LAMOST M dwarfs proposed in this paper works well in clustering of classification. The main research work includes:1. Using Spark platform for early M dwarfs clustering can improve efficiency significantly while not increasing the complexity of programming. Clustering algorithm can be write on Spark effectively.2. According to the experiments of several distance-measurement, Bisecting K-Means has more advantage than K-Means in classification on M dwarfs. And DBSCAN and OPTICS are not appropriate here.3. Implemented a new dimensionality reduction method which is combined by indices, PCA, median filtering and polynomial fitting, and it has good performance on sub classification of early M dwarfs.