Study On Feature Selection Algorithm Based On Least Squares Support Vector Machine And Particle Swarm Optimization

Classification problem is a primary problem in the field of pattern recognition. The accuracy of classification is related with the choice of classifier, the number of samples and the dimension. The feature selection is the key problem. It is important to extract the feature which is useful and effective for classification. Recently, along with the improvement of science and technology, the dimension of samples that obtained in some certain field is large. So the feature selection is becoming more and more important. Lots of researchers are devoting to the study on feature selection.The study on feature selection is focus on the two ways: filter approach and wrapper approach. Filter approaches are usually stated as individual feature ranking methods. They evaluate a feature based on its contribution to the class discrimination without considering its interactions with other features. The feature selection procedure is independent of the classification procedure because a classifier is not built when evaluating a feature. But the wrapper approaches use the learning algorithm itself as a part of the evaluation criteria. The feature selection procedure is combined with the classification procedure. In general, the filter approach costs much less computational time and storage space than that of the wrapper approach. However the predicting accuracy for the filter approach is generally less than that of the wrapper approach.The statistical learning theory which is proposed by Vapnik avoids the shortcomings in artificial neural networks such as the decision of the network structure, over learning, short learning and local minimal. The statistical learning theory is considered as the best theory for small sample classification and function regression. Based on the statistical theory, Vapnik proposed support vector machine (SVM) in 1995. SVM drew more attention and developed in theory studying and algorithm implementing. It has been used widely in pattern recognition, regression analysis, signal process and function regression, et.al. It is becoming the hot topic in machine learning. However the traditional SVM should solve a quadratic programming (QP) with inequation constrains. It is difficult to solve a large sample leaning problem for the solving technique, hardware condition and long learning time. Suykens et.al converted the inequation constrains into linear equation constrains. The learning of SVM is converted into linear equation system. This method is called least square support vector machine (LSSVM). LSSVM simplified the learning of SVM and lowered the difficulty of solving SVM.Kennedy and Eberhart proposed particle swarm optimization (PSO) in 1995, Inspired by the social behavior of bird flocking or fish schooling. This is a heuristic global optimization algorithm and it is an evolutionary computation algorithm based on swarm intelligent. Kennedy and Eberhart initially proposed this algorithm to simulate the simplified social system, study and explain the complex social behave. And then they found that PSO can be used to solve the complex optimization problem. PSO has some difference to genetic algorithm which based on Darwinian evolutionary theory. PSO searches for the optimal by the cooperation in individual. The individual in PSO is a particle without quality and volume and the behavior of particle is preordered. The whole swarm has complex feature and could solve complex optimization problem. PSO is used initially to optimize the continuous function. Kennedy and Eberhart proposed binary particle swarm optimization (BPSO) in 1997 and the BPSO is suitable to solve the combination optimization problem. BPSO encodes the particle with binary. In the BPSO model the number in each dimension of the particle position is constrain to 1 or 0. The velocity of particle does not constrain. The possibility of position change is denoted by the Sigmoid function of velocity. PSO has been developed widely and recognized in the international evolutionary computation fields for its simple theory and easy implement. It has been applied in many fields such as the optimization of electrical system, TSP problem, the training of artificial neural network, the optimization of digital circuitry, function optimization and parameter identification, et.al.In recent years, the evolutionary algorithm based on biology intelligent is developed widely. Meanwhile a lot of feature selection approaches based on intelligent algorithm and hybrid algorithm appeared. An improved PSO and LSSVM are combined for feature selection in this paper. The contents on this paper are as follows:(1) Firstly, introduce the basic principle of statistic learning theory, support vector machine, least square support vector machine. And then introduce the theory and algorithm steps of particle swarm optimization and binary particle swarm optimization. The classification of the feature selection algorithm and the feature selection approach based on swarm intelligence are also introduced. (2) Propose a feature selection approach based on particle swarm optimization and least square support vector machine. This algorithm adopts improved binary particle swarm optimization to select feature and uses least square support vector machine to construct classifier. The accuracy on classification is the main factor to evaluate a feature. In PSO, a particle is a point in feature space and it corresponding to a feature subset. In the initialization step, the number of"1"in a particle is randomly determined and these"1"are distributed randomly on the dimension of a particle. This initialization method could reflect the variety of the feature. The improved iterative equation on position and velocity are adopted. The distance between the two positions is evaluated by the different digital in two strings. In the process of updating velocity, the position vector and velocity vector add and the result model"2"to map into"0"and"1". The improved particle swarm optimization combines with least square support vector machine to construct the feature selection approach. The results of simulation experience show that this method decreases the dimension of sample and improve the efficiency of classification.