Multiobjective Learning And Optimization:Theory And Application

With the rapid development of Internet and information techniques,machine learning has attracted widespread attention in a variety of real-world applications,such as health care,public transport,online shopping,and national security.Recently,many researches have pointed out that the machine learning problems have several objectives.For example,in supervised classification,a classification model should not only minimize the approximation error on the training data,but also improve the generalization ability.The traditional machine learning methods aggregate several objectives into a scalar cost function.In general,it is difficult to design appropriate regularization term and determine the regularization parameter.Evolutionary multi-objective optimization no longer needs to specify the regularization parameter and can obtain a set of nondominated solutions.The decision maker is able to extract knowledge and make decision to select the final solution.Machine learning can also be used to enhance multi-objective evolutionary algorithms.Inspired by the multi-task learning,evolutionary multitasking optimization aims to solve multiple optimization tasks simultaneously for employing the relevant information available in related tasks.This thesis investigates the multi-objective theory from the perspective of learning and optimization,respectively.On the one hand,this thesis models the self-paced learning problem as a multi-objective optimization problem,which is solved by a multi-objective particle swarm optimization algorithm.On the other hand,this thesis establishes an evolutionary multitasking framework to simultaneously optimize multiple sparse reconstruction tasks using a single population.The main works are summarized as follows:(1)Self-paced learning involves easier samples into training at first and then gradually takes more complex ones into consideration.It uses a weight variable to reflect the easiness of a sample and an increasing pace parameter is utilized to control the pace for learning new samples.Therefore,a set of solutions can be obtained and these solutions constitute the solution path of the self-paced learning problems.However,it is difficult to choose the suitable pace parameters and determine where to optimally stop this increasing process.This thesis proposes a multi-objective self-paced learning method to optimize the loss function and the self-paced regularizer simultaneously for avoiding the selection of the pace parameter.In the proposed method,a decomposition-based multi-objective particle swarm optimization algo-rithm is used to simultaneously optimize the two objectives for obtaining the solutions.A satisfactory solution can be naturally obtained from these solutions by utilizing some effective tools in evolutionary multi-objective optimization.Experiments on matrix factorization demonstrate the effectiveness of the proposed technique.(2)In recent years,several efficient self-paced regularizers have been proposed,such as hard weighting,linear soft weighting,logarithmic soft weighting and mixture weighting.Among them,the logarithmic soft weighting and the mixture weighting regularizers have complex forms,from which the pace parameter cannot be separated.Therefore,the logarithmic soft weighting and the mixture weighting regularizers cannot be used in the proposed multi-objective self-paced learning model.This thesis proposes a novel polynomial soft weighting regularizer to address this issue,which has a simple form and can penalize the loss according to the problem requirements.Theoretical studies are conducted to show that the previous regularizers are roughly particular cases of the proposed polynomial soft weighting regularizer family.Experiments on action recognition and media event detection demonstrate that the proposed soft weighting regularizer can achieve the best results in most cases.(3)Most of the classic regression or classification methods can be improved by self-paced learning.Alternative search is used to solve the objective function of self-paced learning.When the weight variable is fixed,the optimization problem is a standard learning problem with weighted losses.This thesis proposes an unsupervised change detection method based on self-paced learning.In the proposed method,a pseudo training set is generated by the unsupervised method.Because some training samples have wrong labels,self-paced learning is used to automatically learn the reliable samples.This thesis deduces the formulas of support vector machine classification under the weighted loss function.The weight imposes an upper-bound on the support vector coefficient.Then this thesis deduces the formulas of artificial neural networks under the weighted loss function.The sample weights can dynamically control the learning rates for converging to better values.Experiments on two change detection datasets demonstrate that the self-paced learning methods are robust to the noisy labels and can obtain satisfactory change detection results.(4)Inspired by multi-task learning,evolutionary multitasking optimization aims to reveal the multitasking potential of evolutionary algorithms.In order to exploit the similar sparsity pattern between different tasks,this thesis establishes an evolutionary multitasking framework to simultaneously optimize multiple sparse reconstruction tasks using a single population.In the proposed method,the evolutionary algorithm aims to search the locations of nonzero components or rows instead of searching sparse vector or matrix directly.Then the within-task and between-task genetic transfer operators are employed to reinforce the exchange of genetic material belonging to the same or different tasks.The proposed method can solve multiple measurement vector problems efficiently because the length of decision vector is independent of the number of measurement vectors.Experiments on the six simulated problems demonstrate the effectiveness of the proposed method.(5)The sparse unmixing model assumes that all pixels in the hyperspectral images share the common set of spectral signatures.However,this assumption rarely holds because the hyperspectral images usually have several heterogeneous regions.The spectral signatures in heterogeneous regions may be different.Therefore,it is more effective to employ the spectral unmixing technique in the homogeneous regions.This thesis proposes a hyperspectral unmixing method based on multitasking sparse reconstruction.In the proposed method,we partition the hyperspectral image into several homogeneous regions.For each homogeneous region,the pixels are very likely to share a common set of spectral signatures.Therefore,solving the sparse unmixing problem in a homogeneous region can be considered as a task.Finally,the proposed evolutionary multitasking sparse reconstruction method can be used to solve these tasks simultaneously.Experiments on the simulated and benchmark problems demonstrate that the proposed method has better performance than other competing algorithms.