Researches On The Auxiliary Problems In Structural Learning

Multi-task learning refers to a methodology, which takes a certain domain as background and utilizes the knowledge derived from the related tasks to solve the target problems in that domain. It belongs to the field of transfer learning and develops vigorously in recent years. In particular, structural learning framework applies the algorithm of Alternating Structure Optimization (ASO) to learn the structures of predictors space shared by multiple related tasks. The experimental results are satisfying in lots of applications. However, the ultimate experimental results largely depend on whether the auxiliary problems (APs) are good or not. To our knowledge, there exist few researches on it.We focus on the principles called principle of relevancy and principle of orthogonality for APs selection and then obtain some valuable conclusions. We also discuss the property of domain adaptation for structural learning. Furthermore, we validate that above principles and property are feasible on the improved cASO (convex ASO) algorithm. Finally, we apply these principles and property to a specific natural language processing (NLP) task,"Chinese semantic role labeling". The experimental results demonstrate that these conclusions are credible and feasible. In addition, the major innovation points in this paper are as follows.1. We focus on the metrics of principle of relevancy between APs and TPs in structural learning framework. It is researched by taking example of Chinese syntactic chunking. Four types of APs are created. Through substantive experiments and analyses, some valuable conclusions with regard to it are obtained. That is, if the APs are predicting head nouns of the sentences, the relevancy of them is greater than that of other types of APs.2. We propose a new principle called principle of orthogonality for APs selection. We first give theoretical analyses on it, and then carry experiments on the task of Chinese syntactic chunking. They both validate the following facts. If the weight matrices of different types of APs are orthogonal or approximately orthogonal, the multi-combinations of them perform better than or equal to any components of them. Even if the total amounts of APs are given, we can also obtain the same conclusion provided the ratios of different types of APs are appropriate in the multi-combinations. Moreover, we draw some conclusions on how to select appropriate total amount of APs. In short, the principle of orthogonality is credible.3. We study the property of domain adaptation in structural learning. Theoretical analyses and experimental results both indicate that the performances are still satisfying if unlabeled data (APs) come from different source domains. Even if the data distributions of source domains and target domain (TPs) are quite different, that conclusion is still established. Namely, there exists the property of domain adaptation in structural learning.4. We do some researches on the convex ASO (cASO) algorithm. The experimental results show that the principles of relevancy and orthogonality proposed by this paper are still established for cASO algorithm. There also exists the property of domain adaptation in it. Moreover, the performances of cASO algorithm are superior to those of ASO algorithm in the same experimental settings.5. The technique of semantic role labeling (SRL) has a wide range of applications in NLP, such as question answering (QA), machine translation (MT), information extraction (IE). We apply the principles of relevancy and orthogonality proposed above to it. Experimental results demonstrate that these principles are reasonable and credible.