Research On Techniques Of Automatic Sematic Role Labeling Of Chinese FrameNet

In order to provide an automatic labeling tools for developing a large-scale resource of Chinese FrameNet, based on the semantic knowledge base of Chinese FrameNet(CFN) self-developed by Shanxi University, this paper study the semantic role(frame element) automatic labeling for given a target word in a sentence and its known frame name.The task of semantic role automatic labeling is conversed into sequential tagging problem at word-level within the entire sentence by IOB strategy. The conditional random fields model (CRF), and the orthogonal array experiment in statistics are employed.The experimental corpus in the paper, selected from current CFN corpus, include 6692 annotated sentences of 25 frames. The corpus is uniformly divided into four parts. Therefore,2-fold cross validation experimrnt can be engaged in the three different groups. We take the cross-validation average F1-value on three groups as the system performance measure. This paper presents the estimator of variance of the system performance measure as well as the significant test method for two different labeling system.Using word as the basic tagging unit, the tagging procedure is divided into three steps:1)identification,2) classification,3)post-processing. The two IOB strategies are adopted, one is conjunction of identifying and classifying, and the other is firstly identifying then classifying. In post-processing step, the final output of the sequential labels is choosen by the largest probability of all labels with a logical IOB sequence in entire sentence.This paper totally extract 26 features, and for each feature set some optional windows. The combination of various features with different windows form the feature templates of CRF model.The best template selection method is given based on orthogonal array in statistics.The three schemes of experiment are adopted. Scheme I: based on 11 word-level features, including word, POS of word, position of word relative to the target word, and the target word etc, the experiment is arranged in orthogonal array L32 (49×24); SchemeⅡ:based on all 26 features, including 11 word-level features and 15 base chunk features about shallow syntax, arranged in orthogonal array L54 (21×325).The base chunk features are automaticly extracted by automatic analyzer of Tsinghua University Zhou; Scheme III:batch orthogonal array experiment, i.e. first using the orthogonal array L32 (49 x 24) on 11 words-level features acquire the best templates, and then join the 15 base chunk features into orthogonal array L54 (21×325). Through the appropriate selection of the levels in orthogonal array L54 (21×325), it ensure that performance measure is not lower than the previous results.Each experiment conduct a detailed analysis.The paper compares the template selection method upon orthogonal array with traditional greedy algorithm, and compares the sequential tagging method at word-level with method upon syntactic parses tree, and also compares with different tagging models, such as support vector machine (SVM) model and maximum entropy(ME) model.Experimental results show:1)Based on 11 word-level features (schemeⅠ), the best average F1-value reach 61.61%.The result is significantly higher than method upon syntactic parses tree which regard the role labeling as a classification of syntactic constituents.On template selection, comparing with traditional greedy algorithm, the two methods dose not have significant differences, but orthogonal array method is relatively sample in calculation, and has some advantage in choice of the general template for any frame.2) Adding the 15 base chunk features (schemeⅡ) can significantly improve the performance. These features mainly have significant effects on role classification, not significant on role identification.3) Batch orthogonal array experiment (schemeⅢ) has a significant higher performance than the schemeⅡ.4) Two IOB stratagies, i.e. each frame training a model with role identification and classification together, or the firstly identifying then classifying, are no significant difference in performance, but the former has less variance of performance measure.5) The experimental results have no significant difference in annotation upon the conditional random fields model (CRF) and upon support vector machine (SVM) model, but CRF model is significantly better than a maximum entropy (ME) model. 6) In total 25 frames of all experiments, for semantic role identification the best average F1-value reach 71.68%. Given semantic role boundary, for the role classification the best average accuracy achieve 84.08%.Given target word in the sentence and its known frame name, the best average F1-value obtain 63.26%. Main innovations in the paper is the first systematic studies of Chinese FrameNet semantic role of automatic labeling, and proposes to adopt orthogonal array to select template. The method is more simple than the greedy algorithm template selection method. For feature selection problems in general sequential annotation, the orthogonal array feature template selection method is also applicable.For labeling performance, this paper's results is better than those based on syntactic parses tree.