Distributed Representations For Cross-lingual Cross-task Natural Language Analysis

Feature representation is the foundation work of statistical machine learning,and also one of the key factors that affect the performance of a machine learning system.In the field of statistical natural language processing(NLP),the most commonly-used feature representation is the discrete symbolic representation,such as the One-Hot representation of words and the Bag-of-Words representation of documents.This kind of feature representation is intuitive,concise and easy to calculate.It has been successfully employed by most of the mainstream NLP tasks,combined with feature engineering and conventional machine learning algorithms,e.g.,Maximum Entropy Model,Support Vector Machine and Conditional Random Fields.Another important feature representation mechanism is known as distributed representation,mostly appearing as continuous,dense and lowdimensional vector representations of features.Typical distributed representation learning approaches include the early-stage latent semantic analysis and the “feature embedding”approach which has gained a lot of interests recently.In recent years,distributed feature representations have been extensively used in deep learning models for NLP.Compared with symbolic representations,distributed representations can be much more naturally combined with deep neural network models for learning high-level task-specific semantic representations through layer-wise representation abstraction.This has been an effective way of bridging the semantic gap in NLP.More importantly,distributed representation provides a universal semantic representation space across different tasks,languages and data modalities,so that training signals from multiple sources can be incorporated effectively,which further promotes knowledge transfer in practical learning tasks.For example,the distributed word representations learned from the neural network language modeling task on plain texts have been proven highly beneficial for a variety of NLP mainstream tasks.Inspired by these characteristics of distributed representation,especially its universality in semantic representation,this paper investigates the key technologies in distributed representation learning for knowedge transfer across languages,multi-typed data and tasks.To be more specific,our contributions include:1.Learning sense-specific word embeddings by exploiting bilingual parallel data.Single word embeddings have been shown poor in representing polysemy.To address this problem,we propose a novel and effective approach for learning sense-specific word embeddings by exploiting the sense-alignment information contained in bilingual resources.The proposed embeddings are expected to capture the multiple senses of polysemous words,and also benefit downstream applications.2.Learning multilingual distributed representations for cross-lingual transfer parsing.The majority of languages in the world are low-resource for dependency parsing,and it's labor-intensive to annotate treebanks for every language.We present cross-lingual word representation learning,to map words from different languages into a common vector space,and thus build a bridge connecting different languages.Therefore,the largescale treebanks of rich-resource languages can be exploited to induce parsers for lowresource languages through transfer parsing.3.A deep multi-task learning framework for transfer parsing across multi-typed treebanks.Various treebanks have been released for dependency parsing,either belonging to different languages or annotated with different schemes.We propose a deep multitask learning architecture with representation-level parameter sharing,in order to distill knowledge from multi-typed treebanks and benefit parsing of the target treebank.4.A unified model for semantic role labeling and relation classification.It is common for different NLP tasks to be related in certain ways.For example,semantic role labeling and(entity)relation classification both involve categorizing the semantic relation between words in a sentence.We propose a unified neural architecture that ties together the task of semantic role labeling and relation classification,and further apply deep multi-task learning to leverage their potential mutual benefits.Overall,this paper systematically and deeply investigates the application of distributed representation learning on knowledge transfer across languages,tasks and multi-typed data.We will show its effectiveness through substantial empirical studies on lexical,syntactic and semantic tasks respectively.In the future,we expect to apply our research achievements to more diverse data and tasks,even to domain adaptation,and finally make a significant difference in the NLP field.