Research Of Exercise Modeling And Application Based On Multimodal Learning

Recent years have witnessed the rapid development of information technology,which brings us great convenience,as well as gradually influences the current education patterns and boosts education informationization.With the continuous advancement of educational informationization,many education systems which not only break the limitation of space and time of traditional classroom teaching but also provide learners lots of open-ended courses and reduce the cost of learning,have boomed.Meanwhile,these education systems have collected and accumulated millions of complex exercises which play an important role in strengthening students' knowledge,assisting teachers'teaching and improving educational management.How to model exercises effectively to understand and represent the semantics of them and apply the semantic information to exercise-based applications,has become one of the important research topics in the field of Educational Data Mining(EDM).Many researchers in EDM have made a lot of efforts on exercise modeling meth-ods and applications.However,there are still some limitations of the existing exercise modeling methods and applications.First,for the research field of exercise modeling,exercises in many disciplines(e.g.Maths and Physics)contain multiple heterogenous data including texts,images and knowledge concepts,but the existing exercise modeling methods simply use the textual content and ignore the semantic information embedded in the heterogenous data,which leads to reducing effectiveness of exercise modeling.Second,for the application of exercise modeling results,the problem of how to apply the modeling results of exercises containing heterogenous data to exercise-based ap-plications remains pretty much open.Therefore,to address the problems mentioned above,combining multimodal learning methods which can handle heterogenous data effectively,this thesis develops a series of researches about the exercise modeling and application.First,for modeling and representing exercises containing heterogenous data,we propose a Multimodal Exercise Representing Model(MERM).Second,for applying the results of modeling exercises consisting of heterogenous data to the find-ing similar exercises task,we propose a Multimodal Attention-based Neural Network(MANN)framework.The major work and contributions can be summarized as follow:1.We propose a Multimodal Exercise Representing Model(MERM).This model can learn a unified semantic representation for exercises from the heterogenous data(i.e.,texts,images and knowledge concepts)in a multimodal learning way.Here,Text-Concept Attention and Text-Image Attention are designed to capture the text-concept and text-image associations in each single exercise,respectively.2.We propose a Multimodal Attention-based Neural Network framework(MANN).This framework can exploit the unified semantic representation of exercises learned by MERM to measure the similar parts in each exercise pair with a Similarity Attention and calculate the similarity score of each exercise pair to deal with the finding similar exercises task.3.Extensive experiments are conducted on a real-world dataset.The experimental results clearly validate the effectiveness of the proposed MERM and MANN and the strong interpretation power of MANN.