Research On Algorithm And System Based On Multimodal Information Fusion To Optimize Mobile Reading Experience

With the rapid development of smart-phone market and popularization of the mobile Internet,more and more people spend a lot of time using smart-phones nowadays.However,smart-phone reading can easily make readers uncomfortable(feeling visual fatigue,nausea,dizziness,etc.)in motion environment(on vehicles,walking,etc.).It is important to improve the user experience in such scenario.The key to improve smart-phone reading in motion environment is to acquire the displacement of a smart-phone.According to the displacement,the position of the content displayed on a smart-phone can be adjusted to keep it stable on screen.In this way,people can avoid frequent eye focus and visual fatigue when reading in motion environment.We try to calculate the screen adjustment offset using time series prediction methods such as Linear Regression and Recurrent Neural Network based on multi-modal data collected from an Android phone.Experiments show that time series prediction methods have good adjustment performance against regular movement,but they are incompetent for irregular movement.In order to improve the generalization ability of the system,an adaptive algorithm based on multi-modal data fusion is introduced to adjust the screen.Specifically,it integrates the acceleration signal,which is collected with high frequency,with the visual data from image sensor,which is precise but with low frequency.The effectiveness of the algorithm is validated through a series of indoor and outdoor experiments.In this thesis,a meaningful exploration is made on how to improve the reading experience of smart-phones in motion environment.The details of the background,models and validation methods are introduced and a demo application is also developed for effective screen adjustment.