| Usability testing has been considered to be an irreplaceable practice that tests user interfaces on real users. Traditional approaches, which rely heavily on human effort and experience, face challenges with the rapid development of mobile applications (apps, mobile web applications, etc.). Due to the competitive market, mobile applications usually have to meet very tight time-to-market requirements, which push traditional development modes to a breaking point. Teams are required to produce value (adequate downloads, positive user reviews) in a very short time (even in weeks). Moreover, people use mobile devices in more contexts, such as walking, in vehicles, sitting still, or connected to different networks. Traditional usability testing can demand a large amount of human effort, expert experience, time and money. Thus, it is difficult to extend the scope of the testing to a larger scale or keep the same pace with frequent upgrades of the software.;Similar to usability experts observing interesting or unexpected behaviors in usability testing, the purpose of this thesis work is to explore methods to detect atypical (we will also use the terms "abnormal" and "unexpected" alternatively in the thesis) behaviors among user interactions automatically. Such abnormal behaviors are usually caused by usability issues leading to user frustration. By identifying factors that can indicate user frustration, it potentially enables us to diagnose use sequences/interactions automatically.;There have been many studies of user behavior that predict a user's mental state working with desktop platforms, most of which focus on webpage browsing behavior such as searching activities. Little work has been done on mobile platforms or specifically with mobile apps. By modeling Dwell Time (the time a user spends on a page), researchers have achieved some success in predicting user satisfaction for information retrieval activities such as searching. As far as we know, comprehensive work in time analysis on the mobile platform was rare, not to mention work on mobile app-using behaviors.;In this thesis, we perform a detailed analysis of mobile app user behaviors from the perspective of user interactions with interfaces and identify factors that could contribute to constructing prediction models of user frustration. We collected data from real-world usability testing of 6 mobile apps under different categories on the Android platform. Choice reaction time is the time the user needs to perform the subsequent interaction with the interface. The results of our choice reaction time analysis lead to novel insights into mobile app user behaviors. We identify both time-relevant features and time-irrelevant features to depict mobile app user frustration. These features are proved to have modest-to-strong correlation with user frustration levels.;We propose methods for mobile app user frustration prediction at two levels. At the sequence level, we elaborate on how to use a regression model to predict users' frustration level for tasks. At the interaction level, we explore both an unsupervised approach and supervised approach to capture abnormal interactions. We discuss in detail unique issues of applying these machine learning techniques to the domain of usability testing. We evaluate our prediction result not only against key metrics in machine learning, but also evaluate the compliance with real user behaviors. Both suggest that our proposed method can achieve satisfactory accuracy, which potentially enables large-scale analysis of real-world usage data and opens a new chapter to explore. |
