Research On Key Technologies For Automatic GUI Code Generation

GUI code generation is a software engineering process that automates the generation of GUI code from GUI images(GUI designs and mockups).As one of the automatic code generation solutions,search-based methods have the advantage of being interpretable and not requiring large amounts of labelled data.These methods involve several sub-tasks such as data preparation,GUI widgets detection and classification,code synthesis and testing.Specifically,data preparation entails gathering and cleaning GUI images from multiple apps to build a dataset for detection and classification tasks.GUI widgets detection entails identifying the bounding boxes of atomic widgets of a user interface from GUI images.The widget classification is essentially an image classification task that entails classifying the identified widgets into their appropriate domain-specific types(e.g.,Button,Progress Bar).Furthermore,code synthesis entails creating a correct hierarchy of classified widgets and detected layouts and translating them into GUI code.GUI testing is important to ensure the quality of the generated GUI code and the complete functionality.Despite the recent development in the GUI code generation techniques,several problems hinder the performance of the existing techniques.These problems include(i)lack of review analysis for GUI code generation and GUI testing literature,(ii)poor dataset,and(iii)much manual effort required to design a widgets classification model.Specifically,for review analysis,several GUI code generation and GUI testing techniques have been proposed to generate GUI code from GUI images and assess the quality of GUI-based apps in the last few years.However,few efforts were made to characterize and understand the GUI code generation and GUI testing domains,particularly for mobile apps.This problem may be an obstacle to new researchers and more research in the area.Also,for the dataset,poor samples,noise and class imbalance are the problems of the existing widgets dataset.Similarly,for widget classification,existing works rely on handcrafted model architecture to classify the widgets.However,designing hand-crafted architecture involves much manual effort and is often prone to error and time-consuming.In order to solve these problems,this thesis proposes a whole set of research schemes.The scheme includes(i)presenting a systematic literature analysis to gain insight into the GUI code generation and GUI testing for mobile apps,(ii)constructing a standard benchmark dataset of labelled widgets,and(iii)proposing an automatic approach to design model architecture for widgets classification.Specifically,for solving review problems,first,a literature analysis is performed to give a review of the existing GUI code generation techniques and an overview of several tasks in the GUI code generation process.Second,a systematic literature analysis is performed again for the GUI testing studies published in the last decade.In this systematic review,an extensive search of relevant literature over seven popular digital libraries was conducted.Next,a careful selection of 86 primary studies published in the last decade from 3,420 candidate studies.Finally,an analysis of these primary studies from bibliometric and qualitative perspectives.For the bibliometric analysis,the popular research topics and their relationships are analyzed.Similarly,the authors’ communities are studied.Also,for the qualitative analysis,the main objective,approaches,and evaluation metrics employed in these primary studies are explicitly analyzed.Moreover,a standard widgets dataset is built with a clean sample and balance class distribution to solve the dataset problem.The standard dataset is built by inspecting and cleaning the large existing GUI widgets dataset.Also,an automated approach,Neural Architecture Search-based Widgets classification(NASW),is proposed to solve the widgets classification problem.NASW uses machine learning techniques to generate a model architecture based on a small prepared sample of labelled standard widgets dataset using Neural Architecture Search(NAS).The generated architecture is trained on a large widget dataset to generate the NASWm classification model.The model NASWm achieves a 5% improvement on multiple metrics.This approach greatly saves the timeconsuming and error-prone task of manually designing an optimal model architecture for GUI widgets classification and improves its performance.In summary,numerous significant findings are presented that might serve as guidance for new researchers and future studies in the domain of GUI testing for mobile apps.Besides,a standard dataset of widgets is built to mitigate the existing dataset problems.Also,for solving widgets classification problems,the thesis proposes an automated approach to generate model architecture automatically that addresses the manual design challenges.