Research On Deep Learning Based Graphical User Interface Component Detection Method

Graphical User Interface(GUI)allows users to interact directly with applications through GUI components such as buttons,images,and text,etc.Many software engineering tasks are based on how to detect them from the GUI interface.For example,when developing GUI for Internet applications,automatic interface code generation deals with the increased development costs because of the difference in domain background knowledge between designers and developers by converting prototype drawings into relevant code.The accuracy of GUI component detection,as a key technology to achieve the task,directly affects the effectiveness of the final code generation.Besides,it plays an important role in GUI automation testing related tasks and more UI inversion related tasks nowadays and its speed and model occupation size have become important evaluation metrics for GUI component detection methods as the application scenarios become more extensive.Computer vision techniques such as Canny is used in the traditional GUI detection methods,with timeconsuming detection process and hard-classifying components.With the development of deep learning,target detection algorithms based on it have been applied to solve the problem,achieving better results.However,without consideration of the characteristics of GUI component detection task,there are still problems of low localization and classification accuracy.This thesis studies and improves the deep learning-based GUI component detection methods for GUI component characteristics,using the novel YOLO series algorithm YOLOX as the basic framework.The main work is as follows.Two improvement methods are proposed for the diversity of GUI component scales: the positive and negative sample matching method Sim OTA of YOLOX is improved.Besides,the method using a diamond as the basic shape to determine the sample candidates is proposed.Instead of Sim OTA,which uses circle,the diamond shape is more consistent with the aspect ratio,achieving a more balanced ratio of the two samples.To promote further fusion of shallow detail features and deep semantic features,a new multiscale feature fusion network structure is proposed,which adds a bottom-up pyramid structure to the original PAFPN.It improves the detection capability of the model with the introduction of a small number of parameters and commutations.Three improvement methods are proposed for GUI components with many small target samples and high similarity: a new attention mechanism Shuffle CBAM based on Shuffle Attention and CBAM is proposed and used in the backbone network to improve small target perception without introducing too many parameters and commutations.To improve the detection performance of small targets,a new Io U loss function is proposed by increasing the penalty on small targets based on the target.An adaptive activation function Auto Si LU is proposed to improve the overall detection effect of the model by introducing learnable parameters being optimized for difficult samples using automatically selected gradients.In this thesis,the dataset is based on and preprocessed the large-scale mobile application UI layout dataset CLAY.The effectiveness of each improved method is verified by ablation experiments with five improved methods being incorporated into YOLOX as the model proposed.The experimental results show that the m AP of the method in this thesis reaches56.61%,which is better than other target detection algorithms.With a 5.84% increase compared to YOLOX,it significantly improves the accuracy and introduces a small number of parameters and computation,effectively applied to current GUI component detection tasks.