Research On The Auxiliary Automated Method For Mobile GUI Design

Graphical User Interface(GUI)is now widely used in all kinds of computer software,mobile applications,and online websites.GUI provides a visual bridge between a software application and users through which they can interact with each other.A well-designed GUI is beautiful,exquisite,concise,and easy to use,which is the key to the success of an app.However,GUI design is still full of challenges even for experienced GUI designers.It is not only very time-consuming but also needs continuous innovation and creativity to catch up with the constantly updated popular elements;moreover,the rapid development of GUI design requirements also increases the workload of designers and developers.Besides,due to the shortage of professional GUI designers in practice,the GUI design work in many enterprises is forced to be handed over to software developers with limited professional skills and design experience,which undoubtedly makes the GUI design and development process more difficult.GUI designers and developers usually look for the designed GUI resources from the Internet as a reference in the initial stage of GUI design.At the same time,people have more and more demand for various automatic auxiliary tools and methods related to GUI,such as automatic search or template generation,because such tools and methods can help them complete their personalized design work.However,at present,there is little research on GUI design.The information mining for GUI design style,GUI structure,and other characteristics is insufficient,which can not associate the user needs with the results they want.Based on the above contents,this article analyzes the information related to the GUI data in mobile applications,including GUI metadata,GUI design style,and GUI structure characteristics.Then,combined with the feature extraction and expression methods,using deep learning model,and reinforcement learning algorithm to design and realize the automation methods of GUI design.Finally,this article realizes a variety of auxiliary application tools for GUI designers and software developers to accelerate the GUI design and development process.The main research contents of this article include:(1)Feature extraction and expression of GUI design style:GUI design style is an important part of mobile application design and development,and also affects the appearance and user experience of the final product,but there are rare researches and methods about it.Firstly,this article puts forward the hypothesis that the homologous GUIs have similar design style and the non homologous GUIs doesn't,which is verified by experiments,and builds the GUI Pair dataset.Secondly,GUI2 Vec is proposed.This method uses the Siamese network based on dual channel CNN to model the design style of GUI data.Through the two-stage training of unsupervised learning and binary classification learning,the model has the ability to learn the features of GUI design style.GUI2 Vec is used to transform GUI design into semantic feature vectors,realizing the construction of GUI design style feature space.Two auxiliary automatic tools are developed:GUI design style search and GUI design style continuity detection.The experiments show that GUI2 Vec has better performance in GUI design style feature extraction and expression than the baselines.The two application tools are proved to be practical by user evaluation.Thirdly,GUIVit is proposed by adjusting the model and optimizing the training process.This method introduces transformer module and multi-task training mechanism.Rico Topic dataset is built for training.The experiments prove GUIVit 's feature learning and expression ability and practical value.Finally,this article studies the influence of natural pictures contained in the GUIs when they are used as the input of the network.(2)GUI design and generation method based on generative adversarial network:GUI automatic generation method is an important means to assist GUI designers to complete their design work,which provides diversified reference samples for designers and developers.Because the traditional GAN methods based on pixel level is difficult to generate GUI images with high definition and regular edges,this article presents GUIGAN,an automatic generation method of GUI design based on confrontation training mode.Firstly,the GUI is regarded as a whole composed of a series of local components.A GUI is decomposed into several subtrees as the smallest component of the new GUI design based on the GUI metadata and structural rules.The Subtree dataset is built.GUIGAN's generator can generate a sequence composed of GUI subtree through the adversarial training.The new GUI design can be reorganized with these subtrees.This article integrates GUI design style information and structure sequence information to jointly optimize the generator parameters during training process,so that GUIGAN can learn to generate new GUI design samples with coordinated style,reasonable structure and diversification.The experiments show that GUIGAN can use the existing app and GUI data to generate new GUI design samples with higher quality and richer diversity compared with the baselines.The human evaluation proves the effectiveness and practical value of the proposed method.(3)GUI wireframe generation based on structural feature learning:In order to improve the structural quality of the generated GUI and make the model generate the corresponding GUI design according to the given target,GUIWFG is proposed in this article.Firstly,the deep learning network model is used to extract the features,and the strategy gradient algorithm is used to update the parameters.The input data of the model is transformed from the GUI screenshot image into a wireframe that can represent the GUI structure.Secondly,the local structure position deviation and the target GUI structure learning object are introduced to optimize the model,so that the it can generate a new GUI design in line with the actual GUI structure according to the characteristics of the target object.Rico Topic dataset built in this paper is used in the experiment.The results show that GUIWFG can generate a new GUI design with higher structural quality according to the given target GUI object,but still maintain the overall quality and sample diversity of GUI under the condition of ensuring reasonable structural layout.The user evaluation experiment also proves the potential practical function of GUIWFG.Finally,the style transfer method commonly used in natural images is introduced to improve the overall style compatibility and expand the style diversity of the generated GUI.To sum up,facing the GUI design and development process,this article studies the relevant information features such as GUI design style and GUI structure in GUI data by establishing a deep learning model and takes this as a support to realize auxiliary automation tools with practical value,such as GUI search,GUI style continuity detection,GUI design generation,GUI structure wireframe generation,etc,It aims to help GUI design developers provide more reference objects and improve their GUI design efficiency.At the same time,it also hopes to attempt the automation process of GUI design and development.