Assembly Process Recognition Based On Deep Learning

As the scale and complexity of industrial manufacturing increases,there are instances where unregulated shop floor operations lead to reduced productivity and even induce safety accidents.Therefore,it is vital to improve the productivity of manufacturing companies by strengthening the supervision and regulation of shop floor workers behavior.The traditional production management model relies on manual operation,which is inefficient and time-consuming.To address these issues,this paper uses deep learning-based vision technology to identify assembly processes in the assembly of components in the workshop,enabling intelligent and digital management of the production workshop.A series of studies are carried out to identify the problems in the assembly process steps.The main work and results are summarized as follows.(1)A motion information-guided shop floor tool detection algorithm is proposed to address the interference of changes in front and back information during shop floor operations on the detection of work tools.Firstly,in order to improve the localization ability and detection accuracy of the target,the foreground image with motion spatial characteristics is obtained by segmenting the hand motion region in the foreground based on the frame difference method,which is combined with the RGB image of the assembly process to form the dual-channel input of the target detection network.Then,the spatial perception module is designed to achieve spatial feature fusion of the dual-channel input and obtain global spatial information.Finally,the feature enhancement module is used to fuse the global spatial information and deep semantic information to enhance the feature response at salient locations.Experimental results show that the algorithm proposed in this paper effectively reduces the false detection rate and can quickly and accurately detect the tools used by shop floor personnel during operation.(2)To address the problem of complex backgrounds and noise information unrelated to assembly operations in videos that affect the process recognition effect,a shop floor work process recognition algorithm based on spatio-temporal features is proposed.The algorithm is based on key point detection to obtain hand key point information of the assembly operation,and fuses the workshop operation tool information to form a deep learning network with multiple information inputs to reduce the interference of irrelevant information in the background.The correlation between different parameters is modelled using one-dimensional convolution and attention mechanisms to improve the extraction of key features.A combination of TCN and BiLSTM networks captures the timing information in the before and after frames of job screens and models the timing features in both directions to achieve shop floor process recognition.The experimental results show that the deep learning algorithm constructed in this paper has a higher accuracy rate compared with traditional deep learning algorithms such as CNN and LSTM.(3)In order to meet the demand for rapid judgement of workshop work process steps,a workshop work process recognition algorithm is proposed to improve the EfficientNetV2 image classification network,which firstly fuses the movement information of workers and constructs a two-stream convolutional network to extract work features in response to the complex environmental changes in workshop operations.Then,the channel excitation module is embedded to enhance the characterization of key information.The feature fusion module is designed based on Ghost convolution to improve the efficiency of feature transfer in the deep network.Finally,the Mish function is introduced to reduce the redundant computation in non-linear activation.Experimental results show that the improved EfficientNetV2 network outperforms image classification algorithms such as MobileNetV2 for the shop floor job process recognition problem.(4)A web-based shop floor work process recognition system was designed and implemented.The system was embedded on Nvidia Xavier AI development board and deployed on the shop floor production line,with an interactive interface presented through a web page.The ultimate goal is to achieve intelligent management of shop floor operations.This paper presents assembly process recognition algorithm based on the deep learning algorithm for the intelligent detection and management of the assembly process on the shop floor.Combining the information of tools and hand key points in the assembly images of the workshop,a temporal data processing model and an image classification model for process recognition are constructed respectively.A shop floor assembly operation system incorporating the deep learning model is designed.The algorithms proposed and the system designed in this paper have implications for the realization of intelligent management on the production floor.