Complex Background Constrained Excavator Action Recognition

Excavator is an indispensable engineering machinery in earthwork engineering,and accurate evaluation of its working efficiency is crucial for controlling construction costs.The identification of different actions such as excavation,swinging arm,and dumping during the excavation process is an important assessment basis for estimating performance indicators such as operating cycle time,downtime,and direct working time of the excavator.However,traditional excavator monitoring often relies on the subjective observation of on-site excavator operators,which consumes a lot of manpower and resources and is inefficient.With the development of deep learning,it is a feasible solution to assist in monitoring excavators by constructing a deep learning-based excavator action recognition model.The construction of an excavator machinery action recognition model faces the following difficulties: 1)the actual working background of excavator machinery is complex,with interference from a large number of unrelated targets such as materials,equipment,and construction personnel;2)the excavator moves slowly and has a lot of jitter during excavation and dumping,and the model is difficult to capture key differences between actions.In response to these problems,this thesis first constructed a video-image dataset of excavator operations from actual construction sites and carried out the following study work based on this dataset:(1)To address the problems of cluttered backgrounds and multiple interferences in the target area in the actual working scenario of the excavator,this thesis proposes a video area extraction-based excavator action recognition algorithm.First,a multi-frame fusion-based video area extraction algorithm is proposed,which adaptively generates the smallest predicted box based on multi-frame detection results to accurately extract the working area of the excavator and exclude the background interference of the construction site.Then,a threedimensional convolutional network is constructed to extract the spatiotemporal features of the target area excavator action and recognize the excavator action.Finally,by comparing and verifying on the excavator machinery dataset constructed in this thesis,the proposed algorithm achieves an accuracy rate of 95.28% and an F1 score of 95.14%.(2)To address the problem that the excavator moves slowly and has a lot of jitter during work,making it difficult for the model to capture key differences between actions,this thesis proposes an excavator action recognition algorithm based on the target reconstruction task.Based on the region extraction,a reconstruction branch network is used to reconstruct the extracted video frames,and in the reconstruction process,key feature differences caused by jitter and slow movement are learned to reduce the model’s misclassification rate for excavation and dumping action categories and improve the effectiveness of excavator action recognition.Finally,the proposed algorithm achieves an accuracy rate of 97.90% and an F1 score of 97.73% on the excavator machinery dataset constructed in this thesis.(3)A excavator intelligent detection and recognition system based on video-images is developed,which is supported by the GUI graphics tool in Py Qt.The excavator action recognition network model proposed in this thesis is applied to the system.The system has four major functions: single-frame image detection,real-time video monitoring,video region awareness,and body action recognition.The real-time results are displayed through the visualization monitoring window,achieving assisted monitoring tasks for excavator operations.