Design And Development Of Remote Assistance System Software Based On Real-time Interaction Of Audio And Video

Remote Assistance System is a real-time video interaction device that enables remote professionals to guide field personnel.At this stage,the professional and technical personnel in some industries of China are in short supply.It is impossible for professionals to handle incidents on site at all times.This has caused delays in event processing or mishandling of non-technical personnel.Regarding the issue above,this thesis designed a remote assistance system with real-time video and audio interaction.The mobile terminal of the system uses the wearable device as a carrier to liberate the hands of the operator.In addition to traditional video and audio interactions,This system also supports the helper to add drawing information to the video screen through the client software,and simultaneously display it to the assisted person.Reduce the operational errors of non-professionals through multi-angle assistance.The main contents of this article are as follows:(1)The overall design of the remote assistance system.The system requirements and the overall framework are determined,and the hardware system design and development platform were built.(2)Design and implementation of mobile terminal acquisition and display software.Using STM32 microcontroller as the intermediate medium for data transmission.Design a serial communication protocol,complete the data interaction function between the external device such as IMX214 image sensor and VL53L0 X distance sensor and the Hisi 3519 control chip.Design multi-process tasks on the Hisi 3519 platform,realize data transmission through inter-process communication,and complete the processing of serial data.(3)Design and implementation of video live broadcast function.Using H.264 video coding and AAC audio coding technology improves the quality of live audio and video,reduces network bandwidth pressure.Resampling is performed to reduce the noise of the audio.Audio and video data is synchronized using a timestamp synchronization technique.The data frame is packaged based on the RTMP streaming media transmission protocol,and the network transmission is performed frame by frame.The SRS is built on the streaming media server to forward the live data directly to the client,which reduces the network latency while satisfying high concurrency.(4)Client software design based on QT platform.For the GUI required by the system,the window class and its method provided by QT are used,and the software GUI design is combined with the signal and slot mechanism.According to the cushion technology,Created buffer and the multi-thread synchronization is completed by the critical section resource and the read buffer,In this way,the smoothness of the audio and video decoding process is improved.