Design And Implementation Of A Video Conference Oriented Audio And Video Mixing System

Video conference system has always been an important application in multi-person real-time communication scenarios.In recent years,the COVID-19 has taken a rapid development of video conference applications.With the application and development of 5G network,network speed will no longer be the bottleneck limiting the user experience.It’s the performance of the device itself determine whether users can get a better user experience on the device and try more functions.In this case,the video conference based on MCU architecture,which means complex compute will be completed in the central server,has more advantages than that based on the SFU architecture,which needs higher device hardware requirements.The traditional MCU architecture has the following problems:First of all,affected by network fluctuations and other conditions,the audio and video stream received by the MCU from the user may lose their original time domain correlation,and the mixing operation will introduce new time domain correlation.As a result,the final video and audio are not synchronized;Secondly,the traditional audio mixing algorithm and simple audio mixing structure will inevitably introduce additional noise,resulting in the superposition of background noise and the attenuation of human voice,which will reduce the audio quality in the conference;Thirdly,when users use different audio coding formats to access the conference,the mixing process may cause some error,introducing noise and delay due to the limitations of the audio mixing algorithm and the requirements of the audio codec technology.Based on this background,this paper designs and implements an synchronization strategy between audio and video streams suitable for MCU server after mixing process according to the mixing characteristics of MCU server and the existing synchronization methods.So that there will be no user perceivable audio and video synchronization during the whole conference process.Besides,this paper proposes an optimization algorithm based on the adaptive weighted mixing algorithm.Compared with the original algorithm,it has more stable output volume and smoother at the overflow point for mixing.At the same time,combined with the design of the new mixing structure,it reduces the superposition of background noise and human voice attenuation caused by mixing.On this basis,this paper analyzes the principle of mixing error,and designs the internal structure of the mixer,which can be well compatible with the mixing of audio with different coding formats.Through the comparison and verification during experiments,the work of this paper has solved several problems of traditional MCU architecture.Finally,this paper designs and implements a video conference oriented audio and video mixing system integrating designs above.After completing the functional and non-functional test of the system,we proves that this work is also feasible in practical application scenarios.