Research And Implementation Of Spatial Audio Generation Algorithm Based On Panoramic Video Content

With the development of virtual reality technology,360-degree panoramic video,as a new form of video,provides a more immersive experience to users due to its feature of full-angle surround view.Generally,in order to obtain a deeper sense of immersion on the basis of panoramic video,typically a matching spatial audio is added to the video.However,it is still a technical difficulty to generate realistic spatial audio to integrate into the video scenes.The built-in audio recording equipment of common panoramic video cameras are usually not able to achieve satisfactory results.On the other hand,manually edit the sound of a panoramic video or utilizing professional spatial audio recording equipment could cost a large amount human and financial resources.For this reason,this work has designed and implemented a set of algorithms for computationally generating spatial audio based on the content of the panoramic video.This method enables users to automatically generate matching spatial audio for their 360-degree video more efficiently.In this paper,firstly we decompose the elements required to generate spatial audio into three main parts:sound objects,room reverberation,and ambient sound.Based on the video content,these elements are produced through multi-target detection and tracking,room parameter regression,and scene classification modules,respectively.And an ambient sounds database is established to facilitate this process.The spatial audio generation system is established using the method proposed in this paper.On this basis we designed and conducted a user study to verify the effectiveness of this algorithm.Through analyzing statistical data,we verified the effectiveness of this algorithm,and we studied the relative importance of each element in the algorithm.When the spatial audio generation algorithm uses the DeepSORT multi-target detection and tracking module,in order to resolve its inapplicability in panoramic video,this paper proposes the S-Deep SORT algorithm.On the basis of the original DeepSORT algorithm,S-DeepSORT introduces Sphere-SSD based on SphereNet as a target detector.Moreover,by employing coordinate conversion and data enhancement methods,the algorithm can be better performed in a panoramic video environment.Through comparison experiments with existing algorithms' performance data,it is shown that S-DeepSORT can achieve better results in multi-target detection and tracking in a panoramic video.Finally,this paper designs and implements a system platform for viewing and editing the spatial audio of panoramic video based on Unity game engine.