| The world we live in is a multimodal environment.We interact with the world with multiple senses.With the development of virtual reality technology,we not only have higher requirements in vision,but also hope to get more realistic audio.A spatial audio contains the spatial information of sound,which is an important way to enhance the user’s sense of reality.In recent years,researchers began to explore how to leverage visual information to convert a mono audio into a spatial audio.However,there are still some problems in the existing methods,such as the generation results need to be im-proved,additional data sets need to be introduced,and the scale of the existing datasets is small.Therefore,we propose a supervised learning method and a direct generation method to solve the above problems.First,we propose a method of generating spatial audio from mono audio using multi-task learning.We train two tasks simultaneously,spatial audio generation task and flipped audio classification task.Our model extracts the information about the spatial position from the input video frames and associates it with the mono audio.Finally we can predict the left and right channels of the spatial audio,and judge whether the left and right audio channels are flipped.Specifically,first,we use Res Net to extract visual features from video frames.Then we use two sub networks based on visual features to perform spatial audio generation and flipping audio classification tasks.Our learning method will optimize the overall loss by weighted summation of the losses of the two tasks.We trained and tested our model on the Fair-Play dataset and Youtube-ASMR dataset.We conducted quantitative and qualitative evaluation.The results show that the proposed method achieves better scores and user preferences than the previous methods,which shows the effectiveness of our method.Next,we propose a direct spatial audio synthesis method based on sound source separation and location.The existing direct synthesis method needs to cut and combine multiple single sound sources into a new multi-source video with spatial audio.For a more direct way,we decompose the synthesis task into two steps: sound source separa-tion and location,spatial audio reconstruction.For a given multi-source video,we first separate the sound source,disassemble it into different single sound sources.Then we calculate their positions in the visual frame.We divide the video frames into different regions along the horizontal direction.Instead of considering the sound source in the physical sense,we regard each region as a sound source.We can also easily calculate the center coordinate of the visual region as the sound source position.According to the sound sources and their positions,Ambisonics and HRIR are used to reconstruct the spatial audio.We can simulate the sound we hear in our left and right ears through our method.We qualitatively show our results and quantitatively compare different param-eter choices.Experimental results show that our method can produce realistic spatial audio,which contains more spatial information than mono audio. |
