Design And Implementation Of A High Efficiency Short Video Processing Framework

For the past few years,the popularity of short video industry is rising rapidly,and the short video traffic of major platforms is growing explodedly.As one of the leaders in short video traffic,Byte Dance attaches great importance to the technologies related to the short video.In face of the increasing number of daily active users and the increasingly complicated processing scenarios,the requirement for short video processing efficiency is getting higher and higher.And there are quite a few disadvantages in original solutions,such as redundant codec,frequent model compilation and single processing mode.In order to solve the problems above,a high efficiency short video processing framework is designed and implemented in this thesis.With a more efficient solution,the framework has become a unified underlying framework of several video departments and is serving for a variety of different video scenarios.In terms of process,the framework makes use of a directed acyclic graph structure to record the processing flow of short video in the memory in advance.By separating the construction and execution of processing period,I/O interaction can be saved and repeated encoding can be avoided as well,which can significantly improve the processing efficiency of short video.In terms of architecture,unification of underlying framework has a good convergence effect on the business architecture of company,which can not only avoid inconsistent versions of dependencies among different systems,but also avoid frequent compilation by encapsulating models into common modules.In terms of processing mode,the framework provides a variety of processing modes,which brings higher processing efficiency for special scenarios such as batch processing.The framework can be divided into service layer,builder layer,engine layer and process layer.Service layer is mainly responsible for the maintenance of batch tasks.Builder layer's duty is the construction and optimization of the stream processing framework.And the engine layer is responsible for module coordination and scheduling during the execution period.Last but not least,the processing layer focuses on particular processing modules.The thesis demonstrates the design and implementation of this high efficiency short video processing framework according to the software development process.First of all,the thesis expounds the necessity of the framework through the project background and introduces the technologies related to the framework.Then demand analysis is showed in detail according to the use case diagram.And then,with the help of "4 + 1" view model,the thesis describes the overall architecture of this framework.After this,the detailed design of the four layers of this framework is demonstrated,and the implementation codes for key points are showed as well.At last,high efficiency of this framework is demonstrated by the test case.At present,the framework has been put into use in the company.Due to the efficiency improvement it brings,the convenience of unified management,and the flexibility of module development,it is being used by more and more video processing departments.