Design And Implementation Of Video Processing Platform

Nowadays,under the catalysis of Internet technology,new media tends to develop in video.The video occupies an indispensable position in entertainment and learning because it carries rich information.The increased demand for video has prompted more and more video sites to sprout.While guaranteeing commercial income,it also brought about problems such as competition between websites and inability to share resources.Large-scale websites attract users to expand daily output by purchasing copyrights,while small-scale websites face the risk of being eliminated when they cannot enrich video resources.With the rise of the era of machine learning,research in the field of video requires a lot of video data,and simply relying on manual data collection has been unable to meet the demand.In addition,downloading videos from major websites is extremely inconvenient.Even downloaded videos can only be played on specific clients,and there are many video processing processes and loose functions,making it difficult to form a system.Considering the above problems,this paper designs and implements a video processing platform.The platform solves the problems of batch video acquisition,video download,video deduplication,video merge and user interaction.First,the platform uses Scrapy-redis distributed crawler technology,design ip proxy pool to solve the anti-crawl problem,and Scrapy's own selector to obtain and parse video information;second,the platform analyzes the obtained video for each website during the video download process Download the real address and use multi-thread to download the video to improve the download rate,and support the interrupted resume of the interrupted download of the video.The video download address acquisition and video download adopt the producer / consumer model to achieve decoupling between systems.Then,the algorithm based on hard subtitles and perceptual hashing is used to deduplicate the video,and the concat protocal protocol is used to merge the segmented video.Finally,by using the Vue.js + Django framework to develop a web platform,the video information obtained by the crawler is displayed to the user,and at the same time,the user is provided with a window for video processing operations such as video downloading,video merging,and video deduplication.For convenience,this platform is currently running normally in the enterprise.