Research And Development Of Platform For Internet Video Data Storage And Analysis Based On Hadoop

With the development of digital media technology,the number of video on the Internet is increasing rapidly.In order to explore the value and the law of the Internet videos,their users behavior information is stored and analyzed.With the rapid accumulation of the information about videos,servers are facing a great challenge on the storage capacity and computing power.What's more,diverse data sources make it more difficult.Traditional solutions for the problems above are based on adding more hardwares,but the improvement is limited and is expensive.This paper proposes a distributed storage and computing solution,which is a low-cost and efficient platform based on the Hadoop,using cheap computers to build cluster.To facilitate the analysis of fine-grained features of Internet videos users behavior data,this platform builds unstructured database.To cope with the diversity of data sources,the platform improves data source extraction methods of MapReduce by packaging functions and interfaces.In order to serve the upper application systems better,a task distribution module has also been developed to monitor the status of the upper application systems,allocate tasks based on real-time status and improve platform performance.The design and development of the underlying storage and computing platform and tasks distribution system have been completed.What's more,by storing real data,the feasibility of the underlying storage system has been verified.By running MapReduce examples,the effectiveness of the computing platform has been demonstrated.Finally through the simulation on the laboratory platform,the reliability of the tasks distribution system has also been verified.