Analysis And Application Of Online Video Service Based On Large Scale Network Traffic Data

Online video service has currently become the most important component of the Internet, thanks to the development of network and communication technologies, together with the widespread of large-screen user equipment and dedicated service applications. It attracts large-scale users, generates massive network traffic and occupies a large proportion of network bandwidth. Meanwhile, online video service is constantly showing new features, such as the comprehensive service type and the mobile network access. Hence, an in-depth understanding of modern online video service is of great importance for a number of tasks,such as optimizing the network resource deployment, designing the service function algorithms and adjusting the advertising strategy, in order to obtain the potential benefits and further improve the user experience.Under this background, based on massive network data collected from real environment, using big data processing, data mining and machine learning technologies, this dissertation makes a detailed analysis of online video service from multiple angles and solves some practical problems directly. The main research contents and innovations of this dissertation are as follows.(1) Analyze the process and characteristics of the communications between users and servers in the video delivery phase, and propose a generic detection method for the delivery servers of online video service.Online video service providers use delivery servers to serve their geographically distributed users. These servers continuously transmit large video files to users, thus are of great importance in the network control and optimization tasks. However, video delivery servers usually have no domain names nor fixed IP addresses, and may be added to or removed from the network dynamically. Hence, they are difficult for network operators to accurately identify. To this end, in this dissertation we first propose the detection problem for the delivery servers of online video service. Based on active measurement experiment, we reveal the generic communication process of online video delivery. And based on large-scale network traffic data, we analyze the communication characteristics between users and delivery servers in depth, including the intervals between HTTP transactions, the content types of transferred files, the redirection behaviors of HTTP transactions, and the URIs and headers of HTTP packets. Based on the analysis results, we further define multidimensional features, combine with the efficient machine learning algorithm, and finally propose a generic detection method for the delivery servers of online video service. We further use real-world data to verify the detection method. The experimental results show that the proposed method has excellent performance: the precision is close to 100%, and the recall is over 85%.(2) Based on more than 17 billion network traffic traces collected from a mobile network in a northeast province in China, analyze the user behavior characteristics of online video service in the emerging mobile network environment from three aspects: data consumption,location movement and service usage.The three analysis aspects are related to the resource consumptions of users in core network, radio access network and video service provider,respectively. In terms of data consumption, we analyze the distributions of user traffic volume and user active time. In addition, we find the existence of heavy users, and further propose a non-parametric detection method. In terms of location movement, we analyze the number and locations of cells that are visited by a user while watching online videos.Based on this, we first propose the notion of mobility pattern for online video service users in the spatial dimension. We further study the trajectories and residence time of each mobility pattern in detail. And in terms of service usage, we analyze the number distribution of videos watched by users, and reveals the characteristics of request time. In addition, we reveal and measure the replay behavior of users. Finally, we analyze the related characteristics of heavy users across analysis aspects,and compare the behavior differences between heavy users and non-heavy users in different aspects.(3) Based on large scale network traffic traces collected from the network operator and long-term video metadata crawled from the video website, for two key types of users, analyze the user preference characteristics of the emerging comprehensive type online video service from three aspects: user activity, video property and user relationship.For the study of user preference, we are the first to conjointly consider the two key types of users (i.e. uploaders and viewers), and perform a comparative analysis work in this dissertation. In terms of user activity, we study the distribution characteristics of users in different time scales, and compare the overall service usage of the two kinds of users. In terms of video property, we analyze the static and dynamic properties of the uploaded videos and the viewed videos, including video category,video duration, video view count and etc. At last, in terms of user relationship, we focus on the follower count for uploaders. We analyze the overall distribution, together with the impact on video view count, of the follower counts. And for the viewers, we built a relationship network based on their video preferences, and reveal the small world characteristic of this network.(4) Based on the monthly view count tracking data of more than 200 thousand newly uploaded videos, characterize the popularity of online videos in depth from both the group and the individual perspectives.For a group of videos, we first analyze the overall distribution of their long-term view counts, and fit it with a Pareto Type 2 distribution.Then, we define different popularity levels based on the long-term view count. Finally, we explore the impacts of video categories and video tags on the long-term video view count. For an individual video, we first analyze the daily increase of view count, and propose the notion of active day. Based on the locations of active days, we further analyze the length of active period for each video. Next, we measure the heterogeneity of video views received in each day, and propose the notion of popularity burst. Based on the number and temporal positions of bursts, we finally define a set of popularity growth patterns, in order to describe different video popularity evolution trends. On this basis, we further study the correspondence between the popularity level and the popularity growth pattern.(5) Based on the analysis of online video popularity, given different predicted objects, propose efficient prediction methods for the future popularity of online videos.Given the huge amount of video content and the high variability of user attention, it is of utmost importance for a number of practical tasks to understand the characteristics of online video popularity and further predict the popularity of individual videos. In this dissertation, we study the predictions of the future popularity levels and future view counts of online videos, respectively. For the level prediction of video popularity,we extract multi-dimensional features from five aspects (video property,uploader property, content topic, text language and historical dynamic),combine with a variety of efficient machine learning classification algorithms, and predict the future popularity level in different scenarios.We show it is possible to predict the popularity at the time of video publication, with an average precision around 74% and recall around 60%.And by introducing the early observation period, the prediction performance can be further boosted to 95% and 91%, respectively. For the numerical prediction of video popularity, we first study the relationship between early video popularity and long-term video popularity. Then, we propose two kinds of prediction methods based on popularity growth pattern and popularity level transfer, respectively.Experimental results show that the proposed approachs can outperform the state-of-the-art baseline methods by over 30% in relative errors.