Designand Implementation Of The Retail Oriented Big Data Downloading And Analysis

Along with the development of the information technology and information security,information and data resource play a pivotal role in the current economic filed.Whether the enterprise could timely retrieve information and data recourse has become the key to success in business activities.As a result,Big Data,as a newly emerging filed,has received more and more attention from major enterprises.For the numerous generated business data and sales data during the enterprise operation,how to analyze them correctly and quickly as well as how to extract the valuable information from them could grasp the market trends and reduce the operating costs.Therefore,it is really significant to grasp the information accurately and timely while using the Big Data.This thesis mainly describes the design and implementation,of the retail-oriented,crawler-based big data download and the data analysis/computing based on OLAP(On-line Analytic Processing)database.Crawler is a technique against the data download,it utilize crawlers to rapidly and completely download data and import it into OLAP database,execute multi-dimensional expression to generate the result set required by customer in a OLAP database,which is also called a multi-dimensional database.The result set will be written,transferred and formatted in a excel format file,and ultimately form a report satisfying customer needs.The application mainly uses SQL Server 2008 Analysis Service and Java to build a traditional multi-tier system mode,the mode similarly includes three-tier applications:UI(User Interface),BLL(Business Logic Layer)and DAL(Data Access Layer)which users are able to flexibly select the report type,data type,browse method(via emails or online).Users can also define report content to provides more intelligence and personality to the reports.The big data download and data analysis/computing system implement two major functionalities:Download big data,query and generate reports from a multi-dimensional database.In addition to these two functions,it can also achieve a functionthrough addition and deletion to change search database witha front page,in order to a better control aboutthe logic behind the operation.The front page includes as follows,one is configuration page which sets data download types and download cycle,the other one is users' rights management page.The front page can be designed by traditional JSP technology.this thesis provides the method for testing ofthe system,in particular the using of the black box model in function testing,by simulating a variety of unusual circumstances,abnormal import simulation data,to see whether the operation of the system is normal or not,checking the system and correcting the problems,to ensure that the system can achieve accuracy,stability,and flexibility.In performance testing,using a pressure test method,such as starting multiple threads simultaneously download a variety of data,viewing the maximum pressure value thatthe system can withstand,determining the system to see if it can withstand the maximum pressure value with monitoring CPU status.Finally,based on test results and summary of system implementation,we proposed a new step in the future about the development and expansion of the system in terms of future...