On Foreign Trade Data Acquisition And Analysis

For the purpose of responding to the national project “One Belt One Road I”,this paper explores the application strategy of foreign trade activities in big data and distributed computing environment,and stretches the chain of foreign trade business process in seeking the opportunity of improving free trade.The first part introduces the eco-environments of collecting and mining the big data in foreign trade,which includes open source distributed computing platform—Hadoop,the infrastructure of transparent distributed system with the core of HDFS and MapReduce(Hive?Hbase?Pig?Sqoop ?Mahout ?Spark ?Shark ?Storm)and open source big data mining platform,TipDM-HB and etc.In the second part,it tells the function and principle of web crawler technology in foreign trade data collection.The following part,by using Classification prediction,clustering analysis,association rules,separatelyestablishes optimization model of trade activities,value classification model of foreign trade customers and foreign trade product related recommendation model for the purpose of solving the massive foreign trade data processing.The forth part provides an empirical case of constructing automatic recommended model of customer favorite products by using the collaborative filtering algorithm.The last part provides feedback evaluation and improvement measures for the structure and quality control subsystem of foreign trade data acquisition and analysis systemin order to realize system model combination optimization and system smooth running.This paper is a discussion on foreign trade data acquisition and analysis of the framework of the program and the macro model,which is finding out the solution to the foreign trade problems based on the theoretical support of big data acquisition and mining theory system and the process of software engineering.