Design And Implementation Of Investors' Trading Behavior Management System In Precious Metal Market Based On Hive Data Warehouse

With the rapid growth of the market size for precious metal,the volume of accumulated trading records is increasing exponentially.It raises new challenges in the management of investor trading behavior.Thanks to recent advances in big data technology and data mining,we are able to collect,store,analyze data more efficiently,thus creates new opportunities for better trading behavior management.This article focuses on the next generation data warehouse system for precious metal exchange company where the author works in.It discusses the design and implementation of trading behavior management system in the warehouse.The new system is based on Hive and aim to solve practical issues in managing everyday trading behavior.The core of the system is a set of evaluation criteria,consisting of a trading activity score and an anomaly detection algorithm.The trading activity score is derived by three variables selected from practical business evaluation.The activity scoring module is implemented by first retrieving data from Hive system via HQL script using Map Reduce mechanism,then going through Impala for calculation.The anomaly detection algorithm is set up by experimenting various clustering algorithms,and the final algorithm is selected based on a performance metric.The trained algorithm is wrapped in to anomaly detection module.In the production,the trading data to be tested firstly flows through a customized data cleaning module following by the anomaly detection module.Besides,the raw data structure to store the trading data is optimized using stratified design to adapt the characteristics of trading data.A scheduling system is developed to realize the planning of script execution from various platform namely HQL,Impala and Sqoop.The front end visualization is implemented based on web application developed by SSM + Vue.js.The system solves the long-lasting pain point in trading behavior management,thus boosting the efficiency of trading behavior management in precious metal exchange company.It also provides a rapidly expanding framework for the analysis and management of other investors' trading behavior in the future.