Design And Implementation Of Data Service System Oriented To Consumer Finance

The traditional consumer finance industry often encounters a situation where the amount of data is insufficient and the model accuracy is low during the risk assessment process.Direct purchase of external data not only consumes a lot of costs,but also causes difficulties in data retrieval and analysis due to data heterogeneity.This article provides a complete set of data service solutions for the consumer finance field,including data acquisition modules,data fusion modules,data retrieval modules,and data analysis modules for visual deployment and monitoring.The data collection module is expanded based on the Scrapy framework,and with the Redis database,distributed data collection is realized on three server nodes.The Scrapyd management tool is used as middleware and ScrapyWeb is used as the operation interface to realize the interface deployment and log monitoring of distributed data collection tasks.The data fusion module combines the traditional database master-slave synchronization principle and the field mapping method in the Elasticsearch data writing process,and integrates some of the internal data stored in the file system and the data stored in the MYSQL and MongoDB obtained by the data collection module,and Write to Elasticsearch synchronously.The data retrieval module combines Elasticsearch's inverted index principle to realize the classification and retrieval of data written to Elasticsearch,and automatically sorts and displays the returned results according to Elasticsearch's scoring mechanism.In addition to basic data retrieval functions,common auxiliary functions such as auto-complete and popular search recommendations are also introduced to optimize the user experience.The data analysis process is an extension of the data services of this system.Based on the Beijing second-hand housing transaction data obtained by the data collection module,the surrounding environment data of the second-hand housing is supplemented by calling the Baidu map interface.Normalize the data through the data preprocessing link,and complete the classification with positive and negative labels.The experimental part uses a logistic regression model,and the final model accuracy rate is 80.66%after tuning.The innovations of this paper mainly focus on the following three points:1.Develop distributed data collection tools based on Scrapy,and can realize real-time monitoring of the data collection process.2.Learn from the principles of traditional database master-slave mode and call Elasticsearch components to achieve data fusion and break down data barriers.3.Facing the field of consumer finance,providing a big data processing platform with automatic collection,data fusion,data retrieval,and data analysis has certain application value.This article analyzes the problem from the background of the subject and proposes the research content and significance of the subject.On the basis of fully investigating the relevant technologies,a detailed demand analysis of the system was carried out.Then divide the five functional modules according to the system function points,elaborate the outline design and specific implementation process of each module in detail.Finally,the usability test of the system is introduced in two aspects,functional test and performance test,and the test results are good.The system has basically achieved the initial demand setting of the system,with stable operation and good performance,which can be promoted and used.