Design And Implementation Of Some Modules Of Finical Big Data Platform

With the development of information technology into the big data era, the big data in financial industry is constantly for development. China’s big data development is in its infancy, many units begin to build the big data platform. A domestic financial research institution hope to build a big data platform to support its research work. Currently, the institution’s data source includes:internal publications, commercial databases, internal papers and domestic and foreign public industry data, the big data platform will be constructed based on the data. The platform’s goal is to build a multi-scale, multi-resolution, multi-species, multi-user basic economic data structure in finical field, deeply develop and use the finical information resources, construct a national authority, the only and general public finical information platform and a consultation platform of finical information, promote the integration, sharing and utilization of the data resource of the finical industry, serve the national financial sector information planning, construction and management services, provide finical services for the government, enterprises, experts and the public.In this paper, based on the above, firstly, introduce the background of the big data platform. Secondly, introduce the key technology used in the platform, such as network spider, webpages parse, Chinese word segmentation, and the data display technology. Thirdly, introduce the requirement analysis guided by the idea of software engineering. Fourthly, complete the general design and introduce the idea of parts of the platform which the author participate inindependent design. Fifthly, introduce the idea of the detail design and complete the journal paper database, gather the network data and two modular of the application platform. Lastly, complete the functional test and non-functional test of the system. According to the characteristics of journal data, the paper introduce the design and realization of the data acquisition, the data pretreatment, data index and search in detail. Use solr to index the papers of journal, and provide the function of basic search, senior search, online reading and the download of paper. In the part of the handle of internet financial products data, introduce the acquisition strategy, store strategy and display strategy of finical products data in detail. Use the network spider technology to gather finical products data from the websites, use the jsoup technology to parse webpages and extract useful information from webpages, use the way that the combination of manual and automatic to gather the updated finical products data. And, use the solr search engine technology to index the data, complete the display of a part of the statistical results of finical products data. Introduce the acquisition strategy, store strategy of the national statistical data, to the data from the website of state data in detail.At present, the finical big data platform is in the development, there is a period of time to launch the platform. The following work is to build a distributed search engine and improve the data mining algorithm, and so on.