Research And Construction Of Securities Database

In recent years,as China's stock market has become increasingly ma-ture,more and more people begin to participate in stock investment.Com-pared with organization,individual is still at a disadvantage due to lack of resources.So in order to help individuals participate in stock investment better,this thesis studies and builds a securities database,including stock data acquisition,processing,storage,as well as the construction of the data platform.The first problem is the source of stock data,in this thesis,a small part of the data is from excel format data manually downloaded from existing stock analysis software and from json format data got from free third-party interfaces.Most of the stock data is from web pages,and web crawl tech-nology is involved.Compared with traditional data fetching scheduling algorithm,this thesis considered multi-level feedback queue scheduling algorithm of operation system to crawl data and the problem of prioritiza-tion is involved.In a multithread scenario,this thesis applied consistent hashing algorithm to the distribution policy of URLs,and the problem that all the URLs's hash map results would change when a thread is added or removed is resolved.At the same time,this thesis did the research on web repeated question and used the BloomFilter algorithm to deal with it based on Bitmap algorithm.Then this thesis discussed stock data's processing and storage,includ-ing the definition of data quality,guarantee of data quality and construc-tion of database tables,etc.Finally,in order to help users check stock data better,this paper realized a stock data platform.The whole data platform based on the Tornado+MySQL+Python technology.In order to solve the design problem of human-computer interaction,the front end of the system uses Bootstrap,JQuery as templates,and the back-end uses the Tornado as web framework and web server.