Design And Realization Of Bidding Information Monitoring System

With the further development of the domestic economy, with theimplementing of "Bidding" and strengthening anti-corruption of the government,bidding market have made a huge development.With the further developmentof the market economy, more come more companies are tender to.selectpartners by a public tender.The number of firms bidding and information publishing platform get moreand more. The public tender information is available in provinces,municipalities and counties, Market is facing an asymmetric supply anddemand. Bidding information publisher hope to capable enterprise take part in.The potential enterprises are also willing to cross geographical segment, tolearn something competitive bidding projects of their own, and timelyparticipating in public bidding. Faced with the vast excess information,companies bidding information concerned is not easy to be selected, whichresulted in supply and demand information asymmetry-the standard newsrelease party can not find enough high-quality bidders, the excellentcompanies that want to participate in the bidding are hard to find good projects.Therefore, market need such a high degree of intelligence monitoring biddinginformation system to resolve the serious problem of information asymmetry.The system can real-time monitor more than2000thousands bidding bidinformation release platform on the network, automatically, intelligentlyobtaining marked the corresponding structured content matching with what thecustomers concerned, allowing users instantly find good bidding projects.The system automatically analyzes and identifes biding news, such as:standard news release region-Region/province/municipality/prefecture-level city, marked the announcement name, marked the file download, standard information snapshots, procurement, tendering agencies,tender content, project budget, issuing time, opening time, sources and so on.Monitoring system through the structured data in real-time structured databasestorage for customers online at any time through the use of various terminals(browse, search, export).The system not only monitors the tender information, but also providesinformation on the tender of the industry, geographical and other statisticalanalysis and statistical analysis as well as provides monitoring and statisticalanalysis services.The system uses a common framework spider algorithm, usingmulti-threaded mode, stable and simple design that can support multi-processsingle-server deployment, or multi-server distributed deployment. Grab thethousands of standard news site data, within half an hour to complete, systemperformance and stability, supporting domestic standard news crawl has beensufficient in the upgraded version of the next phase, we hope to competitiveintelligence and foreign standard information included in the scope ofmonitoring, will spider frame and algorithm upgrades, consider using cloudarchitectures were based on Hadoop cluster deployment spider systems.BMS Spider Spider bidding monitoring subsystem is the use ofevent-driven, multi-threaded spider system pipeline operations, the use of ACEstream, Task and other models and ACE Socket Wrapper Fa ade achievemarked the release of thousands of sentinel sites capture, filter, and otheridentification, indexing and storage capabilities. Data is stored into thestandard information database, for the reception User Platform users to usestandard information platform calls.After starting the system reads from standard news site dictionary seedURL list and push to be crawling marked the URL queue, spiders crawl URL tobe ready after reading this queue to extract URL for DNS resolution anddownloaded by thread after hearing the standard URL to crawl, crawl successfully marked the content page is HTML code, through coding, HTMLcontent parsed, filtered navigation information, advertising, copyright and otherinvalid information, the seeds within the standard news page URL link standardnews link recognition, recognition conform to the rules of the URL into thestandard information a list of libraries, and crawl URL queue length throughstandard information based on URL links to be launching threads, real-timesupplement to be crawling into the URL queue list; marked the URL to crawlsuccessfully, illustrated by3.11after the process was Dom node labeled byfiltering algorithms filter out invalid information, standard information structureddata analysis and extraction, after extraction of structured standard information,enter into the Mysql standard information stored in the content database forcustomers end calls.