Research And Implementation Of Smart Contract Of Crawler Based On IPFS And Blockchain

With the rapid development of computer information technology,crawler technology plays an important role in the outbreak of massive data.At present,crawler technology has been very mature.There are many functional crawler frameworks.The distributed cluster mode will give full play to the performance of crawlers.However,these distributed technology have high complexity,expensive hardware equipment and high investment cost.In view of the above problems,this thesis combines the rapid development of current blockchain technology,designs a crawler scheme based on the existing crawler framework,which can reduce the investment of hardware resources and simplify the deployment.The main research content of this thesis includes the research and design of the crawler data storage module in the scheme by using the IPFS(Inter Planetary File System)technology,as well as the preparation of the smart contract which plays an important role in the scheme.The main research work and achievements are as follows:(1)The theory of blockchain technology,underlying encryption algorithm,consensus algorithm,blockchain risk and smart contract are analyzed,and the principle of previous crawler framework is described.It provides a theoretical basis for the overall design of the project.(2)Combined with IPFS technology,the crawler data storage module is researched and designed.The hardware pressure required by the cluster is allocated to the Internet,and each IPFS node is allowed to contribute its own hardware resources,so as to avoid the distributed cluster mode and reduce a large amount of capital investment in hardware.(3)According to the working condition of IPFS node and the whole scheme design,the appropriate Ethereum smart contract is compiled and the smart contract reward mechanism module is designed.Smart contract can play a driving role,which can motivate the IPFS nodes to grab information,store information,upload information,and logically form a closed loop for the whole research and design.In summary,the research design of this article is based on IPFS technology,blockchain smart contract technology,etc.to transform the existing mature crawler framework.It has stable security,more data sources,and less hardware investment.The practical application of blockchain technology provides a reference.