Dynamic Resource Adjustment Of Hadoop-based Big Data Services

With the development of technology. From the traditional Internet is widely usedin recent years and the explosive growth of the mobile Internet and Internet ofthings,the data which dependent on the network data growing, according toInternational Data Corporation (IDC),EMC company’s research shows that in themobile network equipment and video surveillance, the amount of data around theworld has reached487billion GB, and the amount of data in the2007report that only161billion GB. These data include a large number of telephone, mail, photos,SNS,news, and video content. How to effectively use these data to provide users withhigh-quality user experience in scientific research, with a large number of GPS datacollected by the device data collection and research are urgently need to get technicalsupport.In the future, with the further development of the Internet of Things, a largenumber of services is based on the data of the location-based services (LBS), will alsohave a large number of requests services based on LBS or personal preference. Thisalso requires the services in the future is different from now.Google has a wealth ofexperience in data mining,The Hadoop, which is drawing on Google’s GFS conceptused by many companies whom have a large data store, search and processing.Distributed parallel computing, the processing of large data decomposition to eachnode to do parallel computing, and thus more rapid completion of the data processingis applied to the Web logs, transaction flow, data backup has a huge advantage.Thereserach trying to ming information from structured or unstructured data, summed upthe characteristics of the data, and put data stored in Hadoop cloud computing system,application side access services, issued by the application server request, the nearest access to resources. In this process, Hadoop achieve traversal of resources and toadjust the location of the node and the node number of resource distribution toachieve the best application for the hardware and software resources. Provides a newidea for the LBS-based big data services.The main works are:First, Analysis the big data storage research, including the GFS file system and theHadoop framework, technical support for data stored in large data services.Second, Research data characteristics and the related to data mining algorithmsmining unstructured data summarized read structured data analysis, data hotspotweights initialization.The main results are:First, Based on the existing Hadoop framework.to modify Hadoop storage backupalgorithm right through the resource list of values to achieve the resource file storageresources Points weights in accordance with the data in a modified frame.Second, In this paper, on the basis of Hadoop design resource weight initializationalgorithm, when the resource file access increased or import settings via an external hotweight, calculated and distributed file storage node data resources.Three, Achieve weight adjustment in the process of library resource file hotspotschanging the resource file, and pass the value of the rights to redistribute the dataresource file and adjusted to achieve the service resources amplification algorithm andservice resources contraction algorithm.