The Security Mechanism Of Near-infrared Spectroscopy Big Data Under Hadoop Framework

The technology of near-infrared spectroscopy analysis is lossless, fast detection and widely used in the field of processing analysis. The technology of near-infrared spectroscopy can generate a large volume of near-infrared spectroscopy data one time, and single file is small (less than1MB), low of information quantity. To analysis and mining of the massive near-infrared spectroscopy big data, the best approach is to use cloud computing. In the existing cloud computing, Hadoop cloud computing is one of the most popular platform of big data processing. Using Hadoop to process the near-infrared spectroscopy data should consider its security architecture. Hadoop uses its default data block of64MB to store cloud data, but single near-infrared spectroscopy data file is much smaller than the Hadoop data block. Therefore, there are many intact single near-infrared spectroscopy files in a Hadoop data block, which pose a great threat to the safe of near-infrared spectroscopy data.According to the above security problem of near-infrared spectroscopy data existing in the Hadoop cloud, the paper made a deeply study of the Hadoop framework and the characteristics of near-infrared spectroscopy data, then propose a security mechanism of near-infrared spectroscopy big data under Hadoop framework. Aiming at the massive near-infrared spectroscopy big data, a mathematical model of key-band sets is established to distinguish and simplify the key-band data segments and the others among the massive data. Design the key-band sets detection algorithm by two division spectroscopy data, which can detect the key-band sets high efficiently in the massive near-infrared spectroscopy single file. The algorithm also can manually set the detection length for one time and the detection range of the key-band sets in a single file, then encrypt the key-band sets detected, the algorithm turn the data into encrypted and non-encrypted data coexist in a file. After this processing, the data will be compressed and merged by the LZO dictionary compression algorithm which is compatible in Hadoop, and uploaded to the Hadoop cluster.The security mechanism of near-infrared spectroscopy data under Hadoop framework was not reducing the performance of Hadoop cluster, clients process a suitable file management for the security Hadoop to the massive near-infrared spectroscopy data, which ensure the security of Hadoop cloud data. In the tobacco raw materials near-infrared spectroscopic information sharing system shows that the security mechanism design of the paper is feasible.