Design And Implementation Of Aurora Image Experiment System Based On HBase

With the rapid development of polar research technology and computer technologies, aurora image data obtained by Earth's north and south poles' station is increasing dramatically. The increasing speed of aurora image data has exceeded one million frames per year, about 550 GB. In addition, with the continuous development of the polar camera technology, the amount of aurora image data will obviously increase in the future. How to manage it efficiently becomes a primary problem which scientists explore the aurora phenomenon of natural because traditional ways are expensive and difficult to extend.Hence, we need a scalable and parallel processing model of aurora image data process. HBase and Map Reduce meet the needs naturally. In this paper, we designed an aurora image management and experiment system based on HBase. We propose a method to store massive aurora image data into HBase, and process it using Map Reduce. Based on the study of Hadoop and HBase, we designed a B/S structure system which based on HBase and intends to manage and make experiment on aurora image data. The system runs on hadoop cluster, it uses two HBase tables to store the aurora image data as well as the information which aurora image data generates after running some image processing algorithms. Well, the HAurora Img Table stores the aurora images' byte stream information and the HAurora Data Table stores the aurora image metadata and the results of image processing algorithm.The distributed computing layer of this system uses Map Reduce framework, make full use of the interface of HBase and Mapreduce to imply the algorithms of image processing. In this paper, we implement three image processing algorithms. The first one is the aurora image preprocessing algorithm, this algorithm converts the raw image into bmp format and extract the image metadata. In the process of this algorithm, we have subtracted the dark current, cut the image, stretched the original aurora image's gray scale, rotated the image to make the North Pole upward. The second one is keogram algorithm, this algorithm extract the intensity data along the geomagnetic latitudinal direction of the magnetic meridian and store it into the HAurora Data Table as a result. The third algorithm is LBP algorithm, this algorithm extract features of the image. We put the aurora images into 3 row 6 column, a total of 18 small pieces and use LBP method to extract each piece's feature value, then put the result into the HAurora Data Table.The API layer of the system provides access to the aurora data which stored in Hadoop and HBase. The servlet layer is located above the API layer. The agent software which deployed on the master node of Hadoop cluster can communicate with the servlet layer, the agent software can send execution command to the hadoop cluster.The application layer encapsulates the function of the system, mainly including user registration, user login, data import, running mapreduce experiment, retrieving the result of experiment and demonstration, system help.The system experiment result showed at the end of this thesis. Experiment results show that the system can run on the cluster correctly, besides, illustrate that the speeds of aurora image data import and data process with Map Reduce increase obviously as the cluster of HBase growing.