Research On Distributed Storage And Indexing Technology Of FITS File Based On Bitmap Index

Most of the data produced in astronomy observation is kept in FITS (Flexible Image Transport System) files, which are used for keeping and exchanging data all over the world. Currently, with the application of numerous large scare and multi-waveband telescopes, the amount of FITS files produced in the astronomy observation is rapidly increasing, which could lead to a problem of how to store and search such amazing numerous FITS files efficiently. In the very old days, these FITS files were not indexed. They were once stored on hard disk drives or other storage devices directly. When a storage device is full, it will be replaced with a new empty one, and the replaced one will be stored in a warehouse, which is used for storing the used storage devices. Those replacement operations are done manually, which lead to a waste of human resource. Additionally, those replaced storage devices are offline of course, so the task of querying files on them will be difficult. So it may be able only when the query condition is a date or a period of time. The complex query conditions like conical search were nearly unable to be processed in those days. The problem led by the increasing number of FITS files produced in the observation once was solved by using traditional database management system (DBMS) such as MySQL and Oracle. But when the amount is increasing faster and faster, the traditional database management system cannot follow the step. The time taken in indexing and querying gets too long to endure.In this paper, we developed a way to store FITS files by using distributed storage systems, and introduced and compared several distributed file systems by experiments. By analyzing the results of the experiments, we concluded that the distributed file system like GlusterFS and Lustre which have a better performance in writing are more suitable for storing the FITS files continuously generated in astronomy observation. And finally we decided to choose GlusterFS as the distributed file system used in FITS file distributed storage system.In order to solve the problem of retrieving FITS files, in this paper, we present a way to accelerate retrieving FITS files by using bitmap index technology, and by applying bitmap index technology to distributed systems, we developed the FITS file distributed indexing system to achieve rapid indexing and query mass FITS files. In this paper, we proved the performance advantage of FastBit bitmap index in solving the problem of mass FITS files indexing by experiments, and proved under the condition of FITS files distributed storage, the FITS file indexing and query system based on FastBit bitmap index is able to take advantage of multi-machine collaboration, which is able to improve the speed of retrieving FITS files significantly.