Research On Efficient Storage Of Small Files In Mobile Ultrasound Detection Based On HDFS

Mobile ultrasound detection in the high-speed rail detection, underwater mapping, power monitoring and other applications, produces massive data, and large numbers of small files. Hadoop distributed file system(HDFS) show high performance, high reliability, high scalability characteristics. We can easily build a cloud storage system for mobile ultrasound detection by using HDFS. HDFS uses a master-slave architecture with metadata information of files stored in Name Node memory. A large number of small files generated by mobile ultrasound detection consume a lot of memory in Name Node, which limits the capacity of the HDFS distributed clustered storage.To solve this problem, this paper designs a small file storage module independent of HDFS for mobile ultrasound detection. In this approach, with using spatio-temporal information of small files in mobile ultrasound detection, small files which have geographical proximity are combined into a single large file. And each small file index information is stored in the large file header. The mapping of small files to a large file mapping is stored in HBase(Hadoop Database). We use cache prefetching mechanism for file mapping, indexing information and part of file data to accelerate the efficiency of access to small files.In the implementation of merge strategy that according to small files geographically similar, we aggregate the files with similar geographical location into the same cluster by using clustering algorithm to analyze the small files based on their latitude and longitude attributes. For files of mobile ultrasound detection in clumped geographical distribution, we use grid clustering algorithms combined with hierarchical algorithms for cluster analysis. For files of mobile ultrasound detection in the rail which have the attribute of Rails-like distribution, we design clustering scheme for rail detection files referenced to K-nearest neighbor algorithm.The result of experimental test demonstrates that small file storage module reduces Name Node memory consumption for small files metadata significantly, and accelerates the efficiency of access to small files by using cache prefetching mechanism.