| After years of demonstration and research,the International Astronomical cooperation project,the Square Kilometre Array(SKA)project,officially began construction in 2019.The large telescope project will bring unprecedented big data,and researchers from all over the world will carry out targeted research in the fields of materials,machinery,networks,electronics,energy and so on.In any field,research on the project is at the forefront.There will be thousands of telescopes and millions of antennas in SKA,in the first stage,the amount of data generated per second will reach 160 TB,it is necessary to build a data centers in the region outside the country where the telescopes been built,these reginal centers will provide data analysis and data storage services for regional scientists.Chinese astronomers are actively promoting the formation of the SKA Asia Pacific Regional center.Based on this background,this paper studies the storage and processing of large data for the SKA regional center,to provide practical solutions for key technologies of regional centers,and the research results have a good prospect of application.In view of how to create a regional center to meet the needs of the project,this paper proposes a plan to build an astronomical cloud platform or a cloud data center.We have carried on the concrete research to the data distributed read-write method and the data distributed processing separately,and have implemented,tested and analyzed the proposed scheme.The main work of this paper includes:1)Design the SKA cloud platform architecture: For the computing scale of SKA,the combination of high performance computing and cloud computing is the trend of development.From the perspective of cloud computing,this paper proposes to build an astronomical cloud platform,and gives the specific architecture design.2)Research on distributed data literacy in SKA reginal center: Compared to the growing astronomical data,the development of astronomical data processing software is relatively slow.Take the current general CASA software as an example,the data read and write operations are still serial in its core code,which apparently unable to meet a large number of data read and write requests.In this paper,a method of distributed data read and write is pro posed,combined with the distributed file system,by rewriting the data table storage manager,which is specially used for data storage and management.3)Research on distributed astronomical data processing in SKA reginal center: The processing and analysis of radio astronomical data is very complicated.Taking the most time-consuming mapping part as an example,the distributed processing under cloud platform is studied in this paper.4)Build a cloud platform experimental environment: A experimental cloud platform environment where using the actual observation data for testing and analysis is built by using Open Stack.In the testing process,two processing flows and three kinds of data splitting methods are presented.Finally,the test results are presented.After testing and analysis,this paper can draw the following conclusions:1)In the case of large amounts of data,ompared with the traditional methods,the distributed read and write method proposed in this paper can significantly improve the performance of data literacy.2)The method of distributing astronomical data in the cloud platform is feasible.The efficiency of the whole data processing can be effectively improved by choosing the appropriate granularity of data decomposition. |
PDF Full Text Request