Development Of The System For Metrology Quality Management Based On Spark

With the increasing competition on the market of varification and testing,strengthening the metrology quality management system has become an important strategy for metrology and test institutions to upgrate their overall capabilities.At present,the enormous metrology data generated and demand higher storage space.In addition,the statistics data are more frequently utilized in management decisions and occupy a lot of real-time computing resources.In the face of these two trends,the storage space and the poor scalability of the computing capability of the traditional single server system are becoming more and more prominent,which have brought about the demand for the application of large-scale data processing technology.Because Spark is based on memory and RDD technology and has faster parallel computing speed,this paper proposes to use Spark as the computing platform for large scale statistical data.For the function of large scale data statistics query,Spark periodically calculates the query statistics results of various conditional combinations,and then caches.the results are read directly from the cache when the user is querying.On the basis of using relational database to store the structured data of each module,this paper also uses the Mongodb to store all the certificate reports and the original records.It takes the advantages of fast concurrent reading and writing speed,easy expansion of storage capacity and reliable access interface.The combination of Spark framework and Mongodb constitutes a complete framework for large-scale data processing.In addition to meeting the current application requirements,it can also provide support for future metrology data mining and industry trend research.The system test and application show that by combining Spark,mongodb database,relational database,bar code,VPN,SMS and other technologies,the running of the management system is perfectly supported.The system is able to generate statistical data quickly,facilitates management and decision.It achieve the large-scale storage of certificate reports and original records.In addition,the system has also completed the monitor of the instrument circulation process,satisfied the management needs of the external inspection business,and realized the monitor of completion rate and certificate quality.