Research And Application Of Performance Opti Mization Method In Multi-media Distributed System

With the development of the Internet,software applications will generate massive data content every day.At this time,the storage system needs to take into account the large storage capacity,high performance and low cost.The mixed multi-media storage system becomes the only choice for enterprises to reduce costs and improve performance.Hybrid multi-media storage systems are designed to achieve high cost efficiency and provide fast responsiveness for applications.Improving the performance of hybrid multimedia storage systems has become a hot topic for enterprises and scholars.In this paper,Ceph multi-medium storage system is constructed by using dynamic random access memory,solid state disk based on flash memory and mechanical hard disk.The experimental research shows that the read and write bandwidth of the disk increases with the increase of the read and write block size.In order to improve the performance of HDD devices in multiple media,the paper proposes strategies such as erasure code strip merging and writing,block prefetching,and small file filtering and migration.The research shows that the erasure code of Ceph system has poor performance compared with the copy mode because the erasure code needs to store the data in pieces and split the large file into strips and store them on the disk.The paper proposes to merge and store the stripes in the migration process,bind the continuous stripes according to the fixed size,and bind them to the OSD node.On this basis,the paper proposes a prefetch algorithm based on the data side,predicts the user’s behavior through the sequential read detection algorithm,and reads the data to the data side in advance through the block read mode.In order to save the prefetched data blocks,the paper proposes an offset linked list structure to store the discontinuous data blocks.This structure can adapt to multiple sequential reads and insert the cache at the same time,providing the ability to quickly find,insert,delete,and so on.The experimental results show that the prefetch technology has obvious optimization effect.Compared with the non-enabled prefetch technology on HDD devices,the maximum bandwidth capacity is increased from 916mb/s to 2257mb/s,with an increase of 146%.Compared with Ceph’s original prefetch algorithm,the maximum bandwidth of a single file increased by 32.69% from 1701mb/s to 2257mb/s.In addition,the paper also proposes a small file filtering migration strategy.When migrating new data,only large files are considered to reduce the impact of small files on HDD disk performance.In the migration strategy,the file creation time,size,and the remaining capacity of SSD storage pool are comprehensively considered.This method also improves the read bandwidth of large file pairs.In the highly concurrent small file read scenario,the read bandwidth of large files has been increased from376mb/s to 1896mb/s,and the performance has been improved by 406%.The condition of opening SFFMS policy for small file read OPS is 410% higher than that of not opening SFFMS policy.Finally,the online trial run of the multi-media storage system was carried out.In terms of the overall read delay at the data node side,the overall delay was controlled to about 220 us.The latency of data end reading from memory is 20 us,while reading from HDD device reaches tens of milliseconds,even hundreds of milliseconds.In terms of memory usage,when the overall cluster traffic reaches 15gb/s,the maximum prefetch memory of a single machine is about 15 GB.In addition,in the case of sequential reading of large files,the hit rate of the prefetch algorithm reaches about 1%,ranging from 0.3% to 2%.The application shows that the improvement effect is obvious,and the relevant monitoring indicators meet the expectations.