The Research And Design Of Memory Adaptive System For Hugepage Scene

In big data application scenarios on cloud computing,data intensive tasks bring great memory resource overheads.The memory expansion alone can alleviate the memory resource constraints,but it introduces more hardware address translation overheads.As a result,server vendors provide support for hugepage memory,reducing page table entries to reduce the overhead of address translation on one hand,and reducing the miss rate of TLB to improve performance on the other hand.For hugepage scenes,there are a number of hugepage management systems in Linux.But the hugepage management systems and the base page management system are independent,and the number of hugepage is fixed in booting process.Due to the unpredictability of high-level service workload,hugepage and base page memory cannot be accurately adapted which lead to a certain amount of memory wastes.According to our survey,the work of building a memory adaptive system based on existing hugepage page management hasn't been done by other researchers at present.Based on the original hugepage and base page mechanism of Linux,this paper studies and designs a memory adaptive system for large page scenes.The system can automatically adjust the amount of hugepages based on the memory usage of the hugepages,thus improving memory utilization and saving the cost.In order to complete this system,this paper has implemented the following works:1)According to the hugepage data,designing memory adaptive algorithm:we add interfaces in the initialization,allocation,free,etc.to count the current total pages and free pages.The algorithm predicts future usage based on the historical data of hugepage,and thus determines whether to adjust or not.2)When the number of hugepages is adjusted,designing the migration of page content:the data migration method is designed to ensure the correctness of the program currently using hugepages.3)On considering performance of program and system,the adjustment will be finished by asynchronous threads:this approach reduces the impact on the performance of running programs and avoids system throttle caused by frequent adjustments.4)Deep analysis of the Linux memory management module of Intel x86 32/64 bit and ARM 32/64 bit architecture:design and implementation in multi-platform,and the system can support Intel x86 32/64 bit and ARM 32/64 bit platforms.5)Finally,the system has been tested:benchmark SPEC 2006,STREAM and HUGEMEM-MMAP have been used to verify the function and performance of the system,and we achieve the expected goals.