Research On The Detection Of The Inter-boundary Bugs In The System Software

Hierarchy is the foundation of computer system development.However,the interaction part between different layers is usually the weak point of the system,which tends to have security defects.Based on the current system hierarchy,this dissertation proposes the concept of “system inter-boundary”,including the parallel inter-boundary between the threads,the privilege inter-boundary between the kernel and user application,and the hardware inter-boundary between the operating system and the peripheral devices.This dissertation focuses on the security issues in the inter-boundary,including the atomicity-violation bug in the parallel inter-boundary,the double-fetch bug in the privilege inter-boundary,and the hardware double-fetch bug in the hardware inter-boundary.In summary,this dissertation makes the following contribution.(1)Efficient atomicity-violation bug detection based on prediction.The atomicityviolation bug is one of the most common and significant concurrency bug types.The uncertainty introduced by the thread scheduling makes one atomicity-violation bug can have multiple buggy interleavings to trigger it,making it difficult to detect,reproduce,and fix.This dissertation proposes a prediction-based approach to efficiently detect and reproduce atomicity-violation bugs.Based on a formal definition and the candidate interleavings,this approach can expose all of the potential buggy interleavings for the same atomicity-violation bug to assist a thorough fix.A prototype tool named AVPredictor is implemented based on the proposed approach.Experiments prove that AVPredictor can effectively detect all the known atomicity-violation bugs and one unknown bug from the test cases.AVPredictor has an acceptable runtime overhead(18x for the monitor and31 x for the controller),which is lower than related works such as AVIO and Maple.The grouping strategy AVPredictor adopts can avoid up to 66.7% violation-free executions.AVPredictor has an average speedup of 5.8x to Maple.(2)Double-fetch bug detection based on multi-taints parallel tracking.For the double-fetch bugs between the kernel and user space,this dissertation proposes an approach called multi-taints parallel tracking,which is the first approach that introduces taint tracking to the detection of double-fetch bugs.This approach is specific to the doublefetch bug features and maps the time factor of memory access order to the space factor of different tainted variables,avoiding the exacerbation of symbolic execution state space explosion caused by the thread scheduling.A prototype tool named DFTracker is implemented based on the proposed approach,which has the advantages such as high path coverage and low time overhead.Experiments show that DFTracker can effectively find all of the known double-fetch bugs from the test suite with the time overhead of approximately 2x,no false negatives and only a few false positives.When applied to Linux kernel-3.18,DFTracker finds a new double-fetch bug,and the time overhead of analyzing the entire kernel is 9x.(3)Double-fetch bug detection based on static pattern-matching.Since dynamic approaches have limitations such as require specific hardware support and have low code coverage,this dissertation proposes a static pattern-matching approach to detect doublefetch bugs.This approach can cover the entire kernel(including all drivers)within one execution.When applied to the Linux,FreeBSD,and Android kernels,our approach found six unknown double-fetch vulnerabilities.Besides,this dissertation also abstracts three typical scenarios in which double-fetch situations are prone to occur,provides an exploit PoC based on CVE-2016-6516,and proposes strategies to prevent double-fetch bugs.So far,all of the identified vulnerabilities have been confirmed and fixed by maintainers.Our approach has been adopted by the Coccinelle team and is currently being integrated into the Linux kernel patch vetting.(4)A study of the hardware double-fetch bug in the I/O memory.Operating systems communicate with peripheral devices by reading from and writing to the device mapped I/O memory.Due to the lack of effective validation of the attached hardware,compromised hardware could flip the data between two reads of the same I/O memory address,which disrupts the operating system.This dissertation presents the first dedicated study of the double fetch problem in the I/O memory and names it as the “hardware double fetch”.Based on a static pattern-matching approach,we identified 361 hardware double-fetch situations from Linux.The results are categorized according to the I/O memory feature and each category is analyzed using case studies to discuss the possibility of causing bugs.Finally,four are confirmed as double-fetch vulnerabilities and fixed by the maintainers.This work provides a new perspective to the double fetch problem by increasing the scope to include peripheral devices.