Detection And Diagnosis Of Program Vulnerability In Java Language

Java language is one of the most important languages in the world. According to the programming community index provided by TIOBE, Java language is one of two most popular languages from2002till now. Adopted by Android operating system, Java language is not only widely used in PC and servers, but also popular in smartphones.With the growing popularity of Java language, the number of Java programmers and Java applications increased rapidly. Take Android as an example, till now, there are already more than500,000applications in the official Android market(Google Play) and the number increased by20,000each month. How to manage and maintain these applications is a challenge for both developers and application markets. Functional flaws, also called bugs, and security flaws bring to large losses each year, so detecting, diagnosing and fixing these bugs and flaws is in urgent need.Detection and diagnosis of program bugs and security flaws have always been the major concern of computer systems and been the focus of both researchers and the industry. Generally, previous research efforts can be categorized into two fields:(1) Researches of concurrency bugs, which investigates solutions in diagnosing and fixing concurrency bugs under multicore architecture. For example, deterministic replay systems record and replay non-deterministic events, so that concurrency bugs can be reproduced deterministically.(2) Detection and diagnosis of flaws in smartphones, which aims to find possible bugs and security flaws in smartphone applications, and identify their trigger conditions. Fox example, by using symbolic execution, we can explore the execution space of smartphone applications, dynamically analyze the program behavior, and report possible bugs/flaws and corresponding program inputs for diagnosis. Besides, by analyzing smartphone applications using static taint analysis, we can detect possible information leakage flaws. Because smartphones commonly use application markets to manage and distribute applications, there are tens of thousands new applications each month. Thus, efficient validation in smartphone markets not only needs fast analysis speed, but also needs a complete verification method. Similarly, concurrency bugs and smartphone application flaws are also the most important issues in Java community.However, state-of-the-art detection and diagnosis tools for Java applications mostly rely on the existing techniques for native languages, and unique features of Java language, such as object orientation, managed runtime and runtime events, make execution of Java applications different from execution of native applications. Thus, existing techniques are not suitable for Java. For example, garbage collection techniques, which are commonly used in Java Virtual Machines, introduce huge amount of data movements in heap. Thus, existing deterministic replay systems need to track the massive data movements, and record corresponding dependencies, which is time-consuming and inefficiency. Besides, Java applications in smartphones are usually user event driven, which is different from the non-interactive applications. Existing techniques mostly focus on the program behaviors introduced by data inputs, so when applied to interactive applications, they cannot properly model the event inputs. Thus, some state-of-the-art techniques, such as symbolic execution, must sacrifice accuracy for performance when applied to smartphone Java applications.Based on a detailed analysis on the requirement of software correctness or security for application users, this dissertation performs heavy experiments on the behavior of Java applications for revealing the difference between Java and native languages and its effect on existing techniques for detection and diagnosis of bugs or security flaws. Then, a systematic solution is proposed to face the limitation on existing approaches.Specifically, the proposed solution is composed of the following key techniques and systems that solve different problems in different fields:1. Practical and efficiently improve performance and reproducibility of deterministic replay systems by introducing object oriented deterministic replay. Design and implement ORDER system to record and replay Java applications based on object oriented deterministic replay technique. Using object as the granularity of shared memory access recording, ORDER reduces contention on accessing shared metadata and improve recording locality, thus its recording overhead is much lower than the existing replay systems for Java. Furthermore, by inlining deterministic replay into Java Virtual Machine, ORDER not only covers concurrency bugs producible in existing replay systems, but also covers bugs caused by dynamically loaded classes and libraries.2. The idea of guided symbolic execution for improving performance and accuracy of symbolic execution. Design and implement SymDroid system to improve symbolic execution of Android applications by guided symbolic execution. SymDroid utilizes the result of static analysis, to limit search space of symbolic execution without sacrificing accuracy. Guided by SymDroid, symbolic execution can finish searching execution space of specific program behaivor in only several hours, which is much better than the existing approaches that either do not support event-driven applications on Android or sacrifice code coverage for performance.3. The LeakMiner system which provides both detection and diagnosis support for information leakage flaws in Android applications. LeakMiner is the first time that static taint analysis and symbolic execution is combined to detect and diagnose information leakages in Android applications. With the help of symbolic verification, LeakMiner can detect most of the leakage paths without producing false positives. Besides, inputs set generated by LeakMiner can help programmers to locate leakage paths at the first time. By leveraging the guided symbolic execution provided by SymDroid, LeakMiner can finish its task in less than one hour for each application.