Formal Analysis of Network Protocol Security

As computer networks turn into an indispensable part of technology and entertainment of human life today, security and reliability become the forefront issue of network protocol design and implementation. Any security or reliability flaws in network protocol operation can lead to great loss in private data, business opportunity and reputation. As a result, methods and tools for preventing and detecting such security breach in network protocol design and implementation are urgently needed. Formal methods have proven to be successful in system verification and testing. This thesis is devoted to formally design algorithm and tools to efficiently and effectively investigate security of both protocol specification and implementation.;For protocol specification, we first model a protocol as a state machine. To verify certain security property against the state machine, we need to explore all reachable states in the state machine. However, traditional reachability problem suffers from the classic state space explosion problem. We propose two techniques to tackle this problem: trace inclusion transformation and online minimization. We first transform the original state machine to a simplified machine which is a trace inclusion transformation of the original machine by removing certain system parameters. We apply online minimization to construct a minimal reachable graph of manageable size that is bisimulation equivalent to the reachability graph of the simplified machine. Then we can verify the desired security property against the minimal reachable graph. However, we may introduce false positive and false negative since we have simplified the original machine. In the second phase of analysis, we show that there is no false negative if the simplified machine is a trace inclusion transformation of the original machine. We also design algorithm to eliminate false positive. Then we can conclude if the state machine (the protocol design) is security against the checked security property.;However, even with perfect design, fault can still be introduced during the implementation and integration phase. Due to the black-box nature of protocol implementation and often unavailability of specification, traditional verification techniques do not apply here. In this thesis, we first automatically synthesize a formal model to represent the behavior of the implementation and then use the synthesized model to assist different security flaw detection applications. We study both active and passive synthesize approaches and compare their advantages and disadvantages. We incorporate the formal model to two different security flaw detection applications. In the first application, we build a model based fuzz testing framework in which the formal model is used to guide test input selection and served as test coverage criteria. Our framework significantly improves existing black-box protocol security testing techniques. In the second application, we examine the synthesized model to determine if there is any malicious function hidden inside a protocol implementation and determine the behavior of the hidden function.;We show the proposed approaches are effective with extensive case studies for various scenarios and applications. Nevertheless, the techniques and framework proposed in this thesis is generic and could be easily adapted for different problem domains and applications.