The Research On Stack Buffer Overflow Detection Of COM Component

In order to solve the problem of software reuse, shorten software developing period, reduce maintenance cost and realize software automatic updating, component software design is put forward and has become an inevitable developmental trend. Microsoft's Component Object Model (COM) is a relatively mature one of software component models, which is widely used in Windows operating system and application programs. Along with the prevalence of COM, more and more security problems are exposed, of which buffer overflow constitutes a high proportion.Buffer overflow has been one of the commonest forms of security holes. It mainly exists in the non type-safe languages, such as C and C++. However, new generation languages, such as Java and C#, do not have this problem. One of the most important reasons is that C allow indirect memory access by pointer without boundary check and provide many unsafe functions which may cause buffer overflow. Meanwhile, there are such functions in Windows operating system. Therefore, if effective means can be adopted to detect buffer overflow, security of component software will be greatly enhanced.Buffer overflow detection based on binary code is thoroughly studied in this thesis because in most cases the source code of COM is unknown. A detection algorithm is proposed by modeling the buffer overflow problem and transform it to ranges comparison of integers. First, establish an unsafe function library of COM and use it to distinguish unsafe function calls in assembly code. Second, according to types of arguments passed to unsafe functions, different buffers are distinguished. Third, obtain ranges of integers based on the size of declared buffers and allocated ones. Last, use buffer overflow detection criterion made by this thesis to detect buffer overflow. COM uses virtual table to locate functions. According to this structural characteristics, this thesis realise accurate positioning of user functions and extract their dependence relations by means of IDC script language. Based on the work mentioned above, a prototype system that can detch stack buffer overflow of COM is realised.