Research On The Malware Signature Generation And Cloud-based Malware Detection

With the development of Internet and cloud computing technology,malware presents lots of features,such as huge quantity with high-speed growth,sophisticated polymorphic technology and extensive distribution of samples,and it has become one of the key factors that have serious harm to information security and privacy.Moreover,with the deepening of cloud computing technology applications,cloud-based security protection provides lightweight and efficient malware detection,but also brings the risk of privacy data leakage because of the security issues in the cloud environment.Hence,it is one of the hot issues in cloud security research to study how to achieve high efficiency malware detection and meet the data privacy preservation requirements during the detection.In order to solve the above-mentioned malware-related security and privacy issues,this thesis,which is based on the hashing malware signature,aims at designing the protection system of malware signature generation and malware detection based on cloud computing platform and providing efficiency and reliable security service.The main contributions include the following four aspects:(1)proposing automatic clustering and signature generation for malware based on the network flow.Aiming at the problems of large generation cost and low correlation between the generation results in the existing signature generation techniques,this thesis designs and implements AutoMal,a system for automatically extracting signatures from large-scale malware.Firstly,the system proposes to represent the network flows by using feature hashing,which can dramatically reduce the high-dimensional signature spaces that are general in malware analysis.Then,we design a clustering and median filtering method to classify the malware vectors into different types.Finally,it introduces the signature generation algorithm based on Bayesian method.The system can extract both the byte signature and the hash signature of malware from its network flow with low false positive and zero false negative.(2)proposing cloud-based malware detection based on hashing signature.Aiming at the problems of low accuracy of hashing signature detection and large cost of character signature detection in the existing malware detection techniques,this thesis designs and implements HiScan,the multi-level malware detection mechanism based on the hashing signatures.Firstly,the signatures and file contents are formally initialized in order to generate hashing signature vector and proceed initial filtering.Secondly,by adopting multi-level hashing filters on the partially matched suspicious content,false positives during the initial filtering are further reduced.Finally,the pattern matching algorithm is used to confirm the suspect segments after multi-level filtering,so as to ensure the correctness of the detection results.(3)proposing anti-eavesdropping mechanism of cloud security detection.Aiming at solving the privacy data leakage during the communication and optimizing the interaction consumption,this thesis designs and implements CloudEyes,an antieavesdropping mechanism of cloud security detection.For the cloud server,CloudEyes presents suspicious bucket cross-filtering,a novel signature detection mechanism based on the reversible sketch structure,which provides retrospective and accurate orientations of malicious signature fragments.For the client,CloudEyes implements a lightweight scanning agent which utilizes the digest of signature fragments to dramatically reduce the range of accurate matching.Furthermore,by transmitting sketch coordinates and the modular hashing,CloudEyes guarantees both the data privacy and low-cost communications.Finally,we evaluate the performance of CloudEyes by utilizing both the campus suspicious traffic and normal files.(4)proposing cloud-based privacy-preserving security detectionAiming at solving the problem of achieving practical malware detection under the anti-attack semi-honest model,this thesis designs and implements PriMal,a cloud-based anti-malware system which achieves usable detection performance and protects the data privacy of both the cloud server and the client.In PriMal,a newly designed private malware signature set intersection(PMSSI)protocol is involved to enable both the cloud server and client to achieve malware confirmation without revealing the data privacy in semi-honest model.Moreover,we propose the relevant signature engine to reduce the detection range and overhead.The experimental results show that PriMal offers a practical approach to achieve both usable malware detection and strong data privacy preservation.