A Network Security Analysis System Based On Apache Pig

With the growing scale of computer networks,the approaches of network attacks against network vulnerabilities and security flaws are becoming more and more complex,and the classic security defenses are no longer able to protect the network effectively.Dynamic detection of intrusions and attacks has become an important means to protect the network,network packets capturing and analysis are the key for dynamic approaches.Decoding and analyzing the captured packets can monitor network running status effectively and locate network faults accurately,and it is the basic step of further appropriate measures to deal with aggressive behaviors.Although existing techniques can solve the questions of capturing and storing network packets,there is no solution that achieves both good performance in analysis and diversified capabilities in protection.For example,the network packets capture tool called Wireshark,which is mainly used for LAN protocol analysis,cannot analyze the network status;the intrusion detection system Snort only provides network anomaly intrusion detection.These tools are only for a particular study field,and the provided valuable information are relatively scarce,and they cannot meet the needs of network security analysis.In this paper,we propose a analysis and detection technique for network data,and develop a network security analysis system based on Apache Pig.The system can achieve various functions of security analysis and intrusion detection by integrating packet decoding,intrusion detection,fingerprint identification,protocol analysis and others to analyze network packets comprehensively.Finally,we perform functional tests and validation for the designed system in Pig local execution mode.The experimental results show that our system can not only realize the basic functionality of analyzing packets like Wireshark,such as counting packets,traffic statistics,and protocol analysis,but also perform deeper security analysis like intrusion detection and passive OS detection by integrating open source tools Snort and POf.More importantly,the system is based on Apache Pig platform,it has a very good scalability and aims at Full Packet Capture.It can achieve a comparatively integrated analysis frame,providing the analysts a complete context related to attacks.