Inter-network IP Hosts Traffic Behavior Analysis And Their Relationship Discovery

Internet is a network that consists of millions of networks of local to global scope. It carries an extensive range of information resources and services. The systems and networks that operate in cyberspace have vulnerabilities that present significant risks to both individual organizations and national security. Internet threats is one of the most serious economic and security challenges facing nations. Security researchers proposed different strategies for reducing the impact of Internet threats and improving the resilience to cyber-attacks. A key factor is the accurate and timely detection of attacks. The role of the defenders consists of complex cognitive tasks.Many kinds and different names of security, monitoring, and analysis tools have been used to detect the network penetration and analyze the network performance, such as Antivirus, firewalls, log audit tools, Host-based and Network-based Intrusion Detection Systems IDS, Low and High interaction based honeypots, general purpose and special purpose honeypots, network flow analysis tools, etc. With all of these different sources of security tools, it is becoming more and more difficult for network security engineers to be aware of the huge amount of data produced by these different tools, at the same time it has been proved that depending on one kind of these tools is not enough to protect the networks from being exploited, and to detect previously known threats in addition to zero day exploits. Network security situation awareness NSSA provides a high level security view based upon the continuous monitoring and security alert events. It is the ability to effectively determine an overall computer network status based on relationships between security events in multiple dimensions. This dissertation proposes a contribution to provide background information about the networks environment to setup IP information database to support NSSA. This kind of IP information will be useful in the prediction of the future situation of the network security. We tackled the problem of IP hosts profiling and clustering aiming at identifying dominant and persistent hosts’ behaviors to setup hosts’ profiles and identify groups with similar behaviors. This will enrich the IP characteristics database to accumulate other researches in this SA perception and comprehension levels.IP profiling is done based on traffic patterns of the most significant active observed IP addresses, we present an algorithm to extract most significant IP nodes to be analyzed instead of analyzing the complete list of millions of IP nodes that exist in the trace (Data Reduction). We discuss the features or host behavior communication patterns to be utilized in hosts’characterization to setup profiles. Fifteen traffic patterns related to the IP address traffic patterns are extracted or calculated to be used later as features for machine learning clustering. We analyze IP nodes traffic behavior on relatively long periods of traces, which helped to extract a more stable host’s behavior. While previous studies focus only on host behavior for relatively short periods, we extract host’s behavior patterns over a period of one hour which needs big data analysis to provide results in a reasonable time.IP Relationship is studied based on the social relationship of the managed domain network hosts with the outside IP network. The key idea of this methodology is to split the entire IP address space into Internal (inside the managed domain) and External (outside) ones. The clustering strategy is to group inside IP addresses that communicate with common outside IP addresses, the similarity measure of two inside IP addresses is the unique number of the common outside IP addresses. We propose a novel approach with an approximation algorithm to discover communities on a large scale in the managed domain based on the bipartite networks and one mode projection and the basis of graph partitioning of the similarity graph. Bipartite networks were built using NetFlow datasets collected from a boundary router in an actual environment, and then a one-mode projection has been applied to build a social relationship similarity graph of the inside IP addresses. A new innovative community detection algorithm is used to detect communities of similar behavior. We experimentally validate our approach in terms of IP networking by applying deep flow inspection (DFI) and deep packet inspection (DPI) on related traffic to prove that hosts with the same cluster tend to have some dominant network behavior. We demonstrated the practical benefits of exploring social behavior similarity of IP hosts in understanding application usage, users’ behavior, detecting malicious users, and users of prohibited applications.