The Research And Implementation Of The APT Intrusion Detection Based On Spark

With the continuous improvement of network technology,Advanced persistent threat(APT)become a problem difficult to solve in work security field now because its organizational and premeditated.APT is not adopted advanced technical means to attack network,but it uses the traditional attack technology which has already been for a long time.Such as phishing emails,DDOS,Trojan and so on.But it has a variety of means to attack and hackers spent a lot of time making strategy.Based on these,they can reach the goal of attack.So the tactics of attack are diversified.APT attack mainly has two stages to complete attack:First of all,it detects the target device or network and then it completes attack by using a lot of critical attacks.So it is very import to detect it early.Due to its concealing and camouflage,it is very hard to realize by using traditional security technology.This paper presents a method to detect APT attack.At the first,analyse the network data by using Snort and improve the speed of rule-matching.And the,build the Spark framework.After that,improve the speed of Apriorialgorithm for detecting APT attack and solve the algorithm's problem which need spend a long time on I/O operation.At the last,by using the Spark framework search for association rules from alarm logs and submit the result to the network technicians.By using the DARPA dataset,the improved Apriori algorithm is tested and use the test results to compare the speed and precision with traditional algorithm.In the same time,analyse the test data and validate whether this method can find APT attack.In summary,The main work and innovation points of this paper include:(1)Research and implement an intrusion detection method for APT attacks,which detailedly includes deploying Snort to extract alarm logs,mining association rules between logs by using Spark distributed framework.(2)Improve Snort matching rules to make make it more efficient and Reduce the running time in the matching process.(3)Improve Apriori algorithm to make it more efficient by reducing the process of I/O operation based on the matrix.Apply Apriori matrix algorithm to the Spark distributed platform and do not reduce the accuracy of the algorithm.(4)Improve Apriori for application scenarios to make it more suitable for APT attack mining(5)Before searching for the new frequent items,prune the matrix to make the speed of scanning matrix more faster.Based on that,the speed of Apriori algorithm is more faster too.(6)Confirms the feasibility of this method by an example.And compared the new Apriori algorithm with the traditional algorithm to verify whether the speed of the algorithm is improved.