Defending against malicious nodes in closed MANETs through packet authentication and a hybrid trust management system

Wireless links and lack of central administration make MANETs far more susceptible to attacks than conventional networks. MANETs must provide various levels of security guarantees to different applications for their successful deployment and usage. Their security requirements depend greatly on their architecture.;In this dissertation we are focusing on closed MANETs where only designated nodes are supposed to access the network (e.g., in a military or corporate setting). We define "outsider" nodes as those nodes that are not authorized to access the network, and " insider" nodes as those that are allowed to access the network. The objective of this research is to develop mechanisms that protect a closed MANET against malicious behavior from outsider nodes as well as insider nodes through packet authentication and a Hybrid Trust Management System, respectively.;To defend against outsider nodes, we present a new Hop-by-hop, Efficient Authentication Protocol, called HEAP, which is suitable for unicast or multicast applications. HEAP is independent of the routing protocol used and it is based on a modified HMAC algorithm that uses two keys and is very efficient. We compare the performance of HEAP against other algorithms and provide proofs of its security.;To combat against insider attacks, we propose a new hybrid trust management system that is based on Reputation Systems (RS) and Role Based Trust Management (RBTM). We develop a novel Machine Learning based RS, called EMLTrust and delineate its advantages. We compare its performance against other RSs and demonstrate the improvements in performance. We also highlight the challenges associated with using RBTMs in MANETs and offer some solutions. Finally, we propose a hybrid TM system that combines EMLTrust with RBTM, and evaluate it to illustrate its efficacy in thwarting insider attacks.