Defeating HTTP Fingerprinting

HTTP fingerprint identification has become an emerging topic of applicationsecurity, the HTTP server and HTTP application security has become an important part ofnetwork security from the standpoint of network management, to maintain the web serverand to monitor and track HTTP fingerprint, the HTTP fingerprint identification makes theautomation of information systems and security policy change, has been set based on thespecial audit policy platform or web server, and security testing tools can use HTTPfingerprint to reduce the test configuration.This research describes HTTP fingerprint recognition technology, and the ways toidentify HTTP fingerprint and how to defeat it by using some available systems.Fingerprint is defined as fingertip impression on any surface or ink impression at yourfingertips dipped in ink on paper, left; that can be used to identify things: such as thecharacteristics traces or features to expose the origin of things, that the evidence of theobject or material characteristics.Fingerprint recognition technology is now known of the network security technology.OS fingerprinting is a common task in any network assessment, but now they have a lot ofoperating system fingerprint recognition technology. OS fingerprinting Why? That isbecause each operating system TCP/IP protocol, there is a slight difference, the morepopular is the use of TCP/IP stack operating system identification tools such as the Nmap;there is the tool using ICMP identification of the operating system, on behalf of The suchXprobe.HTTP fingerprint recognition principle is generally the same to record a differentservers to identify small differences in the implementation of the HTTP protocol. HTTPfingerprint identification is more complex than TCP/IP stack fingerprint identification onthe grounds that the custom HTTP server configuration file, adding plug-ins or componentsthat make changes to HTTP response information has become very easy to changedifficulties, which makes identification; custom TCP/IP stack behavior need to modify thecore layer, so it is easy to identify. REVIEW TO HTTP PROTOCOLThe Hypertext Transfer Protocol (HTTP) is an application-level protocol fordistributed, collaborative, hypermedia information systems. HTTP has been in use by theWorld-Wide Web global information initiative since1990. The first version of HTTP,referred to as HTTP/0.9, was a simple protocol for raw data transfer across the Internet.HTTP/1.0, as defined by RFC1945[6], improved the protocol by allowing messages to bein the format of MIME-like messages, containing metainformation about the datatransferred and modifiers on the request/response semantics. However, HTTP/1.0does notsufficiently take into consideration the effects of hierarchical proxies, caching, the need forpersistent connections, and virtual hosts. In addition, the proliferation ofincompletely-implemented applications calling themselves "HTTP/1.0" has necessitated aprotocol version change in order for two communicating applications to determine eachother’s true capabilities.This specification defines the protocol referred to as "HTTP/1.1". This protocolincludes more stringent requirements than HTTP/1.0in order to ensure reliableimplementation of its features.Web Clients and ServersWeb content lives on web servers. Web servers speak the HTTP protocol, so they areoften called HTTP servers. These HTTP servers store the Internet’s data and provide thedata when it is requested by HTTP clients. The clients send HTTP requests to servers, andservers return the requested data in HTTP responses, as sketched in the Figure below.Together, HTTP clients and HTTP servers make up the basic components of the WorldWide Web.You probably use HTTP clients every day. The most common client is a web browser, such as Microsoft Internet Explorer or Netscape Navigator. Web browsersrequest HTTP objects from servers and display the objects on your screen.When you browse to a page, such as http://www.webserver.com/index.html,”yourbrowser sends an HTTP request to the server www.webserver.com (see above figure). Theserver tries to find the desired object (in this case,“/index.html”) and, if successful, sendsthe object to the client in an HTTP response, along with the type of the object, the length ofthe object, and other information.Methods:HTTP supports several different request commands, called HTTP methods. EveryHTTP request message has a method. The method tells the server what action toperform (fetch a web page, run a gateway program, delete a file, etc.).The below able lists five common HTTP methods, Note that some methods have abody in the request message, and other methods have bodiless requests. Requests:The request format for HTTP is quite simple, The first line specifies an object,together with the name of an object to apply the method to. The most commonly usedmethod is "GET", which ask the server to send a copy of the object to the client. HTTPrequest message contains the method followed by the URI and ending by HTTP/version.The client can also send a series of optional headers; these headers are in RFC-822format. The most common headers are “Host”, this header is mandatory in case of requestwith HTTP/1.1,"Accept", which tells the server which object types the client can handle,and "User-Agent", which gives the implementation name of the client. The Request syntax:<METHOD><URI>"HTTP/1.0"<crlf>{<Header>:<Value><crlf>}*<crlf>ExampleGET/index.html HTTP/1.0Accept: text/plainAccept: text/htmlUser-Agent: Yet Another User AgentResponsesThe response format is also quite simple. Responses start with a status line indicatingwhich version of HTTP the server is running, together with a result code and an optionalmessage.This is followed by a series of optional object headers; the most important of these are"Content-Type", which describes the type of the object being returned, and"Content-Length", which indicates the length. The headers are terminated by an empty line.The Response syntax:"HTTP/1.0"<result-code>[<message>]<crlf>{<Header>:<Value><crlf>}*<crlf>ExampleHTTP/1.0200OKDate: Mon,15Aug201111:14:20GMTServer: Apache/2.2Content-type: text/htmlContent-Length:233 Status CodesEvery HTTP response message comes back with a status code. The status code is athree-digit numeric code that tells the client if the request succeeded, or if other actions arerequired. A few common status codes are shown in the below table.HTTP also sends an explanatory textual “reason phrase” with each numeric status code.The textual phrase is included only for descriptive purposes; the numeric code is used forall processing.Now let us take a quick look at the structure of HTTP request and response messages.HTTP messages are simple, line-oriented sequences of characters. Because they are plaintext, not binary, they are easy for humans to read and write. Figure below shows the HTTPmessages for a simple transaction. HTTP messages sent from web clients to web servers are called request messages.Messages from servers to clients are called response messages. There are no otherkinds of HTTP messages. The formats of HTTP request and response messages arevery similar. HTTP messages consist of three parts:Start line:The first line of the message is the start line, indicating what to do for a requestor what happened for a response.Header fields:Zero or more header fields follow the start line. Each header field consists of aname and a value, separated by a colon (:) for easy parsing. The headers endwith a blank line.Body:After the blank line is an optional message body containing any kind of data.Request bodies carry data to the web server; response bodies carry data back tothe client. Unlike the start lines and headers, which are textual and structured,the body can contain arbitrary binary data (e.g., images, videos, audio tracks,software applications). Of course, the body can also contain text.Here in this thesis, I present techniques to defeat HTTP fingerprint in differentphases, as it is known and stated [1] that the process of a Web server servicing a HTTPrequest can be divided into three phases: receiving the request, dealing with the request andsending back the response and Each phase is important for defeating the Web serverfingerprinting.And because of Mod security inability to process the web server response to hideits identity in the sending back the response phase (its measures usually deny or pass), so Iadopted in my research to explain how can use Mod security rules to normalize the requests (receiving the request phase) and explain how to use a proxy server technique(using an open source proxy) to manage a client request and web server response (sendingback the response phase) to Complete my research work aim which is defeating afingerprinting attacks.Firstly we will have an introduction speaks about the improvement of the internetand how the client connect with server to make client-server interaction and there will be alittle information about HTTP fingerprintThen in chapter two we will speak about the concept and theory of fingerprinting fornetwork security and operating system with some explanation for the leak of applicationlayer because of firewall manner for packet inspection.Next we will provide in chapter three a discussion about HTTP fingerprintingmethodology classification and HTTP fingerprinting ways and tools with an experimentaltests to show the different responses from different HTTP servers with some examples tomake a summary table for different servers responses for same requests to each server.After that there will be in chapter four an introduction for the importance and advantage ofOSWAF (Open Source Web Application Firewall) ModSecurity, and how to use anddevelop its rules to defeat and protect servers of the Apache platform against HTTPfingerprint and to make it clear, we used HTTPrint free software as a source for HTTP fakerequests and how to defeat against this software (HTTPrint) by using ModSecurity Rules,and same operations will implemented against HTTPRecon software which is anotherfree software has the ability to send a fake requests to invoke the web server to respond toit’s fake requests and by comparing the outgoing responses from the web server with thedata stored in it’s database then it can identify and recognizes the server name and version.Lastly in chapter five, we will provide an introduction for a web proxy definition andexplain what is the different between Proxies and Gateways, in addition to that we willmention Benefits of Using a proxy server and brief definitions for forward and reverseproxies.Then there will be a discussion and an explanation of how can use an intermediaryproxy between a client and web server to normalize the incoming requests and outgoingresponses in the purpose of decreasing web server information leaking.