This blog post series will be covering the topic of performing Web Application Penetration Tests. An important thing that you should keep in mind during a Penetration Testing engagement is to constantly think about how someone could abuse and exploit the existing functionality of the application. In part one of this series, we will cover the initial steps of our security tests.
HTTP is a protocol which allows the fetching of resources, such as HTML documents. It is the foundation of any data exchange on the Web and it is a client-server protocol, which means requests are initiated by the recipient, usually the Web browser. HTTP functions as a request–response protocol in the client–server computing model. Clients and servers communicate by exchanging individual messages. The messages sent by the client, are called requests and the messages sent by the server as an answer are called responses. A web browser, may be the client and an application running on a computer hosting a website may be the server. The client submits an HTTP request message to the server. The server, which provides resources such as HTML files and other content, or performs other functions on behalf of the client, returns a response message to the client. The response contains completion status information about the request and may also contain requested content in its message body.
The first documented version of HTTP was HTTP V0.9. The first version of the protocol had only one method, namely GET, which would request a page from a server. The response from the server was always an HTML page. HTTP/1.1 is a revision of the original HTTP (HTTP/1.0). In HTTP/1.0 a separate connection to the same server is made for every resource request. HTTP/1.1 can reuse a connection multiple times to download images, scripts, stylesheets, etc after the page has been delivered. HTTP/1.1 communications therefore experience less latency as the establishment of TCP connections presents considerable overhead.
HTTPS is an extension of the HTTP. It is used for secure communication over a computer network, and is widely used on the Internet. In HTTPS, the communication protocol is encrypted using Transport Layer Security (TLS) or, formerly, its predecessor, Secure Sockets Layer (SSL). The protocol is therefore also often referred to as HTTP over TLS or HTTP over SSL. The principal motivations for HTTPS are authentication of the accessed website, protection of the privacy and integrity of the exchanged data while in transit. It protects against man-in-the-middle attacks. The bidirectional encryption of communications between a client and server protects against eavesdropping and tampering of the communication. In practice, this provides a reasonable assurance that one is communicating without interference by attackers with the website that one intended to communicate with, as opposed to an impostor.
HTTP defines a set of request methods to indicate the desired action to be performed for a given resource. Any client can use any method and the server can be configured to support any combination of methods. If a method is unknown to an intermediate, it will be treated as an unsafe and non-idempotent method.
The GET method requests a representation of the specified resource. Requests using GET should only retrieve data.
The HEAD method asks for a response identical to that of a GET request, but without the response body.
The POST method is used to submit an entity to the specified resource, often causing a change in state or side effects on the server.
The PUT method replaces all current representations of the target resource with the request payload.
The DELETE method deletes the specified resource.
The CONNECT method establishes a tunnel to the server identified by the target resource.
The OPTIONS method is used to describe the communication options for the target resource.
The TRACE method performs a message loop-back test along the path to the target resource.
The PATCH method is used to apply partial modifications to a resource.
Robots exclusion standard
The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned. Robots are often used by search engines to categorize websites. Not all robots cooperate with the standard; email harvesters, spambots, malware and robots that scan for security vulnerabilities may even start with the portions of the website where they have been told to stay out. The standard can be used in conjunction with Sitemaps, a robot inclusion standard for websites.
Ncat is a feature-packed networking utility which reads and writes data across networks from the command line. It uses both TCP and UDP for communication and is designed to be a reliable back-end tool to instantly provide network connectivity to other applications and users. This tool is very useful and handy during the Penetration Testing process.
Burp Suite is a suite of tools allowing penetration testers to intercept all requests and responses between the browser and the target application, even when HTTPS is being used. We are going to use one of its tools called “Repeater“, to fingerprint the web server. This is the the Must Have tool during a Web Application Security Assessment.
Web Server Fingerprinting
The reason behind the web server fingerprinting process is that it will allow us to search the internet for known vulnerabilities as long as we succeeded in acquiring the server’s software and version and it will also make our attacks more effective if we are using the appropriate tools and the appropriate technology while targeting the web server in scope.
Fingerprint Web Server using the Ncat tool
Open a terminal and execute the command, “ncat example.com 80”.
HEAD / HTTP/1.1
Hit enter twice.
Fingerprint Web Server using Burp Repeater
Start your Burp Suite. Select the temporary project option.
Use Burp defaults.
Open the Repeater tab.
Configure your target on the right side and type the following in the Request pane:
HEAD / HTTP/1.1
Send malformed requests
Try sending malformed requests or requests of non-existent pages to the server. Create requests with non-existent HTTP methods, HTTP versions and protocols.
Request a non-existent page.
Send a request with a non-existent HTTP method.
Send a request with a non-existent protocol.
Send a request with a non-existent version.
Retrieve robots.txt using your browser
Open your favorite browser and type a URL, e.g. https://www.google.com/robots.txt. Each of these discovered robots.txt directories should be enumerated/brute-forced to discover more hidden/public exposed directories. We will get back to directory brute-forcing at a later blog post.
Retrieve robots.txt using the Burp Repeater
Open the Repeater tab and type the following:
GET /robots.txt HTTP/1.1
Look for default HTML Pages
Search online for vulnerabilities
After discovering web server software version you should search online for any changelog documents and CVEs.