Michael N. Dundas

My daughter has recently become much more interested in some of the social networking sites such as Facebook and Youtube. This is a little concerning for my wife and I. We encourage her to use technology as much as possible, but at the same time there is a inherent risk. There is software you can purchase and install that will download the latest bad sites, look for questionable URLs and even questionable pictures, but I didn’t want to move to this level just yet.  She is not running Windows.

The problem became how could I use some standard networking tools to passively monitor what she is up to on the Internet? I made some basic assumptions.  First, I am only interested in HTTP for now.  Second, I want to extract the sites she visits and do not care about the data that is returned at this point.

We have a Linux box that acts as our gateway to the Internet, so that seemed like the best place to deploy the solution. The first thing was to create a regular expression (regex) that will examine each packet that leaves our internal network and look for commands from the HTTP protocol specification. Any packets matching this will be saved for future analysis. The regex I created is:

^([Gg][Ee][Tt]|[Pp][Oo][Ss][Tt])|([Hh][Ee][Aa][Dd])|([Pp][Uu][Tt])|([Dd][Ee][Ll][Ee][Tt][Ee])|([Tt][Rr][Aa][Cc][Ee])|([Oo][Pp][Tt][Ii][Oo][Nn][Ss])|([Cc][Oo][Nn][Nn][Ee][Cc][Tt])\x20*[Hh][Tt][Tt][Pp]\x2f\x31\x2e

This regex looks for any packet that begins with a HTTP 1.x command such as GET,POST,HEAD,PUT,DELETE,TRACE,OPTIONS, or CONNECT.  The command is separated by a space and then contains the HTTP version number, HTTP 1.  I am aware the regex could be made more optimal.  I chose to not worry about it as this format makes it easier to explain and understand if you are not familiar with regular expressions.  For those with DPI experience, there are more complex and accurate ways to detect HTTP.  For example, ipoque the company that initiated opendpi.org, released some “demo code” that shows some of the ways deep packet inspection (DPI) works.  You can run the demo code on any pre-saved capture files you have and it will attempt to inform you of the protocols that are in the capture file.   If you look at their code for HTTP detection, they have a multi-stage approach that looks at both sides of the flow to determine if the protocol is in fact HTTP.  Any vendors selling DPI equipment today should be doing this type of approach for protocol detection when possible.  However, for the purposes of determining what a individual is doing, I feel this is overkill.  If the situation was a company that was ‘suspicious’ of an employee and just wanted to investigate simple solutions are better.  If criminal activity was found, and the data goes to court you want to be able to explain how you gathered the data, why it is valid and what it means.  Keep the explanation as simple as possible in these potential circumstances.

The only two missing pieces are we need to specify that this is for packets egressing from a particular computer (in this case my daughters).  This can be accomplished by adding a Berkeley Packet filter (BPF) on ngrep which will pre-process the packets prior to the application of the regular expression.  The final command I deployed was:

ngrep -O ./httpWatch1.cap -d eth1 -tq -Wbyline “^([Gg][Ee][Tt]|[Pp][Oo][Ss][Tt])|([Hh][Ee][Aa][Dd])|([Pp][Uu][Tt])|([Dd][Ee][Ll][Ee][Tt][Ee])|([Tt][Rr][Aa][Cc][Ee])|([Oo][Pp][Tt][Ii][Oo][Nn][Ss])|([Cc][Oo][Nn][Nn][Ee][Cc][Tt])\x20*[Hh][Tt][Tt][Pp]\x2f\x31\x2e”  “src host 10.1.1.40 and tcp”

This records all packets to a file called httpWatch1.cap that arrive on my internal interface eth1 where an HTTP 1.x command is encountered and the source of the request is TCP and from my daughters computer.  The screen shot below of the first few packets show what you can expect throughout the file.

I let it capture for approximately 8 days.  In the next few days I will post how to take the data in this file and manipulate it to extract the information I am looking for.