One thing I had identified to the client was that the subject was using a type of VoIP software.  They asked if it was possible to listen in on the voice conversations.  I told them it was, and that I could probably get them a copy of the voice conversations the subject previously had during the time I was monitoring.  I had packet captures, most non-encrypted so it was just work and time.  At the clients request, I extracted the VoIP conversations into wmv files using date and time of the call as a file name.

At the end of the job, I was having a conversation with the CTO.  He was wondering if there was an automated way to keep audio conversations of all the employees.  At the time, this technology was not as prevalent, cheap, and available to the general public as it is today.  I asked him if he thought that was really appropriate.  I explained that I had just listened in on someones private conversations.   Maybe it wasn’t any of the companies business.   Maybe there were legalities if they were to do that (yes, I was annoyed).  His response was very quick.  “The company has a right to view all data, monitor activity that its equipment or network is used for, period”.   He told me all the employees know this and sign a document to that effect.  I said that made sense.   I asked him what he would think if he was in a confidential conversation on the phone with someone in a different province and Bell had listened in on his conversation?  I said that I assume he didn’t have problem with it, after all it is their network, their devices.  Aside from the angry facial expression, he said that was ‘different’ and they shouldn’t be allowed to do that.

Fast forward to now.  Everyone has a video camera or picture camera on them as a result of mobile phones.  If you are serious about it, you can find all kinds of tiny spy cameras.  Rob Spence has implanted a camera in his eye. It amuses me when law enforcement gets all concerned about citizens taking their picture and video taping them.  I guess they feel that they should be able to watch and monitor us, but we shouldn’t be able to watch and monitor them.  Of course if they are not doing anything wrong, then they should have nothing to worry about right? (that statement is an entire topic in and of itself).

Everyone has reasons why a particular person or group of people should or should not be monitored.  It really comes down to the basic premise that we as humans don’t want to be monitored, but we want the ability to monitor others, especially if we deem them as a threat.  Government wants the ability to covertly monitor their citizens but do not want organizations covertly monitoring them.  Police want cameras everywhere so they can monitor what is going on and use it to assist with their job, but they don’t want to be video taped in case they get caught doing something controversial, such as Robert Dziekanski being killed by officers at Vancouver airport. The video once released on the Internet, forced police to change their story.  Businesses feel they have a right to monitor their employees, but would have concerns if employees were monitoring some of their activities.

Personally, I think it is futile to attempt to stop one group from monitoring another, especially in public places.  It will never be successful.  Who do you feel should be able to monitor who?   Under what circumstances and conditions is video or audio surveillance appropriate?

photo credit

My first day in Dubai, I went to lunch with one of the executives of the Telco.  During our lunch, I asked him how they manage encrypted connections.  He explained that they were currently getting ready to deploy a solution to solve that.  The infrastructure was being upgraded to decrypt all SSL sessions and parse the data as required.   Aside from opening my eyes to the difference in privacy between North America and the Middle East from a privacy perspective, I found it interesting that SSL decrypting was so easily available.   Previously, I had seen software that law enforcement used for this purpose.  I myself had done it for clients using available tools during a engagement.  But these tools were designed more for targeted surveillance, not mass scale.  Like all technology, it improves and gets less expensive over time I guess.

Today, there are many more companies in North America and abroad that either have deployed SSL decrypting capabilities or are in the process of doing so.  Security, diagnostics, audit and legal requirements to know what is coming and leaving their networks and being able to log and trace back data transmissions to the originator are some of the reasons.  One driver is Data Leakage Protection (DLP), currently a very ‘hot’ topic with many new vendors jumping on the opportunity with solutions.  In order to look for data leakage, you need to see past any encryption that might be present.  Cisco, Bluecoat, PaloAlto, Fortinet are just a few companies that offer products for SSL decryption.

With Google deploying encryption for Gmail and more recently searching, plug-ins such as the EFF Firefox plug-in to help secure your communications, companies are feeling more and more concerned about what data is coming and going.  What worries me is that all these security, legal and audit requirements companies face are actually not helping them in the long run.   If these companies are decrypting SSL sessions that egress and ingress their network, you can be sure that other companies are doing the same to theirs.  The net result is that everything is decrypted and no one has any privacy.

Next time you connect to your bank, doctor’s office, insurance company, Gmail or any site and see secure indications from your browser similar to these

along with the companies re-assurances that the site is secure, keep in mind things may not be as they appear – today even more so than yesterday.

Do you deploy any type of decryption on your network?  If it is deployed are you aware of it?

photo credit

Over the Easter weekend, I had the opportunity to speak to a friend who has worked for the federal government for over 30 years.  My friend was telling me about a security team who’s  sole responsibility is to be proactive.  This team searches the network looking for vulnerabilities or attacks that are in progress, usually under the radar using a variety of open source and other tools.  My friend was very positive about them, indicating the team has done really good work and produced excellent results.  I was happy to hear that a large organization such as the federal government had a full time team dedicated to this purpose.

In my years consulting for many different industries both large and small, I have seen a very obvious increase in proactive security monitoring, analysis, and investigation.  Most financial industries have teams in place today as well as other large organizations.  Unfortunately, in some cases, these teams are not dedicated full time, rather it is one part of their many responsibilities.  In my opinion, this is where a mistake is being made and the effectiveness of having proactive security teams starts to be a problem.

One of the biggest reasons that proactive security analysis teams are not present, or only part-time is cost and lack of measurable valid metrics.  How do you measure the effectiveness?  It is possible the team might go for weeks, not finding any big vulnerabilities.  Maybe there are not currently any attacks present on the network.  Maybe there are active attacks, but they are currently not looking in the right places?  Maybe they don’t have the expertise required to see the attack in progress?   From a financial perspective, one sees large sums of money for the team of experts and you may or may not get tangible results.  It is a tough justification.  If money gets tight within the organization, this problem often worsens.  Research often falls into very similar circumstances.  There is an intrinsic value to having these types of teams, but how does one represent that financially?  I haven’t figured out an answer to this yet.

For industries that provide infrastructure or financial services, or deal with data that is sensitive, I believe that regulation from government is necessary for this type of activity to be provided with guarantees.  I think as a society we will eventually get there, but it will be a long battle with industries pushing back indicating that they can self-regulate.  Given the types of attacks that are now prevalent, proactive analysis with expert people is absolutely necessary.

If you ask any organization large or small they will all state they take information security very seriously.  But would you expect a different answer?  I have spent the last 8 years consulting, and this has given me an insight into those statements.  In my experience, the reality of those statements contain quite a bit of variance.  From my Consulting engagements in many different parts of the world, I find that this is somewhat geographically based.  If you head over to the middle east for example, I have found that proactive security is present in many organizations and it is not new.  The attitude is different as well.  Proactive security is expected, from senior management down and if you mention the idea of not having it, the reaction is to look at you as if you are nuts and in most cases that reaction is a truthful one,

How serious is your organization about security?  Do there actions match their statements or are they just words?

If you want privacy or anonymity on the Internet, there are many things you can do. Proxies, Tor, encrypted tunnels, compromised systems, and many other techniques are available.  None of these will guarantee you anonymity or privacy, but they each help and the more you can do the better.  There are caveats of course and in several cases while consulting I have come across scenarios where a client thought they were being anonymous but were in fact not as anonymous as they thought.  When you are trying to be anonymous, use of monitoring techniques and system checks really help.

I’ve realized that running a Tor exit node but not using it yourself gives you anonymity.  I’ve always known this inherently, but I’ve realized that it is even better than I thought.  Say you are an evil person doing something evil on the Internet.  If your activities were being tracked by your service provider due to a warrant from law enforcement or laws were put in place that required all service providers to track and retain your Internet surfing activities for a period of time, they would be recording the surfing habits of every connection that selected your Tor node as its exit node.

If they accused you of illegal activity, you could easily say that was not me, it must have been someone using my Tor node.  While this is not a guarantee the criminal would not get caught, it would increase the cost of the investigation significantly.  More investigation time, more forensics to prove that the suspect is the criminal.  Add in anti-forensics on your terminals and systems you use for the crime and the costs for investigation again will increase, forcing them to assess if it is worth the time, money, and resources required.

If countries are going to deploy the retention laws similar to the above, it will only be a matter of time before they will have to outlaw services such as Tor in order to make them effective at catching the serious criminals.  From a Tor network perspective, these laws might help increase the node count of the Tor network on the Internet which is a good thing for them.

I wonder if law makers consider these questions when suggesting these laws?

I reviewed a report where it was indicated that because the data was encrypted it was impossible to determine anything useful.  This is not always the case, but I have seen this conclusion in reports and investigations many times when dealing with encrypted or unidentified data.  Aside from the marketing which says that if my Internet sessions are encrypted then one is safe (nothing could be further from the truth), many network administrators do not understand or have had much experience with behavioural profiling.  Behavioural profiling of networks can be very complex, and research is relatively new in this area.  To give some insight into how one might profile network sessions and show how one can use behavioural profiling to extract information, I decided to walk through a simple example and answer a simple question.  Specifically, what are the differences between an encrypted network session where one is watching a program or video (user providing no input), compared to an interactive type of network session where one is interacting (providing input)?  I used the SSH protocol to illustrate.

I used video over SSH to watch a program.  The program was approximately 24 minutes in duration and was hosted on a server at my ISP.   There were no problems watching the program, it didn’t pause or stop, and it was just like watching a typical television program (in fact I watched it on my flat screen TV).  I used a device to capture the traffic between the server hosting the program and my home for the entire duration of the program.  Finally, I captured an interactive SSH session which was me logged into a server at my ISP, where I was doing some coding and some shell commands.

Attempts to look at the actual data of either of these captures will be useless.  Since the data is encrypted, without access to the session keys knowing what was transmitted is close to if not impossible.  That being stated, what behaviour characteristics can we observe to tell us what might be going on?

I separated the direction of each of two captures which gave me 4 capture files, video received, video transmitted, interactive data received and interactive data transmitted.

Bandwidth

Looking at the chart above, the video watching has a much larger amount of data received than transmitted compared to the interactive session where a similar amount of data is transmitted and received.  Analysis of most video streaming and flows where downloading is occurring will yield a similar results.  The ratio of received to transmitted data will be high.  Interactive sessions tend to have a more balanced ratio of transmitted to received data compared to a video session.  This of course has dependencies on what the user is doing in the interactive session, but typically this has been the case in my experience.

Inter-packet timing

Another interesting metric is the time difference or delta between two packets.  Watching a video or listening to music, the delta between two packets tends to be small in comparison to an interactive type of session.  There are a few reason for this.  Since the video is being viewed, it is important to ensure that the data arrives in a timely manner so as to not have the video ‘freeze’ while being watched.   Some software attempts to write the video data to disk in advance of viewing to help mitigate this problem, but that leaves an exposure where an savvy individual can obtain a copy of the video by simply making a copy of the temporary file.  As a result, newer software tends to attempt to keep the data in memory and not write it to disk.  The result is the need to ensure a smooth delivery of data, minimizing delay between packets (known as Jitter).

I wrote a simple python script which will take as input a capture file, calculates the inter-packet timing for each pair of packets and then outputs among other information, the results you see in the table above.  The Maximum field is the largest time between packets, the mean is the average time between packets, and the standard deviation is a measure of how ‘different’ the inter packet times are from the ‘normal’.  For those that don’t know or wish to have a refresher in standard deviation, here is a good place to start. However, most languages and spreadsheets have functions to calculate this for you if you do not wish to learn the math.  In simple terms and using our specific example, if all the packets had the exact same time between them then the standard deviation would be 0.  The greater the difference in timing between packets, the greater the standard deviation will be.

Notice that the standard deviation is much higher for the interactive session then the video session.  Sessions that stream data, tend to have a low standard deviation for inter-packet timing.  If you think about it this makes sense, as an interactive session you can walk away from the computer, or the program could be waiting for input from the user so data transmission will fluctuate more.

Bandwidth, inter-packet timing, and methods such as standard deviation and mean are just a few things that can be used to narrow down what a particular subjects activities might be.  In corporate or law enforcement investigations, profiling network behaviour can be a useful tool to determine if you need to spend more time on the investigation or if you have the right target.  Using our example above,  suppose a corporation wants to determine which employees are watching streaming videos.  A scan of the network data reveals an individual who has encrypted sessions, but these sessions show a transmit / receive ratio that is in line with typical interactive sessions and not video sessions.  Also, the standard deviation of the inter-packet timing is higher for these sessions, then you can rule them out as an individual of interest immediately.  This has the advantage of focusing your investigation, not encroaching on privacy issues unnecessarily,  and saves time by allowing you to focus on the users that have network sessions with characteristics that fit the behaviour you are looking for.

For those of you that feel comfortable because the data is ‘encrypted’ it can be a false sense of security.  These are two of the many metrics and theorems that can be used on the data.  This area has active research and there are many products that will do this type of analysis in an automated fashion.  For those interested in this, although older now, this is a great paper where an experiment was conducted to determine what movie people were watching even though the movie data was encrypted.  They used behavioural data to fingerprint the movies, then applied the fingerprints to encrypted transmitted data.

The problem became how could I use some standard networking tools to passively monitor what she is up to on the Internet? I made some basic assumptions.  First, I am only interested in HTTP for now.  Second, I want to extract the sites she visits and do not care about the data that is returned at this point.

We have a Linux box that acts as our gateway to the Internet, so that seemed like the best place to deploy the solution. The first thing was to create a regular expression (regex) that will examine each packet that leaves our internal network and look for commands from the HTTP protocol specification. Any packets matching this will be saved for future analysis. The regex I created is:

^([Gg][Ee][Tt]|[Pp][Oo][Ss][Tt])|([Hh][Ee][Aa][Dd])|([Pp][Uu][Tt])|([Dd][Ee][Ll][Ee][Tt][Ee])|([Tt][Rr][Aa][Cc][Ee])|([Oo][Pp][Tt][Ii][Oo][Nn][Ss])|([Cc][Oo][Nn][Nn][Ee][Cc][Tt])\x20*[Hh][Tt][Tt][Pp]\x2f\x31\x2e

This regex looks for any packet that begins with a HTTP 1.x command such as GET,POST,HEAD,PUT,DELETE,TRACE,OPTIONS, or CONNECT.  The command is separated by a space and then contains the HTTP version number, HTTP 1.  I am aware the regex could be made more optimal.  I chose to not worry about it as this format makes it easier to explain and understand if you are not familiar with regular expressions.  For those with DPI experience, there are more complex and accurate ways to detect HTTP.  For example, ipoque the company that initiated opendpi.org, released some “demo code” that shows some of the ways deep packet inspection (DPI) works.  You can run the demo code on any pre-saved capture files you have and it will attempt to inform you of the protocols that are in the capture file.   If you look at their code for HTTP detection, they have a multi-stage approach that looks at both sides of the flow to determine if the protocol is in fact HTTP.  Any vendors selling DPI equipment today should be doing this type of approach for protocol detection when possible.  However, for the purposes of determining what a individual is doing, I feel this is overkill.  If the situation was a company that was ’suspicious’ of an employee and just wanted to investigate simple solutions are better.  If criminal activity was found, and the data goes to court you want to be able to explain how you gathered the data, why it is valid and what it means.  Keep the explanation as simple as possible in these potential circumstances.

The only two missing pieces are we need to specify that this is for packets egressing from a particular computer (in this case my daughters).  This can be accomplished by adding a Berkeley Packet filter (BPF) on ngrep which will pre-process the packets prior to the application of the regular expression.  The final command I deployed was:

ngrep -O ./httpWatch1.cap -d eth1 -tq -Wbyline “^([Gg][Ee][Tt]|[Pp][Oo][Ss][Tt])|([Hh][Ee][Aa][Dd])|([Pp][Uu][Tt])|([Dd][Ee][Ll][Ee][Tt][Ee])|([Tt][Rr][Aa][Cc][Ee])|([Oo][Pp][Tt][Ii][Oo][Nn][Ss])|([Cc][Oo][Nn][Nn][Ee][Cc][Tt])\x20*[Hh][Tt][Tt][Pp]\x2f\x31\x2e”  “src host 10.1.1.40 and tcp”

This records all packets to a file called httpWatch1.cap that arrive on my internal interface eth1 where an HTTP 1.x command is encountered and the source of the request is TCP and from my daughters computer.  The screen shot below of the first few packets show what you can expect throughout the file.

I let it capture for approximately 8 days.  In the next few days I will post how to take the data in this file and manipulate it to extract the information I am looking for.

There has been lots of discussion lately about Flash websites using Local Shared Objects (LSO) to track users selections, browsing habits, and other information.  One of the advantages for websites has been that until now they have not been well known.  From my basic searching they have been around since at least 2004 and probably earlier.  A user may configure their browser to remove or delete all ‘cookies’, but LSOs stay.  According to some, many of the top websites use them.

I tried a little experiment to see how LSOs are stored.  The directory that they are stored varies depending upon your operating system.  For me I use Linux as my primary O/S.  The default directory for LSOs is ~/.macromedia/Flash_Player.

Under the ‘Flash_Player’ there are two directories and under each of these directories are the security configuration and the binary installer for the Flash Air application.  Nothing interesting.  Next, I started Firefox and went to youtube.com and selected a video.  After the video completed, I took another look at the ~/.macromedia/Flash_Player directory.

Under~/.macromedia/Flash_Player we now have two new directories, macromedia.com and #SharedObjects.  If we decend the macromedia.com directory, we find 3 nested single directories called support, flashplayer, and sys respectively.  Under the ’sys’ directory we find a binary file called settings.sol and a subdirectory which is #s.ytimg.com owned by Google.  The #s.ytimg.com directory contains a separate settings.sol which is binary.

Under the #SharedObjects directory, there is a single oddly named directory ‘3BJH4AW6′, then a directory for the website ’s.ytimg.com’, a domain owned by Google.  Below this are two files entitled videostats.sol and soundData.sol, both containing binary data.

I haven’t investigated the format or contents of the .sol files, but it is obviously where the metadata is stored.  I may try to investigate the format or see if anyone else has already figured it out as I am curious.  The bigger question in my mind is how does one properly erase this data.  There is a Firefox add-on called BetterPrivacy which will do just that. It can be configured to delete LSOs on request or remove all the LSOs when you shutdown Firefox.  I installed BetterPrivacy and tried it.  Sure enough, upon shutting down Firefox I was greeted with this window:

Selecting OK, put my ~/.macromedia/Flash_Player directory back to its original state with no LSOs or website directories present.  For the normal user that should suffice.  However, these are files and they have been deleted.  Most people should know that files these days that are deleted are typically still recoverable.   File systems such as NTFS (windows), ext2/ext3 (*nix) all can have files deleted on them recovered.  In the case of ext3, it is a journal file system and the default file system installed on most *nix platforms today.  Without getting into the details in this post, this effectively means that even if you wipe a file it can potentially still be recovered.

If you carry around sensitive information on your laptop, I recommend you create an encrypted volume on your hard drive using a package such as TrueCrypt, PGP.  In the case of my system, I formatted the encrypted file system to be ext2.  This means there is no journaling.  This has the disadvantage of being less ‘recoverable’ but it has the advantage that if you wipe a file with ‘wipe’, ’shred’ or some other wiping software it is unlikely to be recovered.  Next, I point my ~/.macromedia directory to the encrypted file system.

You can see the ~/mndData file which is the truecrypt fileystem.  ~/.macromedia is symbolically linked to the encrypted filesystem.  For those interested, you can see that my Evolution (~/.evolution), Google Desktop (~/.google), Firefox Cache and bookmarks (~/.mozilla), IM client (~/.purple) and Skype (~/.Skype) all write to the encrypted file system.  You have to be able to mount the ~/mndData to get at any of the email, browser cache, bookmarks, IM conversations and now LSOs.  It isn’t fool proof, but it offers another layer of protection so that client data remains unviewable in the event of my laptop being stolen.

The DNS section of the presentation was based on my earlier two posts on DNS analysis which are here and here.   The SSL server_name option was based on my post that is here.  The “Working with service providers” I have never really posted about yet, but have been engaged with service providers all over the world for almost 5 years consistently, so I spoke about my experiences, and thoughts.

The presentation slides are here.

What can you learn from a visualization such as this?  Could you build a profile of the persons in this house just from their DNS queries?  And if you can, what does it tell you?  Twitter is obviously used in the house as the largest number of references are made to ‘twitter’. ‘Sandvine’ is also used often.  There are references to ‘mac’ and ‘apple’.  ‘facebook’ also is large relative to the others.  There are queries to ‘thepiratebay’. What do these all mean?  What can we infer from them, and are we accurate with our inferences?

Using the same dataset with full queries, here it is visualized as a bubble graph .

From this visualization, ‘twitter.com’ and ’search.twitter.com’ receive most of the queries, making it safe to say there is probably at least an active twitter account with an individual in this residence.  The ‘DC-2.sandvine.com’ sheds light that someone reguarily looks up what is probably a ‘Domain controller’ for ‘Sandvine.com’.  If from this you were to infer an employee of Sandvine, well you’d be correct.  You can not actually get to any of those servers without using a VPN, but due to the way DNS works, it often leaks.

Over the next few weeks, I will be working with this data, the graphs above, with other tools and DNS vectors to determine what  else can be inferred from just DNS.

You can clearly see the increase in bandwidth on this one link during the Inauguration.  This has happened before. Twitter has inauguration data that shows the same trend for their micro blog service.

As the Internet becomes more and more the media for information, bandwidth is going to constantly increase and spike when these type of events occur. Service providers need to effectively manage the bandwidth, ensuring fairness, privacy, and deploying appropriate infrastructure to support the trending increase in bandwidth over the next 5-10 years.

I look at my family over the last 3 years. We hardly watch television and any shows we do watch, we watch via the Internet. We listen to the radio via the internet. We get all information and news via the Internet. We communicate almost exclusively via the internet.