Michael N. Dundas

http://www.flickr.com/photos/tiffanyhoran/4288875968/

Most people have come to expect that when an email is sent it will arrive at it’s destination.  Over the last decade, email delivery has become much more reliable due to many factors such as better network architecture, better mail server design, load-balancing and fail over design, all driven by increased reliance on email in todays world.  There is also the ability to request a delivery receipt on most email clients although users typically disable this feature themselves, or the security policy of the organization disables it.  Email however is not a guaranteed delivery service.  The SMTP protocol as well as the process of email delivery on the Internet does not guarantee delivery.

One technique that I have used when someone has either not responded or indicated that they did not receive my email is to check the server delivery logs.  While this does not guarantee that the email was placed in the destination users mailbox, it does indicate acceptance at the mail exchanger of the ISP or company.

Above is an email I sent to a friend last week confirming plans for dinner.  By viewing the headers and looking for the SMTP “Message-ID” field, I can then search for that ID in the log files of the mail server.

# cat maillog | grep -i "4B618D6A.2070804"
Jan 28 08:13:19 mailsvr sendmail[20093]: o0SDDGVS020093: from=<xx@xxxxxxxxxx.org>, size=399,, nrcpts=1, msgid=<4B618D6A.2070804@xxxxxxxxx.org>, proto=ESMTP, daemon=MTA, relay=eee.dddd.ca [216.bbb.ccc.12]
#
# cat maillog | grep -i "o0SDDGVS020093"
Jan 28 08:13:19 mailsvr sendmail[20093]: o0SDDGVS020093: from=<xx@xxxxxxxxxx.org>, size=399,, nrcpts=1, msgid=<4B618D6A.2070804@xxxxxxxxx.org>, proto=ESMTP, daemon=MTA, relay=eee.dddd.ca [216.bbb.ccc.12]
Jan 28 08:13:20 mailsvr sendmail[20098]: o0SDDGVS020093: to=<yyyyyy@gggggggg.com>, ctladdr=<xx@xxxxxxxxxxx.org> (501/501), delay=00:00:01, xdelay=00:00:01, mailer=esmtp, pri=120399, relay=ttttttt.hhhhhhcom. [142.fff.rrr.227], dsn=2.0.0, stat=Sent (Ok: queued as 51E02514002)
#

In this case the server logs are using Sendmail, so depending on your server, the procedure might be slightly different.  Using the SMTP Message-ID field as a search parameter, I obtain the entry of the unique ID of the Sendmail delivery process for that message, in this case “o0SDDGVS020093″.  Searching the log file for that unique ID, shows me the remote mail server that accepted the email for delivery.  The status is “sent” and confirmed by a Deliver Status Notification (dsn) of 2.0.0.

There are many other fields and status messages with server logs, some you can see above, which are useful resources when troubleshooting or doing forensic activity involving an email transmission in an investigation.   Although this might appear to be too technical for a general user, I have used the logs to confirm myself if email is getting to at least the mail exchanger.  These records can assist in determining if the email arrived.  At the very least, you can use it as evidence the email was received by the destination company.  While it is not 100% proof, it is typically a good indicator.

In one instance, I was not getting a response from my daughter’s school concerning a particular issue.  After several attempts, I sent a new email asking why they were not responding, as it appeared obvious the school board was receiving the emails and I attached the log.  I had a response within the hour.  I am sure the users didn’t fully understand each field, but it was enough to get a response.   I don’t know of any service providers or companies that provide an on-line interface to check status of messages, but it might not be a bad service to offer.

http://www.flickr.com/photos/solarider/2255744829/

A few years ago, I was having a discussion with an acquaintance who was involved in an investigation.  One individual they were tracking kept changing his mobile phone every few days.  Each new mobile was typically pay as you go or stolen and personal information connected to the mobile was either false or not available.  Yet the investigators were able to very quickly determine the new number of the individual each time they switched mobile numbers.    How they did this at the time impressed me, and I use the logic to this day.

Throughout the course of the investigation they were able to determine who this individual contacted.  A few of the mobiles that the individual contacted did not routinely change their mobile number.  As a result, by watching the calling patterns of the mobile phones where the numbers did not change, the investigators could quickly determine a new number that suddenly was calling each of the static numbers in a similar pattern.  This of course requires access to mobile network data, but it worked.  Even though this individual thought they were not being tracked,  their efforts to remain anonymous unknown to them were ineffective.   As a side note, there is software that will search for and detect these types of calling patterns automatically.  The same logic here can easily be applied to a Internet connection.

A more common example is when you are ever pulled over by a police officer and you don’t have your license.  Aside from them giving you a ticket for not having your license on your person, they will most likely ask you for your full name and birth date.  The reason for the birth date is to help assure them that when they go back to the cruiser to search on their laptop, the records they obtain are actually yours and not someone else with the same name.   How many Michael Dundas’ are there in Canada?  Not sure, but the number of Michael Dundas’ with the exact same birth date really lowers the probability of a false positive.  This same logic can be applied to social networking and there is interesting research in this area including twitter.

The EFF recently published a post on information theory and privacy.  In it they discuss the concept of Entropy and how it applies to information and privacy.  It touches a bit on some of the math behind it, but if you are interested it is a good explanation of why when you think you are anonymous you may not be, even when you take precautions.  If you skip the math, their example of how a ‘user-agent’ header transmitted by your browser can narrow you down to one of 1500 people can start to give people that are new to information and anonymity a good perspective.

http://www.flickr.com/photos/imuttoo/3935553419/

On the Securosis blog there has been two posts recently (here and here) about security and taking a default deny position as the best approach to securing a particular service or network.  At a high-level, you block every port / service / protocol that is not defined as being required and then wait to see who complains.  As people complain, you investigate the complaints and figure out what policy changes are required and make them.  The end result is a secure policy allowing only the required access.  At least that is the theory.

I believe that “default deny” is a excellent security goal.  That being said, obtaining that goal has to be weighed against other objectives.  Often, I find many security professionals proclaiming that ‘default deny’ must be deployed, everyone has to make it happen, regardless of the cost to the company, regardless of the risk to the project.  The general sense is that if default deny can not be completely reached, the project should not go forward or should be held up.

This sets a very adversarial tone for everyone involved in the project.  It creates a very binary choice, “you are either with us or against us”, there is no in between.  While this is great in the movies, for the most part, it is not real life.  That positioning breaks down communication, it puts the team on the defensive, and it creates a environment where the team does not want to talk, work, or involve the security experts.   They are seen as unreasonable and unrealistic.  Have you ever been ordered by law enforcement to “stand back”, “show me your driver’s license”, or told you can not cross this line with no explanation as to why?  How does it make you feel?  Did this attitude earn your respect or lessen your respect for them?

The default deny stance is easy, minimal work, and most importantly risk free for the security members of the team.  While that is not a bad thing, it often increases the amount of work for others on the team as well as their responsibility.  In a simple case, if on a project by blocking port 1234/tcp, I force the team to have to re-program the socket interface on the application, which in turn generates a code review, which then generate more work for the Q/A team.  If the team overrides the security experts and says we are not doing that work, the security members can now claim they did their part, the team did not listen and so if there is a breech it is not their fault.  This does not promote a collaborative team environment.

Humans naturally fear the unknown.  It explains why as a society we overreact to terrorists that attempt to blow up planes or all rush to get the latest vaccine for a new strain of bacteria.  In both cases we are more at risk of death from being hit by a car in our daily travels yet we show no fear that will occur.  This irrational fear is re-enforced in courses and books on security.  The result is we see “default deny” as a valid and only solution.   The result is security professionals promoting often with a very hard line just that.

“Default deny” ideally assumes that their is an understanding of a service or application in its entirety.  From the end user interface right down to the bits that traverse the wire  in detail under all conditions.  Years ago this was possible, however todays applications are rarely the result of one teams code from the ground up.  APIs of third party vendor systems are called, third party libraries are used for communication, storage, authentication and many other functions and features.   Today, it is unreasonable to assume that a particular team will understand everything at all levels given the nature of how services on the Internet are built and deployed.  Security professionals are correct in pointing out this is a risk, however it is a risk that is not going to go away and security models have to adapt to manage and minimize the risk.  A simplistic “Default deny” does not accomplish this.

I have consulted for several very large tier 1 service providers.  The default position tends to be a “Default permit”.  From there they determine what is ‘bad’ and craft security policies to deal with and minimize the risk.  While enterprises can afford to take a more “Default deny” approach, this will become more and more difficult.  As services are more and more build by external vendors, use third party APIs and libraries, interact more and more with cloud computing, permit access on PDA devices for services, and the many other services available and yet to be available a different approach is needed.   “Default deny” is a great goal for security of a project, however it needs to be prioritized with and assessed from a risk perspective with other goals of a project.

Do you think that the security community of today needs to change their approach, and behaviour?

http://www.flickr.com/photos/room929/428260081/

Nicole Garton-Jones submitted on slaw.ca today a post entitled Practicing Law on the Road: the Role of the Cloud and the Emergence of the Virtual Law Firm.  In it she highlights the idea of working remotely and using VoIP, Cloud computing and virtual desktops along with your PDA and laptop devices.  Especially when it comes to law firms, my experience is they are often slower to adopt to technological changes that other industries due to a combination of tradition and general need to follow government laws, and procedures enforced by their professional organizations.  It is nice to see a lawyer promoting these technologies, I think that is great for the legal industry.

In her post, she discusses cloud computing, laptops and PDAs and touches on the security.  I feel that the security needs to be given a much more serious discussion.  My experience consulting with small companies and law firms is that they typically do not give security enough time, consideration, or expertise before choosing a technology path.  There are many reasons for this, cost, resources, and time being the main factors.  It is usually discussed when a laptop with sensitive data goes missing, someone realizes there is a keystroke logger on their system, or their server data has been compromised and is leaking onto the Internet bypassing the firewall, IDS, anti-virus, and notice of the system administrators or third party companies hired to provide system administration and security.

Cloud computing offers many advantages and cost savings to companies.  It also brings with it the concern of your data being stored off-site, out of your direct control.  With large cloud computing vendors such as Amazon and Google, your data could be across the world in a foreign country and the laws that apply to the protection of that data probably differ from those in your home country.  This has been a topic of discussion for a while now in the Cloud computing arena.  One of the suggestions is to use a ‘private’ cloud.  This is typically a cloud that you own or have more control over where the data is stored.  For example, Canadian Cloud offers a guarantee that “…data are safe and secure on hardware located in Canada, and subject only to Canadian laws and regulations..”  This resolves international issues when it comes to control of data and is appealing.  However, there is much more to consider before choosing a provider.  While Amazon, Google and other large companies are international, they also have the size to attract security professionals that are very knowledgeable and current.  They can afford the resources to properly monitor against attacks to steal your data.  Google recently publicized the discovery of China conducing espionage on its systems.  Will a provider of a smaller cloud offering have the resources to detect such attacks?  If you install your own cloud, do you have the resources to hire individuals capable of detecting these types of attacks?  One could argue that not using Amazon or Google is less secure and you have more risk exposure.  My point is that companies and firms need to consciously assess these decisions based on the sensitivity of the information they are thinking about storing on a cloud system.

Laptop security is still as important weather the cloud is present or not.  It makes sense for an attacker to go after the weakest link and that is almost always the end user device.  Although one may suggest that all the information is on the virtual desktop on the cloud, there may be cases where data needs to be pulled locally.  If this is the case and the data is sensitive you will require encryption.  Even if data is not stored on the laptop ever and therefore there is no need for encryption and the management tasks it brings, installation of malware that will capture keystrokes and gather screen shots is invaluable on the laptop of a lawyer involved in a sensitive case.  This software exists in many places and is easily obtained and deployed.  Proper user device security does not go away.

Between iPhone and Blackberry, currently the Blackberry is much more secure than an iPhone.  Blackberry has the infrastructure including BES servers which allow enforcement of detailed security policies along with a robust management architecture.  BES servers offer the ability to remotely wipe a lost Blackberry as well as the ability to track the location of the phone remotely.  The Blackberry device itself has the ability to wipe all data via a menu option or by simply entering the wrong password a configurable number of times.   By comparison, the current iPhone can have a password in place, but bypassing it is easy once you have the physical device and security policies can be easily overridden by the user of the device.  I fully expect the iPhone to improve in this area as it targets the business market, but currently this is the general state of security with the iPhone.  A company that deploys iPhones or Blackberries needs to consider the type of data on these devices and the required security.  While many users prefer the iPhone over the Blackberry, you are making a security decision when you make this decision as well.  Best to make it consciously and understand the risks you are assuming with your firm and clients data.

Companies and firms need to consciously assess the security requirements of their data independent of any one technology.  Once this is completed, choose and deploy solutions and services that meet those requirements balancing off risk, cost, and convenience.  While there is no such thing as 100% security, you can consciously minimize this exposure, and manage the risk.

How confident is your company or firm that data stored on your local servers, cloud infrastructure, laptops, PDAs and other devices is secure, and can not be extracted or viewed without proper authorization?  If your data was being extracted or viewed without authorization would your security team detect it?  If not, why not?

I posted a couple weeks ago about operators monitoring systems and discovering a serious exploit in progress and determining what to do if no one was available to make a call such as shutting down a service.  What metrics are in place such as length of time, number of phone calls, seriousness of incident, that allow an individual to make a call that might affect the business confidently.  My example was one where it was discovered that a hacker was slowly siphoning off account information at a financial institution. I don’t know what this particular institutions procedures were, but turns out my fictional example happened.  Not surprised as it is a valid scenario in todays world, but thought it was worth commenting.

On Slaw blog, there was a post today about some issues a few lawyers had when they ended up following an individual on Twitter. The post ends by effectively asking if people feel they should follow someone who follows them or not. I added my thoughts into the comments of that post, but thought that would be a good topic for a quick entry in my blog as I have pondered that question for a while.  I’ve added a little more detail here as to my criteria than the comment. The process is not cast in stone, rather a general set of guidelines that I typically use to make a decision.

My goal with social media is to connect and meet other interesting people.  As a general rule, I believe that when someone decides to follow you they are indicating they value your opinion and/or want to start some sort of on-line relationship with you. At least for a majority of people, I believe this to be true. Specifically in my areas of interest (security and networking), Twitter has been very valuable for me in building relationships, getting feedback, and keeping abreast of what is happening.  I also feel that the point of Twitter, Facebook, Linkedin and other social media sites is to connect with others, build relationships and trust.  Accomplishing that requires both parties to give, just like a relationship between two friends.  If it is one sided, what is the point?

That being said, there are those that will use social media for ‘bad’. Bad by my definition in this context, is to attempt to tweet me to death with useless information, send marketing links about products constantly, or use it as an automated tweeting tool where no real person is on the other side.

When someone follows me I typically do the following:

Check their twitter profile

Are others following them?  What is the ratio they have of followers to following?  If not many are following them, then I check how long they have been tweeting.  Maybe they are new.  The ratio of followers to following is a indication to me of how active they are and how interested they are in others.  A low follow rate may indicate they like to say things, but don’t like to hear opinions of others.  Not 100%, but an indicator.

Scan their tweets

I scan their previous tweets.  Are they informative and original or are they all just re-tweets.  Do they appear to be all just trying to sell products? Do they appear to be auto-generated?

Internet presence

Do they have an Internet presence such as a website, blog,  Facebook account,  Linkedin account?  If they have a website does it look legitimate?  Does the website or blog have information that is useful?  Are their opinions?  Is their an ‘about me’ area where they tell the reader about them.  This is extremely important to me.  I like to know who I am building a relationship with.  I don’t need big secrets about them, but a general concept of who you are, what you do, likes dislikes is helpful.  If I am going to read your posts, references to articles, I’d like to know that you are real and have some background and/or experience with the information you post.

General Internet search

I will search Google.  Do they post elsewhere?  Do they have comments and opinions?

Based on the information I find and feedback, I make a decision to follow or not.  This evaluation process is similar for blogs I add to my blog reader.  Again, this is not cast in stone.   There are a few that I follow that do not follow me back and that is fine.  However, for me that is the exception as opposed to the rule.

Do you have a criteria for who you follow on Twitter or what blogs you subscribe to?

Seth Godin was interviewed by Nora Young on Spark.  The interview can be found here.  The part of the talk where Seth describes how many of us were never trained to take initiative but to follow instructions and how that impacts us in our work made a lot of sense to me.  My favourite part was the section on emotional labour, the act of connecting to another human being and making a change even if it is not easy for you to do it in that moment.

A good talk for anyone in a leadership position.

The picture on the right is taken from a Canadian television series called “The Border“.  It follows a team of Canadian customs agents saving Canada from threats.  In this particular episode called “Kiss and Cry“, Slade who is their technical wizard agent discovers that the Chinese secret service has installed a trojan in their system allowing them to monitor their activities.  Upon investigation, discovery of the trojan, and a quick assessment of the risk, he immediately initiates a system wide shutdown of all services.  Given the sensitivity of data they have in their systems, the type of data their systems have access to,  and the nature of their business it was the right call, however I found it interesting that Slade made it.

Although this is a fictional television series, this scene got me thinking about my clients.  I can not think of any client large or small that is prepared for or has a single staff member onsite that could authorize a system wide shutdown quickly.  As an example, let’s take a large financial institution.  One of the technical staff is doing some routine system checks and discovers that every time a customer logs into their bank accounts, the customers login and password information along with other helpful data such as birth date and postal code is transmitted externally to a range of servers.  Being a large financial institution there is presently a new customer login average of one per second.  What should she do?  Should she shutdown all customer access immediately?  Should she investigate?  If she investigates, how long should she investigate for?  Can she get hold of someone who can authorize the shutdown?  What if that person is unavailable?  Can she make the call to shutdown services then?  It is obviously critical.  Should she keep trying others?  If so for how long?  If from discovery through investigation to authorization it takes 10 minutes, that is 600 client compromises in this scenario.

What is important is that the staff clearly understand what they can and can not do in any situation.  They need to feel comfortable they have done the right thing and will not be punished for doing what they ‘perceive’ as the right thing.  In the scenario above, if you asked your employees what they would do in this scenario, do you know what they would answer?  Would they be comfortable answering the questions above and more importantly would the business be comfortable with the answers and the risks associated with those responses?

I know many business people that would indicate this is fictional or ‘far fetched’.  While I would have agreed to some degree a few years ago, I wouldn’t today.  What I would suggest is that they go to a recent technical (not business) security conference or ask your technical team or consultants about latest research into threats and vulnerabilities and their availability.  Don’t ask the vendors (or at least be careful), they are trying to sell you results and are never as advanced as the bad guys.  Also keep in mind that even research is behind.  There are many malicious pieces of software that are ‘underground’, but you don’t need to look there.  Just look at some of the available off the shelf tools available for purchase.

Is your business realistically aware of the current threats to its data?  Are the risk assessments accurate?  Do you have the appropriately qualified staff and procedures in place to deal with current threats and do they have the appropriate authorization to make the necessary calls in the event of an emergency or unexpected event?  Is the business comfortable and accepting of the risk exposure associated with these decisions?

In working with large companies such as service providers, financial and manufacturing institutions, I have come across many common and simple attacks.  I will discuss one that I came across recently while planning for a project.  It is not a new attack as I and most other security professionals have encountered it many times.   The attack itself has been around for years now.  What amazes me is that regardless of how simple, common, and old the attack is I usually find it undetected on most networks.

Before walking through the attack, let me describe the steps used for this attack.  There are many  papers, books, courses and posts by security professionals on how to effectively detect and respond to attacks, the proper methodology, decision points and other variables.  These methods vary to different degrees in application, complexity and point of view.  For example, the methods and steps identified and taken by a first responder will be different than a security architect designing a system.    For the purposes of this post, I’ve chosen a simple set of steps:

• Detection
• Investigation
• Scope
• Assessment
• Mitigation

Detection

I was working on a particular server and router.  I was planning a side project I have an interest in and wanted to check the configurations of the router and server to ensure it would support my project.  During the course of checking the server, I issued a command to check for the current connections being made to the server (netstat).

What immediately jumped out at me was the ssh connection highlighted above in red.  Although SSH is permitted to this system, there is only 3 people that have access and all are members of the same ISP.  This connection was not part of the ISP netblocks.  It is possible someone could have been traveling and accessed it remotely but I was confident no one with access was in China  (where the IP is registered).  Regardless of the source address, the source port ‘36948′ was constantly changing every few seconds, indicating a new connections being spawned.

Investigation

After observing the constant connection attempts, a quick look at the server logs and some basic filtering revealed the following:

Nov 16 00:45:05 serverA sshd[5423]: Invalid user admin from 218.108.234.208
Nov 16 00:45:05 serverA sshd[5424]: input_userauth_request: invalid user admin
Nov 16 00:45:06 serverA sshd[5423]: Failed password for invalid user admin from 218.108.234.208 port 36910 ssh2
Nov 16 00:45:10 serverA sshd[5425]: Invalid user test from 218.108.234.208
Nov 16 00:45:10 serverA sshd[5426]: input_userauth_request: invalid user test
Nov 16 00:45:11 serverA sshd[5425]: Failed password for invalid user test from 218.108.234.208 port 38556 ssh2
Nov 16 00:45:14 serverA sshd[5427]: Invalid user guest from 218.108.234.208
Nov 16 00:45:14 serverA sshd[5428]: input_userauth_request: invalid user guest
Nov 16 00:45:16 serverA sshd[5427]: Failed password for invalid user guest from 218.108.234.208 port 40196 ssh2
Nov 16 00:45:19 serverA sshd[5429]: Invalid user webmaster from 218.108.234.208
Nov 16 00:45:19 serverA sshd[5430]: input_userauth_request: invalid user webmaster
Nov 16 00:45:22 serverA sshd[5429]: Failed password for invalid user webmaster from 218.108.234.208 port 41776 ssh2
Nov 16 00:45:31 serverA sshd[5434]: Invalid user oracle from 218.108.234.208
Nov 16 00:45:31 serverA sshd[5435]: input_userauth_request: invalid user oracle
Nov 16 00:45:33 serverA sshd[5434]: Failed password for invalid user oracle from 218.108.234.208 port 45829 ssh2
Nov 16 00:45:36 serverA sshd[5436]: Invalid user library from 218.108.234.208
Nov 16 00:45:36 serverA sshd[5437]: input_userauth_request: invalid user library
Nov 16 00:45:38 serverA sshd[5436]: Failed password for invalid user library from 218.108.234.208 port 47647 ssh2
Nov 16 00:45:41 serverA sshd[5438]: Invalid user info from 218.108.234.208
Nov 16 00:45:41 serverA sshd[5439]: input_userauth_request: invalid user info
Nov 16 00:45:43 serverA sshd[5438]: Failed password for invalid user info from 218.108.234.208 port 49440 ssh2
Nov 16 00:45:46 serverA sshd[5440]: Invalid user shell from 218.108.234.208
Nov 16 00:45:46 serverA sshd[5441]: input_userauth_request: invalid user shell
Nov 16 00:45:48 serverA sshd[5440]: Failed password for invalid user shell from 218.108.234.208 port 51218 ssh2
Nov 16 00:45:51 serverA sshd[5442]: Invalid user linux from 218.108.234.208
Nov 16 00:45:51 serverA sshd[5443]: input_userauth_request: invalid user linux
Nov 16 00:45:53 serverA sshd[5442]: Failed password for invalid user linux from 218.108.234.208 port 52953 ssh2
Nov 16 00:45:56 serverA sshd[5444]: Invalid user unix from 218.108.234.208
Nov 16 00:45:56 serverA sshd[5445]: input_userauth_request: invalid user unix
Nov 16 00:45:59 serverA sshd[5444]: Failed password for invalid user unix from 218.108.234.208 port 54704 ssh2
Nov 16 00:46:02 serverA sshd[5446]: Invalid user webadmin from 218.108.234.208
Nov 16 00:46:02 serverA sshd[5447]: input_userauth_request: invalid user webadmin
Nov 16 00:46:04 serverA sshd[5446]: Failed password for invalid user webadmin from 218.108.234.208 port 56994 ssh2
Nov 16 00:46:13 serverA sshd[5451]: Invalid user test from 218.108.234.208
Nov 16 00:46:13 serverA sshd[5452]: input_userauth_request: invalid user test
Nov 16 00:46:16 serverA sshd[5451]: Failed password for invalid user test from 218.108.234.208 port 60988 ssh2
Nov 16 00:46:24 serverA sshd[5456]: Invalid user admin from 218.108.234.208
Nov 16 00:46:24 serverA sshd[5457]: input_userauth_request: invalid user admin
Nov 16 00:46:27 serverA sshd[5456]: Failed password for invalid user admin from 218.108.234.208 port 36482 ssh2
Nov 16 00:46:30 serverA sshd[5458]: Invalid user guest from 218.108.234.208
Nov 16 00:46:30 serverA sshd[5459]: input_userauth_request: invalid user guest
Nov 16 00:46:32 serverA sshd[5458]: Failed password for invalid user guest from 218.108.234.208 port 38285 ssh2
Nov 16 00:46:35 serverA sshd[5460]: Invalid user master from 218.108.234.208
Nov 16 00:46:35 serverA sshd[5461]: input_userauth_request: invalid user master
Nov 16 00:46:37 serverA sshd[5460]: Failed password for invalid user master from 218.108.234.208 port 39898 ssh2
Nov 16 00:47:20 serverA sshd[5489]: Invalid user admin from 218.108.234.208
Nov 16 00:47:20 serverA sshd[5490]: input_userauth_request: invalid user admin
Nov 16 00:47:23 serverA sshd[5489]: Failed password for invalid user admin from 218.108.234.208 port 54777 ssh2
Nov 16 00:47:26 serverA sshd[5491]: Invalid user admin from 218.108.234.208
Nov 16 00:47:26 serverA sshd[5492]: input_userauth_request: invalid user admin
Nov 16 00:47:28 serverA sshd[5491]: Failed password for invalid user admin from 218.108.234.208 port 56536 ssh2
Nov 16 00:47:31 serverA sshd[5493]: Invalid user admin from 218.108.234.208
Nov 16 00:47:31 serverA sshd[5494]: input_userauth_request: invalid user admin
Nov 16 00:47:33 serverA sshd[5493]: Failed password for invalid user admin from 218.108.234.208 port 58262 ssh2
Nov 16 00:47:36 serverA sshd[5495]: Invalid user admin from 218.108.234.208
Nov 16 00:47:36 serverA sshd[5496]: input_userauth_request: invalid user admin
Nov 16 00:47:38 serverA sshd[5495]: Failed password for invalid user admin from 218.108.234.208 port 60006 ssh2
Nov 16 00:47:52 serverA sshd[5503]: Invalid user test from 218.108.234.208
Nov 16 00:47:52 serverA sshd[5504]: input_userauth_request: invalid user test
Nov 16 00:47:54 serverA sshd[5503]: Failed password for invalid user test from 218.108.234.208 port 36914 ssh2
Nov 16 00:47:57 serverA sshd[5505]: Invalid user test from 218.108.234.208
Nov 16 00:47:57 serverA sshd[5506]: input_userauth_request: invalid user test
Nov 16 00:47:59 serverA sshd[5505]: Failed password for invalid user test from 218.108.234.208 port 38498 ssh2
Nov 16 00:48:04 serverA sshd[5507]: Invalid user webmaster from 218.108.234.208
Nov 16 00:48:04 serverA sshd[5508]: input_userauth_request: invalid user webmaster
Nov 16 00:48:06 serverA sshd[5507]: Failed password for invalid user webmaster from 218.108.234.208 port 40506 ssh2
Nov 16 00:48:09 serverA sshd[5509]: Invalid user user from 218.108.234.208
Nov 16 00:48:09 serverA sshd[5510]: input_userauth_request: invalid user user
Nov 16 00:48:11 serverA sshd[5509]: Failed password for invalid user user from 218.108.234.208 port 42147 ssh2
Nov 16 00:48:14 serverA sshd[5511]: Invalid user username from 218.108.234.208
Nov 16 00:48:14 serverA sshd[5512]: input_userauth_request: invalid user username
Nov 16 00:48:16 serverA sshd[5511]: Failed password for invalid user username from 218.108.234.208 port 43771 ssh2
Nov 16 00:48:19 serverA sshd[5513]: Invalid user username from 218.108.234.208
Nov 16 00:48:19 serverA sshd[5514]: input_userauth_request: invalid user username
Nov 16 00:48:21 serverA sshd[5513]: Failed password for invalid user username from 218.108.234.208 port 45636 ssh2
Nov 16 00:48:24 serverA sshd[5515]: Invalid user user from 218.108.234.208
Nov 16 00:48:24 serverA sshd[5516]: input_userauth_request: invalid user user
Nov 16 00:48:26 serverA sshd[5515]: Failed password for invalid user user from 218.108.234.208 port 47217 ssh2
Nov 16 00:48:35 serverA sshd[5520]: Invalid user admin from 218.108.234.208
Nov 16 00:48:35 serverA sshd[5521]: input_userauth_request: invalid user admin
Nov 16 00:48:37 serverA sshd[5520]: Failed password for invalid user admin from 218.108.234.208 port 50752 ssh2
Nov 16 00:48:40 serverA sshd[5522]: Invalid user test from 218.108.234.208
Nov 16 00:48:40 serverA sshd[5523]: input_userauth_request: invalid user test
Nov 16 00:48:42 serverA sshd[5522]: Failed password for invalid user test from 218.108.234.208 port 52460 ssh2
Nov 16 00:49:05 serverA sshd[5536]: Invalid user danny from 218.108.234.208
Nov 16 00:49:05 serverA sshd[5537]: input_userauth_request: invalid user danny
Nov 16 00:49:07 serverA sshd[5536]: Failed password for invalid user danny from 218.108.234.208 port 32852 ssh2
Nov 16 00:49:10 serverA sshd[5538]: Invalid user sharon from 218.108.234.208
Nov 16 00:49:10 serverA sshd[5539]: input_userauth_request: invalid user sharon
Nov 16 00:49:12 serverA sshd[5538]: Failed password for invalid user sharon from 218.108.234.208 port 34547 ssh2
Nov 16 00:49:15 serverA sshd[5540]: Invalid user aron from 218.108.234.208
Nov 16 00:49:15 serverA sshd[5541]: input_userauth_request: invalid user aron
Nov 16 00:49:17 serverA sshd[5540]: Failed password for invalid user aron from 218.108.234.208 port 36174 ssh2
Nov 16 00:49:20 serverA sshd[5542]: Invalid user alex from 218.108.234.208
Nov 16 00:49:20 serverA sshd[5543]: input_userauth_request: invalid user alex
Nov 16 00:49:22 serverA sshd[5542]: Failed password for invalid user alex from 218.108.234.208 port 37737 ssh2
Nov 16 00:49:25 serverA sshd[5544]: Invalid user brett from 218.108.234.208
Nov 16 00:49:25 serverA sshd[5545]: input_userauth_request: invalid user brett
Nov 16 00:49:27 serverA sshd[5544]: Failed password for invalid user brett from 218.108.234.208 port 39340 ssh2
...............

From the server logs, we can determine:

• Attack started at 00:45
• Dictionary attack where the attacker is sequencing through names as well as common Unix account ids.
• Rate is approximately 1 id every 1.5-2 seconds
• Source port is reasonably random, or at least random enough to fool basic firewall and IPS technologies.

Scope

What other systems if any on the network are under attack?  To determine this quickly I logged onto an aggregation point and captured traffic that corresponded to the attack in progress for a few minutes.  Next, a command was run to filter the captured data to show the servers that were being attacked.

$ tcpdump -n -r ./sshBfAttack-ispView.cap "src net 218.108.234.0/24 and tcp[tcpflags] & (tcp-syn) != 0" | awk '{print $5}' | awk -F. '{print $1"."$2"."$3"."$4}' | sort -u
reading from file ./sshBfAttack-ispView.cap, link-type EN10MB (Ethernet)
xxx.x0.0.25
xxx.x0.0.4
xxx.x0.0.43
xxx.x0.12.100
xxx.x0.12.101
xxx.x0.12.103
xxx.x0.12.136
xxx.x0.12.142
xxx.x0.12.20
xxx.x0.12.29
$

We now have a list of current targets.  The filter above is a simple filter and it makes some basic assumptions.  Several filters were run on the traffic to ensure the scope of the attack but for the purposes of this post, the concept is what is important.  The type of filters and parameters of the filters one uses will depend on the type of attack, direction of the attack and other factors.

Assessment / mitigation

What most fear when they assess an attack are false positives of actions they perform.  An action that causes a valid request to be denied for example.  In the case of a company such as an Internet service provider, financial institution or any business that makes money using the Internet, this could be detrimental.  How a company mitigates or handles an attack really depends on many factors.  The type of attack, the behaviour of the attack,  the risk of stopping the attack,  the risk of letting the attack proceed are just some examples of questions that need to be asked and answered.

For this specific attack:

• The servers being attacked contained no financial or personal data that was at risk to anyone.
• One of the servers controls some password authentication features
• The attack is external and coming from a specific IP address.
• The service under attack is really not required for external access.

The solution was to deploy an access control list on the routers to not permit connections to that service from external sources.  This effectively mitigated the attack.

Conclusion and thoughts

What amazes me is that these dictionary type of attacks, regardless of service are very common.  Every step I have outlined here can be automated and should be, yet in so many cases this is not true.  I know many organizations that have spent thousands of dollars on projects, vendor equipment, security audits, and consultants, yet you take a look at their network and this simple, known,  attack is still present and goes on undetected.

Has your company spent time and money on security solutions such as audits, penetration tests, and products for security?  If you looked at your network or asked your security folks if the attack here would be automatically detected, reported, investigated and mitigated if it was present on your network would the answer be ‘yes’.  If not, why not?

Seth Godin put together eBook entitled “What Matters Now”.  So far I have only read the first 30 pages.  He contacted a bunch of individuals and asked them to write a page expressing their thoughts and feelings on the future.  Several of the individuals are people I follow on a regular basis.  So far it has been a great read, especially this time of year.   If you are still interested, I’d suggest reading Seth’s blog entry or Michael Hyatt’s posts.  Both are much better writers than I and will do it the justice it deserves.  You can also download the eBook from links in their posts.