Michael N. Dundas

The picture on the right is taken from a Canadian television series called “The Border“.  It follows a team of Canadian customs agents saving Canada from threats.  In this particular episode called “Kiss and Cry“, Slade who is their technical wizard agent discovers that the Chinese secret service has installed a trojan in their system allowing them to monitor their activities.  Upon investigation, discovery of the trojan, and a quick assessment of the risk, he immediately initiates a system wide shutdown of all services.  Given the sensitivity of data they have in their systems, the type of data their systems have access to,  and the nature of their business it was the right call, however I found it interesting that Slade made it.

Although this is a fictional television series, this scene got me thinking about my clients.  I can not think of any client large or small that is prepared for or has a single staff member onsite that could authorize a system wide shutdown quickly.  As an example, let’s take a large financial institution.  One of the technical staff is doing some routine system checks and discovers that every time a customer logs into their bank accounts, the customers login and password information along with other helpful data such as birth date and postal code is transmitted externally to a range of servers.  Being a large financial institution there is presently a new customer login average of one per second.  What should she do?  Should she shutdown all customer access immediately?  Should she investigate?  If she investigates, how long should she investigate for?  Can she get hold of someone who can authorize the shutdown?  What if that person is unavailable?  Can she make the call to shutdown services then?  It is obviously critical.  Should she keep trying others?  If so for how long?  If from discovery through investigation to authorization it takes 10 minutes, that is 600 client compromises in this scenario.

What is important is that the staff clearly understand what they can and can not do in any situation.  They need to feel comfortable they have done the right thing and will not be punished for doing what they ‘perceive’ as the right thing.  In the scenario above, if you asked your employees what they would do in this scenario, do you know what they would answer?  Would they be comfortable answering the questions above and more importantly would the business be comfortable with the answers and the risks associated with those responses?

I know many business people that would indicate this is fictional or ‘far fetched’.  While I would have agreed to some degree a few years ago, I wouldn’t today.  What I would suggest is that they go to a recent technical (not business) security conference or ask your technical team or consultants about latest research into threats and vulnerabilities and their availability.  Don’t ask the vendors (or at least be careful), they are trying to sell you results and are never as advanced as the bad guys.  Also keep in mind that even research is behind.  There are many malicious pieces of software that are ‘underground’, but you don’t need to look there.  Just look at some of the available off the shelf tools available for purchase.

Is your business realistically aware of the current threats to its data?  Are the risk assessments accurate?  Do you have the appropriately qualified staff and procedures in place to deal with current threats and do they have the appropriate authorization to make the necessary calls in the event of an emergency or unexpected event?  Is the business comfortable and accepting of the risk exposure associated with these decisions?

In working with large companies such as service providers, financial and manufacturing institutions, I have come across many common and simple attacks.  I will discuss one that I came across recently while planning for a project.  It is not a new attack as I and most other security professionals have encountered it many times.   The attack itself has been around for years now.  What amazes me is that regardless of how simple, common, and old the attack is I usually find it undetected on most networks.

Before walking through the attack, let me describe the steps used for this attack.  There are many  papers, books, courses and posts by security professionals on how to effectively detect and respond to attacks, the proper methodology, decision points and other variables.  These methods vary to different degrees in application, complexity and point of view.  For example, the methods and steps identified and taken by a first responder will be different than a security architect designing a system.    For the purposes of this post, I’ve chosen a simple set of steps:

• Detection
• Investigation
• Scope
• Assessment
• Mitigation

Detection

I was working on a particular server and router.  I was planning a side project I have an interest in and wanted to check the configurations of the router and server to ensure it would support my project.  During the course of checking the server, I issued a command to check for the current connections being made to the server (netstat).

What immediately jumped out at me was the ssh connection highlighted above in red.  Although SSH is permitted to this system, there is only 3 people that have access and all are members of the same ISP.  This connection was not part of the ISP netblocks.  It is possible someone could have been traveling and accessed it remotely but I was confident no one with access was in China  (where the IP is registered).  Regardless of the source address, the source port ’36948′ was constantly changing every few seconds, indicating a new connections being spawned.

Investigation

After observing the constant connection attempts, a quick look at the server logs and some basic filtering revealed the following:

Nov 16 00:45:05 serverA sshd[5423]: Invalid user admin from 218.108.234.208
Nov 16 00:45:05 serverA sshd[5424]: input_userauth_request: invalid user admin
Nov 16 00:45:06 serverA sshd[5423]: Failed password for invalid user admin from 218.108.234.208 port 36910 ssh2
Nov 16 00:45:10 serverA sshd[5425]: Invalid user test from 218.108.234.208
Nov 16 00:45:10 serverA sshd[5426]: input_userauth_request: invalid user test
Nov 16 00:45:11 serverA sshd[5425]: Failed password for invalid user test from 218.108.234.208 port 38556 ssh2
Nov 16 00:45:14 serverA sshd[5427]: Invalid user guest from 218.108.234.208
Nov 16 00:45:14 serverA sshd[5428]: input_userauth_request: invalid user guest
Nov 16 00:45:16 serverA sshd[5427]: Failed password for invalid user guest from 218.108.234.208 port 40196 ssh2
Nov 16 00:45:19 serverA sshd[5429]: Invalid user webmaster from 218.108.234.208
Nov 16 00:45:19 serverA sshd[5430]: input_userauth_request: invalid user webmaster
Nov 16 00:45:22 serverA sshd[5429]: Failed password for invalid user webmaster from 218.108.234.208 port 41776 ssh2
Nov 16 00:45:31 serverA sshd[5434]: Invalid user oracle from 218.108.234.208
Nov 16 00:45:31 serverA sshd[5435]: input_userauth_request: invalid user oracle
Nov 16 00:45:33 serverA sshd[5434]: Failed password for invalid user oracle from 218.108.234.208 port 45829 ssh2
Nov 16 00:45:36 serverA sshd[5436]: Invalid user library from 218.108.234.208
Nov 16 00:45:36 serverA sshd[5437]: input_userauth_request: invalid user library
Nov 16 00:45:38 serverA sshd[5436]: Failed password for invalid user library from 218.108.234.208 port 47647 ssh2
Nov 16 00:45:41 serverA sshd[5438]: Invalid user info from 218.108.234.208
Nov 16 00:45:41 serverA sshd[5439]: input_userauth_request: invalid user info
Nov 16 00:45:43 serverA sshd[5438]: Failed password for invalid user info from 218.108.234.208 port 49440 ssh2
Nov 16 00:45:46 serverA sshd[5440]: Invalid user shell from 218.108.234.208
Nov 16 00:45:46 serverA sshd[5441]: input_userauth_request: invalid user shell
Nov 16 00:45:48 serverA sshd[5440]: Failed password for invalid user shell from 218.108.234.208 port 51218 ssh2
Nov 16 00:45:51 serverA sshd[5442]: Invalid user linux from 218.108.234.208
Nov 16 00:45:51 serverA sshd[5443]: input_userauth_request: invalid user linux
Nov 16 00:45:53 serverA sshd[5442]: Failed password for invalid user linux from 218.108.234.208 port 52953 ssh2
Nov 16 00:45:56 serverA sshd[5444]: Invalid user unix from 218.108.234.208
Nov 16 00:45:56 serverA sshd[5445]: input_userauth_request: invalid user unix
Nov 16 00:45:59 serverA sshd[5444]: Failed password for invalid user unix from 218.108.234.208 port 54704 ssh2
Nov 16 00:46:02 serverA sshd[5446]: Invalid user webadmin from 218.108.234.208
Nov 16 00:46:02 serverA sshd[5447]: input_userauth_request: invalid user webadmin
Nov 16 00:46:04 serverA sshd[5446]: Failed password for invalid user webadmin from 218.108.234.208 port 56994 ssh2
Nov 16 00:46:13 serverA sshd[5451]: Invalid user test from 218.108.234.208
Nov 16 00:46:13 serverA sshd[5452]: input_userauth_request: invalid user test
Nov 16 00:46:16 serverA sshd[5451]: Failed password for invalid user test from 218.108.234.208 port 60988 ssh2
Nov 16 00:46:24 serverA sshd[5456]: Invalid user admin from 218.108.234.208
Nov 16 00:46:24 serverA sshd[5457]: input_userauth_request: invalid user admin
Nov 16 00:46:27 serverA sshd[5456]: Failed password for invalid user admin from 218.108.234.208 port 36482 ssh2
Nov 16 00:46:30 serverA sshd[5458]: Invalid user guest from 218.108.234.208
Nov 16 00:46:30 serverA sshd[5459]: input_userauth_request: invalid user guest
Nov 16 00:46:32 serverA sshd[5458]: Failed password for invalid user guest from 218.108.234.208 port 38285 ssh2
Nov 16 00:46:35 serverA sshd[5460]: Invalid user master from 218.108.234.208
Nov 16 00:46:35 serverA sshd[5461]: input_userauth_request: invalid user master
Nov 16 00:46:37 serverA sshd[5460]: Failed password for invalid user master from 218.108.234.208 port 39898 ssh2
Nov 16 00:47:20 serverA sshd[5489]: Invalid user admin from 218.108.234.208
Nov 16 00:47:20 serverA sshd[5490]: input_userauth_request: invalid user admin
Nov 16 00:47:23 serverA sshd[5489]: Failed password for invalid user admin from 218.108.234.208 port 54777 ssh2
Nov 16 00:47:26 serverA sshd[5491]: Invalid user admin from 218.108.234.208
Nov 16 00:47:26 serverA sshd[5492]: input_userauth_request: invalid user admin
Nov 16 00:47:28 serverA sshd[5491]: Failed password for invalid user admin from 218.108.234.208 port 56536 ssh2
Nov 16 00:47:31 serverA sshd[5493]: Invalid user admin from 218.108.234.208
Nov 16 00:47:31 serverA sshd[5494]: input_userauth_request: invalid user admin
Nov 16 00:47:33 serverA sshd[5493]: Failed password for invalid user admin from 218.108.234.208 port 58262 ssh2
Nov 16 00:47:36 serverA sshd[5495]: Invalid user admin from 218.108.234.208
Nov 16 00:47:36 serverA sshd[5496]: input_userauth_request: invalid user admin
Nov 16 00:47:38 serverA sshd[5495]: Failed password for invalid user admin from 218.108.234.208 port 60006 ssh2
Nov 16 00:47:52 serverA sshd[5503]: Invalid user test from 218.108.234.208
Nov 16 00:47:52 serverA sshd[5504]: input_userauth_request: invalid user test
Nov 16 00:47:54 serverA sshd[5503]: Failed password for invalid user test from 218.108.234.208 port 36914 ssh2
Nov 16 00:47:57 serverA sshd[5505]: Invalid user test from 218.108.234.208
Nov 16 00:47:57 serverA sshd[5506]: input_userauth_request: invalid user test
Nov 16 00:47:59 serverA sshd[5505]: Failed password for invalid user test from 218.108.234.208 port 38498 ssh2
Nov 16 00:48:04 serverA sshd[5507]: Invalid user webmaster from 218.108.234.208
Nov 16 00:48:04 serverA sshd[5508]: input_userauth_request: invalid user webmaster
Nov 16 00:48:06 serverA sshd[5507]: Failed password for invalid user webmaster from 218.108.234.208 port 40506 ssh2
Nov 16 00:48:09 serverA sshd[5509]: Invalid user user from 218.108.234.208
Nov 16 00:48:09 serverA sshd[5510]: input_userauth_request: invalid user user
Nov 16 00:48:11 serverA sshd[5509]: Failed password for invalid user user from 218.108.234.208 port 42147 ssh2
Nov 16 00:48:14 serverA sshd[5511]: Invalid user username from 218.108.234.208
Nov 16 00:48:14 serverA sshd[5512]: input_userauth_request: invalid user username
Nov 16 00:48:16 serverA sshd[5511]: Failed password for invalid user username from 218.108.234.208 port 43771 ssh2
Nov 16 00:48:19 serverA sshd[5513]: Invalid user username from 218.108.234.208
Nov 16 00:48:19 serverA sshd[5514]: input_userauth_request: invalid user username
Nov 16 00:48:21 serverA sshd[5513]: Failed password for invalid user username from 218.108.234.208 port 45636 ssh2
Nov 16 00:48:24 serverA sshd[5515]: Invalid user user from 218.108.234.208
Nov 16 00:48:24 serverA sshd[5516]: input_userauth_request: invalid user user
Nov 16 00:48:26 serverA sshd[5515]: Failed password for invalid user user from 218.108.234.208 port 47217 ssh2
Nov 16 00:48:35 serverA sshd[5520]: Invalid user admin from 218.108.234.208
Nov 16 00:48:35 serverA sshd[5521]: input_userauth_request: invalid user admin
Nov 16 00:48:37 serverA sshd[5520]: Failed password for invalid user admin from 218.108.234.208 port 50752 ssh2
Nov 16 00:48:40 serverA sshd[5522]: Invalid user test from 218.108.234.208
Nov 16 00:48:40 serverA sshd[5523]: input_userauth_request: invalid user test
Nov 16 00:48:42 serverA sshd[5522]: Failed password for invalid user test from 218.108.234.208 port 52460 ssh2
Nov 16 00:49:05 serverA sshd[5536]: Invalid user danny from 218.108.234.208
Nov 16 00:49:05 serverA sshd[5537]: input_userauth_request: invalid user danny
Nov 16 00:49:07 serverA sshd[5536]: Failed password for invalid user danny from 218.108.234.208 port 32852 ssh2
Nov 16 00:49:10 serverA sshd[5538]: Invalid user sharon from 218.108.234.208
Nov 16 00:49:10 serverA sshd[5539]: input_userauth_request: invalid user sharon
Nov 16 00:49:12 serverA sshd[5538]: Failed password for invalid user sharon from 218.108.234.208 port 34547 ssh2
Nov 16 00:49:15 serverA sshd[5540]: Invalid user aron from 218.108.234.208
Nov 16 00:49:15 serverA sshd[5541]: input_userauth_request: invalid user aron
Nov 16 00:49:17 serverA sshd[5540]: Failed password for invalid user aron from 218.108.234.208 port 36174 ssh2
Nov 16 00:49:20 serverA sshd[5542]: Invalid user alex from 218.108.234.208
Nov 16 00:49:20 serverA sshd[5543]: input_userauth_request: invalid user alex
Nov 16 00:49:22 serverA sshd[5542]: Failed password for invalid user alex from 218.108.234.208 port 37737 ssh2
Nov 16 00:49:25 serverA sshd[5544]: Invalid user brett from 218.108.234.208
Nov 16 00:49:25 serverA sshd[5545]: input_userauth_request: invalid user brett
Nov 16 00:49:27 serverA sshd[5544]: Failed password for invalid user brett from 218.108.234.208 port 39340 ssh2
...............

From the server logs, we can determine:

• Attack started at 00:45
• Dictionary attack where the attacker is sequencing through names as well as common Unix account ids.
• Rate is approximately 1 id every 1.5-2 seconds
• Source port is reasonably random, or at least random enough to fool basic firewall and IPS technologies.

Scope

What other systems if any on the network are under attack?  To determine this quickly I logged onto an aggregation point and captured traffic that corresponded to the attack in progress for a few minutes.  Next, a command was run to filter the captured data to show the servers that were being attacked.

$ tcpdump -n -r ./sshBfAttack-ispView.cap "src net 218.108.234.0/24 and tcp[tcpflags] & (tcp-syn) != 0" | awk '{print $5}' | awk -F. '{print $1"."$2"."$3"."$4}' | sort -u
reading from file ./sshBfAttack-ispView.cap, link-type EN10MB (Ethernet)
xxx.x0.0.25
xxx.x0.0.4
xxx.x0.0.43
xxx.x0.12.100
xxx.x0.12.101
xxx.x0.12.103
xxx.x0.12.136
xxx.x0.12.142
xxx.x0.12.20
xxx.x0.12.29
$

We now have a list of current targets.  The filter above is a simple filter and it makes some basic assumptions.  Several filters were run on the traffic to ensure the scope of the attack but for the purposes of this post, the concept is what is important.  The type of filters and parameters of the filters one uses will depend on the type of attack, direction of the attack and other factors.

Assessment / mitigation

What most fear when they assess an attack are false positives of actions they perform.  An action that causes a valid request to be denied for example.  In the case of a company such as an Internet service provider, financial institution or any business that makes money using the Internet, this could be detrimental.  How a company mitigates or handles an attack really depends on many factors.  The type of attack, the behaviour of the attack,  the risk of stopping the attack,  the risk of letting the attack proceed are just some examples of questions that need to be asked and answered.

For this specific attack:

• The servers being attacked contained no financial or personal data that was at risk to anyone.
• One of the servers controls some password authentication features
• The attack is external and coming from a specific IP address.
• The service under attack is really not required for external access.

The solution was to deploy an access control list on the routers to not permit connections to that service from external sources.  This effectively mitigated the attack.

Conclusion and thoughts

What amazes me is that these dictionary type of attacks, regardless of service are very common.  Every step I have outlined here can be automated and should be, yet in so many cases this is not true.  I know many organizations that have spent thousands of dollars on projects, vendor equipment, security audits, and consultants, yet you take a look at their network and this simple, known,  attack is still present and goes on undetected.

Has your company spent time and money on security solutions such as audits, penetration tests, and products for security?  If you looked at your network or asked your security folks if the attack here would be automatically detected, reported, investigated and mitigated if it was present on your network would the answer be ‘yes’.  If not, why not?

Seth Godin put together eBook entitled “What Matters Now”.  So far I have only read the first 30 pages.  He contacted a bunch of individuals and asked them to write a page expressing their thoughts and feelings on the future.  Several of the individuals are people I follow on a regular basis.  So far it has been a great read, especially this time of year.   If you are still interested, I’d suggest reading Seth’s blog entry or Michael Hyatt’s posts.  Both are much better writers than I and will do it the justice it deserves.  You can also download the eBook from links in their posts.

The Canadian Government announced this morning that Globalive (operating as Windmobile in Canada) is free to enter into the Canadian Market and compete with our Tier 1 providers in the wireless space.  Their ability to compete in Canada has been under fire by Rogers, Telus, and Bell for a while now.  What I am amazed at most is the responses I have been seeing on Twitter, instant messaging, comments on news articles and even a poll.  If it wasn’t obvious before now, Canadian consumers seem to be:

• very happy that the CRTC decision was over turned and Globalive is allowed to compete in Canada.
• Customer anger, frustration, and resentment are very obvious with Rogers, Bell, and Telus.

These feelings don’t just pop up.  They have obviously been building in consumers over time.   I hope this is a wake-up call for the providers.  The anger and frustration being expressed is serious and I am disappointed they either were too naive to see it building in their customers, or just didn’t care.  Either way, it will now probably directly affect them.  My hope is that they learn to value their customers thoughts and opinions in the future.  As a side note, I think this is the happiest I’ve seen Canadians with the current conservative government to date.

I was recently given via Michael Hyatt at Thomas Nelson Publishing a copy of the book Derailed.  The book was written by Tim Irwin.  In the book, Tim discusses what he feels are the reasons why leaders fail as leaders and gives insight into how to avoid these situations.

The first part of the book profiles 6 CEOs of major companies that failed as leaders.   Each leader is analyzed and what Tim feels are their weaknesses and the reasons why they were asked to resign from their position.  While one can argue that the opinions are subjective (and they are), I found his rational to be sounds and made sense.  Regardless of the subjectivity, anyone can learn from the mistakes of these profiled individuals and help themselves be a better leader.  I found myself identifying with the character flaws of these individuals.  I have seen them in many people I have worked with and even myself at times.

The rest of the book discusses the derailment process.  Finally, based on the profiles and the derailment process, Tim Irwin identifies and discusses five lessons that can be learned by anyone in a leadership role and ways to implement these lessons and keep yourself ‘in check’ as a leader.

I found Derailed extremely valuable for myself.  It was well worth the time to read and would suggest it to really anyone that interacts with other people at work.  Although the book profiles CEOs of large organizations, it is very applicable to anyone, even those not in an ‘official’ leadership position.