Catch Me If You Can – Gaps in Log and Event Management Coverage

Catch Me If You Can – Gaps in Log and Event Management Coverage

So I was approached by several friends in the managed services and security operations services fields, last year, with questions about discovering an attacker that is already inside a corporate network. Specifically, both had recently had clients go through internal penetration tests and only had limited success in identifying the hacking attempts. After talking with them, we quickly identified several gaps in their overall security posture as well as logging and event management processes that would have made discovery of an attacker much easier. Sure enough, on this year’s internal penetration test, all of them where happy to announce they had identified and caught the penetration teams within the first 20 minutes of this year’s tests.

What changes did they make?

The first change was in their security posture or the way they listen or monitor for internal threats. They needed to make this first change, because like so many other security operations teams they were assuming the wrong things when looking for an internal attacker. I personally believe that compliancy-based testing is the root of this common misconception. When performing these compliancy-based “penetration tests” most testing/auditing companies immediately start the engagement by firing up the vulnerability scanner and blasting at the devices on the network. Ironically most internal security operations teams get excited about discovering this and start believing this is normal “hacker” activity. The problem is this is not normal activity at all, it is just a compliance test or vulnerability assessment.

Today’s penetration methodology de jour is just the opposite of this. How so? Simple, we do not run vulnerability scanners and rarely run any port scanners at all during a real penetration test. The other key technique to go undetected is to keep all of your activity low and slow to minimize someone putting events together.

Many security operation teams look at internal network and hosts based IDS/IPS solutions and immediately turn off logging for any low or informational ranked vulnerabilities and purely focus on high and medium threats. Why? Because of the noise factor. For every high or medium event flagged by an ids/ips there are 100s if not 1000s of low and informational events that get flagged in a typical network. Knowing this, most pentesters and real attackers will simply stay in the low and information realm of vulnerabilities to map out and attack a network.

Here’s an example – Do you still have NULL sessions enabled in your Windows network settings? Still using SNMP with public or private set as the password to monitor things? Do you still allow null sessions to your LDAP servers? Do you still allow clear-text services like HTTP or Telnet to access internal devices or solutions? All of these threats are technically considered a low severity vulnerability when discovered on the internal networks, but can easily be escalated to a Severe or High threat by an attacker.

So what is the typical internal penetration test methodology? Well there will be some variance from tester to tester, but most happen in the following manner:

• Step 1 – Tester plugs in laptop and gets a DHCP address.
• Step 2 – Tester makes note of the WINS servers provided by the DHCP servers (they are 95% of the time the domain controllers for the Windows network).
• Step 3 – Tester uses a basic script to use null session against the domains controllers and enumerate all of the users and groups in the domain.
• Step 4 – Tester then does basic brute force against all privileged users – username is password or password is blank.
• Step 5 – Tester starts doing slow brute force testing against the domain users – test for the top 5-10 common bad passwords. Test only 2 passwords per day, a few hours apart.
• Step 6 – Tester dumps master browser table to discover systems in the domain that are running SQL Servers.
• Step 7 – Tester starts testing for weak SA password on all SQL servers in the domain.
• Step 8 – Tester does a TCP and UDP port ping for specific services such as Oracle, MySQL, and VNC, etc.
• Step 9 – Tester starts testing for weak passwords on systems that responded to the TCP and UDP port pings
• Step 10 – Tester elevates permissions on all the systems they have access to. This shows they have gained privileged access and starts harvesting all the data they have access to. This will illustrate how easy it was to break into the system.
• Step 11 – Use exploits – Rarely needed!

So using this as a play list of what to look for with your internal network and host based IDS/IPS solutions, you will notice that the majority of this activity will go unnoticed on the typical deployment as all but a few of the various solutions out there label these events as a Medium or High risk threat.

So the first change needed here is to at least log these events. I know there is a lot of events per day, but this will at least give you and your security operations teams the ability to do log analysis to try and identify an attack against your systems.

The second area for improvement is event logging. Now in the example above, some of the tests will generate an event, namely when we have failed login attempts while performing the brute force attacks. The problem however, is that most operating systems and IDS/IPS solutions only flag multiple login attempts against a single account and not single login attempts across multiple accounts. Knowing this, the attacker can easily try the top 5-10 list of bad passwords, until they’ve collected enough accounts to be leveraged to cause real damage in the environment.

To discover this threat, again we recommend leveraging a logging solution, send all of your event logs to a log management solution and leverage a security event solution to set the criteria to generate an alert. The criteria is up to you and your environment. Are five failed attempts within a matter of 15 seconds enough for your environment or is it ten failed attempts within a matter of 30 seconds or a minute? Your choice, but this is the start of identifying real brute force attempts.

The other event to correlate is when a batch of users all log in within a short internal of time from each other. This one is tough (especially during lunch hours), but establishing an alert when say 10 accounts log in within 15-30 seconds of each other is another great way to discover an attacker leveraging brute force attempts to login. If you got an alert that said bob smith , training 1-10, and BESAdmin all logged in at nearly the same time from the same IP or system name, I would hope you would send someone to investigate that.

Now lets say everything up to this point has slipped through our monitor solution. Do we give up? No! There is another layer of alerting and logging we can use, but it’s a complicated one to tackle

Log successful file access.

I told you it was complicated. So in the example pentest above, we assume that we would have seen the tester attempting to access files they did not have access too.

Note – It has been my experience that very few organizations today actively alert on failed file access attempts.

Let’s assume this is true however, but let’s augment this. So we found the failed attempts, but could you have spotted the successful attempts? Namely, does your organization have the ability to spot a user accessing or dumping lots of files? The answer in my experience is no. In over 20 years of doing testing, I’ve only seen two organizations that logged all file access and they were both in the government sector. Even then there are gaps, just ask Bradley Manning and Wikileaks.

The problem here is again, too much noise. But by monitoring truly sensitive or critical files, while leveraging log management and event based alerting we can establish an alerting criteria to catch our tester or thief in the act. Again the criteria will have to be established on a case by case basis. In your office, how many files does a typical user access a day? Start at 50, if too noisy move this to 60 or 75. That way you can spot the day when Bob’s account attempted to grab 100 files at a time. Maybe Bob is quitting tomorrow or his account was compromised. Either way, events above the norm should be investigated.

And if you are wondering, yes, all of my friends spotted the penetration teams due to the successful login attempts, and started investigating the attempts. The best one noticed that 25 accounts all logged in within 30 seconds of each other. They then caught the penetration teams in the act, when they started getting successful file access notifications from the same accounts within the next 2 hours. The best one got notification of a penetration tester trying to grab a folder with 1000 files in it.

Incidentally, they’ve also been able to catch a few employees attempting to “backup” all their account information before their planned departure from their companies. Let’s just say that I was told their departure was accelerated in several instances.

Like I said, these solutions are not easy, but if implemented will provide you with the ability to spot internal attackers, rogue employees, or penetration testers.

   By: Jim Broome   11.26.12