May 31, 2018 7:38:00 AM

3 Tips for Threat Hunting with Splunk

Staying Proactive in Your Security Strategy via Threat Hunting

Differentiating Threat Hunting from Incident Response, and How Splunk Can Help

Threat hunting is all about proactively searching to detect and isolate different threats in your environment that aren’t detected by your tools. The goal is to identify potential issues before they become problems. Incident response is entirely reactive and takes place after potential nefarious activity has already become a problem.

Splunk is an excellent tool to aid in threat hunting, focused on proactive interception. When you have access to all machine data, there’s nowhere for adversaries to hide, which is where Splunk really comes into play. Splunk provides access to all data in your environment, from IP addresses, ingress and egress traffic, network artifacts (flow, packet captures, DNS activity, zone transfers for DNS, endpoint host artifacts and patterns), vulnerability management data and user behavioral analytics.

Time is a very important factor in the threat hunting conversation. One of the greatest features of Splunk is its ability to bring in any sort of data that has a time stamp on it. This gives you the ability to link events based on time, so for example, you could use swim lanes to say, “activity A happened here, activity B happened on a different device, but at the same time,” etc., and you can start correlating events to figure out where threats are coming from, go in and remediate them.

This kind of efficiency is also a major benefit. If you’re in a security environment, you might have ten or more tools, all doing different things to secure your network. Splunk takes data and amalgamates it in one convenient, searchable area. Instead of logging into ten different tools or devices, you can have it all centralized in Splunk, easily accessible.

Real-Life Examples: What I’ve Found Threat Hunting with Splunk

Let’s talk through a couple of real-life examples of threat hunting with Splunk.

I’ve found a plethora of interesting and useful things working with Splunk for threat hunting. In the past, I was an analyst at a federal agency, and I noticed a lot of traffic going outward. I looked at the IP address, and it was for a file sharing program for video streaming. It turned out, this was software someone at the company was using that allowed them to stream movies at work, and this particular software is known for exfiltrating data. Once I dug deeper, I found all these other machines were using this program, so I quickly cut those off and was able to stop what could’ve been a massive government breach thanks to a little bit of curiosity and Splunk, which is where threat hunting comes into play.

You’re really looking for anomalies and things that aren’t supposed to be on your network, or maybe they are, but they look weird. When you find something that doesn’t belong, you could possibly save your organization millions of dollars.

For instance, at another organization, I noticed that someone was constantly accessing their device from an external country. If you logged in from another country, you should’ve automatically gotten your certificate revoked, but this person was connecting initially from an outside network and using the internal VPN to do their work, which meant that they managed to shift around the rule for some time. I was able to display my findings and got their certificate revoked until they returned back to the U.S.

Three Tips for Threat Hunting with Splunk

1. Ingestion: make sure you’re getting ALL data you have available into your Splunk environment. If you have a file log system, IPS or antivirus, make sure you’re getting it all into Splunk.

You can use Splunk as a glass window where you can see everything that’s going on in your network, but it only works as a single point of truth if you’re putting stuff in there to begin with. This all exists in your network, and you can see what happens assuming you have a couple tools, (which, everyone should, in the security shop). So, making sure you’re getting all the logs and data is crucial. It’s what you have to do before you can do anything else. This is almost Step 0—you need something to work off of.

2. Aggregation: build dashboards relevant to your security needs as an organization.

Pull all the data together in dashboards! Refer to the point about efficiency.

Yes, Splunk can pull in all the data, and it can search it, but that’s not time efficient, especially when threat hunting. Time is of the essence. So, being able to aggregate data into dashboards and timed searches, saved searches, and other features Splunk offers makes it an extremely effective tool. You can click a button, log in and see all the failed logins on your system, potentially malicious sites people are going to, potentially malicious emails being received and anything bad happening on your network.

3. Correlation: take steps 1 and 2, and once you find something going wrong, perform a search to find out if it’s happening anywhere else.

If you see one incident, you need to drill down, solve it and find out if it’s happening anywhere else that you didn’t see it—that’s where you could be without Splunk. Plug in the IP address you saw trying to attack, correlate all different events you’re getting into Splunk and paint a bigger picture of an incident.

Splunk offers a number of correlation searches they’ve configured based on common threats and intelligence gathered through looking at logs (that’s the backbone of what Splunk is). Using those searches they’ve started with, you can adjust them and enhance them to make them more directed toward your personal environment, but it’s good to remember that there are ideas for you to work off of so you can see what people have done in the past.

Also note that Splunk is supported by a huge community, and there are always millions of people who have the same issue you might be running into.

Don’t Forget About Remediation!

If I do say so myself, these are three great tips, but let’s talk remediation—consider it a sort of tip 3.5. There are basic things you can do, such as, if you find something that’s an anomaly, but it keeps happening, set a script to remediate with an alert. This way, you can take the hands off the user who would typically have to carry out remediation, and Splunk can start to notify and make adjustments to your environment based on things you learned and tell it to do for future instances, like when other threats that emerge.

Also, keep in mind, once you’ve established something as a known, true positive incident, you can set up alerts based around it, so if anything like it happens again, in an instant, Splunk will let you know so that you can just go in there and chop it off right there. Like you would if you were battling Medusa.

Splunk is a highly effective, easy tool to use in your threat hunting ventures, ultimately resulting in a stronger overall security posture. Want to know more about how to make the most out of Splunk? We’ve got you covered. Connect with an August Schell Splunk specialist today, or call us at (301)-838-9470.