Questions about IOC ingestion and at-scale comparison engine for enterprise SIEM using MISP

General
, , ,
1

I see a ton of companies just selling IOC feeds and there are quite a few free ones, but I rarely see anyone talking about how to actually use those feeds. Sure there are some that you can just feed into your firewall and forget about for some of the paid feeds, but what I’m talking about is actually using them to generate detections and prompt investigations by your SOC.

Let’s assume you’ve got a MISP instance and you’re using Splunk as your SIEM. MISP is configured to ingest feeds and you’re tagging those feeds by the feed name (malwarebazaar or whatever) and your goal is to perform two types of searches in Splunk:

  1. “live” comparisons where you’re comparing the IOCs to recent data to catch active infections.

  2. “retroactive” comparisons where you’re looking for indicators in historical data to see if you’ve missed anything

Both of these scenarios have some pretty interesting technical hurdles to get through, but I’m really only struggling with the live search.

For live search first and foremost, maybe Splunk isn’t the best tool for this since by nature of it being a SIEM the data is stale when it gets there. But for the sake of us not implementing some pre-ingestion time comparison tool, let’s just say you’re ingesting a MISP feed every 24 hours and then running that against 24 hours of Splunk data.

Because the data in Splunk is fresh, you want to compare a large list of IOCs against it. Essentially you want every IOC that isn’t relevant anymore (maybe leveraging MISP’s decay model?). These are expensive searches AND expensive fetches from MISP’s API. If you wanted to compare 90 days of indicators from MISP and you’re using PyMISP w/ something like:

misp.search(tags=[‘malwarebazaar’], timestamp=90d)

this will take forever for MISP to respond and you’ll probably time out. So the alternative is maybe to just keep your own local storage but then you lose the functionality of MISP’s decay. You could just do that misp.search and then parse and drop them into a lookup file that you ship to Splunk… but should you just do it once every 24 hours and append to that list with a the older than 90 day objects falling off as you write it? This feels hacky and I don’t like it. What I’d really want is for MISP to just nicely return those objects and use the exclude_decayed and enforce_warninglist params and let MISP do what it does best. Maybe the timeout problems are on our MISP server side but it just seems to not really be the best way to handle this.

How are other SOCs/Intel teams handling this? We’ve already currated our feeds to only use feeds that we think are high quality but the number of events is still very large and I don’t like being limited by size here because there are other larger feeds out there that I’d like to use.

Once we’ve got the lookup lists or KV stores in Splunk we are then restricted by memory again because there is a maxium number of entries that Splunk will allow to be stored in memory during a search - but this is an easier problem to wrap my head around. I have already planned for having indicator specific and feed specific searches to keep the numbers smaller.

2

Oh wow, I’ve been trying to find time all day to write this reply.

I think the first thing we need to establish is that considering the firehose of your IOCs as a single lookup table is a flat-out error. It shouldn’t be treated that way, and the entire point of MISP’s ridiculous metadata system is to allow you to be more selective.

Put another way, there’s no reason that Phishing atomics should be checked against outbound network traffic. That’s silly. And conversely, your email senders probably don’t need to be checked against C2 IPs. Using MITRE tags or other metadata to create smaller groups is a useful first step.

Also, and I know you probably don’t want to hear this, but MISP isn’t really complete without some custom Python. In particular, using the feed generator template that comes in the box can help you produce the subsets you need for more selective cross-referencing. That’s part one.

Part two is yes, indeed, get more agressive with your decay rules. In fact, get more choosy with your confidence scores and admiralty. Especially if you are using OTX or VirusTotal OSINT, a bunch of that stuff is just noise. You can probably use smaller, higher-quality feed sets and be safe. For example: OTX allows subscriptions to specific contributors. Be bold enough to use smaller lists.

Part three is considering hunting vs. alerting. Your objective should not be, imo, to be building for manual hunting, which I agree is agonizing. Your objective should be to build two streams of data that, if there’s a match on either side, generates an alert about a hit. That is a fundamentally different problem than taking, like, 90 days of Splunk data and comparing it to your entire IOC database.

Instead, as events enter Splunk, batch the events and query against the appropriate indicator subset defined above that hasn’t aged out. Matches produce a new entry in a Splunk index like cti_hits. That is the primary value of joining these datasets. For the manual hunting you’re describing, it’s almost always ad-hoc bidirectional enrichment. So if you’re in MISP, you can ask if indicator X was seen in Splunk, and vice versa.

FWIW, I think OpenCTI—even the Community Edition—is a much better intermediary for this than either Splunk or MISP.

I do think there’s room for improvement in the middleware here though. Both Splunk and MISP’s architecture make this uniquely expensive.

1 Like
3

I think the first thing we need to establish is that considering the firehose of your IOCs as a single lookup table is a flat-out error.

So I think I misrepresented what I was saying here - that is not creating a single lookup table. We are creating a lookup table per feed. Even still this can be pretty massive as some feeds can just drop 10k entries in a 24h period.

We are weeding out lists and planning on using smaller lists but smaller != fidelity. We have a specific IOC list that is reputable and if we somehow ingested all of it we’d have millions of entries.

I think the ultimate solution is we need something doing live matching prior to ingestion into the SIEM like cribl or Apache Pulsar that can do streaming comparisons.

For your 3rd point maybe that is just something we need to consider. Is hunting for matches on IOCs an activity that the SOC should be involved in (search runs, results trigger a detection) or is this something that a dedicated threat hunt should perform (Assign hypothesis or thrunt topic, run search, analyze results, trigger TPs as detections).

I don’t really like that 2nd option because it is not really optimized to how we do things but ultimately my point here is to understand how others are doing things too because we are likely doing some things wrong.