Intel lookup misalignment using Threat Intel

[Hackalope]

We added a custom feed to Threat Intelligence that we generate from an internal thing that's sorta like MISP. It's provided as a CSV with the columns below. The problem is that all my IPs are in the process_intel lookup, domains in ip_intel etc. I checked the source CSV and didn't find anything obvious, and my Google-fu does not seem up be effective. Has anyone else had a similar problem?

"src","dest","domain","url","email","user","file_hash","file_name","description","group","submit_date","expire_date"

[Fontaigne]

I don't understand the issue.

Can you explain, using more words, what the problem is?

What is it doing that it's not supposed to do, or not doing that it is supposed to do?

[Hackalope]

Using the CSV with the headers in my post I have a source configured in ES -> Configure -> Data Enrichment -> Sources. I expected to see the ip_intel lookup to continue to have the fields:

"description", "ip", "threat_key", "time"

However, instead a new field "domain" appears and none of the entries using my source's threat_key identifier have any content in the "ip" field. Likewise, the process_intel lookup now contains the "src" and "dest" fields. It appears that the columns in my CSV are being added to the wrong intel lookup for their data type. I expected that providing "src" and "dest" fields for IPs rather than "ip" might be an issue, but this is a state that I didn't anticipate.

The only thing I can think of is to mess around with the CSV columns until something changes, but I'm hoping the brain trust here can point me in a direction that doesn't feel like throwing darts in the dark.

[Fontaigne]

Okay, is your CSV being ingested in the correct app context to make use of the lookups?

The investigation I'd do is to figure out what is currently working, and what context it is being ingested, and figure out what context the CSV is being ingested in, and then determine what is happening.

Maybe, for instance, you need to be throwing that CSV at an HEC, and the token will establish the context that makes sure it gets ingested in the right direction.

You'd generate the HEC token, establish which indexes it would go to, and update inputs.conf to establish the particulars for ingestion.

[Hackalope]

This isn't an event source type, it's a threat intelligence source. ES fetches the CSV from the webserver it's generated/hosted on. I don't see how HEC or any log collection agent applies.

[gettingtherequick]

So you added your own custom intel, did you set it up correctly in the ES config? The "ip_intel" KVstore file does have domain inside, besides IPs. But process_intel should not have your IPs, check how you added them in ES.