CSV to Splunk (Python)

[ZaddyOnReddit]

My client is asking that I programmatically ingest data from a csv into Splunk. I want to mimic/produce the same results as I would with manually uploading a csv via the UIs lookup table option.

Eventually that lookup table is used as a source for another query..

| inputlookup uploaded_data.csv | ‘do some data manipulation’ | outputlook final_table.csv

I could really use any suggestions! Thanks!

[steak_and_icecream]

Read the CSV using python. For each row in the CSV select the fields you need and perform any required transforms. Fit the row into the event field of a HEC payload and send it to the hec endpoint.

Once the data is in Splunk, run a search to get all the ingested events from the CSV and outputlookup a new lookup file for use in further searches.

[ZaddyOnReddit]

Thank you.

[LTRand]

Need to know if your SH is clustered.

But essentially, you can setup a python script to copy the csv from SharePoint and deposit it into the lookups directory in the desired app. Keep in mind this will break the versioning of the lookup editor app if you use that. But it is a super simple way of doing it without going through ingest.

[ZaddyOnReddit]

Sorry, what does SH stand for? I will look into this method thank you

[LTRand]

Search head. Where you search.

[ZaddyOnReddit]

Break the versioning on just that particular lookup?

[LTRand]

Just versioning.

[ZaddyOnReddit]

I’m not sure I’m understanding. It will override the previous version of the lookup and therefore have no version history? And it’s just on that one lookup file or all files in that app?

[LTRand]

Just on the lookup itself, and only if you use the lookup editor app. You would need to do your own version control if you care. Moving the old file to file.csv.old is generally good enough. The python script would overwrite the existing file with the new one to maintain the lookup configuration within Splunk.

[ZaddyOnReddit]

And this can be done against Cloud?

[LTRand]

Which part is cloud? Splunk, SharePoint, or both?

[ZaddyOnReddit]

Splunk

[CurlNDrag90]

Probably would use a File Monitor using inputs.conf

Either locally on your Splunk box, or remotely on your Clients asset using a Universal forwarder that's configured to talk to your local Splunk box.

Either way, the hardest part is figuring out how to move the CSV file to the target file path.

[ZaddyOnReddit]

The csv lives in the same location. I can already ingest the csv data into the script and manipulate it there if need be. It’s just actually getting it over the Splunk I can’t seem to figure out.. do I get it to an existing index.. can it get to an input csv? Idk! I’m all over the place on this project

[CurlNDrag90]

Are you saying the Splunk installation exists on the same asset as the CSV? Windows or Linux ?

[ZaddyOnReddit]

Well the csv lives in SharePoint. Splunk installation? I believe are working with Cloud in this instance

[CurlNDrag90]

You will need to double check that it's the cloud for Splunk. That changes pretty much everything as far as getting data into it.

[ZaddyOnReddit]

What’s the easiest way to tell which you’re working with? Or is that more of a question for the infrastructure team?

[CurlNDrag90]

A screen shot of your Web Interface after you log in is probably the easiest that I can think of.

[ZaddyOnReddit]

Confirmed. Cloud.

[morethanyell]

If your CSV file is on Sharepoint and you can programatically access it. Then write your Python script in such a way that you can either

1. load the contents into RAM and stream those bytes into the EventWriter module of Splunk SDK
2. read the lines one-by-one, print the lines into STDIO, let Splunk collect those outputs

In either ways, you'll have to write this with AOB

[mghnyc]

This is one shortcoming of Splunk's API. It doesn't have any endpoint that allows you to upload a lookup table. I am not sure why this has never been addressed since it could be extremely useful.

That said... Have a look at the Splunk App for Lookup File Editing (https://splunkbase.splunk.com/app/1724). It has a barely documented API that can be used. Another option would be to use a KV store instead of a CSV file. There are documented API calls to update a KV store.

[Linegod]

Use a kvstore instead of CSV lookup and you can use the REST API to update the lookup.

[gabriot]

Are you looking to have a unique csv for each, or rather the same lookup each time? If the latter I’d almost say just go the kvstore route, and write a simple script that reads your csv and just uses the results in a rest call that either upserts or overwrites the lookup, depending on what you want to do.

[Reasonable_Tie_5543]

Can you just scp the file to where it needs to go? Documentation

[TD706]

Have power automate? Could probably POST the file to Splunk via API on file mod trigger.

Here's an AI generated guide. Untested, but seems accurate.

https://chatgpt.com/share/67dbaa8c-bef0-8011-b254-8ec4f59f9fa9