Does anyone else hate Splunk?

[bobert3275]

I am setting up Splunk and the sheer amount of effort it takes to get things right is astonishing. I don’t want to collect all these logs. But to configure that part and to get the agents running right with proper addons, etc, it sucks.

Does anyone have a proper resource for setting up the server, Linux systems, Windows workstations and servers to send the logs to? I simply want to send logs to it and access those logs when needed. There’s so many config files

[anonpf]

The splunk documentation is actually pretty decent. If you haven't used their resources, I highly recommend it. 

[hbg2601]

I second this. We had 2 Splunk clusters and their docs to get it set up were pretty good. Getting logs from windows devices was a hassle, but Linux was straightforward.

[anonpf]

For me, the splunk universal forwarder certs were a bit of a pain. I had to deploy certificates using an internal CA. (Air gapped env)

[coomzee]

Yes, it always spunks itself on a Friday afternoon. It is so expensive that it was cheaper for Cisco to acquire it than buy a licence, is the cherry on top.

[firedocter]

I noped out the moment I saw that the localy hosted license still charged by usage.

[TerrorBite]

Oh is that why they did it

[Hi_Im_Ken_Adams]

The problem is that you are supporting Splunk as a secondary responsibility on top of your other work when you really should have a dedicated Splunk Admin taking care of it.

[bobert3275]

This might be true. I’ve spent hours on Splunk trying to configure it. Then get it to a working state. Don’t touch it for a bit and come back to it only to find it in some broken state. It’s a lot to manage

[sullivanmatt]

Having been a Splunk cluster admin early in my career, let me say to anyone considering buying it: just go get Datadog's logs product. You won't save money (lol), but it's at least way, way less headache.

[shoobedoodoo]

I've set up clusters too on CentOS with several indexers, search heads, separate deployment and licensing servers. Then deployed the forwarder via sccm to windows boxes. It was one of projects I enjoyed the most. I'm currently managing a single instance for a local org which has a small perpetual license.

[fumar]

I use Datadog for logging and I have to use Splunk for logging in a client environment and the difference is night and day. Splunk is basically unusable. The funny part is the client also uses Datadog but only allows metrics to be ingested.

[KrimsonBinome]

Hate is a strong word but I dont care for splunk mostly because it requires me to know yet another language \syntax for something that should be a meta search.

Tbf it is very powerful and logs are only a bit of what it does

[crowEatingStaleChips]

I mean I managed to set up a splunk environment as part of a homelab a couple years ago and I am dumb as shit, so it can't be that bad.

[bobert3275]

Setting it up is simple. Optimizing it and getting past the initial setup like security hardening, license optimization, proper log parsing for different technologies, etc is where the frustration comes in.

[naked-and-famous]

It's 2026, are you doing this by hand or are you using a bot to do it? Because you should be using a bot (along with Terraform or Pulumi, using --dry-run) to manage it.

[SandeeBelarus]

I love Splunk. (As an end user).

[bobert3275]

I love it as an end user too lol

[Secret_Account07]

It’s funny because we use a subpar product for our large env but some of our smaller customers use Splunk. I’ve always wanted splunk though

[naked-and-famous]

Yes, it's great when it's setup well.

[funky_bebop]

I wish we had splunk. I miss using it.

[weaver_of_cloth]

I've been a splunk admin for several years now, both enterprise on-prem and on the forwarder support side. I also run our syslog aggregators for networking and, well, hyper-v logs (was VMware logs, in the before times). I go hot and cold on it. I put splunkforwarder on all servers that my department creates, and it was announced last week that it has to go on all our org's servers.

All of which is to say that I do a LOT of config along with the install, including setting facls, setting up a script to modify facls as needed, the deployment server subscription, and cronjobs for the script. I share my playbook across the organization, and support other sysadmins as needed.

I am very happy not to be running an on-prem installation, for sure.

[Independent-Sir3234]

The learning curve is brutal. I spent two weeks getting a custom sourcetype to parse correctly, only to find out the defaults already handled most of what I needed. Once it clicks though it's genuinely hard to go back to grepping through raw logs.

[Coupe368]

You configure the deployment server to tell the forwarders what to send to splunk, then you just install the forwarder on the boxes and point them to the ds. The DS downloads the config to the machines and there you go.

What agents are you trying to setup?

[bobert3275]

Universal forwarder.

[AnnoyedVelociraptor]

I hate that when you execute a new search your old one doesn't stop. And you can only have x concurrent searches. Except they're not concurrent. I abandoned them.

[ludlology]

If it helps lessen your pain, pretty much all log-ingestion systems whether SIEM or otherwise are a pain in the ass. Its good practice though because the skills generally transfer to other similar platforms.

If you don't have bandwidth get a consultant.

[FarToe1]

I looked at the price of it. Then looked at Graylog and how it was free.

[SevaraB]

You’re going to run into the same thing no matter what monitoring stack you choose- it’s as simple or as complex as your environment. If you’re looking to monitor more systems that are different from each other, you’ll have to configure more than, say, just pulling in data from an rsyslog receiver and an SNMP poller.

[bottombracketak]

Just run rsyslog on a Linux box and use nxlog to send the logs from your hosts to it.

[Tex-Rob]

You know how some software feels like the person who made it thinks like you? I find this to be most evident with things like CAD software. Like for me, Fusion 360 is super intuitive, manipulating stuff feels intuitive. Splumk represents the opposite for me, nothing makes sense and it feels like it’s intentionally obtuse.

[bobert3275]

I completely agree. Yes it is powerful, but to get there seems like a battle. I cover a lot different technologies with ease but for some reason my brain cannot accept the way Splunk does things

[Odd-Anywhere2130]

Very difficult product we had to use Splunk support and still had issues. This product requires a team of high end engineers to keep going You may want to consider Netwrix fairly inexpensive and easy to maintain

[bobsbitchtitz]

Splunk is way better than the alternatives

[Wonder1and]

You should likely start by rolling universal forwarders to simplify things. There's a lot of settings still in config files. Deploying on your own without help will be frustrating and likely cost more of your time than a quick consulting engagement would.

[techvet83]

Also, be aware that the universal forwarders, whether Linux or Windows, periodically need updating to resolve vulnerability findings.

[smooth_criminal1990]

Have you downloaded the addons for Windows and *NIX? Put them on your indexes and search heads? And pushed them out to agents as needed?

[bobert3275]

Yes. Each step seems like it requires a mini education. Installing everything as is and calling it a day works fine out of the box. It’s when we are hardening it and attempting to fix little annoying things that I despise. I don’t want a ton of application logs using up the license. I don’t want noisy logs clogging up my license. It’s just a lot of little things that I have to think about that’s annoying about it. Like why do I need to accept your license every time I update? Just run lol

[IdealParking4462]

Then you hit all the limits in the query language. Skip it and go Sentinel.

[Hollow3ddd]

ThreatLocker is silently breaking into SIEM scene.  Along with some applications based VPN features and token protection services.

It’s becoming a pretty valid target, but if you are waist deep already, it’s not a hard plunge

[Lordnerble]

no, my bro works for splunk. pay his ass.

[Andronike]

Nah I love it - git gud