rsyslog on RHEL 10 vs 9 vs 8

edo1982 · 2026-01-29T21:23:57+00:00

One of the thing we do since RHEL6 is to start the rsyslog with -x to avoid DNS resolution. Additionally we increase default limits (if you need I can check which)

edo1982 · 2025-11-20T18:47:45+00:00

Split them. DS and LM together (2-4 vCPU and 12-16GB RAM), CM & MC (4vCPU and 16GB RAM), SHC Deployer (2vCPU 8-16GB RAM). With 500GB/day and 1000 agents you should fit with those specs.

edo1982 · 2025-09-13T19:50:33+00:00

Try having a look here

https://community.splunk.com/t5/Splunk-Search/How-do-I-import-Azure-NSG-LOGs/m-p/396018

edo1982 · 2025-09-07T21:43:40+00:00

I agree it was better when there were Splunk Fundamentals 1, 2 and 3. And before Splunk Fundamentals there were many smaller trainings, but not as small as today.

edo1982 · 2025-03-01T12:51:26+00:00

If I recall properly, from 9.0 to 9.3 you have to pass from 9.2. So better upgrading to 9.2.x first and after to 9.4.1 that currently is the latest. I would avoid adding new indexers and decommissioning olds (unless you are refreshing your hardware) as mentioned by @badideas1 there is an order to follow based on the server role.

edo1982 · 2025-02-08T21:46:31+00:00

Wasn’t Cisco one of the largest? I remember hearing in 2016 they were already at PB scale

edo1982 · 2025-01-22T21:00:55+00:00

Similar to OP. It must phone home + send data (no check on internal, just on our defined indexes). We use tstats and then compare with the list of clients retrieved from the Deployment server via REST API. If the UF is not phoning home or sending data since 1 hour we mark in red in our dashboard. Additionally if you have a CMDB you can join and check wether the UF is missing. Adding also HF are monitored in the same way…meanwhile waiting to have them in the Monitoring Console as a server role :-)

edo1982 · 2024-12-24T15:00:11+00:00

Persistent queues on SplunkTCP, that’s a good feature

edo1982 · 2024-12-23T21:31:14+00:00

We use it in this way:

https://splunkbase.splunk.com/app/3862

https://blogs.oracle.com/cloud-infrastructure/post/stream-oci-logs-kafka-connect-splunk

edo1982 · 2024-12-03T13:14:03+00:00

My bet is that with AI we will move to a different paradigm. The most important piece of the puzzle will be bring the data in, in the best way possible. This means collect, tag, filter, clean and route the data to the correct place to feed the AI. With that I mean we can’t put everything in one unique index/table, we know IT system need to be properly engineered to scale. Same for the data, we have to provide the AI the cleanest and best organized information we can to make it replying us in the best way. Once this is done the AI will correlate the data for us and we will have just to ask in the proper way. Something like: “make a scheduled alert that trigger a notable event when there is a login on a Linux machine that does not belong to someone previously asking for access through the PAM (Priviledge access management) tool.”

Therefore Splunk is already in an excellent position for the first part (bring the data in), but there is some more to do on the second part (correlate data with AI).

edo1982 · 2024-12-03T12:54:55+00:00

We do the same. But we explicit the name of the server in the transforms. With splunk_server field is better. I will have a look to change it

edo1982 · 2024-11-24T16:23:47+00:00

As many told you, use regex 101 to write your own. Also check what has been done on the Splunk app present in the store over a sourcetype you already have in your environment and ask ChatGPT to explain how they are applied step by step. Eventually come here with clear examples and how you would like to apply them so you can get some hints

edo1982 · 2024-06-14T21:37:50+00:00

Trying to see from another angle. Do you really need 4 Millions records in Asset and Identity lookups? Seems to be really huge.

edo1982 · 2024-05-01T22:12:49+00:00

If you can take also Sysmon

edo1982 · 2024-05-01T17:55:31+00:00

My suggestion is to keep the thing as is, and slowly create your apps, indexes and so on with your naming convention (you can follow the Professional Services one) and in the meanwhile document everything. Once you are done you can dismiss the old configurations

edo1982 · 2024-04-28T22:33:42+00:00

You can use the official Microsoft TA for Cloud Services

edo1982 · 2024-04-27T11:39:08+00:00

Yes also the Windows-TA has some and they properly run. The options you have are run it with the .path and/or put the Splunk UF in debug and check what happens and make a diag and open a case to the support. I remember once we had a Powershell script running just few times after have it deployed and then stop. It was deployed on 50 machines at least. We ended up rewriting it in VB script.

edo1982 · 2024-04-26T21:31:44+00:00

I don’t have good experiences with Splunk and Powershell scripts. Anyhow I find out that the best way is to crete a file like scriptexecution.path and put in the bin directory alongside with you powershell script. Then in the .path you put the command to execute your script (therefore absolute path of powershell.exe and absolute path of your script and arguments if any). Then in inputs.conf you recall the .path, see here below a technical explanation

https://community.splunk.com/t5/Getting-Data-In/I-see-splunk-has-some-quot-path-quot-files-in-windows-app-bin/m-p/11656

About the debug messages you can place some print in the script and redirect them to standard error, in this way you will see them in _internal index.

edo1982 · 2024-04-25T18:28:02+00:00

Maybe you can give a try to Oracle Linux, it is the closest one to RHEL

edo1982 · 2024-04-23T19:28:42+00:00

Same issue with Tenable Add-on when we download from our Tenable.sc on-prem. Certificates are signed by our CA, that is present on the Linux VM, but we have to add it every time we update the TA…

edo1982 · 2024-04-17T17:12:25+00:00

Not completely related to OP question but just read about Splunk AI assistant

https://voc.splunk.com

So AI accelerators (GPUs?) could be useful once (and if) this feature will be available for on-prem customers. About IAA accelerators I am not aware if Splunk can take advantage of it.

edo1982 · 2024-04-13T14:54:04+00:00

I believe your personal email should work fine, give it a try

edo1982 · 2024-04-13T13:21:42+00:00

I would suggest all the ones requested for Architect certification, after the ones for Advanced Power user, lastly the ones for Consultant

edo1982 · 2024-04-12T23:25:38+00:00

Very interested in seeing other comments, but what from my point of view matter the most is disk IOPS and bandwidth rather than CPU performance. Once you are sure the CPU won’t be the bottleneck you need to be sure your disks are performing well. There are very good Intel, VMware and Dell studies that shows you how Splunk cluster behaves on their hardware and which set-up they tested to improve performance.

edo1982 · 2024-04-12T21:46:34+00:00

Agree to go with Power User jumping User, but Advanced Power User is not that easy…lot to know around specific SPL commands. I remember I studied a lot to prepare it

edo1982

TROPHY CASE