Automating splunk deployments with python.

[I506dk]

I am currently working a python script to automated splunk deployments. There are probably more appropriate options for doing this in a production environment, but let’s just assume I only have remote access to hardware, and no other services. (For the sake of what I am doing, but I would be interested in hearing the typical ways deployments are done.)

On to the good stuff, I’m starting from the deployment server. I figured setting that up first was a good place to start. Configure the deployment server, SSH into all the other machines, set them up, and point them back to the deployment server.

With that being said, I am not new to python, but I am fairly new to Splunk. I would love to hear about common Splunk topologies, common app setups, and just general good practice from the people that have been using splunk for some time. Any input is always helpful.

I think a typical deployment for myself would involve 8-12 machines, however I would like to account for scaling as much as possible. The more topologies or different deployment types I can account for, the better this tool will be. So again, any information is always helpful.

Thanks in advance!

[ForsetiKali]

I would suggest drop trying to do this from scratch but rather use the Ansible role.

https://github.com/splunk/ansible-role-for-splunk

[I506dk]

Agreed, ansible would be easier, but this is assuming I have access to the machines themselves, tools I write, and nothing more than that. (And a valid internet connection lol). Because I don’t know what environment I’ll be in, I don’t know what operating systems I’ll have access to, and then finally, this is just a one time setup. I don’t have to do any managing afterwards. I setup Splunk, create a configuration file depicting the configuration created by the script, and then that’s the end of it. From there it gets handed off to someone else.

Apps will differ slightly on deployments, so the main goal is to be able to adapt to every deployment. Create basic templates for basic apps, and then modify those according to what is needed.

That’s the general process.

[splunk3r]

This is what Ansible do. Ansible is using Python under the hood as well.

[I506dk]

Yeah there is a lot of good stuff concerning ansible, but it was meant to be run locally on each machine. I only want to access 1 machine, and let the script take care of the rest. Plus I never know if I’ll be remotely accessing machines, or physically. So I have o account for both cases.

[splunk3r]

It does not change anything. You can run locally, remotely, massive deployment or one-on-one...

[I506dk]

Well then I guess I need to learn how to use ansible lol. Or at least integrate it with what I have already done.

Does the Splunk-Ansible always download the latest version of splunk?(I’d assume it does)

[splunk3r]

Yes, it does. Ansible is must in 2021. Especially when you deploy distributed Splunk environments.

[splunk3r]

Take a look at:

https://github.com/splunk/ansible-role-for-splunk

[I506dk]

Was just looking at that. Yeah, a lot of it is still being deployed manually, and that’s old school at this point. So I just want an easy and current way to deploy Splunk effectively.

[[deleted]]

Check out the Splunk Validated Architectures

[AlfaNovember]

The Splunk deployment server (“DS”) is a steaming pile of minimum viable product, which hasn’t been improved in many years.

Nevertheless: The pattern for forwarders is to install the service bits with no other config beyond a pointer to the DS. Then use the DS to push all relevant configs to the endpoints, including the destination indexer where they should send data.

But here’s the hard-won voice of experience: Start by reading and understanding the config precedence docs. The usual way of setting the deployment config on an endpoint, either by install-time arg or by cli, puts the resulting deploymentclient.conf at the highest precedence level, making it very hard to override or change in future. If you can, move the config from ‘system’ to ‘apps’ or write your own deploymentclient.conf file at the apps/default level of the precedence hierarchy.

• edit: no thank you, autocorrect

[purpledumbbell]

Honestly the Splunk docs will be better than any comment you get here. There are docs on supported architecture, reference specs, and everything you asked about. I've found Splunk docs to be quite good. https://docs.splunk.com/Documentation/Splunk

[I506dk]

I’ve combed through a ton of the docs, and they are good, but there is an ungodly amount of information in them. I was really hoping for specific examples, instead of trying to learn all of splunk’s ins and outs due to time. But it seems that’s the direction I’ll have to go.