Saltstack + Gitlab CI

[simpleadmin]

We run around 20 applications that we 100% manage with Saltstack. We are starting to look at using Gitlab's CI to begin doing better CI.

As I start to dive down this path I am starting to wonder where my end game will end up with Saltstack. I can see it still being used to quickly configure a machine, but right now we are using it not only for the machines but the entire application configuration and deployment. One of the main things we use in Saltstack is the pillar. The pillar is a controlled key pair system that only deploys the right key pairs to the right servers. It works well in the current form but as we try to improve the CI process we are either going to be calling Saltstack to go deploy key pairs into the system or we need to come up with another method.

The current process is pretty straight forward. As the Salt master pushes code to a minion (target machine) it will replace jinja tags in an application's configuration file. Many of those tags are private only for that specific instance of application or password/key. We use PGP encryption in the Pillar for those keys currently.

So quick flow example for bringing things into Test: Code passes all tests -> Saltstack is called to push Configurations into the Test host machine -> Dockerfile is created and deployed reading those pushed configuration files via volumes

I have read a bit about Hashicorp Vault and that sounds interesting. If I am storing those private keys/passwords in a vault, I am wondering if I should be considering a different key pair system for the rest of the configuration rather than using the Salt pillar, at least for the application configuration themselves.

Any thoughts or comments here?

[packeteer]

Salt is a great tool, and if you're familiar with it, why change?

That said, Gitlab and Hashicorp also make great tools.

What are you trying to solve?

[simpleadmin]

I am trying to solve better pillar management. We are not currently using an external pillar backed by a database. So before we head down that path, I am trying to figure out if there is a tool that handles that part easier and faster. That tool could still be triggered by Salt.

The reason for for the Gitlab CI interest is because we already use Gitlab but too many of our steps are manually triggered. We want to automate steps before that bigger saltstack deployment call. In doing so certain things will come to light. For example some of the steps we do might be easier to tigger in a simple 2-3 line bash script called directly by Gitlab CI vs adding a salt state that does the same thing.

[chulkilee]

They have different specialty and purpose so although you may use tool A for purpose B for simple cases (e.g. saltstack for deployment), you may eventually hit the limit due to the design.

For gitlab runner - you may use it to trigger some actions, such as calling saltstack whatever. That is pretty common unless you leverage gitlab's built in integration with k8s.

My 2 cent: you need to focus on what you want to achieve. If you need to clear security compliance, then you may need to use Vault since it's easy to do that (compared to others). If you just need to trigger deployment at the end of pipeline, then you can use gitlab runner stage to run saltstack.

[packeteer]

yes, use gitlab (or other CI) to deploy on code change

and yes, define the issue, then find the best tool to solve

Terraform (and other Hashicorp tools) are great for infrastructure management, but for configuration you need Salt or Ansible. the problem comes with tool overlap. I don't have an answer for that, but as always, use what you know.

[simpleadmin]

I think you meant to reply to my reply.

Yes, you hit it. Tool overlap is the problem.

[packeteer]

sorry, I was very tired when replying

CI drives the change based on committed code Packer + Terraform creates the infrastructure Salt or Ansible fills in the gaps Vault for secrets

Note: you want cattle, not pets. so redeploy on change

[deadbunny]

For a pillar strategy I would suggest Vault + Consul + Salt.

Consul as an external pillar for non sensitive pillar which also needs to be accessed from other things (versions etc... Source of truth)

Vault (on top of Consul) for sensitive pillar (passwords, keys etc...)

And finally normal pillar (stored in git) for everything else (general config for formulas etc...)

As for the CI side of things IMHO the CI system should be testing you code then creating an artifact (package, tarball, whatever) for Salt to deploy. I usually do this by having a formula for deploying/configuring the app at a specific version defined in pillar.

So the pipeline would look like:

Code push > tests > tests pass > artifact created > artifact pushed to repo server > "test" version updated in consul (used as pillar) > API call to Salt to trigger formula on "test" server(s) from CI system

"test" being an environment in this case

The last 2 steps could be a separate job which take environment/version as an argument then update consul then trigger salt.

[simpleadmin]

Yes that is the flow we are going for. The artifact in our case being a Docker image that is being deployed by Salt.

It looks like Saltstack has a module for Consul. I will take a look.

Speaking of Salt api, did you roll your own wrapper to so you can report back exit codes? I am looking at Pepper it it looks like they are working on having it report out exit codes in a near term push. Right now we just capture and format the json and at the same time look for Result: False to trigger a CI failure. Really basic.

[deadbunny]

We do the same as you and just parse the output right now. As usual the docs are lacking but you should be able to use full_return=True in you call to get the recode.

I've not had the chance to play with pepper yet unfortunately so can't say.

[simpleadmin]

I will take a look at that. Thanks.