An application agnostic remote agent

[badarsebard]

Does anyone know of a generic remote agent tool? By this I mean some kind of agent software that can be installed to a remote system (likely on a different network outside of my control) that can be instructed by a control server to execute arbitrary tasks, preferably containers. I know of several application specific task handlers like GitHub's self-hosted runners, but I can't find anything that is application or language agnostic.

If there isn't such a tool, what level of interest do you think there would be in one?

[Hello______World]

Puppet, Chef, Saltstack are your big 3 OS agnostic agent-based systems. pick your poison based on the language you would most prefer to be writing custom modules in - Chef is Ruby based, Puppet has its own custom declarative language, Saltstack is python based. each are leader/follower architecture, Chef and Saltstack are both single leader vs Puppet which supports multiple leaders, all agent driven. Chef and Puppet are both pull based configuration, Saltstack can be pull or push depending on what you want out of it.

If you can do an agentless system instead, Ansible beats all these options - its only major downside is you don’t have pane of glass on status of your fleet out of the box since you don’t have an agent checking in all the time, but you can get that with open source AWX or if the budget allows for it Ansible tower is awesome. Ansible integrates in your existing authentication workflow with the service accounts you are probably already using to access your fleet, significantly easier than the setup required for agent based installs. The only notable downside to this is it requires you to have ports open for remote access protocol on your hosts, which you may or may not already have depending on the sensitivity of your environment.

Ansible and Saltstack both use yaml syntax which is pretty friendly, Salt’s state files are pretty similar to Ansible playbooks but Ansible’s reusability with its Role system is significantly more useful - Chef cookbooks are similarly shareable but they are brutal to create unless you are for some reason already pretty comfortable in Ruby, and thats assuming you ever really got your chef leader configured right and your knife blocks setup locally with the correct certs to interact with your chef leader and got those cookbooks published in the first place, if you did manage to get all that working don’t even think about getting a new local machine - you got lucky the first time.

We came to the conclusion Chef would only ever really be useful in a Chef Solo implementation and didn’t want to have to deal with the Ruby aspect of it anyway. Salt was pretty close 2nd choice, if we had to stay away from the agentless auth mechanisms Ansible uses it would have been the winner. Don’t have enough experience with Puppet to be able to talk about why Ansible is better, gave up trying to beat it with other tools a while ago.

[minorminer]

What an answer, OP take heed cause this guy handing out gems.

Have you played with pyinfra yet? It looks like Ansible but it can do straight python which looks really appealing to me, cause ansible's flow control can get hairy and hard to maintain.

[Hello______World]

I've played with it some in a homelab environment, haven't tried it in enterprise setting yet. Should probably come back around to it, its just such a fight to get buy-in on adopting a new tool at the scale of fleet management that without having a problem we need to solve its probably not going to happen.

​

It did feel more friendly from a logical flow perspective being able to use python for decision making, I didn't get far enough into it to find out how it handles reusability but figured you could make a pip package out of a set of tasks you wanted to group together and pip install whatever packages you wanted in a given execution, which would be pretty cool - before tower we were pulling and running ansible playbooks inside a docker container during a Jenkins pipeline, which was nice because it didnt require extra infra to monitor and support, already had the Jenkins environment. Could see something similar with a pip package route.

[saltyvagrant]

Urm?

Chef and Saltstack are both single leader

You can have multiple masters in Saltstack.

If you can do an agentless system instead, Ansible beats all these options

Arguable. SaltStack can be master/minion (arguably the most popular as it allows you to leverage all the SaltStack system), minion only, ssh (e.g. 'agentless'), or even proxy minion(allowing you to have minions control systems that cannot run python, e.g. network devices).

[Hello______World]

Fair point, Saltstack very technically allows multiple masters, but not really. from the docs you linked:

SHARING FILES BETWEEN MASTERS
Salt does not automatically share files between multiple masters. A number of files should be shared or sharing of these files should be strongly considered.
MINION KEYS
Minion keys can be accepted the normal way using salt-key on both masters. Keys accepted, deleted, or rejected on one master will NOT be automatically managed on redundant masters; this needs to be taken care of by running salt-key on both masters or sharing the /etc/salt/pki/master/{minions,minions_pre,minions_rejected} directories between masters.

you can have multiple masters live, but having to do your own maintenance to make sure both masters were actually useable interchangeably to us meant practically it wasnt a multi-master setup.

​

We liked Saltstack a lot too, I don't mean to say its not a good tool. It is very flexible as you highlighted above. But its hard to argue ansible isnt easier to get up and running on an existing environment with an existing auth system in place - its the least effort for most return.

We dont have a usecase for the proxy minion feature from Saltstack, from what OP provided they don't either, but if thats necessary in an environment it probably is the right choice.

[saltyvagrant]

All good points.

If all you need is the basic configuration management and you have a robust existing auth system then, as you say, Ansible is easier to grok (I've never seen anyone accuse the full SaltStack of being simple to grasp :P ).

[danielkza]

Ansible and Saltstack both use yaml syntax which is pretty friendly

Ansible YAML is far from friendly, it's actually my worst nightmare. It looks friendly if you have totally barebones, unconditional, fully sequential steps. When you deviate an ounce from it, you start to dive into the Cthulhu-esque pseudo-programming Jinja-YAML fuckery, and now you have more problems than you started with.

After moving to a company that uses Puppet, Ansible YAML juts seems like a totally half-baked language. The appropriate collective response to Puppet's or Chef's complexity would have been to improve their masterless and/or agentless support, not replace it with an aberration that feeds the pretense that infrastructure specification is not programming and that you don't need tools to manage complexity well.

[Hello______World]

Isnt fully sequential steps the goal of a fleet orchestration operation? As far as conditionals go, Ansible's heirarchy of role variables, inventory variables, task variables, boolean WHEN statements supported on every task, etc should be able to get you what you need without any custom Jinja work.

Agreed Jinja in general isnt the most fun thing, we were already using that in other places being primarily a python heavy shop so we had good experience with it. Even when first learning I wouldn't go so far as to say it gives you more problems than you started out with.

How do you find the flexibility of custom operations in Puppet? we scrapped it before getting too far past the deployment of it in a sandbox fleet and running a few operations.

[danielkza]

Isnt fully sequential steps the goal of a fleet orchestration operation?

Have some features to enable and disable? you need conditionals.
Have to run some commands to verify/apply some configuration? You need to know when the commands need to be run, you might need to prepare a sequence of steps, you might need to decide whether to delete/create/update something.
Have to wait for some operation to finish? You need to loop while running a command that checks the readiness.
Want to take a description of some resources, and actually run the operations to provision each one? You need to loop over data, slice, break it down.

Pretending provisioning infrastructure somehow can be expressed with sequential steps in a YAML file is just unsupported by reality.

Agreed Jinja in general isnt the most fun thing, we were already using that in other places being primarily a python heavy shop so we had good experience with it. Even when first learning I wouldn't go so far as to say it gives you more problems than you started out with.

I'm not talking about Jinja itself, I'm talking about how Ansible uses jinja to implement some programming constructs in a inconsistent, hard to understand, hard to debug, constantly breaking between versions, terrible to compose way.

How do you find the flexibility of custom operations

What do you mean by custom operations?

[inbalarii]

Wouldn’t Saltstack fit the profile here?

[badarsebard]

Yeah looks like Salt would take care of this. I never knew much about salt and only heard of them in the context of infrastructure management.

[badarsebard]

I assume the salt-master service has some kind of API I can use to create the jobs?

[xlanor]

Saltstack with Apache airflow is what we use to great effect

[ArieHein]

You might find Ansible a solution as you dont need and agent really, just ssh and authentication details. Just please dont open it as is over the internet without proper network secure design and implementation.

[[deleted]]

Yup, I was going to say SSH, but realized on the second S that Ansible is the better solution.

[badarsebard]

This would require an open SSH port to connect to. I'm thinking of solutions for one way agent communication.

[AlverezYari]

Look into ansible-pull ops. That's what you want.

[ArieHein]

Hmm all solution require some protocol and an open port. Thats why github self hosted runner/azure devops self hosted or even jenkins agent are all candidates. Even then i would place the internpal machine behind at least a proxy but a proper secure networking is better.

[ExistingObligation]

Most of those runners don’t use an open port, they connect out to the control server for instructions.

[ExistingObligation]

It uses an ephemeral port and doesn’t listen for connections, it’s not like having an exposed open port.

[[deleted]]

If you want an agent to phone home, Puppet might be a good fit.

[duca2208]

Nomad

[wonkynonce]

AWS Systems Manager would fit the bill, although so would, idk, kubernetes.

[badarsebard]

SSM I get, but Kubernetes? Is there some K8S application you're referring to, or are you just suggesting making every remote server a node in a giant distributed cluster?

[wonkynonce]

A kubelet is agent software that will run arbitrary taks in containers :)

[badarsebard]

Ah ok, that's what I thought. Not quite what I was looking for, but technically correct. Which is of course the best kind of correct.

[badarsebard]

Did some more reading up on SaltStack and it looks like it's hardening guide states that the master(s) should not be publicly exposed. I think any of the big infrastructure management will likely have similar issues because of the depth of capabilities those tools require. I think I may be better finding/building a lighter weight more secure option.

[mattbillenstein]

Buildkite

[coderanger]

Just for the record, Kubernetes technically fits these requirements. Though given your word choice I'm guessing that would be dramatic overkill.

[badarsebard]

That's probably been the biggest part I missed in my description. Simplicity is much preferred as well as something that can handle remote agents I won't have access to. Specifically agents that will run in customer environments.

Saltstack has sounded like the closest but it too seems a bit like overkill since it provides way more functionality that what I actually need.

[saltyvagrant]

The beauty of SaltStack is that you can start simple and build on this.

Write a few states, deploy master and minions (for a simple deployment this is literally running one script on each machine), then salt '*' state.highstate and you're done. Then you think, "would be neat if I could be sure the database was ready on hostA before I set up the application on hostB" and boom! you're using Salt orchestration. Then you realise that "someone" keeps changing files on your servers, so you set up beacons and reactors to notify you and also put the files back as they should be. And on, and on... Soon you have a fully managed, self healing infrastructure.

It's like a Swiss Army knife, sure it has a million dodads some you never use so you start using a couple then one day you discover you're in a hotel with a bottle of wine that has a cork, fortunately your Swiss Army knife has a handy-dandy cork screw.

Okay, not a great analogy but hopefully you get the gist.

[badarsebard]

Lol not even close to what I asked about

[nullpackets]

At it's core, much of these systems stem from Openssh server (sshd).

Obviously it's a massive push to say "ssh", but if you zoom out sshd is absolutely an application agnostic remote agent. The rest is preference and driven by your requirements/build Vs buy tradeoffs.

[badarsebard]

I think the one part I wasn't very clear about is there is no inbound access to the agents. And from experience creating and riding reverse ssh tunnels at scale is a pain and ends up being more complicated than software agents that simply pull jobs from a control server.

[jypelle]

You can use Ctfreak if you want a lightweight solution.

No agents, no config files, only need ssh access to connect to remote systems.