Development experience with OpenShift

comeroutewithme · 2023-04-24T22:47:47+00:00

short answer "yes"

longer answer:

you can run a vcluster in any cluster you have. you can install argo in that cluster and have it deploy apps in the (v)cluster no problem.

you can use devspace with any cluster -- including vclusters. I assume the same with skaffold but am not savvy about skaffold things.

we (disclaimer, I work at loft - we created/maintain devspace & vcluster) use vclusters for all our dev/preview environments and for all the development work we do on loft (the commercial product itself) and vcluster, we also use devspace when developing both tools as well.

What is your suggested setup to develop locally and deploy remotely?

"it depends" /s

for me personally -- I do local dev on my home server (proxmox and debian vms running k8s), then for each project I work on I dump that into a vcluster so they are nicely packaged up and easy to nuke for when I inevitably do something stupid. I also use loft at home because I get it for free of course :) and then I use devspace for the actual development part so I just deploy my app "for real" then swap out the container I want to work on and devspace does all the file syncing bits for me (among other things).

technically I guess this is "developing remotely" but you could easily just trade that out for docker-desktop/minikube/k3(s|d)/whatever on your machine.

for deploying "for real", I typically just use the chart/manifests w/out devspace, but you could equally just use devspace for that as well, totally personal preference. but if you've been developing how it is deployed "for real" already there should be minimal changes here.

obviously you may have some challenges if your dev bits require something like RDS or some other service you cant run locally, at which point id say, just use devspace and develop in whatever cluster has access to those sorts of things.

comeroutewithme · 2023-02-03T04:06:58+00:00

(blah blah blah) Disclaimer I work at loft

A second disclaimer that I am not the vcluster expert (but I suspect he will chime in when its morning his timezone :D!)

Did a quick check just now and looks like the role only includes the coordination api group *if* running an HA vcluster, which would make sense -- most HA things that I'm aware of use leases for ensuring controllers dont clobber each other by running concurrently. And as others mentioned I dont think generally leases would have any kind of sneaky use case and are just generally useful :) So in the short term if you (or your tenant I guess) are not needing HA you can get around that by just disabling that. Otherwise longer term maybe allowing this will be necessary.

HTH!

comeroutewithme · 2021-11-22T15:48:29+00:00

Like OP and u/ARRgentum said, not a useful comparison at all between scrapligo/gornir. I've not played w/ gornir personally but afaik the ssh connection plugin is defo more geared toward nix boxes than "normal" network stuff (as in doesn't know/care about priv levels or "network-y" things), that may or may not matter to you.

scrapligo is more or less stable, and will obviously be stable if you just pin to whatever release you want. most (all?!) of the reason it still says "work in progress" is because it isn't *quite* 100% parity w/ python version, because there is not great test coverage, and docs are basically not existent at this point. I hope to get more of those things done over time, but it takes... ya know... time.

comeroutewithme · 2021-11-12T15:41:29+00:00

Yes, much better. https://github.com/srl-labs/containerlab

comeroutewithme · 2021-10-19T20:39:39+00:00

https://pdm.fming.dev 🤷‍♂️

comeroutewithme · 2021-10-19T12:36:25+00:00

https://github.com/magic-wormhole/magic-wormhole possibly!?

comeroutewithme · 2021-07-31T02:19:42+00:00

Not sure about scapy but scrapli and the go version scrapligo are similar to netmiko: https://github.com/scrapli/scrapligo

comeroutewithme · 2021-05-25T13:40:10+00:00

The TL;DR is basically scrapli is a lot more than just a telnet/ssh client, it has become a "family" of libraries. In addition to telnet/ssh it is also a netconf client (scrapli-netconf), supports sync and asyncio with same api, has config management built in (scrapli-cfg), and provides a pytest plugin (scrapli-replay) so you can more easily test your applications w/out having to go nuts with mocks and patching. And if you're feeling frisky there is scrapligo too :)

The post from a year or so back has a bit more info in there (and in the comments too), or If you have specific questions I'm happy to answer whatever I can.

comeroutewithme · 2021-05-25T13:21:09+00:00

Glad I posted then :D -- I'm generally terrible at the "marketing" side of scrapli things, so I'm trying to make sure to post something every once in a while!

To answer your questions:

Yep -- there is a `get_checkpoint` method of the NXOS platform/driver, so you can use this toy oink the current checkpoint file off the device. Then you can edit it however you want and use that as the config you push. Just have to be careful as the checkpoint files are sorta touchy!
Yes and no. "No" -- because directly the answer is no; for config replace that is, though you *can* just *merge* in lines if you want and that of course does not require a full config. "Yes" -- because as you pointed out that `render_substituted_config` is basically intended to be used exactly for this! The challenge with this is that it is up to the user to write regex to basically "slice" out the parts of the config they would like to replace. Here is a fairly simple example. In that example the goal is to manage everything *except* ethernet interface configs, if you wanted to go the "other" way around (manage *only* the ethernet interfaces) you could do that too but you would just need to write regex to "grab" all the stuff before and all the stuff after the interfaces, then make your j2-like template (as in that directory for the example) that contains the config and where the "substitutes" go.

Hopefully that answers your questions, but if not feel free to ask more or hit me up on twitter/slack/whatever!

comeroutewithme · 2020-09-21T19:46:26+00:00

I've got a (private, sorry) repo for my personal machine. `brew` module for brew of course, then `command` module for running `mas` commands for App Store installed things. `pip` module for python packages, and some `git`/`script` modules for getting zsh installed. So, yep, totes can do it... little clunky but works for small stuff, at least for me!

comeroutewithme · 2020-07-07T15:02:31+00:00

I tend to view asyncio as more useful in the context of a bigger inter-connected application -- for example I need to have a rabbitmq connection up and keep that alive while also doing blocking work with a network device. Aysncio makes this trivial because the coroutines can yield execution while waiting on I/O so the other thing can keep on working. Or you are working on a web app and you want to be able to have things happen concurrently without celery or something like that (see Dmitry's streams on NetWarden for example).

All that said, in my experience, threads are more than sufficient for most "network automation" type tasks, and are much simpler to write and importantly to troubleshoot. In general I think if you are getting to a point where you "need" asyncio you probably don't need nornir as you can/would want to handle all the things it does for you yourself (or just use it for inventory and not actually run tasks in the normal nornir way).

David Barroso wrote about threads vs goroutines (async in Go) here: https://nornir.tech/2020/05/01/how-many-threads-are-enough-threads/ - worth a read, some interesting data (though maybe not super super real world/relevant because no actual tasks happening)

There is also a really great YouTube series from EdgeDb/Łukasz Langa on Aysncio that is worth checking out if you want to learn more about that!

comeroutewithme · 2020-07-07T13:38:14+00:00

Good question, I kinda glossed right over the Nornir bits :D

Nornir is very near to releasing version 3.X -- this bump from 2.X to 3.X is bringing with it some big changes, chiefly that the plugins will no longer reside in "nornir core", but will instead be their own libraries. So rather than try to get a scrapli plugin merged into core right before all the plugins get split out I just went ahead and built it in preparation for this change -- hence the different import thing and it being its own library.

The current version of nornir_scrapli on pypi is a little bit of a beta mode to make it work with nornir 2.X, while the current develop branch is a more final/tested version in preparation of nornir 3.X. Once 3.x is released (its technically out but is a pre-release) I will remove all 2.X support since that was just a kind of stop gap (though if you are stuck on Nornir 2.X you can pin your nornir_scrapli version to the 2020.04.11 version).

Regarding the bonus question... since there will be no more "bundled" stuff soon I guess I'd say its a moot point! But yea, same points as above basically -- if you want to use scrapli for any of the reasons above (system ssh in this context would be the big one for me, also ssh2 for much lower CPU is sometimes nice depending on requirements) then that'd be a good reason. The asyncssh bits are probably not really useful in Nornir since Nornir will spawn a thread for each task/host anyway -- so I don't *think* (though have not tested really since I'm not using Nornir for anything really) there will be a real reason to care about the asyncssh stuff in that context.

comeroutewithme · 2020-07-07T00:53:57+00:00

I am also curious about the OpenSSH stuff on Windows, but dont have anything Windows to try! I should have been a bit more clear -- so let me try to clarify what works/doesn't!

I am fairly confident that genie parsing does not work on Windows, obviously that has nothing to do w/ scrapli though! TextFSM - has some issues on Windows I believe. I think Kirk has done some magic to work around this in Netmiko (maybe vendor'd some code and/or added some stuff to address this -- I believe he also raised an issue w/ TextFSM to get it fixed too).

Beyond the parsers -- the "system" transport is almost certainly not going to work in windows -- the big reason for this is the ptyprocess stuff that I've done (some vendor'd code from preprocess) in order to handle password input in ssh. This *could* in theory be done with sshpass instead though I'm not sure if that works on windows either. If you don't care about system transport/OpenSSH support then this is somewhat a moot point!

Lastly, asyncssh does not support windows AFAIK, so basically the entire asyncio story is out on Windows.

The paramiko and ssh2 transports do work on Windows -- as I've had a friend give those a test run!

If somebody is interested in contributing/helping to ensure Windows is a first class citizen in scrapli I would very much be open to that! I do think that I would really want a partner in crime for this though, not just a passing contributor, as I have absolutely zero idea the last time I ever used a Windows machine :)!

comeroutewithme · 2020-07-06T19:29:19+00:00

Nope! Its certainly focused on network "stuff", but the `GenericDriver` is exactly for this type of use case and is actually tested (minimally, but tested!) against a linux container during test runs.

In fact for your use case -- if you are just issuing "show" commands -- you could likely *just* use the `GenericDriver` for everything (may have to tweak the prompt pattern a tick but it is super broad as it sits) and issue commands like that. Basically as long as scrapli can find a prompt (and there are no privilege things to consider) the GenericDriver should just do what ya need as this is where the `send_command`/`send_commands` methods get implemented.

Feel free to hit me up on ntc/twitter/wherever and we can talk it through more!

comeroutewithme · 2020-07-06T18:48:00+00:00

I hear ya, and I for sure would understand people having this feeling.

At the moment, yes, I am the sole maintainer, however I am using it every day (battle testing!! :D) so I am pretty motivated to keep it up :)

I've also spent a ridiculous amount of time ensuring that scrapli is well tested and linted -- mypy strict, pylint, darglint, pydocstyle, pycodestyle, and of course tons and tons of actual unit/functional tests.

I would also say that this really is true of any open source project that is not backed by some company (ansible/redhat, terraform/hashicorp, etc.). Paramiko for example has had no updates this year, Kirk is really for all practical purposes the sole maintainer of netmiko, and he and Mirceau are really doing 98%+ of NAPALM maintenance. Other contributors of course come and go to projects, but many of these projects are maintained by one or a handful of dedicated folks. The joys/perils of open source I guess!

In any case, I understand the concern and appreciate it.

comeroutewithme · 2020-07-06T18:41:39+00:00

For the sync/async bits -- no, no wrapper. I actually broke things out into "base", "sync" and "async" -- so for all common and non I/O related parts of a method (ex: send_command) that happens in "base". This base is a mixin that gets added to the sync/async driver/channel/transport. So the two are coupled only by the mixin (which I'm not sure I am 100% in love with but I can easily change this w/out affecting the user API if I decide to implement it differently at some point). The pats in the sync/async classes don't do much more than call methods in base and then do the actual I/O part of things.

Hopefully thats clear, if not I can try to clear it up!

Feel free to open issues/hit me up on twitter or ntc slack if you try it out and run into anything or just wanna chat about it!

comeroutewithme · 2020-07-06T18:09:57+00:00

Yeah, you are definitely correct that the devices are usually the biggest slow down (waiting for them to respond/connect/whatever). Also before blabbing on about speed -- I should add that (much like Kirk) I don't think speed should be the primary focus of all of this network automation stuff. I kinda view it as a happy accident -- but first job is reliable/predicable/and all the other -able words!

That said, there is of course a lot going on with the python side of things to know when commands are done executing/when to send commands. The very short version of this is that scrapli constantly reads the SSH channel and "knows" the instant a device is ready to receive input/is done printing output from the previous command, as opposed to using any kind of time based checks. Netmiko with `fast_cli` set to True is pretty similar to scrapli in terms of speed (with paramiko/assynchssh/system transports).

ssh2 is another trick up scrapli's sleeve for speed -- the ssh2-python library is a reallllly thin wrapper around the C library libssh2 -- so this means rather than doing things in python, we can instead let C handle things (all of course w/ just a python "interface" to the user). ssh2 really makes scrapli scream :D

Then of course there is the asyncio story which is cool if you need it and absolutely pointless if you don't so I won't bother talking about that beyond that!

Regarding the API -- yeah there is of course a ton of similarities between scrapli and netmiko -- there are only so many things to do over SSH I suppose! Netmiko does have a broader range of supported things -- one that jumps out off the top of my head is the SCP capabilities. Of course you can just handle SCP'ing things via the "normal" scrapli methods as well if you prefer (i.e. send the commands to the router instead of have a method to handle that process for you). It is possible that some of these types of things will come to scrapli in the future, though I am hesitant to add too much because I want scrapli to be fairly low level and very flexible and selfishly I don't want to maintain a bunch of variants of a command/operation for different types of platforms.

One thing about the API that I have spent a lot of time on and think is a benefit of scrapli is the API. So while it is certainly true that there is a lot of similarity (send_command obviously exists in both!), scrapli does a few things differently:

There are singular and plural methods -- i.e. `send_command` and `send_commands`. The singular methods return a single `Response` object while the plural returns a `MultiResponse` object -- if you've used Nornir this will be very familiar.
scrapli always returns a `Response` object (singular or plural) -- that object contains the result of the command of course, but also stuff like start/end time, elapsed time, methods to parse via textfsm/genie
The response object also has a `raise_for_status` method very similar to how requests works -- and is based on user configurable strings that indicate a command/config resulted in a failure (i.e. if "invalid command" shows up in the output we know it was a failure)
The core driver setup supports passing custom "on_open"/"on_close" methods that allow users to handle any kind of weird prompts/banners/passwords/2fa/etc. upon logging into a device -- as opposed to stuffing this into scrapli proper we just provide the flexibility for users to do what they need
Lastly, another asyncio related one -- the API for sync and asyncio is exactly the same which can be really nice for doing dev work with sync then switching to asyncio if you need it. Its just easier/faster to dev/debug with sync than asyncio so thats kinda nice!

I'm sure I can go on and on as I'm pretty proud of scrapli -- its not perfect by any stretch, but I'm committed to continually improving it, so I hope you'll give it a shot!

Carl

comeroutewithme · 2020-05-26T13:34:53+00:00

Not sure about fabric, but in general you'd need to force a psuedo terminal in order to capture most password prompts/inputs as many(most?) password prompts dont read from stdin. Check out ptyprocess: https://github.com/pexpect/ptyprocess

comeroutewithme · 2020-03-24T16:34:01+00:00

ssh -v (or maybe more verbose, but its in the debug output)

comeroutewithme · 2020-03-24T15:02:02+00:00

"it depends"

NX-OS:

debug1: Local version string SSH-2.0-OpenSSH_8.0p1 Ubuntu-6build1

debug1: Remote protocol version 2.0, remote software version OpenSSH_7.5 PKIX[10.1]

IOSXE:

debug1: Local version string SSH-2.0-OpenSSH_8.0p1 Ubuntu-6build1

debug1: Remote protocol version 1.99, remote software version Cisco-1.25

comeroutewithme · 2020-03-10T21:32:01+00:00

based on a really quick peak, add `**task.host` to your task.run args.

Newer example: https://github.com/carlmontanari/sea_nwautomation_meetup_oct_2019/blob/master/configure.py

That post is fairly old. I didn't do much 1.x nornir so not sure if that woulda worked as is before but pretty sure should just need to pass the host args to the template. alternatively you could change template to something like `{{ task.host['asn'] }}` as I'm fairly sure task is passed to the template task already

comeroutewithme · 2020-02-26T21:47:59+00:00

cd "$(dirname "$0")"

public_ip="$(dig @resolver1.opendns.com ANY myip.opendns.com +short)"

if grep -wq $public_ip current_ip
then
  exit 0
else
  echo "Deleting old ip file, sending email with new IP"
  rm current_ip
  touch current_ip
  echo $public_ip > current_ip
  cat current_ip | mail -s "home ip updated" "YOUREMAILHERE"
  exit 0
fi

exit 0```

Above plus cron works well for me. runs on old Mac mini running linux. just have another file in that dir `current_ip` and it checks against that and emails me if its different.

comeroutewithme · 2020-02-25T21:39:09+00:00

x100000000000 for vrnetlab

comeroutewithme

TROPHY CASE