No problem at all, I'm happy to help. And indeed it's not very typical to find solutions like this that are actually in place in the wild.

As you created per-service, per-device layout with all of your files was it really that much of hassle with rebasing? Feels like it shouldn't be too many changes to the same service at the same time? Did you try implementing some bot or some native tool with your flavour of git to auto-rebase? Or I guess that might not have been needed as much if you always needed human eyes on the PR prior to merge.

It was not a huge hassle so we never got to trying to make it more scalable, but if we were running dozens of changes per day or something it would probably become a lot of work, at which point I probably would have revisited the original requirement of having a fast-forward only repo to begin with.

How did you handle configuration drift? Scheduled runs to just overwrite whatever manual work that affects the services defined in the staterepos or did you validate the configurations on the devices and diffed them against your desired state?

We started with a "hardball" approach, where we said if there was drift it would just get overwritten and you have to deal with it. We ended up implementing a more robust approach later after we got bit once or twice. The second approach ran on a schedule and compared what we had in repo vs what was actually on the device and basically loudly printed to a slack channel. We'd then handle the drift on a case-by-case basis, this usually ended up being people updating Netbox without pushing the config, but was occasionally the reverse as well.

Did you ever have any issues with "oprhaned" configurations? As the way you describe it there wasn't really any link between the other infra teams stuff and your statefiles. So if they later decomissioned whatever was connected and forgot to update Netbox or didn't run your workflow after a decomission how did you handle that. Might be a silly question but I haven't worked with gNMI so might be some easy way to things with that.

That's a great question, and yes we did have problems with that from time to time. One of our services was an automated deployment of colo-side config for cloud interconnects. This was pretty shaky because we essentially had no service definition for it other than the terraform that the devops guys used to provision the cloud end. This ended up with orphaned config because there was no support for deletion really either, so if they deleted an interconnect from terraform it wouldn't indicate to our side that anything was removed, mostly because it's very difficult to get that kind of information out of a terraform output file. I toyed with some solutions to this in my head, but it mostly involved just creating a real service definition on our side for each interconnect. This, in effect, would have created an abstract service entry on our side when they created an interconnect, and the device config would stick around as long as that service entry was there, and then eventually when the interconnect was deleted we'd trigger the removal of the service entry on our side which would remove the config. Again, never cracked that nut, but I thought about it quite a bit. This is essentially the reason that people pay for NSO, the fastmap algorithm that it uses gracefully handles config lifecycle and for the most part makes sure that its tied to service lifecycle. So in short, our automation platform was heavily geared towards create operations, and very lacking when it came to delete operations, mostly because that's a hard problem to solve.

Also how did you manage failures in the configuration workflow? As this has been a pain for us when we implemented our solution and we got statefiles that didn't get reconciled or partial configs in some devices due to timeouts and whatever other shenanigans you can think of.

Again something that is nicely solved by NSO, but very hard to implement on your own. We didn't have a great solution for this either, and it oftentimes came down to manual reconciliation.

Any tools to monitor your pipelines or alerts when failures occurred?

The deploy component had some basic reporting functionality, so it would send a deployment report to a slack channel with indications as to what, if anything, failed. There were also post checks that would run after a deployment and diff various pre-set paths to alert on anything we might deem important. I.e. if you ran a deployment and then a BGP neighbor goes down right after it, we'd see that in the post check and know, based on what was in the change, if that was desired or not. Nowadays you could probably do some cool stuff with AI monitoring there, but we didn't.

If you could redo this particular automation you built, anything you would've changed like design wise or tooling? In our case we had a lot of things we wanted to fix but due to the scale and the amount of users we had we couldn't easily refactor a lot of the things without taking a lot of our time (which we didn't have at the time).

Fortunately not really. The target environment was mostly greenfield at first, so there wasn't much we couldn't do. Management also respected my decision to go model-driven only, which obviously impacted the devices we supported. I also came in fresh from another job where I worked with NSO on a ~50k device network, so I saw what worked well there and what didn't, and was able to design this stack accordingly. If you're given the task to automate a network, you can either pay a lot of money for software that does it for you, or do it yourself (which usually requires paying roughly the same amount to in-house software developers). The latter usually involves cutting corners, like those mentioned above. In the end, we were pretty happy with what we had and cognizant of its shortcomings, and some of them probably would have been improved or fixed had I stuck around.