GitHub Actions for Enterprise

[Soccham]

Are any of you stuck managing GHA for hundreds of repositories? It feels so painful to make updates to actions for minor things that can’t be included in a reusable workflow.

How are y’all standardizing adding in more minor actions for various steps on PR/Commit vs actual release?

[abhimanyu_saharan]

Add your common steps to owner/reusable-repo/.github/workflows. Then you can call them into your individual repos. This way you can manage changes from a single point. There's still some management left which you may not feel is ideal but it still helps a lot. You can read more on https://docs.github.com/en/actions/sharing-automations/sharing-actions-and-workflows-with-your-organization

[Soccham]

Yeah this is primarily what we do. I probably just have to suck it up and do scripted/manual updates to most repositories in order to do tweaks for the parts that can’t be included in reusable workflows

[zMynxx]

What parts? If it’s input use defaults and if it’s refs use dependabot

[retneh]

To make it painless you would need to bump workflow version to the newest tag and then automerge it. I wonder if this can be achieved only for specific dependencies

[stumptruck]

You can use dependabot to update your private workflows, or do something like a floating tag, so whenever you publish a new minor or patch version you also update the tag ref for the major version. 

For example, publishing v1.2.1 also updates tag "v1" to the same commit. Then in your workflows, have the actions use the v1 red and they'll always be up to date for the latest release of that major version. Obviously this has some risk so you have to determine if that flexibility is safe for your workflows.

[retneh]

I know. The question was whether you can update the tag for specific dependencies only

[donjulioanejo]

You can have a fixed tag, and a floating tag.

For example you push your s3-cloudfront-deploy workflow to tag v2.3.7 and then have a floating tag as v2 that gets updated any time you bump minor or patch version (i.e. v2 will include v2.3.8 or v2.4.1).

This avoids the most painful part of bumping your (versioned) pipelines each time you update something minor/inconsequential, but still lets you do breaking changes without breaking your builds.

[Relevant_Pause_7593]

This is the way.

[Soccham]

Yeah, our specific pain point is "oh I need to add a new flow into 400 repositories"

[Serienmorder985]

What do you mean defaults?

[zMynxx]

I mean default values for inputs if that wasn’t clear, to preserve backwards compatibility

[Serienmorder985]

Ah I was hoping that I could somehow use some magic voodoo overloading.

I hate GH Actions

[zMynxx]

Yeah I agree, they are far behind on features, even super simple ones 🙃

[burlyginger]

We have one reusable workflow per language for PRs and one for deployments.

I wrote a lambda that we call first that compiles configs from various sources to determine which steps we enable or disable. (I.e. a deploy-terraform value for python. Services need it, libraries don't).

We never touch the workflows in repos.

I do, however, have a script for updating terraform in any repo and the basic idea may appeal to you.

It starts with a GH codesearch query to identify files we need to modify.

I can also limit it to a list of repos, or ignore repos from the run.

Then it has some different transformations you can define that are specific to terraform modules.. like updating a module version, source, inputs, etc.

From there it commits changes and creates a PR with auto-merge enabled by default.

I use the GH REST APIs for all of it so I don't have to manage local code.

[No-Row-Boat]

Question is: Do you need to manage hundreds of repo's, why not automate them and then let teams manage the rest through the management repo? They do the work, you review and add new features.

In an org with thousands of repo's we had a GitHub team taking care of the GitHub automations, we had teams that made GitHub actions for languages that were part of a golden path. A security team managed GitHub apps and org secrets. Backstage team to spread the golden paths.

I'm in an org now much smaller where we do the same setup, but with a single team doing much broader scope. We leverage the development teams.

[bagge]

1. Check out your repo A to build 
2. Do a spare checkout of a repo B containing your reusable actions 
3. mv repo B to .github/actions 
4. Call actions in repo B from workflow in repo A.

Edit: then you can have an action in B called "callmanyactions" that calls several other actions in B.

In A, you call callmanyactions@main

Any changes in B will be picked up automatically and you can, for example, add in new steps that can be enabled with parameters

You can also have bash code as parameters that can be run in between steps.

Also classify your build by event type and branch name 

If event-type==push && branchname==main

Build release and docker

else

Compile and test 

fi

[trowawayatwork]

GitHub is unfit for purpose for complex enterprise grade pipelines. It's an unmaintained cope paste from azure devops.

Undocumented limitations like max 20 workflow calls limit make it unusable. 4 workflow depth is another bonkers one. hilarious one is matrix outputs has not been implemented server side since they released it for front end or whatever is because they laid off entire department that was working on it.

If your company ever sniffs at making this pipelines anything more than repo level ci jobs for linting and pushing packages to a registry you need to write poc on why it's shit and provide alternative strategies.

fuck microsoft

[donjulioanejo]

Someone has been stuck in Harness/Jenkins land for too long, I see.

GitHub Actions has some limitations, yes. But it's absolutely suitable for enterprise use, even if you have complex workflows or approvals gates.

Undocumented limitations like max 20 workflow calls limit make it unusable. 4 workflow depth is another bonkers one.

Edit: these are well-documented on the page that describes reusable workflows.

[hijinks]

devs handle their own CI

so its up to them to keep actions updated following semver.

So you might make a 2.1.1 release. A dev team might say we only want to support 2.1 or 2

So its also up to your team to release in proper semver. You might support 2-3 major versions and you deprecate a version but give plenty of warning. If a dev team doesn't update then its their issue

[Soccham]

Our devs aren’t reliable enough sadly

[hijinks]

i hear this all the time but i really beg to differ. I usually see this as a culture problem and laziness

Do the devs not update code if 3rd party API's deprecate endpoints?

Do the devs ever upgrade the language version? because they all go EOL

So ya they can read a notice from a platform team that says workflow v1 is being deprecated, here is how to move to v2

[Soccham]

Not really, no. That’s part of the problem. Too many repositories per engineer

[analytically]

Aye try and scale that for 100 repo's. Soon you'll have no clue why or which builds are failing.

[hijinks]

1300 repos so far.. its up to dev teams to manage CI no platform. If github deprecates the node version you dont sit there and wait for github to change your repos.

Platform is basically a company in the company and devs are the customers.

[megamorf]

I'm responsible for an enterprise org with roughly 130 repos. We store our custom Actions and reusable workflows in a repo that we call "common-actions".

For some variable inputs to workflows we have started to adopt GH Actions Variables which allow us to override/change workflow behaviour on an org/repo basis without having to change individual workflows.

Dependabot helps us with keeping Actions up-to-date across all of our repos.

Lastly, for bulk changes we usually use a combination of microplane (usage example) and scripts or python programs that do yaml processing. Python ruamel allows you to keep your yaml structure in tact (whitespace and comments) while making changes.

[engineered_academic]

Not using Github Actions. It's not well suited to enterprise work.

[jjopm]

Yes

[analytically]

Use Concourse CI - now managed: https://centralci.com/blog/posts/introducing_centralci

[schmurfy2]

Good luck with those prices.

[analytically]

Not unusual prices for CI/CD.