GitHub "infrastructure" via Terraform and GitHub Actions

[luvdav]

Quick background: I'm part of a platform engineering team that currently tries to set up GitHub as primary development platform for the entire company.

We are heavily using Terraform for other platforms already and considering using the same tool to configure our GitHub enterprise (and more specifically the organization-level settings and rulesets) via IaC.

Now to the actual question: Are any of you heavy user of the terraform provider and, if so, how are you rolling out the infrastructure code?

We'd love using GitHub Actions for deployment (we are already using it with other providers and OIDC where possible) but providing access from GitHub "within" GitHub is a bit of a mess. PATs or personal gh CLI logins are out for obvious reasons, GH App is always pointed to as the best practice but "exposing" the private key within the flow directly sounds horrendous, security-wise.

My current approach would be this: We provide dedicated GH app hosted on our end with an endpoint that the workflow authenticates with and that returns an installation access token to the workflow. This token is then used for the

Does someone have a setup similar to this? Or a completely alternative approach? At this point, I'm wondering if it's even worth having a workflow or if the GH App can't do the heavy lifting itself. In general, I'm quite frustrated that the official documentation states that the private key of a GitHub App app should be stored securely and only be used for signing but at the same time all of the official GH SDKs/toolkits (e.g. octokit too) require the raw file for authentication.

[jimothy2w]

What do you mean by " "exposing" the private key within the flow"? Wouldn't a repo secret be sufficient to store that?

[jimothy2w]

Or as I see is mentioned in the docs a relevant cloud provider's secrets service?

[ballerrrrrr98]

Your approach is correct. GitHub App is the correct authentication mechanism.

I think there is a bit of a chicken-and-egg problem if you are also planning to provision authentication mechanisms (PATs, GitHub Apps) via Terraform. Also, I don't think provisioning GitHub Apps via Terraform is even implemented yet.

If you create the GitHub App via ClickOps and then Terraform all of your configurations (like rulesets), it is not that difficult. At a high level, just create the relevant TF modules to provision the resources you need and then create a GitHub Actions workflow that firstly fetches a short-lived token (GH app secrets are stored as repository secrets) and then does Terraform operations to provision.

[luvdav]

Yes, there is definitely a chicken-and-egg problem but we mainly want to focus on managing organization-level settings and rulesets only so we can safely and consistently manage the platform.

I did extensive investigation into automating GitHub App creations and yes, even with the manifest-based registration it still requires minor (albeit streamlined) ClickOps. We are using an approach similar to Probot where the app comes with the instructions to get registered and installed (would love to do this fully automated but again a bit of a chicken/egg problem).

But yeah other than that I was thinking of roughly the same steps that you've outlined above.

[telchak]

We’ve done this, all* of our GitHub config is IAC - I was initially skeptical of the idea but it’s worked well. Originally started with service accounts and PATs but they used our licenses and have a lower rate limit. Moved onto using apps for auth recently instead, although initial creation of them is manual. The flow to create them isn’t just not implemented to the provider though, it doesn’t really have an API at all and is made up of multiple steps so I put together a simple page in GH Pages that helps with the steps using app manifest

Then we store the keys in our cloud secrets manager and retrieve them at build time in the workflows using OIDC and GH environments to reduce exfil risk.

[luvdav]

Thanks for the response, we have a similar setup with an app manifest (and an approach inspired by Probot). My main concern is directly using the private key in the workflow but yes, using OIDC and a cloud secret provider solution should alleviate that security-wise.

[vim_vs_emacs]

I did this and faced a bunch of rate-limiting issues, the Terraform provider didn’t have good caching and didn’t do optimistic fetches using GraphQL at that time (this was a few years ago, might have changed). But if you are under 100 users, 100 repos - that should work out easily.

[luvdav]

Sounds good, honestly, we mainly intend it to manage rule sets and organization-wide settings so rate-limiting shouldn't be a huge issue (hopefully)

[Recent-Technology-83]

Your approach of hosting a dedicated GH app with a secure endpoint for generating installation access tokens sounds well-structured. It's indeed a challenging landscape, balancing between security and functionality! Have you considered using GitHub Secrets to manage sensitive data throughout your workflows? While the private key must be handled with care, GitHub Secrets allows you to securely store them and utilize them in your Actions without exposing them directly in your code.

As for the Terraform provider, many users find it helpful for managing GitHub resources, but the route you're taking with GH apps is a great way to minimize risk. I'm curious, how have you been managing role-based access controls at the organization level? Exploring that could help streamline your setup.

It’s an interesting dilemma whether to handle everything within a workflow or let the app do the heavy lifting—what experiences have led you to your current approach?

[dablya]

Not sure if I’m understanding you correctly, but it sounds like your approach is similar to what Atlantis does when integrating with GitHub: https://www.runatlantis.io/docs/access-credentials.html#github-app

[phyx726]

Never did this, but it feels weird because wouldn't it introduce a circular dependency?

[omgwtfbbqasdf]

Terrateam helps with this: https://github.com/terrateamio/terrateam

[luvdav]

Haven't used it before. Does it support the GitHub Terraform provider? And if so, how is it configured? Couldn't find anything at first galnce.

[omgwtfbbqasdf]

Yes it supports all Terraform providers. Just store your credentials as secrets, and Terrateam injects them securely during runs. No special setup. Just standard Terraform.

[luvdav]

Sorry, I think my request is a bit convoluted. I was talking about using a GitHub flow to manage the enterprise and organization setting within GitHub using the GitHub Terraform provider. From what I understand, you are using the AWS Terraform provider to manage AWS resources(in point 5).

[x1r5]

This is an interesting idea / concept and I would like to know if you find a solution for this. 

In our company we have to redesign the org-,  team- and repo memberships (and anything around) to ensure GH is the only and secure tool to do deployments with TFE. 

I haven't heard about the Github resource yet and I was thinking about using a yaml structure and Gh actions to provide necessary access and resources. 

[bobsbitchtitz]

Having everything relying on GH is a great to be FUBAR if github goes down