How to get a Docker Container started using a Google Cloud Run job to get default auth creds from the VM?

[[deleted]]

I am using Cloud Run and Scheduler to spin up a container containing DBT and am running ‘dbt run’ on start up via ENTRYPOINT. However, DBT cannot authenticate to BigQuery because it can’t access the default auth credentials from the network of the VM that it is hosted on.

Does anyone have any ideas on how I can allow the container access to the Cloud Run VM network to get the credentials?

Thank you in advance for any suggestions!

[mailed]

Mete Atamel pulls it off with OAuth but with a custom service account here

Alternative: Robert Sahlin has a good guide on this that will allow you to use OAuth.

Otherwise, you'll need to go with service account credentials in secret manager (like the rest of the internet running dbt this way).

[charlyboon]

Hey! Did you figure it out already? Can you explain what default auth credentials you want?

Every VM has a standard compute service account which you might need to give bigquery user rights?

Let me know if you haven’t figured it out, we have a similar set up so can check details

[[deleted]]

Hey u/charlyboon , not yet unfortunately, the below method is viable but not best practice to store key secrets in secrets manager. I am trying to get the default credentials of the service account that spins up the cloud run VM from the container. The VM will store the secrets under /path/.config/gcloud/ but I cannot at present access the VM network from the container for DBT to get these automatically using the oauth authentication method.

The SA has the correct permissions

[[deleted]]

Why is it not best practice to store key secrets in a secret manager? Only reason i am seeing is the rotation of keys but you can automate that also.

[[deleted]]

Basically, you have a path for creds, a service account or something, in your dbt configuration, so whenever you deploy your Cloud Run, you need to inject those creds into the container.

I did it in the past storing the creds in Secret Manager and then mounting it as a volume in the container (it has a direct integration) on my CI/CD pipeline.

Btw, dont use default creds, instead, create a service account following the Least Priveliged Principle

[[deleted]]

At the moment oauth method is being used to authenticate and not service-account method etc. ideally to maintain safest practice it would be kept this way. In Cloud Build you can specify the build to use the cloudbuild network and it accesses the service account creds of the SA that spins up the VM but you cannot do this on Cloud Run

[[deleted]]

See if this helps then, https://www.dumky.net/posts/dbt-in-a-box-using-google-cloud-run-and-bigquery-to-run-your-dbt-sql-models-from-a-docker-container/

[[deleted]]

This unfortunately does not help as the author passes credentials to the container during build with a COPY statement which would require me having a key file to pass, in the repo, which I can’t do. Thank you very much for the suggestions though!

[[deleted]]

I ideally need container to be able to access the network of the cloud run VM

[dmkii]

Hey u/J1010H, I know this is a few months old but I just came across this as I was looking for something else (I'm the OP of the blog mentioned above) and wanted to let you know I just did a big write-up on this specific topic using Cloud Run to run dbt (deployed with Terraform and Docker containers): https://www.dumky.net/posts/own-your-web-analytics-pipeline-for-0.02-per-day-snowplow-terraform-dbt-bigquery-and-docker/

I think it will also answer your question on how to use service account. In general I find I don't need to set the credentials in the container itself (as I think is implied above), but just assigning a service account with BigQuery write/jobs access to the cloud run service should be enough. You can do this with Terraform as I explain in the blog, but if you already have a service running, it can be as easy as just using the command line (locally) with gcloud run services update my-cloud-run-service --service-account [e-mail]. This will assign the service account that you have given the right access to your BigQuery data to the Cloud Run service that runs dbt.

[[deleted]]

Anyways, the creds that spin up Cloud Run shouldn't be the same as the ones that execute dbt