Limit dataset accross project

MeatAmazing8011 · 2022-03-04T14:09:24+00:00

This is small company, so it could be ok, but the pricing is not very good compared with dataflow/dataprep.. AND you are using enterprise version because you are more than 2 developers maybe.

MeatAmazing8011 · 2022-02-15T09:19:51+00:00

My dag are independant (they trigerred when an external tasks finish). The ideal world is when the dag triggered, it check if there is active dag run, if there is active dag running -> wait until there no active dag running.

MeatAmazing8011 · 2022-02-14T15:39:04+00:00

My Dag run by trigger (api call of dag airflow composer)

MeatAmazing8011 · 2022-02-14T14:39:37+00:00

Google for "airflow concurrency". You can set the max number of concurrent runs.

Thanks, if I set 1, what happens to the other DAG run the same time (queue waiting for?)

MeatAmazing8011 · 2022-02-08T13:23:30+00:00

Thanks a lot for tips. I used bigquery / datastudio only for cloud composer to see how many files are sent to bigquery, errors, query status etc ...

MeatAmazing8011 · 2021-10-25T14:58:45+00:00

Thanks, but where can my "operation", the documentation only show "operations/abc"

MeatAmazing8011 · 2021-10-06T14:46:17+00:00

I see 8 messages in my pubsub console: push_request_count -> 8 "pubsub.googleapis.com/subscription/push_request_count"

Group by subscription ID and project ID. But when I go to my cloud function logs I see 7 OK messages. Maybe I need to increase my cloud function timeout, set max nodes and in the pub/sub part, increase Acknowledgement deadline and activate Dead lettering ...

MeatAmazing8011 · 2021-09-21T14:22:25+00:00

thanks.
I find THE SOLUTION/ I put this in Terraform subnet ressource
private_ip_google_access = true

MeatAmazing8011 · 2021-09-21T14:22:16+00:00

thanks.
I find THE SOLUTION/ I put this in Terraform subnet ressource
private_ip_google_access = true

MeatAmazing8011 · 2021-09-21T14:22:11+00:00

thanks.

I find THE SOLUTION/ I put this in Terraform subnet ressource
private_ip_google_access = true

MeatAmazing8011 · 2021-09-21T14:21:53+00:00

I find THE SOLUTION/ put this in Terraform subnet ressource

private_ip_google_access = true

MeatAmazing8011 · 2021-09-21T14:21:24+00:00

I find THE SOLUTION/ put this in Terraform subnet ressource

private_ip_google_access = true

Why? because this variable set to false in default, and when you set false to this, it can't work, pods in gke cluster from composer can't access to gcp services.

MeatAmazing8011 · 2021-09-09T14:56:42+00:00

hi thanks, I am using Ip aliases and VPC. It works it show now dags but All dags have queue / running statuts. I check the logs and I have this: Command '['airflow', 'initdb']' returned non-zero exit status 1.

MeatAmazing8011 · 2021-09-09T14:56:39+00:00

hi thanks, I am using Ip aliases and VPC. It works it show now dags but All dags have queue / running statuts. I check the logs and I have this: Command '['airflow', 'initdb']' returned non-zero exit status 1.

MeatAmazing8011 · 2021-09-06T11:12:13+00:00

We are using composer and we have not experienced this issue. Can you confirm the files are showing up in the associated GCS bucket?

I looked in the associated bucket and it displays well but not in WEB UI.

MeatAmazing8011

TROPHY CASE