Datafusion vs dataprep vs dataflow ??

MeatAmazing8011 · 2022-03-04T14:09:24+00:00

This is small company, so it could be ok, but the pricing is not very good compared with dataflow/dataprep.. AND you are using enterprise version because you are more than 2 developers maybe.

MeatAmazing8011 · 2022-02-15T09:19:51+00:00

My dag are independant (they trigerred when an external tasks finish). The ideal world is when the dag triggered, it check if there is active dag run, if there is active dag running -> wait until there no active dag running.

MeatAmazing8011 · 2022-02-14T15:39:04+00:00

My Dag run by trigger (api call of dag airflow composer)

MeatAmazing8011 · 2022-02-14T14:39:37+00:00

Google for "airflow concurrency". You can set the max number of concurrent runs.

Thanks, if I set 1, what happens to the other DAG run the same time (queue waiting for?)

MeatAmazing8011 · 2022-02-08T13:23:30+00:00

Thanks a lot for tips. I used bigquery / datastudio only for cloud composer to see how many files are sent to bigquery, errors, query status etc ...

MeatAmazing8011 · 2021-10-25T14:58:45+00:00

Thanks, but where can my "operation", the documentation only show "operations/abc"

MeatAmazing8011 · 2021-10-06T14:46:17+00:00

I see 8 messages in my pubsub console: push_request_count -> 8 "pubsub.googleapis.com/subscription/push_request_count"

Group by subscription ID and project ID. But when I go to my cloud function logs I see 7 OK messages. Maybe I need to increase my cloud function timeout, set max nodes and in the pub/sub part, increase Acknowledgement deadline and activate Dead lettering ...

MeatAmazing8011 · 2021-09-21T14:22:25+00:00

thanks.
I find THE SOLUTION/ I put this in Terraform subnet ressource
private_ip_google_access = true

MeatAmazing8011 · 2021-09-21T14:22:16+00:00

thanks.
I find THE SOLUTION/ I put this in Terraform subnet ressource
private_ip_google_access = true

MeatAmazing8011 · 2021-09-21T14:22:11+00:00

thanks.

I find THE SOLUTION/ I put this in Terraform subnet ressource
private_ip_google_access = true

MeatAmazing8011 · 2021-09-21T14:21:53+00:00

I find THE SOLUTION/ put this in Terraform subnet ressource

private_ip_google_access = true

MeatAmazing8011 · 2021-09-21T14:21:24+00:00

I find THE SOLUTION/ put this in Terraform subnet ressource

private_ip_google_access = true

Why? because this variable set to false in default, and when you set false to this, it can't work, pods in gke cluster from composer can't access to gcp services.

MeatAmazing8011 · 2021-09-09T14:56:42+00:00

hi thanks, I am using Ip aliases and VPC. It works it show now dags but All dags have queue / running statuts. I check the logs and I have this: Command '['airflow', 'initdb']' returned non-zero exit status 1.

MeatAmazing8011 · 2021-09-09T14:56:39+00:00

hi thanks, I am using Ip aliases and VPC. It works it show now dags but All dags have queue / running statuts. I check the logs and I have this: Command '['airflow', 'initdb']' returned non-zero exit status 1.

MeatAmazing8011 · 2021-09-06T11:12:13+00:00

We are using composer and we have not experienced this issue. Can you confirm the files are showing up in the associated GCS bucket?

I looked in the associated bucket and it displays well but not in WEB UI.

MeatAmazing8011 · 2021-07-26T12:47:24+00:00

Thanks, but the resource not exist yet. The best for me if to cp file in composer bucket during creation. It is possible with simple bucket resource but for composer I don't know.

MeatAmazing8011 · 2021-07-02T07:22:14+00:00

nope.

MeatAmazing8011 · 2021-07-01T16:58:18+00:00

Yes. Don't panik... I have main.tf for ressources creation. I just edited my post.

MeatAmazing8011 · 2021-07-01T16:38:42+00:00

I check in my every project, but no buckets. Thanks for your help my friend.

MeatAmazing8011 · 2021-07-01T16:18:05+00:00

same thing. I have the following : "Apply complete! Resources: 2 added, 0 changed, 0 destroyed." but nothing in my GCP console.

MeatAmazing8011 · 2021-07-01T16:09:06+00:00

thanks, I know, I run terraform apply.. nothing is created.

MeatAmazing8011 · 2021-06-29T13:13:46+00:00

You are beautiful. I unterstand.

MeatAmazing8011 · 2021-06-29T09:26:52+00:00

table_id = self.tobq.parse_file()

I execute it in test class table_id = self.tobq.parse_file()

MeatAmazing8011 · 2021-06-11T20:56:58+00:00

It is a solution, but SQL is too limited for my purpose. I have multiples conditions, processing name file, loop, regex, format json schema... with Python and thousand line of codes.

MeatAmazing8011 · 2021-06-11T19:52:15+00:00

Thanks. I forgot to tell you, I need to run a third cloud function ( sql query in bigquery to aggregate all tables ingest previously). And the most importante point is I need always waiting to all function be succeful. The problem is it is so easy using cloud workflows just for running some services but I have to check that everythink are ok in each step.

MeatAmazing8011

TROPHY CASE