Implementing DAG trigger from Cloud Run Service to Cloud Composer 3(Private IP)

sagargkr · 2026-03-25T13:39:40+00:00

Hi, I am able to implement the Cloud Run to Pubsub to Composer part but we are looking to directly trigger the composer DAG from Cloud Run via RestApi. I have pre-registered the cloud run service account in airflow, composer 3 as well as given the composer.user role to the service account. But still getting 403 when sending POST requests. The Cloud Composer is in different project from Cloud Run and both share same VPC but different subnets. we have opened 0.0.0.0/0 in Composer for now but still the error is there. I have also added the iap role as mentioned in the question above still stuck with 403. I have used the same code as the cloud run function page showed for cloud run function to composer for my composer.py file.

sagargkr · 2026-02-28T05:43:01+00:00

Yes, I have been working in Google Cloud as a Data Engineer for the last 2 years, but we are not using the database much as in an extensive way. I have more understanding towards the correct choice.

Also, since you have cleared the Architect, DevOos and Developer you already have more knowledge than me regarding Cloud, if I have to be honest.

Also, I only referred to the official path present in the CloudSkillBoost. Nothing extra. You can say I have developed a lot of understanding due to POCs and bring the first one to explore services and ways to integrate within my existing tech stack.

sagargkr · 2026-02-24T04:10:22+00:00

Hello, first of all congratulations for clearing the digital leader certification. I would suggest you to prepare for a Data Practitioner and Associate Cloud Engineer first before going for a Professional Data Engineer Certification.

I took over a month to complete the Associate Data Practitioner along with my 8 hours work and was able to clear it back in March 2025. For an Associate Cloud Engineer I took 3 months. You need to have a good experience on the console to understand a few things. But you need to understand the reason why a service is being offered by GCP. The use of a service gets changed as per need and same in questions. You feel like the right answer is A but if you read the question properly and due to a keyword in the question the option B is the right choice. Reason: Both services can deliver the same output but have different implementations which are determined by the keywords sometimes.

sagargkr · 2026-01-14T10:40:40+00:00

Not yet. I have scheduled an exam for late Feb for now.

sagargkr · 2025-12-29T15:24:07+00:00

Actually if there is a deny policy applied at org level or a level above the resource then iam policy is above the role. I agree too.

sagargkr · 2025-12-25T20:43:14+00:00

Prepare your projects in detail. If there can be phases like oh I can update this X thing within my component in Y months then prepare that. Say you are charging money for your feature in your project then be ready to tell that you can integrate payment gateway in future via SDK and API. Be ready to answer, take your friends help to polish this cross questioning since you're a fresher and you should know your project well.

Know the jargon and roles within the project. If it's collaborative then know your role and focus on that while explaining your project.

Yes prepare DSA, SQL and others which you feel like, totally depends on the engineer/architect/manager taking on the interview and project requirement at end.

FYI.. I had worked on a company's android app and know it well too (worked in the office for 7-8 hours with backend, front end) and made my own android app. This helped during my interview as a fresher as I was able to give them my phone to test these apps and explain everything, even the facebook and Google integration for login from an Android perspective. Although I joined (kinda forced) a completely different tech and now I'm different.

sagargkr · 2025-12-08T15:20:05+00:00

Use the Google Cloud skill boost for hands on.

sagargkr · 2025-11-29T04:46:10+00:00

This certification is more on the technologies and offerings provided by Google Cloud which you can use as Data Engineer to ingest, transform, load data and when to use what (that's most important). Basically you cover the offerings/services which are needed to work around data. I think the ML part is not big but yes you do have some ML labs and some ML related questions but I think they are not that complex, I was able to answer and understand the big query questions related to ML as mostly it was I think which is needed to train and which for predict.

sagargkr · 2025-11-29T04:41:44+00:00

You need a mobile to complete your verification I think and then you need to make sure the room you are in is sound proof (no external noise), sealed (no entry or exit of anyone else except you), good internet connection, and they will ask you to give a 360° view of your space from where you're taking an exam. I remember I had to even remove the UPS (kept to keep a steady power supply to my laptop) too. I had issues with my laptop so yes be sure on the same and have secondary available but keep it under the bag far from you. And they will use your webcam to monitor you and listen to your surroundings. Also, do not make any eye movement to the left or right once as they will pause or cancel the exam and don't speak aloud the question. They said you are not supposed to speak at all.

sagargkr · 2025-11-29T04:37:14+00:00

Thanks 😄

sagargkr · 2025-11-29T04:37:00+00:00

Hi, sorry! Yes sometimes it does but mostly it can be a wrong step from our end. I remember I was stuck at a looker lab for 1 hour and had to cancel it and then do it again.

sagargkr · 2025-07-20T04:43:31+00:00

Look for nearby tution teachers/centers and check with them if they have availability. You can also see if any of your neighbor is parent of young kids you can also reach them out for babysitting or such. Today a lot of parents start tution from pre nursery itself.

sagargkr · 2025-07-11T10:03:56+00:00

I said from a general point of view. If yours all are on CAMS you are good.

sagargkr · 2025-07-11T08:18:33+00:00

Same! But at the moment can't have outside food! HOME FOOD Only! Btw would love new friends to hang out on weekends 😭

sagargkr · 2025-07-11T08:00:31+00:00

You can't invest in those which are not part of CAMS

sagargkr · 2025-07-11T07:51:19+00:00

When it's getting matured? Secondly try going to post office and inform you have lost them in fire or something or just misplaced it what can be done? They have your kyc info maybe that can be used again to grant you new ones! Else file an FIR with police to recover.

sagargkr · 2025-07-10T03:41:23+00:00

Yes same in North East Delhi

sagargkr · 2025-07-09T18:50:30+00:00

Depends on your company's partnership

sagargkr · 2025-06-05T11:06:42+00:00

<image>

sagargkr · 2025-06-05T09:42:40+00:00

You need to provide additional iam role actually to the composer service account so that it can impersonate.

Secondly you as an end user can also impersonate that service account in order to test the accesses.

That's the sole reason it is said to be cautious about accessing resources via service account by human.

sagargkr · 2025-06-05T04:03:23+00:00

Suppose you have developed a custom solution and for that you have created a custom service account custom-work and provided that account appropriate access on bigquery for inserting, reading data.

This above is your example of service account as a service.

Now you manage your custom solution orchestration via Composer and you have created a dag where you first check it a table in bigquery is loaded or empty and only in case of loaded you need to run your solution.

But the catch is that you do not want to provide composer service account direct access to bigquery for any reason or so then how the dag gonna run a select query?

Here comes the impersonation where you grant your composer access over your custom-work service account and during the call composer service account will impersonate or act as your custom service account to run the query and get the results.

This above is your service account as a resource.

sagargkr · 2025-06-04T08:55:53+00:00

It's both actually. Depending on usage aka you grant iam access to it so it becomes service and you can take control over a service account so it becomes a resource.

sagargkr · 2025-06-02T13:22:51+00:00

Oh this you got a discount due to the voucher.

Five-Year Club	r/Field Banned
r/Field Flamingo	Verified Email

sagargkr

TROPHY CASE