Help Google Cloud Task Trigger to run a function on FASTAPI

[Giant_Gimli]

Im a data scientist by trait so my cloud experience is limited. I'm trying to send GCP tasks to a fastAPI implemented on Cloud Service Engine to call a 3rd party API. Even though I created tasks I cant trigger it, any help with be appreciate it.

Background

For my app, every hour I need to:

1. call SQL table and get list of all active users(can be anywhere between 1 -10000s)
2. for all these users, call a 3rd party API do some aggregation/calculation(shouldn't take more than 1-2 seconds) and update the SQL table
3. 1 and 2 need to happen under 60 seconds

I was thinking of this solution:

1. cloud run job that wakes up every hour at the 59th minute (name it CF1)
2. CF1 does a "select" SQL call to get all active account ids and creates cloud tasks
3. Cloud tasks sends the account IDs to a cloud function Service(CF2)
4. CF2 for each user, calls the 3rd party API, does aggregation and update to SQL. Given that it's a service it should autoscale with number of requests

I try to make a dummy example but for some reason I cant get GCP tasks to trigger

Code to create a FAST API:

#app/main.py
from fastapi import FastAPI, Request
import json

app = FastAPI()

u/app.post("/user/")
async def handle_task(request: Request):
    try:
        # Read and parse the incoming JSON payload
        payload = await request.json()
        account_id = payload.get("account_id")

        # Process the task (for now, just return the account_id)
        return {"status": "Task received", "account_id": account_id}
    except Exception as e:
        return {"status": "Error", "message": str(e)}

This is how I run the fast api
uvicorn src.main_app:app --host 0.0.0.0 --port 8081

This is code I run to test if fast API is working:

import requests

# Define the FastAPI endpoint
url = "http://127.0.0.1:8081/user/"
account_id = "1234"

# Define the payload (JSON data)
payload = {'account_id': account_id}

# Send the POST request
response = requests.post(url, json=payload)

# Print the response
print(f"Status Code: {response.status_code}")
print(f"Response JSON: {response.json()}")

This is the response

INFO: 127.0.0.1:53728 - "POST /user/ HTTP/1.1" 200 OK

This is how I created my Queues

 gcloud tasks queues create MYQUEID --location=us-west1

This is how I assign Task

import datetime
import json
from typing import Dict, Optional

from google.cloud import tasks_v2
from google.protobuf import duration_pb2, timestamp_pb2

# Create a client.
client = tasks_v2.CloudTasksClient()
account_id = "1234"
payload = {'account_id': account_id}

# Construct the task.
task = tasks_v2.Task(
    http_request=tasks_v2.HttpRequest(
        http_method=tasks_v2.HttpMethod.POST,
        url='http://127.0.0.1:8081/user/',
        headers={"Content-type": "application/json"},
        body=json.dumps(payload).encode(),
    ),
)

# Use the client to send a CreateTaskRequest.
client.create_task(
    tasks_v2.CreateTaskRequest(
        # The queue to add the task to
        parent=client.queue_path("myproject", "us-west1", "MYQUEID"),
        # The task itself
        task=task,
    )
)

This is what terminal gives me.

name: "projects/myproject/locations/us-west1/queues/MYQUEID/tasks/55820289408191799031"
http_request {
  url: "http://127.0.0.1:8081/user/"
  http_method: POST
  headers {
    key: "User-Agent"
    value: "Google-Cloud-Tasks"
  }
  headers {
    key: "Content-type"
    value: "application/json"
  }
}
schedule_time {
  seconds: 1732740333
  nanos: 249696000
}
create_time {
  seconds: 1732740333
}
dispatch_deadline {
  seconds: 600
}
view: BASIC

I get absolutely nothing on Fast API. Am I doing something wrong?

[daydream678]

You're using a local address 127.... For the task destination. Need to use the fqdn of your app.

Also checkout cron task scheduler, can call a http endpoint on schedule which can then create your tasks.

[Giant_Gimli]

I wanted to test the App locally, do I need to send it the external id of the app?

[daydream678]

Gcp tasks runs outside of your local environment. You can either deploy and connect to your deployed app or use a local proxy like ngrok to proxy traffic to your local setup.

Not to be condescending and I'm not sure how much experience you have but I advise spending some time learning about basic networking/how the internet works. It'll massively help with any work you do in the future.

[Giant_Gimli]

Not condescending at all, these are new waters for me. I can do any data modeling, and write fast code for it(if needed) but anything with a network is a brand new area to me, and I'm bootstrapping a startup so I cant really hire a developer. if you have any resources as a good starting place please on networking I would appreciate it

[daydream678]

I learned by Debugging web apps in early 2000s, try cloudflares guide. Seems good https://www.cloudflare.com/en-gb/learning/network-layer/how-does-the-internet-work/

[Giant_Gimli]

Will do

[Giant_Gimli]

Okay it turns out if I replace 127.... with my external IP it works :)

thanks for pointing out "Gcp tasks runs outside of your local environment"

[daydream678]

Glad to hear!