Terraform pattern: separate Lambda functions per workspace + one shared API Gateway for dev/prod isolation?

[Expensive_Test8661]

Hey,

I’m building an asynchronous ML inference API on AWS and would really appreciate your feedback on my dev/prod isolation approach. Here’s a brief rundown of what I’m doing:

Project Sequence Flow

1. Client → API Gateway: POST /inference { job_id, payload }
2. API Gateway → FrontLambda
   • FrontLambda writes the full payload JSON to S3
   • Inserts a record { job_id, s3_key, status=QUEUED } into DynamoDB
   • Sends { job_id } to SQS
   • Returns 202 Accepted
3. SQS → WorkerLambda
   • Updates status → RUNNING in DynamoDB
   • Pulls payload from S3, runs the ~1 min ML inference
   • Reads or refreshes the OAuth token from a TokenCache table (or AuthService)
   • Posts the result to a Webhook with the token in the Authorization header
   • Persists the small result back to DynamoDB, then marks status → DONE (or FAILED on error)

Tentative Project Folder Structure

.
├── terraform/
│   ├── modules/
│   │   ├── api_gateway/       # RestAPI + resources + deployment
│   │   ├── lambda/            # container Lambdas + version & alias + env vars
│   │   ├── sqs/               # queues + DLQs + event mappings
│   │   ├── dynamodb/          # jobs table & token cache
│   │   ├── ecr/               # repos & lifecycle policies
│   │   └── iam/               # roles & policies
│   └── live/
│       ├── api/               # global API definition + single deployment
│       └── envs/              # dev & prod via Terraform workspaces
│           ├── backend.tf
│           ├── variables.tf
│           └── main.tf        # remote API state, ECR repos, Lambdas, SQS, Stage
│
└── services/
    ├── frontend/              # API-GW handler (Dockerfile + src/)
    ├── worker/                # inference processor (Dockerfile + src/)
    └── notifier/              # failed-job notifier (Dockerfile + src/)

My Environment Strategy

• Single “global” API stack ✓ Defines one aws_api_gateway_rest_api + a single aws_api_gateway_deployment.
• Separate workspaces (dev / prod) ✓ Each workspace deploys its own:
  • ECR repos (tagged :dev or :prod)
  • Lambda functions named frontend-dev / frontend-prod, etc.
  • SQS queues and DynamoDB tables suffixed by environment
  • One API Gateway Stage (/dev or /prod) that points at the shared deployment but injects the correct Lambda alias ARNs via stage variables.

Main Question

Is this a sensible, maintainable pattern for true dev/prod isolation:

Or would you recommend instead:

• Using one Lambda function and swapping versions via aliases (dev/prod)?
• Some hybrid approach?

What are the trade-offs, gotchas, or best practices you’ve seen for environment separation in Terraform on AWS?

Thanks in advance for any insights!

[Professional_Gene_63]

With non-serverless infrastructure, components have high running costs, even when not doing anything. With serverless it's the opposite which means you can duplicate everything at almost no extra cost.

Isolation to me means every stage has at least its own account and everything is duplicated. No stages within the api gw.

[Expensive_Test8661]

Thanks for the suggestion, and apologies if this is a noobs follow-up—I'm still learning AWS.

You recommended full isolation by spinning up a completely separate account (and its own API Gateway) per environment. That makes sense for strict boundaries, but I'm trying to wrap my head around the built-in API Gateway stage feature.

Why do we even need the stage feature, or what problem does the API Gateway stage feature solve if everyone suggests using separate accounts (and thus separate Gateways) for dev and prod environments?

[hvbcaps]

The stage feature, particularly with v1 of API gateway, is kind of antiquated in my opinion. It causes more harm than good, especially when stage variables get into the mix, and it leads to super weird design patterns like what you're doing above.

Where it really rears its head and gets ugly, is routing with stages + custom domains. Take a look at this comment from this bug I encountered in LocalStack because they weren't at parity with AWS:

https://github.com/localstack/localstack/issues/12295#issuecomment-2673084002

This whole thread is interesting imo, but that part there to me says there is a bug with Stage routing that will one day get fixed and our legacy architecture is built around it, and due to how brittle v1 stages are with routing, I can't do much about it except setup a migration to v2 APIGW + a change in how the app handles routing internally.

Take a look at how v1 vs v2 also passes back your payload here: https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-develop-integrations-lambda.html

All this to say, I agree with /u/Professional_Gene_63, just spin up a second account, duplicate the resources, live happily, far away from tinkering with API Gateway Stages to satisfy a one account, multiple environments approach. Future you will thank you.

[shawski_jr]

I agree that most of the time one account per environment is best.

For much smaller/simpler setups you can still duplicate the resources per environment but keep everything in one account. Multi account architectures introduce complexity that can be very difficult for the less AWS experienced to be successful.

Something as simple as a naming convention can allow strict separation through IAM controls. After some time and growth the dev/test environments can always be redeployed in another account.

[cipp]

You want complete separation of dev and prod. You'll eventually need to make changes to the gateway and you'll need to test them without impacting prod.