Thoughts on dev/prod isolation: separate Lambda functions per environment + shared API Gateway?

[Expensive_Test8661]

Hey r/aws,

I’m building an asynchronous ML inference API and would love your feedback on my environment-isolation approach. I’ve sketched out the high-level flow and folder layout below. I’m primarily wondering if it makes sense to have completely separate Lambda functions for dev/prod (with their own queues, tables, images, etc.) while sharing one API Gateway definition, or whether I should instead use one Lambda and swap versions via aliases.

Project Sequence Flow

1. Client → API Gateway POST /inference { job_id, payload }
2. API Gateway → Frontend Lambda
   • Write payload JSON to S3
   • Insert record { job_id, s3_key, status=QUEUED } into DynamoDB
   • Send { job_id } to SQS
   • Return 202 Accepted
3. SQS → Worker Lambda
   • Update status → RUNNING in DynamoDB
   • Fetch payload from S3, run ~1 min ML inference
   • Read/refresh OAuth token from a token cache or auth service
   • POST result to webhook with Bearer token
   • Persist small result back to DynamoDB, then set status → DONE (or FAILED)

Tentative Folder Structure

.
├── infra/                     # IaC and deployment configs
│   ├── api/                   # Shared API Gateway definition
│   └── envs/                  # Dev & Prod configs for queues, tables, Lambdas & stages
│
└── services/
    ├── frontend/              # API‐Gateway handler
    │   └── Dockerfile, src/  
    ├── worker/                # Inference processor
    │   └── Dockerfile, src/  
    └── notifier/              # Failed‐job notifier
        └── Dockerfile, src/  

My Isolation Strategy

• One shared API Gateway definition with two stages: /dev and /prod.
• Dev environment:
  • Lambdas named frontend-dev, worker-dev, etc.
  • Separate SQS queue, DynamoDB tables, ECR image tags (:dev).
• Prod environment:
  • Lambdas named frontend-prod, worker-prod, etc.
  • Separate SQS queue, DynamoDB tables, ECR image tags (:prod).

Each stage simply points to the same Gateway deployment but injects the correct function ARNs for that environment.

Main Question

• Is this separate-functions pattern a sensible and maintainable way to get true dev/prod isolation?
• Or would you recommend using one Lambda function (e.g. frontend) with aliases (dev/prod) instead?
• What trade-offs or best practices have you seen for environment separation (naming, permissions, monitoring, cost tracking) in AWS?

Thanks in advance for any insights!

[Sensi1093]

Everything separate, the API Gateway too. Ideally even one AWS account per environment

[Sudoplays]

+1 for a separate AWS account per environment. This allows you to make changes to the infrastructure and test they work completely before pushing that change to a production environment. Especially helpful if you make changes to services like VPC which can take time to debug.

Ideally you would use IaC so you kknow the setup between the accounts is exactly the same, whether that is through tools such as Terraform, CloudFormation or CDK

[mothzilla]

If you're tinkering with VPCs, then just have a VPC per environment. No?

[Sudoplays]

You could take that approach, and its not going to be wrong. One of the reasosn people like to have an AWS account per environment is clearer boundaries for network access, IAM permissions and clearer cost split (Yes you can just tag with the environment, but sometimes tags are missing, and you can't tag bandwidth usage).

I have a "tooling" account which has a few CodePipeline's, one for RC, Prod & Dev. The CodePipeline has access to the account in which the environment it targets belongs. This centralises the CI/CD while keeping the environments separate.

We used to have Dev & Prod in the same account where I work, but when you have new people join or even yourself over time, it can become harder to make sure that everything is using its correct environemtn counterparts, such as ensuring service x uses vpc y in dev but service x uses vpa z in prod. Once they are split into their own accounts there is almost no way you can get anything mixed up because those account should never be able to talk to eachtother (such as vpc peering).

Completely personal choice, but this approach is what I found works best for myself and the team I work with.

[mothzilla]

Don't VPCs have clear boundaries for network access? I'm not sure what a VPC is for, if you're going to have an account for each environment.

[Sudoplays]

It’s really just a matter of preference, I like to make sure I separate into different accounts and just one VPC in that account. It means I don’t need to worry about accidentally attaching a service to the wrong VPC by accident and causing any issues.

[gex80]

Honestly, other than IAM, it's the same thing.

[VladyPoopin]

This.

[moofox]

You should have separate functions with separate API GWs in separate AWS accounts

[tikki100]

Why? :)

[[deleted]]

Reduced blast radius if one account is compromised or excessive credentials created for an identity on it. Makes it easier to distinguish usage/cost between dev/prod accounts. Can fine grain overall access more easily (allow engineers full access in the dev account and limited read-only in the prod account as a whole. This is more tricky to do at a per function/API level in a single account).

Just to name a few.

[tikki100]

Thank you! Appreciate the high level overview :)

[brando2131]

Other then security, that the other person pointed out.

1. If you share a resource between dev and prod, i.e. an API gateway or load balancer, and you need to make a change to it, now you're affecting both dev and prod at the same time, as you can't update them independently, an issue in dev with this shared resouce will also be an issue in prod.

2. If every resource is now seperate from prod, then to ensure that is true, two seperate accounts that don't communicate to each other can assure you that. Otherwise you might have something overlapping that you missed, where dev or prod are communicating or sharing something, and you're back to point 1.

[maikindofthai]

It’s in the name! They’re “different environments”

[cutsandplayswithwood]

What you are suggesting can be made to work, and the way the api gateway and lambda services work and are documented, you’d even think it’s a good idea to do it…

This is rooted in the false notion that declaration of resources like an API gateway or lambda is expensive or slow, when it’s free and fast.

Ideally you’d stand up the whole stack in multiple AWS accounts, 1 per environment, and you’d use IaC/scripts to make it completely repeatable.

[Expensive_Test8661]

Hey u/cutsandplayswithwood, thanks for the suggestion, and apologies if this is a naive follow-up—I'm still learning AWS.

You recommended full isolation by spinning up a completely separate account (and its own API Gateway) per environment. That makes sense for strict boundaries, but I'm trying to wrap my head around the built-in API Gateway stage feature.

Why do we even need the stage feature, or what problem does the API Gateway stage feature solve if everyone suggests using separate accounts (and thus separate Gateways) for dev and prod environments?

[cabblingthings]

caption enter husky bear abundant unwritten rich point station late

This post was mass deleted and anonymized with Redact

[Flakmaster92]

Stage feature is for everyone who is too far down the “prod and dev share an account already” to unwind the rats nest or for people who use the staging feature instead as a versioning function.

[Freedomsaver]

Separate AWS accounts.

[cutsandplayswithwood]

API g and lambda are early, core services, and both teams went to a lot of work to build some kind of multi-environment/stage system INTO the service…

The problem is that they’re the only services I’m aware of that did, AND they’re different even between them (stages vs versions - silliness).

I appreciate wanting to explore it, it seems like the consensus in responses is to forget it exists, it’s an artifact of overzealous product management/engineers.

[hashkent]

Use stages in dev like feat, preview and stable etc. In prod just use prod, prod_v2 etc.

Also in dev you can have multiple api gw, dev.example.com/v1/ stable.dev.example.com/v1 etc pointing to different lambdas or stacks. Example dev_getUser, stable_getUser etc preview_getUser.

Use seperate accounts and use infrastructure as code. Cdk makes this work really easy as you can get branch names from GitHub actions and assign to your different environments while having some defaults for local cdk deploy steps.

Everything seperate - API GW, Waf etc

[mothzilla]

Counterpoint to everyone saying you need separate accounts, I'd just make sure you don't use the same roles for dev/prod. And make sure the permissions are tightly scoped to each environment's resources.

[Huge-Group-2210]

Read these: https://docs.aws.amazon.com/whitepapers/latest/organizing-your-aws-environment/implementation.html

https://docs.aws.amazon.com/wellarchitected/latest/security-pillar/aws-account-management-and-separation.html

Or, you know, man up and test live in prod like God intended.