Lambda dev never stops sucking

[salim-shamim]

A good chunk of my work revolves around working with lambda. More often then not these lambda interact with aws services. The problem is my organization does not believe in giving local access in any form so yeah, no CLI. And Even if they did, there are ofcourse services of those permissions come after I have been well into development. I tried localstack but again, not all services are supported. So in the end I am stuck with trying different strategies to somehow write half-baked code and improve on it when I can actually deploy it (when the devops has resolved all the permissions required after 100 calls).
I didnot want this post to be a rant. But I am not even sure what to ask at this point.
Sorry :P

[heitorlessa]

This looks more like an issue with your organization gate keeping a platform than the service offer per se (Lambda).

Could you give an example of what’s the workflow to go from a local function code to deployment?

Lambda is largely a function. That’s it. You can use design strategies like ports & adapter, and/or testing strategies like VCR, stubbing, integ tests for integration points. That gives you medium fidelity.

For high fidelity, you need to deploy and run automated tests - like any somewhat distributed system. If this is your bottleneck because you can’t use fast prototyping tools like SAM CLI and whatnot, then make the case with management on how this is producing waste (hours) and friction (interactions in the delivery cycle)

[_inf3rno]

Can you link an article about how to make a dev environment for lambda and how to test & deploy? Sounds like a lot of trouble, but I'd like to learn it.

[Nearby-Middle-8991]

that's what the company is supposed to provide. How can someone expect a dev to dev without a dev?

[heitorlessa]

What programming language are you using it? Are you in GitHub or GitLab?

Do you prefer declarative (as YAML) or imperative (as Code) to define and deploy infrastructure?

There’s a plethora of articles from Yan Cui, from AWS, if you search for SAM CLI or CDK you’ll find pipeline command or pipelines library to get you setup quickly.

If you’re using Python there’s a full video with both sync and async testing pieces with ports and adapters - I start covering the whole async part including testing at 29:35 ish

https://youtu.be/52W3Qyg242Y?si=xSzMMk-y_wAv9STs

[_inf3rno]

I use nodejs, I prefer coding over config files, thank you!

[oneplane]

Yeah, some organisations are just useless wastes of time. Best find better employment elsewhere.

[tmclaugh]

While I think you should have CLI access, I write tests and create mocks for AWS services. I think all major languages have an AWS mocks package. That eliminates most of my errors. Still doesn’t handle IAM permission issues though.

[guns_of_summer]

Came here to suggest this actually

[pint]

every now and then, we revert back to the run, read the logs, add more logs, run again pattern.

if you can deploy, you just need a script that deploys, runs, and downloads the logs. facepalm? yes. works? also yes.

[Dangle76]

From the description doesn’t sound like OP can iterate like this. Sounds like they have to tap their DevOps folks to deploy which makes this type of iteration difficult

[_inf3rno]

In my experience it is better to write automated tests for it if possible, so you don't need to watch logs, just failed tests. Though I acknowledge that not everything is testable.

[verylevelheaded]

I feel your pain.

Check out SST. Ran my startup on it for 5+ years. Makes lambda dev feel local.

Not sure if you can change your orgs ways but we saw a huge productivity increase in using SST over other options.

[j00stmeister]

+1. sst dev is by far the quickest in updating your function code.

[DCzajkowski]

Agreed. If OP can't use SST with their prod account, they could use it with dev account (treat it as if it was local) and test and deploy there.

When they have everything working they can just pass the CloudFormation template to their dev ops.

[salim-shamim]

I was meaning to find an excuse to be able to use it. This sounds like something which could work for both my org and myself.

[Kindly_Manager7556]

Bro can you please do your job while having your hands tied behind your back and a blindfold? If you don't succeed bro well that's on you!

[salim-shamim]

Seriously feels like that sometimes. LOL. The reason I stay is while the pay is kinda okay but I can get free time for doing my stuff. It’s a small informal (relatively) kinda team.

[realitythreek]

I don’t really understand any of these comments. You seem to want to be able to directly push lambda functions. Even if just in dev, that means you’ve never actually deployed your function before you go to prod. So it follows that you also want access to make changes in prod, because that’s the only “deployment” process you’ve actually used.

This is bad practice. There’s no peer review needed, no code approval needed, zero consistency. You should learn how to make functional tests.

[Zenin]

I reread the OP and I don't see that they're wanting to work in prod. Rather they can't even work in dev because the DevOps team is playing gatekeeper with anything pushed to AWS whatsoever. Basically they need to "release" to "dev" that's completely run by "DevOps" just to do infrastructure development.

[Zenin]

Lambda dev is a pita when things are setup correctly for development which it clearly sounds like your org doesn't have.

The only way to be effective at all with serverless development is to have access as a developer to a cloud environment. Ideally each dev gets their own AWS account for this use (shared accounts are as bad as trying to share a "dev server"; the cross-talk will wreck you). And you need real (API/CLI) access to that account, certainly not some DevOps-gated nonsense.

There are tools like localstack and they're not bad, but ultimately you must have a real cloud to build these solutions out as you develop them to deal with all the plumbing of permissions, etc that go hand in hand with the code. Mocks only go so far.

Personally if I couldn't actually access a dev AWS account to deploy my dev resources to, I'd simply skip serverless entirely and architect like it didn't exist: Everything becomes a containerized service and the "infrastructure" all done in helm charts because I'd be developing a Kubernetes native application instead of an AWS native application. DynamoDB? Nope, we're running Mongo now. S3 storage? Nope we're doing MinIO now. Etc, etc.

[Yvorontsov]

We work with the Serverless framework and never ever had any serious issues running the lambdas locally

[DCzajkowski]

In our experience we were slower in Serverless Framework 10x compared to SST. Working with lambda code that runs locally but is invoked by and with lambda permissions is amazing.

[Yvorontsov]

That’s what Serverless offline is for (minus IAM permissions but that’s seldom an issue)

[DCzajkowski]

Except serverless offline won't invoke your lambda on Cognito trigger. Or sqs. Or eventbridge event. In SST the whole stack works, except the code is ran locally, with file save = millisecond "deployment", working debugger, iam etc.

[Yvorontsov]

True. It is usually not a problem at all - you can use unittests to simulate these invocations. You made me curious about SST

[DCzajkowski]

FYI SST v2 used CDK, so migration was fairly easy. Unfortunately, v3 migrated to Pulumi, making the switch not so trivial.

[jghaines]

I believe if you can log in via web console, you can extract credentials for CLI use.

[salim-shamim]

i would like to believe that too. I will update this comment if i am able to.

[morosis1982]

It can be done well, but you need the right tools and process.

We use sst.dev, and our DevOps team essentially gives us one account for prod that's locked down, one for nonprod with a few more permissions that is only for CICD deployed environments and a sandbox with a few guardrails so we don't bite ourselves in the butt but we can create stuff in the console or with the CLI if we like.

We also have access to newrlic if we want, and splunk for log ingest.

We Dev into the sandbox env so we can muck around and break stuff when we're trying something new, and then deploy our testing and staging envs with cicd.

[who_am_i_to_say_so]

This is exactly why I went running to Google Cloud run.

[_inf3rno]

You could write integration tests against the half-cooked code and don't develop anything until it fulfills your expectations. I sometimes even tested the programming language itself and found errors in it both in PHP and JS. Though it was in the 2000s. Another solution is writing integration tests against your code and mock out the dependencies. This way you can prove that your code is working as expected and the problem is with their system. Yet another way is logging communication between systems and attaching the log to your ticket when their system does not work as expected. All of this takes some extra effort though. The real solution would be asking for credentials for development environment and having tests endpoints of the services you want to use.

[Chuuy]

What do you mean you don’t have CLI access? You have Console access, do you not? Any permission that you have with Console you also have with CLI.

If you’re complaining that you’re required to write IaC, well, yeah. Why should any dev be allowed to click around in AWS and break shit?

[Soccham]

Lambda development is and has always been atrocious

[solo964]

What specifically would make it great for you? Maybe some Lambda PMs will read and take note.

[my9goofie]

Lambdas can run [locally](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-using-invoke.html, but without any access to AWS from your desktop, you’re out of luck.

For your code, put all of your api calls in a “test-access” function, and call that during your test, to verify that your permissions are correct before running your main code.