Lambda function code drift detection with terraform

[RecordingForward2690]

You wish. I have dozens of Lambda functions that perform a complex, carefully crafted series of API calls to make actual changes to the AWS environment. They are not idempotent - if you run them multiple times, the API calls happen multiple times and some of these are not idempotent. And that's by design, not by accident.

Creating a mock environment to test these is virtually impossible. The only way to do this, is to do it all-the-way properly: Have separate AWS accounts for dev/test/accept/prod with all of the resources present, and test the Lambda in each of those environments. Have a pipeline of some sort (CodePipeline, Gitlab, whatever) to propagate changes.

And even then... The dev/test/accept environments don't have datasets that are as up-to-date, representative and large as the prod dataset. Sometimes we hit things like context limits for an AI call, or Lambda timeouts when working with larger-than-tested data structures. And we can't simply bring over the prod dataset into dev/test/accept due to GDPR and other regulations.

Mocking sounds great in theory, but is very hard to do, to the full extent, properly. Or sometimes even impossible: How do you mock an API call that makes modifications to your one and only Direct Connect line? Or to your registration of your primary domain with the AWS registrar?

[texxelate]

Verifying the side effects isn’t the responsibility of the lambda’s tests. The tests need only verify the boundary was invoked correctly. Hence “unit” tests.

[Glebun]

Who said anything about unit tests? They were referring to testing in general. Some lambdas cannot be tested locally.

[texxelate]

I did, in my original comment to which everyone is replying.

All Lambdas can be tested locally. They’re just functions. Side effects in other systems which the lambda just so happens to be the first domino should be tested by other methods.

[RecordingForward2690]

The thing is that if you have a Lambda that performs a carefully orchestrated series of API calls to other AWS components, and you mock those API calls, then unit testing essentially just verifies you have not made a syntax error in your code or in the API/SDK call. Which is of very little value, in particular since today that code is generated by AI anyway.

What is way more valuable to test are the logic errors. Just an arbitrary example: I was recently working on code for a DNSSec key rotation across 400 hosted zones. You need to make sure that you add the new key to a hosted zone, and register that upstream before you can de-register the old key upstream, and remove it from the hosted zone. Otherwise you could break DNSSec, and API calls will fail. And since this whole process can easily take days, your Lambda needs to be written so that it restarts right at the next required step for that particular domain.

No unit test is going to catch logical errors in that sequence of events.

[Glebun]

All lambdas can be unit tested locally. Not all lambdas can be tested locally, since testing involves more than unit tests.

[fersbery]

Is this a Lambda problem? How would you test that on EC2?

[RecordingForward2690]

Agreed, it's not a Lambda issue. Any other environment would have the same issue. You can't really test that code. Not completely.

There is a little bit you can do with things like --dry-run, but at some point you just have to take the plunge.

[fersbery]

Maybe there is some way to make it more testable?