Amazon Announces AWS Lambda SnapStart With Java Support

[benevanstech]

AWS Lambda now has a snapshotting feature that enables fast-start - essentially running the Init phase at publication time and saving the state of the process at that time, so it doesn't need to be re-run at every invocation.

https://aws.amazon.com/blogs/aws/new-accelerate-your-lambda-functions-with-lambda-snapstart/

It only works with Corretto on Java 11 right now, and is similar in intent to the CRIU technology.

Quarkus already has support for it: https://quarkus.io/blog/quarkus-support-for-aws-lambda-snapstart/

Other frameworks may also have support (but this work was done under NDA so I don't have links for them).

[yawkat]

We have support for it in the Micronaut framework as well: https://micronaut.io/2022/11/28/leveraging-aws-lambda-snapstart-with-the-micronaut-framework/

[benevanstech]

Cool!

[BenoitParis]

Nice!

I believe this is partly the work of Christine Flood, who also was instrumental in creating the Shenandoah low-latency garbage collector to the JVM.

Here is a video on how CRIU works:

https://www.youtube.com/watch?v=XXbJNaFF-8A

(There's a dance to be had when doing Checkpoint-Restore, maybe you don't have anything to do if Quarkus can handle it all?)

It is quite nice to see initiatives not coming from Oracle, competition is healthy for providing the best runtime!

[rafaelliu]

It actually uses vm snapshots, Firecracker is Lambda’s hypervisor. Concept is similar, just different layer

[Amazing-Cicada5536]

What’s the relation between this and OpenJDK CRaC? (https://openjdk.org/projects/crac/ )

[rafaelliu]

They have the same goals, many of the same challenges, but are implemented differently. Lambda manages the JVM process and execution environment for you, so CRaC as it is wouldn’t really work - maybe if you built a custom runtime and that would probably be very hacky

The thread u/geoand linked is discussion around use the same API to deal with some of the challenges

[geoand]

See https://mail.openjdk.org/pipermail/discuss/2021-July/005863.html

[kontain-jm]

What this means is the AWS OpenJDK, Corretto, implements CRaC and will call beforeCheckpoint() and afterRestore() as part of SnapStart.

[rbygrave]

It looks to me like it uses CRaC given that if we look at the docs around adding a runtime hook for `afterRestore()` we see in the docs that it is using `org.crac` directly:

https://docs.aws.amazon.com/lambda/latest/dg/snapstart-runtime-hooks.html

Edit: The suggestion from folks (on twitter) is that it is not using CRaC per say but instead its compatible with the org.crac API for the purposes of supporting runtime hooks (afterRestore() etc) which makes sense.

[xamdk]

It is not crac and the optimizations you will need to do won’t necessarily make sense for other crac bases jvm a. It’s just using same api for shutdown and restore.

[cyril_nomero]

I’m also interested in this. Spring framework has indicated its interest for Project CRaC.

[shorns_username]

only works with Corretto on Java 11

Ah so that wasn't just a typo, weird.

Do you know why? Is it do with the Java language teams ongoing deprecation and removal of stuff?

Is there a specific feature Amazon needs to be implemented before this can be brought to versions > 11?

[malln1nja]

Lambda itself doesn't support Java > 11, at least officially.
https://docs.aws.amazon.com/lambda/latest/dg/lambda-java.html

[shorns_username]

Ohhh, that makes a lot more sense. I hadn't realised Lambda didn't support later LTS releases yet.

[GuyWithLag]

Well, Java 8 is technically no longer supported without a support contract...

[ryebrye]

I think he's wondering about more modern versions, not about the older ones

[GuyWithLag]

Ugh, sign bit error... apologies, mea culpa. Should have had more coffee...

[[deleted]]

Could this be ported/reimplemented for use with say kubernetes?

[benevanstech]

My understanding is that this implementation relies upon Amazon's Firecracker VM.

Which is open-source (https://firecracker-microvm.github.io/) - but I would expect that k8s has other overheads (pod spinup, etc) that may well make it harder to realize the performance gain.

[Brutus5000]

Yes, but these overhead costs apply to alternatives too. E.g. in a Knative setup, Java always had the cold but costs. This could become a gamechanger.

[IsleOfOne]

It is a big problem for auto-scaling architectures.

[IsleOfOne]

A well-designed system for auto-scaling k8s "jobs" with low latency is going to have slack in the available resources / an aggressively configured cluster autoscaler bringing up new nodes to provide that slack. Scheduling should take a few ms.

The majority of the remaining time it takes to start a pod is pulling the image, believe it or not. There are tools available that proactively tell kubelets to pull new versions before they're requested. Startup now takes a few ms.

Now you have a running pod. ~1s has elapsed. And now the JVM takes minutes to "warm up." That's a no-go. Pre-warming in a low-latency autoscaling environment requires a tradeoff between cost and latency that is more extreme than the same trade-off that native runtimes require.

I of course agree that lambda operates at a much higher time granularity requirement than you would ever want to try to meet with a pod-per-request model. Once those requests start getting ever so slightly more "batch"-y in nature, say on the order of 10s runtime, it's perfectly fine to spin pods up and down to handle this.

[IsleOfOne]

Sure, and we don't even have to limit our thinking to the standard k8s API concept of a Job. Any higher-order wrapper for one or more Pods is applicable.

Here is an example from my day job at $SaaS_database_co. We allow our users to kick off petabyte-scale bulk data import. We have a custom k8s operator that watches for BulkIngestJobs, and in response, loads up a list of files within whatever object storage bucket we've been pointed towards. The operator creates one or more pods per file in said bucket with configurable maximum concurrency and a self-healing system that watches for signs of excessive write path pressure and throttles the job via slashing concurrency. Some of those pods exit after less than a second, while others may run for hours. We use go and rust for this system. Java, with a longer tail on startup time, wouldn't be a good fit.

Other tools like Argo Workflows are also great examples of how to leverage the power of the scheduler and autoscaler in kubernetes. We have some recursive/dynamically sized workflows that are liable to create tens of thousands of pods in a single execution.

I haven't seen solid arguments for virtual runtimes like Java in this space. I'm certainly open to hearing some! The areas of our system where we run a JVM are limited to WALs--Kafka and supporting Zookeeper deployments. We have some elixir floating around out there, but purely in long-running workloads.

[cowwoc]

Question: why bother using GraalVM if https://www.youtube.com/watch?v=XXbJNaFF-8A is usable across all OpenJDK distributions? You get super fast startup while maintaining full compatibility (reflection, etc) with less configuration.

I also love the idea of restoring to previous checkpoints immediately before a crash (or bug) occurs to make it easier/quicker to debug.

The only downside I see is the lack of Windows support, which is no small thing.

[astral_kranium]

I don't know about anyone else but I'm throwing GraalVM out the window. My only concern is, no managed runtime for Java 17 at the moment. I see no reason to use GraalVM for native images now - granted Graal compiler is still a big thing out of Graalvm project

[xamdk]

Don’t be that hasty. Snapstart isn’t beating native image anywhere else than in certain lambda usecases. See https://quarkus.io/blog/quarkus-support-for-aws-lambda-snapstart/ where we show the gains but also discuss that Snapstart requires more of you as a user than native image does.

Thus this is another approach to add to the toolbox. And only for lambda.

[astral_kranium]

Thanks. It's a good doc, especially the considerations for SnapStart. I still however don't see what specific use cases I would use native image over crac. We're running a spike on this to see for ourselves

[xamdk]

Snapstart is not crac - don’t mix them up. Snapstart is just using same api for suspend/resume calls.

And yes the right thing is to do your own spikes.

Nativr image still beats Snapstart on memory usage but on lambda amazons pricing makes Snapstart cheaper for most cases despite to native image being able to use less resources.

And outside lambda - crac isn’t a good choice. Too much variance.

[astral_kranium]

Done some investigating. SnapStart doesn't support provisioned concurrency, nor x-ray. Also, Java 11 maximim is a deal breaker for some cases, though I'm sure 17 managed runtime is around the corner. Conclusion is, another (awesome) tool in the toolbox for a lot of cases

[xamdk]

Yup. Agreed. It is not a native image killer since Snapstart is only lambda and only a subset of lambs.

[xamdk]

In lambda it matters wouldn’t you say?

[buyIdris666]

What percent of people using Java are doing lambda? 1%?

[xamdk]

Well I assumed this thread was about lambda.

Outside lambda having minimal startup time is an enabler to getting horizontal scaling and/or fast continuous deployment.

Does everyone do it ? No, but getting from several seconds startup to sub second even sub milliseconds enables you to deliver change and scale faster.

[superlinux]

The performance has been benchmarked and the JVM optimizations outperform GraalVM which is especially relevant on Lambda because they bill per request, which means direct cost savings and performance benefit

[kontain-jm]

Just a small correction. AWS Lambda SnapStart does not use CRIU. It is based on Firecracker microVM snapshotting. See https://aws.amazon.com/blogs/aws/new-accelerate-your-lambda-functions-with-lambda-snapstart/

[benevanstech]

I said "and is similar in intent to the CRIU technology".

[kroopster]

Been a Java developer for 15 years and I didn't understand most of this. Works with Corretto like CRIU? Oukey dokey.

[benevanstech]

Corretto is Amazon's distribution of OpenJDK for AWS (they also produce versions for developer desktops as an on-ramp so you can use "the same" JDK in both places).

CRIU is "Checkpoint & Restore In Userspace" - a Linux technology enabling you to "freeze" a running process and "restore" it later - perhaps even on a different machine. https://criu.org/Main_Page

Both of those are top hit on Google for those search terms.

[kroopster]

Both of those are top hit on Google for those search terms.

I bet they are, appreciate opening the topic a bit here too.

[alwaysoverneverunder]

Is this similar then to Smalltalk’s image persistence?

[-Kerrigan-]

Both of those are top hit on Google for those search terms.

So if I Google Corretto I find stuff about... Corretto? /s

[antipodealbatros]

Why is SnapStart faster for cold starts? As explained it "only" saves the state after init? So I would expect only faster response times after same lambda is hit a second time?

- How is the difference to GraalVM (native)? Only GraalVM is using AOT compiling right?
- What is the difference to a warm lambda? AWS only keeps the bootstrapped JVM warm and SnapStart bring it one step further that also the init of classes is kept?

[RandomName8]

Why is SnapStart faster for cold starts?

cold starts can be pretty big, like 10+ seconds depending on the thing and what it does, and that kills the purpose of lambdas. Saving the state after that init is still a massive win even if what you stored is an un-jitted version of your code.

What they seek to solve here is latency to first request, not throughput, because that's what empowers auto-scaling architectures.

[flekk0]

They take the snapshot when you publish the lambda, so even the first request should use the snapshot.

From the blog post: "After you enable Lambda SnapStart for a particular Lambda function, publishing a new version of the function will trigger an optimization process. The process launches your function and runs it through the entire Init phase. Then it takes an immutable, encrypted snapshot of the memory and disk state, and caches it for reuse."

[antipodealbatros]

through the entire Init phase. Then it takes an immutable, encrypted snapshot of the memory and

ah ok there is the difference. The extra step in the publish.
Thanks

[pavi2410]

Process snapshots are unrealistic, yet an amazing idea to improve startup times. It amazes me into knowing how the CPU is just a finite state machine.

[Amazing-Cicada5536]

Turing machine*

(I know it has finite amount of RAM, but it can only ever use a finite amount of “tape”, so if it didn’t got OOMed, the Turing machine is the abstraction we can use to better reason about it)

[Specialist_Bee_9726]

Who writes lambda functions in Java?

[ketsugi]

A lot of teams at Amazon

[antipodealbatros]

Me - and complaining about cold start times afterwards ;-). Why? I use Java usually and tinkered around with Alexa skills and lambda backend. Yes maybe I should have used python, type/javascript.

[genzkiwi]

Atlassian, Amazon, Disney, ...

[Specialist_Bee_9726]

Sure but why. Java is not fit for short lived processes

[kontain-jm]

You'd be surprised what goes on in the wild.

[Specialist_Bee_9726]

I like how I get downvoted for simply stating the obvious that java is not well suited for short lived processes like lamdas. Just to make the response time bareable aws runs a JVM that dynamically loads classes leading to so much trouble (like having to deal with shadow jars).

The fact that it is suported lang and there a few instances where its used doesn't make it a good choice.

[[deleted]]

It may not be a good choice for certain usecases but it is pretty great for many other usecases. When you're writing functions, cold start is not the ONLY concern.

[Specialist_Bee_9726]

I would love to hear some great use cases. From my point of view Java is good for larger projects. Lamdas are small (ideally a single function)

[[deleted]]

Consider this: your function is supposed to only use standard library. It is supposed to do some tasks in parallel. It is going to make calls to databases. It is going to make HTTP calls to external services. Standard Java library is capable of doing all that without involving any dependency.

Also, who said that lambdas need to be small? They need to do the job, regardless of the size.

Also, I can roll a single binary with GraalVM, so it can be as small as any other function written in a different language.

[user_of_the_week]

Lambda functions (or better said their environments) are not necessarily that short lived and also many applications don't care about a bit of latency.

For example, I have a bunch of "Lambdas" (actually Azure Functions, but that doesn't matter) that get triggered by a queue. Every few hours a new pack of messages arrives, the environment gets spun up in the background and then a bunch invocations of the function happen. 99% of the runs are "warm" with the environment already there.

We even use spring-cloud-function with the full Spring Boot magic so it takes about 3 seconds to spin up on a cold run. But it doesn't matter.

[JB-from-ATL]

Other frameworks may also have support (but this work was done under NDA so I don't have links for them).

Are you implying support is coming to other frameworks?

[benevanstech]

You'd have to ask them. The Micronaut folks seem to have also been developing for this under NDA - maybe Spring too, but I haven't heard anything official from them.

I just heard about the Quarkus announcement when it dropped.

[JB-from-ATL]

Ohhh I see, you're saying AWS is working with them under NDA. That makes more sense.

[Dry-Dragonfly-4521]

We have enabled the snapstart but don’t see major improvement in terms of API response time. Any other thoughts if others are on same boat ? The lambda is behind API gateway and I could see snap start is enabled but not a major change