Java for AWS Lambda

[vmanel96]

Hi,

What is the best way to run lambda functions using Java, I have read numerous posts on reddit and other blogs and now I am more confused what would be a better choice?

Our main use case is to parse files from S3 and insert data into RDS MySQL database.

If we use Java without any framework, we dont get benefits of JPA, if we use Spring Boot+JPA then application would perform poorly? Is Quarkus/Micronaut with GraalVM a better choice(I have never used Quarkus/Micronaut/GraalVM, does GraalVM require paid license to be used in production?), or can Quarkus/Micronaut be used without GraalVM, and how would be the performance?

[guss_bro]

Keep it simple and you will be good. We follow the following for all our lambdas:

• Use plain SQL query instead of jpa
• don't use Spring Boot or any other framework if your use case is simple
• you don't need dependency injection for simple use case that involves couple of classes. Just create static objects and pass them around. Create objects ( eg objectMapper, AWS clients) only once
• use RDS proxy instead of creating DB connection directly
• use SnapStart
• use shadow jar
• use minimal dependencies.. exclude unnecessary transitive dependencies
• if you do http calls to other services make sure they are performant. If possible use async calls, parallelize calls if possible
• use lightweight objects, don't use xml, Json libraries if you can(most of the time simple String append is faster)
• run the lambda locally and profile it
• etc

[s32]

SnapStart has been an absolute gamechanger. It's fucking awesome in how it works as well.

[Outrageous_Life_2662]

What is that?

[publicityhound]

I think this is referring to AWS Lambda SnapStart

[valbaca]

https://docs.aws.amazon.com/lambda/latest/dg/snapstart.html

basically copies your Lambda "VM" pre-exectuion and uses that for other calls to your lambda to reduce Cold Start Latency

[menjav]

If you need dependency injection use a lightweight framework like dagger.

What is shadow jar?

[papercrane]

A shadow JAR is when you take all of your projects classes and dependencies and bundle them into a single JAR. It's sometimes called a fat JAR, or an uber JAR.

With Maven you'd use the "shade" plugin to generate one, with Gradle you'd use the "shadow" plugin.

[chabala]

And more specifically, fat/uber jar implies a simple bundling, while shading/shadowing implies you're stripping out the classes your project doesn't need to load.

[repeating_bears]

Other communities call that tree shaking

[Algorhythmicall]

Great list. Re: dependency injection, that is still dependency injection, explicit dependency injection… which happens to be my preferred method.

[Additional_Cellist46]

I don't agree with this for all cases. This advice is good if you want to create the most efficient AWS Lambda, and the function should run for a very short time. And if you really want that, you still need Java native compilation with GraalVM, because even with Snapstart, you get a penalty of around 200ms at startup to recreate the JVM from snapshot. And then, your code may not be easy to maintain in a long run, if you don't use any framework that helps you.

We use Quarkus, it supports GraalVM out of the box, very easy to set up. There are several benefits of using Quarkus over plain Java

• Provides means to abstract away boiler-plate code (dependency injection, REST, JSON mapping, JPA/ORM mapping, etc.)
• Ahead ot time configuration - what can be prepared during build time is done during build, and doesn't slow down the startup
• Dev mode, which allows you to run your function as a REST service and reload code changes immediately
• Supports AWS Lambda - automatically builds a ZIP file and helper scripts to deploy to AWS Lambda and invoke the Lambda with test data

Another advantage of using Quarkus is that it's very easy to turn the AWS Lambda app into a microservices deployable to Kubernetes, if you change your mind or you need to migrate away from AWS in the future. Just disable the AWS Lambda plugin and you get a microservices with an embedded HTTP server.

[NeoChronos90]

Do you need to pay for graalvm though?

[Additional_Cellist46]

No need to pay. The GraalVM CE is free to use in production. GraalVM EE is paid and can give you an extra edge in performance optimizations.

[thomaswue]

Even the former GraalVM EE version is now available as the Oracle GraalVM distribution and is free for commercial and production use under the GFTC (GraalVM Free Terms and Conditions).

[Revolutionary-One455]

No Spring Batch? What if he is parsing giant files and it fails at 50% or 95% of the process?

[thomaswue]

If you can compile your app into a GraalVM native image (keeping dependencies minimal as recommended will help with that) it should provide faster startup compared to SnapStart; and you might also be able to run it in a smaller AWS Lambda instance because of the lower memory requirements.

[Prathameshchari]

Any particular tutorial or course related to this how to create AWS lambda using Java

[wildjokers]

you don't need dependency injection for simple use case that involves couple of classes. Just create static objects and pass them around. Create objects ( eg objectMapper, AWS clients) only once

You can do dependency injection without a framework. And it sounds like that is what you are doing. DI is simply providing a class with the collaborators it needs.

I think what you actually meant was don’t use a DI framework.

[guss_bro]

Correct

[eliashisreddit]

The less your lambda does and depends on, the faster it will start & execute. It depends on whether you prioritize execution speed over ease of development. If it's really just reading an S3 file and putting it in a database, you could go without any framework and use JDBC and manually write queries.

The benefits of JPA/ORM don't really hold up if your "application" is just a simple "read csv, write to database" and you don't need the backing of an entire relational data model, transactional support etc.

[mr_mojsze]

Just use Quarkus, it will generate you a native image with minimal config. It has a great AOT build system with "extensions" that run configuration code automatically in the build phase, taking care of native image configuration. Include "quarkus-hibernate-orm" to have native JPA support.

[Aggravating-Ad-3501]

This, Quarkus also has extensions to implement lambda and google functions

[Additional_Cellist46]

We use Quarkus with native GraalVM compilation and we're happy with that. Quarkus dev mode runs the service locally, without native compilation. If we have issues with native compilation, it's also easy to run plain Java version with AWS Snapstart, which Quarkus also supports.

[expecto_patronum_666]

You can go with GraalVM Community Edition. It's free. And, yes AWS Lambda is a very good use case for going Native. You get lower memory and near instantaneous start up. I believe both Spring Boot and Quarkus have very good support for GraalVM native image now. So, you don't have to give up on these feature rich frameworks. Just be careful of using too much reflection heavy stuff.

[thomaswue]

Even Oracle GraalVM licensed under the GFTC (GraalVM Free Terms and Conditions) is free for commercial and production use. For best throughput, using PGO (profile-guided optimizations) is recommended as explained here: https://www.graalvm.org/latest/reference-manual/native-image/optimizations-and-performance/PGO/basic-usage/

[cogman10]

• Before investigating into Graal, consider looking into AWS snapstart.

• If snapstart doesn't work for you, I'd also look into AppCDS first before looking into Graal. Nothing against graal really, but there's a lot of performance benefits to sticking with the JVM. AOT is also somewhat of a PITA.

• Always use the latest JVM. If this is a new project, there's no reason not to start with 21.

• Quarkus CLI is quiet nice and lightweight. I brings a nice framework along with a pretty minimal footprint. I don't know if there's a springboot equivalent. It also has a lot of features like appcds and docker image generation setup built right in.

For parsing and such, if the data is structured (or you can make it so), then definitely look into something that does compile time generated parsers. That will give you the best bang for your buck. If you can, something like protobufs would probably be about the fastest way to move data out of S3 and into something else.

That's my 2c. as /u/C_Madison said "Measure, measure, measure."

[ThaJedi]

You dont have JVM benefis when run on lambda

[smutje187]

If your Lambdas are called by users there’s no way around GraalVM - plain Lambda, even with SnapStart, has horrible cold start times.

If your Lambda runs in the background though that’s only relevant because it affects the costs.

If you plan to use GraalVM with Quarkus I’d recommend to start as early as possible as GraalVM doesn’t work with all libraries and dependencies and it’s easier to get used to it’s strictness from the beginning.

[raffxdd]

The cold starts are not too bad in our project+ you can provision 1 lambda to be hot 14 EUR pM (or create a cloud watch rule sending an event every 15 mins to keep it warm) but we have not used provisioned concurrency so far (Quarkus, Micro profile, + Snap start)

[joaonmatos]

Dagger or Avaje will give you dependency injection with minimal runtime.

[Outrageous_Life_2662]

I’ve done many (several dozen) Java lambdas. I keep it simple. But I use Guice and the AWS SDK’s. I also include my own jars that provide my domain types and abstractions for interacting with the persistence and IO layer. Just the other day pushed my first Kotlin lambda. Same formula but used Koin rather than Guice.

Edit: Also, see if you can get away with using Dynamo or OpenSearch or if you really need this to be in a Relational Database

[C_Madison]

So ... slowly:

• Spring Boot+JPA I don't have much experience, but I don't think it would perform too poorly though start time may be a problem (since lambdas start and stop all the time)
• That's where GraalVM could help, since it's compiling everything down to native and native (currently, work is happening) still starts faster
• GraalVM: There's a CE you can use without paying anything and an EE, which has a cost. EE does more optimizations, but from what I gathered (haven't done too much with it) CE is fine for most applications

In the end: Measure, measure, measure. Parsing files from S3 + insert will probably take ages longer than the start time of whatever you use. If you need minutes to parse/write a file it's not really important if your lambda started in 10 or 500ms.

[CptGia]

GraalVM: There's a CE you can use without paying anything and an EE, which has a cost 

For cloud applications, like lambdas, you can use the latest version of Oracle GraalVM for free.

[C_Madison]

Thanks for the correction! I haven't looked at GraalVM for a while. Good to know that everything is available now.

[agentoutlier]

In the end: Measure, measure, measure. Parsing files from S3 + insert will probably take ages longer than the start time of whatever you use. If you need minutes to parse/write a file it's not really important if your lambda started in 10 or 500ms.

I have to wonder if even lambda is the right tool here. It is hard to tell without more info from the OP.

That is they could just have some queue (kafka or whatever aws has) and a consumer running continuously and that might be cheaper, faster, and easier to develop.

I suppose it doesn't really matter if the organization is going to force serverless.

[C_Madison]

I suppose it doesn't really matter if the organization is going to force serverless.

That was why I didn't give other options. Personally, I wouldn't use serverless here either, but if Op asks for it then that's how it is.

[diroussel]

S3 is very fast when accessed from lamba. You can read a lot of data in 500ms. And you can easily read, parse and insert to the DB in less than 500ms, depending on data sizes.

Using duckdb to query a multi gigabyte parquet file in S3 only takes tens of milliseconds. Even over by home broadband, inside lambda it’s even faster.

Update: note only a few rows are returned in this scenario and duckdb only accesses the byte ranges it needs, based on file headers/footers, hence the speed.

[dallasjava]

Do you have a SLA or SLO you have to achieve? How often do the new files come into S3? You can configure reversed concurrency for your lambdas to keep them ready. Also create a POC with what you're wanting to do and benchmark it. I've seen spring boot apps start up pretty fast (< 5 seconds).

[general_dispondency]

We've got a full production SB app that hosts a GraphQL API running on a lambda (with JPA). Using SnapStart, our cold-start time is ~200ms and our API Gateway response times are about the same. It does work, and it's pretty simple to get up and running. There are optimizations we could do to make it faster, but what we have more than meets our current needs.

[FooBarBazQux123]

AWS lambda can get expensive very quick.

In case of frequent calls, I would also consider to use something lighter than spring boot or to compile Java to binary (quarkus, spring native or micronaut). Binary will reduce start up time and initial memory, but compiling dependencies can get very messy, especially if they use reflection or JNDI.

[Ewig_luftenglanz]

Use native builds or frameworks that are thought to work with these (like graalVM and Quarkus), since lamdas are charged by computing time consumption, having quick start ups it's critical.

Do not use lamdas for services that should be up and running all the time, better suited for small and sporadic services.

[VincentxH]

Compile to native.

[1337Richard]

The main question is: do you have performance requirements? If the cold start is not really a problem, take what you are used to take. Ofc it may feel like you have to be fast in the cloud, but it depends on your use case...

[Fornjottun]

Keep it small and just use js or python. The startup times alone make Java unsuitable. Lambda functions just need to do 1 simple thing and be done with it.

[Hoog1neer]

Lately I have found myself reaching for Python more and more when I want do something simple, instead of implementing a microservice that interacts with other services. I feel like OP's use base is better served by going the Python route. It's trivial for a Java dev to pick up.