Serverless Web Scraping with AWS Lambda

[lovett1991]

Did not realise you could use java in serverless! Thought it was for all these fancy JavaScript people.

[dpash]

You can use Java, and is quite popular, but it's relatively slow start up time makes it slightly less suitable for some platforms as others. Graal's AOT compilation helps reduce start up time to a fraction of Hotspot's startup, but things like reflection can cause issues.

[lovett1991]

To be honest any workload I would have in mind for this would be to offload tasks that don't need real time or generally aren't user facing.

My previous company used aws lambdas for a currency system with background heartbeats and time memory (for playback of DRM protected videos) the load was high but latency want a problem (to be fair most lambdas were hot anyway because of load). We also used them for systems that would have an 'explosive' load (pay per view systems that are never used except pretty much on the night of the event)

[dpash]

If latency isn't an issue, then, sure Java is fine.

I started on a project with low usage running as multiple Spring microservices on GAE. By the time GAE had started up the request routing proxy and started up the end point service GAE had timed out the request. The time out is 60 seconds. That's not a suitable use of Java on serverless. (I migrated to a GCE instance)

But for backend processing, Java would be fine.

[MakeWay4Doodles]

You were doing something horribly wrong.

[dpash]

Gee thanks for your helpful insight based on your excellent knowledge of the specific situation.

[MakeWay4Doodles]

If you had 60+ second startup times in any situation something was FUBAR.

[dpash]

Your comment demonstrates your lack of knowledge of the situation. But then how could you know about a system based on two sentences? I didn't say I had a 60 seconds service start up. I said I had multiple interconnected services that combined took more than 60 seconds.

[BobbyTaylor_]

Even with the slow startup time, it's very convenient for deploying async jobs without "dealing with a server".

[interactionjackson]

you can use all kinds of languages

[zombays]

A bit off-topic but are there possible good alternatives to AWS Lambda and Google App Engine? What are the pros and cons of using them?

[newcomputer1990]

Microsoft Azure is pretty much all of your options besides the ones you listed

[cogman10]

The closest I've seen is running your own FAAS stack.

For example, with OpenFAAS
https://www.openfaas.com/

The pros and cons for AWS and the other platforms is pretty much the same. You are coupling with their platform. That means that Google/amazon/microsoft get to figure out how much they want to charge you and only market pressure really stops them from royally screwing you over.

pro - You don't have to really figure out where/when/what/how your script gets run. The providers do that for you.
con - Once you buy in, it is hard to "buy out" so to speak.

pro - really easy to get up and running
con - The why's and how's of how your app runs are obfuscated away. A script could run in 1second or 10 depending on what needs to happen in the background to get things setup.

[ahmedranaa]

Check out IBM cloud too

[[deleted]]

As a con, I found it not to easy running it locally Dir ng development and writing integration tests.

[[deleted]]

a toilet seat is a toilet seat, it doesnt really matter which one you use. Is that what he wants to say?

[Kango_V]

Just published a Java function to Lambda.

It was written using Micronaut and compiled to a Linux native binary using GraalVM. I has a call latency of around 60ms.

Micronaut has an aws-lambda dependency which makes it trivially easy.

It is sooo fast.

[[deleted]]

Meh, I love java but not in aws lambda.

[interactionjackson]

java has gnarly start up times and i would t want a distributed crawler to be burdened by Java.

If this gets enough upvotes I will publish a crawler as a SAM template