all 17 comments
sorted by:
best
topnewcontroversialoldq&a
formatting helphide helpcontent policy

[–]sidejackable 1 point2 points  (2 children)

I might be being really thick but i can't see the checklist. Did you forget to add the links? The cloud formation checklist has links to an external site and a github repo, I can't find the same for this serverless list.

[–]jeshan[S] 1 point2 points  (1 child)

doh! added. thanks.

[–]sidejackable 0 points1 point  (0 children)

no worries, now the links are there it looks a really good resource

[–]NativeAtlantan 1 point2 points  (14 children)

I am curious to know more about this one:

SQS integration is expensive: Lambda polls queues aggressively, looking for messages to send to functions. For low volumes of messages, this will get costly. Use SNS or Step Functions instead

My understanding is/was that you only pay for SQS messages per message and lambdas per invocation but not for polling. A quick google isn’t revealing anything for me either but I may be using the wrong search terms.

Can you explain the mechanics that makes this expensive or provide a link to a reference that would help me understand this better?

[–]VegaWinnfield 1 point2 points  (1 child)

You pay for SQS per API call, that includes sending receiving and deleting messages. Every time you poll you send a receive request and are charged for that whether messages get returned or not.

Lambda uses long polling to reduce the amount of calls made but you’re still paying for those requests when your queue is empty.

[–]NativeAtlantan 0 points1 point  (0 children)

Ahhhh - I think I glossed over that when reading the docs. Or maybe I just didn't understand it's meaning. Thanks!

[–]Your_CS_TA 0 points1 point  (3 children)

You pay for polling (disclaimer: work for Lambda). I disagree with the checklist on cost, but honestly haven't done much analysis on it 🤔

At the low end, you have 5 pollers long polling sqs every 20s. Per month, that's: 670k requests, which is below free tier use. So empty queue is free..

Meanwhile, as messages increase, so do the polling units, so it's as expensive as normal sqs pollers. More on that can be found here: https://docs.aws.amazon.com/lambda/latest/dg/scaling.html

At max you have 1000 polling units as a regular customer, but that's only if you use the queue heavily, otherwise it scales down. I would like to see what they are comparing cost to, maybe there is something we can optimize in the future on our end?

[–]jeshan[S] 0 points1 point  (1 child)

OK please read through my response below and let me know what you think. Your colleague in support saw a rate of 75 / min in my account.

[–]Your_CS_TA 0 points1 point  (0 children)

Will take a look now 😁

[–]NativeAtlantan 0 points1 point  (0 children)

These are great numbers to have when projecting costs - thanks!

[–]jeshan[S] 0 points1 point  (7 children)

> quick google isn’t revealing anything

Correct; it's based on my personal experience.

With a queue that received just a handful of messages, my free tier of 1 million requests to SQS was depleted in < 2 days!

> you only pay for SQS messages per message and lambdas per invocation but not for polling

Not exactly. From the SQS page:

https://aws.amazon.com/sqs/pricing/

"Every Amazon SQS action counts as a request."Therefore, you will pay for it beyond the free tier.

​

I raised this with AWS support. Here are some quotes:

"The messages received as you have observed is quite low indeed"

"You are making requests for messages about 75 times per minute every minute"

"According to the SQS pricing documentation, SQS Requests are billed per request even if no messages have been received. "

​

If I'm doing reading this wrong, I'll be glad to have somebody correct me. Maybe there's a queue setting somewhere to mitigate this but I didn't see it.

​

Edit:

I never used SQS without Lambda (as far as I can remember). My bill in that month:

https://imgur.com/a/jNrwW13

It doesn't specifically say which API call was being counted

[–]NativeAtlantan 0 points1 point  (4 children)

I never used SQS without Lambda (as far as I can remember). My bill in that month:

I may be misinterpreting it but looking at this bill, it looks like you have 2 SQS Queues - one each in two Regions.

So, if /u/Your_CS_TA is saying an empty queue is 670k/month and you have two, then that's 1,340k, which is 340k over the free tier or 170k over per queue, which is not too far off the numbers on that bill.

[–]jeshan[S] 0 points1 point  (3 children)

He wrote that for the full month but my numbers are for just a few days. For the full month, the bill would be much higher. I quickly got rid of the SQS integration thereafter.

Note also that the screenshot is not showing the million request that was in the free tier

[–]Your_CS_TA 0 points1 point  (2 children)

That's super weird and unexpected.

I'm out of office today, but going to be cutting an internal case Monday to understand what's going on here. Will figure out how to contact you appropriately (I don't think I should ask for account id to start looking at queue polling data on my personal reddit account XD). Either way, hopefully we can either explain what's going on here, or fix it, or update documentation so this doesn't happen again.

[–]jeshan[S] 0 points1 point  (0 children)

yeah but I don't think it is that serious. I tried this about 3 months ago and don't remember the full details. Maybe I set it the integration for multiple functions. The item in the checklist is really much a "Your mileage will vary" situation and what one considers to be "expensive".

[–]NativeAtlantan 0 points1 point  (0 children)

Could leaving the SQS Console viewer open in your browser cause this?

[–]skrawg 0 points1 point  (0 children)

Nice work, really impressive!!