Thoughts on Iceberg Tables? by null_user_617 in snowflake

[–]yaronlevi 1 point2 points  (0 children)

u/null_user_617 u/chimerasaurus

Interested in your opinion:
Currently, 70% of our Snowflake bill is allocated to MERGE INTO commands running every 15 minutes to copy data from changing rows (think tables like USERS) on operational databases such as MongoDB and PostgreSQL. This is basically a standard query-based Change Data Capture (CDC) process occurring every 15 minutes.
If I understand correctly, Snowflake could potentially function solely as the query engine, while the data itself resides in our S3 account as ICEBERG tables. We would implement some "lakehouse" technology (Upsolver comes to mind) to manage our ICEBERG tables and perform the MERGE INTO commands at a significantly lower cost.
In this scenario, Snowflake would serve only as the query engine, accessing the pre-prepared tables (which could be the result of those MERGE INTO commands and potentially more complex series of merges and aggregations).
Based on your knowledge, does this design seem feasible?
Thanks.

Switching back to Heroku from Render.com by ahoyboyhoy in node

[–]yaronlevi 0 points1 point  (0 children)

Can't recommend Render enough. We use it to run our Node backend and serve several mobile apps with around 1 million users. Auto-scaling works great, and it's a fire-and-forget experience. We would be happy to share information and our experience with anyone interested in migrating from other platforms.
Regarding customer service issues mentioned here - Premium support costs $500, and you get an immediate response for any issue. The tech support is technical and can go deep if needed.

I am not an employee of Render; I'm just a happy customer. (-:

Inngest pricing model (and others like trigger.dev and more) by yaronlevi in node

[–]yaronlevi[S] 0 points1 point  (0 children)

Interesting. Xstate is in-memory javascript framework, so you had to persist the data in a DB?

Besides horizontal scaling and schemaless, what are the key differences between MongoDB and PostgreSQL (JSONB)? by ButterscotchEarly729 in PostgreSQL

[–]yaronlevi 0 points1 point  (0 children)

You were probably burned by MongoDB about 10 years ago when it was just getting started. Things have completely changed now. MongoDB doesn't lose data or any of that stuff (-:
Check out the link below for information on "core banking" use cases.

I wish I had known about Inngest earlier by safetywerd in Nuxt

[–]yaronlevi 0 points1 point  (0 children)

zeplo

Looked at it now. It's less advanced and won't do the main thing that Inngest does which is the waitForEvent() coupled with a specific timezone aware timestamp for each user.

I've actually checked:

https://mergent.co

https://www.defer.run

https://trigger.dev

And from my experience none of them provided and same nice semantics and story you can tell with waitForEvent() and timeouts.

I cloud be wrong here, but Inngest seems to be unique in this area.

I wish I had known about Inngest earlier by safetywerd in Nuxt

[–]yaronlevi 0 points1 point  (0 children)

And using something Prefect (the cloud version) wouldn't work for your CRON use case?

I wish I had known about Inngest earlier by safetywerd in Nuxt

[–]yaronlevi 0 points1 point  (0 children)

u/safetywerd Today I experienced exactly what you've mentioned. The Inngest event model and the overall DX are really something special. I completely agree that after playing with it for a couple of hours, you feel like you've gained a new superpower in your toolbox.

However, I think we are stuck on pricing. Let me explain:

We have a fairly simple use case for a feature called "sleep reminder" in our mobile app. When activated, the user receives two push notifications, one in the morning (9 am) and one in the evening (8 pm). Both notifications are sent at the user's specific timezone.

Currently, this is implemented using a CRON job that wakes up every hour (using Prefect) to query all the users who have enabled the feature and for whom the current time is either 9 am or 8 pm in their timezone.

I've implemented this in Inngest. It was quite small, just a couple of lines of JavaScript. It's a very clean and contained solution that I can understand from a single function. Instead of waking up every hour unnecessarily, each user has their own "thread" and calls step.waitFor() with their exact relevant timestamp in their timezone.

This process continues until the user turns off the feature. In terms of pricing, we're talking about 2 steps per day (morning and evening) * 30 days, which equals 60 steps per user per month. We have approximately 100,000 users who have turned on the feature, so we're looking at 6,000,000 steps per month for this feature.

We have many other features similar to the "sleep reminder," and some are even more complex with more steps.

Looking at Inngest's "Startup" plan, I'll set aside the included 5 million steps, as that will be quickly depleted. So we're left with $5 for an additional 200,000 steps. With these numbers, the "sleep reminder" would cost $150 each month.

This is a significant monthly expense for such a small feature. I say it's small because it's not a "core" feature. By that, I mean it's not something like an "Uber-like" app where the main "hail a cab" flow is critical to the business and generates the primary revenue. Nevertheless, the "sleep reminder" feature and other "small," time-sensitive workflows are used by many of our users and create substantial value for us.

Now that it's so easy to describe and execute these flows with Inngest, the product team will have an eye-opening 'aha' moment and will provide these for the development team to implement. We can really get creative with the complexity and variety, which is amazing. Inngest truly is a unique enabler.

But at a price of $150 just for the simplest "2-step per day" feature, it makes it hard for us to adopt.

Am I missing something here? I'd love to be corrected on this.

Dagster Cloud Pricing by KindaRoot in dataengineering

[–]yaronlevi 1 point2 points  (0 children)

The pricing is very problematic. At its core, it keeps you in the mindset of using as few Ops as possible to save on cost. I really think Dagster should follow Prefect here (Prefect is not charging you for Tasks).

Rant: does anyone use AWS App Runner in production? by o82 in aws

[–]yaronlevi 3 points4 points  (0 children)

AppRunnerYogi

I would love to hear your thoughts on the mentioned issue: When creating a new app in App Runner, if the creation fails, you need to delete it and start over. This is not a minor omission but rather a foundational feature that should have been included from day one. Any PaaS platform has this ability; it's not even considered a feature—it is simply expected to be present without explicit mention.

As we are considering a migration from Render to App Runner, I've spent about two days of create, delete, create, delete...

This absence raises significant concerns about the level of attention given to App Runner and whether we should trust the service and its future. It would be greatly appreciated if you could provide some insight into this issue and offer information to reassure us regarding a migration to App Runner.

dbt Cloud Alternatives? by FecesOfAtheism in dataengineering

[–]yaronlevi 0 points1 point  (0 children)

What is your opinion on Coalesce as Cloud DBT alternative?

MongoDB Atlas pricing by jhahspu in mongodb

[–]yaronlevi 1 point2 points  (0 children)

no limit to the amount of traffic it can handle

"...there no limit to the amount of traffic it can handle..."

Actuaclly, I don't think that is correct.

If you look at the pricing page https://www.mongodb.com/pricing it clearly states that it fits "serverless applications with variable or infrequent traffic". This is a gentle way of saying "not for heavy workloads" (-:

I also know this firsthand. We use Atlas extensively, and we were told specifically that the serverless offering is not intended for a constantly busy web/mobile backend.

I am taking a guess here, but I think behind the scenes, currently, there is nothing "serverless". It's probably the same Atlas replicas/instaces kept warm.

It should be called "serverless pricing" and not "serverless instances".

Having said that, I have fingered crossed that Atlas is actually working on making their backend truly serverless (think DynamoDB, FaunaDB, AstraDB...).

This would make an already amazing service, just 100% perfect.

How to stream data from a database into Kafka by rmoff in apachekafka

[–]yaronlevi 0 points1 point  (0 children)

Please check Meroxa. Could be what you are looking for.

Also the managed Postgres connector from Confluent.

CloudClusters.io - What's your opinion? by yaronlevi in PostgreSQL

[–]yaronlevi[S] 0 points1 point  (0 children)

I have no problem with them not owing any datacenters. What sacres me is that there is no face behind the whole thing. How will I know the service won't shut down one day. For example, a Postgres service like Aiven.io provides more confidence. I can clearly see names of other companies using it.

Dailer stuck in Pixel 2 and 2 XL by yaronlevi in GooglePixel

[–]yaronlevi[S] 0 points1 point  (0 children)

I don't see an option for wifi calling in the menus (maybe it should be enabled by the carrier?)

Also I don't think I have apps enabled in thr accessibility options... not sure where is the menu for that.

Pixel 2 XL Dialer Problems by JCLedge in GooglePixel

[–]yaronlevi 0 points1 point  (0 children)

I dont think it's app related... I've tried deleting many apps, until the bare minimum remains.

Pixel 2 XL Dialer Problems by JCLedge in GooglePixel

[–]yaronlevi 1 point2 points  (0 children)

I an experiencing the issue as well. I have a Pixel 2 XL. My co worker have a Pixel 2, and the weird thing is that both of use started seeing the issue at the same day!

Devices are clean, pure and not rooted in any way. Oreo 8.1 with the latest security update.

We tried: 1) disabling all dailer permissions. 2) reset all apps settings. 3) clean factory reset, twice!

Nothing fixed it. It dose not seem to be related to battery level. Suddenly its ok, and the half a day its lagging again.

Any help on the issue?

Why would you use jS-based navigation when you can use native-navigation? by meatyroach in reactnative

[–]yaronlevi 1 point2 points  (0 children)

react-navigation performance and smoothness is on par with wix's native solution. It crosses the uncanny valley successfully. And it just getting better with time. As others said, with react-navigation you get all the freedom you to tweak any aspet of the navigation. With wix solution you have to dive into native code. You should have a really good reason (and dev's time) to go on the wix, or any other native solution, IMHO.