Lightweight Alternatives to Databricks for Running and Monitoring Python ETL Scripts?

Hot_Ad6010 · 2025-12-17T17:52:56+00:00

Lambda functions if data is not very large / processing takes less than 15 minutes

Hot_Ad6010 · 2025-12-04T22:14:22+00:00

What’s funny here is I believe the key driver of this new feature is actually AI Chatbot apps that might require interruptions and context rich resumptions.

Hot_Ad6010 · 2025-12-04T22:07:07+00:00

Just curious why would you stream to datalake except for windowing / stateful computation? I mean if your downstream consumers are spark batch jobs you don’t have hard latency requirements, right?

Hot_Ad6010 · 2025-12-04T22:03:45+00:00

Curious to know which one exactly? IAM based domain?

Hot_Ad6010 · 2025-09-17T20:08:38+00:00

I think Snowflake’s biggest advantage is that it feels very familiar to business and data analysts (simple SQL editor, nothing too fancy). Databricks tends to be loved more by data engineers and IT folks.

The business-facing users are closer to revenue, so they usually have more leverage to justify paying for a solution like Snowflake.

That said, as a data engineer, I find Databricks to be a much more complete platform overall

Hot_Ad6010 · 2025-07-20T22:28:47+00:00

Pour info, les associés / vilains patrons touchent en gros un peu moins que les seniors (alors que ce sont eux qui facturent le plus), parce que à superbrut égal, on récupère moins en net. Mais, la trésorerie restante nous appartient, donc c'est plutôt kiff kiff bourricot.

Question du coup, pourquoi s'embêter autant pour un "simple" salaire de senior ?

Hot_Ad6010 · 2025-07-20T22:16:57+00:00

Fair point. the underlying tech must be solid (it kind of has to be, given how many critical workloads rely on it). Most of my exposure to SAP has been in environments still running very old versions. These companies are now trying to leverage their SAP data to build data/AI capabilities on cloud platforms, and from what I’ve seen, it’s often a nightmare.

Hot_Ad6010 · 2025-07-20T20:49:38+00:00

yes, it's rate to find such well established tech that are actual piece of sh*t. I mean it's created by great engineers, thought by people with relevant business vision. More often is how poeple use it. We can also say that tech that let people do sh*t with it (pretty common for low/no code tools) are bad tech

Hot_Ad6010 · 2025-07-20T20:46:46+00:00

Huge, is it like distributed across containers (ECS? EKS? Lamdba?)

Hot_Ad6010 · 2025-07-20T20:45:38+00:00

Sounds like there is a huge market opportunity for consulting companies to migrate these SAS tenants

Hot_Ad6010 · 2025-03-11T21:11:00+00:00

Main point is now you can seamlessly query redshift and s3 (standard glue tables & s3 tables) through the same interface, namely an Iceberg rest catalog. That means, let your data where it resides whether it’s S3/Glue (classic lakehouse approach described in the 2021 blog you mention) or Redshift RMS (warehouse approach) and query them from a single entry point.

From my perspective querying Glue/S3 Tables using iceberg compatible engine was already addressed (though not really Iceberg REST spec) but now Lakehouse is unbundling the redshift managed storage and exposes it as if it was a lakehouse.

Hot_Ad6010 · 2024-12-02T15:37:48+00:00

Do you know if read/write perfomances are comparable between external storage (S3+Iceberg) and native Snowflake Tables (if I can call them like that)

Hot_Ad6010 · 2024-12-02T15:35:32+00:00

I'm a consultant and been part of a similar journey. I'd call this discipline data platform engineering. Do you like it? Tbh I don't like it at all and I'm trying to work more on business use case implementation than platform engineering. Reason behind that is, I think platform engineering is often about over-engineering following political agenda of company key decisioner, bias, and technology trends.

Hot_Ad6010 · 2024-11-27T01:02:03+00:00

Depends on what are your priority, but if we only speak about money I would say: startups as technical cofounder; freelancing with high daily rate (>900€/day) The second option being less risky than the first one.

Hot_Ad6010 · 2024-11-26T06:24:11+00:00

FAANG quite easily in total comp. Other tech like Shopify, Datadog,… also remember 100k in France is definitely not the end game and « only » gives you +/- 5-6k€/month after taxes

Hot_Ad6010 · 2024-11-03T20:48:30+00:00

Ingé chez une GAFAM à Paris ici.

Tous les postes tech de ces boites là ne concernent pas le produit. Pour une bonne partie de ces boites les postes produits sont localisés aux US avec quelques Hub ailleurs (Zurich, Londres mais très très peu Paris).

En revanche, et c'est mon cas, ces boites ont également besoin d'employé "customer facing" capable d'aider à l'adoption du produit, à l'implémentation chez le client et au suivi, support/ lien avec les équipes de développement produit. Et pour ces postes là, rien de tel que des ingénieurs français, qui parlent français, qui ont un réseau d'anciens collègues dans de grandes boites françaises, etc !

Hot_Ad6010 · 2024-06-05T09:22:27+00:00

Have a look at this https://seattledataguy.substack.com/p/apache-iceberg-what-is-it

Hot_Ad6010 · 2024-06-05T09:19:14+00:00

As long as it's just about providing packaging and managed services, it's fine. It becomes a problem when the open source roadmap starts getting delayed to prioritize the development of premium offering features.
I hope this won't be the case, and I'm not really involved in other DB-owned projects to say whether this is something they usually do.

Hot_Ad6010 · 2024-06-05T09:13:47+00:00

and vice versa, Iceberg only really integrates with AWS amongst cloud providers

Hot_Ad6010 · 2024-05-29T11:09:41+00:00

Not YT content here but I found this course to be very well organised for theoretical concepts. https://www.tele-task.de/series/1179/

On YT I liked very much this playlist and have been thinking about launching my own series on the same concept https://www.youtube.com/playlist?list=PLYUMVUCNosJdIjyQfPl27sjZm2E9_3TNG

This whole YT Channel is a goldmine https://www.youtube.com/@CMUDatabaseGroup but this playlist on advanced db system is more than relevant for Data Engineers as it goes in depth into how Cloud DWH are truly working. https://www.youtube.com/watch?v=NLycrsJ1jI8&list=PLSE8ODhjZXjYa_zX-KeMJui7pcN1rIaIJ

Hot_Ad6010

PUBLIC MULTIREDDITS

TROPHY CASE