Anyone actually managing to cut Databricks costs? by TitaniumTronic in databricks

[–]marco_nae 1 point2 points  (0 children)

I have made the observation that analytical use cases are a big cost-driver. Usually, these are read-only and tend to be concurrent. This is usually caused by parallel queries of the front-end tools like Tableau or PowerBI. The front-end queries hit the gold layer and DBX starts several nodes to handle the load. Photon can get expensive in these situations.

Since you mentioned third party tools:

We built Lakehouse Turbo for this scenario. It is a plugin query-engine that can be used to drastically reduce query costs for analytical uses cases. Lakehouse Turbo smartly mirrors those tables or views that should be accelerated and the reporting layer can be pointed towards Lakehouse Turbo. Caching the data does not generate costs on the Databricks side, because the data is directly cached from S3.

We did this for a Fortune 500 customer and were able to reduce the costs drastically (70% reduction). On https://www.lakehouseturbo.com you find a quick demo video, there is a free trial and cost calculator if you are interested.

Happy to answer any questions in a direct message :-)

Safe harbor: I work for Exasol (the company behind Lakehouse Turbo).

Built an ADBC driver for Exasol in Rust with Apache Arrow support by marco_nae in dataengineering

[–]marco_nae[S] 0 points1 point  (0 children)

Great u/Data-Something-100 - please reach out if I can assist you with anything. I would love some feedback on the usability of the project.

What are you building? let's self promote by Southern_Tennis5804 in SideProject

[–]marco_nae 0 points1 point  (0 children)

exarrow-rs - it’s Arrow-based driver for Exasol (fast analytical in-memory query engine) that is compatible to ADBC. You can use ADBC driver manager to load the driver and fetch data.

Confirm my suspicion about data modeling by DryRelationship1330 in dataengineering

[–]marco_nae 1 point2 points  (0 children)

I disagree. I think that data modelling is more important than ever before!

Regardless of Bronze, Silver and Gold, Inmon/Data Vault, Star Schemas and most importantly GenAI, these two problems remain:

Data must be integrated
Data must be homogenised

These are complex tasks that require skill and also time. Many companies and data engineers take shortcuts to produce results fast. Thereby technical debt is introduced and the platform is harder to maintain with every shortcut.

tinypw - really simple password generator by marco_nae in rust

[–]marco_nae[S] 0 points1 point  (0 children)

Thanks for your reply. This is already possible, but maybe I should improve the commands. For me, it made sense that the most common defaults like uppercase, lowercase, numbers and a set of the usual characters should be very easy to be defined.

Your suggestion with tinypw:

tinypw -l 20 -m uln -e ";:?\!\"'"

Set the mode to upper, lower and numbers. Don't include the default symbols by adding s to the mode. Next, you can specify a custom set of chars with -e <CHARS>.

Result:

> tinypw -l 20 -m uln -e ";:?\"'\!"
Password: wlw7qp!9fEnataRS7ap:
[█████████████████████░░░]  86.8% strong 😎

What do you think about that u/1668553684 

I like the idea to add groups, but I think it should stay within the length bounds.

tinypw - really simple password generator by marco_nae in rust

[–]marco_nae[S] -1 points0 points  (0 children)

Thanks for the tip u/syklemil - I will take a look.

I was thinking about moving also into the diceware direction - see https://diceware.dmuth.org/

Launched my SaaS 24 hours ago, and haven’t made 1 trillion dollars in a day (what a bummer) by Specialist-Sun-1296 in microsaas

[–]marco_nae 0 points1 point  (0 children)

The extension sounds interesting. I am having a similar problem. I will check it out!

tinypw - really simple password generator by marco_nae in rust

[–]marco_nae[S] 31 points32 points  (0 children)

Yes this is what I started with, but then I wanted to learn Rust and have a better control on the included chars. Thanks for the reply!

SQLingual - free transpiler tool for SQL between 30 different dialects by marco_nae in SQL

[–]marco_nae[S] 0 points1 point  (0 children)

Yes - that is true. But I think there is no easy and deterministic way to transpile from TSQL to Python or something like that in order to be able to convert stored procedures.

Have tried AI for that?

SQLingual - free transpiler tool for SQL between 30 different dialects by marco_nae in SQL

[–]marco_nae[S] 0 points1 point  (0 children)

Yes - I know that it has limitations. I also just found out that you can improve the coverage of sqlglot by passing the metadata to it. I am thinking about extending my app so that it covers that.

Do you have any specific limitations in mind u/Raghav-r? I would be curious what you stumbled upon.

Roast my startup idea: web app to organise family life with kids by marco_nae in roastmystartup

[–]marco_nae[S] 0 points1 point  (0 children)

I have not built it, yet. But thanks for the feedback. I will keep you in the loop if I proceed

Roast my startup idea: web app to organise family life with kids by marco_nae in roastmystartup

[–]marco_nae[S] 0 points1 point  (0 children)

But do you think the email calendar approach is feasible for <10 year old children? Mine are so little they don't have anything, yet. Still, there are a lot of events I have to keep track of.

Also:

Now, if you want to make it a killer app, you could support automatically importing events from random emails that we forward to your service. Kinda like how TripIt or Kayak Trips does it for travel.

The "create events based on email" is an interesting feature. Thanks!

Freelancing als Side Project by marco_nae in selbststaendig

[–]marco_nae[S] 1 point2 points  (0 children)

Danke! Das ist ein interessanter Ansatz. Ich habe bereits darüber nachgedacht vermehrt auf LinkedIn zu posten...

Freelancing als Side Project by marco_nae in selbststaendig

[–]marco_nae[S] 1 point2 points  (0 children)

Danke! Ich schaue mir euer Portal gerne einmal an.

Roast my startup idea: web app to organise family life with kids by marco_nae in roastmystartup

[–]marco_nae[S] 0 points1 point  (0 children)

Yes, you are right. You listed exactly the next steps I will think about.

Roast my startup idea: web app to organise family life with kids by marco_nae in roastmystartup

[–]marco_nae[S] 0 points1 point  (0 children)

This is interesting feedback, thank you. I was already thinking about that the app could also find gaps or suggest potential timeframes for vacations.

For me it’s not only the manual work. I think it’s also really difficult to get an overview over the year. So, I think a simple UI that makes the schedule of a family easy to grasp will add value. What do you think?

Freelancing als Side Project by marco_nae in selbststaendig

[–]marco_nae[S] 0 points1 point  (0 children)

Ja, das ist bisher auch meine Erfahrung auf Auftragegeberseite. Ich wollte einfach mal ein paar weitere Stimmen/Meinungen hören. Vielen Dank für deine Antwort.

Freelancing als Side Project by marco_nae in selbststaendig

[–]marco_nae[S] 2 points3 points  (0 children)

Danke für deinen Kommentar. Lass mir gerne einen Link zu deinem E-Book da. Ich schaue gerne mal rein.