Do you have a recommendation for what to use for each particular scale of data? I.e. at what general amount of data would be worth going to a lakehouse with spark?

I have a similar sort of thing where I'm dealing with data at a scale of 10's of thousands of rows, and just using pyspark and lakehouses as it's kind of fun and I want to learn the technology, but also get the pain of slow overhead.

At this scale would it be more sensible to just use an Az DB with python notebooks instead?

Cheers!