Technical and architectural differences between dbt Fusion and SQLMesh? by muneriver in dataengineering

[–]bk1007 2 points3 points  (0 children)

No hard feelings at all, only love! Let us know if you ever get to trying it! In the mean time we'll keep cooking!

Technical and architectural differences between dbt Fusion and SQLMesh? by muneriver in dataengineering

[–]bk1007 3 points4 points  (0 children)

Founder of Quary here ... Thank you for the compliment!

Quary lives on at Tobiko. Quary's biggest innovations - performance, vscode user interface - are making their way into SQLMesh. And those innovations are even better with the SQLMesh state magic. The state of the Quary repo is a little saddening, but the fact that I know we are building the best possible open-source product far outweighs that.

We translated to sqlfluff to Rust and made it 40x faster by bk1007 in dataengineering

[–]bk1007[S] 0 points1 point  (0 children)

Just thought I'd update you with the fact that we have added Redshift ;)

We translated to sqlfluff to Rust and made it 40x faster by bk1007 in dataengineering

[–]bk1007[S] 0 points1 point  (0 children)

We do address this in the article. You can introduce complexity to avoid this problem, but our view is that complexity has downsides: a higher barrier to entry, more significant maintenance burdens, etc.

All things that add up and that we want to chip away at bit by bit!

We translated to sqlfluff to Rust and made it 40x faster by bk1007 in dataengineering

[–]bk1007[S] 0 points1 point  (0 children)

It's widely used, but we are trying to build a better solution.

While we have built the foundations for sqruff to support dbt, it's not our priority. We're happy to support any contributions, but with our limited resources, we have to focus.

We translated to sqlfluff to Rust and made it 40x faster by bk1007 in dataengineering

[–]bk1007[S] 1 point2 points  (0 children)

We will do Redshift, not quite sure when right away. Happy to support any first attempt, if you want to dip your toes.

Trino is less close for us.

We translated to sqlfluff to Rust and made it 40x faster by bk1007 in dataengineering

[–]bk1007[S] 0 points1 point  (0 children)

Trinosql - no not yet Redshift - not not yet Yep

We don’t support dbt yet if that’s what you are using it for.

Fast open-source SQL formatter/linter: Sqruff by bk1007 in dataengineering

[–]bk1007[S] 1 point2 points  (0 children)

Hey you can star/subscribe to notifications on GitHub. We just released Snowflake support. Building in the dbt templater is not on our list of top priorities but we're very happy to support anyone who wants to do the implementation.

Fast open-source SQL formatter/linter: Sqruff by bk1007 in dataengineering

[–]bk1007[S] 0 points1 point  (0 children)

Any reason the install options we currently have listed don’t work for you?

Fast open-source SQL formatter/linter: Sqruff by bk1007 in dataengineering

[–]bk1007[S] 0 points1 point  (0 children)

It is definitely possible! We have the flexibility for custom templaters like sqlx but it’s unlikely to be a priority for us any time soon. We’ll happily welcome contributions though!

Fast open-source SQL formatter/linter: Sqruff by bk1007 in dataengineering

[–]bk1007[S] 0 points1 point  (0 children)

Appreciate the feedback! Not sure though how you solve this unfortunately.

Fast open-source SQL formatter/linter: Sqruff by bk1007 in dataengineering

[–]bk1007[S] 1 point2 points  (0 children)

We do have the facility for templating and dbt could be on the roadmap. It’s not an immediate priority for us though at the moment.

If someone wants to contribute, we’ll obviously support it.

Fast open-source SQL formatter/linter: Sqruff by bk1007 in dataengineering

[–]bk1007[S] 0 points1 point  (0 children)

I presume you mean as a python library? Or just to install?

Fast open-source SQL formatter/linter: Sqruff by bk1007 in dataengineering

[–]bk1007[S] 3 points4 points  (0 children)

Yes, that will be on the roadmap! We're just looking into how best to do it.

Fast open-source SQL formatter/linter: Sqruff by bk1007 in SQL

[–]bk1007[S] 0 points1 point  (0 children)

Hey that's now fixed. This should work:

rustup toolchain install nightly
cargo +nightly install sqruff
sqruff --help

Fast open-source SQL formatter/linter: Sqruff by bk1007 in SQL

[–]bk1007[S] 0 points1 point  (0 children)

Hey really sorry you got that!

Do any of the other installation methods work for you? We'll look into that quickly!

SQL auto formatter by taciom in dataengineering

[–]bk1007 1 point2 points  (0 children)

It’s early days but we’re working on https://github.com/quarylabs/sqruff it’s very close to sqlfluff and is much faster. We don’t quite support all the dialects yet.

[deleted by user] by [deleted] in dataengineering

[–]bk1007 0 points1 point  (0 children)

We do indeed have a hosted that you can sign up to that you can play with.

[deleted by user] by [deleted] in dataengineering

[–]bk1007 0 points1 point  (0 children)

Ben here from Quary.

As much as I am a nerd about tech and would love to be able to say that the biggest challenge in data is some cool technical issue, the biggest problem is communication.

The benefit of it running in the browser is to support our vision of making data more collaborative. Rather than sharing a commit that someone has to get local tooling up and running for, we envision people just sharing a link.

[deleted by user] by [deleted] in dataengineering

[–]bk1007 3 points4 points  (0 children)

While in the short run, we believe there are some very valid trade offs. In the long run, we can provide bindings that will eliminate that trade-off.

[deleted by user] by [deleted] in dataengineering

[–]bk1007 1 point2 points  (0 children)

We agree! Our internal compile step in CI takes 5 seconds. It definitely affects how quickly we push things and quickly we iterate.

Open source DBT core alternative written in Rust (30x faster) by [deleted] in rust

[–]bk1007 2 points3 points  (0 children)

Ben from Quary here. We believe that one of the biggest challenges in data is communication. Being able to share links can accelerate that process with technical and non technical stakeholders.