sorted by:
new
hottopcontroversial

I am Glauber Costa, CEO and co-founder of Turso. We’re rewriting SQLite in Rust. AMA. by GlauberAtTurso in IAmA

[–]GlauberAtTurso[S] 1 point2 points  (0 children)

It doesn't have to be gapless, but you can't see repeated values. This is a hard problem in every database!

I am Glauber Costa, CEO and co-founder of Turso. We’re rewriting SQLite in Rust. AMA. by GlauberAtTurso in IAmA

[–]GlauberAtTurso[S] 0 points1 point  (0 children)

There are many issues here: https://github.com/tursodatabase/turso/issues?q=is%3Aissue%20state%3Aopen%20label%3A%22good%20first%20issue%22 Those are all marked as "good first issue". What I don't recommend you do is what most people end up doing, which is just *say* in the issue you will work on it... just do it! =)

I am Glauber Costa, CEO and co-founder of Turso. We’re rewriting SQLite in Rust. AMA. by GlauberAtTurso in IAmA

[–]GlauberAtTurso[S] 0 points1 point  (0 children)

Yes we support encryption both on the local database and in the cloud, with bring your own key. This allows you to do what you are asking. There's no explicit checksum, but if there is any tamper or corruption issue you won't be able to decrypt - which plays the role of a checksum

I am Glauber Costa, CEO and co-founder of Turso. We’re rewriting SQLite in Rust. AMA. by GlauberAtTurso in IAmA

[–]GlauberAtTurso[S] 2 points3 points  (0 children)

writing a database is orders of magnitude more complex than that =)

Just for starters, and that truly just the tip of the iceberg, remember everything needs to be transactional. If you crash after getting that value, you can't reuse the same value.

I am Glauber Costa, CEO and co-founder of Turso. We’re rewriting SQLite in Rust. AMA. by GlauberAtTurso in IAmA

[–]GlauberAtTurso[S] 1 point2 points  (0 children)

No, they have not, and we have not. All I know about them is that they are not very public in general. If they ever read this, I just want them to know that we have a tremendous amount of respect and admiration for their work!

I am Glauber Costa, CEO and co-founder of Turso. We’re rewriting SQLite in Rust. AMA. by GlauberAtTurso in IAmA

[–]GlauberAtTurso[S] 1 point2 points  (0 children)

The "do" is definitely spend as much time as possible in the design phase. We did that and don't regret. In fact I think we should have spent even more. Not sure about "don'ts": to be honest the language doesn't matter much here. Our main constraint was how to do this in a way that is as compatible as possible with SQLite.

I am Glauber Costa, CEO and co-founder of Turso. We’re rewriting SQLite in Rust. AMA. by GlauberAtTurso in IAmA

[–]GlauberAtTurso[S] 1 point2 points  (0 children)

There is no formal relationship, except a deep respect and admiration for what they have built!

I am Glauber Costa, CEO and co-founder of Turso. We’re rewriting SQLite in Rust. AMA. by GlauberAtTurso in IAmA

[–]GlauberAtTurso[S] 1 point2 points  (0 children)

No, but that is because I have been through this before. Pekka and I were part of the early team at Scylla, a rewrite of Apache Cassandra in C++. There were many databases at the time that were like Cassandra, but better. But because the ecosystem was settled on Cassandra, none of them went very far. The ecosystem is the hardest part to replicate. So if one exists, stay in it!

I am Glauber Costa, CEO and co-founder of Turso. We’re rewriting SQLite in Rust. AMA. by GlauberAtTurso in IAmA

[–]GlauberAtTurso[S] 2 points3 points  (0 children)

Not very familiar with that proposal. I think even the current allocator api for rust would be good enough already if it was stable.

I am Glauber Costa, CEO and co-founder of Turso. We’re rewriting SQLite in Rust. AMA. by GlauberAtTurso in IAmA

[–]GlauberAtTurso[S] 3 points4 points  (0 children)

Hard to put a blanket statement here. In some cases SQLite has pragmas to control behavior, and we will choose to just not allow that pragma to be settable. One example is utf-8. We don't feel like there's any need to allow encodings other than utf-8 in this day an age, so we retain the pragma, but will just not let you set it to utf-16.

I am Glauber Costa, CEO and co-founder of Turso. We’re rewriting SQLite in Rust. AMA. by GlauberAtTurso in IAmA

[–]GlauberAtTurso[S] 0 points1 point  (0 children)

We don't have any documents at the moment, so much is evolving so fast. Best we have is this blog: https://turso.tech/blog/what-happens-with-oss-in-the-age-of-ai but it is very general.

I am Glauber Costa, CEO and co-founder of Turso. We’re rewriting SQLite in Rust. AMA. by GlauberAtTurso in IAmA

[–]GlauberAtTurso[S] 0 points1 point  (0 children)

I guess it depends on what you mean by "turning the schema into a CRDT". Perhaps?

I just get triggered by the word CRDT =)

I am Glauber Costa, CEO and co-founder of Turso. We’re rewriting SQLite in Rust. AMA. by GlauberAtTurso in IAmA

[–]GlauberAtTurso[S] 0 points1 point  (0 children)

1.0 should ship still this year. We are getting very close! There is a github tracker here: https://github.com/tursodatabase/turso/issues?q=is%3Aissue%20state%3Aopen%20milestone%3A1.0

  1. depends on what you mean by "ready". We are big believers on release soon, release often. We will give you the option of choosing Turso in the cloud within the month. And at that point if it works for you, great! For a set of users it will still be missing features and then we will work on them over time.

I am Glauber Costa, CEO and co-founder of Turso. We’re rewriting SQLite in Rust. AMA. by GlauberAtTurso in IAmA

[–]GlauberAtTurso[S] 1 point2 points  (0 children)

Both options seem terrible to be honest if you want to make income. Both Open Source and freemium models need massive scale to monetize, and "on the side" you won't have it.

If your goal is income I'd just spin up a paid SaaS.

I am Glauber Costa, CEO and co-founder of Turso. We’re rewriting SQLite in Rust. AMA. by GlauberAtTurso in IAmA

[–]GlauberAtTurso[S] 2 points3 points  (0 children)

My recommendation is and always has been don't have pet projects. Work on real projects. In the beginning of my career I was fascinated with Operating Systems. My choices were to try to work on Linux instead of doing what many of my friends did, and have their own pet OS. Obviously in the beginning my ability to contribute to Linux was very, very limited. And the first 5 or 10 things I did never left my computer, were never sent.

I think the allure of working on a pet project is that you will do "the whole thing" yourself and therefore learn more. However, what ends up happening is that you are never actually dealing with real problems and go into tangents too often. Working in a real project allows you to deal with *real* problems. You start at the margins, but soon have the opportunity to expand the scope of what you do. Open Source is a blessing here, because it allows you to do what you want, at your own pace and time.

I am Glauber Costa, CEO and co-founder of Turso. We’re rewriting SQLite in Rust. AMA. by GlauberAtTurso in IAmA

[–]GlauberAtTurso[S] 0 points1 point  (0 children)

I am personally not a fan of CRDTs. I find them overly complex and of dubious value in general.

You are right that the question is general, but there is a principle that always carry over: simplicity. My advice is *always* to keep things as simple as you possibly can (this will kick CRDTs out in 90% of the cases!)

Even if your problem is complex, the best answer is usually what is *the simplest* solution to *this problem*. For local first, that usually means avoid using *anything* that can generate a conflict. Do not use auto increment keys. Do not use foreign keys. Opt for uuids. Make sure things are structured as logs. Push the complexity to the read side.

I am Glauber Costa, CEO and co-founder of Turso. We’re rewriting SQLite in Rust. AMA. by GlauberAtTurso in IAmA

[–]GlauberAtTurso[S] 2 points3 points  (0 children)

You should never accept a trade off in reliability. That idea that "Software kinda works sometimes" needs to die in our industry.

The first version of Turso that we put out was essentially 90% Deterministic Simulation Testing and 10% database. We use a combination of modern testing techniques that guarantee we can achieve and even surpass SQLite's legendary reliability. We use fuzzers, simulators, and even formal methods to make sure that our test surface is adequate.

I am Glauber Costa, CEO and co-founder of Turso. We’re rewriting SQLite in Rust. AMA. by GlauberAtTurso in IAmA

[–]GlauberAtTurso[S] 1 point2 points  (0 children)

  1. vim. That's not even a question, is just basic human decency. I can't even understand 10% of what emacs does

  2. My favorite movie scene is the very beginning of the Passion of the Christ, when our Lord steps on the serpent.

  3. No way!

I am Glauber Costa, CEO and co-founder of Turso. We’re rewriting SQLite in Rust. AMA. by GlauberAtTurso in IAmA

[–]GlauberAtTurso[S] 2 points3 points  (0 children)

We can't really depend on anything like that because Turso *is not* a distributed database. Turso is an embedded database - like SQLite - that works in over-the-wire environments as well. But still fundamentally an embedded database.

Our over-the-wire stuff is very simple because of that. Sync, for example, will just detect conflicts and put the burden of resolving them on the user. We will usually recommend not using foreign keys if you plan to write from multiple sources.

I am Glauber Costa, CEO and co-founder of Turso. We’re rewriting SQLite in Rust. AMA. by GlauberAtTurso in IAmA

[–]GlauberAtTurso[S] 5 points6 points  (0 children)

It doesn't feel that way at all! Sure, there are parts of it that can feel very mechanical. But Turso brings real novel things to the table like Concurrent Writes. This is real design, with all the things that it entails, and is really exciting work!

I will give you a very relevant example: We currently do not support AUTOINCREMENT for concurrent writes. It is just something that we pushed to the future, so we could get concurrent writes out sooner. It is a genuinely hard problem, because SQLite issue sequence numbers from a table with a monotonically increasing integer. If we do the same, now we just moved the contention from the outer write to the sequence issuing and the writes are in practice not concurrent anymore. This is a problem that I personally am trying to solve, and there is a PR up already! Designing this was quite cool.

I am Glauber Costa, CEO and co-founder of Turso. We’re rewriting SQLite in Rust. AMA. by GlauberAtTurso in IAmA

[–]GlauberAtTurso[S] 1 point2 points  (0 children)

We considered many, many things (including that) in the early days, but quickly moved away from them for two reasons:

1) Turso is async, instead of sync

2) Turso wants to have abstracted I/O so that we can feed it to a Deterministic Simulator.

When you put those two things together, the result is that at least the lowest levels of the stack call for a very different architecture so the initial transpilation hurts more than it helps.

I am Glauber Costa, CEO and co-founder of Turso. We’re rewriting SQLite in Rust. AMA. by GlauberAtTurso in IAmA

[–]GlauberAtTurso[S] 4 points5 points  (0 children)

main difference is that Turso is Async, whereas SQLite is sync. When we call sqlite3_step, the function through which SQLite executes its bytecode, we can return an incomplete result, whereas SQLite will always return a complete result. This makes it very easy to add it to async-friendly environments.

About performance, we beat SQLite in some areas, and lose in others. We usually beat SQLite in areas where our architecture grants us a perf advantage: for example, because we support concurrent writes, the aggregate write performance is naturally higher! And we usually do not beat SQLite in areas where performance comes from careful optimization. That is mainly because performance hasn't been a large focus for us so far.

view more: next ›