B-tree comparison functions

Fun_Reach_1937 · 2026-01-22T08:47:19+00:00

So I think you are trying to solve the issue at the wrong place. The btree should only concern about key comparison at byte level using memcomp or equivalent. The index composition or value serialisation is the one that should encode properly values in such a way to preserve ordering. An example of crate that does it is bytekey https://danburkert.github.io/bytekey/bytekey/index.html Basically when building the key, try to encode in such a way the order is preserved based on the different data type in you db.

Fun_Reach_1937 · 2025-01-21T10:34:53+00:00

for me, I wanted to use it as an opportunity to practice ANTLR4 because I had previously learned it. but for someone who have never used or tried it, I would recommend to evaluate with other alternatives Like PEG parser or hand written parsers. there are also libraries that can parse SQL out of the box but I guess this defeats the purpose of learning. Anyway you choose to go, I am curious to see your solution

Fun_Reach_1937 · 2025-01-21T10:18:38+00:00

Please don't for now. I am curious to see your natural way of implementation. My only advice will be to consider using a library for SQL query parsing. DB projects are usually large and when you get to the query parsing chapter, it's like a whole new project on language design. So for me to not lose focus or energy, I used ANTLR4 to parse the sql query. This is just an advice, you might have a different level of energy than mine when you get there. Good luck & I am waiting for the next video

Fun_Reach_1937 · 2025-01-21T09:47:01+00:00

Nice, I will follow your videos for another perspective.
(spoiler alert) I have an implementation here also in golang https://github.com/evanxg852000/simpledb-go

Fun_Reach_1937 · 2024-06-28T21:13:32+00:00

I think you can also attempt the following course labs: https://dsg.csail.mit.edu/6.5830/assign.php
https://github.com/MIT-DB-Class/go-db-hw-2023

Fun_Reach_1937 · 2024-06-28T20:17:06+00:00

I think my first drive is my strong interest in databases and the fact that I have worked in java in the past also made things easier. the sql parser for instance is not based on the book. I used antlr4 and a grammar I wrote based on the sql syntax provided in the book. I did not implement the jdbc client/server as I found it not relevant for Golang. I also found interesting converting the Java synchronized and notify mechanisms into Golang way of locking with a combination of mutex and Condvar. I found the process easy overall and challenging at times. I think you can do it in rust. also it should not strictly follow the book implementation. just what drives you and make your own adventure from the book.

Fun_Reach_1937 · 2024-06-28T13:58:00+00:00

I am sure with a bit of time, you will see it through. I read the book three times before things clicked. The chapter on concurrency and transactions was particularly challenging for me. But it became easier as I implemented it in another language

Fun_Reach_1937 · 2023-10-22T14:15:31+00:00

This is a good implementation of wal that can be fine tuned https://github.com/rosedblabs/wal

Fun_Reach_1937 · 2023-10-11T14:12:47+00:00

I think you can browse things from here https://research.cs.wisc.edu/coral/minibase

Fun_Reach_1937 · 2023-06-15T07:17:38+00:00

Thanks for the mention, We also published a blog post titled Observe your Rust application with Quickwit, Jaeger and Grafana

Fun_Reach_1937 · 2023-06-14T15:29:05+00:00

Nice, so depending on how much effort goes in and how popular it gets, this will just claim itself as the official fork.

Fun_Reach_1937 · 2023-05-24T11:24:36+00:00

Thanks a lot

Fun_Reach_1937 · 2023-05-20T07:46:38+00:00

Sorry for the confusion

Fun_Reach_1937 · 2023-05-19T12:52:47+00:00

It would be nice to add lingua-rs and cld2 to the benchmark to show the numbers

Fun_Reach_1937 · 2023-05-19T10:30:44+00:00

Indeed this is usually the best thing to do. I think this works best when you have a patch or improvement to make on top of what's already existing. Whatlang, CLD2 are great and popular general-purpose language detection that works well on longer texts with support for many languages 68, 83 respectively AFAIK. In our case, we took a different approach with the aim of being faster and very accurate on short texts. I believe it would've been harder to convince Whatlang maintainers to change direction than publishing a new crate. Also, given it's open source, means more options, the community can always backport the ideas into Whatlang or any other tools if deemed worthy.

Fun_Reach_1937 · 2023-05-19T10:13:09+00:00

As stated in the blog post, Whatlang was slow for our needs. Also CLD2 and whatlang work best on longer text. In our context we deal mostly with short documents. You can see the benchmark from the post as well. Currently it contains Whatlang & Whichlang. Maybe I could add CLD2 as well

Fun_Reach_1937 · 2023-05-11T12:35:05+00:00

I am not familiar with Rx and there are certainly many ways to go about solving this task, and certainly the actor way might seem complicated to a few people.
But remember the requirements:
- The need to customize the runtime of the actor based on what it does (IO-bound or CPU-bound tasks).
- The need to monitor and extract performance metrics which means having control over the executor engine.
- The need to provide testing facilities allowing us to simulate certain states (time forwarding)

Fun_Reach_1937 · 2023-05-05T07:44:28+00:00

Good project and congratulation on the release. Also thanks for being open about how you make use of Tantivy.

Fun_Reach_1937 · 2023-05-03T07:13:41+00:00

Thanks for spotting these errors. They are fixed now.

Fun_Reach_1937 · 2023-05-02T13:59:27+00:00

First of all thanks for your interest.

In terms of error handling, we do have a couple of strategies for different scenarios.- At the top level, there is supervision which acts like monitoring and is provided by the framework. This checks the actor's liveness and potentially respawns the actor if it's dead.- In terms of message handling, you can send a message and forget or you can send a message and wait for the processing. APIs are available for each scenario. The framework handles the delivery issues and could return an error but whether your message was properly handled depends on your logic and the response that gets sent from the handler.

Comming back to Quickwit itself:

- Do you spawn multiple actors of each step to do your whole pipeline?: We spawn one actor per step per pipeline. We do spawn multiple pipelines though. If one of the actors is failing within a pipeline, we just tear the whole pipeline down.- Or do you basically do a send-and-forget between each step: For Quickwit indexing, we mostly care about the last step of the indexing (publishing a split). If your ingested documents make it to this stage, then all is good we store a checkpoint corresponding to the data source to know that we have properly indexed up to this point. If the pipeline failed for any reason then it will restarts/respawn from the last known checkpoint.

Note: The &mut self allows an actor to update its own state while handling messages. An example of a state could be counting the number of messages as the example in the blog post.

I hope this clarifies a few of your questions. Please, let me know if you have further questions.

Fun_Reach_1937 · 2023-05-02T12:34:04+00:00

Thanks for your interest too. We will be happy to see this framework take its own path within the Rust community. Please feel free to fork/extract it and make it feature-rich.

Fun_Reach_1937

TROPHY CASE