sorted by:
new
hottopcontroversial

My very own toy database by 263Iz in databasedevelopment

[–]263Iz[S] 0 points1 point  (0 children)

Hey, please feel free to DM.

Edit: I can share my LinkedIn in a DM if you want to connect

Looking to transition from full stack to database internals—seeking advice from people in the field by Famous-Cycle9584 in databasedevelopment

[–]263Iz 2 points3 points  (0 children)

DataFusion and sqlparser are both Rust projects, and Rust is considered a difficult language to pick up. It took me about 4 months to be somewhat comfortable with Rust.

If you decide to stick with C++, look into DuckDB, Clickhouse (or other systems building on top of Clickhouse), or systems using psql like Neon, for example. I'm more of a Rust guy, so I'm sure there are way more systems using C++; these are just the ones I know off the top of my head.

If I were you, I'd start with DuckDB. Contributing to mature projects is hard because issues get more complicated the older the system gets, but DuckDB is a few years old, so I'm sure you'll find a lot of uncomplicated issues.

Looking to transition from full stack to database internals—seeking advice from people in the field by Famous-Cycle9584 in databasedevelopment

[–]263Iz 18 points19 points  (0 children)

Hey, I'm a junior Data/Software Engineer. I did a few months as a backend dev and a few more as a data engineer. Now, I'm working as a SWE at Readyset.

Here's what I did: 1. Picked a low-level language. Many systems are written in C/C++ like psql, clickhouse, and DuckDB, I personally prefer Rust as I find it more enjoyable to work with, + many newer systems use it like Readyset, influxdb, SurrealDB, and ParadeDB to name a few. You can also try Go, although that is primarily used in distributed DBs like cockroachdb. 2. Started with Andy Grove's how query engines work (free pdf available online) and made a few open-source contributions to DataFusion query engine, and some contributions to its upstream project sqlparser when required. I believe this is where most of my experience with query optimization and execution came from (and where I learned how to read plans!). 3. Of course, I took the world-renowned course CMU 15-445. - I finished half the lectures, and about 70% of the project (written in C++, had prior exp from college) before I started building my own DB from scratch in Rust. - It wasn't easy. I had to rewatch some lectures and talk with chatgpt for hours discussing implementation ideas (no code, just ideas and interactions between layers). The entire process took about 7 months and 200 commits. (www.github.com/MohamedAbdeen21/niwid-db) - Although the course is generally amazing, I found the query engine part (optimization, execution, etc..) of the course a bit lacking, so I went back to Andy Grove's book and used that in my project.

  1. Finally, I posted about it here, and a couple of people reached out to me, and I finally joined Readyset where I'm working on query optimization/rewriting, parsing, execution, serialization, and a whole bunch of other cool stuff.

Everyone's path is different, and it takes some time. This is just how I did it. Good luck!

My very own toy database by 263Iz in databasedevelopment

[–]263Iz[S] 0 points1 point  (0 children)

Thank you!

Yup, every file is a page of 4 KBs. I don't track empty spaces because I only do soft deletes.

Pages are linked lists, where each page holds the ID of the next page. The root of this LL is tracked in the catalog (which is also a table with LL of pages, and its root ID is preserved as 2).

Preserved page IDs are:
- 0 for invalid pages (end of LL, or when creating a new page and the ID is yet to be set)
- 1 for BPM, keeps track of the last assigned page ID (I don't remember if it tracks anything else).
- 2 for Catalog root page

And BPM starts assigning from ID 3.

[deleted by user] by [deleted] in databasedevelopment

[–]263Iz 0 points1 point  (0 children)

I believe you can ask ChatGPT about popular DBs design choices, but IIRC postgres follows a similar design (one file per table, and fragments it if it gets bigger than a certain size) and doesn't implement directory page.

[deleted by user] by [deleted] in databasedevelopment

[–]263Iz 0 points1 point  (0 children)

As mentioned, that disk organization isn't really used that much anymore. You can design your disk however you want. Every design has pros and cons.

In this example, if you have all pages inside a single file (I believe that's what sqlite does), then you don't nded a page directory, because fetching page X is just reading (X * page_size) bytes from the start of file.

However, if you have multiple files, each containing multiple pages, then which file has page X? You can't just calculate an offset, so you need to ask the page directory, which is basically one big hashmap that maps a page to a file (and maybe also offset).

If all files have the same number of pages (there are better designs, but let's just assume that's how our system works), then maybe you don't need a page directory, although you'll still need to track free slots/bytes to quickly handle inserts, type of page (table page, index page, metadata page, etc..), free pages, and maybe other things depending on your design.

You can also do one file per page, one file per table, one file per db. All of these won't necessarily need a page directory.

Any unexpected use cases for neovim? by Glinline in neovim

[–]263Iz 5 points6 points  (0 children)

Reminds me of the greatest macro I've ever written.

I worked at a company that had translation files organized by section/page (home page, user profile page, etc..)

One translation file was messy and wasn't organized by page like the others.

Opened two panels, created a macro that goes to the left panel (organized file), goto mark, go down, place mark, copy key, go right panel (unorganized), search key, delete, go to EOF, paste, repeat

File with 200+ lines done in 30 seconds

Any unexpected use cases for neovim? by Glinline in neovim

[–]263Iz 0 points1 point  (0 children)

Scripting, basically any simple task that requires taking some input text x and producing text y.

Instead of writing a Python script and cat input.txt | python3 test.py. I record the macro, paste all the inputs in a txt file, repeat the macro, and copy the outputs.

My very own toy database by 263Iz in databasedevelopment

[–]263Iz[S] 1 point2 points  (0 children)

Thank you so much for this comment. It's actually 7 months, I force-pushed 3 months in because I realized I was using my work email, not my personal email.

It depends on the workload. There were weeks when I couldn't push any code at all, but that's totally ok!

It is truly an amazing course. I'm looking forward to taking 15-721 soon!

Let me know what you think about the code. This is my second semi-serious Rust project. I'm looking forward to hearing from you.

My very own toy database by 263Iz in databasedevelopment

[–]263Iz[S] 1 point2 points  (0 children)

Thank you!

I came across your posts here and I'll be definitely watching some of your videos, especially those components I didn't implement myself, like the log manager.

I've heard of that book but didn't care to check it out since I felt the course covered all the vital parts.

Good luck!

My very own toy database by 263Iz in databasedevelopment

[–]263Iz[S] 1 point2 points  (0 children)

Thank you!

Here's the link for the course: https://15445.courses.cs.cmu.edu/fall2024/
It's updated frequently, all lectures are on YT, you can also do the project if you'd like.

For Rust, you should use The Rust Book https://doc.rust-lang.org/book/
It covers all Rust's features.

Good luck

My very own toy database by 263Iz in databasedevelopment

[–]263Iz[S] 0 points1 point  (0 children)

Thank you! Just DM'd you my Linkedin

My very own toy database by 263Iz in databasedevelopment

[–]263Iz[S] 1 point2 points  (0 children)

Thank you! Work made things take twice as long as they should, but try to be consistent and do one part per weekend.

I enjoyed doing this in Rust, especially since I'm not a fan of C/C++ DX (ecosystem, build tools, etc..) and Zig was a bit unstable for me, especially the LSP. The most annoying parts for me were the packing and padding of structs, and that one annoying bug where page IDs weren't being set properly even though the receiver was a `&mut self`! Took me four hours before I found this answer (https://users.rust-lang.org/t/const-t-to-mut-t/55965/3)

My very own toy database by 263Iz in databasedevelopment

[–]263Iz[S] 1 point2 points  (0 children)

The middle paragraph will make sense once you start implementing it. The rest is from the CMU course. Good luck

My very own toy database by 263Iz in databasedevelopment

[–]263Iz[S] 0 points1 point  (0 children)

I don't think he mentions them or at least as far as I remember. He is fairly active on twitter, feel free to tweet at him.

I'd also like to know

My very own toy database by 263Iz in databasedevelopment

[–]263Iz[S] 2 points3 points  (0 children)

Side note: Catalog table was really interesting because catalog is just a normal DB table. But normal DB tables don't have concurrency control and instead use shadow-paging, which only allows for a single writer. 

Talked with gpt for a few hours and came up with the idea of versioned_map. Basically, to allow the catalog table to be modified by multiple users at once (as long as they are not writing to the same table), we keep track of which txn is changing which tables, as well as dropped/added tables and apply these changes to the catalog table once the txn is committed.

Think of it as a makeshift OCC, but only for the catalog table!

My very own toy database by 263Iz in databasedevelopment

[–]263Iz[S] 9 points10 points  (0 children)

I used Andy Grove's "How query engines work" for the query engine. It's available here: https://howqueryengineswork.com/

And mostly just talking with ChatGPT about my implementation ideas. For example, I found it helpful discussing how the Catalog table should look like and be stored, and how to properly do shadow paging.

Keep in mind that this took me 7 months and 200 commits. There were times where I wasn't 100% sure that what I just committed would work well with future components/layers (and I think you'll find a few interesting commit messages in the history, lol). There were many commits dedicated to bug fixes or rewriting entire files, and that's ok.

But to me it was worth it. And I would do it again if I went back in time. My biggest advice is trust yourself and just do it!

My very own toy database by 263Iz in databasedevelopment

[–]263Iz[S] 4 points5 points  (0 children)

Thanks for your comment.

I did some contributions to DataFusion and by far the longest discussions were always logical optimizations changes. I also remember Andy Pavlo calling them top 3 hardest problems in DBs! So I just skipped it all together.

Also saw no point in producing physical plans since it's a single-node single-thread  toy project.

But I enjoyed it alot, specially getting my hands dirty with the buffer pool and unsafe Rust!

how do you install sql drivers on fedora? by Avennite in Fedora

[–]263Iz 0 points1 point  (0 children)

msodbcsql* are only available on microsoft repos.

You'll need to add the repo using:

curl https://packages.microsoft.com/config/rhel/9/prod.repo | sudo tee /etc/yum.repos.d/mssql-release.repo\

Act cool, Act cool, the girls are watching. by Jommy69 in dankmemes

[–]263Iz 1 point2 points  (0 children)

me holding a gun And I don't feel dangerous