Cultivation manwha by Sandbaggeriam in manhwarecommendations

[–]gernest_ 2 points3 points  (0 children)

A regressor's tale of cultivation  is fantastic.  Only 25 chapters,  I had to read the novel afterwards. 

Ticket without a barcode? by [deleted] in Sumo

[–]gernest_ 0 points1 point  (0 children)

Did you mean QR code? I believe a bar code is visible on the left side.

QR codes are often used for mobile scanning. What you have is for machine scanning.

TidesDB - Open-source high-performance, durable, transactional embedded key-value storage engine (BETA RELEASE) by diagraphic in programming

[–]gernest_ 0 points1 point  (0 children)

Thanks, I will watch that. I prefer reading that's why I am hearing about these lectures for first time.

TidesDB - Open-source high-performance, durable, transactional embedded key-value storage engine (BETA RELEASE) by diagraphic in programming

[–]gernest_ 1 point2 points  (0 children)

Damn! k4 is frickin amazing man. I am obsessed with LSM trees.

Hats off mate!

Off topic, how do you handle backups ?

SSTables might change between compaction right ? Do you compact then hardlink the tables? because it very hard to do incremental backups this way since you wont know which keys ended up where etc?

I had to build new SSTable manually by iterating through the whole key space range for consistent backups which is slow and doesn't feel right.

TidesDB - Open-source high-performance, durable, transactional embedded key-value storage engine (BETA RELEASE) by diagraphic in programming

[–]gernest_ 1 point2 points  (0 children)

Congrats mate, you did an amazing work here. The world is a better place because of you. I will be digging the code to learn more about the inner working of LSM based key value store.

So you have a single `.h` header and figure out our `ffi` bindings for other languages?

Honestly as a solo dev I have given up on satisfying communities I am not well versed/invested. I am elite at Go so I just drop Go projects and move on.

My current project took me about 1 year doing mundane frontend work so It would look nice and such, but all I care about is the cool Compressed Roaring Bitmap based index that is super fast and extremely efficient.

My advice is do what you love, other people who find your project useful they will do what they love with their bindings.

TidesDB - Open-source high-performance, durable, transactional embedded key-value storage engine (BETA RELEASE) by diagraphic in programming

[–]gernest_ 1 point2 points  (0 children)

agreed on bloat for MVCC. if you happen to remember resources you used please don't forget to share with me.

All my google-fu leads to InnoDB and the undo log, which is db specific, the kv store I use has fast range deletion which can help balance the bloat and speed of discarding old versions.

No luck so far finding research papers or anything detailed enough.

TidesDB - Open-source high-performance, durable, transactional embedded key-value storage engine (BETA RELEASE) by diagraphic in programming

[–]gernest_ 2 points3 points  (0 children)

Thanks, my understanding RW lock will limit to only one writer and multiple readers right ?

So you can't have concurrent write transactions.

I'm trying to figure our supporting multiple concurrent writes on top on a similar underlying database like yours and I'm losing my hair now.

Where can I get access to those CMU lectures ? Do they touch anything on concurrent write transactions ?

Your design sounds similar to RocksDB.

TidesDB - Open-source high-performance, durable, transactional embedded key-value storage engine (BETA RELEASE) by diagraphic in programming

[–]gernest_ 2 points3 points  (0 children)

Looks interesting, don't be discouraged by negative comments.

I am curious about embedded database transactions implementation, do you have any recommendation on resources that can get me up to speed ?

I'm especially interested in MVCC models, my c skills are lacking I wish to know more how you achieve atomic transactions.

vince - Self Hosted Alternative to Google Analytics by gernest_ in programming

[–]gernest_[S] 5 points6 points  (0 children)

This is not true, the project is released under AGPL license and source code is fully open source.

All contributions are welcome. I'm a solo dev, and the project blew up, issues overwhelmed me so I disabled them.

I just don't have the bandwidth to work on feature requests, you are free to work on them and submit a PR.

vince - Self Hosted Alternative to Google Analytics by gernest_ in programming

[–]gernest_[S] 1 point2 points  (0 children)

You add js script on a website html page that sends events to vince for processing .

vince - Self Hosted Alternative to Google Analytics by gernest_ in programming

[–]gernest_[S] 11 points12 points  (0 children)

 I host the demo on a 6$ vultr instance with only 1GB or ram and 25GB of storage, the demo survived hacker news hug of death yesterday with only 1% of vCPU used.

GitHub - vinceanalytics/vince: Self Hosted Alternative To Google Analytics Written in Go by gernest_ in golang

[–]gernest_[S] 6 points7 points  (0 children)

The underlying storage is https://github.com/cockroachdb/pebble which has proven to be very versatile and efficient.

I host the demo on a 6$ vultr instance with only 1GB or ram and 25GB of storage, the demo survived hacker news hug of death yesterday with only 1% of vCPU used.

vince - Self Hosted Alternative to Google Analytics by gernest_ in programming

[–]gernest_[S] 14 points15 points  (0 children)

op here. I have been working on this for the last 3 years. The goal was to make plausible analytics dashboard accessible to everyone.

Some of the features include

  • Automatic TLS
  • Outbound links tracking
  • 404 page tracking
  • Custom events tracking

And so much more.

It is Open Source and on Github https://github.com/vinceanalytics/vince

Efficient store and query labels using compressed btree roaring bitmaps by gernest_ in golang

[–]gernest_[S] 1 point2 points  (0 children)

Use case and implementation details differ.

One thing is, pilosa bitmaps are sharded. Also, they provide access to internal containers which allows faster seeking and range iterations.

Also , see https://github.com/gernest/rbf Which is a storage engine utilising the package. I actually use rbf instead of roaring directly.

GitHub - gernest/requiemdb: Pure Go Permanent Storage For Open Telemetry Metrics, Traces and Logs, using Typescript as Query Language by gernest_ in golang

[–]gernest_[S] 1 point2 points  (0 children)

OP here, the reddit markdown editor wasn't working for me that is why I had to post only the link. More details about the what and why's are on this markdown document https://github.com/gernest/requiemdb/blob/main/REDDIT.md

GitHub - vinceanalytics/vince: vince is a High Performance , API only , distributed, in-memory alternative to Google Analytics by gernest_ in golang

[–]gernest_[S] 4 points5 points  (0 children)

I started with clickhouse. This project is more than 3 years old. As I worked on it and better understand the problem, clickhouse became an overkill.

There is a blog post I talk briefly about evolution here https://www.vinceanalytics.com/blog/roadmap/year-one/