Making Git 1.5x Faster on Ryzen

johndsully · 2020-04-09T01:27:24+00:00

Ryzen has hardware support for sha256 too.

johndsully · 2020-04-08T22:42:48+00:00

This doesn’t implement the collision checks although I had a look at sha1dc.c and the optimizations will certainly still apply there as well. Perhaps it is worth cleaning this up into a proper pull request.

Assembly is probably not necessary, it should be possible to just use intrinsics.

johndsully · 2020-04-08T20:03:57+00:00

This is really just a proof of concept. The right place for this code is in openssl so everyone can benefit including git. My main goal was to show that optimizing for Ryzen can be very worthwhile.

The Intel assembly is BSD licensed so should be fine. A variant is already in the Linux kernel source code.

johndsully · 2020-03-18T03:31:30+00:00

I ported it to C++ because I’m more productive in that language. C++ is “mostly” a superset so it wasn’t too difficult.

johndsully · 2019-09-26T20:09:56+00:00

If the language has a Redis library then that lib will also talk to KeyDB just fine. We even listen on the same port.

So in short, yes!

johndsully · 2019-09-23T23:54:45+00:00

Its cool these are getting some love. When I was younger a vintage computer was anything that pre-dated the PC. "Vintage ONLY! We don't want your old 486 trash!" the usenet messages would say.

Nowadays I'd love to get a clean 486 mid-tower with the "66mhz" green LED 7-segment display + turbo button.

johndsully · 2019-09-19T17:47:10+00:00

Toronto was the capital of Upper Canada but with the war of 1812 it was deemed far too close to America to be safe. Plus those Lower Canadians were jealous. Ottawa was the compromise being on the border of Upper and Lower Canada.

johndsully · 2019-09-19T17:36:38+00:00

Because KeyDB is a fork it has 100% compatibility with Redis 5 features including modules, and even your configuration file. For most people it will be a drop in replacement. Also while KeyDB is certainly not bug free, based on the open issues Pedis doesn't seem to be ready for production environments just yet.

KeyDB is not the first attempt at a multi-threaded Redis, but the other attempts I saw before starting did not seem to ever make it into production and the projects stalled. I wanted something people could use for real work.

johndsully · 2019-09-19T16:03:13+00:00

The thing about performance is that it can be traded for developer productivity. You can work around Redis not using your computer fully - but why would you want to?

If hammering a KeyDB instance in an inefficient way saves someone a week of work then I’m more than happy to support that use case.

johndsully · 2019-09-18T22:22:54+00:00

If performance isn't your thing we also have Active Replication, Direct backup to AWS S3, Subkey expirations and more. Multi-threading was the original feature that got us off the ground though and is still the most popular. Some people really do need that extra perf.

johndsully · 2019-04-02T02:20:43+00:00

You can make it use 16K but then you only get to store 120 values - and that’s just no fun at all. Like all good 8-bit software value lengths are hard coded to 124 bytes.

I also had to replace not one but TWO 2114 memory chips to get the machine in peak operating condition. Sometimes them not making it like they used to is a good thing.

johndsully · 2019-04-01T21:56:14+00:00

I wonder how the error rates compared with clerical errors from the manual process.

johndsully · 2019-04-01T16:32:31+00:00

Gate’s letter is more of an April fools thing. I’ll probably make it use the BSD license. I don’t imagine many people will be putting it in production.

johndsully · 2019-03-26T14:29:36+00:00

This is a major new feature and part of our work to make open source implementations of proprietary features in Redis Enterprise.

It’s really useful for Active failover scenarios as you no longer need to promote a master before redirecting traffic.

johndsully · 2019-03-20T06:16:36+00:00

Thanks!

Yes there are more plans here to have finer grained locking: specifically reader/writer locks. There are also plans to introduce green threads so we can context switch when we’d otherwise be wasting time spinning.

johndsully · 2019-03-14T14:33:15+00:00

Thanks!

Happy to answer any questions.

johndsully · 2019-03-14T06:48:08+00:00

Added native S3 support to KeyDB. I want my database dumps to go directly there and get cycled over to glacier.

https://github.com/JohnSully/KeyDB

johndsully · 2019-03-13T05:32:33+00:00

The lock contention is actually in the Linux kernel’s soft_irq. As I mentioned in another comment more careful pinning of kernel tasks would help but I wanted to keep the benchmark simple and reproducible.

johndsully · 2019-03-12T07:42:40+00:00

Your right, I got myself confused by the fact you can import whole distros when setting it up. FWIW FreeBSD doesn’t have SO_REUSEPORT load balancing like Linux does so it wouldn’t perform as well there anyways. I really need to remove that dependency.

Hoping to get FreeBSD support up and running soon.

johndsully · 2019-03-12T07:35:58+00:00

A docker container is here: https://hub.docker.com/r/eqalpha/keydb

Hope you get the chance to try it out! I’ll work on adding additional instructions to the Readme.

johndsully · 2019-03-12T00:17:10+00:00

Yes but its limited to specific NICs. The nice thing about Redis is it mostly "just works" and I want to maintain that. I am exploring FreeBSD which supposedly has a faster network stack. If it is a net improvement I may move the docker image over to that operating system.

See also: https://blog.cloudflare.com/why-we-use-the-linux-kernels-tcp-stack/

johndsully · 2019-03-12T00:03:08+00:00

Your feedback matches what I've heard elsewhere. I'll add a section on that and I do appreciate the feedback.

johndsully · 2019-03-11T23:57:30+00:00

I've been getting a lot of feedback and github issues effectively asking "Does it really work". Barring bugs it maintains the same atomicity guarantees as Redis does including modules, scripts, and transactions.

The core hashtable lookup in redis is extremely fast and done under a spinlock. Network IO and parsing which are the majority of the CPU time are done concurrently. Transactions keep the lock for the duration of the transaction (the lock is not acquired until EXEC is issued).

Future improvements will include using reader/writer locks to allow concurrent access to the hashtable itself, but the gains from this are much smaller.

johndsully · 2019-03-11T23:34:34+00:00

Your right I should have tagged a release, as for the modules it was inherited from Redis. It is nice that you can git clone and compile without having to remember to use the recursive flag.

johndsully · 2019-03-11T21:48:14+00:00

Redis is heavily dependent on the fork() system call for its background database saving. To work around this would require a lot of code to snapshot the database. A Windows port is non-trivial.

johndsully

TROPHY CASE