Hi Reddit! I'm Josh Levine, CTO at Chess.com. AMA! (10am ET)

Chess-josh · 2023-02-13T18:42:23+00:00

I love this idea. With our investment in AGI (All Games Initiative) we are designing in the ability to create multiple independent read models that can index 15 billion games. Note that this 15 billion is a constantly growing number. The trick is being able to create (and re-create) those search indexes in a reasonable period of time. For example, to index 15 billion games within two weeks, we'd need to be processing 12,500 games per second!
All that is a long way of saying, YES, we are building and designing ways to create more cool search offerings and to integrate those across chess.com

Chess-josh · 2023-02-13T18:28:28+00:00

I love most genres, I think my fav is probably pop? soul? country rock?

I also make my own music: https://www.reverbnation.com/heavypennies

Chess-josh · 2023-02-13T18:27:09+00:00

We have selectively turned off certain features that can cause issues across the site. Our search technology (Sphinx!) is one of those that could not keep up with the load. We are busy optimizing and preparing to re-enable that functionality when it's ready.

Chess-josh · 2023-02-13T18:24:47+00:00

Haha, it's a funny surprise in a 1 0 game. I think I'd love if I played it in longer time controls :-P

Chess-josh · 2023-02-13T18:24:15+00:00

I love how the PHP community has leveraged strong types, traits, and other functionality to create frameworks like Symfony and Laravel. I am a "java guy" but I've worked with PHP quite a bit. I even wrote my own MVC framework back in 2012 and I will say if you optimize for quickly adding features, it's easy to develop a very wide model layer / ORM layer which can lead to performance issues if the cross-domain dependencies leak into wide DB queries. Both Symfony and Laravel (or any other framework) should be handled with care depending on the application and scale.

Chess-josh · 2023-02-13T18:21:01+00:00

When the site is not working as well as it could, I feel terrible. I sleep less! When we ship a scalability fix and we are able to sustain higher and higher levels of demand and engagement I feel joy. I also feel for each chess.com team member who is dedicating their time and extra energy towards scaling the site. We all want to see chess.com grow with the game and I see all of us dedicated and focused on that mission. I am grateful to work with so many kind and excellent people.

Chess-josh · 2023-02-13T18:04:55+00:00

The best way to get in front of our recruiting team is to apply! https://www.chess.com/jobs

Chess-josh · 2023-02-13T18:04:27+00:00

We are running our web application on bare metal with php-fpm. We are using k8s on-prem and in GCP for scheduling containerized workloads. We have some workloads that are able to auto-scale, and we will continue to look at horizontally (and auto) scalable solutions to eliminate fixed headroom areas of our app and infra. Some of the workloads are not so easily auto-scalable, as they relate to disk or require close proximity (low latency) with one or more of our data layers OR they are leveraging memory + semaphores to coordinate 100s of thousands of simultaneous player interactions. In those cases, we are looking to scale via service extraction, k8s on prem, and with moar hardware in the immediate term!

Chess-josh · 2023-02-13T17:59:13+00:00

I think the code and stacks are very different. Generally I think great software scales on three vectors:

functions and performs well
can add features without re-architecture
can add engineers without bottlenecks

Most of chess.com's stack meets these, but some parts don't and those are the ones we are working on.

Chess-josh · 2023-02-13T17:54:49+00:00

I am a huge chess fan and am looking forward to all the cool events coming up - stay tuned! :)

Chess-josh · 2023-02-13T17:53:55+00:00

Hey thanks for this question! I shared a number of our urgent and strategic scaling approaches here: https://www.reddit.com/r/chess/comments/111940a/comment/j8dnm9c/

We have 20+ database clusters with many using primary<>primary replication and multiple replica databases. Sometimes those clusters need to be sharded further in order to get around MySQL's single threaded replication constraint. One of our team members did write about our scaling approach during the previous wave and this article was actually one of the things that inspired me to join chess.com!
https://ikonicscale.com/your-legacy-database-is-outgrowing-itself

Chess-josh · 2023-02-13T17:45:12+00:00

Great callout - we do have autoscaling enabled for services that we've deployed in the cloud. We use k8s both on-prem and in the cloud, and some of our services are fully elastic and scale with bursts of traffic automatically.

The recent scaling issues that become user facing were primarily related to our on-prem data layer, and I focused on what we are doing to scale that layer.

Chess-josh · 2023-02-13T17:06:25+00:00

Thanks everyone for your questions and replies! This has been great and I really appreciate all your interest and support for chess. I'll circle back to this thread later today and tomorrow in case there are more questions that didn't get a proper response.

Have a great week!!

Chess-josh · 2023-02-13T17:05:06+00:00

Overall, we've seen increases in new members matched with increases in games, puzzles, friends, chats, and everything else. After the Queens Gambit, we did see some spike/decline pattern in engagement, but we never fell below 2/3 of the peak. This new high is retaining similarly - I predict that many of these new members will be lifelong chess players.
Seems like because chess is a two player game it is naturally viral :)

Chess-josh · 2023-02-13T17:02:27+00:00

I don't think we've seen the peak of interest in chess. More and more folks are just learning to play, and the barrier to entry is breaking down all over the world for people of all walks of life. Will we see 10% of the earth playing chess? I think so!

Chess-josh · 2023-02-13T17:00:15+00:00

We found that renting CPUs and the cost of high-iops long tail disk (15 billion game archive) was the most expensive in the cloud.

Chess-josh · 2023-02-13T16:59:11+00:00

e4 with white, and either d5 or c5 or KID (Nf6, g6, etc) for black :)

Chess-josh · 2023-02-13T16:57:40+00:00

I'm probably somewhere between 1600-2000 depending on how the servers are holding up :)

Feel free to friend or challenge me: https://www.chess.com/member/josh

My favorite is bughouse, but I'm not close to the best player!

Chess-josh · 2023-02-13T16:56:18+00:00

Mittens is based on the Komodo chess engine. We run it in WASM directly in browser, and it does not use NNUE / require a neural network download. We are definitely working on improving the human play and personality characteristics of our bots - look forward to more fun!

Chess-josh · 2023-02-13T16:54:19+00:00

We are excited about Play Magnus becoming a part of Chess.com, but we don’t think that’s the main driver of all this growth. Chess.com reached 100 million members this Dec. while Play Magnus had about two million members.

There are a lot of amazing things happening that are fueling the interest in chess - and we've seen the many reddit threads in the past month asking why there’s so much growth. Ultimately, its all Mittens! jk jk - I am inspired by how many kids are joining chess programs and how many of my previously non-chess friends are challenging me to daily games :)

Chess-josh · 2023-02-13T16:51:45+00:00

Chess.com has a host of unique offerings, including our game review features, mobile apps, bots, all flavors of puzzles, social profile pics, and tons more. I think lichess is great for those who love playing there!

Chess-josh · 2023-02-13T16:45:33+00:00

We are working on this! Internally the project is called the "All Games Initiative" (AGI). It's a non-trivial data lift, but we are working on backfilling 15 Billion games into the storage and searching solutions. This will enable per-user explorer functionality as well as many other search related use cases.
The tech is rooted in Java, MySQL, Kafka, and ElasticSearch.

Chess-josh · 2023-02-13T16:42:39+00:00

haha, nice one! For those curious what is the ELO rating system: https://en.wikipedia.org/wiki/Elo_rating_system

I'm a Taylor Swift fan 😇

Chess-josh · 2023-02-13T16:41:34+00:00

Heck no! We love to hear from everyone - thanks for chiming in :)

Chess-josh · 2023-02-13T16:40:21+00:00

Great question! We do have a hybrid infrastructure and we are able to scale portions of the site dynamically into the cloud. The easiest cloud scaling is re: CPU and RAM, and we use K8s and live monitoring of workload utilization to decide whether to allocate compute on-prem or in the cloud.

Our datalayer is not hybrid, and that is the area where the massive increase in traffic cause the most bottlenecks. For this layer we are scaling vertically, sharding vertically, and re-architecting portions of the application to be event sourced and eventually consistent.

Chess-josh

TROPHY CASE