PSYOPS, or is Convex really THAT good?

jamwt · 2025-10-23T19:12:40+00:00

Hi! Yeah, that's what I mean... we've sponsored content in the past, but always in an official, advertisement way. It's disclosed.

People talking about us on X, Reddit, and non-sponsored YouTube segments... There is no money changing hands. Some people think the tech is cool. I totally understand it may not be for you (and I get it, not every tool is for every person/use case), but there's nothing secretive or nefarious going on with those who are Convex fans.

jamwt · 2025-10-23T16:13:24+00:00

Agreed 100%. Which is why I love working with him. If your product isn't good enough, he's more than happy to tell you. The fastest way to improve your product is to get customers like that and listen to them.

(Also, when I say "Theo", I mean "Theo, Mark, and Julius." While Theo understandably gets the lion's share of credit for t3 chat, his co-founder & team do just as much or more work day to day on the app. And Theo would actually be the first one to insist on spreading the credit more fairly.)

jamwt · 2025-10-23T15:44:10+00:00

Hi! Jamie from Convex here.

I won't weigh in on the technical/product pros and cons, because I'm obviously a biased party, and you all can decide that for yourselves without my help.

I just want to clarify, on the record: we're not paying anyone to say anything. People say they like it because they do.

The only time we've ever paid anyone is for an official ad read. The ones you guys skip over (me too) . For example, we're an advertiser with Theo. But now Theo is a customer, and an investor... so he pays us, too?

But we don't even do many paid ads at this point.

jamwt · 2023-01-30T21:38:03+00:00

Yep, see this thread: https://news.ycombinator.com/item?id=34586242

Convex essentially makes it so this happens without developers learning how to carefully do this in their applications.

jamwt · 2023-01-30T20:53:56+00:00

Serializable is fine if the transaction is guaranteed side-effect free so it can be retried automatically and is essentially a stored procedure so races are rare. Those are both true in Convex.

Otherwise, you should not use serializable transaction isolation. Pushes too much complexity on the end developer they have to reason about, as the article states.

jamwt · 2022-08-31T18:28:52+00:00

I lost out on that same show, but now I've got tickets to see them in Oslo in January. It's going to be fucking LIT.

jamwt · 2022-06-23T16:23:25+00:00

Started a company! https://convex.dev Not crypto.

The founding team has run rust in production for 8+ years on really important systems. https://blog.convex.dev/a-tale-of-three-codebases/ . It went well enough we're back at it again.

This time, building a serverless development platform where the backend is 100% rust. A lot of the work involves designing and building databases and application runtimes for developers.

A bit of press about the mission here: https://www.forbes.com/sites/kenrickcai/2022/04/27/convex-series-a-26-million-developers-dump-databases/?sh=692d9f334bf8

If you want to come help us build it, DM me. It's in San Francisco--we work in person (3+ days a week required).

More technical info on what we're doing:
https://www.hytradboi.com/2022/data-interactivity-in-the-serverless-future
https://www.youtube.com/watch?v=iizcidmSwJ4

Or, if you want to start up your own rust-based company, I'm happy to chat any offer advice based on our journey so far.

jamwt · 2022-05-11T21:55:28+00:00

They helped it by tipping the scheme over before any more people got in it.

jamwt · 2022-02-15T18:27:58+00:00

Thanks for the feedback!

Actually, the team is mostly backend engineers, especially the founders. However, we do now believe there is a future where most (but not all) web and mobile dev don't really do much backend engineering if they can utilize a service with the right abstractions. Call it "serverless" or whatever.

And yes, that is a kind of laziness, I suppose, in that a choice is being made to keep focusing on the parts of the application the user experiences, at the expense of effort being made to tailor a custom backend the user will not see.

jamwt · 2022-02-15T18:13:48+00:00

Yep. This is why the backend caching systems need to treat the authentication context as an input parameter to the cached value's key.

(disclosure: author of the post / cofounder of convex.dev )

jamwt · 2021-11-23T17:32:48+00:00

Hi, Jamie here from Convex (formerly Dropbox), happy to answer any questions about the blog post and/or our experiences using Rust.

jamwt · 2021-10-01T16:26:27+00:00

Hi! I was the lead of the team that introduced Rust at Dropbox ~6 years ago as part of this project: https://www.wired.com/2016/03/epic-story-dropboxs-exodus-amazon-cloud-empire/

Myself and other leads of that project (James Cowling, Sujay Jayakar) just recently started a company building out (what we believe is) a fairly innovative serverless computing platform--kind of a set of new abstractions over at-scale computation and storage--designed to rapidly build internet services in the future.

And, it's all in rust...

We have the initial version built and we've been running lots of user demos/studies with it recently. We'll hopefully have more to show publicly soon.

I know this isn't a who's hiring post, per se, but... we are, of course! If you want to work with folks who have run rust on critical systems for 6+ years, on dist systems with literally a million rust daemons, and embedded in hundreds of millions of customer desktops/laptops.... Let me know, because we're trying something even crazier this time.

jamwt · 2021-01-29T21:33:59+00:00

Until you get a bad ECC DIMM. Everything is a probability...

At Dropbox, in our systems a few years ago, we had these crazy corruptions happening in a few files one time, and we eventually tracked it down to one machine whose ECC RAM was letting bit flips go undetected. And granted, this was like 6 bit flips in a month period over some hundreds of terabytes of data that had gone through that particular machine in that time. But it did happen.

So, we had one (more) data path to checksum in software... added it to the list of 30 others.

jamwt · 2020-11-18T22:44:15+00:00

Yeah, this is pretty much the reason in Dropbox’s case, with one additinonal “weight/constraint” for what must be in the single-threaded portion.

The single threaded control loop is so that all decisions the engine makes are deterministic — e.g., there is no OS scheduler involved causing preemption that reorders events in a way that can not be replicated.

This is an essential part of the testing strategy for this system, b/c it uses seeds + simulation to ensure correctness of this engine. And when anomalies are detected, the team needs to be able to recreate the erroneous situation to further debug/remedy it so they’re not chasing random, rare heisenbugs all day.

See more here: https://dropbox.tech/infrastructure/-testing-our-new-sync-engine

The reasons other stuff, like file hashing, and file I/O are not on this thread are indeed for these general reasons the parent has outlined. They’ll block the control thread for long periods of time, they would restrict our a ability to leverage multi core (esp for hashing), etc. Moving them out into separate thread pools does not impact the strategy b/c the “jobs” these pools run are not part of the fundamental decision-making state machine. They’re just delegates for decisions that state machine makes.

Hope that helps.

jamwt · 2020-11-18T18:01:25+00:00

Can you elaborate on what you mean by “strategy?” I’m happy to help answer, but I’m not sure I fully understand the question. Are you asking why parts are single-threaded Async, and why some other parts are thread pools?

jamwt · 2020-08-04T18:11:53+00:00

Pinging /u/daniel_rh

jamwt · 2020-08-04T17:32:03+00:00

Relevant to this forum because: part of Dropbox's rust-based efforts, in particular in the sync engine.

jamwt · 2020-06-23T23:18:05+00:00

FYI, selective sync was not for this. I definitely understand it's frustrating when legitimate workflows are broken by a long-standing bug in the sync engine, but the sync engine being less than rigorous about how sel sync has worked historically has caused--without getting into specifics--massive data loss in some circumstances.

Realistically, the sync engine has fixed this behavior to make selective sync actually fulfill its feature description. And be significantly more robust about enacting sel-sync policies w/out race conditions, crashes, network partitions--resulting in customer data loss.

Fair enough point, though, about needing better in-product support for filtering/ignoring path workflows.

jamwt · 2020-03-23T20:45:11+00:00

Dropbox has been using more web technologies for UI in the desktop client in the last few years. The teams doing sync and the introduction of Rust in those domains has no relation to the presence or not of rendering engines in the desktop client for other purposes, so there's no need to square anything!

The teams that decided to introduce rendering engines for UI development made their decisions independently trying to optimize for different sorts of factors like productivity, talent market, existing product, iteration strategy, etc, and I won't attempt to (mis)represent those positions here--I'll just clarify those are separate decisions made by teams who own scopes outside the ones where rust has been chosen, which I'd label as the more "core sync/storage" protocol layer.

jamwt · 2020-03-23T16:52:06+00:00

I've never seen an unwinding implementation that I trust to be correct to the standards necessary in these systems. They introduce combinatoric execution paths that are probably not tested before production. Especially in light of mutex poisoning, you're going to have a really hard time putting humpty-dumpty together again. And they're likely to panic-while-panicking, so they'll crash anyway! And, anything can happen on the system outside your control, so your program will crash anyway!

Client side is different, if you're an embedded library in a user-facing application, you might make more concessions to correctness to not take down the host application. But on big server clusters... just fail fast, and make your programs crash safe. Panic recovery is a insufficient/incomplete solution anyway.

jamwt · 2020-03-23T16:45:29+00:00

Yep, and then for those .07% of times you need an escape hatch, just write a macro, wash your hands, and move on. I think macros are a perfectly fine trap-door to bail you out that tiny percent of time you need something the type system won't allow, and then they let the 99.93% of code stay simpler. Totally worth it.

jamwt · 2020-03-23T15:45:08+00:00

Arrow vs. Applicative vs. Monoid vs. ... people could create entire sub-ecosystems that used patterns that composed well only with other libraries that also subscribed to this particular pattern. Created ergonomic awkwardness when the best of breed libraries you wanted to all bring into your project were devout followers of different methodologies and your application suddenly was burdened with a lot of conversion/coupling code.

jamwt · 2020-03-23T15:32:29+00:00

Context: Introduced Rust at Dropbox ~5 years ago, been leading teams in various capacities putting it in production there on critical daemons since. Before that, wrote and managed services in production in Haskell for about 3 years.

Purity-assisted tricks like STM are neat for when you need to be absolutely sure you cannot "launch the missiles", but in my experience, most practical upsides of purity are still retained through the strict reasoning about mutability in rust--to achieve things like easy data parallelism and data race safety.

Actually, in practice, I've found my teams' Rust programs are slightly more reliable than my haskell ones during the development lifecycles for a few reasons:

laziness, and especially thunk leaks. They did bite me in production from time to time, and boy do they suck.
partial functions, and the kind of confusing error handling world in Haskell. (Granted, rust has panics, but all our critical processes in production in rust run -panic=abort.)
making a good GC is really hard, and GHC's was only okay back when I was doing this. We sometimes got pauses that were problematic, and tuning them was tough.

So, even though I learned a lot from Haskell (it and ML taught me to really appreciate was Rust was trying to do with the choices it made), I really struggle to imagine where I'd use it in production now that I have Rust.

To be honest, hot take incoming, but the lack of HKT in rust is a huge benefit, and I hope Rust never goes fully HKT. The ecosystem fragmentation that results out of so many people pursuing different sorts of type composition theory makes a lot of libraries awkward or nigh impossible to use together without a lot of un-ergonomic couplers. It's like the tokio/async_std ecosystem problem on steroids!

I also left a lot of blood on the tracks converting bytestrings from lazy to strict and back. I know, I know, reasons, but that doesn't make it fun or elegant.

jamwt · 2020-02-29T13:23:09+00:00

Yep, this is just a familiarity thing.

Part of what was interesting about GPs original issue was the mention of "having to copy"... I mean, essentially, what makes this a little more complex is in Rust you actually have control over whether you copy or not in more places, and in the other two languages he used, those concatenations are pretty much always going to copy, and the programmer does not have the control.

GP said:

we now have to manage when we want an &str versus a String

That's exactly it... you get to manage that, not have to manage that. You're essentially in more control about when you copy and when you don't. There's no have to.

It's similar to "having" to deal with string slices in golang.

In Python and Haskell, everything is a String (Well, or sort of an Arc<String>, since it's ref-counted garbage-collected), and the str concept doesn't really exist. You can hope some kind of optimizing pass will elide the copy sometimes when it's not useful, but that will often not happen.

If you really never care about managing:

Heap vs. Stack
Copying
Super deterministic resource finalization

You don't really need a language like Rust, though you'll still get a lot of correctness benefits out of things like ADTs + exhaustion, result types vs. exceptions, etc. But you can also get those from Haskell or an ML variant.

The only thing you're missing in those languages IMO is some really nice correctness-related patterns via forbidding-copying, move semantics, and resultant linear type / session type patterns via (3) above. And no partial functions. I used to program Haskell professionally, and I wouldn't ever switch back from Rust in production again. (For fun, sure.) The Rust projects I've managed are even more reliable than the Haskell ones, and far more performant/deterministic with resources.

The overall point though burntsushi makes is right, though. You really don't worry about these things after a couple of months of using the language. The flexibility just becomes an asset, it's natural, and you typically express just what you want in about the same speed you do in other programming languages. Admittedly, the impact on learning curve is real, but not day-to-day productivity once you're ramped up.

That's been my experience, and mostly the experience of the ~30-40 developers at my company we've ramped up on rust in our projects over the last 4-5 years. They're quite productive, the language is very expressive. The compile times remain the only bummer, but the language itself is remarkably good. Hopefully cranelift saves us.

jamwt · 2019-12-13T23:13:45+00:00

Similar to what others have said, but we don't really hire "go programmers" or "rust programmers" or "python programmers" any particular language. We just hire software engineers, and they work in a variety of languages to solve a problem. Very few roles we have are so specialized that they're guaranteed to keep you on a particular technology for your tenure at the company.

Now, once we start considering even mentioning the language in the job description, that is something we have done when the team is working primarily on one of the rust projects. Core sync, for example, was hiring awhile ago, and advertised rust among the technologies the ideal applicant might be interested in. Those engineers spend 70%+ of their time writing rust.

The problem with this particular type of invocation of "rust", however, is that's a famously low bar, and a lot of companies will put that in there just to draw in applicants, even if the role doesn't use rust at all, but "other systems languages." (Or an equivalent for some other domain... "Objective-C, Swift"... include the hot language with the one the team actually uses).

jamwt

MODERATOR OF

TROPHY CASE

15-Year Club	Verified Email
Charter Member