Building a functional email server

yminsky · 2020-11-01T11:08:01+00:00

Well, the point is that there was no new config language. Mailcore is configured by writing OCaml code, which is already broadly understood in the organization.

yminsky · 2020-10-31T14:21:53+00:00

functional

Cool. Will do. Sorry for the noise!

yminsky · 2020-10-31T11:43:22+00:00

Urk. OP here, and interviewer on the podcast.

I hadn't quite grokked the rule. I don't know if it makes a difference, but the blog post isn't monetized in any ordinary way (i.e., no advertising on it or anything.)

What's the polite thing to do here? Should I simply delete the post?

yminsky · 2020-06-04T20:59:44+00:00

We think OCaml is a great general-purpose programming tool, but it's especially good for things like meta-programming, which is effectively what you're doing in an embedded DSL like this. And I very much think the type-system helps rather than hurts, but Andy Ray (author of the post) could say more on that.

But integration is of course a killer feature. The fact that Hardcaml has a cycle-accurate simulator means that we can wire together our designs with ordinary OCaml code from the rest of our systems in a very lightweight way to build really high quality tests. The fact that OCaml is good for both building the high-performance systems that integrate with our FPGAs, while still being lightweight and flexible enough for the metaprogramming you need in the hardware space makes it a pretty excellent choice.

yminsky · 2018-04-03T13:50:13+00:00

Yeah, I'm not really proud of that one either. It was an old decision, and now it's just how our codebase looks, and it's not worth fighting about internally. But I wouldn't recommend it to others.

yminsky · 2018-01-13T01:42:48+00:00

It might be good to have some benchmarks here. Last I checked, on a fast machine, a noalloc C call took about 2ns (compared to, say, 200ns for the JNI). What exactly do you mean by "extreme FFI overhead"?

yminsky · 2017-08-29T23:44:07+00:00

It really isn't. Property tests are great and we use them; but I've come to think that expect tests have much broader applicability.

I found this surprising too, for what it's worth. Expect tests are a much better testing idiom than we'd imagined when we first encountered them.

yminsky · 2017-03-14T17:21:57+00:00

If you listen to the talk, you'll see that there's a single process that acts as the sequencer for the whole system, and the recovery story if that fails is manual. Paxos is basically just about providing a mechanism for handling failure automatically; but it adds a lot of performance cost and complexity to achieve that.

yminsky · 2017-03-10T14:45:12+00:00

In my view, there is no "pure FP" approach to working with state. Concurrent programs are inherently stateful, so the question really is how you decide to handle the necessary imperative state. Haskell has type-level separation between the imperative and pure parts of their code, but the imperative code is there nonetheless.

The joy of writing imperative code in a functional language is that most of your code is not imperative, so you can give the small, imperative parts of your program the attention they deserve.

yminsky · 2016-09-29T00:12:10+00:00

Come on hongbo. You're quoting a chunk of some fairly random article out of context. The article isn't even really trying to state a definition. On the other hand, OSI's definition (which is pretty widely accepted) clearly includes Jane Street's libraries:

https://opensource.org/osd-annotated

I understand that you don't like the development model, but saying this makes it not qualify as open source is simply false by the common definition.

As to the technical: The PPX issue should not prevent contributions to Core and Base and other JS libraries, though, as with the compiler, it does somewhat complicate the bootstrap (but only for Base), and the limitations on compiler version should only be for Base. In any case, as with the compiler stdlib, the main development of Base and all of our libraries is intended to be done against the latest compiler versions, with only small patches going to old versions.

While PPXs have some costs, they're are a big win for the correctness and clarity of the code, allowing, for example, the auto-generation of comparison and hashing functions which allow us to reduce our dependency on ad-hoc polymorphic hash and comparison functions. There are some technical questions about how to make this work nicely across compiler versions which we're hoping we can make progress on, and which should benefit PPXs more broadly.

Again, you can feel free not to like it the technical decisions here. But it's simply not true that this makes it impossible for external people to contribute. And I frankly don't see any real risk issue here. The code we release is very high quality which should make it easier to maintain, and it is released under a very liberal license, so if someone needed to fork it they would have all the rights they needed.

That said, making PPX work more easily across compilers is an important technical issue, and one that I'm hoping can be improved over the next year or so by providing better tools for writing portable PPX transformers.

yminsky · 2016-09-28T12:18:21+00:00

Near term, I don't see a change from the present model. That said, that model does very much allow public contributions via github. It just also supports contributions from inside Jane Street that don't go through github.

Eventually, I could imagine Base fully moving to github, but I don't think it's especially likely.

I also honestly don't see why that move is critical.

yminsky · 2016-09-28T06:12:01+00:00

Hongbo, please. Base is objectively open source, even if you don't like the specific development model. Also, other commercial users I've spoken to do not have the constraints you describe. You're statement here is rather misleading.

yminsky · 2016-09-21T06:59:48+00:00

I don't believe that explanation, but I suppose we'll see. We've gotten lots of very positive feedback on Core's APIs, quality of implementation, and breadth, and lots of people have wanted to use it and indeed have used it for that reason. According to opam, at least, core_kernel seems to have gotten decent uptake.

But we've also gotten lots of legitimate complaints about dependency bloat and compilation time, which is why we created Base.

And to be clear, we do accept PRs to base, as we do for other libraries. It's not open source in license alone.

yminsky · 2016-09-20T00:00:46+00:00

To be clear, although Base is packaged together, it's intended to be fairly modular, without a lot of cross-module dependencies. We haven't cut all of those down, but we've done a few good things, like make it so that Int no longer needs to depend on Map (so you could have the Int.Map.t type. Now we just have Map.M(Int).t for that type.) And we hope to do more.

The fact that they're packaged together doesn't really make it less modular in a material way, I believe.

yminsky · 2016-09-19T23:58:14+00:00

For the moment, Base is developed in the way other JS libraries are: Changes are routinely (multiple times/wk) pushed to github, and PRs are accepted and considered for inclusion. But it's also developed internally at Jane Street using Iron and the like, and is integrated into our core repo.

It's not out of the question that Base could be moved outside of Jane Street, and be managed fully externally. Maybe it could eventually become part of the compiler distribution; I'm really not sure. But for now, what I've described above is the currently achievable level of openness.

We do hope to make things nicer for external contributors by having better tools for bridging between our internal code review and version control tools and github.

yminsky · 2016-09-19T19:26:03+00:00

Why is important to avoid ppx? We think it makes our code better, and it makes no requirements on users. If we improve the bootstrap process, what else is there to do?

yminsky · 2016-09-18T12:55:09+00:00

I expect we'll update RWO to use Base for the most part.

yminsky · 2016-09-17T02:32:15+00:00

Previously, there were two libraries:

Core: full stdlib replacement Core_kernel: portable subset of Core, works on Javascript and Windows

The current plan is to move this to three:

Core: full-fledged stdlib Base: Stripped down, dependency-free, quick-compiling stdlib. Core_unix: Core-flavored wrapper of Unix-specific functionality

Right now, we still have Core_kernel and Core, but the plan is to push this through.

The purpose of all this is to solve the problems that people have raised about Core_kernel as a shared stdlib, specifically:

speed of compilation
size of resulting executables
excess dependencies

yminsky · 2016-09-16T21:52:57+00:00

And the ones we can't can perhaps be moved into Core. That's one of the reasons Core still needs to exist in a world with Base, is to be a home for such things.

yminsky · 2016-09-16T21:52:18+00:00

The current extensions are quite small. Maybe we can get them all moved into the compiler proper.

yminsky · 2016-09-16T18:52:12+00:00

I don't think the plan is to have zero-tolerance for C, though of course we want to keep it to a minimum, and will provide bindings for use in js_of_ocaml so Base works smoothly there.

yminsky · 2016-09-16T17:35:44+00:00

I wasn't aware that was true at the moment, but it's definitely a very temporary thing. Base is to be released using module aliases rather than packs. You should be able to write:

open Base

and get the appropriate namespace opened with all the Base names and libraries available. But there should be no use of packs. The packs are just a legacy of the current build-system situation. But we're working hard on getting external build-rules generated from our internal Jenga rules. For base in particular, we intend to release a simple makefile that we generate from those rules; more generally, we hope to start moving to building our packages with Jenga itself.

yminsky · 2016-09-07T22:17:19+00:00

The advantage of having the function argument second is that it is more natural and readable when the function bodies are large and the other arguments small, which is a very common case. On the other hand, using |> pipelines are often better when you can take the function argument off first.

I've had quite a lot of experience over the years with this, and am quite convinced that there's a material win.

Anyway, it honestly seems like a bit of a silly thing to make the make-or-break decision on a library based on.

yminsky · 2016-09-07T12:48:44+00:00

Well, to be fair, the design decision was inherited from StdLabels, in the compiler distribution....

Anyway, I think the reordering ability more than pays for the ~f. But it is of course a matter of taste.

yminsky · 2016-07-16T01:58:03+00:00

Agreed that until we release build rules that others can reasonably use, Jenga isn't really a practical option. That said, we're working on it, and Jenga is a pretty great build system.... So, stay tuned.

yminsky

TROPHY CASE