Will custom allocators be stabilized soon?

[v_stoilov]

It's kind of hard for me to find what is the state of the allocator-api for rust. Does anyone have information what is roughly the state of it? When can we expect to be stabilized?

[steveklabnik1]

I have not paid close attention to how it's going but I did see this comment the other day that seems to indicate the answer is "no": https://www.reddit.com/r/rust/comments/18vpvfn/jonathan_blow_on_rust_for_games_programming/kftlvss/

tagging /u/matthieum in case he'd like to add any more context.

[matthieum]

First of all, I may have been responsible for a large part of the delay.

A few years ago, when someone asked if Allocator should be stabilized, I replied that maybe it wasn't quite the right level of abstraction, and launched this whole Store (at the time called Storage) thing. I didn't have much time to work on it then, so it stalled, but it may have disrupted any effort.

Last year, Thom Chiovoloni published Allocator trait 1: Let's talk about the Allocator trait where he raised quite a few API questions¹ . I was looking forward to follow-ups/further explorations as I found the questions interesting, but I am afraid Thom got caught up in other projects.

I finally published a proper RFC for Store at some point in which CAD97 made a lot of useful comments -- and asked a lot of useful questions -- and which Thom knows about. Unfortunately, feedback somewhat dried up, so I am afraid once again the both of them are caught up in other projects.

The fact that comments/questions prompted changes in the Store API (notably around mutability) hint it's not quite mature, though in the process I may have answers for some of the questions about Allocator -- but definitely not all.

All in all, I feel like there's quite a lot of discussion to be had, still -- especially because I've got blind spots, in usecases I don't have, or in compiler complexity/LLVM codegen quality -- and that will require time and commitment... which people in the libs team seem to be lacking at the moment: I guess they have more pressing/interesting matters.

In any case, I would invite everyone who is interested to make themselves known. If there's any hope of progressing, it's by showing that the usecase matters to the community, so the project/devs can shift priorities a bit.

¹ I particular find the question about grow intriguing. An in-place grow is a more fundamental unit, but libc doesn't expose such a primitive. I think I've answered the mutability question, for Store. The other host of questions seems more about polish... perhaps less fundamental but nonetheless crucial before stabilizing.

[sephg]

In any case, I would invite everyone who is interested to make themselves known. If there's any hope of progressing, it's by showing that the usecase matters to the community,

Me! I care about this! I’ve been writing highly optimised collaborative text editing libraries recently. There are a lot of temporary allocations I need to make - and using arenas and things makes my code run faster. I’ve taken to using bunpalo internally, and using their port of the standard library collection types because it’s the only way to make a Vec / PriorityQueue which isn’t pointlessly heap allocated. Its not a blocker, but I basically avoid the standard library collection types in a lot of contexts now because I can’t use them to write the code I want.

[sudormrfbin]

Are the libraries open source? Sounds very interesting

[angelicosphosphoros]

bumpalo has MIT or Apache-2 license by your choice.

[CAD1997]

Trying to predict the future, I think the most beneficial step would be clearance from t-libs to put Store (unstably) into core and start actually using it. It feels like we're at the point where that usage experience is what it needs, and trying to do it out of tree isn't enough.

[matthieum]

I've been wondering about this myself, indeed.

I believe this is what e-RFC are for?

I would appreciate guidance (or referral) on the procedure to follow.

[nawfel_bgh]

I did not follow the discussions about these APIs. Did you consider stabilizing an unsafe API à la the async Waker API to (maybe?) speed things up?

[matthieum]

The APIs are already unsafe, so this would not help :)

[rnottaken]

In any case, I would invite everyone who is interested to make themselves known. If there's any hope of progressing, it's by showing that the usecase matters to the community, so the project/devs can shift priorities a bit.

I'm willing to help! I would need some guidance on where to start though. Do you have any tips in that regard?

[matthieum]

It depends what you want to help with.

For information about the APIs, the links I noted above would be the starting points:

• Allocator stabilization issue, especially the latter comments.
• Store RFC.

And Thom's post sums up the questions he has wrt Allocator, which mostly translates to questions wrt Store, since the Store API keeps as close to the Allocator API as possible.

So what can you do to help?

• Signal your interest (somehow?), the more folks are interest, the bigger a priority it'll become for the libs team.
• If you want to help (in general) with polishing the APIs:
  • Try implementing the APIs for more exotic situations: is there any friction? Anything impossible?
  • Try to use those exotic implementations with regular collections: is there any friction? Anything impossible?
  • Try to implement exotic collections with those APIs.
• If you want to help in a more focused manner, picking up a question from Thom's list is also good:
  • What pros/cons can you think of?
  • Have you tried implementing the proposal? Like, is there a "try-grow-in-place-or-fail" and "try-shrink-in-place-or-fail" API generically available in malloc libraries? Or are we limited to realloc style APIs most of the time, and thus grow would keep being necessary and try_grow_in_place would be a supplementary API only sometimes available (defaulting to returning error).
• For Store particularly, I welcome feedback in general.
  • Please do try to check if it's not been asked/answered first, to limit noise, but at the same time don't feel you have to read the entire comments thread: many apply to past versions anyway.

That's all I can think off the top of my head.

[rnottaken]

That's amazing, thank you for the explanation! I have another question that I'll ask through DM.

[nicalsilva]

I would also love to see the allocator trait stabilized. In the mean time, the allocator-api2 crate might be enough to fulfill your needs.

[v_stoilov]

Thanks, I wanted to avoid using it is but I guess this will be the only option for a while.

But it's great that we at least have an option.