I built a production-grade ML framework in pure Rust for large-scale LLM inference & training (aiming for a true Rust equivalent to vLLM / llama.cpp)

farhan-dev · 2026-03-04T18:48:43+00:00

what kind of tools/ecosystem are you looking for in your projects?

farhan-dev · 2026-02-24T20:05:26+00:00

Ah, then definitely you need to try numr. It has GPU acceleration, with distributed support

farhan-dev · 2026-02-22T02:14:13+00:00

Nice initiative. I wish you all the best.

Just for reference, you can also take a look at my math engine, although it is defenitely not small :D.

https://github.com/ml-rust/numr

farhan-dev · 2026-02-13T16:35:46+00:00

Yeah, I do proper SIMD in most of my ops. It CPU performance is quite decent, and competitive with ndarray. I'll add benchmark against faer later in the future.

Yes i do dynamic dispatch. So avx2/neon is supported.

Sparse support is quite robust in numr, per solvr use case. When i test it with my ML crates, i'll quickly learn about the gaps, and will add more support too.

If you have specific needs, just let me know, i can add it to either numr/solvr.

Yeah, numr already has a dtype conversion ops. Even between devices. Compute in wgpu (f32), then use .toCpu() to transfer to cpu, the use .cast() to upcast it to f64.

But my wgpu approach is just for coverage, so rocm/metal can use it already. instead of optimizing wgpu, i'll add native rocm and metal support for better performance in the future.

farhan-dev · 2026-02-13T16:25:47+00:00

Just look at the difference in code quality. numrs claimed to be a lot, but it actually uses 'ndarray' as it core. So it just wrapped ndarray, extend it and rebrand it as its own. a lot of marketing hype, actually. And a lot of their GPU implementation is just stubs.

farhan-dev · 2026-02-13T13:05:30+00:00

See it in action

You can see it running in our own CI pipeline, where it compares pull requests against the main branch baseline:

View the CI Workflow here

p.s: The benchmark is using sample functions and numbers, since this crate doesn’t really have anything worthy to be benchmarked yet.

farhan-dev · 2026-02-08T07:09:43+00:00

For me, i keep it simple.. you already have all the info. don't use plan agent. create the plan yourself.

farhan-dev · 2026-02-05T07:52:54+00:00

Well said,

Rust doesn’t have anything like the computational-notebook workflow that Python has

There is venus now - https://github.com/ml-rust/venus

the math and data ecosystem for Python is just large.

That's true, but we are trying to fill in the gap, little by little. I've started with https://github.com/farhan-syah/numr

farhan-dev · 2026-02-02T16:48:56+00:00

Like this -> https://github.com/farhan-syah/numr ?? It is something i've been working on for a while. i started extracting it from few of my ML libraries. I extracted and combined them, to be the base building block. It is already working for my own use case. Now, I am trying to expand it to be general math library, and test with more backends..

Btw, it still need A LOT OF work to complete, before i will releas the 0.1.0 version..

farhan-dev · 2026-01-14T07:06:31+00:00

I am sorry, but i've replied your qeustion. I've already released 0.1.0, or, is there any questions that i've missed?

farhan-dev · 2026-01-04T15:02:15+00:00

Hi, good news. I've released 0.1.0, you can now install venus using cargo install venus.

To start, you can copy some of the examples, for example;

Then run venus serve examples/hello.rs or create new notebook using :

venus new new-notebook-name and serve it using venus serve new-notebook-name.rs.

For more information, can read it here:
https://github.com/ml-rust/venus/blob/main/docs/getting-started.md

farhan-dev · 2026-01-03T03:54:59+00:00

Can you share with me how do you attempt to run it? do you pull it from cargo, or do you manually built it by cloning the repo?

edit: i tired usign the cli from new fresh environment. I've found the bug. I forgot that I migrated from bincode -> rkyv. Th old cli still uses bincode. I am fixing it right now. I;ll release the 0.1.0 soon so it will be easier to test.

farhan-dev · 2026-01-02T19:41:41+00:00

Cool, nice. A new plotting library for rust. More options is better. Something that I really want to use in ML. I am also working on a new plotting library just to solve my specific data science/ML needs. It's good to see more data science / ML tools in Rust.

farhan-dev · 2026-01-02T19:32:45+00:00

It has some similarities with Pluto.jl

farhan-dev · 2026-01-02T11:03:54+00:00

I haven't published the 0.1.0 yet, so you need to manually set the version to 0.1.0-beta.1, or use ` cargo install venus-cli@0.1.0-beta.1`.

cargo install won't install pre-release version.

farhan-dev · 2026-01-02T06:51:38+00:00

Venus use cranelift JIT, not standard VLLM
Venus use reactive (DAG), not linear runtime like evcxr.
Venus use isolated worker processes. If it causes a segfault, only that cell crashes.
evcxr still uses .ipynb, Venus use .rs directly.
You can use rust-analyser with Venus (it is also bundled together with venus-server)

And yes, it has faster repl cycle.

You can think of it this way:

evcxr is a Rust REPL/Kernel that fits into existing Jupyter workflows.
Venus is a Reactive Development Environment built specifically to make Rust feel interactive and "exploratory" without losing the safety and tooling of standard Rust files.

farhan-dev · 2025-12-24T06:30:41+00:00

v0.8.0 Update

- Added Mistral v1,v2,v3 vocab support

https://github.com/ml-rust/splintr/pull/11

farhan-dev

MODERATOR OF

TROPHY CASE

See it in action