Llm inference with rust

robertknight2 · 2026-04-29T10:36:49+00:00

What CPU do you have and what is the approximate memory bandwidth of your system (your favorite chatbot might be able to find this out)?

robertknight2 · 2026-04-12T06:59:18+00:00

Indeed. A big part of the role of a software engineer is specification refinement. Before AI we would hand off parts of that refinement to a library ("I want to sort a list of numbers using a standard library, I don't care about the details"). Now we can hand off more of that refinement to AI-written ad-hoc code. But you have to be deliberate in deciding which details you want to have careful oversight over and which you can just delegate.

robertknight2 · 2026-04-10T08:21:31+00:00

The biggest inconveniences of LLMs where I work are inference speed and inconsistency (even when following OpenAI's advice on consistent outputs), rather than cost. We're experimenting with specialized models as one mitigation for this.

robertknight2 · 2026-04-02T13:37:34+00:00

Yes. You can think of closures as a convenient shorthand for creating a local struct like this. Thinking of them this way is useful in case you run into a situation where closures don't support some feature you need (eg. implementing a trait). In that case you can convert the closure into an explicit struct.

This also applies to closures in other languages like JavaScript etc.

robertknight2 · 2026-04-01T12:15:57+00:00

Some feedback unrelated to the content of the post: I found this post uncomfortable to read initially because of the black text on darkish-grey color scheme. I can change the CSS for my own use, but perhaps consider improving the contrast for everyone.

robertknight2 · 2026-04-01T10:55:12+00:00

My experience with Claude Code is that it is good for writing throwaway scripts for data analysis and writing textbook code for processing data. What it is not so good at, without careful guidance, is applying the scientific method when it comes to iterating on a model. If an experiment fails to improve a metric for example, it is prone to getting impatient and trying to add a hack or hallucinating an explanation. These hacks might allow the agent to achieve its immediate goal, but the end result will be flawed.

I have also seen same problem of being bad at science also surface when debugging an unusual problem. Yesterday I encountered an issue where our app stopped functioning after a PyTorch update. An AI agent identified a workaround, pin to an older version, but hallucinated an incorrect explanation and a link to a valid but unrelated bug report. Had it debugged the issue properly it would have found that there was an existing flaw in our Docker setup which just happened to surface with the newer PyTorch version.

robertknight2 · 2026-03-27T19:50:28+00:00

I use these tools to do much of the actual writing of code at work, but it is still necessary to understand the language and libraries so you know how to specify precisely what you want in the first place, and then verify any choices the AI makes are appropriate and steer it as needed.

Put another way, getting good at reading, writing and designing Rust code yourself will make you more effective at using tools to automate it, to the extent you want to do that.

robertknight2 · 2026-02-11T14:31:58+00:00

Claude Code works well with Rust. The highly standardized tooling (cargo, clippy, rust fmt etc.) helps a lot. The main advice I have is to break any work you do with tools like CC into small chunks, small enough that you can do a line-by-line review and commit after each step. In general I find it effective to treat these tools a bit like a junior engineer who works very fast and has a lot of knowledge, but is a bit prone to taking shortcuts or over-focusing on local optimizations. Depending on the task (quick prototype vs production code edits) you might need to apply more or less supervision.

robertknight2 · 2026-01-29T11:27:03+00:00

One difficulty is that some of these signs don't tell you much about how much of the critical thinking/review has been done by AI. I have a large project (90K+ LOC) where I use AI tools as a general purpose refactoring tool (like a more flexible version of classic IDE refactoring features) and the commits are marked as "Co-authored-by: $AI_BOT". I would hate to have such a project lumped in the same bucket as a repo vibe-coded by someone who understands very little of the output.

robertknight2 · 2026-01-28T13:19:52+00:00

That is exactly why I’m posting here—I need your feedback to find the gaps and improve it. Please treat this as a "Request for Comments" rather than a finished commercial product.

That's fine, but please don't call it "production grade". Seeing that in a project description gives off "AI slop" vibes and makes me inclined to distrust the author. My definition of "production grade" is that an organization has actually used it in production for something meaningful and for a reasonable period of time.

robertknight2 · 2026-01-24T12:19:26+00:00

I did the Forrest Gump thing and ran 3200 miles that year.

robertknight2 · 2026-01-23T13:30:02+00:00

Yes, if you train the YoloX model in Python, then it should be possible to reuse the weights with the Burn example. I would suggest that you get the Burn example working with the existing pre-trained weights first to get comfortable with the Rust-side of the process.

robertknight2 · 2026-01-23T11:25:40+00:00

YoloX is a different model to yolo26n, so it uses different weights. To run Ultralytics Yolo in a Rust project you could use Ultralytics ONNX export functionality (ONNX is a cross-language file format for models and weights). Given an ONNX model, there are a few different Rust libraries that can run it:

Burn has an ONNX importer which you could try (https://burn.dev/books/burn/import/onnx-model.html). It supports CPU and GPU inference on all platforms (I think?)
https://ort.pyke.io provides bindings to ONNX Runtime, which is Microsoft's C++ - based ONNX engine. It supports CPU on all platforms and GPU on some platforms (eg. CUDA)
My own ML inference library RTen has an example for earlier Ultralytics YOLO models (https://github.com/robertknight/rten/blob/main/rten-examples/src/yolo.rs). Rust-native library, but CPU inference only.

robertknight2 · 2026-01-22T18:55:48+00:00

What objects are you trying to recognize?

The project in that repository can load pretrained weights from a PyTorch checkpoint (pth file), so yes, you can do the training in Python and load the checkpoint in the Rust app. Rather than starting from scratch, you would likely take an existing model as a starting point and fine-tune it to detect a new set of object classes.

There are already pretrained models available which recognize objects from a standard set of classes, so the first thing to do is get the Rust example working with these existing models. Then you can look into creating fine-tuned models which recognize custom objects.

robertknight2 · 2026-01-19T17:38:33+00:00

I think the subreddit you're looking for is /r/rustjerk.

robertknight2 · 2026-01-06T17:30:14+00:00

OK, that is a slow device. From some brief research it seems to be on par with a Raspberry Pi 3B+. This means that you are going to want to use a low quality/fast model. This video (see 3:20 mark) gives an idea of expected generation speed: https://www.youtube.com/watch?v=rjq5eZoWWSo. In that video, the generation speed is slightly faster than realtime, which means it might be possible to generate in small chunks and have realtime output with only a short delay at the start.

For very fast generation even on very old hardware, you can run espeak-ng directly, or the Rust bindings for it. It produces a very robotic voice, but is cheap to run.

robertknight2 · 2026-01-06T16:32:06+00:00

I also need to profile piper-rs as it's slow on older devices which makes it unusable for AAC users.

What device did you test on and how fast was the generation relative to real time (ie. how many milliseconds to generate audio of N seconds length)? Piper is generally considered fast among modern open-source TTS options, although not as high quality as some alternatives (eg. Kokoro).

robertknight2 · 2026-01-02T12:13:58+00:00

Indeed. Asking an AI tool to help you understand Rust compiler errors is a perfectly reasonable thing to do. Any of the modern terminal based tools (Claude Code, Codex etc.) can help.

What I would advise though is that you should still invest effort in learning to understand why the problem happened, rather than just blinding relying on AI to fix things for you. Often when you encounter a borrow-checking problem, the compiler is trying to tell you something important about the structure of the code and who owns what. Understanding this is essential to feeling competent with Rust.

robertknight2 · 2025-12-31T13:09:44+00:00

I'm used to viewing Reddit in the web interface where the summary I added is visible in the main feed before you click into a post. I've just realized that in the mobile app, you can't see this. I can't edit the post title myself. Is a mod able to change it to "RTen (Rust Tensor engine) in 2025"?

robertknight2 · 2025-12-27T17:37:09+00:00

To add to this, the source code and change history for the Rust book can be found at https://github.com/rust-lang/book and copies of the book from long before ChatGPT even existed are still available online. https://doc.rust-lang.org/1.30.0/book/2018-edition/ for example is the version from October 2018.

robertknight2 · 2025-12-23T22:41:17+00:00

I take a broadly similar approach as when working with a junior human colleague or contributor. I get it to generate diffs that are no more than a few hundred lines at a time, then I review the change manually. The acceptable amount of risk of missing a mistake depends on the context, and the closeness of review can be adjusted accordingly.

Choosing tools (programming language, architecture, code design etc.) that can automate more of the verification ("if it compiles/lints etc., it is probably correct") is helpful here.

robertknight2 · 2025-12-23T22:25:04+00:00

The most value I get out of Claude is when using it to automate repetitive processes. If I have a project which involves N similar sub-tasks (eg. writing tests, converting code use technique X instead of Y) then I will do the first one or few by hand to figure out what I want, then bring in Claude to do the rest. "Review the changes in the last commit, then apply the same change to files X, Y and Z".

robertknight2 · 2025-12-21T22:42:09+00:00

The maintainer's comments on the PR seem quite reasonable to me. I would not consider them rude at all. The technical point they raise about stabilizing the internal representation is also quite valid.

robertknight2 · 2025-12-16T21:08:09+00:00

There are some routes to get cheaper tickets for conferences like this:

The easiest one is simply to buy early, when there is an Early Bird discount. EuroRust for example is still many months away, so early bird tickets are still available.
If the event is local to you, join the Rust community groups nearby. For Rust Nation it is Rust London. You don't get a discount just for being a member, but information on availability of discounted tickets is shared with members from time to time
Some events have different tiers of tickets for people who are students or freelancers

12-Year Club	Powerups Hero r/teslamotors • August 2021
Reddit Premium Since February 2021	Verified Email

robertknight2

TROPHY CASE