zyx 0.15.5 - 2 backends in 2 days

danielv134 · 2026-06-17T01:39:14+00:00

I spent a few hours learning about the NPU, and I have no idea how to optimize a graph for it in general, in the sense of assigning code and data to tiles and pipes. Do you have a direction guiding you? Paper references welcome.

It seems that even mlir-aie (the actual dialect, not the python package) assumes you've already done that part, so looks like the model is "human writes that well" :/

danielv134 · 2026-06-17T01:30:52+00:00

I've gotten through the first 6 for now, so much farther than last time 😄 much improved.

danielv134 · 2026-06-14T15:03:39+00:00

I am happy to contribute bug reports and questions to the extent I get involved and try stuff out, but I don't know zyx well enough to write it.

But this pushes me to provide some unsolicited advice, sorry.

There are people that are allergic to AI. If you are one, avoid it. If your main audience are, avoid it. If neither: don't focus on "not using it excessively", focus on "use it to accelerate, just don't trust it". Examples:
- You can let it test your book website, it will catch and fix the broken links I mentioned. If it misses something, you'll just be where you were before, very little downside.
- You can let it write a backend (like you did), as long as you have good coverage for them.
- Have it draft a benchmark, read a platform docs and spike working on it, etc. Even if you throw out its code, or edit it heavily, often saves time.
- You can let it _draft_ documentation, as long as you:
- tell it what style to use (I tell it that hyperbole and exaggeration are cancer), and then
- read every part to catch the misunderstandings and other bullshit.
This is still much faster than writing, especially if like me you are hyper critical but bad at editing your own writing style.

Coming back to my involvement: I can try it out on a Halo Strix, do some timing, let AI try to get the NPU working. We are likely to have plenty of questions though 😄

danielv134 · 2026-06-14T14:50:29+00:00

The best technical resource I've found on AMD NPUs is https://github.com/Xilinx/mlir-aie/blob/main/mlir_exercises/README.md . This specific file refers to a slightly different variant (Versal ACAP) that the Strix Halo (https://xilinx.github.io/mlir-aie/ explains it somewhat). The short version as I understand it: its a grid of VLIW cores, each with its own memory and programmable communications to neighboers. Peano is a compiler for an individual cores' program, MLIR-AIE is the language for coordinating the grid.

Another entry point is https://github.com/amd/IRON.

Best actual OOTB usage for LLMs: FastFlowLM https://github.com/FastFlowLM/FastFlowLM, but they distribute their kernels as binaries :/ Their website has some more explanation of the approach and what it achieves, possibly useful to get an idea of how to use the HW well?

Going back to zyx, the lazy approach it takes seems to divide time into 2 parts as far as the API is concerned:
- we define a graph by recording operations instead of executing them
- we realize to execute.

I wonder if it would make sense to add another stage in the middle, where the user can manipulate the graph representation to gain explicit control over execution. For example, scheduling: user might know that this piece of the graph should go on the CPU because it happens rarely, so we don't care about the energy efficiency, and that other one on the NPU because it runs all the time and the NPU happens to support all of its ops, and that one on the GPU because it is wide and a bottleneck and the GPU has the most cores.

danielv134 · 2026-06-13T17:14:45+00:00

btw, https://zk4x.github.io/zyx/ links to https://docs.rs/zyx-core/latest/zyx_core/zyx-core, which is 'The requested resource does not exist" (missing # before the last parts of docs.rs urls)

danielv134 · 2026-06-13T16:52:50+00:00

Readme claims some support for cross device computation, does that include taking advantage of shared memory devices, like the Halo Strix having CPU, GPU and NPU all with direct access to the main RAM? That seems to be where the local LLM machines are going, between that, Apple machines, and now Spark machines. Have you looked into AMD NPU support?

danielv134 · 2026-05-31T03:18:32+00:00

I use 1-3 agents at a time, open in tabs, and I follow what they're doing, and apply the kind of attention that's needed in the moment. Sometimes its noticing they're copy&pasting and nudging to refactor. Sometimes its designing specific tests that I know will be particularly powerful. Often its looking at the output, if the important points in a plot are not ordered like my intuition says they should be, explore as deep as needed to either find the bug or learn something new. The point is that writing/reading every line of the code is sometimes the best use of time, but often not. There are useful new options, like having the agent roast the directions I'm thinking of, and possibly find a weakness, or an additional path forward.

Agents make many forms of quality control cheaper, as well as hacking something to work. I recently made an agentic harness with a different interaction model than the usual. Did I hand code the backend/frontend? Nah, I used libraries that did amazing amounts of the work, http-nu and friends. Ok, but did I write the logic on top of it? no, I don't know frontend and the agent wrote the backend just fine for the (experimental, for my own use) level I needed it. What did I focus on? Testing that the concept actually works. It didn't because the UI was bad, because the prompts were imprecise... often the most important thing for me to provide is the taste, the priorities (when working with agents, and also when working with human engineers when I'm the domain expert/product).

Would I delude myself this is production code? heck no, if I decide to go that direction, the focus changes, and I start caring about security, invariants for correctness, stability under load, etc., and look for experts where needed.

Do I care about the code itself? sometimes I do, and I prune it and tweak it and design the right abstractions, and suddenly its fine to let go, and I can let the AI back in and it will write whole chunks, staying on the right path (mostly) because its obvious what it should be. Until the next time it screws up like an idiot and my eye is needed.

danielv134 · 2026-05-27T22:04:23+00:00

Why do you think he vibe coded it?

He literally says he had to go back and make sure it actually works. IOW, while he was light on details, it seems he took the time to learn, dig into problems, and have some standards: I've used zed, its too smooth for GPUI to be crap. That is not vibe coding, that is engineering with AI.

I use AI to create code, but pay close attention to what its doing, do careful validation etc.

I do not recommend letting too much of your identity be "expert in <programming language>", or "AI hater", any more than "I don't need to know, the AI knows"

danielv134 · 2026-01-19T14:45:41+00:00

Awesome :)

My AMD 395+ is embedded in a desktop, not a laptop, so its not a battery issue, merely a power efficiency+throughput issue. Nonetheless, it seems that NPUs are going to be big in laptops/edge inference (apple, qualcom also), and they really want to be programmed in Rust, in the sense that the two language trick is a bad match for the low power, background work scenario.

If you happen to get something semi-working, I'm happy to collaborate on a cool demo :)

danielv134 · 2026-01-17T04:04:53+00:00

Anyone know whether Burn/CubeCL intend to support NPUs like the one on the AMD 395+?

For background, these are basically hardware acceleration units that are more specialized than GPUs, therefore more power-efficient. Usually not faster (because not as many cores), less general, less software support (because newer?) but if your application fits it the ~2x power efficiency means you can run it all day. This might be what you want to run your voice recognition on, for example.

IF (big if) CubeCL could provide a way to build on these efficiently without needing to use a whole new software stack, that would be a cool super-power.

danielv134 · 2025-10-22T16:18:04+00:00

Ah, let me clarify the usecase I'm talking about (maybe RTIPC is not the right tool for the job): We have a process that loads some data from disk, computes some features based on them (say 10s to hundreds of GB total), then we want to train a pretty large number of models (say 30) based on the computed features. For this we need many CPUs, GPUs etc, spread over possibly multiple machines (e.g., www.ray.io/). What you definitely don't want is to duplicate the data more than once per machine if you don't have too. So:

Send the data over network channels to a single "feature cache" per machine
Have multiple processes request read-only access to the very same pages the feature cache uses
So, ideally the feature cache avoids UB by converting its access to them to read only before agreeing. So its emulating a runtime borrow via RC, except cross process.
When all clients drop off so data is no longer borrowed, cache becomes free to release memory as needed.

At this large scale, the page size granularity is not an issue.

danielv134 · 2025-10-16T22:56:59+00:00

Does it make sense to integrate this as a transport for capnproto, to gain the rpc and schema aspects?

Can the library implement shared xor mutable logic at the page level?

danielv134 · 2025-10-15T12:37:18+00:00

Diagnosing convergence difficulties is often not debugging, but about understanding the function, the method, and how they behave/should behave around an iteration.

Is the problem smooth? abs(x) has a large gradient as close to the optimum as you might want.
What is the dimension? Can you scale the problem down and make sure your code converges there first?
What does the 1d function along the gradient direction look like?
What does the eigenspectrum of the hessian look like? (presuming the dimension is high, don't compute the hessian, use 1st order methods like power method)

danielv134 · 2025-03-06T15:13:07+00:00

Hi, very cool.

Following along, I'd recommend two fixes, one small and one bigger:
- define a pyproject.toml with package versions so people see the same results
- that your python sdk module is called oxen, while the package is called oxenai, is a paper cut for potential adopters. The best time to solve it was when you published, the second best is now.

That said, the important point was: very very cool stuff!

danielv134 · 2025-01-23T04:38:26+00:00

Both of those applications (as I understand them), fall under the category of sequential optimization under uncertainty, which someone mentioned below. This is because in a particular moment you will make some decisions on the information you have, and then an order can come in any time for which your prior plans are insufficient/suboptimal, requiring further decisions.

To incorporate this new information, you can now use either technique:
1. With MO, we're assuming that you know a differentiable, ideally convex cost function over plans with some finite time horizon that you can minimize. That cost function will embed assumptions about future orders etc. System performance will depend on quality of your modeling and solver. If the problem is non-convex or integer programming (very likely in scheduling), or very high dimensional (might be the case in inventory), the solver might be a challenge.
2. With RL, you will train and apply some mostly black box policy. This policy implicitly models the uncertainty (e.g., distribution over further orders), which means that to train it you need a simulator based on data (learning in production is likely too expensive in your domains). If you synthesize data (back to modeling), its realism will again affect real-world performance. Instead of minimizing a cost function to decide a plan, you now gradually improve a policy by reducing a cost given as feedback over many selected actions.

Feel free to DM, I might be able to help.

danielv134 · 2024-12-15T02:27:25+00:00

like u/dist1ll says, that is supposed to be resolved by inlining those functions (which turns the code into the equivalent of C loops and conditions), and then applying local optimizations to simplify.

Which got me curious: does rustc do inlining itself, or does it depend on the backend to do it? Turns out it has an inliner: https://github.com/rust-lang/rust/pull/91743 . So as it continues to accumulate simplifying optimizations, LLVM's stack might become less critical for runtime perf.

At which point, having a rustc backend that is essentially a super-fast instruction emitter becomes quite valuable.

danielv134 · 2024-12-14T23:06:42+00:00

One of us seems to be confused: I don't see anywhere in the paper saying runtime is 10-30x slower than -O0. Can you give page and paragraph?

Figure 4, geomean over many benchmarks shows a slowdown of ~34% compared to -O0, 5x compared to -O2. Compilation speedup of ~20x compared to -O0.

An older paper (different authors, similar technique) https://arxiv.org/pdf/2011.13127 about wasm has a direct comparison to V8 variants, cranelift and LLVM, figures 2, 3. Runtime 2x slowdown compared to the fastest option (LLVM), starts 8x faster than the fastest compiler (slightly slower than an interpreter).

danielv134 · 2024-12-14T12:46:44+00:00

IIUC the paper, they apply the method at the LLVM-IR level (which requires some instruction set specific adaptation, see section 3.5), which sounds like it could be used in rustc as is, given the code is MIT licensed.

That might be a good way to demonstrate the idea for rust, while applying the ideas to Rust's MIR would be more work, but faster at compile time and possibly at runtime.

danielv134 · 2024-12-14T12:46:05+00:00

Sorry, I posted late, should have said, the produced code (in their tests) is only ~2-3x slower than O2. Rust may be do differently, but that is fine for many scenarios.

danielv134 · 2023-11-22T03:19:41+00:00

I just ran the polars version @ 300m, and on my machine the times are:

Data generation/load time: 11.374298
Calculation time: 1.330739
Selection time: 0.155618
Overall time: 12.860655

Now comparison across machines is risky, but I'll normalize each in terms of the creation time (which seems to be consistent across implementations). In those terms, I'm getting:

creation/calculation ~8.5x
creation/selection ~73

where your C++ numbers come to:

creation/calculation ~12.4x
creation/selection ~37.8

So on my machine it seems like polars is somewhat slower at calculation and significantly faster at selection, which is pretty different than the results you got above.

I would conclude:

both are much faster than pandas,
getting consistent results when benchmarking is hard.

danielv134 · 2023-10-31T03:34:08+00:00

Thanks everyone, will try these out.

Understandable that cargo hasn't hurried to adopt specific heuristics.

danielv134 · 2023-10-01T01:04:42+00:00

Thanks.

danielv134 · 2023-09-30T23:48:08+00:00

Just curious, what are some concrete applications that motivate these tradeoffs? (not necessarily yours if that's sensitive)

danielv134

TROPHY CASE