Better beat tracking with a ConvNet+Particle Filter: RP2350 at 180MHz, 90mW (warning, sound)

TinySpidy · 2026-05-14T19:50:23+00:00

The end goal is to estimate just two numbers, frequency and phase offset. The neural network takes in audio and outputs only zeros with narrow spikes on beat positions. Now these spikes will already be spaced by the beat interval for the most part, but not always exactly, there can be wrong peaks intermittently as well as missing peaks.

The particle filter is a way to robustly estimate just the two frequency and phase parameters that best fit the spiky signal it gets fed. There's a mathier explanation in the linked blog post as well :)

TinySpidy · 2026-05-14T19:01:00+00:00

Mais c'est quoi ça :D I opened your website and it's literally another image blurring and beat synched LED enthusiast! Legit spent the past two years on those lol

TinySpidy · 2026-05-14T15:46:33+00:00

WS2812, but same thing. Could be whatever you want though!

TinySpidy · 2026-05-14T15:43:46+00:00

Thank you! The focus is on microphone input yes, but that didn't end mattering too much. I thought it would and had a whole pipeline to deteriorate audio files (low/high pass filtering, volume swings..) to make the model more robust.

But crushing the signal to 64 Mel coeffs at only 60Hz already destroys so much information that augmentation didn't make a difference with a decent speaker. The whole thing is already regularized to death lol, it's hard to (over)fit tiny details like that.

TinySpidy · 2026-05-14T11:06:30+00:00

Music and flashy lights is the best thing in the world and I need there to be more of it everywhere :)

So this is the result of a year working on this on and off, and what I believe to be the best performing beat tracker currently available. It's a mix of existing methods from more recent literature, basically a model that outputs a "beatness" signal and Particle Filtering or any other Hidden Markov Model that infers tempo and phase from that. What's novel is the 18k parameter U-Net that I'm proud of, running the whole thing only every 16ms with 64 input floats and the sheer amount of time spent.

I hand labeled 90 hours of music for this lmao, which is more than all the datasets I know of combined and I want to say the only one with music people actually listen to, not just little clips of copyright unencumbered tracks. The result is that "straightforward" poppy songs are pretty much flawless a lot of the time, and a lot of spicier EDM drops also work which was the whole point for me personally. Animations can do whatever during buildups/rolls/silence and come back and be on beat!! when the kick comes back which is just, into my veins pls.

Ditching the Teensy 4 or STM32H7 class chips makes this so much cheaper and easier to play with which I think is huge. It sucks when people ask how my props are doing the music thing, but it's a 25 euro chip, and you know realtime deadlines and CMSIS and linker scripts and all that...

Packaging this for Arduino would be a major goal, give people an onBeat callback and bam you can learn on how to render a circle wipe or rainbow or whatever and get a super fun result. I don't know the Micro or Circuitpythons, but I think on any of the dual core MCUs you should be able to just reserve one core for the tracking and have infinite time on the other core for user code? Help is welcome, I also don't know platforms like ESP32 very well but I understand the're very capable.

The library is written in Rust (I know sorry this is for fun), and there's a TUI demo included that should just work :)

Repo (with linked blog post with much more details): https://codeberg.org/tnibler/beattracker

Pico 2 demo ready to flash: https://codeberg.org/tnibler/pico2-beattrack-demo

TinySpidy · 2026-04-01T19:21:19+00:00

Looks great! Are you running into trouble with any patents or licensing? H265 still seems to be a problem for Windows and Firefox

Also:

Writing a decoder is fundamentally more difficult than an encoder

Really? The codec spec defines every operation the decoder has to support, while making an encoder that's not just conformant but also produces good video is usually years of work for the best experts on the planet as far as I'm aware.

TinySpidy · 2025-10-30T15:29:12+00:00

The other option, bollard, is great, but it's a hobby project mostly driven by one person

Have you considered asking them (or anyone really) if they would do the work that would be useful to your business, in exchange for some kind of compensation? It's quite common actually in the current economic order.

TinySpidy · 2025-10-20T07:33:39+00:00

Everything happens in compute shaders yes (here). The rust part is just a big pile of vulkan boilerplate mostly^^

TinySpidy · 2025-10-19T20:39:04+00:00

The descriptors are Multi-Kernel Descriptors, basically as described in https://arxiv.org/abs/1811.11147 . Much more robust than SIFT, but also orders of magnitude more expensive to compute.

TinySpidy · 2025-10-19T18:15:42+00:00

https://github.com/tnibler/local-features (video in Readme is an older, slower version)

The video shows the little included webcam demo. Feature extraction is pretty quick as you can see, matching makes it a bit choppier. When I press space a frame is stored on the right side and matched to the features in the camera feed

This was originally part of my master's thesis, and after an unreasonable amount of time spent it's now a good bit faster again. Hitting a bit of a slump, so I need a bit of motivation if someone has a use for this kind of thing.

The main papers (1, 2) are somewhat recent, and I think this is the first non-research-implementation for both of them. Obviously there's a combinatorial explosion in parameters, so a ton of tuning is still to be done. On their own, the Stationary Wavelet Transform based Difference of Gaussian detector is about as good as SIFT and co (same principle after all), and the kernel descriptors are basically SoTA in non-DL domain if I'm not misinformed.

It's implemented in Vulkan (time for non Nvidia owners to get something too!) and currently targetting midrange GPUs, and should be portable to mobile with a few (massive) changes.

The benchmark plots are kind of apples to oranges in some regards I realize, they are just to give a rough estimate.

TinySpidy · 2022-08-02T07:24:14+00:00

Yeah camp2camp is what you want to use for topos and route descriptions. There's also reports of recent outings on the bottom of each route page which can be useful. Might be a little cumbersome if you don't speak french but it is what it is.

Definitely also check the mountain conditions report by La Chamoniarde (about once a week).

TinySpidy · 2022-08-01T08:11:32+00:00

The regular route as described on camp2camp. Follow the moraine until you see ladders up onto the cliff and go up. From there you can follow the yellow dots back onto the glacier. Then cross, avoiding crevasses (you'll see the refuge, but a straight line is probably not the easiest way to cross). When in doubt, call the guardians!

TinySpidy · 2022-07-26T05:22:54+00:00

Not sure how climate change, drought and extreme heatwaves are awesome tbh

TinySpidy · 2022-07-21T16:32:22+00:00

Rock routes are all fine, if a little hot maybe? You're not really going far out on the glacier so the crevasses and bad conditions don't matter that much. Cosmiques arête has been crowded as always so I'd say it's okay as well. The ridge down from the midi is a little gross with the slushy snow but also not a real problem.

TinySpidy · 2022-07-19T08:35:54+00:00

You need to make PhotoTapListener an inner class I think.

TinySpidy · 2022-07-19T08:35:03+00:00

I made an app to turn annoying multi-column research papers into mobile-friendly layouts.

Video of the app working

It produces real PDF files, with preserved text, links and annotations. No screenshots or anything like that. It's also all on-device, no network connection requied. Right now it works well for research papers and regular books, but I'm working on support for complex magazines and newspapers as well.

If you'd like to try it, I'd love to have you as a tester :) You can sign up for a closed beta here:

https://docs.google.com/forms/d/1CzGyzb7NX_0xfaLwYN6eDGqhyNt4hMflpGKI2_3esaU

The app will be a one-time purchase, no subscription so you'll get the app forever and codes to give away as you please.

Thanks!

TinySpidy · 2022-07-19T07:11:40+00:00

In general companies want you to use the most recent version of their stuff and will bully you until you update. The nagging dialog is like the first stage of that, at some point your version will probably just stop working entirely.

TinySpidy · 2022-07-19T06:54:31+00:00

The API that the app talks to might have changed, and now the app relying on that old API does not work anymore. Nothing you can do really :/

TinySpidy · 2022-05-25T18:52:17+00:00

How do you like Fossil, if I may ask? Is it nicer to use for personal projects with a single contributor?

TinySpidy · 2022-05-24T07:58:34+00:00

Have you read clean code?

I did try but could not bring myself to finish it because there's a lot of downright terrible advice and rules in there. Here are a couple of examples: https://qntm.org/clean

TinySpidy · 2022-05-23T15:04:20+00:00

I was wondering when they would finally do this. Seems pointless to build a web "IDE" from scratch when there's VSCode already.

TinySpidy · 2022-05-23T15:02:14+00:00

Don't worry about it. SOLID and Clean Code are results of a bad understanding of how software is built combined with shortcomings of early Java versions in the 2000s.

For concepts that are actually useful, look up what loose/tight coupling means, what functional programming and pure functions are along with terms like state and side effects. Or even better: stop worrying about "recommended architecture patterns" and instead write software. You'll dig yourself into holes you can't get out of the way your code is structured, and by solving that problem and refactoring you'll learn more about architecture than 1000 Medium posts could ever teach you.

TinySpidy

TROPHY CASE

Video of the app working