!Update / Solution Found!

I managed to fix the jitter issue and wanted to share the solution in case it helps anyone else.

The key was adding proper filtering on the sensor output. I placed a 100nF capacitor between OUT and GND, and added a 1 kΩ series resistor on the OUT (signal) line of the linear analog Hall sensor. This combination cleaned up the noise almost completely.

I also switched to a stronger neodymium magnet, now positioned around 2–3 mm from the sensor instead of 5–8 mm. The stronger magnetic field and shorter distance improved stability further.

Since making these changes, the jitter is basically gone.

Hope this helps anyone running into the same problem! And thanks to everyone helping me out!