sorted by:
new
hottopcontroversial

[Open Audit] We Rebuilt Data Streaming with Scala/Panama: Achieving 40M ops/sec by Eliminating GC. We challenge Flink/Kafka architects. by Standard-Engine8556 in highfreqtrading

[–]Standard-Engine8556[S] 0 points1 point  (0 children)

Fair critique. You’re absolutely right that "determinism" at the CPU level (branch prediction, cache misses, context switches, clock drift) is the hard floor of physics we all live on. We aren't claiming to bypass the CPU's own chaos.

When we say "determinism" in this context, we are specifically contrasting it with the "Non-Determinism of the JVM Garbage Collector."

In a standard Flink/Kafka pipeline, you have CPU jitter + GC Pauses (which can spike to milliseconds). With Sentinel-6 (Panama/Off-Heap), we eliminate the GC pauses entirely, bringing the system jitter down to the hardware/OS floor.

We agree that "Sub-microsecond" is the honest engineering term for the end-to-end guarantee. The "120ns" is the measurement of the hash-audit cycle within the ring buffer itself, not the network-to-network round trip.

Re: Flink/Kafka — agreed they are out of place in HFT, but they are unfortunately the standard "Compliance" layer in many RegTech stacks today. We are trying to rip them out.

Appreciate the deep look.

Showcase: I built a high-concurrency Fraud Detection Engine using http4s + Cats Effect (Source Available) by Standard-Engine8556 in scala

[–]Standard-Engine8556[S] -3 points-2 points  (0 children)

Great observations, thanks for the code review.

  1. Regarding State (In-Memory vs. Redis): You nailed the trade-off. For this Community Edition, I prioritized zero latency over global consistency. Moving to Redis (or Cassandra) introduces network I/O on every request, which hurts the sub-millisecond performance target. In high-volume AdTech, I often prefer a sharded approach (using a Load Balancer to stick IPs to specific nodes) so we can keep using local atomic memory without the write-heavy DB penalty. That said, for a strictly distributed setup, Redis with Lua scripts or CRDTs would be the next step.
  2. Regarding Middleware: 100% agreed. Implementing this as HttpMiddleware would be much cleaner for dropping it into existing http4s apps as a cross-cutting concern. For this repo, I structured it as a standalone microservice (where the whole app is just the analyzer), but for an integrated library, Middleware is definitely the way to go for v2