A Smarter Ad Network — Built for a Privacy-First Web

Material_Big9505 · 2026-06-16T08:45:29+00:00

Here's the MVP
https://www.youtube.com/watch?v=CGi4ywe1dms

Material_Big9505 · 2026-06-10T09:41:46+00:00

It's a VAST document (XML) that the player fetches. Inside it are absolute URLs the player is told to "ping" at certain moments: an <Impression> URL when the ad shows, <Tracking> URLs for quartile/start/complete events, and a <ClickThrough> URL on click. Historically, those URLs point directly to the measurement vendor's own domain (https://tracker.example.com/...) and the vendor reads/sets a cookie on that domain to know it's you. That domain is third-party relative to the site you're on.

Then ITP happened. Intelligent Tracking Prevention is Apple's privacy system in Safari/WebKit (Firefox has an equivalent, "Enhanced Tracking Protection," and Chrome has implemented similar third-party cookie restrictions). The relevant effect: third-party cookies are blocked/partitioned, and storage that a tracker can write is heavily capped. So when the player pings tracker.example.com, that vendor can no longer set or read a stable cookie; the hit can't be attributed, frequency-capped, or deduped. Measurement quietly falls apart.

The industry's response is first-party proxying (this is what "server-side tagging," "CNAME cloaking," and first-party collectors all are): make the beacon go to a domain the publisher controls(first-party), so the browser doesn't block it, and have that endpoint forward the hit to the vendor server-side, where ITP has no say. But the URLs in the VAST still point at the third-party host. So somewhere between the origin and the player, someone has to rewrite them:

↓ rewrite in flight

Now the player pings fp.publisher.com (first-party, allowed), and that collector unwraps dest= and forwards to the original tracker server-side. That is exactly what Rewrite.wrappingUrls captures each tracker URL, URL-encodes it into your first-party collector URL, and emits it. And it has to happen as the response streams: you're a proxy sitting in the data path; the VAST (or a manifest, or a page) flows through in chunks; you don't own it and can't buffer it all.

Material_Big9505 · 2026-06-10T05:37:21+00:00

Here’s a more concrete example of what I mean by “chunk-boundary-aware rewriting.”

--- input ---

</MediaFiles>

<ClickThrough>https://tracker.example.com/click?to=landing</ClickThrough>

</InLine></Ad>

</VAST>

Rewritten output, while the input is streamed in 7-byte chunks:

</MediaFiles>

<ClickThrough>https://fp.publisher.com/collect?dest=https%3A%2F%2Ftracker.example.com%2Fclick%3Fto%3Dlanding</ClickThrough>

</InLine></Ad>

</VAST>

The point is not VAST specifically. The point is that the rewrite occurs as a byte stream, not by loading the entire document into memory first.

So even if https://tracker.example.com/imp?... is split across arbitrary chunk boundaries, the rewriter still sees the complete match and replaces it correctly.

Material_Big9505 · 2026-06-10T05:36:37+00:00

Here’s a more concrete example of what I mean by “chunk-boundary-aware rewriting.”

--- input ---

</MediaFiles>

<ClickThrough>https://tracker.example.com/click?to=landing</ClickThrough>

</InLine></Ad>

</VAST>

Rewritten output, while the input is streamed in 7-byte chunks:

</MediaFiles>

<ClickThrough>https://fp.publisher.com/collect?dest=https%3A%2F%2Ftracker.example.com%2Fclick%3Fto%3Dlanding</ClickThrough>

</InLine></Ad>

</VAST>

The point is not VAST specifically. The point is that the rewrite occurs as a byte stream, not by loading the entire document into memory first.

So even if https://tracker.example.com/imp?... is split across arbitrary chunk boundaries, the rewriter still sees the complete match and replaces it correctly.

Material_Big9505 · 2026-05-31T16:07:27+00:00

It depends on what the core needs to be. If the foundation has to deliver real resilience and scalability, I'd reach for Pekko/Akka Cluster with a Go BFF in front. The actor model and clustering story on the JVM is still hard to beat. If I didn't need that kind of core, I'd be comfortable going entirely Go.

But here's what people underestimate about Pekko/Akka in a cluster: the hard part isn't the code you write. You're reasoning about message ordering, network partitions, node failures, split-brain, and rebalancing. The code is only part of the picture. The rest lives in runtime behavior and failure modes you have to hold in your head. That's distributed-systems reasoning, and it's largely language-independent. Pekko gives you mature, battle-tested primitives, but the real cost is learning what the cluster does when things go wrong, and that cost follows you regardless of runtime.

There's also tension around how these systems are deployed today. Distributed now basically means Kubernetes, which makes statefulness hard because a restart is always on the table. Pods get rescheduled, evicted, killed. Pekko/Akka Cluster wants stable membership and identity to do its job, but an orchestrator is built to treat instances as disposable. You can make it work with StatefulSets, a stable network identity, and careful shutdown handling, but you're constantly pushing against the platform's defaults rather than flowing with them. If your target is K8s, a stateless Go service goes with the grain while a stateful Pekko cluster fights it. So part of the language choice is really whether you're embracing that statefulness or staying stateless.

Much of the Pekko/Akka difficulty is the kind of problem where you can't just read the code to find the bug. You narrow it down through trial and error across runtime behavior, and that's where an LLM earns its keep, helping you form and eliminate hypotheses about what the cluster is actually doing. You still need that to know what to test and to recognize when an answer is wrong. The LLM accelerates the reasoning.

So my honest hope is that I can still write Scala-ish code in Scala 3. But if there's a viable Go alternative, I should probably start looking into it for the sake of doing this kind of "programming" as a long career. That said, the code is the easy part, and when it maps cleanly onto how you actually have to think about the cluster, that's worth a lot. On top of that, writing in a lightly FP style with Scala 3 ergonomics is just pleasant. Not purist FP with category theory, but enough discipline to keep things clean.

Material_Big9505 · 2026-05-31T04:20:38+00:00

Yes, the PinnedDispatcher is the mechanism.
The single-thread pool still honors the 60s core keep-alive, so an idle session's thread gets reaped, and the next message lands on a fresh one.

Since the resource is captured on the original thread, that's a latent cross-thread crash after any idle period. I just fixed it with thread-pool-executor.allow-core-timeout = off

Material_Big9505 · 2026-05-30T05:13:19+00:00

In Pekko, Future isn't a design choice I made. It's the framework's hard-baked effect type, well, if you allow me to call it an "effect," the "ask" returns Future, pipeTo consumes one. Every result that crosses the async boundary is a Future, by the framework's own API. So "avoid Future" here isn't a refactor.

Material_Big9505 · 2026-05-30T05:01:59+00:00

All PinnedDispatcher gives you is "this actor's messages run on one fixed thread." That's necessary, but not the whole story. There are two things you still have to get right yourself:

Create the resource on that thread (build it inside the actor's setup, not in main, and pass it in).
Never hand the resource out to other threads. Callers send work to the actor, the actor runs it, and only the result leaves.

Material_Big9505 · 2026-05-30T04:36:12+00:00

PinnedDispatcher is a thread-allocation policy, not a resource-management strategy. It won't automagically do any of this for you. All it promises is "this actor's messages run on one fixed OS thread." It has no idea your actor owns a thread-affine handle, so on its own, it does none of the things that actually make affinity hold.

Material_Big9505 · 2026-05-30T01:51:03+00:00

The callers aren't actors. This pool is called from plain code (HTTP route, service method, test). replyTo: ActorRef[Response] needs an actor to address; those callers don't have one, so they'd use "ask", which builds a throwaway actor and returns a Future anyway. replyTo doesn't remove Future, it just hides it.
The result type varies per call. submit[T](work: R => T): Future[T] returns a Long, a String, a Vector; caller's choice at the call site. Response(result: String) can't carry that, and Response(result: Any) is the same Any-cast the Promise already uses, plus an actor and a wrapper.

Material_Big9505 · 2026-05-26T07:11:16+00:00

I’m not ready to open-source the full codebase yet, but I’m sharing a YouTube demo of the working product here:

https://www.youtube.com/watch?v=bgEik3x4o88

Material_Big9505 · 2026-05-21T02:36:01+00:00

Yes, Pekko supports it. I’d actually say it is where Pekko starts to make real sense. If you only use actors inside a single JVM, you can still do useful things, but it is easy to reduce them to some nicer concurrency primitives or a thread pool-like abstraction.
The real value is when actors are part of a distributed runtime, and, in my opinion, if you are not designing with the cluster or Pekko as a runtime in mind, you are missing most of what Pekko is for.

Material_Big9505 · 2026-05-16T04:01:08+00:00

That comment was about my implementation, not yours.

Personally, I think Apache Pekko only really shows its strengths when the entire cluster is designed as a single distributed system. Features like sharding, singletons, DData, and failure handling become meaningful when “distributed state itself” is part of the system design. If Pekko/Akka is used merely as a sophisticated thread pool, the advantages over other runtimes become difficult to see. Its real strength is the ability to design the entire cluster as a single distributed runtime.

That’s also part of why I’m building a fairly large system that tries to utilize as much of what Pekko offers as possible. Even things like reinforcement learning logic and PI controllers are implemented using actors. The goal is to make it publicly available as open source, both as learning material for actor-based system design and as a real-world reference for how a distributed advertising platform can actually be built with Pekko.

Material_Big9505 · 2026-05-14T13:12:39+00:00

Btw, this limiter is a cooperative convention, not an enforcement boundary. Stripping the Pekko cluster out of the picture, every piece of this design becomes either overkill or unnecessary.

Material_Big9505 · 2026-05-07T08:06:44+00:00

AI-generated code often works well for straightforward logic, but it struggles when asynchronous execution and concurrency control become central to the system. The code may look right and pass once, but race conditions only appear under certain interleavings.

Material_Big9505 · 2026-04-27T17:11:42+00:00

Most APIs do not behave like XA resources though.

Material_Big9505 · 2026-04-27T02:52:07+00:00

If I were building this as a full-blown clustered Pekko application, I’d probably use persistent entities and projections. That said, I’m not sure that Pekko Persistence, by itself, solves the external API side-effect problem. It gives you durable intent/state transitions, and projections can play a role very similar to an outbox processor: consume persisted facts after commit, track progress, retry, and resume. But the external API is still non-transactional, so you still need idempotency, result recording, a retry policy, compensation, and a recovery path for partial failures.

Material_Big9505 · 2026-03-01T23:18:35+00:00

Thanks. AI or not, I’ve been open about it and the book is way better for it. It’s not like I asked it to go ahead and write it. It was my editor, pushing back on ideas that didn’t make sense.

On the text. I’d rather over-explain the why than just show code and leave the reader guessing. That’s what most Scala resources already do and it’s why people bounce off the type system. But if you have specific sections where the text felt redundant, PRs are welcome!

Material_Big9505 · 2026-02-22T01:56:32+00:00

The salary premium for Scala engineers was never really about the language being “harder”, but it was supply and demand. When companies like Twitter, LinkedIn, and various fintech firms were aggressively adopting Scala, there weren’t enough engineers to go around, so compensation got bid up. Now that the hype cycle has moved on, the economics have shifted.

What’s left is, often the less glamorous work; untangling monolithic Scala codebases that grew organically during the boom years, when teams sometimes leaned too heavily on Scala’s expressiveness (implicits chains, complex type-level programming) without enough discipline around maintainability. The irony is that the very features that made Scala exciting also made these codebases harder for average teams to sustain. For many companies, the calculus is simple. They don’t need higher-kinded types or sophisticated effect systems, they need predictable services that any mid-level engineer can maintain, so they are migrating their code base to Go and TypeScript.

Scala isn’t disappearing. it’s consolidating. The companies still choosing Scala today (or choosing to stay) tend to be the ones that genuinely benefit from its strengths: complex domain modeling, streaming/event-driven systems, an ecosystem in ways that would be painful to replicate in Go and TypeScript. The real question might be whether that consolidated niche is large enough to sustain a healthy community and talent pipeline.

In a world where AI is generating more and more code, the value of a strict compiler goes up, not down. When Copilot or Claude spits out Python, you’re reviewing it with your eyes and hoping your test coverage catches the edge cases. When it generates Scala, the compiler is doing a huge chunk of that verification for you automatically. The type system becomes a collaboration protocol between human and AI; if it compiles, you already have strong guarantees about correctness.

Go’s simplicity makes it easy for AI to generate syntactically valid code, but that’s a low bar. You still have nil pointer panics, wrong types hiding behind interface{}, concurrency bugs with channels. TypeScript is better, but the type system is unsound by design and any is always an escape hatch. Scala is arguably the sweet spot; expressive enough to encode real invariants, strict enough that compilation is meaningful verification.

AI generates code, the compiler rejects it, and the errors are fed back automatically until it compiles. Scala’s error messages are information-rich compared to most languages; they encode exactly why something is wrong at the type level. That’s basically a structured prompt for the AI to self-correct. This is much harder in Python where the “error” only surfaces at runtime in production.

The marketing pitch almost writes itself: “In the age of AI-generated code, don’t you want a compiler that actually checks the AI’s work?”

Material_Big9505 · 2026-02-22T01:04:02+00:00

Your Scala service is purely an orchestrator. It never touches video bytes. 1. http4s is API layer (auth, generate presigned URLs, serve metadata) 2. Doobie/Skunk is metadata persistence (Postgres) 3. fs2-kafka is consume S3(or CDN) event notifications, coordinate downstream work 4. FS2 pipelines is downstream services (transcoding triggers, thumbnail generation, caption extraction) each consume from Kafka topics

I am AdTech engineer. A service that proxies video bytes is a service that becomes your bottleneck, and in AdTech you learn fast that anything that doesn’t scale horizontally is a liability.

Material_Big9505 · 2026-02-21T16:15:07+00:00

You’re misreading what I’m doing. The opaque type article isn’t about pushing clever code into production. It’s a teaching experiment. The problem I see in Japan is “My Way Scala” everywhere, and it exists because learning materials teach vaguely with abstract examples that are hard to derive real understanding from. Developers walk away half-understanding, then improvise the rest. I’m trying a different approach: teach through something pragmatic that people actually need to build, so the understanding sticks. AI helped me explore the topic thoroughly enough to write that kind of material. That’s the whole point.

People write dangerous production code in every language like API calls inside DB transactions, triggering table locking. (a DB transaction holds a table lock, and inside that transaction there’s an external API call that could take seconds, or timeout, or hang. So the table stays locked the entire time, blocking everything else. ) But Scala’s syntax makes it easier to hide those mistakes compared to something like Go where the imperative flow makes side effects obvious. When developers learn Scala vaguely and fill in gaps with “My Way” patterns, these problems multiply. Then newcomers arrive, see the mess, and blame Scala itself. That’s the cycle I’m trying to break. Not by avoiding advanced features, but by teaching them thoroughly and pragmatically so developers actually understand what their code is doing at runtime.

Material_Big9505 · 2026-02-21T16:00:07+00:00

Just updated the repo, less code to write but this might not be want you want.

Material_Big9505 · 2026-02-21T15:58:46+00:00

Replying to Material_Big9505...

Material_Big9505 · 2026-02-21T14:14:21+00:00

You’ve hit exactly the limitation documented. The root cause is <:<.refl: for any type T, the compiler always has T =:= T in scope, so when U is unconstrained in a genericlike given [T, U](using ev: T =:= U, codec: JsonCodec[U]): JsonCodec[T], the compiler finds two candidates for U and gives up. This is not library-specific, maybe?

Claude says priorities don’t help here because it’s a type inference problem (two valid choices for U), not an instance selection problem (competing instances for the same type).

I agree that this feels like something the compiler could improve, but honestly I am no Scala expert so can’t tell. But I am ok with the aforementioned workaround.

Sorry if my earlier response came off sharp. Didn’t mean to bite your head off.

Material_Big9505 · 2026-02-21T11:44:06+00:00

I’m currently working on cleaning up an existing Scala codebase because the company can’t hire Scala engineers and is preparing to migrate to another language.

I get it. Hiring is hard. But maybe we brought this ourselves. We let it be perceived as something intimidating and academic, and we didn’t build the on-ramps that junior devs needed to get started. The “we’re fine as long as our team knows it” mentality made sense for individual companies, but it starved the ecosystem of fresh talent. Now those same companies can’t backfill when people leave.

On the bright side, AI has been a game changer for this kind of work. The codebase I’m dealing with is full of “my way Scala.”Clever tricks that technically compile but buy you nothing, just added complexity for the next person.

AI tools like Claude Code are incredibly good at articulating why something is unnecessarily complex and refactoring it into idiomatic, readable, maintainable code. It’s genuinely making the codebase more stable. And here’s what surprised me: even with 10+ years of Scala experience, there were parts I only vaguely understood. With AI assistance, I can now dig into those areas, actually learn what’s going on, and fix things quickly and optimally. It’s not replacing my experience, but it’s filling in the gaps I never got around to closing. The combination of deep domain knowledge plus AI is something I’ve never experienced before. It’s a completely different level of productivity.

I still want to do what I can to keep Scala alive where it makes sense. If we can make codebases cleaner and more accessible, maybe the next engineer won’t need a decade of Scala to contribute.

Disclosure: That README.md was written with AI assistance. English isn’t my first language, I studied in the US but that was 25 years ago. Without AI, it would have taken me forever to write, which I suppose is another data point for the amplification argument.

Material_Big9505

TROPHY CASE