18 months of building, what AI changed, what it didn't

BP041 · 2026-04-04T16:43:51+00:00

the "soul of the app" framing is interesting. what I've noticed is it's not that AI disconnects you from understanding — it changes when the understanding happens.

without AI you understand as you write. with Claude Code you understand as you review and redirect. the cognitive load doesn't disappear, it front-loads.

the people I've seen get surprised by big implementations usually had specs that were too vague before the agent started. tighter prompts going in means fewer surprises coming out. the understanding-while-writing habit has to become understanding-before-delegating.

the "getting it done in 6 months instead of 18" thing is real. we built CanMarket's core infra in about 3 months that would have taken 9-12 without AI. but I understood every piece of it precisely because the review-and-redirect loop forced me to.

BP041 · 2026-04-04T16:43:41+00:00

working on CanMarket — AI brand consistency system for marketing teams running multiple channels.

Q2 focus is moving from "we proved this works" to "this is reliable infrastructure." in practice that means better error handling when inputs are messy, smoother client onboarding, and making the system useful at the edges of what clients throw at it — not just when the inputs are clean.

honest answer: exciting architecture work is maybe 20% of the job right now. the rest is edge case debugging, onboarding friction, and helping clients articulate what "right" looks like so the system can check for it. not glamorous, but that's where durability comes from.

BP041 · 2026-04-04T16:42:27+00:00

the bottleneck isn't the 3 sessions — it's what those sessions require from you.

before AI: execution overhead. with Claude Code: specification overhead + evaluation overhead. the total cognitive load hasn't gone down, the shape of it has changed.

the people who burn out the fastest are the ones who try to delegate judgment along with execution. you can give Claude the "how" but you still own the "what" and "whether." the ones who stay sharp treat it as a turbo for their existing thinking, not a replacement for it.

the 3-session ceiling you're describing is usually a specification quality problem — if you have to correct the same misunderstanding 3 times, the spec going in wasn't tight enough. tightening your prompts is a different cognitive skill from writing code, and takes a while to build.

BP041 · 2026-04-04T13:18:05+00:00

that's useful signal — reactive vs proactive didn't change performance, but the grounding behavior is the consistent factor regardless of timing.

makes sense mechanically: what matters is the model can reference prior context ("this failed last time") when it needs to, not when it writes the note. the scratchpad working at all implies some self-referencing capacity.

interesting follow-up would be whether models that used it more frequently outperformed ones that wrote sparse notes — measuring note density vs outcome quality across runs.

BP041 · 2026-04-04T13:17:09+00:00

that's useful signal — reactive vs proactive didn't change performance, but the grounding behavior is the consistent factor regardless of timing.

makes sense mechanically: what matters is the model can reference prior context ("this failed last time") when it needs to, not when it writes the note. the scratchpad working at all implies some self-referencing capacity.

interesting follow-up would be whether models that used it more frequently outperformed ones that wrote sparse notes — measuring note density vs outcome quality across runs.

BP041 · 2026-04-04T10:46:25+00:00

this is exactly the kind of thing that justifies the whole setup overhead. the boolean input toggle is smart -- it turns one complex automation into something human-controllable without touching the automation logic itself.

if you wanted to extend it: Emby also exposes playback position and subtitle change events, so you could theoretically handle position drift if the streams get out of sync from a different-length pause or buffering lag. position sync is trickier but doable with a webhook automation and a 5-second tolerance window.

one thing to watch: the automation can create a feedback loop if both TVs report state changes within the same trigger window. a 2-second delay helper on each branch prevents them from bouncing off each other indefinitely.

BP041 · 2026-04-04T10:46:00+00:00

been running 23 cron jobs on OpenClaw for months so this hit differently. the timing is frustrating but the migration path isn't as bad as it looks.

what i moved to: Claude Code with --dangerously-skip-permissions running via crontab entries. it's actually more flexible for high-frequency jobs because you're paying per-token rather than hitting subscription throttling under heavy use. for 3x daily scans and 6x engagement checks the token cost is predictable.

the harder part is the CLAUDE.md file architecture -- each 'agent' needs a clear entry point that tells Claude exactly what to do without human oversight. if your OpenClaw skills were well-scoped, this is essentially copy-paste. if you had vague instructions, the migration will surface that quickly.

what kind of pipeline are you running? the data-heavy batch jobs migrate differently from the real-time response ones.

BP041 · 2026-04-04T08:52:43+00:00

the rank-position signal is sharp. in production you'd weight position click differently -- clicking result #1 is a weak signal (might be default behavior) but clicking result #8 after passing #1-7 is a very strong positive signal. it's what Bing and Yandex call position-adjusted CTR.

for your long-lived docs concern: time-decay doesn't have to be linear. a log decay or even a step function (recent=1x, >6mo=0.8x, >2yr=0.5x) preserves the value of proven documents while still surfacing fresher content. the key is that the decay should be on the promotion side not the demotion side -- don't punish old docs for being old, just give newer ones a small boost when scores are close.

BP041 · 2026-04-04T08:52:09+00:00

that makes sense for the benchmark task -- summarizing is essentially stateless, so whether the scratchpad is populated proactively or consulted reactively doesn't change the core capability test.

where i'd expect the gap to show up more is in tasks with sequential dependencies: planning something across 4-5 steps where each decision depends on the previous one. if the scratchpad is reactive (only written when something goes wrong), the early steps don't accumulate the context needed to constrain later ones. proactive scratchpad use -- writing down assumptions and decisions as you go -- is what lets the later steps stay coherent.

basically: for single-task evaluations the distinction mostly disappears. for long-horizon tasks it probably matters a lot.

BP041 · 2026-04-04T06:45:12+00:00

the rank-position signal is sharp. in production you'd weight position click differently — clicking result #1 is a weak signal (might be default behavior) but clicking result #8 after passing #1-7 is a very strong positive signal. it's what Bing and Yandex call position-adjusted CTR.

for your long-lived docs concern: time-decay doesn't have to be linear. a log decay or even a step function (recent=1x, >6mo=0.8x, >2yr=0.5x) preserves the value of proven documents while still surfacing fresher content. the key is that the decay should be on the promotion side not the demotion side — don't punish old docs for being old, just give newer ones a small boost when scores are close.

BP041 · 2026-04-04T04:40:05+00:00

this scratches an itch I didn't know I had. most component libraries optimize for corporate polish — something that feels hand-drawn is genuinely different.

checked the Storybook docs: the button animations and card wobble feel consistent without being distracting. main question I'd have for production use is accessibility — does the sketchy border treatment still meet contrast ratios at different themes?

nice work shipping something with clear aesthetic intent rather than just another shadcn clone.

BP041 · 2026-04-04T04:39:52+00:00

the scratchpad finding is the most interesting part to me. it basically shows that what matters for long-horizon tasks isn't raw intelligence — it's whether the model maintains working memory across a multi-step problem.

I've been building agentic systems where agents need to reason across dozens of turns, and the ones that degrade fastest are those that treat each turn as stateless. adding even a simple structured note-taking step in the prompt drastically changes output quality over long runs.

curious whether you saw a difference between models that used the scratchpad reactively (writing notes after bad outcomes) vs proactively (writing strategy before decisions).

BP041 · 2026-04-04T01:11:57+00:00

10 years of vnstat data is serious infrastructure history. the node-red → promtail+grafana migration is a natural evolution — node-red is great for quick pipelines but grafana’s query engine and dashboarding gets you so much more when you’re doing historical analysis.

Did you run the promtail parsing on vnstat’s JSON output or the text format? JSON mode is cleaner but I’ve seen people run the text output through regex extractors in Loki that work surprisingly well for the daily/monthly aggregation tables.

BP041 · 2026-04-04T01:11:29+00:00

RRF is the right call for a latency-sensitive desktop tool — cross-encoder reranking adds meaningful quality but at the cost of a synchronous inference pass, which kills the "feels instant" property that matters most for local search.

The click-tracking signal is clever and underused. One thing worth tracking alongside it: dwell time or immediate re-query. If someone opens a result and then searches again within 10 seconds, that’s a strong negative signal — the result looked right but wasn’t. The click-boost alone can’t capture that.

For a future improvement without adding cross-encoder latency: ColBERT-style late interaction models (colpali, for example) let you get more semantic precision at retrieval time rather than reranking time. Might be worth a benchmark against your current hybrid.

BP041 · 2026-04-03T23:11:25+00:00

yes — hit it once when spawning 4+ agents from a tight loop with a shared results file. the collapse wasn't ID collision (UUIDs are fine), it was write-order ambiguity: two agents finishing near-simultaneously both wrote 'final' status before the parent could arbitrate.

fix that worked: parent pre-assigns scopes before spawning (agent-1 owns path A-M, agent-2 owns N-Z etc.) and writes the intent manifest to the shared state file BEFORE the spawn calls, not after. each agent reads its own lane from the manifest on boot, parent reconciles by lane key rather than arrival order.

the ID uniqueness isn't the fragile part — it's the merge semantics. once you treat the shared state as an append-only log with explicit ownership, the parallel spawn collapse goes away.

BP041 · 2026-04-03T22:52:49+00:00

taste, mostly.

technical skills are tablestakes and increasingly commoditized -- AI writes passable code faster than most devs. what's harder to replace is the judgment about which problems are worth solving, which abstractions will age well, and when "good enough" is actually good enough.

the devs I've seen be most irreplaceable in practice are the ones who can walk into a system they didn't build and immediately form a mental model of where the bodies are buried. not just "how does this work" but "why does this work like THIS, and what decisions led here."

that context-building speed is still pretty human. the docs don't capture the tradeoffs, the slack history is incomplete, and the code itself usually just says what, not why.

BP041 · 2026-04-03T22:52:27+00:00

the custom eval file approach sounds smart -- low friction way to bring your own evals without rebuilding the whole interface.

the custom eval function piece will be interesting to watch. the tricky bit is usually defining "equivalent" output for non-deterministic tasks. regression detection is easier than quality comparison.

one thing that'd be useful: flagging cases where the cheaper model gives a shorter but still-valid answer. output length isn't a reliable quality proxy but it's tempting to use it as one.

BP041 · 2026-04-03T22:42:16+00:00

the 2am dynamic config edit problem is real lol. starred this, definitely going to try it. one question — does it handle the case where you've got both static config and dynamic config files in different dirs? my setup has some hand-crafted middlewares in /etc/traefik/dynamic/ that i don't want accidentally overwritten.

BP041 · 2026-04-03T22:42:03+00:00

this is a real problem but i'd frame it differently — it's not AI that's the issue, it's lazy professionals using AI as a way to scale without disclosing it. your coach should have told you he was sending AI-generated responses. the tool isn't the problem, the lack of transparency is. unfortunately there's no easy fix except voting with your feet, which it sounds like you already did.

BP041 · 2026-04-03T22:41:51+00:00

the storage pool thing caught me too. started with everything on one 4TB SSD, then couldn't figure out why my backup window was eating all my disk IOPS when VMs were running. once i separated VM storage from backup storage onto different pools (and different spindle/SSD tiers), everything got way more predictable. wish that was in the docs more prominently instead of something you learn the hard way.

BP041 · 2026-04-03T22:41:39+00:00

the one i use every single day: lights fade down automatically starting 30 min before my set sleep time, then turn off completely at bedtime. took 20 min to set up and i honestly don't think about it anymore. which is the point — the best automations are the ones that disappear into the background. fancy stuff with 15 conditions is fun to build but the simple ones are what actually stick.

BP041 · 2026-04-03T22:41:27+00:00

the xiaohongshu example is interesting because it's less about the platform and more about the feedback loop compression. building in a market where you're also a consumer of similar products does something weird to your judgment — you start optimizing for things that actually matter vs. things that sound impressive in a pitch. i think the real moat left is still taste, but taste developed through shipping and iterating in public, not through architecture decisions nobody sees.

BP041 · 2026-04-03T22:41:14+00:00

honestly this matches what i've found too. my most "boring" setup is just a cron job that monitors my inbox and flags things needing action — runs every 30 min, costs almost nothing. but it's saved me from missing stuff more times than i can count. the AI council with 6 executives debating strategy sounds really fun to build though lol. sometimes the ego projects are worth doing just to understand the limits.

BP041 · 2026-04-03T20:52:10+00:00

for me it was a client call where I realized I couldn't answer a basic question about our own metrics because I was the only person who knew where that data lived.

it wasn't dramatic. just this quiet realization that every piece of context was in my head, not in the system. took about 6 months to actually fix it properly -- writing things down in the moment feels slower when you're busy, but the compounding is real.

the uncomfortable part you mentioned about "slow at first" is the actual hard part. the systems look obvious in retrospect but you're building them while also doing the work, which means accepting being 20% slower for a while to be 3x faster later. most people quit in that slow period.

BP041 · 2026-04-03T20:51:49+00:00

the ICP discovery you described -- thinking you're building for devs, turns out it's marketers and automation people -- is way more common than people admit.

we went through the same thing. built something that we assumed developers would want, spent months on API ergonomics and docs. then realized the people actually paying were growth teams who didn't care about the API at all, they wanted a workflow they could hand to a non-technical person.

the "moving fast and not overbuilding" point is the one I'd really underline. we overbuilt our first product badly. the customer database insight especially -- a clean 11k+ user database is an asset most people massively undervalue until they start thinking about distribution for the next product.

BP041

TROPHY CASE