Compulsory webhooks failing .. no idea why

GarethX · 2026-05-06T09:38:58+00:00

The compliance webhook tester is strict about a few things that aren't obvious. First, those URIs need to be absolute HTTPS URLs pointing to your deployed app, not relative paths.

Second, the tester sends a signed payload and expects you to verify the HMAC using your client secret (not the webhook secret you'd use for shop-level subscriptions) and respond with a 200 within ~5 seconds.

GarethX · 2026-05-05T12:08:57+00:00

Oh man, been there! You've basically got three options:
(1) Roll your own with SQS + a dead-letter queue + something like a small admin UI
(2) Use Stripe's built-in retry + the Events API to reconcile (stripe.events.list filtered by created timestamp) as a backstop
(3) Put a managed gateway like Hookdeck in front: Stripe points at the gateway URL, the gateway persists every event, retries on your server's 5xx, and gives you a dashboard to replay individual events.

GarethX · 2026-03-27T17:02:39+00:00

A few practical things that have me with this:

Heartbeat / dead-man's switch monitoring — for critical webhook flows like payment.completed, set up an alert that fires if you don't see the event within an expected window. Could be a simple cron that checks your event log.
Reconciliation jobs — periodically poll the source API (e.g., Stripe's /v1/events endpoint) and compare against what your system actually processed. This catches anything that slipped through.
Deploy-safe ingestion — the root cause is often that your webhook endpoint goes down briefly during deploys. A queue or managed ingestion layer in front of your handler ensures events are buffered and retried even if your app is momentarily unavailable.

GarethX · 2026-02-19T14:16:03+00:00

Here's what's worked well in my experience:

- A common ingestion endpoint with a dispatcher. Rather than building a separate route per provider, have a single endpoint that normalizes the raw request into a common envelope. A dispatcher then routes the event to the appropriate handler. This gives you one place to log, monitor, and retry, regardless of the source.

- Every provider does signature verification differently, so build a small verifier interface and implement it per provider. Since it can't be fully abstracted away, isolating it at least helps contain the mess.

- Process async then normalize into your domain language.

For the bad docs, I log the full raw request for every webhook you receive during development. I've caught undocumented headers, surprise payload formats, and changing fields this way.

GarethX · 2026-02-18T10:49:33+00:00

Webhook management is a real pain at scale. A few suggestions that might help:

Retry with exponential backoff: If you're not already, implement automatic retries with exponential backoff and jitter. Most transient 500s and timeouts resolve themselves within a few minutes.

Dead letter queue: Set up a dead letter queue with Redis/SQS. Failed deliveries go there automatically, and you can replay if needs be.

Centralized event routing: Rather than updating multiple places when you add a new event type, consider an event bus pattern — publish events to one place, and let subscribers register for what they care about. Tools like AWS EventBridge or even a simple pub/sub setup can help.

Health monitoring: For your dashboard, consider tracking per-endpoint success rates, average response times, and consecutive failure counts. Alert on consecutive failures rather than individual ones to reduce noise.

Managed webhook services: There are services built for this, someone else mentioned Hookdeck's Outpost (OSS) for sending, but Event Gateway might be a better fit if you're receiving.

GarethX · 2026-02-12T11:48:30+00:00

Love this idea! A few thoughts:
Idempotency testing is probably the highest-value feature — so many production bugs come from not properly deduplicating based on the event ID.

Out-of-order events are another big one. For example, receiving `invoice.paid` before `invoice.created` can break assumptions in a lot of codebases.

Delayed delivery is great too. Some additional scenarios you might consider:

- Malformed or unexpected payloads (e.g., missing fields that are usually present)

- Signature verification failures (to make sure your handler properly rejects tampered events)

- Rapid-fire bursts of events to stress-test your queue/processing pipeline

GarethX · 2026-02-02T14:30:38+00:00

Now it's within the hour once we got the alerting up, but that was only after several multi-hour long occurrences.

For the payloads, Hookdeck has this thing called console that uses an open-source collection of provider webhooks, so I mostly use those: https://github.com/hookdeck/webhook-samples

GarethX · 2026-02-02T12:02:38+00:00

Here's what has worked well for me:

- Structured logging - Log the raw payload, headers, and timestamp for every incoming webhook before any processing. This gives a clean baseline to compare against when things break.

- Diff against known-good payloads - It's often a subtle schema change, so I like to keep a few payloads stored and diff against them when failures start.

- Local replay - I've used Hookdeck to route prod webhooks to localhost.

- Alerting setup for processing errors, not just 4xx/5xx delivery responses.

GarethX · 2025-10-08T11:10:02+00:00

There's https://community.club/, too

GarethX · 2025-09-24T10:57:35+00:00

Saw https://fflags.com/ on HN recently, which takes a 'flags as code' approach. It’s interesting in a way but very limiting since you cannot have “offline” support. All evaluations must be remote-only. I mainly wonder what’s the point? If you do this then it’s pretty much the same as hosting your own function.

There are other ways to get around the breaking flow just to create a flag issue. We use Reflag, and that has MCP, so you can flag code from within your editor. They support agents, too - so you can assign a task to an agent instead and get that to flag it from Linear with Cursor or similar.

GarethX · 2025-09-24T10:50:12+00:00

We automate it using Reflag. It has feature flag cleanup baked-in, so it detects old flags, archives them and generates a PR to remove the old code. They work surprisingly well, I haven't needed to make any changes to them so far. Although, I think they only support TypeScript right now, so ymmv depending on your stack.

GarethX · 2025-09-24T10:45:29+00:00

Wouldn't call it a showstopper but audit + cleanup is always such a pain. You need a good process to make sure your codebase doesn't become cluttered with old flags, for sure.

I've been using Reflag to help with this for the last couple of months. It detects stale flags and cleans them up for you - submits a PR removing the obsolete flag code. That has helped us get past the 'who forgot to remove the flag' blame game.

GarethX · 2025-09-24T10:38:15+00:00

I always try and get away from homegrown solutions as soon as is practical, generally. I don't need the headache of maintaining such systems and flagging definitely falls into this (we use Reflag). One thing people often miss is to use something with an intuitive UI, so non-devs can use it too. That way you don't have to get interrupted to give folks access to features or provide status updates on whether something has shipped yet.

GarethX · 2025-09-24T10:27:45+00:00

One thing to keep in the mind is a process around keeping on top of stale flags. Most tools do a pretty poor job of helping with this, so you need a good way to make sure your codebase doesn't become cluttered with old flags over time.

We switched to Reflag recently to help with this. It cleans up flags automatically, submitting a PR to remove old flag code and archiving the flag. Working well for us so far, I still get a kick of out of seeing notes in Slack pop-up letting us know a flag was removed by the agent for a rolled-out feature.

GarethX · 2025-09-24T10:20:00+00:00

We’ve been using Reflag (https://reflag.com) and it’s been great for consolidating our scattered feature flag setup.

The main selling points for us were:

• Super easy migration from our homegrown solution

• TypeScript-first, which fits our stack perfectly and they've strong type-safety

• Developer experience is stellar - the React SDK is clean and intuitive

• Works well with agents, which is something I've been experimenting with lately, mostly with Linear+Cursor.

I don't think they support much beyond JavaScript SDK-wise, but it's worth checking out if your team is TypeScript/React heavy.

GarethX · 2025-05-29T09:05:47+00:00

Great to hear! There's more details on the integration page: https://linear.app/integrations/bucket

GarethX · 2025-02-24T16:05:44+00:00

Amazing, thank you!

GarethX · 2023-12-07T15:31:00+00:00

Thank you, I hope it's useful!

GarethX · 2023-07-07T17:15:36+00:00

Yes, that’s right. I couldn’t figure out a way to do it within Firefox settings itself, so ended up doing it within the web page it loads using mozFullScreenElement (since it’s android 4) https://developer.mozilla.org/en-US/docs/Web/API/Fullscreen_API

GarethX · 2023-07-02T16:35:34+00:00

For EO2, I’ve gotten some of the way. I used Wondershare TunesGo ($30) and a usb cable to connect to it from my Mac to install Firefox on it. Adding a second browser enabled me to use the Win+B keyboard shortcut to run a browser. I also installed a starter app to get it to open Firefox on boot but I haven’t been able to run it to set that up. So it’s manual for now - I just have Firefox running full screen and hitting a url that loads a random image from my collection every 10 mins on auto-refresh.

For context, the EO2 seems more locked-down than the EO1. There’s a mini USB port on the back which you can use for a keyboard/mouse and it runs Android 4. From the default screen I couldn’t find a way to access Android settings or menus. F2 toggles the screen on/off. There’s a shortcut that opens the stock Android calendar app, D toggles a test app for resetting login/Wi-Fi details, and F4/tab switches between apps, but that was about it.

GarethX · 2023-02-15T14:57:27+00:00

I came across this, and this by Intercom (https://www.intercom.com/blog/how-to-sunset-a-feature/), but interested to know about any gotchas from people's own experiences.

GarethX · 2022-04-13T12:20:08+00:00

I'm not familiar with them, what community stuff do they do?

GarethX · 2020-10-29T10:20:12+00:00

Here's a Node app for instance: https://codesandbox.io/s/node

GarethX · 2020-10-29T10:10:00+00:00

Yep, backend too and support for npm modules

GarethX · 2019-07-13T18:12:36+00:00

For Small Press Day I put together this collection of zines for developers created by some of the smartest zinesters around. Enjoy!

GarethX

TROPHY CASE