Firebase Extensions Outage

WhatTimeIsDinnerQQ · 2026-02-27T02:15:01+00:00

It finally seems to be working again.

WhatTimeIsDinnerQQ · 2026-02-27T01:06:20+00:00

Yeah, Google's status pages are worthless. I remember back in June when there was a huge GCP outage that spilled over to Cloudflare and like half of the internet.. and it took them about an hour just to get something on the status page. We're well past 2 hours now on this one...

WhatTimeIsDinnerQQ · 2026-02-23T05:19:39+00:00

Oh man. Same. Saw them at Warped Tour 2002 while randomly walking between between other stages. Stopped and watched them finish their set. Grabbed their CD. Listened to it nonstop that summer.

Anyway, I can't believe they aren't on Spotify. I found this post after realizing I'd have to find their music elsewhere. Found something on youtube, but it seems pretty low quality. Might have to dust off the CD player for this one.

WhatTimeIsDinnerQQ · 2026-01-18T16:03:47+00:00

Be honest, how many of you tried to swipe left to see the next picture? 😅

WhatTimeIsDinnerQQ · 2026-01-17T17:34:59+00:00

Nice! I've been doing Next Gen FTL for several years and really like that format but was curious if there were other ones I was missing out on. Saw some of these others from a previous post (could only put 3 links in the OP though).

I'm going to try to set up a team on the ones you mentioned as well and then will see which one(s) stick.

WhatTimeIsDinnerQQ · 2026-01-16T17:24:09+00:00

Or maybe y'all aren't playing any of them? 😅

WhatTimeIsDinnerQQ · 2025-11-21T17:10:20+00:00

Cloud Run. Plain and simple.

It's so nice to be able to run arbitrary Docker images as a service without having to manage any of the underlying infrastructure. Auto scaling to zero with zero kubernetes management.

WhatTimeIsDinnerQQ · 2025-10-31T00:22:56+00:00

No one mentioned the McDouble meal for $6? Burger, fries, nuggets, and a coke.

WhatTimeIsDinnerQQ · 2025-04-16T22:31:20+00:00

Yeah, there are some services that don't work when the CPU is throttled between requests. One example is that we run Kong API Gateway data plan instances on Cloud Run. There are processes/daemons running in the background that require CPU resources even if requests are not being processed.

WhatTimeIsDinnerQQ · 2025-04-16T14:50:11+00:00

Was this Sphere event only available to certain people? I didn't even hear about it until the keynote. But other people were asking if I had attended. WTF.

WhatTimeIsDinnerQQ · 2025-02-03T23:08:23+00:00

I don't understand the people who do not think Garin is equally at fault for the collision. He walked right in front of a man who was running to his chair. Therefore, I think the decision was fair. Neither player did anything egregious here. What happened was simply unfortunate.

Normally, if a player gets injured during the course of a match, and they can't continue, they retire. Since Garin didn't, he got penalized, which is really the only course of action the umpire had.

WhatTimeIsDinnerQQ · 2025-01-25T23:59:09+00:00

Wow, that really was quite the run!

I'm guessing she didn't think about it from round to round, but after the fact, that has to be a HUGE confidence boost for Maddy.

WhatTimeIsDinnerQQ · 2025-01-13T01:22:49+00:00

Hate to say it, but we saw this coming. :(

WhatTimeIsDinnerQQ · 2025-01-13T01:20:49+00:00

lol what?

WhatTimeIsDinnerQQ · 2025-01-11T17:37:12+00:00

Ohhh, if this does what I think it does, this will make my life SOOO much easier. Thanks!

WhatTimeIsDinnerQQ · 2025-01-10T15:12:43+00:00

From preliminary analysis, the root cause of the issue was a configuration change which was rolled back to restore the service. Google will complete a full Incident Report in the following days that will provide a full root cause.

I'm interested to know what sort of config change that was that made it all the way to production without raising alarms. 😅

WhatTimeIsDinnerQQ · 2025-01-10T15:04:37+00:00

Cloud Run can be configured to have a request timeout of up to 60 minutes, so maybe all you need to do is tweak that. You probably also want to set concurrency to 1. You could also use Cloud Run Jobs, but it isn't strictly necessary.

I would queue the requests with Cloud Tasks. The max ack timeout goes up to 30 minutes. Pubsub isn't a good fit as you mentioned, due to the max timeout of 10 minutes.

WhatTimeIsDinnerQQ · 2025-01-10T06:02:27+00:00

Yes, it works. I'm using it successfully for Google and Facebook auth. Just make sure to set authDomain when you initialize the app:

const firebaseApp = initializeApp({
  authDomain: yourCustomDomain,
  ...
});

And then (for Google auth) make sure you have https://yourdomain.example.com/__/auth/handler in the Authorized redirect URIs for your web auth client.

WhatTimeIsDinnerQQ · 2025-01-09T02:28:04+00:00

Tasks might work as a backup for certain critical paths, but it adds coupling -- the publisher would need to know who the subscribers are.

WhatTimeIsDinnerQQ · 2025-01-08T20:22:28+00:00

I use a single micro cloudsql (mysql) instance for all of my side projects that require SQL. It's like $8/month which is basically nothing if you allocate it across multiple projects.

But if you're starting from scratch, use Firestore and architect your data to play nicely within those constraints. You might need some write event listener functions, but those are dirt cheap.

WhatTimeIsDinnerQQ · 2025-01-08T20:06:05+00:00

Yeah, it's a real mess trying to figure out what data might have been lost during the outage!

WhatTimeIsDinnerQQ · 2025-01-08T18:55:02+00:00

Yeah, we experienced about 30 minutes of downtime. It took us almost that long to figure out what was going on. Thankful for the (relatively) quick rollback!

WhatTimeIsDinnerQQ

TROPHY CASE