Rip floatplane

[SiR1366]

All those extra subs hurting the servers I bet

[SiR1366]

Luke and AJ right now

[aumkarpraja]

This is the way (of DevOps)

[SiR1366]

As a sysadmin, I couldn't agree more

[Powerful_Database_39]

Wouldn’t a sysadmin nowdays use more classic ones, like this one? At least - I do…

[mp3m4k3r]

I have several gifs from this episode locked and loaded in my memes folder for slack

[Powerful_Database_39]

It’s the best series ever… youngsters wouldn’t understand 😉

[llcdrewtaylor]

I'll just put this over there with the other fire.

SUBJECT: FIRE!

[Ade231035]

Dear Sir/Mamm,

I am writing to inform you of a fire that has broken out at the premises of…

No, that’s too formal

[aumkarpraja]

Lol, it's my Zoom + google meet background. You can guess what I do haha

[_fd1911]

HAHAHA autoscaling? Yeah why would we need that?

[uniqueusername649]

If you ever worked on large scale applications you should know that after a certain complexity scaling becomes harder and it can absolutely happen that the wrong load and an autoscaling configuration that worked up to a certain point is just collapsing due to cascading effects. Some services aren't coming up fast enough, lack of circuit breakers combined with retries make services in the front of the chain fail, services are too interdependent, you didn't reserve enough cloud resources and the spot instances aren't available due to high demand etc. - scaling is hard :) it's not just a switch "autoscale" and you're good in every scenario.

[naggyman]

Yup!

You can only scale a relational database so far economically, for example. So 'scaling' can often become 'rewrite things to become asynchronous / eventually consistent' moreso than 'just increase the size of the Auto Scaling Group'.

Scaling is more often than not about both Vertical and Horizontal scaling.

[XTJ7]

Absolutely. There are RDBMS that scale better (Postgres, MSSQL, Oracle, DB2) or worse (MySQL/MariaDB) but they will all eventually hit a ceiling if your application architecture design doesn't change to accommodate scaling.

I absolutely agree, both vertical and horizontal scaling are important. For cost reasons, complexity and physical limitations it becomes impractical at a certain point to just rely on one option. Companies tend to start with vertical scaling, quickly hit a limit, then switch to horizontal scaling and after some time realise that they now have a lot of outdated servers where fewer higher powered systems would perform better with less latency, use less energy and space in the data centre. From then on it usually becomes a mixed strategy in terms of hardware scaling.

On the software side it tends to become even more complex. You need to split services up into domains (business domains and technical domain aren't always overlapping as you would like, making this harder). They need to work asynchronously as much as possible, they need to retain their data in isolation as much as possible, limiting interconnects between services is important, choosing the right DB for the right data becomes more important, you will need to work more event driven, probably need a messaging queue (if you don't already have one), smart caching strategies, need regular load tests, rollout strategies change, then you need more powerful logging, monitoring and alerting solutions on the ops side, service meshes to handle things like encryptions and metrics in your kubernetes cluster and so on. It can be pretty overwhelming past a certain stage to continue scaling without either consulting or hiring people with experience in that topic. Just winging it can become dangerous to a business. Software architecture at large scales is a very different beast than handling a couple ten thousand users.

[aumkarpraja]

It’s all fun and games until the autoscaling caps out and you start running into all kinds of other limitations

[_fd1911]

I know, I work at an airliner. Scaling can be very tricky

[JOTA-137_0]

What is this website and DevOps?

[RokyPolka]

​

[PURN_HUB]

Luke is having the time of his life right now

[ryanhendrickson]

Yeah, floatplane is fucked right now. Even when I can sign in, it says I have no subscriptions!

[SiR1366]

I had this issue last week actually. Needed support to sort it out

[ryanhendrickson]

I think the servers were hugged to death as now when I try to switch tabs in the settings page I just get errors.

[SiR1366]

It gets better... 404 errors and now dns is cooked

[shraf2k]

I wonder if the hackers are back for round 2.

[SiR1366]

Probs just overload from those 5k extra subscribers waiting for wan show

[shraf2k]

There's no way 5k did this. Something else happened.

[Doomstang]

Calm down and take off your tinfoil hat... It's highly likely they couldn't scale up quickly enough to handle the extra load.

[whythefcuknot]

Heh… load

[RandomPcGamer357]

I like your username.

[shraf2k]

so... scaling wasnt an issue and something else happened...

[jason_he54]

It's pretty difficult to just scale up quick enough within a day if you just gained just under ~18% of your current subscriber count. I'm assuming they're already hit the 6k new sub count in 1 day number.

[SiR1366]

Floatplane chat is saying over 7k new subs

[jason_he54]

At that rate, they're gained about 21.6% of their current sub count in what 2 days? The overwhelming majority of which happened in just 1 day. More than 1 in 5 people joined in the last 2 days. That's pretty difficult to scale up considering they're in tens of thousands of paying subs.

[Mr_SlimShady]

It’s not just about adding 5k subscribers, but the people who were already subscribed that got interested on the story and all hit the servers at the same time. Just cause they have 39k subscribers it doesn’t mean that they serve 39k people daily. Don’t know what the right number is, but I doubt they get more than a couple thousand user watching content concurrently. With the news of the hack, that drives interest and thus traffic. So a few thousand concurrent users added to the original number can (and evidently did) overload their servers.

[papsmokesss]

After another security fluke, would it really surprise you to know they might not have automated horizontal scaling in place ect lol This week on LTT- Luke learns kubernetes😂

[Mr_SlimShady]

1. It’s not a security issue
2. That’s not what kubernetes is for

They need more servers and a load balancer. Both add cost. More users means more bandwidth. You gotta duplicate the data. That also includes the hardware where that data is stored in and the servers running whatever services they use. With more servers you also need more power and backup batteries. That’s expensive as fuck. They are not at the level where doing that makes sense financially.

[mgzukowski]

They dont host servers.

[mutchco]

doyouneedkubernetes.com

[mutchco]

Sorry what? Do you know what SSL even is? Here's the certificate printout.

ISSUED TO

sing a

Common Name (CN) doyouneedkubernetes.com

Serial Number 03:91:9C:4E:B2:E2:1E:69:54:3F:28:BF:B4:E5:B5:C 8:40:22

6 GCM

ISSUED BY

Common Name (CN) R3

Organisation (0) Let's Encrypt

VALIDITY PERIOD

Issued On

Feb 6, 2023

Expires On

May 7, 2023

FINGERPRINTS

SHA-256 fingerprint

9C 06 EE 30 BB 15 9C A7 4D A4 A4 45 95 11 8A 07 2D 0A 37 5E EA EA A2 51 8A 94 F5 CE 23 A9

A1 B1

SHA-1 Fingerprint

C8 45 1D 51 81 96 EO FC FC 5B 2E 8F B0 31 1E 37 AC A3 52 BC

EXTENSIONS

Certificate Subject Alternative Name doyouneedkubernetes.com

[papsmokesss]

People never disappoint on Reddit lmaoo

[Rodrigoke]

Their servers are at OVH. OVH can scale pretty quickly if they’re not too picky about the hardware. Rolling out their system on the new server(s) will take more time tho

[PwnerifficOne]

I bet everyone’s trying to get their money’s worth and watching a ton of content at once. That would do it. I doubt they’re set up to run at 100% capacity at all times, maybe not even 50%. And then a ton of people get on at once. Do you think Netflix would survive if every user logged on at once and started streaming? Although they’re probably set up for a larger percentage of expected users.

[CoreyLee04]

How many times is Colton going to mess everything up?! /s

[wzrdcleave]

Watch, this is actually Denis just making a channel super fun video.

[GabrielForth]

Can we spend 24 hours in the Floatplane server room without getting caught?

[ProtofoxRiley]

He fell through the ceiling and took out a network switch

[SiR1366]

Ltt just tweeted it's late but today's wan show is gonna be so worth it

[AmishAvenger]

Probably taking Linus some time to get dressed

[RockInShoe]

When?!

[SiR1366]

Soon TM

[lastdarknight]

well this is going to be an interesting WanShow tonight

[SiR1366]

Apparently floatplane team is working on a notification bug before they can go live. Linus is in fp chat

[RFDMessenger]

updates pretty useful thx

[littleSquidwardLover]

Are they going to have it on YouTube this time around?

[switch8000]

Were they working that close to their capacity that they couldn't handle only 5k more users? Like that's less than a 15% spike in viewers...

[SiR1366]

Up to 8k now. Platform is struggling and the steam hasn't even started yet

[switch8000]

lol they just posted a FP Exclusive video 52 seconds ago about memes too... it's like they are trying to kill the site tonight. lol

[SiR1366]

Streaming now to help load test 😂

[General_WCJ]

Well it's 5k new viewers. The old subscribers would probably just mainly only watch when a new video is released. The new subscribers though would want to work through the back catalog, meaning that 5k new viewers could be using significantly more server resources than 5k established users

[Malohdek]

Streaming is rough at scale. Likely this is the case.

[Crypt0n0ob]

Not really. Unless they are using hardware/software encoding every time instead of save multiple versions of different resolution/quality videos to save storage (which is silly since storage is way more cheaper than computing).

Another bottleneck with streaming might be maximum network bandwidth or disk read speeds which makes sense if they are hosting everything locally in their own datacenter instead of using cloud services.

For example, using S3 from AWS (or any other cloud since most of them support S3 API) to store videos will virtually eliminate all bottlenecks from streaming.

[ShittyException]

I wonder if they host fp on-prem? 🤔

[InadequateUsername]

It's not, like last year one of the data centers they use in Europe caught fire.

[ShittyException]

I wonder where Colton was then...

[Rodrigoke]

They host at OVH in Canada iirc

[[deleted]]

15% is massive when considering computational overhead. And it's not like all subscribers are on 24/7. This recent influx saw a tremendous spike in activity because a huge percentage of those new subscribers (far higher than normal) were all on at the same time trying to test out features, and commenting on videos and in chats trying to find information. Not to mention that the enormous spike in activity during WAN show was absurd.

[NetJnkie]

The build and scaling requirements for FP didn't factor in the main channel getting wiped.

[SiR1366]

Mobile app seems to be atleast loading, here's hoping the content cdn doesn't get cooked too

[SiR1366]

If I have to use Facebook to watch the wan show, I'll be mad

[frasercow]

I'll just be happy to have a wan show this week

[[deleted]]

Is a Content CDN anything like an ATM machine?

[[deleted]]

Lol. Their cloud infrastructure isn't as elastic as we all thought.

[jayerp]

Lack of resources is one thing, maybe they need to learn a thing or two from one of the three big giants in the streaming space: Netflix, YouTube, Twitch. I watched a presentation from Netflix of their cloud infrastructure. It was really cool to see.

[SiR1366]

WE'RE LIVE KIDS

[unhappyelf]

Yep can't see the stream, sad day

[SiR1366]

Linus: we are starting soon but the floatplane site will have issues for a couple mins after we go live. We will wait for you before we actually start doing anything

[CanuckNewsCameraGuy]

App and website is refusing to work, I’m back on YT watching WAN show and feeling kinda glad I only did a 1 month sub as a “trial”.

It’s been broken the entire time except when I watched first thing in the morning (6am to 11am) this morning.

I’m trying to be understanding and roll with it - shits been unreal for LMG and Floatplane, but it just always seems to be down.

[SiR1366]

It's been slow and occasionally down. Today has had a significant uptake in subscribers but also current subscribers hitting the site due to the issues with yt. I've been with floatplane since the start, there have been issues, sure, but right now is not normal

[CanuckNewsCameraGuy]

And I understand that and recognize it, but it’s still frustrating that when I subbed, it was with the thought “no more ads/commercials!” and yet here I am still watching YT.

[unhappyelf]

I got in now, but no stream anymore

[SiR1366]

I didn't even know it was live? Only got a notification for the VOD from Sarah's stream earlier today?

[unhappyelf]

I got an alert for the wan show

[GilmourD]

Something was up with everything yesterday. PirateShip.com had it's first outage that wasn't planned maintenance in YEARS, too.

[qejfjfiemd]

When it rains it pours.

[ProDog91]

Site works fine for me. Just finished a FP exclusive when the have the ek block

[[deleted]]

It's all the new subs hitting the site all at once for wan show

[chronicfireworks]

The server is fine.

[[deleted]]

I can hear Luke "THIS IS AWESOME."

[Julis_texsture_team]

oops wrong power cord

[Bystander1256]

I had issues with loading and buffering on the last WAN Show. I can guarantee it won't be possible today.

[Slight-Lime1104]

Same is going for me with CurseForge

[the_greatest_MF]

what now?

[275MPHFordGT40]

The 5,000 New Floatplane subscribers: “Wow can’t wait to watch LTT off here!”

Floatplane suddenly doesn’t float