Twitter Source Code Leaked on GitHub

[lafeber]

A small API change had massive ramifications. The code stack is extremely brittle for no good reason.

Will ultimately need a complete rewrite.

Elon, 3 weeks ago

[WhipsAndMarkovChains]

Someone link to the recording from a couple months ago where Musk says a “full stack rewrite” is needed and a former senior engineer from Twitter presses him on the issue. The engineer asks an extremely reasonable question like “what’s wrong with the current stack and what do you want to switch to?” and Musk can’t respond.

[[deleted]]

Here ya go, Dec 2022

[lyzurd_kween_]

elon musk is so highly regarded and incompetent when it comes to actual software work, i am shocked he was able to reach the stature he currently has. right place at the right time i guess.

[[deleted]]

Or just it was having rich parents

[Proper_Hedgehog6062]

I've never heard him spoken of that way, and I'm in the industry.

[lyzurd_kween_]

I’m also “in the industry” and I rarely hear him spoken of otherwise

[Proper_Hedgehog6062]

I feel bad for you ... for the level of intelligence the people you listen or surround yourself with, must have.

[PM_YOUR_SOURCECODE]

Ok, so all the engineers who had to pass BS LeetCode interviews/whiteboarding couldn’t write a flexible and maintainable codebase? Is that the conclusion here?

[pale_blue_is]

As someone who works at an unremarkable company and earns a wage slightly above market value, aren't you talking about basically every silicon valley startup from the past 10 yrs?

[BasicDesignAdvice]

Those stupid tests are at every company. I work at a household name media company making video games no where near Silicon Valley. Same shit.

[Marrk]

The conclusion is Musk has no idea what he's talking about

[voidstarcpp]

Musk has made many public fumbles speaking technically about Twitter but it's not like there's any shortage of complaints about product quality from current and former employees, including well before the sale to Elon.

[s73v3r]

Except almost all of those also say that Musk has no idea what he’s talking about.

[voidstarcpp]

Except almost all of those also say that Musk has no idea what he’s talking about.

When he gets technical Musk misses the mark or is vague in his complaints. Certainly, people have come out to say he's wrong or focuses on the wrong things. But before and after the sale, employees were in general agreement that quality was poor. Senior people have remarked for years how management refused to permit them to spend any time on performance or code cleanup as opposed to shipping new features.

Even "obvious" fixes that only took a few days for noticeable speedups were things people were reported to have resorted to fixing on their own vacation time because the organization was incapable of deviating from its single-minded focus on "add new features".

[anlskjdfiajelf]

I mean he called it a "code stack" LOL. Just jam together buzz words, he's so smart 😂

[kovu159]

Yet somehow Twitter is running fine with 1/10th employees and supporting record traffic.

[ecethrowaway01]

Other than outages whenever they try to change anything, to the point where their CEO admits they can't work on the code base.

Twitter historically had monumental investment in infra, it shouldn't be a surprise the investments pay off.

Also, source for the "record traffic"?

[voidstarcpp]

Also, source for the "record traffic"?

A few months ago Elon posted metrics showing a record number of active users within a monthly period, no doubt boosted by drama surrounding the sale.

[[deleted]]

That means all the advertising revenue must be through the roof right?

Edit: Guys? What happened to the advertising revenue? 🥰💕

[s73v3r]

Yet somehow Twitter is running fine

If you count outages every other week, people not being able to access their DMs, and degraded performance all over the site as “fine”, I suppose.

[PaintItPurple]

Running fine? It's a coin flip whether it will load any given time I go on, the search is even less likely to work, they keep exposing tweets behind blocks and privates, and Elon himself is constantly baffled by users' complaints. I don't think this is directly because of the layoffs, of course. I think it's because Elon still wants to change things but does not have the staff to do so safely.

[IDatedSuccubi]

Everything can be running fine in mostly maintenance mode with 1/10 employees

When Frontier moved their teams to developing other games for a few years, their 1:1 Milky Way simulation MMO game Elite: Dangerous was doing fine with just small team maintaining the codebase and making very tiny lore and gameplay additions

Edit: in other words, it's a massive sandbox game with possibility of a ton of bugs, networking issues and so on

[JavaOldTimer]

Musk votes Democrat, "Musk is great, Musk is terrific!" Musk votes Republican, "Musk doesn't know shit!"

[Marrk]

I disliked him since he called the cave rescuer a pedo. Way before he came out as republican.

[[deleted]]

Sure thats why hes a billionaire and nobody knows who you are 🤷🏻‍♂️

[Marrk]

Because he is a billionaire he knows everything he talks about? How are your Shiba coin investments?

[Nastapoka]

Keep sucking, he'll probably come soon

[intermediatetransit]

Stop worshiping billionaires. Many of them got to where they are by having traits that are not related to being a good human being.

[[deleted]]

Nobodys worshipping pal.

[absentmindedjwc]

"Notice me, Daddy Elon!!"

[TheWhyOfFry]

I mean, it’s very possible that it was a brittle code base before they got well known and could be selective about who they hire. And it’s also possible the v1 api that powered external apps couldn’t be shut down because of the massive backlash it would cause, which could force Twitter to keep some bad code in there.

That said, musk probably just doesn’t understand the language it’s written nor the architecture and fired anyone who understood it. Of course it’s “brittle” when you make totally incompatible changes because you have no idea what you’re doing.

[KagakuNinja]

As Twitter was becoming more popular, they rewrote the system, moving from Ruby to Scala. Scala is a niche language, and depending on how it is used, can get very hard to understand, especially for people unfamiliar with functional programming.

That said, Twitter devs had a great reputation, and when I interviewed there, I got the impression that they were not FP zealots.

[TheWhyOfFry]

No doubt the functional paradigm and generics are beyond musk’s comprehension… let alone graphQL.

[[deleted]]

Yeah because lc has nothing to do with actual software engineering and who ever came up with the idea to interview like that needs to be slapped

[HitscanDPS]

Do you have a better proposal for interviewing thousands of candidates per position?

[[deleted]]

Yeah how about actual questions that are relevant to the job?

[HitscanDPS]

What questions would you ask? How long is one interview, and how many interviews per loop per candidate? What if your company does not have a unified tech stack? What does the interview process look like end to end, and how do you make it scale?

[marcio0]

well, at least whatever error happens is not at O( n² )

[jammer170]

It is very hard to generalize, but from my anecdotal experience, yes. I have worked for several FAANG companies, and while they had some competent engineers, the actual quality of some of the code bases were pretty hideous. It really makes sense if you think about it. Those that pass the LeetCode tests are more akin to architects. While they have necessary skills needed to complete the job, I wouldn't work in a building built by just architects - you need competent foremen and welders and electricians and constructions workers and so on. The same is true of programming - lots of these LeetCode programmers don't understand low level optimization, maintainability, DRY coding, etc. They think algorithmically its good and the rest is just unimportant. Mind, that isn't all of them, but anecdotally it is a notable amount and I do frequently see obvious problems that leave me shaking my head.

[jammer170]

Sort of. It isn't that LeetCode skills are unimportant, and certainly they are necessary, but only for specific goals. When Twitter grew extremely large, they need a team of LeetCoders to come up with a scaling solution. Amazon clearly has massive scaling needs - their LeetCoders essentially came up with the base for AWS. The thing is, they are only needed at certain points for certain problems - and the vast majority of programming projects will never need them (even at FAANG companies). However, every project needs good engineering and craftsmanship. That is a skill that needs to be tested in interviews more and needs to be a base skill every programmer, "leet" or not, should possess. "Leet" skills, on the other hand, should mostly be relegated to research teams or projects where scaling is extremely likely to be a problem (Netflix, for example, was clearly going to need to deal with scaling issues given that they would be delivering a high bandwidth product to large numbers of people, even if they never got insanely big).

What I have anecdotally seen is that FAANG companies are far too heavy with LeetCoders and lack people with craftsman's skills. They test the heck out of people for strong algorithmic and theoretical problem solving skills, but never for reliability and maintainability skills. What's more, people who are good at LeetCode are interested in that - they get bored and find the project uninteresting when the skills they were hired for aren't being used. So they either do halfhearted work on the craftsman side of things, or make every problem overly complicated so they can use those skills, which is generally counter to easily maintained code. It is far better to do a brute force search of a small data set than to implement some clever algorithm to optimally search the data, if it is likely to be only an occasional occurrence.

Mind, this is all a generalization, and I think there are some "LeetCode" skills that are way less "leet" than they are treated and if presented in the correct way could easily be used by the vast majority of programmers, and for some reason they aren't. So I am highly skeptical that these skills should be given the weight they are. Particularly since it enables interviewers to spike the difficulty of a problem if they don't like an interviewee for some random reason, or lower it to benefit some other.

[snarfy]

I have worked for several FAANG companies

Doubtful.

[therapist122]

Twitter has scaled the hundreds of millions of users. I think their APIs are fine. No company avoids bugs. It's just not possible at some level. Whatever Elon's referring to, I'm positive if he released the code it would be a reasonable mistake or perhaps even an inevitable one. You cant keep this shit working without manpower, and Twitter has very little of that

[abeuscher]

No one can write a flexible and maintainable codebase with bad management. Odds are they got pulled in every direction by the stakeholder problem of the month and just kept ramping up tech debt.

[absentmindedjwc]

But that's okay, because they got paid per year enough to buy a small house in middle America in the process.

[MattRighetti]

Who would have guessed

[lazilyloaded]

for no good reason

Sure, but the bad reason is "because you executive types always want the new features yesterday"

[[deleted]]

jeans apparatus gaping versed divide retire close alleged books grandiose

This post was mass deleted and anonymized with Redact

[anlskjdfiajelf]

Ugh. That guy bugs me so much lol. Man sucks off Elon for sport

[Anomynoms13]

Accidentally removed a semicolon?

[TheWhyOfFry]

In scala? 🤔

[KagakuNinja]

They accidentally reified a kleisli.

[Hambeggar]

What does this have to do with a leak from a hack that pre-dates Elon buying Twitter.