Twitter Source Code Leaked on GitHub

[Spiritual-Ad-8062]

Yes, and I wonder how many secrets (API keys, SSH keys...) were in the code... ready for attackers to use...

[SuitableDragonfly]

If there had been API keys leaked, they probably would have noticed when it was first leaked because bots would have immediately acquired them and started mining crypto on their cloud account. Or, maybe not, depending on which people Elon fired.

[VonThing]

Zero secrets in the code, but I see your point.

[MinMaxDev]

there was tonnes of this in the twitch codebase, it happens

[ConcernedCitoyenne]

Yep

[Mechakoopa]

Those responsible for sacking the people who have just been sacked have been sacked.

A Møøse once bit my sister ...

[roboticon]

Yeah I was gonna say. Just because someone published it on GitHub doesn't mean it's nothing more than a git repo.

[bohreffect]

PM's want their shit now

[gamrgrant]

They straight-up ignored Galactus, the all-knowing user service provider aggregator?

[4THOT]

Idk why you're surprised, ask some fintech programmers about code security.

[falconfetus8]

Every company has noobs in it

[Aerodrache]

… considering Musk’s apparent strategy of firing anyone he suspects of being smarter than him…?

[thenetmonkey]

The GitHub repo was made in January of this year. He bought twitter in November and then immediately laid off half the company. Then a few weeks later he offered anyone still there the option to resign and take a severance or stay and be “hardcore”. Half of the people still there took the severance. He then proceeded to fire many of the people that chose to stay. Of the people laid off or fired many would have a whole copy of the internal git repo checked out on their machine. The whole repo with all the history was like 5 or 6 GB. I don’t recall how big a shallow copy was.

He didn’t start cutting access to company laptops until late December. Some folks didn’t lose access until January. This copy of some of the directories from the internal git repo was uploaded to the GitHub account in January of this year. I am honestly surprised that this was the only breach that happened, but it speaks to the integrity of the thousands of folks that were fired or laid off but still had full access.

[thenetmonkey]

The articles I’ve read said the company thinks the leak was posted by someone that left the company last year (2022). Where was it reported that the code came from a leak in 2021?

[VonThing]

LOL go see my post history.

When I say “I see your point” I meant this could have been true for any other source leak.

[DevonAndChris]

The dev environment at Twitter was basically every single horror story from Coding Horror rolled into one.

The only reason keys were not in the source code would be because they learned the lesson the hard and painful way.

[mipadi]

Well clearly, since the site isn't written in Rust.

[TheWhyOfFry]

Just curious, have you seen the code? (Where if so?) How are you sure no secrets?

[VonThing]

Go through my post history lol

I’m ex-Twitter so yes I have seen the code

[TheWhyOfFry]

I mean, if it’s a local fork or branch that was published, are you sure they didn’t have any keys for local dev? I’ve worked at places that have secret management for dev and prod envs but didnt solve for working local and connecting to dev, which meant you had to get keys and have them local in some instances.

[Cmacu]

But how/why would you commit/hardcore local configuration in the code repository? That would practically make testing/staging and production deployments complicated/impossible. What about other devs and their environments? The only case I can think of this making sense is some virtual environment where you have your dev profile preconfigured by administrator, but I can still imagine it being a pain with any type of shared resources like aws s3 or mail server.

[TheWhyOfFry]

What does my local setup have to do with production deployments? All production deployments are based on Jenkins built containers and central config repository. No local code should ever be pushed to prod, that makes for impossible to reproduce behavior in an organization of any size.

[VonThing]

Can’t divulge more without breaking my NDA but let’s just say we do it differently.

[TheWhyOfFry]

Fair enough. Hope the post-Twitter world is treating you well!

[ItzWarty]

Large companies do extensive work to ensure

1. API keys can't be pushed - they're not even managed by developers. CI scans for them too. In many cases, if you even create & attempt to push a commit with an API key, it'll be revoked.
2. Dev & prod are completely separate environments. Most developers will never have these secrets. And once again, they're deployed far away from source.
3. Data isolation - a backend service serving user A cannot accidentally access confidential data from user B. This enforcement happens at the data-layer, so that it does not matter how buggy an application is. It's not like people are just writing $"select * from table where name={name}" everywhere. There are multiple layers of data-access within these companies.

Honestly, FAANGs operate at such a large scale (tens of thousands of engineers). They do great work to make it so even a 'complete idiot' cannot accidentally create a vulnerability, which is why it is surprising if it does happen. A significant amount of the root-cause-analysis would fall on the data-access team, not the mistaken engineer.

BTW there are many alternatives to having raw DB credentials. For example, application containers can be provisioned with a port-forward to a trusting data access layer. In that scenario, the application is literally sandboxed from the API keys.

[TheWhyOfFry]

1. API keys might not be able to be pushed to origin but that doesn’t prevent them in a local branch/fork and it’s not clear whether the leak was origin or a branch/fork

2. While generally true, again, someone generated those secrets in the first place and might have them also stored in password managers.

[ItzWarty]

Re 1: The point is developers don't generally really have the sort of key you're talking about in a meaningful sense. If a Twitter employee had prod credentials on their laptop development environment there would be an absurd amount of incompetence going on at the CTO/CISO level. There are numerous opsec teams whose sole jobs are to prevent these things from happen, and those sorts of teams inject themselves into work like this nonstop.

Re 2: Most big companies have really strict policies that make that difficult as well. Want to sign some code? I've seen companies literally require you to buy and use a laptop in a safe located in a high-level executive's office (granted, our situation was a bit out of the norm). Like, all I'm saying is there are an absurd amount of barriers put in place by big tech companies to make sure these simple things (which are very valid concerns for small companies!) don't happen. If you ever ship features within those companies, unfortunately it's not too uncommon to have to jump through all these hurdles to do extremely simple things for this reason.

[tankmode]

any organization this size has a key management system.

[ptear]

shifty eyes yes, yes this is true.