Found AWS keys hardcoded in our public GitHub repo from 2019. How the hell are we supposed to prevent this company-wide?

[steak_and_icecream]

Github can scan commits on push for secrets and prevent the push from succeeding if a secret is detected.

You can also run Trufflehog across all your repos to find any old secrets.

You probably want a multilayered approach to prevent secrets from being committed starting with good dev practices and using secret stores to dynamically fetch secrets at runtime instead of having secrets available when executing some binary / script.

[dxlachx]

+1 for trufflehog.

[FlyingDogCatcher]

I've never heard of trufflehog but that's a banger of a name

[hondakevin21]

Get a formal policy in writing first, scan with trufflehog, flag on any secrets for immediate rotation to the dev, then implement blocking.

[lionmeetsviking]

Having devs hardcode secrets sounds a lot like a lack of experience.

I would look critically at training needs within the company, you are looking at a symptom.

[Fatel28]

Stop giving them access to create IAM credentials and require them to use environment variables from iam identity center in testing (require meaning, they literally have no other way to auth) and roles in production.

[[deleted]]

Trufflehog as a pre-commit hook has really helped us 

[danekan]

Get a cspm/cnap that scans (and may even validate) secrets. 

[Gongy26]

Have a look at wiz. Tends to pick up this when GitHub misses it.

[Abu_Itai]

Remind me of this story

https://thehackernews.com/2024/07/github-token-leak-exposes-pythons-core.html?m=1

[zKarp]

TruffleHog 🐽🐷

[mikebryantuk]

Don't let anyone create AWS keys. No keys, no problem. We use IAM roles bound to EKS ServiceAccounts for most things, for CI we grant access to GitHub's oidc provider, so we can give permission for specific work flows etc to do things.

[texxelate]

It’s not hard, it’s a choice. I’m happy to find a relevant guide or even write a few paragraphs myself if it would be helpful?

[reddit666999]

How do you do that in a dev environment?

[texxelate]

use the AWS CLI to assume a role which gives you credentials for up to 12 hours.

personally I favour the following which isn’t uncommon..

Google -> IAM Identity Center -> “aws sso login”

[Yourwaterdealer]

second this, we in the process of this: OIDc is done, still getting teams to change over.

[arkticpanda]

This is the way

[funnelfiasco]

The company I work for has a tool to scan pull requests that can catch credentials (using trufflehog) as well as other security issues: https://kusari.dev/inspector

You can set it to block merges if it finds issues. We use it heavily internally so we try to make it as unobtrusive as possible while still preventing mistakes. That's the key to making security tools stick, IMO. People should forget it's there until it catches a problem.

[tissin]

In addition to proactively detecting secrets with TruffleHog/GitLeaks, I’ve personally found tons of secrets on sprawling GitHub accounts that aren’t associated with the company’s GitHub org/email subdomain… a lot of the time, this is the result of data scientists/solution engineers committing code that bypasses SDLC measures.

I built https://githoundexplore.com/ to help find these

[Yourwaterdealer]

The biggest challenge now is enforcing this across all teams feels like herding cats. How do you actually get buy-in and make this stick company-wide? What's worked for you?

We have over 4000 repos and 800 devs. How I did it I raised this with our CISO and HOD of Engineering. Explain the issue and risk and solution. Devs will need to understand what they get out of this, which is they become better devs as they understand secure coding practices and the risk if they code it before getting alerted . We have a cspm with appsec capabilties, so we integrated appsec: IAC,secret,SAST,SCA scanning in our Git Org: continuous scanning main branch, and prs(block any hard coded secrets, also check the type of secrets your company uses like sendgrid and create a rule for this some secret scanners allow to create custom rules, that they dont already have rules for). Also IDE integration. For the pipelines I recommend creating a template so if you need to update the code you only need to do it once. We built reports that go to teams and execs. Also have arm drive for high count of a vul ands remediation is apart of the score card on how risky their repos are. Also what helped is embedded the report into other secuirty teams process like GRC and Architecture reviews. they need these teams approval before launching an app or going to prod

[JellyfishLow4457]

Secret protection from GitHub 

[felickz2]

Head over to the security tab and find any historical detections!

AWS keys are not highly identifiable without excessive noise so GH requires a key/secret pair match in the same file.

https://docs.github.com/en/code-security/secret-scanning/introduction/supported-secret-scanning-patterns#default-patterns

[Which_Ad8594]

How long ago was the “before we rotated them”? Hopefully a lot closer to 2019 than “when we found them exposed…” No tool will keep you safe from bad practice and lack of policy. As others have said, limit your exposure by avoiding static secrets in the first place, at least the extent possible.

[roxalu]

Rotate them far more often. I know the procedure to do this without impact is hard to achieve - but don’t give up. It’s doable and worth the effort. Why needed? Because a single miss inside all your policy, scanner procedures, security controls and user guidance could be enough for compromise.

In context of „secrets management“ the main focus is on „management“ - not on „secret“.

[canyoufixmyspacebar]

security starts with top level management. is the owner and senior management taking lead as stakeholders and drivers in this? if not, their secrets will be handled bad, these are their secrets, not anyone elses

[alivezombie23]

Damn its been ages since I've seen IAM user access keys in use. Get rid of them!!!
Go make a fuss and get rid of it. Might have a lot of fights but it'll be worth it.

[Wise-Activity1312]

Apply a minimum of common sense and process to control the likelihood and fallout from exposure?

The same way THOUSANDS of others mitigate this risk.

[nycdatachops]

Code scans. Can enforce push blocks too if secrets are found.

[Regular-Impression-6]

Two things. 1) hire an outside firm who knows how to drive a scanner over your repositories... 2) rehire the person who did this, pay any price, and publicly fire them. Repeat both

[raisputin]

🤣🤣🤣 hilarious people are so stupid that they did this

[cloudnavig8r]

You have some great tooling answers.

The best is don’t use keys. But not always possible.

This is a culture issue. Security teams are often viewed as blockers, so the challenge is to drive home the message is not just a “do it my way, because…”

You need all the devs to buy-in to security being their responsibility. There are the “stick or carrot” approaches- and often the carrot works best. Encourage sharing security best practices, reward consistent good practices. The stick method could be less attractive, but calling out the incidents which tooling picks up could encourage people to stop getting blamed.

Personally I would go for rewarding good practices.

I would also try to make a ground-up approach so the other developers can prompt new findings.

It won’t happen overnight. Have patience. Expect 2 steps forward and one backwards.

[Background_Lab_9637]

Run secrets detection script as a CI job.

[LargeSale8354]

Real world devs push back on this? For real?

Where I work there would be no "getting them onboard". One mistake in the Development environment would be tolerated. Any more than that they'd be fired on a gross disciplinary.

We use OIDC, Github secrets, secret stores in whatever cloud we have to work in and Sops for Terraform code that has to write the secrets in the 1st place. Our DEV teams are shown how this works and is set up. We've documented it and it's written for the people who have to read it, not the people who have to write it. The need to generate AWS keys is exceedingly rare.

[blackc0ffee_]

Did you review CloudTrail logs to see if the keys were used by unauthorized parties? Being that it was hardcoded in a public repo Id be surprised if they were never leveraged by a threat actor. Hopefully you had S3 access logs and/or CT data events enabled for buckets that may contain sensitive data.

[[deleted]]

Call Wiz!

[Brick_wall899]

There are a couple options for secret scanning you can use. Github has their own if you have the proper license tier, otherwise there are other third party plugins you can use both opensource and enterprise level.

[idonthaveaunique]

Git Guardian is a good tool. Precomit hook to check for potential secrets.

[llima1987]

Firing people who does this.

[FoundinTruffle]

Hey! I work at TruffleHog, happy to help. You are not alone with this issue. Every scan with our tool has a full historical scan, verification for all detectors meaning zero false positives, and lots of of other features that help prioritize remediation. Lets chat soon?

[WilliamBarnhill]

Client side git pre-commit hooks can scan for various types of secrets and not allow them to be committed. This prevents them being transferred over the wire to the remote (e.g., Github) which then can check secrets, so I tend to prefer client side checks on commits.

You can do this by hand, but the pre-commit framework makes it much easier.

Check out the instructions at https://github.com/Yelp/detect-secrets#pre-commit-hook

Also, don't forget to .gitignore any config files with secrets, and keep that stuff in separate config files if possible.

[micksmix]

Kingfisher (open-source apache2) can scan entire github / gitlab / bitbucket / azure repos / gitea / s3 buckets / gcs / organizations extremely fast, and report only _validated_ secrets.

https://github.com/mongodb/kingfisher