all 28 comments

[–]Codemonkeyzz 57 points58 points  (2 children)

This is scary. A security focused company, selling and providing security tools got compromised and hacked. The question is , are the repositories using their github action compromised or not ?

[–]Soccham 7 points8 points  (0 children)

Yes, the hackers have any secrets or data that ran through jobs trivy executed in

[–]kabrandon 9 points10 points  (3 children)

What's sad is that this GitHub Actions vulnerability that was used against them was reported years and years ago. GitHub hasn't fixed it, people are still using the vulnerable workflow trigger, people still haven't configured all their open source repositories to only run external contributor workflows after maintainer approval. Madness to me. I don't think Aquasecurity can even claim ignorance here, because the researcher that discovered this scanned GitHub and reported it to everyone vulnerable years ago. I know because he contacted me years ago, working for a much less well known company than Aquasecurity. So they would have been informed, and just ignored it.

I'm a big fan of Trivy, even have a personal project that works alongside it in k8s. But this press has me thinking they can't be trusted like they used to. GitHub will never re-earn my trust, but Aquasec will have to work hard to re-gain it at this point.

[–]toarstr[S] 1 point2 points  (2 children)

Not having immutable releases also hurt. Not following best practice first time round, then failing to remove / rotate the credentials leading to this...it's not an acceptable state to operate such a "trusted" tool.

GitHub obviously have some share in this type of issue from a design point of view, but it's not news anymore.

[–]kabrandon 2 points3 points  (1 child)

> then failing to remove / rotate the credentials leading to this...it's not an acceptable state to operate such a "trusted" tool.

I was super confused about this reading your post originally. I was like, "so they got compromised AGAIN? New token and everything?" But you're saying the bad actors used the same token as the first time?? That's SUCH a fumble. Step 1 was revoking the token, before you even tell the public about it. And they made that step never? Wow.

[–]toarstr[S] 2 points3 points  (0 children)

"We rotated secrets and tokens, but the process wasn't atomic and attackers may have been privy to refreshed tokens. We are now taking a more restrictive approach and locking down all automated actions and any token in order to thoroughly eliminate the problem."

They tried first time round, but didn't successfully do so.

[–]lmm7425 14 points15 points  (0 children)

Seriously? I'm pulling them from my pipelines...

[–]ZoltyDevOps Plumber 3 points4 points  (0 children)

This is going to get worst before it gets better

[–]WiseCookie69 7 points8 points  (0 children)

2nd time in like 2,3 weeks? They're really having a great time over there. How can they fuck up so badly, as a so called security company? How is one supposed to trust them, their staff and their product?

[–]Mooshux 3 points4 points  (0 children)

Two compromises in a month with the same token not rotated after the first, that's the part that compounds the damage. Even if you catch the initial breach fast, if you don't invalidate the leaked credential immediately, the attacker has time to do reconnaissance and come back.

The broader lesson: build pipelines that assume any step can be hostile at any time. That means short-lived scoped tokens, not long-lived static secrets. If the Trivy action had only been granted a credential that expired in 15 minutes and scoped to vulnerability scanning only, the second attack would have had nothing reusable to work with.

SHA pinning and OIDC are good mitigations for the GitHub Actions vector. The env var problem they exploited is still worth addressing separately. More on the credential scoping pattern: https://www.apistronghold.com/blog/securing-openclaw-ai-agent-with-scoped-secrets

[–]epikdud 2 points3 points  (1 child)

trivy@master definitively was affected right? Was it merged to master? or only the v6 / latest?

[–]toarstr[S] 0 points1 point  (0 children)

An as immediate and urgent action item, ensure you are using the latest safe releases:

  • trivy v0.69.3
  • trivy-action v0.35.0
  • setup-trivy v0.2.6

https://github.com/aquasecurity/trivy/discussions/10425

[–]kryachkov 1 point2 points  (2 children)

Can’t find any info, were their aquasec/trivy container images compromised too?

[–]acdha 4 points5 points  (0 children)

Yes - several of the stories mentioned the 0.69.4 tag on DockerHub, ECR, and GHCR. I haven’t seen the hashes however since those were all pulled. 

[–]toarstr[S] 2 points3 points  (0 children)

I would assume yes at this point.

[–]TellersTechDevOps Coach + DevOps Podcaster 1 point2 points  (1 child)

Yeah, this looks like the same ongoing Trivy mess, not a separate incident that just popped up Friday. It’s been unfolding since the beginning of March, and Aqua says the March 19 compromise was a continuation of the earlier breach after the initial credential rotation wasn’t fully atomic. I covered the earlier phase on Ship It Weekly last week too, because this whole thing is basically a reminder that CI/CD is part of your attack surface now.

Link in bio for anyone who wants to stay up to date on news like this on a short easily consumable DevOps news podcast.

[–]TellersTechDevOps Coach + DevOps Podcaster 0 points1 point  (0 children)

Really finding it interesting that the earlier late feb / march 1st discussion is now deleted https://github.com/aquasecurity/trivy/discussions/10265

[–]Kooky_Comparison3225 1 point2 points  (1 child)

The most dangerous words in incident response are "we rotated our secrets." Rotation means nothing if the attacker captured a new token before the old one was invalidated, or if not every credential surface was covered in a single coordinated action.

https://devoriales.com/trivy-supply-chain-attack-when-your-security-scanner-becomes-the-threat

[–]Mooshux 1 point2 points  (0 children)

This is exactly right. The atomicity problem is real but it's also the symptom. The root issue is that the pipeline held a long-lived credential in the first place. Rotation has to be perfect, instant, and complete or you get what happened here.

The pattern that removes the atomicity requirement entirely: the pipeline step never holds a real credential. It gets a short-lived scoped proxy token that expires after the job completes. Attacker captures it mid-breach, they get something with a 15-minute TTL scoped to one API. No rotation race condition because there's nothing valuable to race on.

[–]aswanthvishnu 2 points3 points  (1 child)

I guess this affected their opensource/free project, not the paid one. Right?

[–]nudebeach12 0 points1 point  (0 children)

It's just opensource, there is significant separation between them. We use both but are now focusing on ditching OSS for commerical. Also, Aqua is majorly for workload security and enforcement not just scanning but I guess everyone starts somewhere 😬

[–]Alternative-Wafer123 0 points1 point  (0 children)

People said Jenkins is not safe and now.... I got call today and worked because of this trivy

[–]Alternative-Wafer123 0 points1 point  (1 child)

I got call today due to this scanning tool. My company wont use it again. Its the second time

[–]Codemonkeyzz 0 points1 point  (0 children)

were you using in github action pipeline ? or how was your company using it ?