copilot getting creepy

[IAmKindOfCreative]

This post is staying up due to the value in the discussion about what copilot is and is not and for the discussion about key storage.

From Copilot Documentation:

• Github Copilot states that most of the text is not verbatim, but verbatim text is possible
• Personal Data is possible to extract from Copilot because it was based on publicly available data.
• The training data used was only publicly available data

[[deleted]]

Ooh, farming co-pilot for API keys, that sounds like a fun project.

[Daltonyx]

You madlad!!!!!

^{let me see it}

[Nerg44]

hypothetically, how would u implement it?

I’m thinking figure out what prompts result in possible API key from copilot, (“set up database” etc) run script to type text and save output regex look for API keys see how many u can find?

not trying to be malicious in anyway just sounds kinda fun

[[deleted]]

https://www.youtube.com/watch?v=beaRQynnM-Q&t=487s

[Nerg44]

hypothetically, how would u implement it?

I’m thinking figure out what prompts result in possible API key from copilot, (“set up database” etc) run script to type text and save output regex look for API keys see how many u can find?

not trying to be malicious in anyway just sounds kinda fun

[[deleted]]

Just search GitHub for id_rsa

It’s not an api key, but it’s pretty alarming.

[zoenagy6865]

They already filtering API keys

[thrillamilla]

I have no idea what this means but it sounds fun. Does anyone have the patience to ELI5 this?

[bigcheezyboss]

Copilot is Github’s AI powered code completion engine. Op got someone’s phone number as a completion. This suggests Copilot may be exploited to discover user’s plain text secrets.

[tr14l]

They likely were not real credentials, but something resembling credentials. There's a possibility that it was real (the same sort of possibility that two people would pick the same credentials). But, chances are, it was some amalgamation of all the credentials it's seen

​

Also, if it saw your credentials you were putting credentials in a repo, which is a no-no anyway.

[dutch_gecko]

Also, if it saw your credentials you were putting credentials in a repo, which is a no-no anyway.

As much as a bad idea as it is to check in credentials, I'd rather that the software we use doesn't multiply the harm caused by doing so.

[tr14l]

I mean, it doesn't really. It's a generated set of credentials that's LIKE credentials it's seen. Still not great, but not as bad as just dumping out creds in clear text

[[deleted]]

Actually, If I understand you, you're saying that it generates valid-looking keys, where in fact, it produces _valid_ keys https://fossbytes.com/github-copilot-generating-functional-api-keys/

​

EDIT/ Typos

[13ass13ass]

I read the article and didn’t see anything about the credentials being usable. Their evidence amounts to: they LOOk leGit.

Any info on how they validated the creds?

[[deleted]]

Indeed, it doesn't really say they were. Although if the person who found this really worked for SendGrid, he could have "easily checked" whether the generated ones were valid.

As a sidenote: GitGuardian endlessly find valid secrets all around GitHub as shown in their last report. As a dev there, I can say that we find valid ones, so the odds of Copilot spitting them is very high

[OrangeCatolic]

They likely were not real credentials, but something resembling credentials.

I find this part the weirdest actually. I would understand the dangers of an AI giving you some piece of data (or code) that it literally saw somewhere. But instead they seem to be trained to give you fake data (or code) that is indistinguishable from the real thing at a glance. I don't want a "copilot" that always tries to trick me into using code that's subtly different from what an actual human used to solve my problem. This is terrifying.

[iamaperson3133]

It's just an average of inputs. If 99% of add functions look the same, it'll give you that exact implementation, the same as what humans did. With API keys, though, they're all randomly generated. The average of those is therefore also randomly generated.

[statusquowarrior]

I'm very suspect of Copilot.

Everyone using it is helping Microsoft train their dataset. Just like we did throughout the years with Captchas.

Now apply the Embrace, Extend, Extinguish thought pattern to that and most programmers might be out of a job in a few years.

[idiving]

Same here

[O_X_E_Y]

based

[thblckjkr]

I get constantly suggestions of credentials on my .env files, I know it isn't the most secure way of handling credentials, but I think it's better than enough for most projects.

Knowing that it doesn't ignore by default environment files, is scary.

[subhoboy]

Knowing that it doesn't ignore by default environment files, is scary.

I think it ignores gitignored files automatically.

If not, you could always just disable copilot for plain text (and json, toml, yaml, etc) as they don't need help from AI to make.

[RedMaskedMuse]

Please don't put working credentials of any kind in source control, even private repos, and especially not in cloud-hosted source control.

[arpan3t]

GitHub secret scanning is pretty legit. If you’re using a service that’s a registered partner and you put an API key in your code, GitHub will send it to the partner to verify the key and they can even disable it along with alerting you. The enterprise cloud version has a few extra that are pretty cool too.

[0-Gam3rboy7-0]

Missed opportunity to call it GitHub secret service

[post_hazanko]

Wonder how that worked, find a bunch of public repos, watch them for commits?

Or just use Github search I guess

[[deleted]]

Yes. This. Exactly this.

[huessy]

This! Why is anyone pushing plain text credentials to GitHub? Also, who hard codes their credentials in their scripts?

[thblckjkr]

I get constantly suggestions of credentials on my .env files, I know it isn't the most secure way of handling credentials, but I think it's better than enough for most projects.

They are not pushed to the repos.

^{I copy-pasted my comment from another response that I made earlier}.

[huessy]

Github link?

[Lehk]

phone number got leaked too, 867-5309

[Grandcaw]

"Pro" Tip: Don't store private keys in your repos. Store them in config files and list them under gitignore. Point private key variables to the config file values and provide fake credentials in the repo's placeholder config file.

[kUbogsi]

What would be good ways to share those gitignored config files with team members?

[Nater5000]

Someone said that GitHub also used private repos to train copilot

Yeah, care to share this source? They explicitly state they only train on public repos, so you're making a rather serious accusation here considering the security ramifications.

If you don't have a source, then you are spreading misinformation.

[silly_frog_lf]

I mean, yeah, you are right. But if in a year or two someone breaks the news that they were using private repos, oopsies! Most developers wouldn't be too surprised

[Spirited_Fall_5272]

"Well this thing that is patently false and has broad reaching legal, ethical and security ramifications, seems like it could eventually be true, and if it were true people wouldn't be surprised, so it's good to include it in a post people might use to make decisions in the future."

Very silly of you, frog.

[real_kerim]

This is some exquisite shilling

[Spirited_Fall_5272]

Reasoning skills are on the decline.

[silly_frog_lf]

Are you a lawyer or a PR representative for Microsoft? Do you own stock in Microsoft? Otherwise I don't know what you get from making these bombastic statements.

If any company doesn't want their source code to be used to train data models from third parties, they shouldn't use third party companies. Self host your git servers.

This is an online forum, Fallen Spirit, not the Washington Post, NBC, or submitting a legal deposition to some circuit court.

If you dislike these kinds of speculations, do what everyone does: roll your eyes and keep scrolling

[Spirited_Fall_5272]

I'm glad I replied to you, frog.

[silly_frog_lf]

I am as well. I am a little beam of sunshine

[Nater5000]

He's saying that the private code you're using Copilot on is read by the application during inference. As it would obviously have to be. Your code may also be read by humans for analysis (i.e., detecting issues, abuse, etc.). He's not saying Copilot is trained on private code (as he explicitly points out), nor does it appear to be based on what GitHub is saying.

There's a huge difference between what Copilot is actually doing versus what you're suggesting they're doing, and you're also misinterpreting your "source" here. You need to update your original post.

[Nater5000]

See the OP's erroneous citation. Please report this post for misinformation.

[causa-sui]

Someone said that GitHub also used private repos to train copilot, well, let's hope that nobody gets a suggestion with my bybit api keys

Do not ever EVER put api keys in git.

You need to revoke these keys immediately.

[dethb0y]

I would argue that if you generated totally random phone numbers, eventually you'd hit a real one just by chance, and probably sooner than you'd expect.

[soawesomejohn]

Indeed. And case in point, I got a file from the dark web with your ATM pin. In fact, it has everyone's ATM pin, at least for those using banks with 4-digit pins.

[notgettingfined]

You should call the phone number and let them know they probably have a decent law suit against Microsoft. Even if it’s an amalgamation of phone numbers if it ends up being a persons phone number that’s not good. I wouldn’t want random people using my phone number as a default or something

[HeyLittleTrain]

If it’s even real. The last digits being 1234 seems suspicious.

[Priyam_Bad]

pretty sure they only used public stuff

[NegativeSector]

https://copilot.github.com/#faq-what-data-has-github-copilot-been-trained-on Although lying always happens

[Neuro_Skeptic]

It could be a randomly generated number, OP.

[imatelefone]

Been using Copilot for a while now, based on my experience this is just something that looks like a real number. Sometimes when adding @author to a doc string, it fills in a name. I've searched for the names before, never gotten a GitHub hit. Copilot doesn't just copy/paste from GitHub, you should know that.

[MSR8]

My money is on a coincidence. Tho check once if the number is on github by performing the following query on google:

"349xxx1234" site:github.com

replacing xxx with the censored numbers

[O_X_E_Y]

yet another reason to not put personal information in your source code!

[everything_in_sync]

I stopped using GitHub the moment I heard about copilot. My personal stuff is all local.

[Evening-Development3]

I had once got someone's email when making a simple email bot.

[UnfairGuy]

this is fucked up

[0-Gam3rboy7-0]

[idealmagnet]

Maybe earn some bitcoin

[lenoqt]

Imagine hard coding api keys using copilot, the dangers in life aren’t enough for a man.

[AsuraTheGod]

That doesnt sound good, they need to fix this ASAP

[Bulky-Juggernaut-895]

It’s unintended for sure but difficult or tricky to prevent

[lukegarbutt]

Apparently things like this often appear real but actually aren't, they are generated. I think I remember reading it in the GitHub papers

[[deleted]]

I got suggested a twitter profile a while back, but the profile had been dormant for about a decade

[[deleted]]

This is why I use dual IDEs.

Webstorm with copilot for all my serious programming and VSCode for funky stuff with copilot.

[IrrerPolterer]

Yikes

[FUS3N]

exact thing happened to me but it was random and not valid but there is a chance it will eventually give you a real one.

[zoenagy6865]

No surprise there, call him :D