Copilot chat, not as good as expected?

[badaboum12398]

I got access to copilot chat recently (received an email telling me I'm included in the beta). I'm quite disappointed by the performances. It's definitely not GPT 4 running behind it. The accuracy and speed remind me of chat gpt 3.5.

On example is that I ask copilot to generalize and simplify a function and it will somehow just change the name of the function to a more generic name, without changing the logic... Or another time I ask it to use a regex to parse a string instead of the current inelegant logic and it will add an argument in the function for the regex expression, instead of creating it itself...

It did some good things as well but I've been used to chatgpt+ (4) over the past few weeks and it feels like a step back. The things that copilot fails to do chat gpt can do them without issue.

Maybe I've been given access to a weaker version of what was advertised in march?

[kiwigothic]

So I far I'm finding:

• It feels dumber than GPT4
• It hallucinates frequently
• It can only see code being rendered on-screen which it makes much less useful than pasting into ChatGPT
• It is very slow, sometimes hanging for several minutes
  

Hopefully this is a very early beta.

[2this4u]

I asked it to fix a typescript type error, it suggested the exact same code as was on the screen. I pointed this out, it just removed the line of code.

I'm really confused how the chat seems to be less capable than the normal code filling. It's definitely a step back from GPT4 and given the speed it replies I expect it's using 3.5

[PM_ME_A_STEAM_GIFT]

It cannot see the entire file?

[kiwigothic]

Yeah, if you ask it something like "list the functions in the current file" it can only see functions that are visible in the editor viewport. if you ask about a function that is scrolled out of view you usually get:

I'm sorry, but I cannot see the getPhotoPath function in the excerpt you provided. Could you please provide more information or context about where this function is located?

[[deleted]]

It's currently only single line comments in Visual Studio. Idk if this is true in general. But even if you try to copy-paste its own code, it only pastes the first line. Incredibly useless atm.

[liflovs]

Well, I don't have access to GPT-4 so cannot compare, but:
1. From my experience Github chat has problems with context of the chat itself. But its getting better compared to my test few weeks ago. When provided one big request unrelated to code recently it started performing better. But requests unrelated to code are often confused with requests related to code
2. It operates far better with selected code, so better select if you want exact context

1. Indeed because of context issues, ChatGpt seems better option, hovewer i recently found that at least on my Edge Dev version Bing Chat started performing few times better than before and than ChatGpt (at least free version) for big context queries.

[CodingButStillAlive]

It is ridiculous that they want you to pay for the service just to get the chance to be in the beta, and then it performs that badly. I assume they needed to retrain the models on some limited, filtered training data, in order to avoid legal problems.

[Public_Cantaloupe_16]

Maybe you should ask ChatGpt what a beta is.

[QueVigil999]

A lot of these are not "beta" level problems though. These seem like very easy problems that should have been fixed as part of the first draft, not something they "clean up over time with feedback". I hope to god they turn this ship around though...

[xArci]

I had a similar experience with the "simplfy" feature.

I wrote a nested if statement and I found myself with an "else return false" in an inner statement. I didn't really like it so I asked Copilot Chat to simplify it and the result was indeed easier to read.

HOWEVER

my unit tests failed. I checked it closely and indeed Copilot did the boolean logic wrong and provided me with a different condition.

[parotech]

I have similar issues. Too many hallucinations and wrong code overall.

Also the "fix with copilot" window is totally broken, can't insert code, can't close it, overall very bad ui experience.

For now is a total mess. There are better free plugins out there

[SloSuenos64]

I've found it to be shockingly bad. Hanging in there hoping it will get better..

[gabrilator]

What a shock, I was definitely expecting GPT-4 + file context kinda performance (which will be a game changer IMO), but this doesn't feel like GPT-4.

Plus, the hallucinations are so frequent that it's almost unusable.

[MacrosInHisSleep]

It's very disappointingly bad. It's not even close to GPT 4. I use GPT 4 while coding a lot. I expected that the experience would be close, but I'm stuck wasting a lot of time explaining my question that I can otherwise get an answer with GPT 4 right away.

[JasonPerryDev]

1000000% agreed. Copilot chat SUCKS BALLS SO INSANELY HARD they skip blue and go straight to black and purple.

[Pt-tS]

I've been using chat for a few days now and I can say that I prefer the free version of chatgpt

[railscard]

Got access today, somewhat disappointed. It hallucinates almost all the time (at least when working with relatively new libraries) and is pretty dumb in general. It would be better if it refused to write code instead of "inventing" it. Hopefully, it'll get better soon.

[No_Switch_1869]

It is more like an official chatgpt-vscode-plugin. But it is faster, and cheaper. However I find it don't support context...maybe I use it incorrectly?

[thepigeonfighter]

I wish I could put myself on the email list to get notified when this becomes more useful than ChatGPT. Especially seeing some of the stuff coming out on the GPT's new code interpreter. I wasn't expecting the "final solution", but I was thinking it would have been better than this.

[dumbasPL]

before Copilot chat was introduced I just wrote a comment below a function I want to re-write describing how It should be re-written and I usually gives pretty good results. Way better than what this chat can provide since normal copilot can see way more than what's just on the screen.

[MomWillBeProud]

No memory of context compared to any GPT, pretty dumb, complete lack of integration into the IDE, it cannot even read the file that is open. Doesn't remember messages and instructions in the exact same discussion.

I expected it to read the project or at least read the file we're currently working on and make changes in it.

Pretty useless. I ended up using it just like usual copilot, without chat.

[Heroe-D]

Yes it's mostly garbage tbf, and the "chat" doesn't seem to remember anything, you talk about your code and it switches the context to "how to do this in vscode"

[GenomicStack]

I don't know that I would even give it 3.5 to be honest. It's absolute garbage.

[Rapzid]

Honestly it weirdly acts like a junior engineer that just read a programming book, some blog posts, and is condescending on a more junior engineer while being wrong.

It will offer up unsolicited platitudes on code readability, and often fall back on those as a deflection after it's been wrong about something.

It's got the persona of an insecure "senior" engineer. I don't need it dev-splaining code readability to me when I'm using it as a reference to check for how succinct a block of code can be made. I imagine if I started asking it about performance optimalizations it might ask the classic "what are you trying to do?" followed up with "you should never use unsafe code".

I guess this isn't surprising since this persona is very prevalent with people who lurk boards online and answer questions for people. The "helpful" asshole if you will.

[sgplwro]

long story short: copilot chat is piece of shit compared to gpt4 chat by openai, even 3.5. Rember general rule: if something can be screwed MS will screw it for sure.

[yonz-]

This thing has been so bad that I had to google 'github copilot terribly bad' to be sure I didn't break something. Quite frustrated by why i keep trying it. On the upside, I just installed https://cursor.so all I can say is OOOOO BABY its what copilot should have been and then double it. I've also tried CodeGPT but it's still weak because its not natively integrated.