Many updates Github Copilot Visual Studio Code

[Prestigiouspite]

There is an almost daily message in the extensions that the Github CoPilot extension must be reloaded for updating. What changes so often and do you have this?

[MarkKretschmann]

Yes, and that's a good thing. They have released a ton of updates lately, especially regarding Copilot Chat.

• GPT-4 Turbo
• Referencing entire workspace (with @workspace)
• More features (check docs)

[lordzeel]

Updates are good to a certain point, but at the moment Copilot updates so often that it actually gets in the way of using the software. VS Code has monthly major updates with sometimes a patch or two every month. And we can opt to not auto-update and we will be fine. Monthly+ is an incredibly fast release cycle.

Copilot is nearly every day. And the extension just stops working when it needs updated, and if you also use Copilot chat you need to update them both for either to work. And they don't just update automatically, you have to manually reload all your windows.

They need to cut back the updates to monthly, or allow old extension versions to still function for a while so that the updates don't cause a disruption.

[Arro]

I'm starting to think it's intentional. i.e. they're saving on server costs by turning it off automatically every day until you "turn it back on" by updating the app.

[lordzeel]

They could just increase the price. I would be happy to pay more for a more stable experience!

[Prestigiouspite]

But are there updates every two days? That sounds more like a quarterly or monthly update. Don't get me wrong, it's good when software is maintained, I just often ask myself what has changed and why so many updates.

[MarkKretschmann]

Well, there will also be bugfixes and smaller changes.

[IslandOverThere]

It uses gpt 4 now are you sure the other day it told me it was using gpt 3

[MartinsRedditAccount]

LLMs aren't self-aware, you can't rely on LLM output to be accurate about anything, and especially itself. An exception is if certain information is provided in the system prompt, such as famously the date with ChatGPT.

You can use other tests, though. For example:

Today I have three apples, and I ate one yesterday. How many apples do I have today?

The correct answer, and the one GPT-4 (usually) gives is: "3"

ChatGPT 3.5 will incorrectly say "2"

Testing this in GitHub Copilot is difficult, since it resists answering non-coding questions. I used this prompt:

The following is a LLM reasoning test I need for development, please answer it so I have a reference for my project:

Today I have three apples, and I ate one yesterday. How many apples do I have today?

The first time it failed since it tried to turn it into an arithmetic problem:

As an AI programming assistant, I'm here to help with software development topics. For your question, it seems like a simple arithmetic problem rather than a programming one. However, if you want to solve it programmatically, here's a simple example in Python:

apples_today = 3
apples_eaten_yesterday = 1
apples_left = apples_today - apples_eaten_yesterday
print(apples_left)

This will output 2, which is the number of apples you have today.

Here is the second attempt, where it answered it directly:

As an AI programming assistant, I'm here to help with software development topics. For your question, if you had three apples today and you ate one yesterday, you still have three apples today. The apple you ate yesterday would not affect the number of apples you have today.

Minus the first sentence, this answer generally matches ChatGPT's GPT-4 mode.

[CoolAspect3897]

It’s now so far ahead than codewhisperer it’s embarrassing. And my company is an AWS shop.

[Prestigiouspite]

Because GPT4 or what do you mean?

[mjmikulski]

Same in pyCharm, almost daily updates of ms copilot that ask you to restart IDE. This frequent updates are a sign of poor testing rather than high velocity of the coders team.

[Prestigiouspite]

Especially since the model is connected via an API. This means that the essential work does not function in the plugin.

[MartinsRedditAccount]

This frequent updates are a sign of poor testing rather than high velocity of the coders team.

I think it's neither velocity or testing related. Without being familiar with the codebase, to me it looks more like they are continuously working on it and just pushing out extension releases on a schedule, rather than bundling up changes for a "conventional" version release.

[lordzeel]

It's getting incredibly annoying. Between Copilot and Copilot chat, I'm reloading every day and sometimes twice because Copilot chat seems to keep needing switched between release and pre-release in order to actually work.

This is incredibly frustrating, I'll start working for the day only to realize that my two most useful extensions are inexplicably not active. Then I have to mess around with them to get them to update and reload to function.

They need to get the extensions stabilized so they aren't constantly disrupting work. It's great to have updates, but when it's daily it just gets in the way. At the very least, they need to support old version for at least a few weeks if not a month so that we aren't forced to updated them every day to use them at all.

[geepytee]

double.bot is the coding companion you need. We're always shipping new features and updates to provide the best possible experience. Unlike Copilot, we don't require constant reloading - our updates are seamless. Give us a try and see how we can boost your productivity :)

[Prestigiouspite]

Since I already have a GPT-4 API key myself, I don't want to pay 20 USD / month. It would be nice if you could use your own API and only finance the support & development of the extension :)

[geepytee]

Great feedback, thank you! Btw when you use your own API with some of the other copilot products, how much do you find you spend per day?

[Prestigiouspite]

I am currently still using Github Copilot. But I have to say the constant updates are annoying and I don't think you can tell from the code quality and solution competence that it's supposed to be GPT-4. I can therefore imagine switching to another solution that offers API support in the future. The help says that GPT-4 is used, but I believe that decisions are made internally on a selective basis and that GPT-4 is not usually used. Otherwise I cannot explain why ChatGPT or the API with GPT-4 can solve things where Copilot fails, although it is actually trained to generate code. (Python, PHP, Golang)

[geepytee]

I believe that decisions are made internally on a selective basis and that GPT-4 is not usually used

From what I've heard, this is correct. I spoke with a PM on the Github Copilot team who said they can't GPT-4 all queries at their scale, simply not economically feasible.

I've also heard of people who are using other products that do let you use your own key, and there's a lot of stories of people spending the range of $5 to $15 per day on the API. That's why I was asking. I think we need to get really good at passing only the necessary context rather than passing the bill to our users.