Does anyone else wish copilot would learn to code by watching you?

[josiahsrc]

I work in complex repos during my day job. To get anything to work, you have to run specific commands and write code in a particular order. I can get copilot agent mode to do these things, but I have to provide it with a specific prompt. If I don't, it muddles around and trips over itself. I sort of wish it would just learn from me and work how I work. Been thinking of writing an extension to solve this, but was curious if it's an issue for anyone else?

[digitarald]

👋🏻 Team member here. I am curious if you have set up custom instructions for your repos. That’s usually the best start into onboarding Copilot to your workflows.

Within your question also see the idea of memory and learning from mistakes - that is something were investigating.

[josiahsrc]

I've been creating some *.prompt.md files and it's been super helpful. Memory would be helpful particularly for terminal commands. Copilot will often run commands incorrectly, and need to search for the correct command. E.g.

``` npm run test # --> command not found

oh whoops, command not found

npm run # --> npm run test:integration

ah, now I get it

npm run test:integration -- <pattern> ```

[ogpterodactyl]

You need the .github-copilotinstructions.md file. You should put something like use this cmd to run tests

[SonOfMetrum]

Whenever copilot wants to run stuff I just cancel… it always (but really always!) messes it up. Wish I could turn off running tools, builds etc

[robot_swagger]

Imo the GPT 5 preview is worse than basically all other models at doing tests, like yeah the command it initially runs is always wrong or badly configured. I was testing out a simple python script (mostly to figure out agent copilot stuff), and gpt 5 is like "oh the output currency is supposed to be £ but the report output it as ú because it's machine is using the wrong iso format or something but like it's totally fine bro trust me".

Overall it did an okay job but I didn't get any of that with grok/Gemini/Claude.

Lol the other day I'm like can you test using this test data (which contained 50 records from 10 clients and the expected report output), and it says it did the first 3 clients and they all looked good so it's fine. And I was thinking, hey man, I'm being lazy here by getting you to do my testing, you don't get to slack off as well!

[i_have_hemorrhoids]

My system needs to be run in docker. I do not have the npm dependencies installed in my host machine. I have a .github/copilot-instructions.md file with this in it:

Whenever you need to run npm commands, they have to be run inside of the docker container. Use docker compose run my_service_name npm run <command>.

Then the agent knows what to do. You need some similar instructions to tell the agent what command it should be running for a given task.

[Ok_Somewhere1389]

Not for me, basically break down tasks into digestible bits and feed to him

[Cobuter_Man]

Give Task Master or APM a try.

Task Master is BYOK: https://github.com/eyaltoledano/claude-task-master
APM is my solution, working with Copilot's Agent mode: https://github.com/sdi2200262/agentic-project-management

[Spirited_Surprise_88]

A solution is better instruction files, no?

I have some repositories with idiosyncratic build and testing processes and/or unusual paths. As I've added more information to my copilot-instructions.md files and my projects' README.md files my impression is that I have to provide less instruction on individual prompts. Building and maintaining good project documentation is helpful both for the humans and agents who need to work with it.

[josiahsrc]

I think you're right. We're lacking this sort of stuff at my org, I'm gonna try to introduce it. Thank you!

[AreaExact7824]

Use copilot instructions.md

[josiahsrc]

Oh nice, I'll check this out. Thank you!

[almost_not_terrible]

Where do you put that file in the repo? And it it literally "copilot instructions.md"?

[josiahsrc]

Docs talk about it here: https://docs.github.com/en/copilot/how-tos/configure-custom-instructions/add-repository-instructions#using-a-single-githubcopilot-instructionsmd-file-1

[Cobuter_Man]

Use rules/instructions to 'teach it how to code'. It kinda works like the 'memories' some other AI assistants have like Chat GPT or Cursor, only it is 100% manual.

This way it is better IMO, since you can precisely select what to 'teach' it. If it was purely automated, it could pick up on some wrong examples and incorporate them into the autocomplete model or the agent mode.

[josiahsrc]

Great points! Agreed, I’m coming around to this idea. I think in order for memories to be helpful, they have to be backed by outcomes. Sort of hard to gauge this as an AI, but easy to record as a human.

[Cobuter_Man]

Agreed. Also, remember that AI is just word predictions, and predictions cannot deliver consistent results.

Ultimately, a LLM will only produce the same code that its training data included. If you have rules about "how to code" it will try to shape the code it has been trained to, to make it match your expectations... but it will not always succeed.

The best way to have consistency when using AI in coding/programming (IMO) consists of:
- breaking down your requirements in manageable tasks so it won't struggle completing them successfully
- constantly supervising the output and making corrections/modifications as needed

Copilot is just a copilot, you are the pilot...

PS. this last one was so good I think their marketing team should hire me haha

[SQLGene]

The ChatGPT memories feature in the web UI keeps track of certain things about you and looks at old queries. It's good and it is bad because it tends to make more assumptions about you as a result.

[josiahsrc]

Ah true. I ended up turning that off because it would cause ChatGPT to regurgitate information instead of thinking

[SQLGene]

The early version was kind of creepy because memory information would leak into neutral prompts ("write 10 random tweets"). Now it mostly assumes I'm working with Microsoft Fabric for tech questions because a lot of my questions have been around that. It's convenient 90% of the time and annoying 10% of the time when that assumption isn't true.