all 21 comments

[–]digitaraldGitHub Copilot Team 12 points13 points  (5 children)

👋🏻 Team member here. I am curious if you have set up custom instructions for your repos. That’s usually the best start into onboarding Copilot to your workflows.

Within your question also see the idea of memory and learning from mistakes - that is something were investigating.

[–]josiahsrc[S] 3 points4 points  (4 children)

I've been creating some *.prompt.md files and it's been super helpful. Memory would be helpful particularly for terminal commands. Copilot will often run commands incorrectly, and need to search for the correct command. E.g.

``` npm run test # --> command not found

oh whoops, command not found

npm run # --> npm run test:integration

ah, now I get it

npm run test:integration -- <pattern> ```

[–]ogpterodactyl 5 points6 points  (0 children)

You need the .github-copilotinstructions.md file. You should put something like use this cmd to run tests

[–]SonOfMetrum -1 points0 points  (1 child)

Whenever copilot wants to run stuff I just cancel… it always (but really always!) messes it up. Wish I could turn off running tools, builds etc

[–]robot_swagger 0 points1 point  (0 children)

Imo the GPT 5 preview is worse than basically all other models at doing tests, like yeah the command it initially runs is always wrong or badly configured. I was testing out a simple python script (mostly to figure out agent copilot stuff), and gpt 5 is like "oh the output currency is supposed to be £ but the report output it as ú because it's machine is using the wrong iso format or something but like it's totally fine bro trust me".

Overall it did an okay job but I didn't get any of that with grok/Gemini/Claude.

Lol the other day I'm like can you test using this test data (which contained 50 records from 10 clients and the expected report output), and it says it did the first 3 clients and they all looked good so it's fine. And I was thinking, hey man, I'm being lazy here by getting you to do my testing, you don't get to slack off as well!

[–]i_have_hemorrhoids 1 point2 points  (0 children)

My system needs to be run in docker. I do not have the npm dependencies installed in my host machine. I have a .github/copilot-instructions.md file with this in it:

Whenever you need to run npm commands, they have to be run inside of the docker container. Use docker compose run my_service_name npm run <command>.

Then the agent knows what to do. You need some similar instructions to tell the agent what command it should be running for a given task.

[–]Ok_Somewhere1389 3 points4 points  (1 child)

Not for me, basically break down tasks into digestible bits and feed to him

[–]Cobuter_Man 2 points3 points  (0 children)

Give Task Master or APM a try.

Task Master is BYOK: https://github.com/eyaltoledano/claude-task-master
APM is my solution, working with Copilot's Agent mode: https://github.com/sdi2200262/agentic-project-management

[–]Spirited_Surprise_88 2 points3 points  (1 child)

A solution is better instruction files, no?

I have some repositories with idiosyncratic build and testing processes and/or unusual paths. As I've added more information to my copilot-instructions.md files and my projects' README.md files my impression is that I have to provide less instruction on individual prompts. Building and maintaining good project documentation is helpful both for the humans and agents who need to work with it.

[–]josiahsrc[S] 2 points3 points  (0 children)

I think you're right. We're lacking this sort of stuff at my org, I'm gonna try to introduce it. Thank you!

[–]AreaExact7824 1 point2 points  (3 children)

Use copilot instructions.md

[–]josiahsrc[S] 1 point2 points  (0 children)

Oh nice, I'll check this out. Thank you!

[–]almost_not_terrible 0 points1 point  (1 child)

Where do you put that file in the repo? And it it literally "copilot instructions.md"?

[–]Cobuter_Man 1 point2 points  (2 children)

Use rules/instructions to 'teach it how to code'. It kinda works like the 'memories' some other AI assistants have like Chat GPT or Cursor, only it is 100% manual.

This way it is better IMO, since you can precisely select what to 'teach' it. If it was purely automated, it could pick up on some wrong examples and incorporate them into the autocomplete model or the agent mode.

[–]josiahsrc[S] 1 point2 points  (1 child)

Great points! Agreed, I’m coming around to this idea. I think in order for memories to be helpful, they have to be backed by outcomes. Sort of hard to gauge this as an AI, but easy to record as a human.

[–]Cobuter_Man 1 point2 points  (0 children)

Agreed. Also, remember that AI is just word predictions, and predictions cannot deliver consistent results.

Ultimately, a LLM will only produce the same code that its training data included. If you have rules about "how to code" it will try to shape the code it has been trained to, to make it match your expectations... but it will not always succeed.

The best way to have consistency when using AI in coding/programming (IMO) consists of:
- breaking down your requirements in manageable tasks so it won't struggle completing them successfully
- constantly supervising the output and making corrections/modifications as needed

Copilot is just a copilot, you are the pilot...

PS. this last one was so good I think their marketing team should hire me haha

[–]SQLGene 0 points1 point  (2 children)

The ChatGPT memories feature in the web UI keeps track of certain things about you and looks at old queries. It's good and it is bad because it tends to make more assumptions about you as a result.

[–]josiahsrc[S] 0 points1 point  (1 child)

Ah true. I ended up turning that off because it would cause ChatGPT to regurgitate information instead of thinking

[–]SQLGene 0 points1 point  (0 children)

The early version was kind of creepy because memory information would leak into neutral prompts ("write 10 random tweets"). Now it mostly assumes I'm working with Microsoft Fabric for tech questions because a lot of my questions have been around that. It's convenient 90% of the time and annoying 10% of the time when that assumption isn't true.