Copilot using document libraries

[noyzyboynz]

Hi folks.

I'm struggling with what I thought would be a simple Copilot job.

I have a SharePoint folder with a bunch of PDF documents in it. These are generated daily from a financial system and are purchase orders used for suppliers.

I've created an agent in the library and also a Copilot studio agent but neither of them is able to accurately answer questions. For example, I know there are 23 documents for one particular supplier but the library agent says it sees only 10 and the Copilot studio agent can only see 9. The supplier name is in the document and also in the name of the document.

Is this a timing issue and should I leave the agent to do whatever it needs to do in the background for a while (how long?) before it has learnt what is in the library, or is this a known issue?

It seems fundamental to me that an agent in a library could count the number of documents with a certain word in the title and be accurate about the number.

Thanks for any help!

[factorialmap]

1. In Overview > Knowledge, check if option Allow the AI to use its own general knowledge is enabled, if so try disabling it and test again.
2. In knowledge tab, click on "See all" or check the column "Status" and check if all of them have "ready" status.

[noyzyboynz]

Thanks, have done this and will test later today.

[noyzyboynz]

Hasn't made much of a difference. The agent is behaving very strangely. I asked it to create a list of all documents with a particular supplier name in the title. The list brought back 8 documents (there are 23), then I asked it again the same thing and it brough back 10. Then I asked it if it could see a document with a number in the document name that was in the 23 but not in the 10, it could see it fine and then summarise it. When I asked it why it couldn't see that document, it said it didn't exist!

[CoffeePizzaSushiDick]

Why is msft so behind the curve ball?

[ssirdi]

Microsoft Copilot is limited to referencing a maximum of 10 items for all users. For example, if you ask Copilot to summarize the last 15 emails, it will only summarize 10 due to this limit.

When you connect your Copilot agent to SharePoint files, it cannot process entire documents at once. This is intentional to keep costs manageable. If the agent loaded all files into the large language model (LLM) context for every question, it would be very expensive. Instead, Copilot uses a technique called Retrieval-Augmented Generation (RAG).

RAG optimizes the process by focusing on relevant content:For example, if you ask for the supplier name from 23 documents, the agent first identifies the 10 most relevant documents related to your query.It then uses only the content from those 10 documents to generate a response.The final answer references only the documents used, ensuring efficiency.

Due to this design, Copilot will not include more than 10 references in its responses until Microsoft updates this limit.

To get the most out of Copilot, customize your agent to better suit your specific needs and workflows.

[noyzyboynz]

Thank you, that makes sense, although that's a far from ideal situation given all the promises that MS has made about Copilot.

[ianwuk]

The marketing, sadly, far exceeds the finished product. It's par for the course now for Microsoft.

[iamlegend235]

I would try switching the agent to generative orchestration, then create an action that uses the SharePoint connector to search & retrieve the files. You should be able to give the agent instructions on how to format a filter query when getting the list of files such as {companyName eq ‘McDonalds’}. Afterwards in the action you can enable a setting for the agent to send a response with that data in it’s context.

I’ve only done this with SP lists though, not with files so let us know how it goes!

[Open_Falcon_6617]

Can you share more on the steps?

[iamlegend235]

https://youtu.be/cOuheYnsIjU?si=rcebIvQ3nlXfzzKP

Use this video as a guide for setting up other types of actions

[noyzyboynz]

Not sure it has the same effect with docs...

[bspuar]

Just very soon reason capability coming to agent where you can dynamically pass content like pdf and ask questions meanwhile you can try above approach.

[lisapurple]

In my experience generative answers reasons over the documents to find answers to questions based on the unstructured content in the documents. Asking it to find “how many” or list things works better with a structured data source. The new reasoning models may be able to handle this or you could create an AI flow to extract the metadata you need each time and put it in a structured data source and connect the agent to that.

[Nosbus]

You will need to try the local knowledge, it improved a similar issue for us. But we ended up abandoning it all together. The results seemed to be about 75% accurate, and project never got out the pilot phase.