What is actually inside a Copilot Studio agent
I built AI agents in Copilot Studio without ever seeing what was underneath. So I took one apart, piece by piece, to understand what each box in that interface really does.
I built AI agents in Microsoft Copilot Studio for a while. They worked, and people used them. But four things kept bothering me, and I could never quite explain any of them.
I could not control how my documents were split up before the agent searched them. I could not see why it gave the answer it gave. When it chose a passage, I could not tell why it picked that one when a clearer passage was sitting right there in the same documents. And none of it could run anywhere except Microsoft’s cloud.
For a long time that did not matter much. Then someone asked me a simple question.
They did not use Copilot Studio. They had no cloud infrastructure at all. But they had their own documents, and they wanted to know whether I could still build them an AI assistant.
I said yes. Then I realised I could not honestly explain what I would be building, because I had never seen underneath the interface I had been using.
So I went and found out.
Every box in that interface is a real thing
The first surprise was that nothing in Copilot Studio is mysterious. Each part of the interface stands for something specific, and once you can name it, you can understand it.
| In Copilot Studio | What it does | Technical term |
|---|---|---|
| Knowledge sources | Cuts your documents into pieces, turns each piece into numbers that represent its meaning, and stores them so they can be searched | Chunking, embeddings, vector database |
| Instructions | A block of text quietly added to every question | System prompt |
| Topics | Decides which path a question should take | Routing, intent detection |
| Actions and Power Automate flows | Lets the model ask your own code to go and do something | Function calling, tool use |
| The model behind it | The part that reads and writes, a file of numbers | Large language model (LLM) |
| Publish to Teams | Where people actually type their questions | Frontend, channel |
| Sign in | Checks who is asking | Authentication |
Copilot Studio assembles these for you and connects them to Microsoft’s cloud. That is real value, and for most people it is the right trade. But it also means every one of my four frustrations was hiding inside one of those rows.
The rest of this post is about the first row, knowledge sources, because that is where three of the four were.
Two employees in a private office
The picture that made it click for me is a small private office inside your own building. Nobody outside sees it. You hire exactly two people.
A Filing Clerk. She works quietly in the back room and never speaks to visitors. Her only job is to read every document once and file it in a particular way.
An Explainer. He sits at the front desk and is excellent at reading a page and turning it into a clear answer. But he has never read a single one of your documents. On his own, he knows nothing about your policies.
Neither can do the other’s job. The Clerk cannot hold a conversation, and the Explainer cannot find anything. Together they can answer questions about your documents.
That is the whole system. What follows is how the Clerk files things, and how the two of them work together.
Chunk: cutting the documents into pieces
Before the Clerk can file anything, the documents have to be cut into smaller pieces. Each piece is called a chunk.
You cut them up because a whole policy is too much for one search entry. Squeeze twenty topics into one entry and you get a vague average that matches nothing well. Small pieces, one topic each, stay sharp, and they let you point to the exact place an answer came from.
But the size matters in both directions. Cut too large and unrelated topics blur together. Cut too small and a piece loses the context that gave it meaning, so a sentence ends up on its own without the paragraph that explained it.
This was my first frustration. In Copilot Studio I could not find a way to decide where those cuts happened, and they decide a great deal about what the agent can later find.
Embedding: giving every piece a position
Next the Clerk turns each piece into a long list of numbers. That list is called an embedding, and it represents what the piece means.
The easiest way to think about it is a map. The numbers work like coordinates, except they describe meaning instead of location. Pieces about similar topics end up close together. Pieces about different topics end up far apart.
This is what lets the system find a policy about hand hygiene when the question talks about washing before touching a patient, even though the two share no words at all. An ordinary keyword search would fail there. A search by position on this map does not.
Retrieval: finding the nearest pieces
When a question comes in, it gets turned into numbers in exactly the same way, so it lands on the same map. Then the system simply takes the pieces sitting closest to it. Usually three or four. That step is called retrieval.
This is the answer to my third frustration, and it was the most useful thing I learned.
Retrieval picks what is nearest in meaning. It does not pick what answers the question best. Those are usually the same thing, but not always.
A piece that uses the same kind of language as the question can sit closer than a piece that actually contains the answer but phrases it differently. So the system can hand over three passages that are about roughly the right subject and miss the one passage that settles it, even when that passage is sitting just outside the circle.
That is exactly what I had been seeing in Copilot Studio and could not explain. It was not the agent being careless. It was the distance on the map, and I had no way to see the map.
RAG: the whole pattern
Put those pieces together and you have the pattern behind nearly every document assistant, including the knowledge sources in Copilot Studio. It is called RAG, which stands for Retrieval Augmented Generation.
Retrieve the nearest pieces. Augment the question by pasting those pieces into it. Generate an answer from them.
The house rule matters as much as anything else in the diagram. The Explainer is fluent, and fluent people sometimes fill a gap rather than admit they do not know. So he gets one instruction with every question: answer only from the pieces in front of you, and if the answer is not there, say so.
This is where my second frustration lived. The answer can only ever be as good as the pieces the Clerk retrieved. If you cannot see which pieces those were, you cannot see why the answer came out the way it did. You are looking at the end of the process with no view of the middle.
And the fourth one: the cloud
The last frustration turned out to be the simplest.
The Explainer, the language model, is not really a service you have to connect to. It is a file, a few gigabytes of numbers, that a program loads and runs. Put that file on your own machine and there is nothing travelling to anyone else’s server.
Which meant the answer to the person who asked me was not “you need the cloud”. It was “you need the pieces, and you can run all of them yourself”.
So I am going to build one
Knowing the theory is one thing. I wanted to know whether I actually understood it.
So I am going to build a document assistant from scratch, on an ordinary laptop, with no cloud and no graphics card. Every piece in this post, built by hand, where I can see what happens at each step.
If the theory is right, it should work with the network cable pulled out.
Update: I built it. It runs completely offline, answers in English and Arabic, shows the exact paragraph behind every answer, and taught me a great deal that this post could not. The full write-up, with a video of it running: An offline AI that answers from your own documents.
← Back to Notes