Building an AI Assistant From Scratch, No Cloud Required

I was talking to someone about the AI agents I had built in Microsoft Copilot Studio. They asked a simple question. They do not use Copilot Studio. They have no cloud infrastructure. But they do have their own data. Could I still build an AI solution for them?

I said yes. I knew the theory well enough to explain how it would work.

Then I actually built it, and learned more in a week than the theory had taught me in a year.

The biggest realisation came early. Copilot Studio is a visual way to configure things I could configure in code: instructions, knowledge sources, topics, actions. It is not doing anything magic. It is doing specific, nameable things, and hiding them behind a clean interface. Once I understood what those things were, the interface stopped being a black box and started being a shortcut.

This post is what those things actually are, explained the way I wish someone had explained them to me.

First, what “no cloud” really rules out

Less than most people assume.

It does not rule out AI. It rules out one delivery route. The real question is not cloud or no cloud. It is where the model runs.

Every AI product you have used sends your text to a company’s servers, where their model reads it and sends a reply back. That round trip is the thing a bank, a hospital, or a government department often cannot allow.

But the round trip is a business arrangement, not a law of physics. The model is a file. Put the file on your own machine and there is no round trip at all.

The private office

Here is the picture that made everything click for me.

Imagine a small private office inside your building. Nobody outside ever sees it or hears what is said in it. You hire exactly two employees.

A Filing Clerk. Slow, quiet, works in the back room, never speaks to visitors. Her only job is to read every document once and file it in a very particular way.

An Explainer. Fast, well spoken, sits at the front desk. He is excellent at reading a page and turning it into a clear answer. But he has never read a single one of your documents. He knows nothing about your policies.

Neither can do the other’s job. The Clerk cannot hold a conversation. The Explainer cannot find anything. Together, they can answer questions about your documents.

That is the whole system. Everything else is the filing cabinet and the house rules.

The Explainer is a file you download

The Explainer is a large language model, or LLM. It is not a service you connect to. It is a file sitting on a disk, a few gigabytes of numbers, that a program loads into memory and runs.

I used Qwen, an open-weight model published free for anyone to download. I picked it for four practical reasons: it is small enough to run on a laptop CPU with no graphics card, the licence allows commercial use, it handles Arabic as well as English, and swapping it for a different model later is a one line change.

To run it I used Ollama, which is not a model at all. Ollama is the office building. It is a program you install that loads model files and lets other programs talk to them over a local address.

The test that settles every security conversation takes ten seconds: download the model, unplug the internet, ask a question. It still answers. Nothing that works with the network cable pulled out is sending your data anywhere.

The four words worth actually understanding

A chunk is a small piece of a document, roughly 500 words. You cut documents up because a whole 20 page policy is too much for one search entry. Squeeze twenty topics into one and you get a vague average that matches nothing well. Small pieces stay sharp, and they let you cite an exact page.

An embedding is a list of numbers that represents meaning. Mine were 768 numbers per chunk. Think of it as a GPS coordinate, but for meaning instead of location. Text about similar topics gets coordinates close together. This is what lets a search for “how do I clean my hands” find a document that says “hand hygiene protocol,” even though not a single word matches. Keyword search fails there. Coordinate search does not.

Retrieval is the search step: turn the question into its own coordinate, then find the three or four chunks sitting closest to it.

RAG is the name for the whole pattern, Retrieval Augmented Generation. Retrieve the right chunks. Augment the prompt by pasting them in. Generate an answer from them. That is it. It sounded intimidating before I built it. It is two employees and a filing cabinet.

The house rule that makes it trustworthy

The Explainer is fluent, and fluent people sometimes fill gaps rather than admit they do not know. That is the real risk in this design, bigger than any privacy concern.

So he gets one instruction, attached to every question:

Answer using only the pages in front of you. If the answer is not there, say “I could not find this in the available documents.” Never guess.

I tested this by asking about a policy that existed in none of my documents. It refused, correctly, instead of inventing something plausible. That refusal is the most important thing the system does. A tool that confidently makes things up is worse than no tool at all.

What Copilot Studio was doing all along

This is the part that reframed everything for me. Every box in that interface maps to something specific underneath.

In Copilot Studio	What it actually is
Knowledge sources	Chunking, embedding, vector storage
Instructions	The system prompt
Topics	Routing logic
Actions and Power Automate flows	Function calling
The model behind it	The LLM itself
Publish to Teams	Your own frontend
Sign in	Your own authentication layer

Copilot Studio is not doing something you cannot do. It is assembling these pieces for you, connecting them to Microsoft’s cloud, and giving you a clean interface over the top. That is real value when the cloud is available. When it is not, you assemble the same pieces yourself.

Knowing this changed how I use the no-code tool too. I stopped guessing why an agent behaved strangely and started asking which underlying piece was misbehaving.

Three things that surprised me

The model was the easy part. Almost every quality problem traced back to the documents or the chunking, not the LLM. A PDF exported from a dashboard extracts as scattered numbers with no grammar, and no model can rescue that. The skills that mattered most were the ones I already had from years of data work: knowing whether source data is fit for purpose.

Refusing well matters more than answering well. Anyone can demo a correct answer. The trustworthy behaviour is admitting ignorance, and it has to be designed in deliberately.

Slow is often fine. On my laptop, answers took several seconds. That is a poor experience next to a cloud service, and a perfectly good one next to no AI at all.

Build it yourself

The whole thing runs on a normal laptop. No graphics card, no cloud account, no API key, no cost.

The complete project is on GitHub, with the documents, the code, and setup instructions: github.com/dataworksjouhar/learnedge-assistant

The core of it, ingestion and the API, is a little over 200 lines of Python. That number surprised me more than anything else.

Next in this series

This post was the tidy version. The real build was messier, and the mess is where the interesting parts are.

In Part 2 I will cover what happened when I stress tested it: the real numbers from running an LLM on a weak CPU, and a retrieval bug that produced a completely correct looking answer while quietly failing underneath, which I only caught by checking the numbers behind the answer instead of the answer itself.

← Back to Notes