I built an AI assistant that already knows everything I know
The problem with Claude, ChatGPT, and every other assistant: they don't know who you are.
Every session, you start over. You re-explain your stack. You restate your conventions. You paste in your context. And in the next session — nothing. Memory wiped.
I have 172 notes in my Brain vault. 46 work sessions with an AI agent. Years of documented decisions, validated patterns, snippets that work.
All of it was sitting dormant in Markdown files. The AI had no access to any of it.
The false problem
The instinctive reaction: fine-tune a model on this data.
That's the wrong solution. For two reasons.
A model fine-tuned on 172 notes learns surface-level patterns — it doesn't "memorize" facts. And on a 2B-parameter model (the only realistic option on an i5 with no GPU), fine-tuning makes things worse. Small models have a reasoning ceiling that more data doesn't move.
The real problem isn't the model. It's access to context.
RAG: giving memory to an LLM that has none
RAG — Retrieval-Augmented Generation. Before every query, we search a knowledge base for the relevant passages and inject them into the prompt.
The model doesn't memorize anything. It reads.
That distinction changes everything. Memorizing = learning statically at training time. Reading = dynamically accessing information at every query.
The Brain vault becomes a vector database. Each note is split into ~800-character chunks, encoded with all-MiniLM-L6-v2 (80 MB, CPU), stored in ChromaDB. When I ask a question, the proxy computes the embedding of my question, searches for the 8 nearest chunks (cosine distance ≤ 0.55), and injects them into the system prompt before sending it to Groq or Ollama.
The model sees my notes. Without ever having "learned" them.
The architecture
Open WebUI (localhost:3004) — Claude-like interface
↓
FastAPI Proxy (localhost:8080)
├── 1. RAG: query ChromaDB → Brain context injected
├── 2. Groq API (llama-3.3-70b-versatile, free)
└── 3. Fallback to Ollama → qwen3.5:2b on rate limit
ChromaDB (localhost:8001)
└── 1,891 chunks from ~/Brain/
Three Docker services. A Python indexing script with an inotify watcher. A systemd service that starts everything at login.
The interface is Open WebUI — open source, visually identical to ChatGPT, running entirely locally.
The fallback isn't optional
Groq's free tier is generous. But it has rate limits. When you hit a 429, the request has to go somewhere.
The proxy handles it in two lines of logic: if Groq returns 429, block Groq for 2 minutes and route every request to local Ollama. In streaming mode, if the 429 hits mid-generation, an invisible transition chunk gets injected and the response continues on Ollama. The user sees nothing.
The local model is a qwen3.5:2b. It's not llama-3.3-70b. It's less capable. But it answers at 10-14 tokens/second on CPU, and it knows the Brain context — which makes it more useful for my project-specific questions than any large model without that context.
What this actually changes
I ask a question about Pixelmart's architecture. The proxy retrieves the project notes, the roadmap, the Moneroo integration snippets. The model answers with real context — not an invented generality.
I ask how to center a modal without a Framer Motion conflict. It retrieves the exact snippet I documented after fixing that exact bug.
I mention a coding convention. It already knows I never write any in TypeScript, that my commits are in French, that I use Zod at system boundaries.
This isn't magic. It's vector search over my own notes.
The honest limits
The local model doesn't replace Claude. Not even GPT-4o-mini. On complex reasoning tasks, it hits a ceiling.
RAG isn't perfect. If the information you're looking for isn't in the 8 retrieved chunks, the model never sees it. Chunking by Markdown section sometimes misses important information buried deep in a long document.
The hardware caps the comfort level. An i5 with no GPU = 10-14 tok/s. Fine for a fluid conversation. Slow for generating 500 lines of code.
These limits are known. They don't invalidate the system. They define its scope of use.
The real value
A general-purpose AI asks you questions.
An AI that has read your notes starts giving answers.
The difference isn't technical. It's in the time you no longer spend giving context — and in answers that match your reality, not some average reality extrapolated from the internet.
The Brain vault was already useful as a personal archive. Now it's an external memory, queryable in natural language.