I already had a cloud RAG pipeline running on Apify -- it chunks text, generates embeddings, and stores vectors in a cloud database. That works great for general content. But when I'm coding and want to search across all my repos, I don't want to push code to the cloud and wait for results. I want answers from my own codebase, instantly, without anything leaving my machine.
So I built a fully local version. It indexes all my GitHub repos, embeds them with a model running on my CPU, stores everything in a local vector database, and plugs directly into my editor through MCP. This post covers how it works and how you can build something similar.
What This System Does
In simple terms: you ask a question in plain English, and it finds the relevant code across all your repos.
For example, I can ask "how do I handle retries in my scrapers" and it returns the actual retry functions from my codebase -- with file paths, line numbers, and relevance scores. No keyword matching, no manual searching. It understands what you mean, not just what you typed.
The pipeline has four steps:
Clone your repos
-> Break code into meaningful chunks
-> Turn those chunks into vectors (numbers that capture meaning)
-> Store them in a searchable database
Then an MCP server sits on top and lets your AI assistant query it.
The Stack
Everything runs locally. Here's what you need:
| Component | What It Does |
|---|---|
| Ollama | Runs the embedding model on your machine (no cloud API) |
| nomic-embed-text | The embedding model -- turns text into 768-dimensional vectors |
| Qdrant | Vector database, runs as a Docker container |
| FastMCP | Exposes search tools to your editor via the Model Context Protocol |
GitHub CLI (gh) |
Lists and clones your repos automatically |
Total cost: $0. No API keys, no subscriptions, no data leaving your network.
How It's Different from My Cloud Pipeline
I wrote previously about building a RAG Content Chunker and a full RAG pipeline on Apify. Those tools split text into token-sized windows -- great for documentation and web pages, but not ideal for code.
The local version takes a fundamentally different approach to chunking. Instead of cutting code into fixed-size windows of 512 tokens, it actually understands code structure:
- Python files get parsed with Python's built-in AST module. Every function and class becomes its own chunk. Methods get qualified names like
MyClass.my_method. - JavaScript/TypeScript files get parsed with regex patterns that extract functions, arrow functions, and class declarations.
- Markdown files get split on headings. Each section is its own chunk.
- Config files (JSON, YAML, etc.) are stored as whole files.
Why does this matter? Because when the embedding model processes a complete function, it captures what that function does. When it processes the middle third of a class that got cut at a token boundary, it captures noise.
Setting It Up
1. Get Ollama Running
Ollama lets you run AI models locally. Install it and pull the embedding model:
curl -fsSL https://ollama.com/install.sh | sh
ollama pull nomic-embed-text
2. Start Qdrant
Qdrant is the vector database. Run it with Docker:
# docker-compose.yml
services:
qdrant:
image: qdrant/qdrant:latest
restart: unless-stopped
ports:
- "6333:6333"
volumes:
- qdrant_storage:/qdrant/storage
deploy:
resources:
limits:
memory: 2G
volumes:
qdrant_storage:
driver: local
docker compose up -d
3. Install Dependencies
The Python dependencies are minimal:
qdrant-client>=1.12.0
pydantic>=2.0.0
httpx>=0.27.0
mcp>=1.0.0
python-dotenv>=1.0.0
The Interesting Parts
Rather than walking through every file, here are the design decisions that actually matter.
AST Chunking for Python
The chunker uses Python's ast module to parse source files into a syntax tree, then walks that tree extracting every function and class:
import ast
tree = ast.parse(source_code)
for node in ast.walk(tree):
if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)):
# Extract this function as its own chunk
chunk_text = "\n".join(lines[node.lineno - 1 : node.end_lineno])
If a file has a syntax error (maybe it's a template or legacy code), it falls back to indexing the whole file as one chunk rather than silently dropping it.
Batch Embedding with Ollama
Ollama's /api/embed endpoint accepts a list of texts and returns all embeddings in one call. Sending 32 texts at once is dramatically faster than 32 individual calls -- especially on CPU:
response = client.post(
"http://localhost:11434/api/embed",
json={"model": "nomic-embed-text", "input": batch_of_texts},
)
embeddings = response.json()["embeddings"]
If a batch fails (Ollama occasionally chokes on weird content), the embedder retries with exponential backoff, then falls back to one-at-a-time as a last resort. One bad chunk doesn't kill the whole run.
Incremental Indexing
The system tracks the last indexed commit hash for each repo. When you re-run the indexer, it compares the current HEAD against what it last indexed. If nothing changed, it skips that repo entirely. State is saved after each repo, so if the process gets interrupted, it picks up where it left off:
if prev_commit == current_commit:
logger.info(f"[SKIP] {repo_name} - unchanged")
continue
First run across 27 repos takes about 10 minutes on CPU. Incremental runs where only a few repos changed finish in seconds.
Deterministic Chunk IDs
Each chunk gets a UUID5 generated from its repo name, file path, and line range. Same code at the same location always produces the same ID. This was a lesson from my cloud pipeline -- Qdrant requires UUIDs or unsigned integers as point IDs, and hex strings get rejected.
The MCP Server
This is where it all comes together. The MCP server exposes four search tools that any compatible editor or AI assistant can call:
| Tool | What It Does |
|---|---|
search_codebase |
Semantic search with optional repo/language/type filters |
get_repo_overview |
Structured summary of a specific repo |
find_similar_patterns |
Find similar functions and classes across repos |
list_indexed_repos |
Show all indexed repos and chunk counts |
When you ask a question, the server embeds your query using the same nomic-embed-text model, then searches Qdrant for the most similar chunks using cosine similarity. Results come back with the actual code, file path, line numbers, and a relevance score.
To connect it to your editor, point your MCP config at the server:
{
"mcp": {
"ratbyte-rag": {
"type": "local",
"command": ["python", "-m", "mcp_server.server"],
"enabled": true
}
}
}
That's it. Now you can ask things like:
- "How do I handle authentication across my scrapers?"
- "Show me how retry logic works in my projects"
- "What does the RatVault repo do?"
And get actual answers pulled from your codebase.
What I Learned
- AST chunking beats token-window chunking for code. When your chunks are actual functions instead of arbitrary text windows, the embeddings capture meaning instead of noise. Retrieval precision goes way up.
- Batch embedding matters on CPU. The difference between sending texts one at a time and batching 32 per call turned a 45-minute indexing run into a 10-minute one.
- Delete-then-insert beats surgical updates. When a repo changes, delete all its vectors and re-insert. Simpler than computing diffs, and avoids orphaned vectors from renamed files.
- MCP is the right interface. Exposing search as MCP tools means any compatible editor can use it. No REST API to maintain, no custom integrations, no auth to manage. Just tools over stdio.
What's Next
- Automatic re-indexing triggered by git hooks
- Hybrid search combining vector similarity with keyword matching
- A web UI for browsing the indexed codebase outside the editor
Research Resources
- Ollama - Run AI models locally
- nomic-embed-text - Open-source embedding model (768 dims, 8K context)
- Qdrant - Open-source vector database
- Model Context Protocol - Open standard for AI tool integration
- FastMCP (Python SDK) - Build MCP servers in Python
- Python ast module - Built-in syntax tree parser