AI · Networks & servers

Private AI at home: an agent that decides what goes to the cloud

An AI agent in my homelab that decides, based on the content of each message, whether a local or a cloud model processes it. Confidential content never leaves the network.

Year: 2026
Status: In daily use
Role: Architecture, development and infrastructure
Repository: Repository coming soon

01 · Problem

I wanted to use an AI agent (Hermes) from my phone without my private documents ending up in third-party services. The model gateway I use (Bifrost) cannot route by message content: its rules never see the prompt.

02 · Approach

I built a “router shim” in FastAPI that sits between the agent and the gateway: it reads each message, applies configurable rules (keywords → model, reasoning effort and local or cloud) and, when in doubt, routes locally (fail-closed). The agent, the gateway and the shim run in Docker inside a Proxmox LXC; local models run on Ollama.

03 · Result

Confidential messages are handled by local models on Ollama and the rest by cloud models. There is a web panel to edit the rules, an MCP server to change them by chatting with the agent, another one to manage Proxmox and DNS filtering without exposing credentials to the model, and voice-note transcription with faster-whisper inside the local network.

04 · Stack

Python
FastAPI
Docker
Proxmox (LXC)
Ollama
MCP
faster-whisper
Telegram

The flow

Me (Telegram, phone)
   │
   ▼
Hermes (agent) ──► router shim (FastAPI) ──┬──► Ollama  · local models  [confidential]
                                           └──► Bifrost · cloud models  [regular]

Golden rule: confidential always wins. If a message matches a privacy rule —or the shim fails— it is processed locally. Nothing leaks by accident.

Why the shim was needed

The first attempt was to write routing rules directly in the gateway with conditions on the text. It did not work: according to the official docs, the gateway’s rule engine only sees the model, provider, headers and parameters, not the message content. The keyword rules were invalid and confidential prompts went to the cloud. The shim is the only piece that reads the prompt, so it is mandatory.

What makes it useful

Rules without code. A web panel edits rules (keywords → model + effort + local/cloud) with a test box and the live list of available models.
Configured by chatting. An MCP server with 6 tools lets the agent list, create, edit, delete and test rules.
Homelab without exposing keys. Another MCP server manages Proxmox (overview, SSD health, containers) and DNS filtering; credentials live in the container and the model never sees them.
Local voice. An OpenAI-compatible transcription server runs faster-whisper on CPU: on my PC, the small model transcribes 7 s of audio in about 5 s. Audio never leaves the house.

Takeaway

Before writing code, I checked against the documentation what each piece could and could not do. That explained a bug that looked like a configuration issue and changed the whole design.