Three degrees of local: what "self-hosted" actually means in a meeting assistant

So here is the precise version: locality in this system is a spectrum with three positions, and you choose your position per component, in the admin panel. This post walks the pipeline and names exactly which bytes leave your machine at each position. Everything below is verifiable in the repository; I cite file paths instead of adjectives.

The pipeline, end to end

The browser captures microphone audio with getUserMedia, converts it to PCM16 16 kHz mono, and streams it over a WebSocket as binary frames with a 1-byte track prefix -- 0x00 for mic, 0x01 for tab/system audio (frontend/src/hooks/useAudioCapture.ts, backend/app/ws/audio_handler.py). Capturing the two tracks separately matters later: remote participants arrive on the system track and get their own speaker identities, so "who said what" survives even though everyone is coming out of one pair of speakers.

From there, four stages: voice activity detection and diarization, transcription, live interim captions, and the LLM agent layer. Each stage has a different locality story, and being honest about the seams is the point of this post.

Stage 1: diarization is always local, in every configuration

Speech segmentation and speaker attribution never touch a network. Silero VAD (ONNX) finds speech at a 0.6 threshold, segments run 750 ms to 15 s with a 600 ms silence gap, and each segment gets a speaker embedding from a WeSpeaker ResNet152-LM model, also ONNX, with Kaldi fbank features computed by kaldi-native-fbank with mean-only cepstral normalization -- matching the frontend the model was trained with (backend/app/services/speaker_diarizer.py, backend/app/config.py). Embeddings are matched against per-track speaker profiles at a 0.68 cosine-similarity threshold.

There is no "diarization API" in this system to route audio through. This was a deliberate constraint: the biometric part of the pipeline -- voice embeddings, the part that is literally a model of the people speaking -- runs on your hardware in all three configurations, full stop.

Call audio is also written to disk locally as per-segment WAV (backend/app/services/audio_store.py), which enables a feature I have not seen elsewhere in this category: POST /api/sessions/{id}/retranscribe replays any stored call through a different transcription model later, so you can test a new local model against your own real audio instead of against a benchmark.

Stage 2: transcription -- degree one and degree two

This is the first fork. Batch transcription routes through a factory (backend/app/services/local_transcriber.py): model ids beginning with local- run ONNX Whisper or Parakeet on your machine via onnx-asr, with weights downloaded once to the data directory; anything else wraps the diarized segment as WAV and sends it to Gemini on your API key.

Degree one, the default: local diarization, cloud transcription. Audio segments leave your machine, but to a provider you chose, on a key you hold, under that provider's API terms -- not to a meeting-AI vendor with its own retention and training policies layered on top.

Degree two: local transcription. No audio leaves the machine at all. The tradeoff is real and I will not launder it: hosted frontier models transcribe difficult audio -- crosstalk, accents, jargon -- better than compact local ONNX models. That is exactly what the re-transcription endpoint is for: run a meeting both ways and read the diff before you decide what your accuracy budget is.

Live interim captions follow the same fork. The default is a Gemini Live session acting as a silent listener that relays interim text (backend/app/services/gemini_live.py). Set the audio gateway model to local-parakeet-live and captions come from an on-device Parakeet captioner instead (backend/app/services/local_live_captioner.py) -- no cloud, works under Privacy First. It is experimental and CPU-heavy, and the app ships a fit test that projects whether your machine can sustain it, because "it works on my machine" is not a deployment plan.

Stage 3: the LLM agents -- degree three

The analysis layer is a crew of agents -- an analyst, a low-latency objection handler, a synthesizer, an opportunity specialist, a strategic signals agent, and a three-lens post-call briefing -- each with an editable model, prompt, and trigger. Agents consume transcript text, not audio. So in degree two, what leaves your machine is text, on your key.

Degree three closes that last seam. The backend speaks to any OpenAI-compatible server: register an endpoint (LM Studio, Ollama, vLLM, LiteLLM -- there are presets), it lists its models, and every served model becomes a first-class entry in the model registry with the id endpoint:<slug>:<model>, selectable in every model picker next to the cloud options (backend/app/services/custom_endpoints.py, backend/app/services/llm_endpoint.py). Point the agents at a model served on your own box or your LAN, use local ONNX transcription, and the entire pipeline -- capture, diarization, transcription, live captions, agents -- runs with no API key from anyone and no packet leaving your network.

Privacy First mode is the enforcement mechanism rather than a promise: it locks every model picker to models flagged as running locally, and it admits self-hosted endpoints on loopback, private-network addresses, or LAN hostnames (frontend/src/lib/modelOptions.ts). Flip it on and a cloud model is not a worse choice; it is not a choice.

What I am not claiming

Honesty about the boundary is worth more than the boundary itself, so:

• Degree one sends audio to your model provider, and degrees one and two send transcript text to it. If your threat model includes your own API provider, only degree three answers it.
• Local model quality is your problem to evaluate. A 7B instruct model will not write the same objection analysis a frontier model writes. The per-agent model picker exists precisely so you can spend cloud budget where it matters and keep the rest local -- but I will not pretend the outputs are interchangeable.
• There are no user accounts or authentication in the application. It is a single-operator tool designed to sit behind your own network boundary, not on the public internet. Provider keys are encrypted at rest (Fernet, with a master key in the data directory), but access control is your reverse proxy's job, not the app's.
• There are no compliance badges. No SOC 2, no ISO, no HIPAA. What I can offer instead is an MIT-licensed codebase where every claim in this post is a file path you can read. For some reviewers that is stronger than an attestation; for procurement it is usually weaker. Know which reviewer you are.

Why the spectrum, instead of just shipping degree three

Because the honest engineering answer is that the degrees trade accuracy against locality, and that trade belongs to you, not me. A sales team that wants maximum insight quality with audio kept on-premises runs degree two. A therapist, a lawyer, or anyone whose transcripts are the sensitive artifact runs degree three and accepts local-model output. A homelab user benchmarking Parakeet against Gemini on their own standup recordings runs all three in an afternoon through the re-transcription endpoint.

A hosted product has to pick one position on this spectrum for all of its customers, and its incentives push it cloudward. A self-hosted one can hand you the slider. That -- not a bot-free gimmick, not a compliance page -- is what "self-hosted" actually buys you.

The code is at github.com/talberthoule/backchannel under MIT, and it runs with docker compose up --build. If you try degree three with a local model I have not tested, the issue tracker is where I would genuinely like to hear about it.