Audio Pipeline

Everything in the live path operates on PCM16, 16 kHz, mono audio.

Capture and framing

frontend/src/hooks/useAudioCapture.ts captures microphone audio with getUserMedia, downsamples it to 16 kHz PCM16 in the browser, and sends binary WebSocket frames. Tab/system audio, when enabled, is captured as a second track. Each frame carries a 1-byte track prefix:

• 0x00 – microphone
• 0x01 – system/tab audio

Because PCM16 payloads are always even-length, the backend detects the prefix by frame parity: odd-length frames are prefixed, even-length frames are treated as legacy mic audio (_decode_audio_frame in backend/app/ws/audio_runtime.py). The WebSocket layer is split across backend/app/ws/audio_handler.py (the endpoint and call lifecycle) and its companions audio_messages.py, audio_pipeline.py, audio_runtime.py, and audio_persistence.py.

On the backend the two tracks are mixed into a single stream (backend/app/services/track_mixer.py) for the interim audio gateway and the session recording, while each track is diarized separately so remote (system-audio) speakers get their own identities, prefixed sys_.

During split-track capture, microphone segments resolve directly to the sole configured is_user speaker and stay out of the remote auto-speaker map. System-audio speakers retain normal diarization. Mic-only capture, or sessions with zero or multiple is_user rows, retain normal mic diarization. The client sends system-track state changes explicitly, and that state is snapshotted on each queued segment so stopping sharing cannot relabel in-flight transcription.

Voice activity detection and segmentation

backend/app/services/speaker_diarizer.py runs Silero VAD over the incoming stream and cuts speech into segments. Defaults live in backend/app/config.py and can be adjusted at runtime through the Admin diagnostics endpoints (/api/diagnostics/diarization):

Setting	Default	Meaning
VAD_THRESHOLD	0.6	Silero speech probability required to count as speech
MIN_SEGMENT_MS	750	Segments shorter than this are dropped
MAX_SEGMENT_MS	15000	Force a segment boundary after this much speech
SILENCE_GAP_MS	600	Silence long enough to close the current segment
SPEAKER_SIMILARITY_THRESHOLD	0.68	Cosine similarity required to match an existing speaker embedding
MIN_NEW_SPEAKER_MS	4000	Minimum speech needed before enrolling a new voice profile
MAX_SPEAKER_PROFILES_PER_TRACK	4	Safety cap for auto-enrolled profiles on each audio track

Both ONNX models run on CPU with an explicitly bounded ONNX Runtime thread pool rather than ORT’s one-thread-per-core default, which charged most of its CPU to pool overhead and ignored container CPU quotas:

Setting	Default	Meaning
DIARIZER_VAD_ONNX_THREADS	1	Intra-op threads for Silero VAD (nothing in it parallelizes)
DIARIZER_EMBED_ONNX_THREADS	min(4, cores / 2)	Intra-op threads for the speaker embedding model
DIARIZER_EMBED_ONNX_SPIN	false	Whether ORT may spin-wait between embedding calls

See Configuration for the measured trade-offs.

A diarizer slower than realtime would otherwise buffer incoming audio without bound until the process is killed for running out of memory. The ingress path therefore caps the queued diarization backlog at 30 seconds of dual-track audio and sheds the oldest queued frames past that cap (_shed_diarization_backlog in backend/app/ws/audio_runtime.py, applied per frame in backend/app/ws/audio_messages.py), so an overloaded diarizer loses some speaker attribution instead of ending the call. The client is told once via a diarization_overloaded status message, and the shed count feeds the call-health block of the live agent activity feed.

That guard and the thread bounds above address the same failure from opposite ends. Under a 2-CPU container quota the unbounded pool ran the pipeline at 134 percent of realtime - slower than the audio arriving, which is what drove the backlog into the shedder in the first place. Bounding the pool brought the same audio to 14 percent, so the shedder is now a genuine backstop rather than a routine occurrence.

Speaker identification

Each closed segment gets a WeSpeaker ResNet152 embedding, compared against the per-call SpeakerRegistry. A match reuses that auto ID; otherwise a new auto_N identity is created after enough speech is available. Short unmatched segments reuse the closest established profile without changing its centroid, and each input track has a bounded profile count. The WebSocket handler maps auto IDs to database Speaker rows, auto-creating “Participant N” (or “Remote Participant N” for sys_ IDs) rows when a new voice appears. A ghost filter (backend/app/services/speaker_ghost_filter.py) defers short one-off segments that would otherwise create a spurious new speaker.

The default live diarizer is the lightweight VAD+embedding pipeline (LIVE_DIARIZER=lightweight). An NVIDIA Sortformer diarizer can be enabled for GPU deployments (see Deployment); backend/app/services/diarizer_factory.py chooses the implementation from runtime configuration.

Required ONNX models are expected at backend/models/silero_vad.onnx and backend/models/voxceleb_resnet152_LM.onnx (the legacy backend/models/ecapa_tdnn.onnx is used as a fallback when the new file is absent); fetch them with backend/scripts/download_models.py (the Docker build does this for you).

Diarization modes, benchmark, and voice profile

The Diarization Capability card (Admin -> Transcription & Audio) controls which live diarizer runs and how strictly speakers are matched:

• Fallback is the lightweight VAD+embedding pipeline described above. It runs on CPU everywhere and is the default.
• Enhanced runs NVIDIA NeMo Sortformer. It stays locked until the machine passes a sustained benchmark. One 15-20 second input is replayed for three live windows, and passing requires 3x measured throughput: 2x for mic plus system audio with a 1.5 load reserve for transcription. The result reports raw and contention-adjusted per-track real-time factors, measured headroom, and the per-instance peak resident-memory increase. The highest observed peak is retained so a warm rerun cannot erase a cold load measurement. These values persist across restarts for the aggregate capacity planner. The benchmark accepts an uploaded file or a fresh mic recording.

The speaker matching slider on the same card adjusts SPEAKER_SIMILARITY_THRESHOLD at runtime: lower values merge more (fewer, broader speaker identities), higher values split more.

My voice profile

Recording a short clip (4-10 seconds) enrolls your voice for mic-only speaker matching: the backend extracts a speaker embedding and keeps only that encrypted voice signature – the calibration audio itself is discarded. Recording again replaces the profile; Delete removes it. Browser recordings for both voice enrollment and the mic benchmark arrive as WebM/Opus. The ffmpeg fallback streams them from memory to memory; it does not create a temporary source-audio file. Windows and Linux desktop bundles include ffmpeg, while macOS and source runs need it on PATH as described under Audio file import.

Batch transcription

Diarized segments are transcribed in original audio order through OrderedTranscriptionQueue. The transcriber is picked by model ID in create_transcriber (backend/app/services/local_transcriber.py):

• local-* model IDs (for example local-whisper-base, local-parakeet-tdt-0.6b) run ONNX Whisper/Parakeet locally via onnx-asr. Weights download to DATA_DIR/asr-models/ on first use; no API key required. These models transcribe only; the analysis agents run on text models instead – cloud, or self-hosted via an endpoint:<slug>:<model> registry entry (see Configuration).
• OpenAI-provider registry IDs route to OpenAI (backend/app/services/openai_transcriber.py): the specialized transcribe models (gpt-4o-transcribe, gpt-4o-mini-transcribe) go through /v1/audio/transcriptions (OpenAITranscriber), and the audio chat models (gpt-audio-1.5, gpt-audio-mini) go through Chat Completions with an input_audio content part (OpenAIChatTranscriber).
• Everything else goes to Gemini: the segment is wrapped as WAV and sent with a transcription prompt (backend/app/services/batch_transcriber.py).

The active model comes from the persisted transcription.batch.model_id app setting (Admin panel), falling back to BATCH_TRANSCRIBER_MODEL. Filters drop low-energy segments, known phantom phrases (common hallucinations on near-silence), and single-word outputs before anything is saved.

Interim transcription

Independently of the batch path, the mixed stream is forwarded to the audio gateway, which sends back interim text. The gateway is a cloud streaming session (Gemini Live or OpenAI Realtime, sub-second partials) or, when the audio_gateway agent is set to local-parakeet-live, the on-device local captioner (backend/app/services/local_live_captioner.py): it batches the incoming audio into short, non-overlapping chunks committed roughly every 3 seconds (chunks under 1 second are held for more context) and transcribes each with local Parakeet ONNX. No cloud call is made, so it works under Privacy First; it is experimental and CPU-heavy, and latency is about one commit interval rather than sub-second. Interim text is display-only; the diarized batch transcript is the source of truth that agents analyze and that gets persisted.

Audio storage and re-transcription

Mixed call audio is appended per call segment to DATA_DIR/audio/<session_id>/segment_<n>.wav (backend/app/services/audio_store.py); the path is stored on the call_segments row when the segment closes. Split-track calls also retain time-aligned segment_<n>_mic.wav and segment_<n>_sys.wav files, including silence on the absent side. This uses about three times the PCM storage of a mixed-only call. Auxiliary files are created only after a system track is observed, so mic-only calls stay at one copy; startup removes unreferenced auxiliary files left by an interrupted split-track call.

Migration 016_add_call_segment_track_paths adds nullable mic_audio_path and system_audio_path columns, and startup schema patching adds them to older local databases. Existing rows with only audio_path remain playable and re-transcribable.

Because raw audio is retained, a session can be re-transcribed later through any batch-capable model with POST /api/sessions/{id}/retranscribe (destructive to existing transcript entries). Retranscription prefers the split paths when present so mic speech stays bound to the sole local user and remote voice matching continues across call segments. Split-track turns are ordered by their source speech start rather than diarizer completion time. Individual mixed segment recordings can be fetched from GET /api/sessions/{id}/segments/{n}/audio.

Audio file import

POST /api/sessions/{id}/import/audio accepts .wav, .mp3, .m4a, .ogg, and .flac, decodes with soundfile first and falls back to ffmpeg for compressed formats, then runs the file through the same diarization and transcription pipeline as a live call. The ffmpeg fallback resolves the executable from BACKCHANNEL_FFMPEG first (the desktop launcher points it at the copy bundled with Windows and Linux desktop builds), then from PATH, and reports a clear “FFmpeg is required” error when neither is available.