Two Tiers.
One Explicit Contract.
The Gateway owns every model, audio, auth, and storage decision. The Flutter App renders and captures — nothing more. Between them: a single authenticated WebSocket carrying a typed JSON envelope and binary audio frames.
Name the Planes. Give Each One Job.
Connect, don't absorb
NvAgent, Speaches & Keycloak are external services the Gateway integrates with — never re-implemented here.
Thinks vs. renders
All intelligence is server-side. The client is a presentation surface: it draws, captures, and plays.
One explicit contract
A small, framework-agnostic WebSocket protocol we own end to end — testable and stable.
Stateless & scalable
Gateway pods hold no session state; scale horizontally with sticky WS by session id and Postgres LISTEN/NOTIFY.
The Gateway —
Python / FastAPI
A stateless engine that exposes one real-time WebSocket plus a small REST surface for uploads, history, and health. It shapes the thinking channel, routes to a backend adapter, and runs the whole voice turn.
- WS Session & Event Router — per-connection fan-in / fan-out
- Voice Engine — STT → LLM → TTS with text cleaning
- Model Router — adapter select, stream normalize, thinking events
- Auth Guard — JWT today, Keycloak JWKS + RBAC next
- Persistence & Storage — asyncpg prefs + S3/MinIO signed URLs
The App —
One Flutter Codebase
A thin client for web, iOS, and Android. It renders chat and panels, captures the mic with on-device VAD, plays audio, and runs the login — identical regardless of which backend the Gateway routes to.
- Chat & streaming render — markdown, elements, live tokens
- Thinking panel — reasoning, tools & skills as they happen
- Native DAG monitor — Cytoscape-style, pan / zoom / drag
- On-device VAD + mic capture — Silero neural, native & web
- Responsive layout — split view on wide, chat-only on phones
One WebSocket, a Typed Envelope
Log in for a JWT, connect to /ws?token=…, and the whole turn flows over a single socket — the user message, streamed answer, agent progress, files, and spoken audio.
Client → Server
| Type | Purpose |
|---|---|
user_message | Send a chat turn (text, session, attachments) |
audio_blob | A spoken clip captured by on-device VAD |
resume | Rehydrate & continue a past thread |
monitor_set | Toggle the DAG monitor |
ping | Heartbeat (Gateway replies pong) |
Server → Client
| Type | When |
|---|---|
connection_ack | On connect — user, monitor & voice state, welcome |
task_start | A turn began |
reasoning · tool_call · use_skill | Live thinking-channel events |
final | The agent's final answer (markdown) |
files | Agent-produced output files |
audio | Spoken reply (base64 WAV) |
turn_complete | The turn finished |
Thinking-channel verbosity is gated by display levels: 0 answer-only, 1 progress while you wait (default), 2 everything. Reasoning is buffered and consecutive repeats collapse into a single × N line.
From a Laptop to Thousands of Users
The stack runs as gateway + app + speaches via Docker Compose for development, and scales out with stateless Gateway pods behind a load balancer.
- Sticky WebSocket routing by session id
- Postgres LISTEN/NOTIFY for cross-pod events
- Long-turn resilience — no wall-clock ceiling on multi-sub-agent turns
- 8 s server keepalive + 15 s client heartbeat keep sockets alive
- Phase-2 hardening — Keycloak OIDC/PKCE SSO, TLS/WSS, multi-tenant RBAC
Bring NvAssistant to
Your Enterprise
Voice-first AI, one Flutter codebase across every device, and a gateway you fully own. Let's talk about a pilot, a keynote, or a build.