ShadowTalk AI Documentation
ShadowTalk AI is a BYOK-first agentic workspace: bring your own provider key, feed it documents with local-first Stealth RAG, run approved local tools via MCP, and talk to a living voice presence in LIVE Mode — with on-device memory, honest token telemetry, and zero training on your data.
What is ShadowTalk AI?
ShadowTalk AI is a production-grade, high-performance agentic workspace designed for engineers, researchers, founders, and enterprises. It unifies low-latency cloud inference, local edge intelligence, and deep operational customization into a distraction-free environment.
Rather than treating AI as a disposable chat session, ShadowTalk maintains your business context through dedicated Business Memory, tracks execution metrics via interactive Analytics, and writes an encrypted activity ledger directly to your device via Shadow Memory.
Core Architectural Foundations
1. BYOK-First Multi-Provider Engine: Your provider API key stays in your browser and streams directly to the provider — 16+ supported providers with per-provider context budgeting, honest token telemetry, and optional on-device WebLLM inference when the model/device path is available.
2. Zero-Cloud Shadow Memory (/shadow-memory): An on-device activity ledger built on IndexedDB with automatic local storage fallback, guaranteeing zero telemetry leakage to external servers.
3. The Intelligent Workspace: Stealth RAG ingests and retrieves your documents locally, the Dynamic Token Optimizer compacts context non-destructively, and the MCP Bridge runs your local tools behind explicit approvals.
Privacy & Zero-Knowledge Architecture
ShadowTalk enforces a strict client-side privacy perimeter. User API keys, prompt injection filters, and on-device memory ledgers remain in your browser sandbox. We never train public foundation models on your proprietary business conversations.
The Intelligent Workspace (Stealth RAG · Token Optimizer · MCP Bridge)
Stealth RAG: drop a document into chat and it is extracted, chunked, embedded and indexed entirely on your device (IndexedDB, isolated per user and project), then retrieved automatically into the conversation. The default embedder is honest on-device feature hashing; neural embeddings are an explicit opt-in via your own BYOK key on documented provider endpoints only.
Dynamic Token Optimizer: per-provider context windows and output caps for every BYOK provider, CJK-aware token estimation, and non-destructive compaction — decisions, requirements, open questions and your recent message floor are protected, summaries are computed locally at zero cost, and your stored conversation is never rewritten.
MCP Bridge: a zero-dependency Node companion pairs with your browser over loopback (127.0.0.1 only, token-gated, DNS-rebinding defended) so approved local tools can be invoked from chat — every tool starts at ask-every-use, and write/execute tools always prompt.
LIVE Voice Mode — A Presence, Not a Chatbot
LIVE Mode is a full-screen voice conversation with a living avatar: a canvas-rendered presence that breathes, blinks, and morphs with real microphone and TTS amplitude, its aura re-tinting per phase (idle, listening, thinking, speaking).
Interruption is hands-free: while the AI speaks, the mic keeps metering and sustained speech hands you the floor instantly (echo-cancelled, with a grace period and cooldown so the AI never interrupts itself). Tap-to-interrupt remains as a fallback.
Replies stream in as they generate — the first sentence is spoken roughly a second after you stop talking — and a rhythm engine strips emoji and markdown before TTS and inserts punctuation-aware pauses between sentences.
AI Provider Routing & Hybrid Intelligence
ShadowTalk is BYOK-first: the browser streams directly to your configured provider (OpenAI, Anthropic, Google, Groq, Mistral, OpenRouter, Together, Fireworks, DeepSeek, xAI, Cohere, Perplexity, Hyperbolic, Novita, DeepInfra, and more) using your own key from Settings → Personalization → BYOK.
Streaming is native: provider deltas (onDelta) feed both the chat surface and LIVE Mode's speech pipeline, so voice replies begin speaking the first completed sentence while the rest is still generating.
LocalAIProvider: Manages fully offline inference using the WebLLM engine, dynamically falling back to local models when disconnected. A shared-gateway path exists for accounts without a key — with honest errors, never fake responses.
Web Worker Inference Engine
To guarantee 60fps UI responsiveness during offline execution, the LocalInferenceEngine isolates the WebLLM runtime entirely in a background Web Worker (webllm.worker.ts).
Model weights are cached locally via Cache Storage/OPFS, preventing expensive re-downloads, while the main thread remains free from heavy tensor computations.
Activation & TTFV Telemetry
ShadowTalk measures Time To First Value (TTFV) through the ActivationAnalytics service.
Every session is tracked from initial load to the first successful generation (both online and offline), logging anonymized metadata to Firestore while keeping prompt content strictly on-device.
Product pillars
BYOK Multi-Provider Engine
Direct browser-to-provider streaming across 16+ providers with per-provider context budgeting and on-device WebGPU fallback.
Intelligent Workspace
Stealth RAG document ingestion and retrieval, non-destructive context compaction, and approval-gated local MCP tools.
Shadow Memory Journal
100% on-device IndexedDB activity ledger with parameter inspection and multi-format exports (/shadow-memory).
Real-Time Telemetry & Analytics
Production Recharts dashboard measuring tokens, response latency, and feature usage (/analytics).
Tech stack (summary)
Frontend: React 18, Vite, TypeScript, Tailwind CSS, Framer Motion, shadcn/ui, Recharts
AI Intelligence: Multi-Model Turbo Engine (Groq Llama-3.3 70B, DeepSeek R1, OpenAI GPT-4o)
Edge & Offline: In-browser WebGPU runtime + Transformers.js zero-network execution
Telemetry & Audit: On-device tamper-evident cryptographic ledger + downloadable JSON reports
Platform: Desktop PWA, mobile-responsive layout, and developer REST APIs