Run an LLM in your browser — no install, no server
The short answer
With WebGPU, a modern browser can run quantized LLMs — Llama 3.2, Qwen 2.5, SmolLM2 — directly in a tab, on your own GPU. ShadowTalk AI turns that capability into a product: a device-tiered model manager, resumable per-shard downloads, a watchdogged inference worker, and automatic escalation to cloud models when a task is too heavy — free.
Browser LLMs spent years as demos. This page explains how they actually work, which models fit which machines, and what separates a packaged product from a proof-of-concept — with the specific engineering ShadowTalk ships for each gap.
How a browser runs a language model
Three pieces make it work. First, WebGPU: the browser exposes your GPU's compute shaders to JavaScript, which is what makes billion-parameter models interactive on consumer hardware. Second, an inference engine — ShadowTalk uses MLC's WebLLM — running in a background Web Worker so the interface stays responsive while the model thinks. Third, storage: model weights stream down as shards and are cached per-shard in the browser's Cache API, which is what makes downloads resumable and repeat visits instant.
- WebGPU compute shaders execute the transformer on your hardware — prompts never leave the device
- The engine runs in a Web Worker: the UI keeps 60fps while tokens stream
- Per-shard caching means a dropped connection pauses the download; it resumes, never restarts
- GPUs without the shader-f16 feature automatically get q4f32 quantized model twins — the swap happens at activation time
Models you can run, and what they need
ShadowTalk's model registry pre-checks your GPU features and storage quota, then offers the tiers that fit. All sizes below are approximate on-device footprints:
| Model | Strength | Size (q4f16 / q4f32) |
|---|---|---|
| Qwen2.5-0.5B-Instruct | Nano tier — modest devices | 600MB / 700MB |
| Llama-3.2-1B-Instruct | Instant default — fast, capable | 880MB / 1.1GB |
| SmolLM2-1.7B-Instruct | Balanced everyday chat | 1.68GB / 2.0GB |
| Qwen2.5-Coder-1.5B / 3B | Code generation on-device | 1.5GB / 2.3GB |
| Llama-3.2-3B-Instruct | Pro tier — needs 8GB+ devices | 2.1GB / 2.4GB |
| Qwen2.5-3B-Instruct | Pro tier — strongest local chat | 2.3GB / 2.6GB |
From demo to product: the gaps most browser LLMs leave open
Running a model in a tab is easy; shipping it is not. What ShadowTalk adds around the raw engine:
- Tiered, consent-gated background install with storage-quota pre-checks and auto-retry with backoff — offline pauses and resumes cleanly
- Watchdogged engine lifecycle: 45s init watchdog, generation watchdogs, and a recovery ladder that caps worst-case wait at about three minutes
- A tested hybrid router (30+ unit-tested cases): small talk and quick replies stay on-device, heavy tasks escalate to your chosen cloud model — Apple-Intelligence-style routing, in a browser
- A supply chain you can audit: zero third-party runtime calls, self-hosted fonts, and model weights plus engine WASMs mirrored on the project's own git branch — Hugging Face is not on the critical path
Developer notes
The stack is Vite + React + TypeScript with @mlc-ai/web-llm 0.2.x, vite-plugin-pwa for the precached shell (10MB-per-file cap, weights excluded by design), and a puppeteer-verified offline smoke test in CI that asserts zero third-party hits under a network cut. The mirrored 0.5B model twins and their model_lib WASMs live on the repo's orphan weights branch, byte-identical to upstream, with an env override for self-hosters.
Frequently asked questions
Keep reading
ShadowTalk vs web-llm-chat
The demo vs the product, scored
ShadowTalk vs LM Studio
Desktop local AI vs browser local AI
Offline AI chat — works with no internet
What keeps working after the download
Private AI chat — encrypted, no account
The three privacy tiers
Documentation
Full feature and setup reference
Free to use — no account, no card.
Works offline. Syncs when you're back. Your keys, your models, your chats.
Run your first browser LLM — free