What is on-device AI? The plain-English explainer
The short answer
On-device AI means the model runs on your own hardware instead of a cloud server: your prompt, the context and the reply never leave the machine. In a browser this works through WebGPU — your GPU executes a quantized model (roughly 600MB-2.6GB) inside the tab. ShadowTalk AI is a free workspace built around it: on-device chat with zero network egress, offline queueing and automatic sync, plus cloud models when a task needs them.
"On-device" is the phrase behind private AI, offline AI and edge AI — and it's often used loosely for products that still phone home. This page defines it precisely, shows how a browser can run a model, and lays out what on-device AI is genuinely good for and where cloud models still win.
On-device vs cloud AI
The two architectures differ in where the model physically runs — everything else follows from that:
| Aspect | On-device AI | Cloud AI |
|---|---|---|
| Where inference runs | Your CPU/GPU — in a browser via WebGPU | The provider's servers |
| What leaves your device | Nothing (zero network egress) | Your prompt, context and identity |
| Works offline | Yes, after a one-time model download | No |
| Model capability | Practical ceiling ~3B-class in a browser tab | Frontier models, hundreds of billions of parameters |
| Cost of inference | Your electricity — nothing else | Metered or subscription |
| Latency | Speed of your hardware, not your signal | Network round-trips |
How a browser can run a model
Three pieces make it work. WebGPU exposes your GPU's compute shaders to JavaScript. An inference engine (MLC's WebLLM class) executes the transformer in a background Web Worker so the interface stays responsive. And model weights stream down as shards cached in the browser — which makes downloads resumable and repeat visits instant. ShadowTalk wraps that capability with a device-tiered model registry, quota pre-checks and watchdogged workers.
The privacy math
On-device chat has zero network egress: no prompt, no context, no metadata leaves the tab, so there is nothing to intercept or store server-side. That is the tier where 'nothing sees my prompt' is literally true. Stored history is a separate question — ShadowTalk's Private Mode encrypts chats end-to-end (AES-256-GCM) with keys derived on your device, and default chats are stored server-side when Private Mode is off. Both facts are documented, because privacy claims die in the fine print.
What on-device AI is good for (and not)
Good for: everyday chat, drafting, summarizing, code help at small scale, anything sensitive, and any situation with bad or no connectivity. Not (yet) for: frontier-level reasoning, very long context, or the newest model releases — those need cloud models, which is why ShadowTalk routes heavy tasks upward automatically instead of pretending a 3B model is a frontier model.
Frequently asked questions
Keep reading
Free to use — no account, no card.
Works offline. Syncs when you're back. Your keys, your models, your chats.
Try on-device AI — free