For the first two years of the LLM boom, “AI” meant “call an API.” In 2024 small models started running usefully on laptops. In 2025 that became a product category — privacy-first assistants, offline tools, developer-side IDE features, voice agents with local inference. In 2026 it’s approaching mainstream.
This post covers the infrastructure side of edge LLM inference: what runs where, the runtimes that matter, deployment tradeoffs, and the privacy/product patterns emerging.
Three distinct tiers:
Each tier has its own runtime ecosystem. The product patterns differ wildly.
Apple’s MLX framework is the default on M-series chips. Released late 2023, matured fast.
Strongly recommended for any Apple-first product.
llama.cpp (Georgi Gerganov’s project) is the community standard for local inference. Runs on CUDA, Vulkan, Metal, CPU. GGUF quantization format ubiquitous.
For production Windows apps with NVIDIA, CUDA direct via PyTorch or TensorRT. ExecuTorch (PyTorch’s edge runtime) is the newer managed path.
Ollama wraps llama.cpp with model management and an API. Best DX for “download and run” scenarios. Macs, Windows, Linux. API is OpenAI-compatible.
Default recommendation for any user who wants to run models locally without building infrastructure themselves.
Running models in the browser via WebGPU. Transformers.js (HuggingFace) handles small models; WebLLM (CMU) scales to ~13B on capable machines.
Use cases: browser-side PII redaction, offline tools, privacy-first SaaS features.
Mobile inference is dominated by 1B–4B specially-designed models: Apple’s foundation models, Google’s Gemini Nano, Llama 3.2 1B/3B, Phi-5, Qwen3-1.5B, DeepSeek Edge, TinyLlama.
Models we routinely run with decent performance (5–20 tokens/sec):
Quality is genuinely near-frontier for most tasks. Privacy use cases (legal, medical, code generation on private repos) are real.
Fast enough for interactive use, useful quality. This is the sweet spot for desktop/laptop apps.
Quality is task-dependent. Great for summarization, classification, structured generation. Less reliable for open-ended chat.
Everything runs on device. No network calls during inference. Privacy-first.
Example: a local note-taking app that runs a 7B model in the background to generate tags and summaries.
Tradeoffs:
Local model handles sensitive operations; cloud handles complex ones. Routing happens on-device.
Example: an enterprise assistant runs PII redaction locally, sends redacted queries to a cloud model for reasoning.
Tradeoffs:
Default to local; fall back to cloud when local fails (doesn’t know the answer, hits latency budget).
Example: IDE code completion — local model on keystroke, cloud model when user requests “explain” or “refactor.”
Tradeoffs:
Not for privacy — purely for user experience. Local streams first tokens fast; cloud model provides the final answer.
Example: voice agent that starts responding immediately from local model while cloud model generates a better answer.
Tradeoffs:
Edge models are not API models. For reasoning-heavy tasks, a 7B local model is no match for GPT-4o or Claude Sonnet.
What edge models are competitive at:
What they struggle with:
Product design matters. Good edge AI products scope what’s asked of the local model to its capability band.
The single best reason to go edge in 2026 is privacy. Several product categories emerged specifically around local-first AI:
For B2B in regulated industries, “runs on your laptop, never leaves” is a serious product positioning. Worth real investment.
Edge inference doesn’t have server-side ops — but it has its own ops problems:
1. Model distribution. 4GB to 40GB of weights have to reach user devices. CDN strategy matters. Delta updates help.
2. Version fragmentation. User on version 3.2 has model v1; user on 3.5 has v2. Both need to work.
3. Hardware capability detection. Choose the right model for the device. Fallback to smaller model if user’s GPU is insufficient.
4. User education. First run needs to download the model. Users need expectations set.
5. Telemetry. You can’t see failures server-side. Opt-in telemetry, error reporting, usage metrics all become harder.
6. Quality measurement. Your eval pipeline needs to run on-device, not just server-side.
Tools that help:
What’s new in 2026:
A user’s local model is tuned on their data. Private, personalized. Emerging in note-taking and coding tools.
Local models adapt based on user interactions. Privacy-preserving by design (data never leaves device).
Local models handle vision + text in the same pass. Especially strong on Apple Silicon (unified memory).
Small agents running local tools on the device itself. File search, calendar, notifications — without cloud round-trips.
Local small models enable genuinely real-time voice agents. End-to-end latency under 500ms achievable.
If your product has a privacy dimension and you’re not considering edge inference, you’re missing a capability the market increasingly expects.
Designing an edge-AI product? Talk to us — we’ve shipped local-first AI across desktop and mobile.
AI platform engineering is a distinct discipline from ML ops and generic platform engineering. A practical guide to scoping, staffing, and operating an AI platform team — from first hire to org-wide enablement.
When GPU spend crosses $500k/month, informal cost discipline stops working. A FinOps playbook for large AI compute bills — attribution, commitments, workload placement, and the structural changes that matter.
Prefill is compute-bound, decode is bandwidth-bound. Running them on the same GPU wastes capacity. Disaggregated inference separates them — 30–50% throughput wins on real workloads.