The 24 GB question.

Start here

Choose a task, then compare the assumptions behind the answer.

Use cases

What local AI is actually good for, from a private voice assistant to a robot arm.

• A private voice assistant for the home

  Wake word, speech recognition, a small language model and a voice, all on a box in the cupboard. Nothing you say leaves the house.

• Offline document search with RAG

  Ask questions of your own PDFs, notes and manuals, with citations, on a machine that never phones home.

• A local coding copilot

  Autocomplete in the editor and an agent that edits files and runs tests, with your code staying on your machine.

• Receipt and paperwork scanning

  Photograph receipts and letters, get searchable text, titles and tags, and keep the archive on your own disk.

Rankings

24 GB class, on a used RTX 3090 at 8k context. Speed and quality are listed side by side, never merged into one score.

Builds

Three of our six curated rigs, with parts lists, power and break-even.

Complete systems, listed parts only; tax, shipping and peripherals excluded. Electricity: load power × 4 h plus idle power × 20 h daily, 30 days at $0.15/kWh. Cost comparison only; capabilities differ.

Local vs. subscription

How long a $3,499.99 Strix Halo box takes to pay for itself against each plan.

Sourced hardware price; Sourced power input. Complete mini PC purchase; tax, shipping and peripherals excluded. Electricity assumes 120 W × 4 h/day, 30 days at $0.15/kWh, with no idle draw. Build cards include idle draw. Cost comparison only; capabilities differ.

Months to break even = hardware ÷ (plan price − electricity). Heavy tier means the vendors' high-usage plans (5×, 20× or top tier); standard is the ~$20 plan. Local AI pays off on cost only against heavy tiers, or when privacy and unlimited use are the point. Run it with your numbers.

Learn

Explainers that start in plain language and go as deep as you want, with links to the data.

Why is memory bandwidth the speed limit for local AI?

The one-line formula that predicts tokens per second, why a 3090 beats a 128 GB mini PC on dense models, and why MoE models flip the result.

What is a KV cache, and why does it eat my VRAM?

The part of memory that grows with every token you add, why it can outgrow the model itself, and the three tricks that keep it small.

What is quantization, and how much does Q4 hurt?

Why a 54 GB model becomes a 16.5 GB file, what Q4_K_M actually means, and how to choose a quant for your card.

• What do context length, RoPE and YaRN mean, and how much context do you really get?

  Why "128K context" on a model card doesn't mean 128K in your app, how RoPE scaling stretches a model past its training, and what each extra token costs in memory and speed.

• What is a mixture-of-experts model, and why does "30B-A3B" run so fast?

  Why a model can be 117 billion parameters and still type quickly on a mini PC, what "active parameters" means for your memory and your speed, and when MoE is the wrong choice.

• GGUF, EXL2/EXL3, AWQ or GPTQ — which quantized file should you download?

  The four file formats you'll see on every model page, which runtime each one belongs to, and a one-line rule for picking the right file for your hardware.

• What are RAG and embeddings, and how does a local model "read" your documents?

  How retrieval-augmented generation lets a small local model answer from your own files without retraining, what an embedding actually is, and the four settings that decide whether it works.

• What is speculative decoding, and is it a free speed-up?

  How a small "draft" model can make a big one type 2–3× faster with identical output, why the speed-up depends on what you're writing, and how to switch it on in llama.cpp and vLLM.

• What are tool calling and MCP, and can a local model use them?

  How a model "uses" a calculator, a search engine or your file system without ever running code itself, what the Model Context Protocol standardises, and what to check before you trust a local model with tools.