Early, heavy and priced premium. I estimate it will take ~5 years for most practical AI to run on our own machines, with a few super brains left in the cloud.
I have been putting some effort into making local AI as consumer friendly as possible on macOS with this project https://github.com/pdparchitect/noodle/ but I will be the first to admit that the hardware you need is beyond most people's budgets. Hell, perhaps beyond most enterprises comfort levels too.
For now, subscriptions are the most practical way in.
Respectfully disagree. The bitter lesson has so far never failed us. And if in 5 years we'll be able to run models as capable as current SOTA on ordinary consumer hardware, imagine how much more powerful the models in datacenters would be
I actually believe it's going to be the other way around. Model progress is towards more efficient models, and mainly better compression, while the overall volume of data relevant to models' use-cases isn't increasing. In fact better models shouldn't depend on having all the data models have today either and should work efficiently without requiring all the factoids models encode today.
Provided better models are going be available, reliance on models as a service should decrease considerably.
Hopefully the point is that local models will be good enough for most day-to-day needs. They need more coherence and less hallucinations but that might happen within a year.
No doubt. The models in data centres will be ginormous, and impractical for everyday work, for the same reason nobody runs their spreadsheets on a supercomputer.
That's a matter of perspective. Almost every computer used to run spreadsheets is orders of magnitude more powerful than that task actually requires. Compute got so cheap that one could reasonably afford to run spreadsheets on a supercomputer.
Don't forget that your phone is several times faster than the world's best supercomputers just a decade or two ago.
> Don't forget that your phone is several times faster than the world's best supercomputers just a decade or two ago.
A decade? Not even close to true in the context of the applications supercomputers are designed to work on.
A20 Pro in the iPhone 18 Pro, 2026: a shade under 3 TFLOPS.
Even two decades ago: IBM BlueGene/L, 2006: 280 TFLOPs, 360+ peak.
You have to go back three decades, to 1996 to find a machine it is generally "several times faster" than — ASCI Red in its first configuration.
For sure, AI compute is an area where a phone has advantages. And, ironically, single core compute, because then ASCI Red is just one Pentium Pro. But ASCI Red had a terabyte of RAM, calculated in true 64 bit floating point, had vastly better distributed memory bandwidth etc.
Theo@t3.gg has an interesting take that open source models won't catch up to frontier models, at least in the foreseeable future, as there are major roadblocks in place today that weren't there less than a year ago.
I'm hugely pro-local LLM, but I'm a little suspicious of this. People used to say the same thing about Opus 4.8 or 4.5 (GPT 4 as well?).
Thing is, if you want to have a coding partner to implement things, the local models today are enough. But the frontier of what's possible is growing outwards to include things like superhuman code review, debugging, optimization, and one-shot program implementation.
Who's to say, by the time we have a local equivalent to Neon/Opus 5.5, that we have a whole bunch of new tasks that the massive models are capable of that we now need to be replicated in local versions before we make the switch?
You could definitely be right. But, “it’s been true before” isn’t a convincing argument; it’s essential that the space of tasks that Opus can do meaningfully better than local models stays large. There’s only so good you can get at the common 90% of coding.
People keep saying that and it keeps not being true because they get accustomed to the frontier models.
I'm certain that once you can run an Opus 5.5 quality model locally, whatever frontier model you can use in the cloud will outclass it so much that there will be no point to even touching your local model from a get-shit-done point of view.
(of course, maybe you want ZDR and offline access and fine tuning and thus have reasons to run it locally, but that's a different conversation)
I think "practical ai" is a bit of a frontier model, wrapped in a harness wearing a trench coat in this comment.
We already have useful local LLMs for voice to text, text to voice, translation, transformation/summation (transcript to formatted meeting notes) ... Stuff which is certainly practical.
But local ai is still trailing behind frontier models.
Since Spring, I've been working on a locally hosted, containerized web application to narrate my audiobook using a full cast of characters (skip to 9m47s for a reading of the demo book):
All weights are either permissively licensed or free until fantasy-level monthly recurring revenue sets in. The system runs entirely on the following hardware:
Text analysis uses Qwen3.6-35B-A3B MTP (Q4_K_XL GGUF) for speaker attribution, voice descriptions (not voice audio), emotive directions, and grammar. Both Muse Glimmer and GRMR cannot compete with Qwen3.6 for grammar checking. I haven't tried Gemma4, but I suspect Qwen3.6 remains the champ. My hope is that Inkling is more accurate for grammar than Qwen3.6, which is being tested at time of writing.
Voice design and cloning uses FireRedTTS3, OmniVoice, Qwen3-TTS, VoxCPM2, and Ming-3.0-tts-tiny. Most are approximately equivalent. For narration, nothing comes close to Drama 3 Expressive Medium fine-tune in terms of quality, consistency, or pacing.
Unsloth's GGUFs have been absolutely instrumental in making the system a reality. Highly recommend using them.
while it's probably on the way to being something good I found that just a single good voice and a CPU-only PocketTTS is sufficient for a good experience. just make sure to process line by line, cut intelligently (periods, commas, quotes, etc) to fit into max_tokens and insert pauses in between lines.
there is a transition point somewhere between kokoro 82M and PocketTTS 100M where it just suddenly becomes good enough. kokoro lacks the tiny human touch that makes prolonged listening-for-pleasure tolerable and PocketTTS has enough of it.
you do need to find a good voice for PocketTTS, for some reason the default voices they chose to include are pretty bad. some people swear that the right voice by chance makes PocketTTS perform better due to some quirk of training or something.
here is a demo of PocketTTS 100M with the default kokoro voice cloned (in practice, I take one from audible): https://vocaroo.com/19piBx5VT5qy (speed it up to taste, mine is 2x)
A suggestion for the calculator: allow the user to set a baseline VRAM usage instead of a fixed 1GB "overhead" when checking "Card also drives my monitors" - for me, /usr/bin/X sits at ~139MiB VRAM on nvidia-smi, and if I don't run Firefox, that is the only thing consuming VRAM.
Set this up this weekend after AMD NPU support was announced for Unsloth. My ThinkPad x13 with the Ryzen AI 350 and 32 GB RAM is now serving my older GhostBSD ThinkPad running crush. Using Qwen3.5-9B (npu2) the quality of course is not comparable to the latest frontier models. Basically no result can be trusted. And 12 tokens per second are quite relaxing but it somehow feels 'right'.
I run Basalt with Qwen 3.8 flash next (default Basalt NVFP4 quant) on 5070ti (16GB VRAM) with 96GB RAM. Prefill ~2000tok/s, generation ~100 tok/s. Definitely doesn't feel sluggish but it's not a laptop
Processor FLOPS is probably only relevant if you’re doing some form of model parallelism and using your CPU for a little extra speed boost (a lot more complexity for maybe like, 5-10% speed up I’m guessing). Otherwise, your CPU is really just issuing instructions to move memory around, which is bottlenecked by PCIe bandwidth, not processor FLOPS. Processor power (watts) is pretty irrelevant im guessing.
The sweet spot for VRAM for local model hosting is 128gb (Mac Studio or DGX spark). That gets you qwen-coder next at like Q8, so not lobotomized too bad, at 50 t/s, and a generous context window.
As with most things, the real answer is “it depends”. All the VRAM in the world won’t save you if your HBM bandwidth is low
I just use Claude Code with DeepSeek 4.1F 1017. Works fine. DeepSeek has its own harness, which I'm sure is also just fine, but using CC with it makes it easy to go back and forth between DS and Opus/Fable.
The guidelines in claude.md apply equally to DS, which is another benefit to sharing the same harness. No need to maintain two separate policy files.
So pretty soon 10,000+ people could be running decent models on their home PCs with unsecured LAN access, connected to their residential modem? We won't just have residential proxy networks, we'll have residential ARTEX swarm clouds.
Doesn’t sound to far off the simplexity chatted to ChatGPT and Claude yesterday.
When entire operating systems and software stacks to run end to end business processes, including engineering is a reality, Bill’s word “optimise” is close to the ultimate fulfillment.
There is too much compute in data centres, when multiple machines can be replaced with a functionally intelligent pi.
Early, heavy and priced premium. I estimate it will take ~5 years for most practical AI to run on our own machines, with a few super brains left in the cloud.
I have been putting some effort into making local AI as consumer friendly as possible on macOS with this project https://github.com/pdparchitect/noodle/ but I will be the first to admit that the hardware you need is beyond most people's budgets. Hell, perhaps beyond most enterprises comfort levels too.
For now, subscriptions are the most practical way in.
Provided better models are going be available, reliance on models as a service should decrease considerably.
Don't forget that your phone is several times faster than the world's best supercomputers just a decade or two ago.
A decade? Not even close to true in the context of the applications supercomputers are designed to work on.
A20 Pro in the iPhone 18 Pro, 2026: a shade under 3 TFLOPS.
Even two decades ago: IBM BlueGene/L, 2006: 280 TFLOPs, 360+ peak.
You have to go back three decades, to 1996 to find a machine it is generally "several times faster" than — ASCI Red in its first configuration.
For sure, AI compute is an area where a phone has advantages. And, ironically, single core compute, because then ASCI Red is just one Pentium Pro. But ASCI Red had a terabyte of RAM, calculated in true 64 bit floating point, had vastly better distributed memory bandwidth etc.
[0] https://youtu.be/B2rFx4Fy1us
Like many others here, Theo has a vested interest in my experience not being real today. I don't really care what he has to say because he's wrong.
Thing is, if you want to have a coding partner to implement things, the local models today are enough. But the frontier of what's possible is growing outwards to include things like superhuman code review, debugging, optimization, and one-shot program implementation.
Who's to say, by the time we have a local equivalent to Neon/Opus 5.5, that we have a whole bunch of new tasks that the massive models are capable of that we now need to be replicated in local versions before we make the switch?
I'm certain that once you can run an Opus 5.5 quality model locally, whatever frontier model you can use in the cloud will outclass it so much that there will be no point to even touching your local model from a get-shit-done point of view.
(of course, maybe you want ZDR and offline access and fine tuning and thus have reasons to run it locally, but that's a different conversation)
We already have useful local LLMs for voice to text, text to voice, translation, transformation/summation (transcript to formatted meeting notes) ... Stuff which is certainly practical.
But local ai is still trailing behind frontier models.
https://www.youtube.com/watch?v=Q9PKT6bdKnc
All weights are either permissively licensed or free until fantasy-level monthly recurring revenue sets in. The system runs entirely on the following hardware:
Text analysis uses Qwen3.6-35B-A3B MTP (Q4_K_XL GGUF) for speaker attribution, voice descriptions (not voice audio), emotive directions, and grammar. Both Muse Glimmer and GRMR cannot compete with Qwen3.6 for grammar checking. I haven't tried Gemma4, but I suspect Qwen3.6 remains the champ. My hope is that Inkling is more accurate for grammar than Qwen3.6, which is being tested at time of writing.Voice design and cloning uses FireRedTTS3, OmniVoice, Qwen3-TTS, VoxCPM2, and Ming-3.0-tts-tiny. Most are approximately equivalent. For narration, nothing comes close to Drama 3 Expressive Medium fine-tune in terms of quality, consistency, or pacing.
Unsloth's GGUFs have been absolutely instrumental in making the system a reality. Highly recommend using them.
while it's probably on the way to being something good I found that just a single good voice and a CPU-only PocketTTS is sufficient for a good experience. just make sure to process line by line, cut intelligently (periods, commas, quotes, etc) to fit into max_tokens and insert pauses in between lines.
there is a transition point somewhere between kokoro 82M and PocketTTS 100M where it just suddenly becomes good enough. kokoro lacks the tiny human touch that makes prolonged listening-for-pleasure tolerable and PocketTTS has enough of it.
you do need to find a good voice for PocketTTS, for some reason the default voices they chose to include are pretty bad. some people swear that the right voice by chance makes PocketTTS perform better due to some quirk of training or something.
here is a demo of PocketTTS 100M with the default kokoro voice cloned (in practice, I take one from audible): https://vocaroo.com/19piBx5VT5qy (speed it up to taste, mine is 2x)
The sweet spot for VRAM for local model hosting is 128gb (Mac Studio or DGX spark). That gets you qwen-coder next at like Q8, so not lobotomized too bad, at 50 t/s, and a generous context window.
As with most things, the real answer is “it depends”. All the VRAM in the world won’t save you if your HBM bandwidth is low
Do you reuse Claude's plugins/skill files?
Do you use a special harness or system prompts?
The guidelines in claude.md apply equally to DS, which is another benefit to sharing the same harness. No need to maintain two separate policy files.
you don't like how the harness works? ask your agent to change it
this works surprisingly well
> I'm using an RTX 5090 with 32GB VRAM for this setup.
$10K GPU. Yeah, sure, not much at all.
When entire operating systems and software stacks to run end to end business processes, including engineering is a reality, Bill’s word “optimise” is close to the ultimate fulfillment.
There is too much compute in data centres, when multiple machines can be replaced with a functionally intelligent pi.