I’m getting more and more convinced that we will end up running LLMs in our personal computers. Which makes me wonder where Anthropic/OpenAIs moats will come from.
1. in order to run LLMs, especially the best ones, you need complicated devices which are expensive
2. if you buy one for your personal use, you are probably not going to utilize it all the time and it will be idle a lot
It seems to me that it will always be more economical that the LLM-running devices are in a datacenter where it is easier to make sure they are always utilized
If a model is substantially better than most humans at most tasks, the human isn't going to be able to perceive the difference between Claude Opus 7.7 and 8.7. Humans at some point aren't going to be able to perceive the difference on benchmarks either, because they are going to get wildly abstract.
AI vendors are really going to struggle to shift tokens far beyond the frontier of human capabilities. It's reasonable (not guaranteed) to assume that, if the trend of frontier models (doubling capabilities on benchmarks every n months) holds, then the same trend will hold for local models, and those local models will meet and exceed the perception frontier. This would mean a human cannot tell the difference between Mistral-Open-2030 and Claude Opus 2030.
That's a bunch of "ifs", but there's nothing exceptional about those "ifs". They're basically the scenario if nothing changes between now and ~2030 with regards to capabilities trend attainment.
The trend over the past three decades of personal computing has been for devices to become exponentially more powerful regardless of the actual computing needs of users. The excess computing power has famously been requested by projects such as SETI@Home and Folding@Home, and been exploited by bad actors for crypto mining. The most basic laptop today used only for web browsing and word processing would be a powerful workstation 20 years ago, when the most basic laptop was also used only for web browsing and word processing (and arguably for more things, as it was all mostly local software).
There is no ceiling to the power of consumer hardware. If it's cheap enough, it will be bought.
I think you missed the point of my message. Web browsing still happens by connecting to data centres, so why are consumer laptops so much more (unnecessarily) powerful today than they were 20 years ago? All the more so given that, at that time, you were running MS Office locally rather than using Office 365 or Google Docs remotely.
Even two or three years people were pointing out "The ChatGPT subscriptions you can buy with $2000 give you much more compute than whatever home setup you come up with" on r/LocalLLM. I did my own elementary school maths and came to the same conclusion.
Yet till this day people still boast how their beefy M4 Pro/Max machine with 32+GB RAM (which is not at all a "normal person's setup" and costs $2000+) runs LLMs smoothly, and "that's the future".
Someone needs to re-learn basic maths and take a walk around Best Buy to understand what "consumer laptop" looks like.
If there end up being useful workflows where you keep stuff running in the background or overnight that's one advantage, compared to a data center that might cut off your access during peak hours or etc.
Think of it like having a graphics card at home versus using a cloud gaming stream? Technically subscribing to GeForce is much cheaper up front than getting a card, but people still do that. So will the audience of people running agents at home be as large as PC gaming? I think that's kind of plausible.
>2. if you buy one for your personal use, you are probably not going to utilize it all the time and it will be idle a lot
I think consumers are primed for that type of behaviour though. I have an iPhone on my desk. It has something like 2-3tflops CPU+GPU, which is double that of the largest super computer on earth when Jurassic Park came out, and is probably more computing power than existed on earth when I was born in the 80s.
I use this device for around 1hr per day to write text messages.
It's inevitable. What might be a prosumer device today priced at 4000$ will be a regular consumer device in 10 years and models only get better.
Local models today are fine for a lot of mundane tasks and will continue to be so. The use cases where paying for frontier models is worth it, will continue to shrink for folks not doing frontier work.
While I agree with that in principle, it is very worrisome that the prices of personal computers, especially of any personal computer that is not a big desktop, have been increasing continuously.
The price of a mini-PC with Intel Panther Lake is at least double in comparison with the price of a mini-PC with Arrow Lake H having similar specifications, and I am talking about barebones, before adding DRAM and SSDs, whose prices have risen even more.
The rise in prices is somewhat obfuscated by the confusing names of CPUs, i.e. some old and new CPUs may seem to be at similar prices and they have similar names, but the new CPU actually corresponds to a lower segment of the market, by having e.g. a smaller GPU and a lower clock frequency, while the CPU model that really corresponds to the old is named such that it seems to belong to the class corresponding to its present price.
As a concrete example of this obfuscation, which may confuse the buyers of laptops or mini-PCs, I have an ASUS 15 Pro with "Core Ultra 5 225H". If I would buy an ASUS 16 Pro now, the corresponding CPU model, the cheapest which is not worse than what I have, would be "Core Ultra X7 358H".
The best open weight LLMs don't run on this computer, or almost any consumer grade computer. Even the memory requirement for Gemma 4 is out of reach for most consumers (by which I mean those who are not on HN). Unless there is some magic that would make high quality LLMs consume no more than 8GB RAM which makes them usable on a 16GB laptop (which is the norm these days), "local LLM for personal computing" is mostly just a myth.
i think a lot of that is for government and enterprise use. even for personal computers themselves (i.e.: laptops) they're usually loss leaders, they don't turn profit. You can run a server (and many do) on laptops, but that didn't replace cloud services or server hosting. You can't store enormous amounts of data on your laptop/phone for the llm to use, or access tools the app dev wouldn't want exposed on untrusted devices.
The whole replacing people angle is just the short term use case the more ghoulish executives are thinking about. In practice, lots of lots of new use cases have been made possible by LLMs. A lot of which can be done locally. But whatever capacity you have locally, they can have more of and for cheaper, and they manage the model instead of you doing it yourself. I think you put it nicely though, their moat will be thinned, and I doubt they'll be as profitable as their funding suggests, but at the same time the demand for them won't go away either. I don't know if OpenAI and Anthropic will be viable, but I'm nearly certain Deepseek is.
The tipping point will be power usage, if a local llm can run the same workload for less power that would be a game changer. Nvidia might get decimated, but even Google and others have moved on from GPUs already, they have faster and more power efficient TPUs. Add to that network bandwidth and availability issues, their moat remains. Also consider that even for graphics capabilities, user devices just don't have a consistent spec to make things like widespread 3d graphics and webgl usage viable. Someone's cheap android phone will never run a local llm reliably,same as it won't a 3d game. even if they have a high-end iphone, network providers aren't always performant as they are in western countries, and then there are people that won't want to install your app or local software, and then browser based exposure of the capability to sites which will have similar hardware spec issues, OS instabilities, competing tabs,etc...
We're hitting the atomic limits of what's possible with minimum feature size in silicon. It's also very hard to remove 1 kW of heat from a laptop, let alone do it quietly or on battery.
My experience over the past decade has been being subsequently burned by being reliant on one provider's ecosystem after another. This is great until Reolink starts doing something shady to pad the bottom line and then it's on to the next.
I wanted the ability to run whatever cameras on a VLAN and own the stack.
I'm guessing that they are using Fargate which is an OSS NVR. It supports a little addon USB stick you can buy for about $30 that will run common computer vision tasks for object detection. Stuff that we've been able to do with WebAssembly and Canvas for a long time now.
Zig is still a moving target with big fundamental changes being made to the language from version to version - nowhere near v1. When rust was at this stage of its development you wouldn’t have been able to name many projects either.
Doesn’t seem like it is in the same adoption realm. I wasn’t aware Ghostty was written in Zig and I’m not aware of any Nim project ever reaching the heights of Ghostty (or indeed Bun). Plus as others state, Zig is still pre-1.0.
Things do look significantly better for Zig adoption-wise than for Nim as far as I can tell.
One thing is sure: if they can be fooled, adults will figure out a way to fool it. (Young enough children might not, but that doesn't help security). And once they're in, their child victims and their parents will be all the more likely to assume they're a child.
Isn’t the real danger now not the ability to find security vulnerabilities, but rather, the ability of anyone to ask an LLM agent to rewrite your open source project in another language and thus work around whatever license your project has?
This is happening quite a lot actually. People just feed an existing project into their agent harness and have it regenerate more or less the same with a few tweaks and then they publish it.
I'm not sure how this works in the legal sense. A human could ostensibly study an existing project and then rewrite it from scratch. The original work's license shouldn't apply as long as code wasn't copy & pasted, right?
What happens when an automated tool does the same? It's basically just a complicated copy & paste job.
A lot of open source projects already have licenses that allow forking and selling the fork, it hasn't been a problem most of the time... there's a lot more to operating open source as a business beyond just shipping the code
> A lot of open source projects already have licenses that allow forking and selling the fork
If we go by the OSI's definition, a project that doesn't allow this is not "open source". So all open source projects -- not just "a lot" -- allow this.
Yep, precisely this. There are many languages out there that would have remained niche if it wasn't for a company sponsoring their development. Go is one of those. Rust too, and it just barely managed to get the critical mass it needed, likely because Mozilla had nowhere near the might of Google.
Nim had a real chance at gaining a foothold, it just needed a company to back it. I think sadly that ship has sailed by now.
reply