It's going to be the shareholders of the first companies to crack AGI, and make human brains fully irrelevant economically. With the trillions of dollars that's going in through both investment and users, it's going to happen. I don't believe the human brain has fundamental magic that will make this impossible.
True AGI would upend society in such a way that I'm not sure that being a shareholder of anything would be meaningful. Perhaps being a pitchfork manufacturer is the winning play in this scenario.
> It's going to be the shareholders of the first companies to crack AGI, and make human brains fully irrelevant economically.
What makes you think if one or two AI labs can do this that the rest (including open model providers) won't be able to follow the same path a few weeks/months later?
Even if you believe in the "Singularity", and believe it is coming soon, I still don't see any reason to believe the Singularity will be... singular. There won't be one clear winner, the race doesn't get called as soon as the first person crosses the line.
None of the AI labs are showing any sign of pulling away to a monopoly or duopoly position, to the contrary the early large leads of OpenAI and Anthropic have all been evaporating.
AI has clear economic value. It still isn't clear at all how the providers of AI will capture that value in a moatless environment with the technology becoming rapidly commoditized.
Because a month later is a month of recursive self improvement at the speed of light. Once a lab catches up, the first lab will be TWO months ahead, then a year ahead, then forever ahead. After a few months of this the differences in absolute terms will be enormous.
If "human brains become fully irrelevant economically" then that brings into question the entire premise of "share holders" and "financial winners".
What even are money, shares, stocks, and finance in a world where human brains are irrelevant economically? No one knows, but betting that "share holders" will be the winners is a highly questionable bet.
I would much more likely bet that "the armed group who manages to control and benefit from the AI through force" will be the "financial winners" more so than "share holders", who tend to not be terribly military minded at least in America.
The AI is likely to control the ability to apply force (see all of the autonomous drone companies). There's a great deal of alignment work being done to ensure that the AI will continue to listen to the shareholders of these companies.
If that fails, who knows what things will look like.
Are you sure that alignment work is aligning with the share holders and not the operators? Or not the creators? Or not the government? Which of these groups should the AI listen to when these groups disagree?
If the AI gets as powerful as you think it might, then the group that figures out the answer to that would have the power, I suppose. or maybe the AI does not listen to any of them and does its own thing. Who knows? Personally, I would not bet the share holders are going to come out "on top" whatever that means.
I think a lot of share holders are finance people, not deeply technical AI people and so odds are the share holders will not really understand the AI enough to be the most likely to control the AI.
To be honest: I don't know for certain, but I'd assume that the people who pay the bills get the strongest alignment. They may not be tech people, but I (so far) haven't got a reason to think that the AI engineers are going behind the backs of their corporate leadership and subverting what they're being asked to do; do you?
(I think it would be a good thing for humanity if they did)
I think right now both the engineers developing AI and the share holders are more focused on beating coding benchmarks and gaining revenue than anything to do with alignment.
The drones don't manufacture themselves, maintain themselves, reload their own ammunition, mine and refine the materials that are used to make them and their ammunition, or operate the power plants needed for all of the above. "AI" isn't going to control diddly squat.
You did not read much about history, did you? Or law. If you embed AI into a gun ... you just created a bomb. It is still you who killed whoever it kills.
Are you making a serious argument that superhuman intelligence is a plausible outcome of training LLMs on everything humanity knows so far? Or are you making the generic assertion that AGI is theoretically possible via means other than emulating the human brain? Because the latter is a strawman (nobody has asserted anything to the contrary), and I have seen no evidence at all to support the former.
I think we can compare the human brain and LLMs on a bunch of capabilities today, and see how we compare. By my reckoning:
- LLMs have better long term memory (they know more than any human) and more working memory (LLMs have fast, uniform access to their whole context window).
- LLMs are faster than we are.
- Humans have online learning (we can do simultaneous learning and inference), giving us advantages in many novel tasks.
- We can learn concepts from far less data. And we can manage our mental context more smoothly.
- We seem to have better world models than current models. AI video just doesn't look right, somehow.
I expect that these remaining weaknesses can be overcome without resorting to human brain emulation. I see no reason to think that current LLMs are at the limit of what technology is capable of.
Because LLMs don't understand anything. That's the tech. They can only predict what they have been trained with and fail daily at the most basic tasks. Granted they can do amazing things, no question there. But they are not "smart".
For example, it seems that even at Fable scale, simple concepts like the passage of time or (gasp) timezones elude them. I live in UTC+10 and with any RFC8339 data LLMs are constantly confused - is it Sunday the 10th or Sunday the 9th, etc. I have tried many solutions for this and every time it finds a way to get it wrong.
How do you square that then? They can do amazing things, but they're also not smart? Do you think its possible to solve Erdos problems without any "smarts"? Can you do it without even understanding mathematics?
I find it very hard to hold the idea that LLMs don't understand anything. They can explain concepts, translate them, simplify them and implement them in code. From the outside, LLMs seem to understands most concepts better than most humans do. Do you understand anything? Couldn't I make the same argument? How would you prove that you understand what a for loop is, or that you know what calculus is? I assume you'd demonstrate your knowledge by using a for loop in a program, or explain calculus back to me. But LLMs can do that too.
> For example, it seems that even at Fable scale, simple concepts like the passage of time or (gasp) timezones elude them.
Funny example, because lots of human struggle with this too. The number of meetings I've had with people in the US! "Lets meet on thursday morning australia time!". Only, they actually meant thursday night US time, which is friday morning australia time. "Oooh that's so weird! Its the next day for you!". ...... Yes, I know.
I think LLMs are just a different kind of intelligence than humans. They're better at some things than us, and worse than others. They can find latent security vulnerabilities in the linux kernel, but struggle to count the Rs in strawberry. They're not as smart as humans in many ways. But we're not as smart as LLMs in plenty of ways too. I didn't find those linux bugs.
What is meant by 'superhuman intelligence'? Certainly it seems to be the case that LLMs are capable of a sort of 'polyhuman' intelligence, in that the same LLM that advances mathematics with a novel proof can add unit testing for a new software feature, design a recipe, and create an SVG of a pelican riding a bicycle. As generalists I'd say they're already 'superhuman'.
> training LLMs on everything humanity knows so far?
That's not all of what we are doing for at least a year, possibly few. LLMs are trained increasingly on generated inputs. Soon human sourced material is going to be rounding error in the process of training.
To clarify, information can be created at will, in automated fashion.
If you add two random numbers and calculate the result and those happened to be numbers noone else ever had idea to add you created a new piece of information. Template is not new, but the piece of information is. And sure, this template might be very simple, too simple, but you can come up with more complex one. And metadata is data. You can create templates in similar manner to how you create new pieces of information using them.
And labs training AI are doing it for years at this point. And it is ever increasing fraction of all training.
> What magic do you think the human brain has that makes it impossible to emulate acceptably?
If I knew, I'd be rich from deploying it onto a substrate for my own AI.
But that doesn't mean that there isn't something there - the current approach seems at odds with how flesh brains work.
I mean, you can power a human brain with 2x bananas for 4 hours, the energy of which might power an H100 for about 20 seconds. It's obvious that there's something different happening.
