I find it amazing that, for all the evidence we have of generative models having some fairly complex internal model of the world, people still insist that they are mere "stochastic parrots" who "don't really understand anything".
From the other side however if you really think about it, our understanding of everything must be stochastic as well. So perhaps this sort of thing yields in many complexities that we are not aware of. How would I know I am not a stochaistic parrot of some sort. I am just neural nets traine on current envrionment while the base model that is dependent on DNA, through evolution and natural selection of the fittest. Same as currently competing LLMs where the best one will win out.
You're not wrong, but the "stochastic parrot" claim always comes with another, implicit one that we are not like that; that there's some fundamental difference, somehow, even if it is not externally observable. Chinese room etc.
In short, it's the good old religious debate about souls, just wrapped in techno-philosophical trappings.
I dont think that's the core of the objection at all. I've never seen it made by people pushing the idea that AGI is impossible, just that AI approaches like LLM are a lot more limited than they appear - basically that most of the Intelligence they exhibit is that found in the training data.
But in which way isn't most of our Intelligence not what is from training data?
1st evolutionary algorithm and then the constant input we receive from the World being the training data and we having reward mechanisms rewiring our neural networks based on what our senses interpret as good?
As someone teaching their five year old to read, I think people way underestimate the amount of training data the average human child gets. And perhaps, since we live in a first world country with universal education, and a very rich one at that, many people have not seen what happens when kids don't get that training data.
It's also not just the qualia of sensation, but also that of the will. We all 'feel' we have a will, that can do things. How can a computer possibly feel that? The 'will' in the LLM is forced by the selection function, which is a deterministic, human-coded algorithm, not an intrinsic property.
In my view, this sensation of qualia is so out-there and so inexplicable physically, that I would not be able to invalidate some 'out there' theories. If someone told me they posited a new quantum field with scalar values of 'will' that the brain sensed or modified via some quantum phenomena, I'd believe them, especially if there was an experiment. But even more out there explanations are possible. We have no idea, so all are impossible to validate / invalidate as far as I'm concerned.
You could leverage the exact same accusation against the other side - we know fundamentally how the math works on these things, yet somehow throw enough parameters at them and eventually there's some undefined emergent behavior that results in something more.
What that something more is is even less defined with even fewer theories as to what it is than there are around the woo and mysticism of human intelligence. And as LarsDu88 points out in a separate thread, there are alternative explanations for what we're seeing here besides "We've created some sort of weird internal 3D engine that the diffusion models use for generating stuff," which also meshes closely with the fact that generations routinely have multiple perspectives and other errors that wouldn't exist if they modeled the world some people are suggesting.
If there's something more going on here, we're going to need some real explanations instead of things that can be explained multiple other ways before I'm going to take it seriously, at least.
> somehow throw enough parameters at them and eventually there's some undefined emergent behavior that results in something more.
But the other side doesn't see it that way, specifically not the "something more" part. It's "just math" all the way down, in our brains as well. The "emergent phenomena" are not undefined in this sense - they're (obviously in LLMs) also math, it's just that we don't understand it yet due to the sheer complexity of the resulting system. But that's not at all unusual - humans build useful things that we don't fully understand all the time (just look at physics of various processes).
> which also meshes closely with the fact that generations routinely have multiple perspectives and other errors that wouldn't exist if they modeled the world some people are suggesting.
This implies that the model of the world those things have either has to be perfect, or else it doesn't exist, which is a premise with no clear logic behind it. The obvious explanation is that, between the limited amount of information that can be extracted from 2D photos that the NN is trained on, and the limit on the complexity of world modeling that NN of a particular size can fit, its model of the world is just not particularly accurate.
> we're going to need some real explanations instead of things that can be explained multiple other ways before I'm going to take it seriously, at least.
If we used this threshold for physics, we'd have to throw out a lot of it, too, since you can always come up with a more complicated alternative explanation; e.g. aether can be viable if you ascribe just enough special properties to it. Pragmatically, at some point, you have to pick the most likely (usually this means the simplest) explanation to move forward.
> But the other side doesn't see it that way, specifically not the "something more" part. It's "just math" all the way down, in our brains as well. The "emergent phenomena" are not undefined in this sense - they're (obviously in LLMs) also math, it's just that we don't understand it yet due to the sheer complexity of the resulting system. But that's not at all unusual - humans build useful things that we don't fully understand all the time (just look at physics of various processes).
You might not, but you don't have to look far in these very comments to be met with woo and mysticism on the emergent phenomena side, either.
> The obvious explanation is that, between the limited amount of information that can be extracted from 2D photos that the NN is trained on, and the limit on the complexity of world modeling that NN of a particular size can fit, its model of the world is just not particularly accurate.
I think the more obvious solution is that it's not modelling the world, because... why would it be? It seems obvious to me that this is a significantly more complex way to handle their given task than every other explanation that does not require them create a model of the world.
> Pragmatically, at some point, you have to pick the most likely (usually this means the simplest) explanation to move forward.
I agree completely with this, which is why the idea that this is modelling the world is absolutely bizarre to me, particularly when our understanding of their training is that it's all weighting for pixel nearest neighbors via ranking how well it does at denoising 2D images.
It's not like de-rendering is something new, either. ShaderMap has existed since the mid 2000s and could get similar results from 2D images without any AI/ML, and we have other models that can generate it without anyone suggesting they model the world, e.g. https://arxiv.org/abs/2201.02279
People like LeCun don't think that even models where the stated goal is world simulation are able to do it, e.g. sora, either - https://twitter.com/ylecun/status/1758740106955952191 - so it seems even more unlikely when that isn't the goal that this has happened due to some emergent phenomenon.
The fundamental difference is qualia, which is physically inexplicable, and which LLMs and neural networks show no sign of having. It's not even clear how we would know if it did. As far as I can tell, this is something that escapes all current models of the physical universe, despite what many want to believe.
How do we know that they don't have qualia? Qualia are by definition private and ineffable. You can kind of sort of infer that other persons have them like you do on the basis of their public responses to them; but that is only possible due to the implicit assumption that other persons function in broadly the same way that you do, and thus qualia give rise to the same visible output. If that assumption doesn't hold, then your ability to infer presence of qualia to reactions (or lack thereof) to them is also gone.
But also, the very notion of qualia suffers from the same problem as other vague concepts like "consciousness" - we cannot actually clearly define what they are. All definitions seem to ultimately boil to "what I feel", which is vacuous. It is entirely possible that qualia aren't physically real in any sense, and are nothing more than a state of the system that the system itself sets and queries according to some internal logic based on input and output. If so, then an LLM having an internal self-model that includes a state "I feel heat" is qualia as well.
> All definitions seem to ultimately boil to "what I feel", which is vacuous. It is entirely possible that qualia aren't physically real in any sense, and are nothing more than a state of the system that the system itself sets and queries according to some internal logic based on input and output. If so, then an LLM having an internal self-model that includes a state "I feel heat" is qualia as well.
Is it "possible"? Absolutely. However, I have no means by which to measure, as you say, where at least with humans and animals I posit that their shared behaviors do indicate the same feeling, so I have some proof.
With an LLM, the output is a probability distribution.
Moreover, if it is as you say it is, then computers have qualia as well, which is scary because we would be committing a pretty ethically dubious 'crime' (at least in some circumstances).
Again, I just don't see it. Anything is possible, as I said, but not everything is as likely, by my estimation. And yes, that is entirely how I feel, which is as real as anything else.
I have several thoughts on the concept of 'hallucination'. Firstly, most people do it regularly. I'm not sure why this alone is indicative of not understanding. Secondly, if we think about our dreams (the closest we can get, in my view, to making the human brain produce images without physical reality interfering), then actually we make very similar hallucinations. When you think about your dreams and think about the details, sometimes there are things that just kind of happen and make sense at the time, but when you look further, you're kind of like 'huh, that was strange'.
The images we get from these neural networks are trained on looking pleasing, for some definition of pleasing. That's why they look good on the whole, but get into that uncanny valley the moment you go inspecting. Similar to dreams.
Whereas obviously real human perception is (usually) grounded in reality.
No, you can have abstract representations of a concept which would fit understanding in a certain sense of the word. You can have “understanding” of a concept without an overarching self aware hypervisor. It’s like isolating a set of neurons in your brain that represent an idea.
Yes I think so. Understanding would involve understanding that it's a short form for adding. Remembering multiplication tables allows you to use math but it doesn't infer understanding.
Training themselves is necessary. All learning is self learning. Teachers can present material in different ways but learning is personal. No one can force you to learn either.
You can check if a model (or a kid) understands multiplication by simply probing them with questions, to explain the concept in their own words, or on concrete examples. If their answers are robust they understand, if their answers are fragile and very input dependent, they don't.
"To understand" is one of those poorly defined concepts, like "consciousness", it is thrown a lot in the face when talking about AI. But what does it mean actually? It means to have a working model of the thing you are understanding, a causal model that adapts to any new configuration of the inputs reliably. Or in other words it means to generalize well around that topic.
The opposite would be to "learn to the test" or "overfit the problem" and only be able to solve very limited cases that follow the training pattern closely. That would make for brittle learning, at surface level, based on shortcuts.
> It means to have a working model of the thing you are understanding, a causal model that adapts to any new configuration of the inputs reliably
The weasel word here is "reliably". What does this actually mean? It obviously cannot be reliable in a sense of always giving the correct result, because this would make understanding something a strict binary, and we definitely don't treat it like that for humans - we say things like "they understand it better than me" all the time, which when you boil it down has to mean "their model of it is more predictive than mine".
But then if that is a quantifiable measure, then we're really talking about "reliable enough". And then the questions are: 1) where do you draw that line, exactly, and 2) even more importantly, why do you draw the line there and not somewhere else.
For me, the only sensible answer to this is to refuse to draw the line at all, and just embrace the fact that understanding is a spectrum. But then it doesn't even make sense to ask questions like "does the model really understands?" - they are meaningless.
(The same goes for concepts like "consciousness" or "intelligence", by the way.)
The reason why I think this isn't universally accepted is because it makes us not special, and humans really, really like to think of themselves as special (just look at our religions).
> It obviously cannot be reliable in a sense of always giving the correct result, because this would make understanding something a strict binary
Our capacity to make mistakes does not necessarily equate to a lack of understanding.
If you’re doing a difficult math problem and get it wrong, that doesn’t necessarily imply that you don’t understand the problem.
It speaks to a limitation of our problem solving machinery and the implements we use to carry out tasks.
e.g. if I’m not paying close enough attention and write down the wrong digit in the middle of solving a problem, that could also just be because I got distracted, or made a mistake. If I did the same problem again from scratch, I would probably get it right if I understand the subject matter.
Limitations of our working memory, how distracted we are that day, mis-keying something on a calculator or writing down the wrong digit, etc. can all lead to a wrong answer.
This is distinct from encountering a problem where one’s understanding was incomplete leading to consistently wrong answers.
There are clearly people who are better and worse comparatively at solving certain problems. But given the complexity of our brains/biology, there are myriad reasons for these differences.
Clearly there are people who have the capacity to understand certain problems more deeply (e.g. Einstein), but all of this was primarily to say that output doesn’t need to be 100% “reliable” to imply a complete understanding.
Indeed, but I wasn't talking about mistakes at all, but specifically about the case when "A understands X better than B does", which is about their mental model of X. I hope you won't dispute that 1) we do say things like that all the time about people, and 2) most of us understand what this means, and it's not just about making fewer mistakes.
Ah, thanks for the clarification; I think I misread you here:
> The weasel word here is "reliably". What does this actually mean? It obviously cannot be reliable in a sense of always giving the correct result, because this would make understanding something a strict binary
What did you mean by "it cannot be reliable in a sense of always giving the correct result", and why would that make understanding something a strict binary?
I do agree that some people understand some topics more deeply than others. I believe this to be true if for no other reason than watching my own understanding of certain topics grow over time. But what threw me off is that someone with "lesser" understanding isn't necessarily less "reliable". The degree of understanding may constrain the possibility space of the person, but I think that's something other than "reliability".
For example, someone who writes software using high level scripting languages can have a good enough understanding of the local context to reason about and produce code reliably. But that person may not understand in the same way that someone who built the language understands. And this is fine, because we're all working with abstractions on top of abstractions on top of abstractions. This does restrict the possibility space, e.g. the systems programmer/language designer can elaborate on lower levels of the abstraction, and some people can understand down to the bare metal/circuit level, and some people can understand down to the movement of atoms and signaling, but this doesn't make the JavaScript programmer less "reliable". It just means that their understanding will only take them so far, which primarily matters if they want to do something outside of the JavaScript domain.
To me, "reliability" is about consistency and accuracy within a problem space. And the ability to formulate novel conclusions about phenomena that emerge from that problem space that are consistent with the model the person has formed.
If we took all of this to the absurdist conclusion, we'd have to accept that none of us really understand anything at all. The smaller we go, and the more granular our world models, we still know nothing of primordial existence or what anything is.
Neither would I. But if someone who can translate a French sentence into English preserving the meaning most of the time, I think it would be reasonable to say that this person understands French. And that is what we're talking about here - models do produce correct output much of the time, even when you give them input that was not present in their training data. The "stochastic parrot" argument is precisely the claim that this still does not prove understanding. The "Chinese room" argument takes it one notch further and claims that even if the translation is 100% correct 100% of the time, that still doesn't prove understanding.
How many people years existed between the time people discovered multiplication and then exponentiation? Did it happen in wall clock time in a single generation?
> How many people years existed between the time people discovered multiplication and then exponentiation?
We don't know, because these concepts were discovered before writing. We do know that far larger jumps have been made by individual mathematicians who never made it past 33 years of age.
While I agree with the hidden statement of the utility in online reinforcement learning here, it should be pointed out that for some snapshot of a system, which may have already been trained with a large amount of data - the structure of the learned space and the inferences within it seem like a reasonable definition for 'understanding'.
I don't know many people that would suggest that knowing about ones family structure, such as their mom/dad/uncle, and how their history relates to them, is required to be _completely reconstructed_ every time they interact with their environment somehow from first principles.
Online reinforcement learning can have large merits without resorting to stating it's required for learning. Just as self awareness can occur independently of consciousness can occur independently of intelligence is independent of empathy and so on. They are all different, and having one component doesn't mean anything about the rest.
I don't buy that definition of understanding. Yours is closer to learning. If you had an accident and lost the ability to learn new things, you would still be able to understand based on what you have learnt so far. And that's what these models are like. They don't retrain themselves because we haven't told them to.
