We are living in a ZIRP-like era where builders at the fastest pace layer have misattributed their velocity to exponential gains in model capability. In fact, they are surfing on decades of careful effort to build a robust foundation of highly reusable software libraries.
This strategy will seem to work really well until the economy that enabled that foundation to form is hollowed out. Then, there will be a reckoning (but we will have no choice but to march forth from there).
It's not just software libraries. Specs, applications (the browser!), expectations, device integrations, operating systems, etc. So much that starting from scratch seems impossible.
I'm not agreeing or disagreeing with you, but my brain cannot comprehend how machines can advance such interconnected systems while keeping humans in focus.
Perhaps I shouldn't have watched the Animatrix again.
> This strategy will seem to work really well until the economy that enabled that foundation to form is hollowed out. Then, there will be a reckoning (but we will have no choice but to march forth from there).
There will only be a reckoning if models don't get much better.
If they do get much better you can just have them refactor, fix bugs in, or replace the existing codebase.
The concept of tech debt is sort of meaningless if you anticipate intelligence gains in models to continue.
Its already speeding up human decision processes, and while ethics / alignment may seem unique to humans we also see normative expressions in monkeys or apes (like the experiment where one is given a grapes, the other cucumber).
A lot of ethics is based on symmetry: symmetric relations, equal rights, equal voting power, ... symmetries sound rather mathematical if you ask me, and decision structures have historically been pressed towards democracy (or at least depiction of it). One could say that modeling humanity as an empire with a king, ignores the will of sometimes hungry farmers with pitchforks. To prevent the occasional "implicit democracy" (royaltycide), it turned out in the interest of the king to recognize the powers of those farmers, and to formalize it in the decision making process. Or at least pretend to.
I believe machines will be able predict the preference sentient creatures would prefer in terms of decision structures, but I don't believe it will be able to predict (without human exposition) those novel preferences that stem not from sentience but from being specifically human properties (i.e. irritants which are quasi universal for humans, etc.), some of them humans know how to make predictions for (we can run expensive simulations modeling what happens when protein X is exposed to substance Y, and then make heuristic predictions of the effect on a full human in a realistic environment). So at a fundamental level I agree: machine learning models are not guaranteed to help much in predictions concerning entirely unexplored territory, neither by humans nor by natural selection. But it will definitely be capable of replacing the average human job, which doesn't involve consensual exploration outside of the homeostasis required in the implicit job description, that seems entirely automatable, regardless if its physics, mathematics, (harder than computer science), let alone programming.
It won't be able to magically systematically correctly predict out of distribution datapoints, it could only explore it like humans could by trial and error.
For everyone claiming that this is a trope of LLM text because it is a trope in the training data: how do you know this trope doesn't emerge during RLHF?
> What is reality? Obviously, no one can say, because it isn’t words. It isn’t material—that’s just an idea. It isn’t spiritual—that’s also an idea; a symbol. Reality is this: [GONG]. You see? We all know what reality is, but we can’t describe it. Just as we all know how to beat our hearts and shape our bones, but cannot say how it is done. - Alan Watts
Brand new industry, massive capital, dropping inference costs, increasing availability of compute, cost centers / subsidized subscriptions are common in SaaS, heavy competition, no public information on actual utilization rates.
How much is Waymo burning a year? 3B on 300M ARR? Anthropic is what 5B on 20B ARR? Waymo is 3x older. Why don't we hear such confident statements about how subsidized their rides are?
It's one thing to speculate it's another to parade it as fact. Even if the S1 reveals an unprofitable business today, you can still only claim it's unlikely.
> How much is Waymo burning a year? 3B on 300M ARR? Anthropic is what 5B on 20B ARR? Waymo is 3x older. Why don't we hear such confident statements about how subsidized their rides are?
We do. We hear it less often because no-one is talking about how Waymo changes how we all need to work or whatever, that's all.
Do people commonly argue Waymo isn't subsidizing rates?
Also, we do have some evidence for my position:
- We know that the consumer Claude plans provide _way_ more tokens than you could get if you were paying API prices. This is a huge part of why Anthropic's limits on other harnesses for subscription customers is such a big deal. So either their profit margin on API tokens is absurdly high, most consumer subscribers don't come anywhere near their rate limits, or they're losing money on the consumer subscriptions.
- It appears that complains about people running into rate limits are common, which suggests the "consumers usually don't use much of their subscription" explanation is incorrect.
- We also know that Anthropic has just become profitable, almost certainly driven mostly by enterprise customers. This rules out the "they make a very high profit margin on the API" explanation, since if that was the case they'd likely have been profitable much earlier.
Taken together, I think the case that their consumer subscriptions lose them money on net is pretty strong, even though their enterprise subscriptions (and API pricing) does make them a profit.
> I think the case that their consumer subscriptions lose them money on net is pretty strong, even though their enterprise subscriptions (and API pricing) does make them a profit.
To be clear I'm not arguing against this position, just questioning the confidence with which people claim that the current consumer subs are not a sustainable offering and a merely temporary.
Whether sustainability is achieved by raising prices or hoping that costs can be brought down, you have to acknowledge that the status quo is unsustainable. If it were sustainable nether change would be necessary.
With respect, you were manipulated (either by founders or by investors). Startups leverage employees' pro-social leanings to make them feel good about a fundamentally anti-social enterprise.
AI has pushed me to arrive at an epiphany: new technology is good if it helps me spend more time doing things that I enjoy doing; it's bad if it doesn't; it's worse if I end up spending more time doing things that I don't enjoy.
AI has increased the sheer volume of code we are producing per hour (and probably also the amount of energy spent per unit of code). But, it hasn't spared me or anyone I know the cost of testing, reviewing or refining that code.
Speaking for myself, writing code was always the most fun part of the job. I get a dopamine hit when CI is green, sure, but my heart sinks a bit every time I'm assigned to review a 5K+ loc mountain of AI slop (and it has been happening a lot lately).
I agree. I’m using copilot more and more as it gets better and better, but it is getting better at the fun stuff and leaves me to do the less fun stuff. I’m in a role where I need to review code across multiple teams, and as their output is increasing, so is my review load. The biggest issue is that the people who lean on copilot the most are the least skilled at writing/reviewing code in the first place, so not only do I have more to review, it’s worse(1).
My medium term concern is that the tasks where we want a human in the loop (esp review) are predicated on skills that come from actually writing code. If LLMs stagnate, in a generation we’re not going to have anyone who grew up writing code.
1: not that LLMs write objectively bad code, but it doesn’t follow our standards and patterns. Like, we have an internal library of common UI components and CSS, but the LLM will pump out custom stuff.
There is some stuff that we can pick up with analysers and fail the build, but a lot of things just come down to taste and corporate knowledge.
I've been using it to do big refactors are large changes that I would simply avoid because, before, the benefits don't outweigh the costs of the doing it. I think half the problem people have is just using AI for the wrong stuff.
I don't see why it doesn't help with reviewing, testing, or refining code either. One of the advantages I find is that an LLM "thinks" differently from me so it'll find issues that I don't notice or maybe even know about. I've certainly had it develop entire test harnesses to ensure pre/post refactoring results are the same.
That said, I have "held it wrong" and had it done the fun stuff instead and that felt bad. So I just changed how I used it.
I read a lot of AI generated code these days. It makes really bad mistakes (even when the nature of the change is a refactor). I've tried out a few different tools and methodologies, but I haven't escaped the need to babysit the "agent." If I stepped aside, it would create more work for me and others on the backend of our workflow.
I read anecdotes of teams that push through AI-driven changes as fast as possible with awe. Surely their AIs are no more capable than the ones I'm familiar with.
I read all the code and it sometimes make mistakes -- but I wouldn't call it really bad. And often merely pointing it out will get a correction. Sometimes it is funny. It's not perfect but nothing is perfect. I have noticed that the quality seems to be improving.
I still think whether you see sustained value or not depends a lot on your workflow -- in what you choose to do or decide and what you let it choose to do or decide.
I agree with you that this idea of just pushing out AI code -- especially code written from scratch -- by an AI sounds like a disaster waiting to happen. But honestly a lot of organizations let a lot of crappy code into their code-base long before AI came long. Those organizations are just doing the same now at scale. AI didn't change the quality, it just changed the quantity.
Arguably the ad business is to blame. It created a perverse incentive. They maximized pay-to-play. The losers were authors that previously published on a passion budget (and would/could never pay for ads). AI is just the last nail in the coffin.
This strategy will seem to work really well until the economy that enabled that foundation to form is hollowed out. Then, there will be a reckoning (but we will have no choice but to march forth from there).
reply