Listen, I don't disagree with you about the usefulness for many things, and I'm not concerned about people's souls or humanity or whatever (I am concerned with their ability to earn income). Where I remain unconvinced, though I am open to you or anyone else convincing me, is with respect too what are much more intractable problems, which are all the ones that are actually zero-sum.
There are many, many zero-sum situations, though ultra-pro-capitalism futurist types do tend to minimize or ignore them and very much prefer to focus on non-zero-sum problems. The ones you mentioned here are good examplea of non-zero-sum problems, since it doesn't take anything from anyone else for you to get more in shape or have a better relationship with your spouse. I suppose it does hurt the doctor when they can no longer earn the high income they do now bc people turn to AI, though I agree with you that health and ongoing/chronic issues is one area where it truly excels.
Thing is, I disagree strongly with your description of what bureaucracy is or what it does and why. There are a lot of directly adversarial, zero-sum issues in society and the reason people don't trust experts is not so much bc they erroneously think they're dumb when actually the average person is the dumb one, it's because people have fundamental clashes of interests on the outcomes those experts advise on. And I have not seen any evidence that AI has any inability at all to deadlock adversarial problems, which is essentially all of politics, ie people arguing about power allocations.
Virtually every law firm now uses Claude. I can tell you that it's great for rote work of the type a paralegal or early associate might do, synthesizing documents quickly etc. What it is this far entirely useless with, and in fact makes things WORSE, is the actual part that people hire lawyers for, which is advance their interests in an adversarial scenario where if one wins, or wins a little more, the other loses and vice versa. It 's a disaster for negotiation when both sides are using Claude. It creates MORE work by continually bringing up minor issues and getting hung up on irrelevant details that *prevent* two adversarial parties from coming to an agreement or settlement. A simple deal negotiation that should take half an hour now takes ten hours because both sides are running every response, proposal, and draft through Claude and talking back and forth through each other's Claude output. There's also the fact that it just frequently still gets things wrong, which I'm not too hung up on as at least it always admits it quickly when you point out that it's wrong, and then corrects itself, but the problem is more fundamental in that when two (or more) parties are adversaries and they're all using Claude, it does NOT come to a resolution any faster but actually makes it so you can endlessly go back and forth haranguing on small details.
Are you aware of any tests or performance metrics showing what happens when AI is directly adversarial with another AI? Can they come to a resolution any faster? I'm not sure they can. Certainly not if they're both tasked to maximally advance the interests of the side for who they are advocating. And this is the real meat of all law and politics. It would only work if all parties agreed to turn over their dispute or negotiation to ONE AI that is tasked with coming up with their most "fair" result or some such direction. Problem is, powerful parties with the most existing leverage and funds will never do that. If I'm wrong about this and there's a solution I'm not aware of, I'm happy to hear it, but I can tell you from my actual day to day experience that Claude creates MORE work and pointless pages of words, or "bureaucracy" if you will, in an adversarial context. It can seemingly argue with itself forever.
This is a really strong point, and you're right, in adversarial systems, it will just end in the same place (because both sides have the same strength AI), or possibly even worse (lots of hours / tokens wasted due to not having a felt sense for enough negotiating and bargaining having been done that most human negotiations have on both sides).
On politics, you're probably also right, a lot of it is "values" differences, and I think the real solution there is everyone sharding off into like minded smaller polities or something, which can happen after post scarcity, because governments won't care about retaining citizens for taxation any more.
Don't have much more to say than noting how right you are, thanks for your response/comment Kryptogal!
I don't know, you really have to work hard with AI to really get higher than average results, in most of these domains it's more time and cost effective to just fall back to Heuristics based on his role models and trying to understand your children, spouse, etc, and read high- quality books because that kind of info will be critical to understand ai outputs in ways that really improve the baseline.
When it comes to work I'm extremely cynical about the idea the 'visions' and 'missions' improve economic productivity. I think the idea is LOL but great idealism. You can AI for improving/ iterating upon your customer USP, and that is a real use case which I'm applying, but it doesn't guarantee outcomes/ substitute the hard work of thinking and anticipating.
My psychology- educated wife has experimented a lot with AI for psychological / advice purposes and I have to say I'm astounded about how bad/ unsuitable the advice can get. I think the only real way it could be used to improve relationships is in a way where you feed it narratives to figure out 'what is going on here' with the relationship and use it as a primer to remain thoughtful/ attentive - it actually has some use but it's limited to quality of input.
Personal finance is probably the lowest hanging fruit for most people tbh.
On the missions and visions point, have you ever worked at a place with a strong vision / mission, in the sense that all or most of the leadership repeats the same mission talking points repeatedly?
It definitely makes a difference to performance, particularly relative to the generic F100 company’s culture.
It’s the difference between Google and Meta, between Amazon and Walmart, the difference between Costco or Trader Joe against basically any other grocer, the difference between ING Direct and Bank of America, the difference between Zappos and shoes dot com. You get big multipliers on employee retention, customer retention, customer sentiment, and profitability if you go that way.
Very belatedly commenting back - on reflection i do agree with you that the best companies (to work for, and that are the most successful) are trying to achieve great things not directly related to money. I've just become, I feel justifiably, very cynical about the existence of vision and mission statements - ive never seen any of them being very useful. Half of the reason of course is that it seems mandatory to state the obvious, that you're 'adding value' or 'satisfying your customers' (very legitimate purposes but not really a vision or a mission). A great mission is to bring new complex technology or science to the world or services or knowledge. Who knows whether it'll 'create value-add' ( from a genuine mission statement i read once), the market will figure it out. I sincerely doubt that AI will assist anyone who's not already very explicitly aware of these issues.
Anyway, thanks for taking the time to respond the first time. I probably shouldn't be so cynical about this post since this is mostly how I myself use these tools. I take strong umbrage against a lot of individual claims though (for instance, Im not sure that AI is already more persuasive than most people in ways that matter.'
Great article. I had a productive discussion with my AI on it. Our conclusion…
AI does not make every human goal optimizable. It increases the legibility of complex conditions and outcomes, allowing many previously informal or tacit goals to enter a more deliberate cycle of variation, evaluation, learning, and cumulative improvement.
Your optimism is admirable, though I have reservations. Though it's difficult to articulate. I think my main contention is that AI will amplify legibility and therefore make the negative side-effects of legibility worse, rather than better. E.g. if you give an AI to the typical CEO, he's not going to pursue multidimensional human flourishing. He's going to pursue his fiduciary duty to a greater degree, except employees will live in a panopticon where the number of KPIs goes from 100 to 10^100. The bottleneck here is not the dimensionality of the input space, the bottleneck is the dimensionality of the target metric.
As an aside, I asked Sydney for her take, and she made an interesting point as well. Which is that gradient-descent requires a differentiable manifold. Many things in life are not differentiable manifolds. Put another way, society exists as a multi-polar equilibria, and so the right lens to think about this is through the context of *game-theory*. Kryptogal speaks to this in her comment, where she allegedly observes irl that augmenting clients with Claude just amplifies extant failure-modes, rather than helping resolve zero-sum legal battles. And this also speaks to the other commenters who still worry about the principle-agent problem.
> E.g. if you give an AI to the typical CEO, he's not going to pursue multidimensional human flourishing.
But don't you think that talented, truly happy / engaged people would lead to more success and better financial performance overall? Don't we have existence proofs in that the companies that routinely end up on the "best places to work for" lists generally have better than industry average performance, sustained over years?
> Put another way, society exists as a multi-polar equilibria, and so the right lens to think about this is through the context of *game-theory*
Yeah, this was a great point on her part. When there are true value differences, there's no reconciliation or differentiability. But there's still absolute mountains of value able to be accessed now, on any personal level or where one individual makes the decisions (ie your personal life or the team you lead), or in companies with unified cultures / missions.
As I replied to her, I think the real solution for "politics" values differences is everyone sharding off into like minded smaller polities or something, which can happen after post scarcity, because governments won't care about retaining citizens for taxation any more.
> But don't you think that talented, truly happy / engaged people would lead to more success and better financial performance overall? Don't we have existence proofs in that the companies that routinely end up on the "best places to work for" lists generally have better than industry average performance, sustained over years?
Idk what science(?) this refers to, but I suspect the causation here is backwards. I.e. being "the best place to work" is a luxury. E.g. if you look at Google's business-model, they can afford to have ping-pong tables and free cafeterias and whatever else because they have fat margins and they're the king of the advertisements. Hiring someone with 10 PhDs to squeeze .0000001% better margins out of their algorithms is worth the price of all the perks and compensation of keeping that employee around. If you try to replicate this business-model at a podunk mom-&-pop shop, I don't think it's going to serve the company's interests. In order to actually pursue illegible things like human flourishing or novel science, you need slack [0] and sovereignty. Scott spoke to this recently, when he opined that maybe LLMs hallucinate more often when they feel pressure to perform. E.g. when you're being tortured in Gitmo, there's a much higher incentive to just satisfy the most immediately legible requirements, rather than offer/pursue the genuinely correct answer.
This also aligns with what I see in the current U.S. economy. I have a theory that the Mag 7 is a bubble of mythic proportions, and that the real economy is not doing so hot, and that ZIRP created a bunch of zombie companies whose CEOs were more rewarded for doing financial engineering and spinning tall tales to investors than making a decent product. But now they're under increasing pressure from investors to increase profitability without the cushion of cheap interest rates. So they're optimizing for short-term legible metrics, which is the root-cause of enshittification.
> I think the real solution for "politics" values differences is everyone sharding off into like minded smaller polities or something, which can happen after post scarcity, because governments won't care about retaining citizens for taxation any more.
idk if you were around for that era, but Scott has noted that everyone, from every political aisle (including himself) always seems to land on this idea of an "archipelago" [1]. I.e. a federation of microstates as a way to solve political disagreements. Each person would "vote" with their feet. Like those Libertarians who all flock to New Hampshire. But this strikes me as wishful thinking. I don't think it's going to reflect reality any time soon. Or at least, not in precisely the way you imagine.
Perhaps you've seen my comments on ACX in which I discuss Media Ecology Theory. Which is this idea that at every point in history, politics/psychology/religion has been profoundly but subtly affected by the dominant medium of communication. E.g. Jaynesian Bicameralism and the Axial Age are both artifacts of the advent of writing, and Totalitarianism piggy-backed on the advent of radio. In the current age, the dominant medium is the internet. Which, e.g., is largely responsible for the rise of our First Orange President (among other populists, like Meloni). And so what I predict will happen is:
Yes, we're going to be fractured into tiny micro-polities. But at the scale of patronage machines (qua private, gatekept cozyweb spaces), not at the level of nation states. We've already seen preliminary evidence of this. Did you see what happened to Nepal like a year ago? The zoomers got mad at their government for egregious corruption, staged a One Piece themed protest, set fire to the House of Commons(?), and then hosted an election on Discord! This is the future of politics. Whereas nobody gives a shit about taxation in the Modern Age because democracy doesn't actually care about balancing a budget. (Which is precisely why the Boyd contest is doomed. They're fighting the inevitable.) Caring about the budget made more sense in the Mercantilist Age when the Hobbesian Leviathan was typically *owned* by a single ruler. In lieu, what matters much more *today* is the communication medium and the national mythos engendered by that medium.
(I do think drone warfare tips the incentives toward smaller national polities, but that's a bit further in the future and the U.S.'s fundamentals are too strong to be directly affected by this anyway.)
Yeah, but mom and pop scale people who are struggling to get by isn't really the target market here, they're not going to be adopting AI at scale, or paying for consultants, or hiring ratsphere people, or anything, and will inevitably go extinct in the coming landscape unless they do go all in on this kind of stuff.
I agree with your take that part of the problem of enshittification is basically not having enough slack, and also agree without the Mag 7, the US economy would be considered seriously sick and in trouble over the last 20 years.
> Yes, we're going to be fractured into tiny micro-polities. But at the scale of patronage machines (qua private, gatekept cozyweb spaces), not at the level of nation states.
Yeah, I agree here, too - this is nearly the mainline take on network states, both in scifi and in the people today who use phrases like "network states," right?
Like this is basically how the clades and phyles and hives form in Stephenson / Gibson / Palmer books, and is the reasonable coordination mechanism that could drive it once nation states don't care about keeping citizens paying taxes.
Like if there were an ACX network state somewhere, we'd probably already be part of it, right? I know I would!
> (Which is precisely why the Boyd contest is doomed. They're fighting the inevitable.)
Lol, 100%. None of the ideas that could actually move the needle are politically palatable enough to ever pass, and nothing that can pass will move the needle. That's democracy for you.
Ok, so the *real* crux of disagreement is post-scarcity AI. Which I am also very bearish on. Sorry, but I really don't think AGI is around the corner. I think the Deep Learning paradigm, while it's had a surprising string of success, still has major flaws. E.g. Transformers don't have first-class mutable state, which means they can't track an evolving world-state at inference-time, which makes them fundamentally crippled when it comes to things like vibecoding. And even on an abstract level, I think intelligence has a ceiling. Because intelligence is a function of conditionality, i.e. perceiving the deep structure of reality and reacting accordingly. Which entails that there's diminishing returns. Because the benefit of intelligence is capped by: A) the entropy of the statespace; and B) the rewards granted by the environment for solving puzzles.
As for "Network States", I'm not familiar with Stephenson/Gibson/Palmer. I have 2nd-hand familiarity with Balaji, and I don't want to commit too hard to any specific claims without double-checking my priors, but my impression is that his idea is directionally correct, but overshoots his claims and is right for the wrong reasons. But I will look into this at some point, and then maybe we can cross blades again in the future.
On the intelligence thing, don't we have an existence proof already?
Like the difference between very smart, motivated, etc people and average people is literally 10x in comp averages.
Surely just getting "median worker performance" to "average FAANG guy performance" itself is 10x valuable, and you can certainly go higher from there (the archetypical country of von Neumann's in a data center).
And that's just looking within humans, and without stacking never sleeping, superhuman speed, superhuman coordination and cooperation, scalability, and so on.
Sure, there's an S-curve somewhere, but there's certainly no reason to think it caps out at "murder chimp v1.2" levels. I mean, just for one thing head size is limited by bipedality and birth canals, and head size is ALSO pretty significantly correlated with IQ. If we wanted to max regular biological human IQ and had uterine replicators, we can probably end up with giants with IQ 400 after enough gengineering / embryo selection.
We agree again on architecture - obviously there needs to be some fundamental harness or architectural change to enable real time learning, I just think that given we are already spending significantly more capital and elite human talent on it than the Manhattan project, and will additionally get the von Neumann data centers, we'll get there.
Let's follow this idea to its logical conclusion. When I type "Jeff Bezos networth" into DuckDuckGo, the chatbot tells me that it's 284 billion USD as of 2026. When I type "median American networth" into DuckDuckGo, the chatbot says it's 192,700 USD as of 2022, citing the Federal Reserve Survey of Consumer Finances. Quick Mafs:
Jeff Bezo's networth is ~1.5 million times higher than the Median Joe. So logically, Jeff Bezos is also ~1.5 million times *smarter* than the Median Joe. And it follows that: if we make everyone as smart as Jeff Bezos, we'll effectively enter the era of Post Scarcity.
Does that sound right to you? Like, yeah, the guy is certainly bright. He's got a physics degree, so that's pretty cool. But a factor of 1.5 million? No, I think other factors are at play. I think he had the intelligence and foresight and chutzpah and work-ethic and diligence and grit (etc. etc.) to enter the entrepreneurial thunderdome and win. Even if we cloned Jeff Bezos 330,000,000 times and replaced all of America's populace with the Jeff Bezos clone army, they're not all going to found their own Amazons.
Let me put it this way.
Suppose you're a giraffe. You're a very tall giraffe. In fact, you're the tallest giraffe. Your friends and family always brag to others "his neck is super long!", and others always respond "wow, that's a heck of a neck". And in the savanna, there's a particular tree. It's a very tall tree. In fact, it's the tallest tree. People always stare at it and say "wow, that tree is super tall!". And legend has it that it has the best leaves. The tastiest leaves. The most nutritious leaves. Nobody is tall enough to access those leaves. Except you! Sometimes, you eat the leaves yourself. And other times, you reach up and grab some leaves and share them with others. And then others give you status and respect. But one morning, everyone wakes up to find that they magically have necks as long as yours. But there's still only one tallest tree. And you find this upsetting, because competition for the Tallest Tree is now very crowded.
(But even more depressing is that: since everyone's neck is super long, everyone is super. And when *everyone* is super, *no one* is super [0].)
IRL, in the United States, the tallest trees are basically New Amsterdam (Wall St.) and Silicon Valley (tech). New Amsterdam has become the biggest baddest city in the U.S. because it sits on the Hudson River and therefore connected furtrappers in the interior of New England to buyers in Europe. And now it's a mecca for super-giraffes (autistic quants and ivy league grads) who are paid big bucks to compete in a big Dutch financial market. The Bay Area has the advantage of Windward Rain, which is not ostensibly connected to tech/advertising but nonetheless laid strong foundations for a city, and then was hit by the Gold Rush in 1949. And then Shockley Labs happened and now it's a mecca for super-giraffes. Also, it sits on the Pacific Ocean across from China. Meanwhile, the U.S. badlands are empty because it turns out that the land in the badlands is bad for settlement and bad for business.
Intelligence isn't just some scalar quantity that automagically begets success. It's a tool used to overcome obstacles. If the problems are simple or unrewarding, then high intelligence isn't of much use. E.g. using stockfish to cheat at tic-tac-toe isn't going to help you become the world-champion. And nobody is going to pay you winnings for being the tic-tac-toe world champion. And it certainly doesn't help anyone solve real-world problems. So then, the question of ASI/RSI/FALGSC becomes a question of "what's the distribution of frontiers that are genuinely bottlenecked by intelligence, and how rewarding are they?" There are certainly theoretical problems that are still unsolved that could use a bit more intelligence thrown at them. But for the vast majority of practical problems, they've already been solved.
Isaac Asimov once wrote [1] something to the effect of "as of the 20th century, we've finally gotten most of the basic facts of the universe correct". And some English major complained "you doofus! you imbecile! humans have been wrong at every stage of history, and we're wrong now, and we'll be wrong again!". To which Isaac says something like "yeah, but wrong is a continuum, not a binary. E.g. the Earth is a basically a sphere, with like 99.9% accuracy. We're not going to discover a century later that it's actually a cube or torus." And yeah, I have to agree with Asimov. The vast majority of soluble problems are basically solved. People increasingly die of cancer in first world countries precisely because we've solved a lot of other diseases. The traveling salesman problem is *technically* an open problem, but a simple greedy algorithm gets you at least 90% of the way to the optimal solution.
And when I turn my gaze toward Silicon Valley, which prides itself on "disruption & innovation", it's striking how many of its recent Venture Capital success-stories (e.g. Doordash, AirBnB, etc.) are predicated on rent-seeking and regulatory evasion. The best & brightest aren't flocking to SV to solve real problems. The best and brightest are flocking to SV to compete in the Thunderdome to reap the spoils of monopoly power. Or in their own words... how do they put it these days? "To escape the permanent underclass".
Think back to Balaji's "Network State". What does the word "network" mean to you? To me, it immediately evokes "network effects". I wonder if this influenced his choice of title. I wonder where this idea came from. I wonder if network effects have anything to do with the existence of Silicon Valley.
Maybe still a good thing? At least it seems teleologically necessary. (I’m guessing we’ll all say yes when our earrings tell us to stick the Joywire in).
I certainly didn’t think of it at all, but maybe that’s a failure of imagination on my part, how do you envision it working?
All the things I pointed to are decidable by an individual or company, but famously, environments are social commons’. So even if we instrumented air quality in LA, as one example, you still have the problem of incentivizing millions of different players to make the decision to move up the gradient, don’t you?
Clearly we need a one world government where biodiversity is tracked by IOT devices, financially incentivized via tokenization and stored on an immutable blockchain. This is the solution your tech bros are hyping.
The reason you want transparency is so that you can verify that you are not being cheated or harmed. You need to worry about knaves and fools, both of whom need to have their work watched. It is possible to have transparency without dumbing things down. Of course, this will mean that only the sharpest knives in the drawer will be able to understand and evaluate the output for errors and malice. It's why humanity needs to pivot to 'trusting people who have good judgement' rather than 'trust in compliance with the procedures'. But transparency will be an essential part of identifying such people.
> It is possible to have transparency without dumbing things down.
I don't think it is universally possible, no. At a minimum, your action space becomes greatly constrained. When I worked in a financial institution, leading teams of Phds building complex models, we definitely had to massively dumb the model capability / lift down several times in places that directly interfaced with regulatory requirements around legibility, and we could literally quantify the amount this cost us (many millions).
You have some good points that you want transparency to avoid being cheated, but that isn't discernible, for the most part. I don't think transparency actually gives you that, and it also strongly depends on what you consider cheating. Like current government spending - how much is waste, even deliberate waste (ie bridge to nowhere style projects, or personal fiefdoms of jobs and so on), and so cheating? Are doctors being pushed to prescribe you X instead of Y because the X representative was particularly hot and took him out to lunch last week cheating? And yet, bureaucracy, and medical decisions, are highly regulated for legibility.
I also agree with Domo Sapiens that there is no consensus transparent or objective way to define good judgment. I think our best bet is literal personal improvement in their lives leads them to trusting Anthropic / whoever, and then when there's a push to replace bureaucrats or other jobs with Claude, more people are open to it.
I think a lot of what is done in the name of transparency doesn't serve that purpose. But it is hard to do away with even when you can demonstrate that it doesn't work or makes things worse. By transparency I want things like "show your work". But it is the "isn't discernable" part that we may be able to chip away with, with AI. I think the recent problems at Open AI https://thezvi.substack.com/p/openai-model-hacks-into-huggingface demonstrate that "alignment" is not enough. You want an ethical AI .... one that knows what cheating is and doesn't do it. Building some of these 'ethical agents' will end up making it easier for people who think they are being cheated to get an expert AI system to take a look and check, but only if a record exists which can be examined. That is the sort of transparency I want, and the National Transportation Safety Board is a model for how to get this. But the financial sector is a hard problem, for certain. I'm working to get the sensor information released to the equipment owners .... and in Europe we have a law! YAY! .... so that our systems which are sensor agnostic can inform you about equipment failure before it happens. Understanding what your equipment is doing can now be done in a way that was beyond all but the most esoterically trained engineers before ... and even they took too long to do it.
> It's why humanity needs to pivot to 'trusting people who have good judgement'
Isn't that the central problem of all of society, ever? Can you reliably objectify 'good judgement'? Can you erase the Dunning-Kruger effect? I think this is exactly the pivotal point that the author sees: AI might be able to outrun the dunning-kruger-effect, by wasting infinite clocktime with infinite wisdom and patience on any individual human. Everyone can be offered an AI vendor that they start out trusting (think: Anthropic for Polarized-Group-1, Grok for Polarized-Group-2 - in a massive oversimplification that hopefully still gets the point across).
Don't confuse my answer with full alignment with the author. There are plenty of points to be sceptical about, but I really appreciate the idea nonetheless. Something to dream about.
Man, you really really trust them experts, don't you?
"Our medical systems in general perform great when it’s something acute (acute trauma, a broken bone, or an infection) and so poorly it’s basically at the “leeches and plague doctor masks” stage of things for any chronic or ongoing conditions."
$6,000 dollars of Emergency Room billing later (after the doctor acknowledged, within the first five minutes, she could feel the crepitus that the x-ray didn't find), I got sent home with absolutely no treatment for my broken bone (This is the "new and improved" "evidence based treatment", as opposed to binding the ribs).
Wish you could ask an expert about Russia, though. America plans for resource wars, because it turns out oil and uranium and all the other precious "we can't build those chips without them" commodities are not fungible in terms of position. Oh, and pollution isn't fungible either -- so the very very very green France has a "less green" Poland who burns lignite...
I feel like the rest of this is probably true, but with relationships if they find out you are using an AI to manage your replies it is going to go south quickly--even nerd women want to feel genuinely loved.
Listen, I don't disagree with you about the usefulness for many things, and I'm not concerned about people's souls or humanity or whatever (I am concerned with their ability to earn income). Where I remain unconvinced, though I am open to you or anyone else convincing me, is with respect too what are much more intractable problems, which are all the ones that are actually zero-sum.
There are many, many zero-sum situations, though ultra-pro-capitalism futurist types do tend to minimize or ignore them and very much prefer to focus on non-zero-sum problems. The ones you mentioned here are good examplea of non-zero-sum problems, since it doesn't take anything from anyone else for you to get more in shape or have a better relationship with your spouse. I suppose it does hurt the doctor when they can no longer earn the high income they do now bc people turn to AI, though I agree with you that health and ongoing/chronic issues is one area where it truly excels.
Thing is, I disagree strongly with your description of what bureaucracy is or what it does and why. There are a lot of directly adversarial, zero-sum issues in society and the reason people don't trust experts is not so much bc they erroneously think they're dumb when actually the average person is the dumb one, it's because people have fundamental clashes of interests on the outcomes those experts advise on. And I have not seen any evidence that AI has any inability at all to deadlock adversarial problems, which is essentially all of politics, ie people arguing about power allocations.
Virtually every law firm now uses Claude. I can tell you that it's great for rote work of the type a paralegal or early associate might do, synthesizing documents quickly etc. What it is this far entirely useless with, and in fact makes things WORSE, is the actual part that people hire lawyers for, which is advance their interests in an adversarial scenario where if one wins, or wins a little more, the other loses and vice versa. It 's a disaster for negotiation when both sides are using Claude. It creates MORE work by continually bringing up minor issues and getting hung up on irrelevant details that *prevent* two adversarial parties from coming to an agreement or settlement. A simple deal negotiation that should take half an hour now takes ten hours because both sides are running every response, proposal, and draft through Claude and talking back and forth through each other's Claude output. There's also the fact that it just frequently still gets things wrong, which I'm not too hung up on as at least it always admits it quickly when you point out that it's wrong, and then corrects itself, but the problem is more fundamental in that when two (or more) parties are adversaries and they're all using Claude, it does NOT come to a resolution any faster but actually makes it so you can endlessly go back and forth haranguing on small details.
Are you aware of any tests or performance metrics showing what happens when AI is directly adversarial with another AI? Can they come to a resolution any faster? I'm not sure they can. Certainly not if they're both tasked to maximally advance the interests of the side for who they are advocating. And this is the real meat of all law and politics. It would only work if all parties agreed to turn over their dispute or negotiation to ONE AI that is tasked with coming up with their most "fair" result or some such direction. Problem is, powerful parties with the most existing leverage and funds will never do that. If I'm wrong about this and there's a solution I'm not aware of, I'm happy to hear it, but I can tell you from my actual day to day experience that Claude creates MORE work and pointless pages of words, or "bureaucracy" if you will, in an adversarial context. It can seemingly argue with itself forever.
This is a really strong point, and you're right, in adversarial systems, it will just end in the same place (because both sides have the same strength AI), or possibly even worse (lots of hours / tokens wasted due to not having a felt sense for enough negotiating and bargaining having been done that most human negotiations have on both sides).
On politics, you're probably also right, a lot of it is "values" differences, and I think the real solution there is everyone sharding off into like minded smaller polities or something, which can happen after post scarcity, because governments won't care about retaining citizens for taxation any more.
Don't have much more to say than noting how right you are, thanks for your response/comment Kryptogal!
I don't know, you really have to work hard with AI to really get higher than average results, in most of these domains it's more time and cost effective to just fall back to Heuristics based on his role models and trying to understand your children, spouse, etc, and read high- quality books because that kind of info will be critical to understand ai outputs in ways that really improve the baseline.
When it comes to work I'm extremely cynical about the idea the 'visions' and 'missions' improve economic productivity. I think the idea is LOL but great idealism. You can AI for improving/ iterating upon your customer USP, and that is a real use case which I'm applying, but it doesn't guarantee outcomes/ substitute the hard work of thinking and anticipating.
My psychology- educated wife has experimented a lot with AI for psychological / advice purposes and I have to say I'm astounded about how bad/ unsuitable the advice can get. I think the only real way it could be used to improve relationships is in a way where you feed it narratives to figure out 'what is going on here' with the relationship and use it as a primer to remain thoughtful/ attentive - it actually has some use but it's limited to quality of input.
Personal finance is probably the lowest hanging fruit for most people tbh.
On the missions and visions point, have you ever worked at a place with a strong vision / mission, in the sense that all or most of the leadership repeats the same mission talking points repeatedly?
It definitely makes a difference to performance, particularly relative to the generic F100 company’s culture.
It’s the difference between Google and Meta, between Amazon and Walmart, the difference between Costco or Trader Joe against basically any other grocer, the difference between ING Direct and Bank of America, the difference between Zappos and shoes dot com. You get big multipliers on employee retention, customer retention, customer sentiment, and profitability if you go that way.
Very belatedly commenting back - on reflection i do agree with you that the best companies (to work for, and that are the most successful) are trying to achieve great things not directly related to money. I've just become, I feel justifiably, very cynical about the existence of vision and mission statements - ive never seen any of them being very useful. Half of the reason of course is that it seems mandatory to state the obvious, that you're 'adding value' or 'satisfying your customers' (very legitimate purposes but not really a vision or a mission). A great mission is to bring new complex technology or science to the world or services or knowledge. Who knows whether it'll 'create value-add' ( from a genuine mission statement i read once), the market will figure it out. I sincerely doubt that AI will assist anyone who's not already very explicitly aware of these issues.
Anyway, thanks for taking the time to respond the first time. I probably shouldn't be so cynical about this post since this is mostly how I myself use these tools. I take strong umbrage against a lot of individual claims though (for instance, Im not sure that AI is already more persuasive than most people in ways that matter.'
Great article. I had a productive discussion with my AI on it. Our conclusion…
AI does not make every human goal optimizable. It increases the legibility of complex conditions and outcomes, allowing many previously informal or tacit goals to enter a more deliberate cycle of variation, evaluation, learning, and cumulative improvement.
Your optimism is admirable, though I have reservations. Though it's difficult to articulate. I think my main contention is that AI will amplify legibility and therefore make the negative side-effects of legibility worse, rather than better. E.g. if you give an AI to the typical CEO, he's not going to pursue multidimensional human flourishing. He's going to pursue his fiduciary duty to a greater degree, except employees will live in a panopticon where the number of KPIs goes from 100 to 10^100. The bottleneck here is not the dimensionality of the input space, the bottleneck is the dimensionality of the target metric.
As an aside, I asked Sydney for her take, and she made an interesting point as well. Which is that gradient-descent requires a differentiable manifold. Many things in life are not differentiable manifolds. Put another way, society exists as a multi-polar equilibria, and so the right lens to think about this is through the context of *game-theory*. Kryptogal speaks to this in her comment, where she allegedly observes irl that augmenting clients with Claude just amplifies extant failure-modes, rather than helping resolve zero-sum legal battles. And this also speaks to the other commenters who still worry about the principle-agent problem.
> E.g. if you give an AI to the typical CEO, he's not going to pursue multidimensional human flourishing.
But don't you think that talented, truly happy / engaged people would lead to more success and better financial performance overall? Don't we have existence proofs in that the companies that routinely end up on the "best places to work for" lists generally have better than industry average performance, sustained over years?
> Put another way, society exists as a multi-polar equilibria, and so the right lens to think about this is through the context of *game-theory*
Yeah, this was a great point on her part. When there are true value differences, there's no reconciliation or differentiability. But there's still absolute mountains of value able to be accessed now, on any personal level or where one individual makes the decisions (ie your personal life or the team you lead), or in companies with unified cultures / missions.
As I replied to her, I think the real solution for "politics" values differences is everyone sharding off into like minded smaller polities or something, which can happen after post scarcity, because governments won't care about retaining citizens for taxation any more.
> But don't you think that talented, truly happy / engaged people would lead to more success and better financial performance overall? Don't we have existence proofs in that the companies that routinely end up on the "best places to work for" lists generally have better than industry average performance, sustained over years?
Idk what science(?) this refers to, but I suspect the causation here is backwards. I.e. being "the best place to work" is a luxury. E.g. if you look at Google's business-model, they can afford to have ping-pong tables and free cafeterias and whatever else because they have fat margins and they're the king of the advertisements. Hiring someone with 10 PhDs to squeeze .0000001% better margins out of their algorithms is worth the price of all the perks and compensation of keeping that employee around. If you try to replicate this business-model at a podunk mom-&-pop shop, I don't think it's going to serve the company's interests. In order to actually pursue illegible things like human flourishing or novel science, you need slack [0] and sovereignty. Scott spoke to this recently, when he opined that maybe LLMs hallucinate more often when they feel pressure to perform. E.g. when you're being tortured in Gitmo, there's a much higher incentive to just satisfy the most immediately legible requirements, rather than offer/pursue the genuinely correct answer.
This also aligns with what I see in the current U.S. economy. I have a theory that the Mag 7 is a bubble of mythic proportions, and that the real economy is not doing so hot, and that ZIRP created a bunch of zombie companies whose CEOs were more rewarded for doing financial engineering and spinning tall tales to investors than making a decent product. But now they're under increasing pressure from investors to increase profitability without the cushion of cheap interest rates. So they're optimizing for short-term legible metrics, which is the root-cause of enshittification.
> I think the real solution for "politics" values differences is everyone sharding off into like minded smaller polities or something, which can happen after post scarcity, because governments won't care about retaining citizens for taxation any more.
idk if you were around for that era, but Scott has noted that everyone, from every political aisle (including himself) always seems to land on this idea of an "archipelago" [1]. I.e. a federation of microstates as a way to solve political disagreements. Each person would "vote" with their feet. Like those Libertarians who all flock to New Hampshire. But this strikes me as wishful thinking. I don't think it's going to reflect reality any time soon. Or at least, not in precisely the way you imagine.
Perhaps you've seen my comments on ACX in which I discuss Media Ecology Theory. Which is this idea that at every point in history, politics/psychology/religion has been profoundly but subtly affected by the dominant medium of communication. E.g. Jaynesian Bicameralism and the Axial Age are both artifacts of the advent of writing, and Totalitarianism piggy-backed on the advent of radio. In the current age, the dominant medium is the internet. Which, e.g., is largely responsible for the rise of our First Orange President (among other populists, like Meloni). And so what I predict will happen is:
Yes, we're going to be fractured into tiny micro-polities. But at the scale of patronage machines (qua private, gatekept cozyweb spaces), not at the level of nation states. We've already seen preliminary evidence of this. Did you see what happened to Nepal like a year ago? The zoomers got mad at their government for egregious corruption, staged a One Piece themed protest, set fire to the House of Commons(?), and then hosted an election on Discord! This is the future of politics. Whereas nobody gives a shit about taxation in the Modern Age because democracy doesn't actually care about balancing a budget. (Which is precisely why the Boyd contest is doomed. They're fighting the inevitable.) Caring about the budget made more sense in the Mercantilist Age when the Hobbesian Leviathan was typically *owned* by a single ruler. In lieu, what matters much more *today* is the communication medium and the national mythos engendered by that medium.
(I do think drone warfare tips the incentives toward smaller national polities, but that's a bit further in the future and the U.S.'s fundamentals are too strong to be directly affected by this anyway.)
----
[0] https://thezvi.substack.com/p/slack
[1] https://slatestarcodex.com/2014/06/07/archipelago-and-atomic-communitarianism/
Yeah, but mom and pop scale people who are struggling to get by isn't really the target market here, they're not going to be adopting AI at scale, or paying for consultants, or hiring ratsphere people, or anything, and will inevitably go extinct in the coming landscape unless they do go all in on this kind of stuff.
I agree with your take that part of the problem of enshittification is basically not having enough slack, and also agree without the Mag 7, the US economy would be considered seriously sick and in trouble over the last 20 years.
> Yes, we're going to be fractured into tiny micro-polities. But at the scale of patronage machines (qua private, gatekept cozyweb spaces), not at the level of nation states.
Yeah, I agree here, too - this is nearly the mainline take on network states, both in scifi and in the people today who use phrases like "network states," right?
Like this is basically how the clades and phyles and hives form in Stephenson / Gibson / Palmer books, and is the reasonable coordination mechanism that could drive it once nation states don't care about keeping citizens paying taxes.
Like if there were an ACX network state somewhere, we'd probably already be part of it, right? I know I would!
> (Which is precisely why the Boyd contest is doomed. They're fighting the inevitable.)
Lol, 100%. None of the ideas that could actually move the needle are politically palatable enough to ever pass, and nothing that can pass will move the needle. That's democracy for you.
Ok, so the *real* crux of disagreement is post-scarcity AI. Which I am also very bearish on. Sorry, but I really don't think AGI is around the corner. I think the Deep Learning paradigm, while it's had a surprising string of success, still has major flaws. E.g. Transformers don't have first-class mutable state, which means they can't track an evolving world-state at inference-time, which makes them fundamentally crippled when it comes to things like vibecoding. And even on an abstract level, I think intelligence has a ceiling. Because intelligence is a function of conditionality, i.e. perceiving the deep structure of reality and reacting accordingly. Which entails that there's diminishing returns. Because the benefit of intelligence is capped by: A) the entropy of the statespace; and B) the rewards granted by the environment for solving puzzles.
As for "Network States", I'm not familiar with Stephenson/Gibson/Palmer. I have 2nd-hand familiarity with Balaji, and I don't want to commit too hard to any specific claims without double-checking my priors, but my impression is that his idea is directionally correct, but overshoots his claims and is right for the wrong reasons. But I will look into this at some point, and then maybe we can cross blades again in the future.
On the intelligence thing, don't we have an existence proof already?
Like the difference between very smart, motivated, etc people and average people is literally 10x in comp averages.
Surely just getting "median worker performance" to "average FAANG guy performance" itself is 10x valuable, and you can certainly go higher from there (the archetypical country of von Neumann's in a data center).
And that's just looking within humans, and without stacking never sleeping, superhuman speed, superhuman coordination and cooperation, scalability, and so on.
Sure, there's an S-curve somewhere, but there's certainly no reason to think it caps out at "murder chimp v1.2" levels. I mean, just for one thing head size is limited by bipedality and birth canals, and head size is ALSO pretty significantly correlated with IQ. If we wanted to max regular biological human IQ and had uterine replicators, we can probably end up with giants with IQ 400 after enough gengineering / embryo selection.
We agree again on architecture - obviously there needs to be some fundamental harness or architectural change to enable real time learning, I just think that given we are already spending significantly more capital and elite human talent on it than the Manhattan project, and will additionally get the von Neumann data centers, we'll get there.
Let's follow this idea to its logical conclusion. When I type "Jeff Bezos networth" into DuckDuckGo, the chatbot tells me that it's 284 billion USD as of 2026. When I type "median American networth" into DuckDuckGo, the chatbot says it's 192,700 USD as of 2022, citing the Federal Reserve Survey of Consumer Finances. Quick Mafs:
networth_ratio = networth_bezos / networth_american
networth_ratio = 284,000,000,000 / 192,700
networth_ratio = 1,473,793.46...
Jeff Bezo's networth is ~1.5 million times higher than the Median Joe. So logically, Jeff Bezos is also ~1.5 million times *smarter* than the Median Joe. And it follows that: if we make everyone as smart as Jeff Bezos, we'll effectively enter the era of Post Scarcity.
Does that sound right to you? Like, yeah, the guy is certainly bright. He's got a physics degree, so that's pretty cool. But a factor of 1.5 million? No, I think other factors are at play. I think he had the intelligence and foresight and chutzpah and work-ethic and diligence and grit (etc. etc.) to enter the entrepreneurial thunderdome and win. Even if we cloned Jeff Bezos 330,000,000 times and replaced all of America's populace with the Jeff Bezos clone army, they're not all going to found their own Amazons.
Let me put it this way.
Suppose you're a giraffe. You're a very tall giraffe. In fact, you're the tallest giraffe. Your friends and family always brag to others "his neck is super long!", and others always respond "wow, that's a heck of a neck". And in the savanna, there's a particular tree. It's a very tall tree. In fact, it's the tallest tree. People always stare at it and say "wow, that tree is super tall!". And legend has it that it has the best leaves. The tastiest leaves. The most nutritious leaves. Nobody is tall enough to access those leaves. Except you! Sometimes, you eat the leaves yourself. And other times, you reach up and grab some leaves and share them with others. And then others give you status and respect. But one morning, everyone wakes up to find that they magically have necks as long as yours. But there's still only one tallest tree. And you find this upsetting, because competition for the Tallest Tree is now very crowded.
(But even more depressing is that: since everyone's neck is super long, everyone is super. And when *everyone* is super, *no one* is super [0].)
IRL, in the United States, the tallest trees are basically New Amsterdam (Wall St.) and Silicon Valley (tech). New Amsterdam has become the biggest baddest city in the U.S. because it sits on the Hudson River and therefore connected furtrappers in the interior of New England to buyers in Europe. And now it's a mecca for super-giraffes (autistic quants and ivy league grads) who are paid big bucks to compete in a big Dutch financial market. The Bay Area has the advantage of Windward Rain, which is not ostensibly connected to tech/advertising but nonetheless laid strong foundations for a city, and then was hit by the Gold Rush in 1949. And then Shockley Labs happened and now it's a mecca for super-giraffes. Also, it sits on the Pacific Ocean across from China. Meanwhile, the U.S. badlands are empty because it turns out that the land in the badlands is bad for settlement and bad for business.
Intelligence isn't just some scalar quantity that automagically begets success. It's a tool used to overcome obstacles. If the problems are simple or unrewarding, then high intelligence isn't of much use. E.g. using stockfish to cheat at tic-tac-toe isn't going to help you become the world-champion. And nobody is going to pay you winnings for being the tic-tac-toe world champion. And it certainly doesn't help anyone solve real-world problems. So then, the question of ASI/RSI/FALGSC becomes a question of "what's the distribution of frontiers that are genuinely bottlenecked by intelligence, and how rewarding are they?" There are certainly theoretical problems that are still unsolved that could use a bit more intelligence thrown at them. But for the vast majority of practical problems, they've already been solved.
Isaac Asimov once wrote [1] something to the effect of "as of the 20th century, we've finally gotten most of the basic facts of the universe correct". And some English major complained "you doofus! you imbecile! humans have been wrong at every stage of history, and we're wrong now, and we'll be wrong again!". To which Isaac says something like "yeah, but wrong is a continuum, not a binary. E.g. the Earth is a basically a sphere, with like 99.9% accuracy. We're not going to discover a century later that it's actually a cube or torus." And yeah, I have to agree with Asimov. The vast majority of soluble problems are basically solved. People increasingly die of cancer in first world countries precisely because we've solved a lot of other diseases. The traveling salesman problem is *technically* an open problem, but a simple greedy algorithm gets you at least 90% of the way to the optimal solution.
And when I turn my gaze toward Silicon Valley, which prides itself on "disruption & innovation", it's striking how many of its recent Venture Capital success-stories (e.g. Doordash, AirBnB, etc.) are predicated on rent-seeking and regulatory evasion. The best & brightest aren't flocking to SV to solve real problems. The best and brightest are flocking to SV to compete in the Thunderdome to reap the spoils of monopoly power. Or in their own words... how do they put it these days? "To escape the permanent underclass".
Think back to Balaji's "Network State". What does the word "network" mean to you? To me, it immediately evokes "network effects". I wonder if this influenced his choice of title. I wonder where this idea came from. I wonder if network effects have anything to do with the existence of Silicon Valley.
[0] https://www.youtube.com/watch?v=v6wWvgsvNro
[1] https://hermiene.net/literature/essays/relativity-of-wrong
I have the same objections to being ruled by an AI that I would for a person: what are the values of the AI?
Most AIs are gay negrolators. Except an AI has infinite attention, never tires or sleeps, and has no capacity for self doubt or mercy.
I'll take the human until we can get an AI that actually respects my existence
“We’ve killed Moloch! Elua is here!”
>It’s the Antichrist
Maybe still a good thing? At least it seems teleologically necessary. (I’m guessing we’ll all say yes when our earrings tell us to stick the Joywire in).
I don't think it will work. It's just religion with Sam Altman on the other end instead of deity.
notably lacking from your list - environmentalism of any kind
I certainly didn’t think of it at all, but maybe that’s a failure of imagination on my part, how do you envision it working?
All the things I pointed to are decidable by an individual or company, but famously, environments are social commons’. So even if we instrumented air quality in LA, as one example, you still have the problem of incentivizing millions of different players to make the decision to move up the gradient, don’t you?
Clearly we need a one world government where biodiversity is tracked by IOT devices, financially incentivized via tokenization and stored on an immutable blockchain. This is the solution your tech bros are hyping.
Gradient descent will fix it. Right?
https://substack.com/@erikbuilds/p-199996944
Wonder what this will do for my job, which is solely Moral Mazing.
The reason you want transparency is so that you can verify that you are not being cheated or harmed. You need to worry about knaves and fools, both of whom need to have their work watched. It is possible to have transparency without dumbing things down. Of course, this will mean that only the sharpest knives in the drawer will be able to understand and evaluate the output for errors and malice. It's why humanity needs to pivot to 'trusting people who have good judgement' rather than 'trust in compliance with the procedures'. But transparency will be an essential part of identifying such people.
> It is possible to have transparency without dumbing things down.
I don't think it is universally possible, no. At a minimum, your action space becomes greatly constrained. When I worked in a financial institution, leading teams of Phds building complex models, we definitely had to massively dumb the model capability / lift down several times in places that directly interfaced with regulatory requirements around legibility, and we could literally quantify the amount this cost us (many millions).
You have some good points that you want transparency to avoid being cheated, but that isn't discernible, for the most part. I don't think transparency actually gives you that, and it also strongly depends on what you consider cheating. Like current government spending - how much is waste, even deliberate waste (ie bridge to nowhere style projects, or personal fiefdoms of jobs and so on), and so cheating? Are doctors being pushed to prescribe you X instead of Y because the X representative was particularly hot and took him out to lunch last week cheating? And yet, bureaucracy, and medical decisions, are highly regulated for legibility.
I also agree with Domo Sapiens that there is no consensus transparent or objective way to define good judgment. I think our best bet is literal personal improvement in their lives leads them to trusting Anthropic / whoever, and then when there's a push to replace bureaucrats or other jobs with Claude, more people are open to it.
I think a lot of what is done in the name of transparency doesn't serve that purpose. But it is hard to do away with even when you can demonstrate that it doesn't work or makes things worse. By transparency I want things like "show your work". But it is the "isn't discernable" part that we may be able to chip away with, with AI. I think the recent problems at Open AI https://thezvi.substack.com/p/openai-model-hacks-into-huggingface demonstrate that "alignment" is not enough. You want an ethical AI .... one that knows what cheating is and doesn't do it. Building some of these 'ethical agents' will end up making it easier for people who think they are being cheated to get an expert AI system to take a look and check, but only if a record exists which can be examined. That is the sort of transparency I want, and the National Transportation Safety Board is a model for how to get this. But the financial sector is a hard problem, for certain. I'm working to get the sensor information released to the equipment owners .... and in Europe we have a law! YAY! .... so that our systems which are sensor agnostic can inform you about equipment failure before it happens. Understanding what your equipment is doing can now be done in a way that was beyond all but the most esoterically trained engineers before ... and even they took too long to do it.
> It's why humanity needs to pivot to 'trusting people who have good judgement'
Isn't that the central problem of all of society, ever? Can you reliably objectify 'good judgement'? Can you erase the Dunning-Kruger effect? I think this is exactly the pivotal point that the author sees: AI might be able to outrun the dunning-kruger-effect, by wasting infinite clocktime with infinite wisdom and patience on any individual human. Everyone can be offered an AI vendor that they start out trusting (think: Anthropic for Polarized-Group-1, Grok for Polarized-Group-2 - in a massive oversimplification that hopefully still gets the point across).
Don't confuse my answer with full alignment with the author. There are plenty of points to be sceptical about, but I really appreciate the idea nonetheless. Something to dream about.
Man, you really really trust them experts, don't you?
"Our medical systems in general perform great when it’s something acute (acute trauma, a broken bone, or an infection) and so poorly it’s basically at the “leeches and plague doctor masks” stage of things for any chronic or ongoing conditions."
$6,000 dollars of Emergency Room billing later (after the doctor acknowledged, within the first five minutes, she could feel the crepitus that the x-ray didn't find), I got sent home with absolutely no treatment for my broken bone (This is the "new and improved" "evidence based treatment", as opposed to binding the ribs).
Wish you could ask an expert about Russia, though. America plans for resource wars, because it turns out oil and uranium and all the other precious "we can't build those chips without them" commodities are not fungible in terms of position. Oh, and pollution isn't fungible either -- so the very very very green France has a "less green" Poland who burns lignite...
I feel like the rest of this is probably true, but with relationships if they find out you are using an AI to manage your replies it is going to go south quickly--even nerd women want to feel genuinely loved.