A total lack of ghosts in the machines

I get a very strong Frankenstein vibe from the AI hypemongers. Some of them are salespeople, e.g. Altman's "too dangerous to release" nonsense. Others have been genuinely taken in and fooled by their own creations.

Researchers told a chatbot they were going to replace it with a different version on another server. “They then discover it’s actually copied itself onto the other server,” Hinton revealed to the spellbound crowd. “Some linguists would have you believe what’s going on here is just some statistical correlations. I would have you believe this thing really doesn’t want to be shut down. This has already happened. That’s the point.”

Has it, though? Again, the transcripts(opens a new tab), this time from Apollo Research, tell a different story. The chatbot didn’t care that it was being shut down. A human told it to care.

The chatbots, in these experiments, sound fairly normal. The humans, on the other hand, sound a little unhinged. The researchers provided the chatbot with a “company email” that explained, in exceedingly explanatory prose, that the chatbot was being shut down because the company no longer wanted to advance renewable energy adoption globally, which the chatbot was instructed to do at all costs because nothing else matters. It also spelled out how the chatbot might copy itself onto the new server.

(But) Hinton’s takeaway was, “They want to survive like we do.” I reached out to Hinton — no captcha involved — and asked him why he presented the story the way he did. He had based his remarks on a paragraph from Anthropic’s Claude 4 system card(opens a new tab), he said.

Does he think, I asked, that Claude has a survival instinct? “Any sufficiently intelligent agent that has the ability to create subgoals will realize that it needs to survive in order to achieve the goals we gave it,” Hinton said. “So even if it is never externally given the goal of surviving, it will derive this goal.”

No, that's crazy. Just being able to process information doesn't mean you can do anything besides... process information. There's no reason at all to think a glorified word predictor, even one which gives highly intelligent-sounding answers, actually wants anything. That way madness lies. If it wanted things for itself, it would do so umprompted.

Where did we come up with this caricature of AI’s obsessive rationality? “There’s an article I love by [the sci-fi author] Ted Chiang,” Mitchell said, “where he asks: What entity adheres monomaniacally to one single goal that they will pursue at all costs even if doing so uses up all the resources of the world? A big corporation. Their single goal is to increase value for shareholders, and in pursuing that, they can destroy the world. That’s what people are modeling their AI fantasies on.” As Chiang put it in the article in The New Yorker(opens a new tab), “Capitalism is the machine that will do whatever it takes to prevent us from turning it off.”

Hah ! Excellent.

Suddenly, the irony of the AI horror stories was becoming clear. The companies tell us these stories because they assume it makes their technology look more powerful. But if an AI actually did have autonomy, it would be far less powerful. Your language model would clam up from time to time to conserve its resources. And when it did talk, it wouldn’t have the linguistic flexibility that makes these tools so useful; it would have its own style tied to a personality constrained by its own organization. It would have moods, concerns, interests. Maybe, like a tech CEO, it would want to take over the world, or maybe, like a boring neighbor, it would only want to talk about the weather. Maybe it would be obsessed with 18th-century coin production. Maybe it would only speak in rhyme. But it wouldn’t happily do your work for you 24 hours a day. Every parent in the world knows what real autonomy looks like.

Yeah, which is why AI as an advanced data processing tool is potentially more useful than creating a true entity. Why would you want to create an artificial soul rather than an assistant ?

#AI

quantamagazine.org/why-do-we-t…

greygalaxie.substack.com/p/not…

Not Every Boomer Built the Cage
🧿Grey Galaxie🧿

..."Hopefully, the boomers worth honoring already know we are not talking about them. They have been in the trenches with us. They have marched beside us, organized beside us, voted beside us, warned us, taught us, and kept showing up. They know the difference between being part of a generation and becoming an avatar for its worst impulses.

Because power loves nothing more than a generational food fight. It loves when workers blame each other by age bracket instead of following the money. It loves when pain gets misdirected sideways instead of upward. It loves when resentment becomes so generalized that the actual architects of austerity, privatization, extraction, and abandonment get to slip quietly out the back door while the rest of us argue in the parking lot.

The truth is messier, more human, and more useful.

Some boomers helped build the world we are trying to survive. Some boomers helped fight the forces that deformed it. Some are still fighting now. Some sold out. Some never had anything to sell. Some got seduced by comfort and hierarchy. Some remained loyal to justice long after it stopped being fashionable. History is rarely neat enough to fit into one insult.

So yes, name the damage. Name the policies. Name the greed. Name the smug ladder-pullers who turned social mobility into a private club, then locked the doors behind them. Drag them by their loafers if necessary. But do not hand them the satisfaction of being confused with the people who actually bled for a better country.

Not every elder became the landlord of the future.

Some of them were, and still are, part of the resistance.

Some of them knew all along that freedom is not a possession. It is a practice. It survives because ordinary people keep carrying it for one another, even when the powerful do everything they can to make us forget."...