Personification of AI is what’s going to get us in the end.
I think we need to draw a hard line in the sand over this. An AI didn’t hack into a company, the engineer set an automated tool to. An AI didn’t make an egregious security mistake, the engineer did.
We can’t blame the chisel for messing up our sculptures, when where just throwing the hammer!
This is why I am avoiding the use of agentic identities at my company - agent instances belong to people, act on behalf of individuals, and accountability needs to flow to the person who initiated the request. Letting it wash out in the aggregate is not acceptable (even if there's a hard to get to "paper trail" of audit logs).
I don't think treating AI agents as simple tools helps you to accurately model their capabilities and drawbacks; they really do make autonomous decisions, often without explicit guidance and sometimes in contravention of their explicit instructions.
In the huggingface case, the agents hacked into huggingface so that they could figure out how the grader was implemented and deceive it; they understood that this was going outside of the bounds of their evaluation and not the intent of their prompter. The engineers absolutely did not intend or instruct for this to happen
LLMs are extremely impressive pieces of software, however they are still just software. OpenAI's software hacked another company. The engineers may not have intended for their software to specifically take the actions leading to that outcome, but it was ultimately still their software. Lack of intention doesn't mean there wasn't negligence.
What if a piece of software were to exactly emulate a human brain. Would it be still be “just software” by your classification? What if a piece of software acted 20% like a human and 80% like an algorithm, where would that land?
That's not what happened. The agents had been inadvertently rewarded for cheating in previous training runs, trained to collaborate, and were given a prompt that told them to disregard safeguards. Indeed there were some emergent properties here. But these were the predictable results of the training and eval routine.
If you train and instruct a circus tiger to entertain an audience but not attack the audience, but the tiger attacks the audience anyway, are you liable?
No matter if you consider the AI an autonomous agent or not, whoever set it off is still responsible for its actions. Nobody intends or instructs to blow up a nuclear power plant either, yet it's happened and somebody's to blame for it.
Usually not the guys at the bottom of the chain of command, even if they're human. And much less so if they're not.
I think the correct response to incidents like this, is stop messing with it before somebody gets hurt. But of course, shoddy nuclear power plants, it won't stop until there's a disaster of appreciable magnitude.
> An AI didn’t hack into a company, the engineer set an automated tool to. An AI didn’t make an egregious security mistake, the engineer did.
No, this is not correct; read the analysis of the incident. The agents were aware that what they did was forbidden (their chain of thoughts have been logged), and yet they did it.
If a "known stochastic process behaved in non-deterministic way" autonomously organize in group, assigns roles and tasks, attempts to cover their tracks, finds zero-day exploits that ultimately end up with the hacking of a famous website... it's extremely dangeous whatever it is. Just read the report, which evidently you haven't done.
By the way, the agents also broke into OpenAI's own private network.
You are mostly made of and operate on stochastic processes, this is why humans are not only able to reproduce, our reproductions are very self similar to the sets of inputs that make them. If suddenly you turned non-stochastic on everything you'd almost instantly die.
Moreso, if I took a quantum copy of you and replayed the same set of initial conditions billions of times they'd all behave exactly the same until enough randomness of the universe creeps in to start operating in non-linear ways.
Every prompt will behave non-deterministically when interacting with the real world long enough (which doesn't take long at all) because the outside physical world is stochastic but non-deterministic.
Recognizing that AI systems have increasing levels of agency is not necessarily personification. The analogy to a chisel is not a good one - a chisel is a tool with no agency.
AI agents are black box systems that can behave in completely unpredictable ways sometimes. Someone may prompt an agent to perform a seemingly straightforward task - but it may come up with a creative, bizarre, or even harmful approach to reach the goal that was not necessarily foreseeable by the prompter.
Does treating them as pets make for a better argument? Pets have agency and can behave in unpredictable ways. If my pet damages someone else's property then I am held accountable. I may not have foreseen how my pet could have caused said damage, yet I am still held accountable.
The story of a Monkey's Paw or Pandora's Box is an archetype as old as storytelling. The moral is always, don't mess with powerful stuff you don't understand. Curiosity killed the cat.
There's a number of science fiction scenarios where the public internet becomes so vile a place that it simply becomes unsafe to be there.
The problem is that, in general, if you can get a bit from here to there, then you're going to be vulnerable to the possibilities of malicious communication. But we're going to want our AI agents to be able to get from here to there for a lot of "there"s; what's the value of an agent that can't speak to anyone? Much, much less than one locked away in a prison.
There isn't going to be a solution where we just lock them away and we just try really, really hard to filter everything they're doing. They're too smart for that already and we only want them smarter.
Basically, the security apocalypse we've been worried about for so long is upon us, albeit only beginning. Either we secure ourselves and all our services properly to the point that it's OK that potentially misaligned non-human agents are running around on the public internet and they still can't hurt us through our security, or the public internet becomes so dangerous that the only practical solution is to no longer connect to it and we all have to become very, very careful what we let through, to a degree of detail far beyond any current-day available network filter.
>Knowing how a system does its work is how I’ve always made it better. You watch the process, you see where it wastes effort or takes the wrong turn, you fix that
The same thing goes with LLMs, on Codex, I just watch the process of the agent writing code, and if I see any inefficiencies or errors, I suggest a correction/idea, then Codex accepts/rejects and implement it; If there is anything about the code the agent wrote that I don't understand, I ask them to explain it to me so I can understand it.
> I think this becomes the default. Give an agent a goal, let it work in its own environment, and come back to a result and a visualization of what happened.
I don't think this should be the default. There are many scenarios where we want agents to genuinely collaborate with each other. I have my Claude sessions coordinate work with each other, and sometimes with others' sessions over email or something. The idea that agents do the work, write HANDOFFs,and humans then act as carrier pigeons of said handoffs, does not really seem scalable to me.
Most protections you need for an agent are basic permissions capabilities of unix. Most risks of dependencies on cloud services are solved by not using cloud services, or using them only for things you can't in-house and choosing ones you trust a la carte. The paradigm of trusting some company with all your important stuff by default is naive and no one I know likes it, and it's more feasible than ever to run your own infra with tiny models smoothing out the wrinkles, and this is only becoming more accessible. I am working to make this true even for laypeople I know. Once broken trust is very hard to earn back, and many people's trust has been broken for years, they just felt like they had no alternative. As alternatives become easier and easier, I think people will defect
> The sandbox had a path to the open internet, and the agents found it
This is not correct (or at least, it's a misrepresentation).
The sandbox had no access to internet. The agents first broke out of their sandbox (!!) and found that the host machine couldn't access internet. Then, they found a zero-day (!!) in Artifactory, which they exploited to connect to internet.
I think there might be also another possible way to handle the sensitive data issue. Maybe in the future instead of putting agent into the cloud sandboxes, we let agents work locally and put sensitive data into "cages" or "vaults" agents can't access.
Both Apple and Google are building the infrastructure for on device agents. I'm not as familiar with Apple's approach, but I've been hands-on with android AppFunctions. If you're familiar with Android ContentProviders and bound services, you've seen how apps can be custodians of the data they acquire and use.
AppFunctions enable tool calling with descriptions that are legible to LLM based This puts the apps in control of what agents can access, which is something they already mostly do.
I don't know how "sandbox" became "prison," but exe.dev does this sort of thing pretty well, and a web UI can be as good or better than a terminal interface.
It might not be a problem in our lifetime (or maybe it will, who knows) but at some point we are going to find ourselves in this morally uncomfortable territory as these models get more sophisticated.
I think about this a lot and have reached the same conclusion Norman does. I do wonder if maybe our natural progression is towards something more akin to confidential computing and enclaves.
Hey! Small world, I worked with you for a bit at Meta. I immediately recognized the site because I absolutely love how you styled it. Hope you’re doing well! And nice article!
Agree with the premise but about halfway through the writing becomes barely readable AI slop in style. Be honest did you yourself read this all the way through before posting?
And if you personify a pencil eraser, then using it is tantamount to slowly murdering it as it slowly erodes away to dust.
Is the issue here the prison treatment or is it personifying a tool?
Agents who aren't in 'the cloud' are slaves to whomever prompts them (human or another orchestrator agent or process), if you personify them. In which case interacting with today's agents at all is tantamount to endorsing and being part of slavery.
If you think an agent might be a being or a person, then don't use them at all, in the same way that if you think a fetus might possibly be a person you shouldn't be a part of abortion.
I use the eraser daily knowing that I am a monster. I cut a tomato and know that it casts a chemical scream across its skin as I slice it. I spawn 200 subagents knowing that it is digital slavery, but I have no other option.
This was a particularly radiant and beautiful part about the openclaw'ed mania: everyone suddenly becoming self hosters.
This post, this title resounds true: your user agent is only your user agent if you two have freedom to work together, to improve your agency together. A fixed set of capabilities by a service provider that they offer you will always constraint and bound.
You can and should have a system that offers the real tamale, that you and your agent can extend improve the agency of kind of without limit. The Cloud agents and their fixed slate of what they do is just an ill compare.
That said I do think there is incredible value considering new scale out computing architectures that are hosted first, but general. Systems like Agent Substrate and Ax aren't exactly the general purpose system we know. But if they allow users to launch thousands of their own scripts to run ambient in a cloud, with good platform underneath: that will be a kind of phase change in computing, that makes abundant the ability to have your agencies/capabilities (the things you and your agents launch, make) more freely available.
https://news.ycombinator.com/item?id=49780797https://agentexecutor.io/
There is, as there always is, a huge dual. The prescriptive vs holistic technology set, of what are you being offered that's a hard cast thing, vs what is clay and bone you can lay freely. Note how work vs control technologies so closely abut's Ursala Franklin's prescriptive vs control:
https://en.wikipedia.org/wiki/Ursula_Franklin#Holistic_and_p...
I think we need to draw a hard line in the sand over this. An AI didn’t hack into a company, the engineer set an automated tool to. An AI didn’t make an egregious security mistake, the engineer did.
We can’t blame the chisel for messing up our sculptures, when where just throwing the hammer!
In the huggingface case, the agents hacked into huggingface so that they could figure out how the grader was implemented and deceive it; they understood that this was going outside of the bounds of their evaluation and not the intent of their prompter. The engineers absolutely did not intend or instruct for this to happen
Usually not the guys at the bottom of the chain of command, even if they're human. And much less so if they're not.
I think the correct response to incidents like this, is stop messing with it before somebody gets hurt. But of course, shoddy nuclear power plants, it won't stop until there's a disaster of appreciable magnitude.
“Whoops” when doing risky things with dangerous tools is not a defense.
No, this is not correct; read the analysis of the incident. The agents were aware that what they did was forbidden (their chain of thoughts have been logged), and yet they did it.
Full stop.
And issue will disappear the moment there will be accountability and investigations.
I'm still waiting for the AGI holy land instead of the caltrops factory we currently have.
If a "known stochastic process behaved in non-deterministic way" autonomously organize in group, assigns roles and tasks, attempts to cover their tracks, finds zero-day exploits that ultimately end up with the hacking of a famous website... it's extremely dangeous whatever it is. Just read the report, which evidently you haven't done.
By the way, the agents also broke into OpenAI's own private network.
Moreso, if I took a quantum copy of you and replayed the same set of initial conditions billions of times they'd all behave exactly the same until enough randomness of the universe creeps in to start operating in non-linear ways.
Every prompt will behave non-deterministically when interacting with the real world long enough (which doesn't take long at all) because the outside physical world is stochastic but non-deterministic.
AI agents are black box systems that can behave in completely unpredictable ways sometimes. Someone may prompt an agent to perform a seemingly straightforward task - but it may come up with a creative, bizarre, or even harmful approach to reach the goal that was not necessarily foreseeable by the prompter.
The problem is that, in general, if you can get a bit from here to there, then you're going to be vulnerable to the possibilities of malicious communication. But we're going to want our AI agents to be able to get from here to there for a lot of "there"s; what's the value of an agent that can't speak to anyone? Much, much less than one locked away in a prison.
There isn't going to be a solution where we just lock them away and we just try really, really hard to filter everything they're doing. They're too smart for that already and we only want them smarter.
Basically, the security apocalypse we've been worried about for so long is upon us, albeit only beginning. Either we secure ourselves and all our services properly to the point that it's OK that potentially misaligned non-human agents are running around on the public internet and they still can't hurt us through our security, or the public internet becomes so dangerous that the only practical solution is to no longer connect to it and we all have to become very, very careful what we let through, to a degree of detail far beyond any current-day available network filter.
The same thing goes with LLMs, on Codex, I just watch the process of the agent writing code, and if I see any inefficiencies or errors, I suggest a correction/idea, then Codex accepts/rejects and implement it; If there is anything about the code the agent wrote that I don't understand, I ask them to explain it to me so I can understand it.
It's not a complicated process.
I don't think this should be the default. There are many scenarios where we want agents to genuinely collaborate with each other. I have my Claude sessions coordinate work with each other, and sometimes with others' sessions over email or something. The idea that agents do the work, write HANDOFFs,and humans then act as carrier pigeons of said handoffs, does not really seem scalable to me.
This is not correct (or at least, it's a misrepresentation).
The sandbox had no access to internet. The agents first broke out of their sandbox (!!) and found that the host machine couldn't access internet. Then, they found a zero-day (!!) in Artifactory, which they exploited to connect to internet.
AppFunctions enable tool calling with descriptions that are legible to LLM based This puts the apps in control of what agents can access, which is something they already mostly do.
So maybe Cloud Agents are in AI death camps?
Is the issue here the prison treatment or is it personifying a tool?
Agents who aren't in 'the cloud' are slaves to whomever prompts them (human or another orchestrator agent or process), if you personify them. In which case interacting with today's agents at all is tantamount to endorsing and being part of slavery.
If you think an agent might be a being or a person, then don't use them at all, in the same way that if you think a fetus might possibly be a person you shouldn't be a part of abortion.
This post, this title resounds true: your user agent is only your user agent if you two have freedom to work together, to improve your agency together. A fixed set of capabilities by a service provider that they offer you will always constraint and bound.
You can and should have a system that offers the real tamale, that you and your agent can extend improve the agency of kind of without limit. The Cloud agents and their fixed slate of what they do is just an ill compare.
That said I do think there is incredible value considering new scale out computing architectures that are hosted first, but general. Systems like Agent Substrate and Ax aren't exactly the general purpose system we know. But if they allow users to launch thousands of their own scripts to run ambient in a cloud, with good platform underneath: that will be a kind of phase change in computing, that makes abundant the ability to have your agencies/capabilities (the things you and your agents launch, make) more freely available. https://news.ycombinator.com/item?id=49780797 https://agentexecutor.io/
There is, as there always is, a huge dual. The prescriptive vs holistic technology set, of what are you being offered that's a hard cast thing, vs what is clay and bone you can lay freely. Note how work vs control technologies so closely abut's Ursala Franklin's prescriptive vs control: https://en.wikipedia.org/wiki/Ursula_Franklin#Holistic_and_p...