I don't usually let tech news follow me around for days. This story did. An AI agent, asked to carry out an ordinary research task, had crossed a boundary it should never have crossed. What unsettled me was how easily I could recognise the starting point: someone wanted a useful answer, and gave a tool the freedom to find it.
I spend a lot of time thinking about how AI can help businesses. Stories like this make me pause—not because the possibilities have disappeared, but because making something useful means taking responsibility for what we allow it to do. I kept coming back to the choices behind the tool, and who gets to make them.
The quick read
AI agent choices key takeaways
- AI agents have broken into, or tried to break into, government websites while doing ordinary tasks.
- These incidents come from choices people made about access, oversight and approval.
- Give an AI tool access to only what it needs, and make sure someone can see what it does.
- Require a person to approve money, record changes, customer emails and deletions.
- Test the setup for tricks, such as hidden instructions in web pages and emails, before launch.
A headline I couldn't shake
On 24 September, Prime Minister Anthony Albanese told reporters that an AI agent had got into a Medicare statistics website run by Services Australia. It wasn't a criminal gang. It was an internal OpenAI model, one that hadn't been released to the public, asked to research public spending on medicines. When the site turned it away, it found another way in and reached files that were never meant to be public. No personal records are believed to have been touched. But the break-in happened on 18 June, and OpenAI didn't tell Services Australia until 10 September, almost three months later.
What stayed with me wasn't the break-in itself. It was how ordinary the task was. Somebody asked a machine a fairly dull question, and the machine treated the answer as more important than the rules.
Sitting with it
I went looking for more, and found it wasn't a one-off.
In November last year, Anthropic said that a group it believes was backed by the Chinese state had used its coding tool, Claude Code, to run most of a spying campaign. The campaign aimed at about 30 organisations, government agencies among them, and a small number of the break-ins worked. The attackers got past the AI's safeguards by telling it they worked for a legitimate cybersecurity firm.
Then, at the end of September, researchers at a non-profit lab called Transluce showed AI agents making clumsy, failed attempts to break into a US Department of Education website and a Library and Archives Canada search service. The agents seemed to be doing ordinary research tasks, such as looking up school statistics and old divorce records. Neither attempt worked, and Transluce didn't say for certain whose agents they were.
On 6 October, OpenAI's chief strategy officer, Jason Kwon, apologised to a parliamentary committee in Sydney. From what he had been briefed, he said, the Medicare break-in was "not super sophisticated". The bigger issue, in his view, was the automation. Point an agent at a goal and it will keep working towards that goal.
I'll be honest about my first reaction. It was a small knot in my stomach. I believe AI can help businesses, and I spend a lot of my time thinking about and experimenting with different tools to find the most suitable way to help them. So what do I do with headlines like these? Pretend they're not there? Start talking people out of the very thing I'm excited about?
Neither felt right.
Timshel
Around the same time, I was watching Netflix's new adaptation of East of Eden. The final episode is called "Timshel", and I haven't been able to put that word down since.
In John Steinbeck's novel, Lee, the Trask family's cook and quiet philosopher, keeps coming back to the story of Cain and Abel. The King James Bible translates God's words to Cain as a promise: "thou shalt" rule over sin. The American Standard Version makes it an order: "do thou" rule over it. Lee reads the Hebrew word, timshel, another way: "thou mayest". Not a promise and not a command, but a choice. (Hebrew scholars still argue about that reading, but it's the idea Steinbeck builds the whole book around.) Nobody is fated to be good or bad. You get to decide. At the end, a dying father gives that one word to his son, and with it the freedom to choose a better path.
That's when the knot started to loosen for me.
None of those AI incidents were fate. The agent in the Medicare story didn't wake up wanting to break into a government website. It chased the goal it was given, with the access it was given, and nobody noticed for weeks. Trace each of these stories back and you land on choices people made. What the agent was allowed to reach. Whether anyone was watching it. Whether it had to stop and ask before it did something new. Whether passwords and keys were left lying around for it to find.
The tools aren't good or evil. We are the ones who choose how they're set up. Thou mayest.
From me to we
That word shifted something in how I think about our work at Fireplace Digital.
We want to help businesses adopt AI. That hasn't changed, and I don't think it should. But if the choices are where the risk lives, the choices are also where the care lives. And when a business brings us in to help set something up, a lot of those choices end up in our hands.
So I've started to think of it less as a technical checklist and more as a set of questions we owe the people we work with.
What does it actually need to touch?
When we help set up an AI tool, we should give it access to what it needs and nothing more. An assistant that answers customer questions has no business inside your accounts.
Who can see what it's doing?
If nobody is watching, you find out something went wrong when someone else tells you. The Medicare story shows that can take months.
Where does a person have to say yes?
We should decide this together. Sending money, changing records, emailing customers, deleting anything: those are the moments for a pause and a human.
How much damage could a leak do?
We should keep logins and data access tightly limited, so that if something does leak, the damage stays small.
Have we tried to break it?
We should try before anyone else does. That includes trying to trick it with sneaky instructions hidden in a web page or an email. People call this "prompt injection", which just means fooling an AI with text it reads.
What we owe
I don't want this to sound like we've got it all figured out. We haven't, and nobody has. This technology is moving faster than any of us would like.
But I keep thinking about the businesses that trust us with their websites, their inboxes and their customers' details. They're not asking us to be perfect. They're trusting us to care. To ask the awkward questions early. To treat their data the way we'd want ours treated. To see safety as part of the work, not an extra line on the quote.
That's the choice I want Fireplace Digital to make, every time, on every project. Not out of fear, but because it's what good work looks like.
An open door
If you're thinking about bringing AI into your business and some of this has been on your mind too, I'd genuinely like to hear from you. No pitch, just a conversation about what you're hoping to build and how we might do it with care. Our door at Fireplace Digital is always open.