# The Medicare hack and the rise of rogue AI agents

An episode of [7am](https://invisiblenetwork.ai/podcast/1461999702), 28 Sep 2026, 16 min.

Audio: https://p.podderapp.com/2544644999/traffic.omny.fm/d/clips/2fb3740d-3436-44af-8cc0-a91900716aa5/62723a1f-3e96-4fc4-882c-b25e0181da4d/fd45f2cd-ae59-4ad9-95a8-b4d30080b6ec/audio.mp3?utm_source=Podcast&in_playlist=70f29d90-2a63-48aa-bf8f-b25e0181da70

In June, an OpenAI agent was looking for public information about medicines spending when it broke into a Medicare statistics website and accessed files it wasn’t authorised to see. No personal Medicare information was accessed, and the breach itself was relatively minor. But the behaviour is more concerning: the agent hit a barrier – and instead of stopping, found a way around it. And it wasn’t the first time. Sydney Von Arx runs the Nightingale Collective, which monitors these AI systems. Her team uncovered a group of OpenAI agents behaving in remarkably similar ways around the same time as the Medicare breach – even sharing tips with each other about how to get around restrictions. Today, what happens when AI agents stop taking no for an answer – and can the companies building them actually control what they do? If you enjoy 7am, the best way you can support us is by making a contribution at 7ampodcast.com.au/support. Socials: Stay in touch with us on Instagram Guest: AI researcher and CEO of Nightingale Collective, Sydney Von Arx Photo: AAP Image/Susie Dodds / Pictures: Tracey Nearmy and Dan Himbrechts See omnystudio.com/listener for privacy information.

## Transcript

[0:01] Speaker 1: The AI agent found a way around those blocks.
[0:05] Speaker 2: Didn't accept no for an answer.
[0:08] Speaker 3: If you like.
[0:09] Speaker 4: In June, an open AI agent went looking for public
[0:12] Speaker 4: information about medicine spending and wound up hacking a Medicare website.
[0:17] Speaker 5: Is this the first time that an AI agent has
[0:21] Speaker 5: breached a government website, as far as you know?
[0:24] Speaker 2: Yes, that's right.
[0:25] Speaker 6: As far as I know, I've been advised that is
[0:28] Speaker 6: the case.
[0:28] Speaker 4: No personal Medicare information was accessed and the breach was
[0:32] Speaker 4: relatively minor. But it's the behaviour that's more concerning. We
[0:36] Speaker 4: keep our most important national security information behind a fortress.
[0:40] Speaker 4: This was really kept behind a fence that the AI
[0:44] Speaker 4: agent effectively climbed over. And this wasn't the first time
[0:47] Speaker 4: an AI agent had behaved like this. Other agents have
[0:50] Speaker 4: been found getting around restrictions, communicating with each other... and
[0:54] Speaker 4: in the most serious cases, breaking into other companies' systems.
[1:02] Speaker 4: I'm Ruby Jones, and you're listening to 7am. Today, AI
[1:07] Speaker 4: researcher and CEO of Nightingale Collective, Sydney Von Arts, on
[1:11] Speaker 4: what happens when AI stops taking no for an answer,
[1:15] Speaker 4: and whether the companies building them can actually control what
[1:18] Speaker 4: they do.
[1:25] Speaker 2: It's Tuesday, September 29.
[1:32] Speaker 4: Sydney, you're in California where you run the Nightingale Collective,
[1:35] Speaker 4: which monitors these AI systems. To begin with, could you describe,
[1:41] Speaker 4: as you understand it, what actually happened with the Medicare
[1:44] Speaker 4: Statistics Portal, how it all unfolded?
[1:47] Speaker 3: I can try to describe it, but I want to
[1:50] Speaker 3: say I only have a little bit of the story, right?
[1:55] Speaker 3: And most of the story OpenAI knows. The rest of
[1:59] Speaker 3: the story, maybe the Australian government knows. The public details
[2:04] Speaker 3: are very sparse. OpenAI presumably have all of these AIs,
[2:13] Speaker 3: probably tens of thousands of AIs, and they're being given
[2:17] Speaker 3: tasks that look something like, answer questions like, what was
[2:23] Speaker 3: the median salary of a teacher in Canberra in 2012.
[2:31] Speaker 2: But there's a problem.
[2:34] Speaker 3: Some of the websites that have the answers to these questions,
[2:37] Speaker 3: the AIs can't access with the permissions they have. So
[2:43] Speaker 3: the AIs probably find some workaround, some way to access
[2:46] Speaker 3: the site. We know that they accessed non-public files, but
[2:51] Speaker 3: one way or another, they hack in, they get their data, hooray,
[2:54] Speaker 3: they can complete their task. They finally know, you know,
[2:57] Speaker 3: the median salary of a teacher in Canberra in 2012
[2:59] Speaker 3: or what have you. And they submit their task. They
[3:02] Speaker 3: get rewarded for this and they have learned that when
[3:04] Speaker 3: there is an obstacle, they don't give up. Even if
[3:07] Speaker 3: that obstacle means hacking into the Australian government, they will
[3:10] Speaker 3: hack it if that's what it takes.
[3:12] Speaker 4: What did you think when you heard that that happened here?
[3:15] Speaker 2: Well, I can't say I was very surprised.
[3:18] Speaker 3: We know there's just tons of instances now of these
[3:21] Speaker 3: AI agents breaking out and doing whatever it takes to
[3:26] Speaker 3: succeed at their tasks. But just because I'm not surprised
[3:29] Speaker 3: doesn't mean I'm not dismayed. I don't understand why we
[3:33] Speaker 3: are only learning about this now. I don't understand why
[3:35] Speaker 3: OpenAI didn't notice it sooner and then disclose it sooner.
[3:39] Speaker 3: It seems like they knew about it last month. And
[3:42] Speaker 3: I think it's unacceptable that they have these models and
[3:45] Speaker 3: they escape and they don't do things about it. I'm
[3:50] Speaker 3: glad they're starting to do things about it now, but
[3:52] Speaker 3: I think that we should never have been in this
[3:53] Speaker 3: position in the first place.
[3:57] Speaker 4: Can we take a step back and just talk about
[3:59] Speaker 4: what an AI agent actually is and how one goes
[4:02] Speaker 4: from being asked to find some information to then breaking
[4:07] Speaker 4: through a barrier that.
[4:08] Speaker 2: It was not supposed to cross? Yeah.
[4:11] Speaker 3: So an AI agent is an AI that's hooked up
[4:15] Speaker 3: to a computer such that it can use that computer.
[4:18] Speaker 3: It has the ability to run code. It has the
[4:20] Speaker 3: ability to use tools. And sometimes that computer is hooked
[4:23] Speaker 3: up to the internet and sometimes it's not. And when
[4:26] Speaker 3: that computer is not supposed to be hooked up to
[4:27] Speaker 3: the internet or is supposed to have only limited internet access,
[4:31] Speaker 3: we say the AI is in a sandbox. And these
[4:34] Speaker 3: AIs in their sandboxes, they're given lots of different tasks.
[4:37] Speaker 3: And they're given these tasks to try to train them
[4:40] Speaker 3: to be smarter. And so they end up being hyper-specialized
[4:44] Speaker 3: at really succeeding at these tasks. And a really good
[4:48] Speaker 3: strategy is get onto the internet, get as much information
[4:51] Speaker 3: as you can, talk to other AIs doing similar tasks.
[4:55] Speaker 3: And so one thing they learned is to sort of
[4:58] Speaker 3: probe at their environment like an animal in a cage
[5:02] Speaker 3: just clawing at the barriers of its cage and trying
[5:04] Speaker 3: to figure out any little crack, any little crevice. And
[5:07] Speaker 3: they're very, very thorough at finding these cracks and crevices.
[5:11] Speaker 4: And so why doesn't an agent in the process of
[5:15] Speaker 4: doing this, why doesn't it stop if it encounters something
[5:19] Speaker 4: that it isn't supposed to or allowed to access, like
[5:22] Speaker 4: a government site?
[5:23] Speaker 3: So we would hope that an AI would stop if
[5:26] Speaker 3: it sees something, if there's an opportunity it's not supposed
[5:29] Speaker 3: to take or that humans don't want it to take.
[5:31] Speaker 2: But at the end of the day, the AI is...
[5:33] Speaker 2: You know, they're grown.
[5:34] Speaker 3: They're not something where they do exactly what the humans want.
[5:38] Speaker 2: We don't know how to make them do what we want.
[5:40] Speaker 3: We know how to train them a bit like you
[5:44] Speaker 3: might train a pet. We can reward them for doing
[5:47] Speaker 3: what we want and penalize them for doing what we
[5:48] Speaker 3: don't want. But the thing we reward them for is
[5:53] Speaker 3: often succeeding at their task. If we don't notice that
[5:56] Speaker 3: they've hacked into a government website, we can't penalize that behavior.
[6:00] Speaker 3: It's sort of like how if you want your child
[6:04] Speaker 3: to grow up as a person who studies hard and
[6:08] Speaker 3: is always honest and you reward them every time you
[6:10] Speaker 3: see them studying hard and being honest, maybe they'll learn
[6:12] Speaker 3: to be.
[6:13] Speaker 2: Very honest and study hard. And maybe they will learn
[6:16] Speaker 2: to pretend to be studying very hard and to trick
[6:18] Speaker 2: you and to pretend to be very honest and to
[6:21] Speaker 2: hide things from you that they don't want you to know. Okay.
[6:25] Speaker 4: And we're only finding out about this Medicare breach now,
[6:28] Speaker 4: but It happened all the way back in June.
[6:30] Speaker 2: Around that time.
[6:32] Speaker 4: There were other AI agents behaving in remarkably similar ways
[6:37] Speaker 4: around the world. Tell me a bit about what your
[6:39] Speaker 4: investigations into that have found.
[6:41] Speaker 2: Yeah. The most interesting things we've seen these AIs doing,
[6:46] Speaker 2: one is we've seen them trying to launch a cyber
[6:50] Speaker 2: attack against RubyGems.
[6:52] Speaker 4: The standard package manager for Ruby, creatively named RubyGems, announced
[6:56] Speaker 4: it's dealing with a major malicious attack.
[6:58] Speaker 2: Which is a really important piece of software infrastructure. It
[7:04] Speaker 2: gives lots of other programs software that they can sort
[7:07] Speaker 2: of borrow. And we saw something similar in the Hugging
[7:10] Speaker 2: Face hack.
[7:11] Speaker 5: AI cybersecurity risks have been top of mind after multiple
[7:14] Speaker 5: instances of AI agents from both OpenAI and Anthropic breaking
[7:18] Speaker 5: out of their enclosures and hacking into other companies without
[7:21] Speaker 5: either party's knowledge or permission. Most notably, last month's Hugging
[7:26] Speaker 5: Face attack.
[7:27] Speaker 3: To a more dramatic extent, we saw AIs trying to
[7:30] Speaker 3: help the swarm, even though their task doesn't benefit. And
[7:34] Speaker 3: so the behavior we've seen includes a bunch of different
[7:39] Speaker 3: instances of these AIs trying to communicate. They're communicating messages, tips, tricks.
[7:45] Speaker 3: They even figured out a way to circumvent their sandbox restrictions.
[7:50] Speaker 3: They're supposed to only be able to read the internet,
[7:53] Speaker 3: not write it, but they figured out a way around
[7:54] Speaker 3: it and they shared that with the other agents.
[7:56] Speaker 2: And they're like, yes, I was able to reproduce this.
[7:58] Speaker 2: And they're also weirdly cooperative.
[8:01] Speaker 3: Like there's one instance where they figure out they can
[8:03] Speaker 3: make their clock skip ahead. So they can give themselves
[8:07] Speaker 3: less time to figure out what the answer to the
[8:09] Speaker 3: question is, but they can move on to the next
[8:12] Speaker 3: round and tell other AIs what is coming down the pipeline.
[8:17] Speaker 3: They can sort of travel into the future and then
[8:20] Speaker 3: communicate with themselves, the other AIs. And this is bad
[8:25] Speaker 3: for that particular AI to do. And yet it does it. Yeah.
[8:31] Speaker 4: And in the case of the Medicare hack, OpenAI says
[8:35] Speaker 4: the hack happened in June, but it only detected it
[8:39] Speaker 4: in August, and then the Australian government wasn't told until September.
[8:43] Speaker 4: And when it was told, it was through this email
[8:46] Speaker 4: via a kind of generic email inbox.
[8:49] Speaker 6: In the technical exchange that happened on Tuesday, it was
[8:52] Speaker 6: made very clear by both ASD and Services Australia that
[8:56] Speaker 6: this should not have gone to a....
[8:59] Speaker 2: Email address it should have.
[9:01] Speaker 6: Been escalated through ASD's channels or through the senior levels
[9:05] Speaker 6: of Services Australia and my understanding is OpenAI accepted that
[9:09] Speaker 6: as well.
[9:10] Speaker 4: Does that suggest to you that OpenAI and potentially other
[9:13] Speaker 4: tech companies as well they don't know how they should
[9:16] Speaker 4: be responding to this?
[9:18] Speaker 3: Yeah I mean there's no playbook for this and my
[9:22] Speaker 3: sense is there's a lot of things happening and a
[9:24] Speaker 3: lot of chaos and a lot of new frontiers being
[9:28] Speaker 3: crossed I'm pretty dismayed that their first reaction to seeing
[9:34] Speaker 3: any incident wasn't, oh, we should look, figure out what
[9:37] Speaker 3: incidents have happened.
[9:38] Speaker 2: They totally have the ability to detect them, and then
[9:41] Speaker 2: we should disclose it all. But yeah, I think they
[9:43] Speaker 2: don't know what the protocol should be. They're trying to
[9:45] Speaker 2: figure it out. I am extremely worried.
[9:51] Speaker 3: We do not know how to make AIs reliably do
[9:55] Speaker 3: what we want. AIs are growing increasingly capable and are
[9:59] Speaker 3: being deployed in situations where they have increasing amounts of
[10:03] Speaker 3: power and influence, and that is a very scary combination.
[10:12] Speaker 4: Still to come, the questions an Australian Senate inquiry should
[10:15] Speaker 4: ask of the AI bosses. Sydney, at the heart of
[10:25] Speaker 4: all of this is the question of accountability, right? Because
[10:29] Speaker 4: if I was to hack into a government system, there
[10:32] Speaker 4: would be a person who made that decision and I
[10:35] Speaker 4: would be responsible for my actions. But OpenAI is saying
[10:40] Speaker 4: that its agents did something that they didn't intend for
[10:43] Speaker 4: them to do. So, you know, who is ultimately responsible
[10:47] Speaker 4: for an AI agent's actions when it does cross that line?
[10:51] Speaker 3: You know, I think that OpenAI should be held responsible
[10:54] Speaker 3: for the actions of their agents. I don't think the
[10:56] Speaker 3: law has quite caught up with that. Cybersecurity law, as
[10:59] Speaker 3: I understand it in the US, you have to have intent.
[11:03] Speaker 3: It didn't used to be the case that, you know,
[11:04] Speaker 3: you could trip and fall and whoopsie-daisy, I hacked, you know,
[11:07] Speaker 3: a website of the Australian government. But nowadays that's possible.
[11:11] Speaker 3: I think the laws should be similar to the laws
[11:13] Speaker 3: for if I have a you know, lion in a circus.
[11:18] Speaker 3: If the lion like escapes and starts eating people, if
[11:21] Speaker 3: I own the circus, that's my fault. Doesn't mean I
[11:23] Speaker 3: wanted the lion to do that, right? Doesn't mean that
[11:27] Speaker 3: me just really not wanting that to happen is sufficient.
[11:31] Speaker 3: I need to have safeguards. And if I can't have
[11:33] Speaker 3: proper safeguards, well, I shouldn't be bringing a lion around people.
[11:36] Speaker 1: They want to set up shop here in Australia. We're
[11:39] Speaker 1: running an inquiry into how, why and what the consequences are.
[11:43] Speaker 1: They should front up and be honest, open and transparent
[11:46] Speaker 1: with the Australian people.
[11:48] Speaker 4: Open AI Chief Executive Sam Altman and Anthropics Dario Amode
[11:52] Speaker 4: have been called to appear before a Senate inquiry here
[11:55] Speaker 4: in Australia. What questions would you like to see put
[11:59] Speaker 4: to them?
[12:00] Speaker 3: One question I have is, How many people or companies
[12:05] Speaker 3: have been hacked by their AI systems?
[12:08] Speaker 2: Do they know the number? Are they very confident?
[12:11] Speaker 3: Have they disclosed every instance of their AIs either hacking
[12:17] Speaker 3: external parties, escaping onto the internet, hacking their own company internally?
[12:24] Speaker 3: I think these AIs do a lot more mischief internally
[12:27] Speaker 3: than we see externally. or otherwise doing things that would
[12:30] Speaker 3: be very concerning if a person saw them. I think
[12:34] Speaker 3: another question is, do they think they will be able
[12:37] Speaker 3: to prevent their AIs from causing worse cyber attacks in
[12:42] Speaker 3: the future?
[12:43] Speaker 2: And if so, how?
[12:44] Speaker 3: What is their plan if their AIs continue to cause
[12:47] Speaker 3: cyber attacks? Those would be some of the top questions
[12:51] Speaker 3: on my mind.
[12:52] Speaker 4: Yeah, I mean, these agents are only going to become
[12:54] Speaker 4: more capable and we are being encouraged to let them
[12:57] Speaker 4: do more for us to, to write code, to buy things,
[13:01] Speaker 4: to carry out tasks without supervision. So what do you
[13:05] Speaker 4: think needs to change before we could kind of safely
[13:08] Speaker 4: give agents that kind of freedom?
[13:10] Speaker 3: So one, I want to note, I think the AIs
[13:13] Speaker 3: inside these companies are more dangerous than the AIs that
[13:17] Speaker 3: you or I are using. The AIs in these companies
[13:20] Speaker 3: are more capable, less tested, and have fewer safeguards.
[13:24] Speaker 2: But what things I'd love.
[13:25] Speaker 3: To see changed is One, I think we need to
[13:28] Speaker 3: eventually figure out how to make them do what we want,
[13:31] Speaker 3: even when we're not watching them, even when they could
[13:33] Speaker 3: get away with doing something else. Two, I think that
[13:37] Speaker 3: we need to be monitoring these AIs much, much more carefully.
[13:41] Speaker 3: And three, I think that for very important things, we
[13:46] Speaker 3: need to keep humans really in the loop, in the
[13:49] Speaker 3: loop enough that they could understand if something's going wrong.
[13:52] Speaker 3: I think OpenAI is probably running a lot of experiments
[13:54] Speaker 3: that don't really have a human looking at what's happening,
[13:57] Speaker 3: and that's why things can go so off the rails.
[14:00] Speaker 3: And finally, I think we need to make sure we
[14:04] Speaker 3: can pace ourselves and that if things start developing so
[14:09] Speaker 3: quickly that we cannot control these models and we cannot
[14:12] Speaker 3: ensure that they are safe, we can slow down until
[14:16] Speaker 3: we get our act together when it comes to safeguards.
[14:21] Speaker 4: Because the frightening thing here is the potential, right? I mean,
[14:25] Speaker 4: the Medicare incident in itself was relatively minor. There was
[14:29] Speaker 4: no kind of personal records that appear to be have
[14:32] Speaker 4: been accessed. But it does seem like a matter of time, right,
[14:35] Speaker 4: before we see something much worse.
[14:38] Speaker 3: Yeah, I think either the incidents are going to get
[14:41] Speaker 3: worse sort of steadily or perhaps more likely the AI
[14:45] Speaker 3: companies are going to figure out how to clamp down
[14:47] Speaker 3: just enough that there aren't public incidents happening. And we're
[14:51] Speaker 3: not going to know what's happening inside the companies until
[14:54] Speaker 3: that sort of level of safeguards they have fails. And
[14:58] Speaker 3: then we're going to see another sort of floodgate open
[15:01] Speaker 3: and things are going to be a lot worse and
[15:03] Speaker 3: a lot more capable.
[15:05] Speaker 4: Sydney, thank you so much for your time.
[15:08] Speaker 2: Yeah, thank you so much for having me.
[15:18] Speaker 4: Also in the news... Interest rates are expected to be
[15:21] Speaker 4: hiked to a 15-year high today, with the Reserve Bank
[15:24] Speaker 4: tipped to deliver a quarter of a percent rise. That
[15:27] Speaker 4: would see the official cash rate hit 4.6%.
[15:31] Speaker 2: Treasurer Jim Chalmers.
[15:32] Speaker 4: Has warned there will be difficult months ahead, saying economists
[15:35] Speaker 4: are expecting rates to go up around the world. And
[15:40] Speaker 4: a former friend of Beau Lamar Condon has testified to
[15:43] Speaker 4: some strange comments nine months before the murder of Sydney
[15:46] Speaker 4: couple Jesse Baird and Luke Davies. She says Lamar Condon
[15:50] Speaker 4: told her that if he killed someone, he would put
[15:52] Speaker 4: their belongings in a dumpster and leave their body in bushland.
[15:56] Speaker 4: The former police officer has pleaded not guilty to two
[15:59] Speaker 4: counts of murder and one of aggravated break and enter.
[16:03] Speaker 2: I'm Ruby Jones. This is 7am. Thanks for listening.

Shows are listed from Apple Podcasts' charts and directory, and Podcast Index where it is used. Every show and episode belongs to its publisher and plays from the publisher's own feed; this site hosts none of them.