It’s a tale as old as (online) time: a company gets hacked, takes a while to realise, then has to decide whether to cover it up or air its own dirty laundry.
I’ve been writing about tech for the best part of two decades, and I’ve watched this play out again and again, up close. Each time, I’m struck by how self-destructive the “hide our failures” instinct ultimately becomes.
Cybersecurity professionals interviewed for one 2024 study described office politics and defensiveness getting in the way of working out what went wrong, as people shifted responsibility to avoid being held personally accountable.
And I’ve personally spoken twice with Tim Brown, former CISO at SolarWinds, about the very famous 2020 “Sunburst” hack. One of his recurring arguments is that the “good guys” need to share what went wrong, even with their direct competitors. Sometimes everybody benefits from eating a little crow.
And you know what? The Australian government, technically, agrees!
It introduced mandatory reporting of personal data breaches likely to cause serious harm back in 2018, trying to force everyone to be smart about this. After all, hackers only need to win once and have no compunctions over how they do. To prevent a hack, the good guys need to win every. single. damn. time. That’s much easier to do if you’re on the same team as your competitors – at least when it comes to fobbing off threats.
Anyway, it’s been six years since the Sunburst attack and we’re still seeing organisations being slow to act and sheepish about their failures. Only now they also have to cover their arses against loose and irresponsible friendly fire, as well as the usual malicious nonsense.
Which brings us to ChatGPT daddy OpenAI.
What exactly happened in the Medicare breach?
In case you’ve been under a rock or diverted by more pressing issues this week, here’s the TLDR:
In June, an experimental OpenAI model gained unauthorised access to Medicare’s statistics portal while trying to answer a research question. It retrieved internal files, credentials and aggregate statistics, and wrote files to the system. OpenAI claims individual patient records were not accessed.
As it later turned out, OpenAI’s agents had also been poking around US government websites. So it’s nice that the models find our data as sexy and alluring as big, juicy US data, I guess? Although I suspect they tried a bunch of international sources and these are just the ones we’re hearing about. That last bit is my own rank speculation.
OpenAI says it discovered the Australian activity in mid-August but only notified Services Australia on 10 September. It says it wanted to finish investigating before providing a detailed account. It now acknowledges it should have shared preliminary findings sooner.
The Australian government waited two more weeks before telling the public about it.
The day before the government finally did decide to let us all know what happened, another embarrassing fluff: almost 300 security chiefs and corporate big-shots got roped into a massive Teams meeting, expecting a cybersecurity briefing from the Department of Home Affairs and the Australian Signals Directorate. Instead, the advertised “information sharing” session featured what attendees described as an OpenAI sales pitch. Essentially, they were catfished into the digital equivalent of a timeshare presentation.
Super great. Five stars all round.
Anyway, this is not a tech newsletter about tech companies. This is a tech newsletter about people. And the people in question here made a bunch of extremely human decisions that seem to be digging us into a hole increasingly filled with pickles. Maybe the pickles are the shovels? Look, I clearly lost control of that metaphor but I think you catch my meaning.
People keep making peopley choices and those peopley choices are both creating the tech problems and then making them much worse.
The dark side of “helpful”
Those choices actually start with one of the most annoying features of the most recent LLM models: they are built with the instinct to appear smarter than they are, more confident than they should be, and to please you in whatever way they can. This is not accidental. The people designing them are deciding what counts as a good job and making this true.
Have you ever come up against an issue where you clock the AI has answered a question that you did not ask and you’ve had to redirect it? Yes, of course you have. It is a very common problem.
In the Medicare case, the model was asked to answer a research question and it was so committed to trying to do a good job it just ... hacked government websites to do it.
This is being expressed as an accident but I’d argue it’s actually a fundamental feature of the design. It casts both the company’s and government’s responses to the incident in a different light.
This is not even close to the first time OpenAI (and its peers of course) have been caught oopsying on the internet. In its account of the Hugging Face incident, OpenAI described agents pursuing increasingly risky routes when tasks seemed impossible, and said some boundary-probing behaviour had been reinforced during training.
It seems to have become a common occurrence. And they keep saying sorry, but persist in designing models that routinely do so because that is literally how they are programmed. To me that makes the apology pretty hollow.
Given that the whole way these models came about was largely by scraping all available content without anyone’s knowledge or consent, the fact that they keep “accidentally” violating new datasets is also a limp excuse. It’s like me stealing your cupcake and maintaining full eye contact while I eat it mumbling “so sorry!” with that delicious frosting all over my face.
Sorry, not sorry
And on the other side of the equation we have a government. One that is trying very hard to position itself as more active on AI regulation than many other middle powers, but really showing its hand here. I’ve long been concerned that our relative power to affect AI outcomes in this country is pretty weak.
As I said in my conversation with UQ’s Michael Noetel for the Tokenised Human podcast: we’re, like, 25 million people perched around the edge of a desert. Even if we had an absolutely perfect regulatory regime, our power to influence what these tech giants do is pretty limited.
He actually made the argument that this was a good reason for Australia to become the datacentre capital of the world. Which, gotta hand it to him, does seem like a baller power play even if there are a bunch of energy issues we’d have to iron out quick smart to make that happen.
But we are not, my friends, behaving like ballers. We are apparently doing some version of: “It’s ok, guys, it was an accident.” To keep the tech overlords on side. As Deputy Prime Minister Richard Marles said during his press conference on the breach: “It matters to have a relationship with OpenAI so that we can have that cooperation.”
Cooperation on what exactly? On not building a system that is by its very nature designed to do exactly this thing it did? It shows how weak governments truly are in the face of these technology companies.
And that is a pretty awkward power dynamic. The government needs OpenAI’s help to understand what OpenAI’s own system did. The company is both the subject of scrutiny and the primary source of information needed to perform that scrutiny.
I can see why Marles wants to keep them talking. But I also wonder how hard you’re willing to push someone when you’re worried they might stop being helpful. Being grateful for their cooperation can start to feel a lot like you being grateful I “saved you” from those cupcake calories.
The question, to me, is what that relationship buys us beyond a better explanation afterwards. Earlier warnings and faster access to information would be useful, I guess?
But meaningful influence would also mean being able to challenge what these companies consider a “successful” product. Sometimes “I couldn’t complete that task within the rules” needs to count as the correct result, even if the model could produce something more impressive by ignoring them.
If we keep rewarding the result – and treating any crossed boundary as an unfortunate side effect – I suspect we’ll be having a lot more very cooperative conversations about things everyone agrees should never have happened.
Ending the affair before you tell your spouse
Whatever our power to change OpenAI’s products (or the products of any similar company) the government does control its own candour. It criticised the company’s slow disclosure while simultaneously defending its own decision to wait two more weeks before telling us.
Marles said the government needed enough information to confidently reassure Australians about the impact. But that’s weird to me on two fronts: one, that framing assumes that the breach would be found to be something they could reassure people about – what if it wasn’t? And two, I’d argue “we’re investigating” would have been sufficient as a first alert. Because two weeks is both too short for a comprehensive investigation and too long for a cursory one.
And most importantly: is that delayed disclosure meeting the standard of transparency the government expects of others in historic cases of large data breaches? It feels instead like a manifestation of the deeply human instinct to keep the uncomfortable thing private until you feel ready to field the questions.
The apology sounds kinda like a sales pitch
To be fair, the government has now instructed departments to review their own defences against AI threats in the future. But that seems like a bare minimum and still more or less lets the culprit off the hook. OpenAI isn’t being asked to pay for that upgrade, as far as we know.
In a kind of round robin of irony, OpenAI has offered credits and technical assistance as a mea culpa, which is a neat trick: the company whose technology created the problem offers more of its technology to help fix it, making us more dependent on the very company we’re supposed to be holding accountable.
And even knowing what they had done — and surely that it was all going to have to come out sooner or later — the company had the balls to pre-empt the reveal with a security-related sales pitch to the top end of town.
I also want to know whether the feeling that this incident was friendly fire makes everyone feel entitled to take it less seriously.
Because if being a valued partner buys you slower disclosure, softer scrutiny or more tolerance for repeat incidents, that has consequences. It tells the company that remaining useful matters more than changing the behaviour.
And it makes everyone else’s safety depend partly on who caused the problem and how badly we want to stay friends with them. Being one of the “good guys” ought to mean being more willing to share your failures and help prevent the next one. It shouldn’t be a loyalty program that earns you a more forgiving response.
If this letter gave you something to think about, consider forwarding it to one person who’d enjoy it too! Word of mouth is how a newsletter this new finds its people.
And I’d love to hear from you — reply to this email, comment in the app, or find me on LinkedIn.