I got an email from my kids’ school this week urging parents to please, please, please not buy their children Meta’s latest pervert device, the Muse Charm, or any of the countless similar AI companion toys already on the market (or likely to hit over the next year or so).
It made me sigh. Loudly.
The timing is particularly interesting given that Australia is on the brink of introducing new legislation — it might even drop today if the rumours are to be believed — that would ostensibly throw the onus for protecting people online back on tech companies.
(When you hear the words “digital duty of care” and your brain tries to glaze over, I encourage you to slap that brain and tell it to pay attention.)
As one of the bill’s advisors, human rights lawyer and Wotton Kearney partner Leanne Ho, told an executive round table I was at yesterday: “It is about shifting responsibility from the individual, if they have been harmed in the online world, to the service provider.”
It would require tech companies to identify and address risks of serious harm to all users, with additional protections for children against harmful content and potentially damaging design features such as infinite scroll.
The irony that Meta — founded by a man who started out building websites to rate the hotness of women, before moving on to vague gestures at “staying in touch with your friends” — now wants to replace those friends with a little intrusive surveillance companion is not lost on me.
As any first-year philosophy major who’s just discovered Foucault will tell you, the panopticon is a prison designed so that inmates can never be sure whether they are being watched, so they eventually learn to police themselves.
In the case of AI, we’re actually living a different and perhaps even more dystopian fever dream than that: a world where we know we’re being watched all the time because we pay for the privilege and complain when the tool isn’t watching us well enough.
(“Why doesn’t it remember my favourite colour is green and I like oat milk more than soy?!”)
A ‘friend’ with a conflict of interest
The fact that Meta claims this product is for adults doesn’t really put my mind at ease.
Things can be marketed as being for adults and still appeal to children. If we wanted to be really cynical we might even suggest there are loads of products that are marketed to children while being labelled as specifically for adults — candy floss vapes or alcoholic lolly water, for example. Moreover, manufacturers can know that children will be attracted to them and simply not care.
And AI companions specifically designed for children are already on the market. In January, Common Sense Media tested three AI toys and identified problems including inappropriate responses, some sexualised content, unreliable safeguards, extensive data collection and features designed to encourage emotional attachment.
But I’d argue the problem is much bigger than whether these devices might say something inappropriate to a kid.
The content itself doesn’t need to be harmful for addiction to develop, for the design of a tool or platform to encourage that addiction, or for negative social impacts to appear.
Substituting a device for friendship or parenting is problematic in a deep sense.
Not teaching them how to shoot up some drugs is the lowest possible bar. It could produce nothing but the most benign, polite responses to every single question and still end up a net negative to the emotional, psychological and social wellbeing of that kid.
We actually don’t know for sure if AI companionship like this is ultimately damaging. But we do have some data that flashes warning lights.
One 2025 study from the MIT Media Lab found adults using ChatGPT more frequently tended to report greater emotional dependence, more problematic use, less human socialisation and greater levels of loneliness.
Another analysis of several papers on the subject, published in the Journal of Consumer Research last year, found that while using an AI companion did immediately alleviate feelings of loneliness for adult participants, the effects were “momentary in nature”.
And its not like we haven’t been through anything like this before. We have two decades of experience with the impacts of social media to draw on. A randomised experiment published in the American Economic Review in 2019 found temporarily deactivating Facebook improved subjective wellbeing, increased offline socialising and reduced political polarisation.
A separate paper published in Science in 2018 found disinformation spread faster than true information on Twitter.
This isn’t to say social media hasn’t provided important social outlets in particular circumstances. Queer kids living in the country, for example, can find communities and connections that would otherwise have been inaccessible to them.
But I don’t think we can look at the past two decades and conclude that social media has improved people’s social wellbeing on the whole.
In 2026, for all the promises that social media would connect us, people are, in general, more socially isolated. According to the Household, Income and Labour Dynamics in Australia (HILDA) Survey, the estimated proportion experiencing episodic social isolation rose from 16.9 per cent in 2001 to 29.9 per cent in 2023.
The ABS General Social Survey found that the proportion of Australians seeing friends or family outside their household at least weekly fell from 68 per cent in 2019 to 53 per cent in 2025.
And so I would argue that any AI-enabled toy exposes a child not just to the specific dangers of inappropriate content, but to a cascade of second- and third-order effects, some of which we cannot firmly predict.
Adulthood isn’t a safety feature
Which raises a broader question — one that I know will make some people kind of uncomfortable.
In Australia we have pushed through world-precedent-setting legislation to (attempt to) restrict children’s access to social media. We police the consumption patterns of children because we recognise certain products and behaviours as dangerous or potentially harmful. But why are we so reluctant to take similar measures to protect ourselves?
Phone addiction and social media addiction don’t just affect children, and loneliness is a problem affecting adults of all ages.
Whenever I raise the idea of restricting adults’ access to social media, whoever I am talking to will inevitably argue that adults should have the freedom to make their own choices. That further regulation would be a nanny-state initiative or government overreach.
And hey, I’m a journalist. I love me some freedom of speech. Importantly, regulating how a technology is designed and operates is not the same as telling people what they can and can’t say. (Also, we regulate all kinds of speech, but let’s not get off track.)
But we place restrictions and disincentives on antisocial and harmful conduct in our societies all the time.
We tax cigarettes heavily and place horrible graphic warnings on the labels to dissuade adult consumption. We ban drink driving even though alcohol is legal. We outlaw heroin and meth and cocaine. We regulate car safety, fire standards and ban hoverboards, even restrict where people are allowed to ride their bicycles and whether they wear a seatbelt or a helmet.
We restrict or disincentivise products and behaviours because we have deemed them unsafe or harmful, including when products are used exactly as designed.
There are, for example, circumstances in which potentially harmful substances might be completely appropriate and allowed. Morphine during surgery or stimulant medication prescribed to a patient with ADHD, for example.
I don’t know why tech gets treated like a special category.
You can check out any time you like
For some reason, we have decided that social media serves a meaningful or important widespread function — connecting people or marketing — even though I would argue that it hasn’t delivered on those promises for a very long time.
And so now small businesses are at the mercy of an algorithm.
I personally find it perverse that my physiotherapist has to be both an excellent healthcare provider and a part-time social media influencer against her will, just to keep the lights on and the doors open.
It’s not how she’d like to spend her time, but choosing not to participate in this market environment could kill her business.
That’s a form of dependence that has very little to do with whether someone wants to be on social media or gets any social benefit out of it at all.
Relying on perfect individual choices gives us a false sense of autonomy because many of our systems require community cooperation to function. Trying to quit alone can be functionally impossible.
It also puts the onus back on individuals to fight against unfair structures or fix problems that are not their responsibility and that cannot be solved by individual choice. Hence the plan for the digital duty of care update.
The whole thing reminds me of recycling. Look how easily they made waste a matter of personal responsibility rather than requiring manufacturers to produce less damaging packaging. Like you could ever shop your way out of climate change if you just made enough perfect choices in the supermarket aisle.
And it reminds me of the TV show The Good Place. (Spoiler alert!)
In the final season they find out everyone on earth is going to Hell because even seemingly harmless individual choices have consequences that people cannot reasonably anticipate or control. No one can be good enough because the systems are inherently evil, and no individual alone has the power to fix the problem through their own choices.
So, you can delete your social media accounts, but you can’t make everyone else leave. You can’t individually change the business models or design incentives of the platforms, and you may not be able to avoid the consequences of everyone else’s participation.
How free is a choice you can’t stop making?
And then there’s the issue of addiction.
Adults would complain if we shut down all social media. They would call it a nanny-state initiative and government overreach. But is that because we are addicts defending the availability of our drug of choice?
And even if someone recognises that their relationship with social media is unhealthy, that doesn’t mean they can simply walk away. They may be psychologically dependent on the platform, socially dependent on it, economically dependent on it, or some combination of the three.
We’re asking people to resist products specifically designed to capture their attention while also continuing to depend on them in other areas. We don’t accept that as a sufficient response to any other product of addiction.
It’s this dependence that brings us back to AI, because we know people are already worried about becoming over-dependant on these tools to do their jobs — and other things too.
Are there, or will there be, use cases where it is a reasonable or beneficial tool? Of course.
Does that mean I should have an AI companion in my pocket, available to talk to me every moment of the day?
Probably not.
If this letter gave you something to think about, consider forwarding it to one person who’d enjoy it too! Word of mouth is how a newsletter this new finds its people.
And I’d love to hear from you — reply to this email, comment in the app, or find me on LinkedIn.