San Francisco’s attorney general is done waiting for the app store gatekeepers to police themselves. David Chiu fired off cease-and-desist letters this week to Apple and Google, ordering them to yank 13 apps that use AI to generate non-consensual explicit imagery — the kind of tools that let anyone transform a fully clothed photo into a deepfake nude in seconds. Wired obtained the letters, which warn the tech giants they’re violating California laws that prohibit supporting services that create deepfake pornography.
Chiu pulled no punches when describing the scale of the problem. “We were absolutely horrified by the ubiquitous nature of this technology and how it was being advertised and marketed,” he told Wired. The victims are overwhelmingly women and children, and the fallout isn’t abstract. “These images are used to bully, humiliate, and threaten women and girls,” Chiu said. “This industry has a horrific impact on one’s reputation, mental health, loss of autonomy. There have been victims who’ve been suicidal.”
The numbers back up his urgency. One of the flagged apps had racked up more than a million downloads and openly advertised features to sexualize images of women or generate “free and uncensored” videos. While Wired declined to name any of the 13 apps — a sensible call to avoid giving them free advertising — Chiu’s office estimates Apple and Google have likely made millions in fees from these apps by looking the other way rather than building better detection systems. That’s the core of his legal argument: the stores aren’t neutral platforms when they’re actively profiting from tools designed to harass.
Google, at least publicly, is playing ball. Spokesperson Dan Jackson told Ars Technica that the five apps Chiu flagged have already been suspended from the Play Store for violating policies against harmful content. He added that Google has suspended “hundreds of violating apps” and restricted search terms like “nudify” on the store. The question Chiu’s action raises is why it took a threat of legal consequences to get there. For years, these apps have sat in plain sight, their very existence a signal that platform moderation moves at a pace that’s wildly out of sync with the speed of AI-powered abuse.
