OpenAI monitoring flagged AI Dungeon players prompting GPT-3 into stories of child sexual abuse; Latitude's emergency filter set off a user revolt

In April 2021, a new OpenAI monitoring system revealed that some players of AI Dungeon were typing prompts that made the game generate stories depicting sexual encounters involving children. OpenAI told Latitude, the Utah startup behind the game, to take immediate action. "Content moderation decisions are difficult in some cases, but not this one," CEO Sam Altman said. "This is not the future for AI that any of us want."

Latitude had been running on GPT-3 since July 2020 and was one of the few OpenAI API customers not required to use the provider's own filters. When it switched on a moderation system in late April, users revolted. The filter threw warnings on the phrase "8-year-old laptop," and Latitude's plan to manually review flagged stories drew accusations that the company was reading private adult fiction. One player who exploited a since-fixed security flaw to download several hundred thousand adventures from four days in April said 31 percent of the 188,000 he sampled contained words suggesting they were sexually explicit. Latitude is now required to use OpenAI's filtering technology.