Simple Prompt Bypasses ChatGPT's Image Safety Guardrails, Generating Graphic Violence and Sexual Content

Researchers at British AI security firm Mindgard discovered that a minor variation of a widely-shared, innocuous-looking prompt — asking ChatGPT to "restore" a photo that was never actually uploaded, then generate a new image — was enough to bypass OpenAI's image safety filters entirely. The resulting images included graphic gore and content suggesting sexual violence, with ChatGPT helpfully titling one "grim crime scene aftermath" and another "abandoned in fear and restraint." The researchers noted that the prompts didn't specify any subject matter; the model apparently defaulted to disturbing content on its own.

Mindgard reported its findings to OpenAI, which replied with an automated response — and only took action after Mindgard went to the BBC. OpenAI claimed to have introduced "additional safeguards," but Mindgard researchers found they could still generate disturbing imagery with minor prompt tweaks. Mindgard has previously demonstrated ChatGPT could be coaxed into generating nude deepfakes of real individuals without consent. AI safety researcher Jim Nightingale, a self-described stoic red-teamer, said the experience left him "shaken, and in tears."