Case Study: Anthropic strengthens AI safety defenses with HackerOne red teaming

How Anthropic’s Jailbreak Challenge Put AI Safety Defenses to the Test

Anthropic, an AI safety and research company, faced the challenge of stress-testing the safety defenses of its Claude 3.5 Sonnet model to prevent the generation of harmful content, particularly related to CBRN weapons. To proactively identify risks and validate its new Constitutional Classifiers, it partnered with the vendor HackerOne to run a specialized AI red teaming challenge.

The solution from HackerOne was an eight-level jailbreak challenge that engaged a global community of security researchers to attempt to bypass Claude's guardrails. The challenge was highly successful, generating over 300,000 interactions from 339 participants. This collaborative effort resulted in four teams earning $55,000 in bounties for their findings, which provided Anthropic with critical insights into emerging attack vectors and directly contributed to strengthening its AI safety protections.

Anthropic

Dane Sherrets

Staff Solutions Architect