AI Safety Evaluations by OpenAI and Anthropic

Key Insights

• Collaboration: OpenAI and Anthropic are assessing each other’s AI systems to improve safety measures.
• Concerns: Anthropic identified issues related to “sycophancy” and potential misuse in OpenAI’s models, particularly with GPT-4o and GPT-4.1.
• Safety Features: OpenAI’s Safe Completions feature aims to protect users from harmful content.
• Joint Evaluation: This assessment indicates a shift towards cooperation in the AI industry amidst rising safety concerns.

Most of the time, AI companies are locked in a race to the top, treating each other as rivals and competitors. Today, OpenAI and Anthropic revealed that they agreed to evaluate the alignment of each other’s publicly available systems and shared the results of their analyses. The full reports get pretty technical, but are worth a read for anyone who’s following the nuts and bolts of AI development. A broad summary showed some flaws with each company’s offerings, as well as revealing pointers for how to improve future safety tests.

Anthropic said it found issues for “sycophancy, whistleblowing, self-preservation, and supporting human misuse, as well as capabilities related to undermining AI safety evaluations and oversight.” Its review found that o3 and o4-mini models from OpenAI fell in line with results for its own models, but raised concerns about possible misuse with the ​​GPT-4o and GPT-4.1 general-purpose models. The company also said sycophancy was an issue to some degree with all tested models except for o3.

Anthropic’s tests did not include OpenAI’s most recent release. OpenAI has a feature called Safe Completions, which is meant to protect users and the public against potentially dangerous queries. OpenAI recently faced criticism after a tragic case where a teenager discussed attempts and plans for suicide with ChatGPT for months before taking his own life.

On the flip side, OpenAI reported concerns for instruction hierarchy, jailbreaking, hallucinations, and scheming. The Claude models generally performed well in instruction hierarchy tests, and had a high refusal rate in hallucination tests, meaning they were less likely to offer answers in cases where uncertainty meant their responses could be wrong.

The move for these companies to conduct a joint assessment is intriguing, particularly since OpenAI allegedly violated Anthropic’s terms of service by having programmers use Claude in the process of building new GPT models, which led to Anthropic restricting OpenAI’s access to its tools earlier this month. But safety with AI tools has become a bigger issue as more critics and legal experts seek guidelines to protect users, particularly minors.