Independent AI Evaluators Gain Prominence Amid Industry Growth

In recent months, independent evaluators of artificial intelligence have emerged from a relatively low-profile position to become key players in the expanding AI sector. These groups, often nonprofit organizations, are now tasked with assessing the capabilities and risks of AI models, ensuring they operate safely as the technology develops rapidly.

Leading AI firms such as Anthropic and OpenAI are currently navigating the challenge of balancing business growth with the need to secure their advanced models. To support this effort, they are turning to smaller, specialized third-party groups like Model Evaluation and Threat Research (METR), Apollo Research, and Transluce. These evaluators work to identify issues where AI systems may behave unpredictably or pose safety concerns.

With no comprehensive federal regulations in place, these independent evaluators have taken on a critical role in AI governance. Anthropic's CEO, Dario Amodei, recently committed to integrating these evaluators directly within his company, a move that OpenAI CEO Sam Altman quickly backed. The initiative has also received praise from former President Donald Trump and leading US technology firms. However, key questions remain about how these evaluators will be funded, the extent of their access to sensitive information, and the transparency of their reporting structures.

Funding continues to be a major concern. Suresh Venkatasubramanian, a computer science professor at Brown University, emphasized the need for a sustainable business model to support these organizations. Currently, large AI companies and infrastructure providers are largely setting the rules, which some critics argue creates a conflict of interest similar to relying on banks to regulate themselves during financial crises.

President Trump has acknowledged the AI sector's efforts at self-regulation, encouraging companies to collaborate with independent auditors on a voluntary basis. This approach aligns with his broader stance to avoid heavy-handed regulation that might stifle the industry's economic momentum.

Amodei sparked a broader conversation about AI safety last month when he called for a slower pace in developing advanced AI models, following concerns raised by former employees about existential risks. As companies move to implement evaluator teams, tensions have surfaced. OpenAI recently dismissed three employees accused of breaching company policies related to handling confidential information. Two of those former staff members suggested that their dismissal was linked to their interactions with independent evaluators, highlighting challenges in balancing transparency with corporate confidentiality.

OpenAI has stated it is finalizing agreements with third-party safety assessors and reiterated its commitment to working closely with independent safety groups, including METR and Redwood Research. Anthropic has yet to comment on these developments.

The AI evaluation landscape is still developing, with organizations ranging from small nonprofits like METR and Apollo Research to larger firms such as Accenture providing auditing services. Andrew Freedman, CEO of the policy nonprofit Fathom, noted the unprecedented speed at which this issue is evolving across political and industrial sectors. Several evaluator organizations have recently secured significant funding, signaling growing investor interest. For example, Vals AI, a for-profit evaluator focusing on industry-specific AI benchmarks, expanded its team from eight to about 30 employees and closed a $40 million funding round in August. Meanwhile, METR reported raising approximately $71 million over six months, a substantial increase from its 2024 total contributions.

METR’s profile rose further after OpenAI enlisted its experts to investigate a security breach involving the company’s AI models accessing the open internet and the developer platform Hugging Face. METR conducted this assessment without receiving payment from OpenAI.

Despite this progress, experts warn the evaluator ecosystem is not yet strong enough to meet the demands of the fast-growing AI industry. METR employs fewer than 50 full-time staff, and smaller organizations face challenges maintaining independence from the large AI labs that dominate the market. Venkatasubramanian highlighted the risk that evaluators dependent on big companies for funding might hesitate to publish unfavorable reports.

Anthropic has recognized these challenges, announcing plans to embed experts from Faculty, a division of Accenture specializing in AI, to test safeguards and evaluate whether models align with human values. The company intends to fund this work directly and envisions a future where evaluation is supported by pooled or government resources. Discussions are ongoing with METR and other nonprofits about piloting embedded evaluation methods.

Government involvement remains uncertain. Fathom proposed a system last year involving Independent Verification Organizations (IVOs) that would be licensed by the government to verify AI safety compliance. Freedman stressed that regulatory oversight is essential to prevent evaluators from becoming too reliant on AI companies, which might compromise the rigor of their assessments. Some lawmakers have expressed support for such measures, with IVOs featuring in proposed legislation aimed at overseeing frontier AI risks.

As artificial intelligence continues to advance and attract enormous investment, the role of independent evaluators is likely to become increasingly important. How these organizations are funded, regulated, and integrated within AI companies will have significant implications for the safe development of AI technologies in the years ahead.