Anthropic CEO Dario Amodei’s proposal for independent evaluators to receive broad access inside leading AI companies is drawing scrutiny over the longstanding personnel, funding and institutional connections between Anthropic and the nonprofit METR.
By yourNEWS Media Newsroom
Anthropic CEO Dario Amodei’s push for independent organizations to scrutinize the world’s most advanced artificial intelligence laboratories is facing questions over the independence of one organization positioned to perform that work.
Amodei proposed giving outside evaluators “employee-like access” to leading AI developers so they could examine company safety practices from inside the laboratories, an approach he outlined Saturday in an essay titled “We Must Pace the Frontier.”
Attention has since turned to the Model Evaluation and Threat Research organization, or METR, and a network of professional, financial and personal relationships connecting the nonprofit with Anthropic and people associated with the Effective Altruism movement.
A New York Post examination detailed those connections after Amodei promoted greater use of outside evaluators as part of his proposed framework for slowing the development of increasingly capable AI systems.
METR, pronounced “meter,” originated as ARC Evals within the Alignment Research Center before becoming an independent nonprofit and adopting its current name in 2023.
The Alignment Research Center was founded by AI researcher Paul Christiano, who has been associated with Effective Altruism and previously worked with Amodei at OpenAI.
Christiano and Amodei were also housemates and research collaborators during the 2010s. Christiano later became one of the original five trustees of Anthropic’s Long-Term Benefit Trust, a governance body established by the AI company.
The network extends into METR itself.
METR founder and CEO Beth Barnes previously worked under Christiano and participated in Effective Altruism activities while attending college.
Ajeya Cotra, who works at METR, is married to Christiano and has been an organizer in the Effective Altruism community.
METR Chief Scientist Hjalmar Wijk and staff members Megan Kinniment and Pip Arnott also emerged from a research community associated with the University of Oxford that has had substantial connections to Effective Altruism, according to the Post.
Wijk describes his work on LinkedIn as “trying to reduce worst-case risks from future models.”
Kinniment previously worked at the Center on Long-Term Risk in London, where her LinkedIn profile described her work as examining how “malevolent actors could increase the likelihood of poor AI outcomes.”
Jasmine Dhaliwal is listed as a policy staff member at METR.
Those personnel connections have become part of a broader debate over whether an organization with close historical ties to AI safety researchers, Anthropic personnel and Anthropic investors can function as a sufficiently independent evaluator if companies adopt Amodei’s proposed oversight model.
The connections extend beyond METR’s staff.
Amodei has pledged to donate 80% of his personal wealth. His sister and Anthropic co-founder Daniela Amodei is married to Holden Karnofsky, who founded two organizations associated with Effective Altruism.
Former cryptocurrency executive Sam Bankman-Fried, who became one of the movement’s best-known financial supporters before his conviction on fraud charges and imprisonment, was also among Anthropic’s early investors.
METR and organizations connected to it have received financial support from donors associated with Effective Altruism, including people who have also invested in Anthropic.
Those include Facebook co-founder and Democratic donor Dustin Moskovitz and Skype co-founder Jaan Tallinn.
METR announced in August that it had raised $71 million in new outside funding during the previous six months.
Tallinn’s Survival and Flourishing Fund has earmarked as much as $752,000 for METR since its creation, according to records cited by the Post. That total consists of $324,000 in general funding and a matching commitment worth as much as $428,000.
Moskovitz’s Coefficient Giving, formerly known as Open Philanthropy, provided $1.515 million to the Alignment Research Center in 2022, before METR became an independent organization under its current name.
METR disputes suggestions that those relationships compromise its independence.
“We don’t think any small group should have authority over what happens with AI, and our goal is to get information from inside these companies into the public domain – external investigators can help bring more information to light,” a METR spokesperson told the Post by phone.
The organization said it does not receive payment from AI laboratories for its evaluation work and does not accept donations from executives or employees working at AI companies.
METR also said Moskovitz’s 2022 contribution to the Alignment Research Center was kept separate from its operations and was not used to fund METR.
“Our funders have no say in the projects we work on and we do not accept any funding from AI companies or their employees,” the spokesperson said.
Critics argue that formal funding restrictions do not resolve concerns created by the overlapping relationships among researchers, donors, investors and organizations.
A source described by the Post as familiar with White House thinking characterized the relationship Tuesday by saying, “METR is essentially Anthropic in another form.”
Perry Metzger, chairman of the Alliance for the Future, questioned whether METR could credibly be treated as independent from the companies and investors connected to its professional network.
“Saying you want to be audited by the ‘independent’ people at METR is a complete joke. It’s people who are funded by your buddies. It’s completely ridiculous,” Metzger said.
David Sacks also challenged the organization’s independence following Amodei’s essay.
“Stop pretending METR is independent when it is intertwined with Anthropic’s investors and staff,” Sacks wrote.
Metzger compared the issue to financial institutions selecting auditors with extensive relationships to their own personnel and investors.
“If you went out tomorrow and you’re a big bank and you had your audits done by an organization consisting entirely of your own former employees, which was funded by the same investors that are investing in you, do you know what the feds would do with you? It would be positively medieval,” he said.
Amodei’s proposal comes amid a broader dispute within the technology industry over whether frontier AI companies should voluntarily slow increases in model capability while stronger safety systems are developed.
His suggested use of embedded independent evaluators would give outside researchers considerably greater access to internal systems than conventional public testing permits, potentially allowing evaluators to examine development practices and risks that cannot be assessed from commercially released products alone.
That level of access makes the question of evaluator independence particularly significant.
METR’s position is that its organizational safeguards, refusal to accept money from AI companies or their employees and commitment to publishing information provide the necessary independence.
Its critics contend that the personal, professional and financial networks surrounding METR and Anthropic warrant greater examination before an organization with those connections is treated as an outside check on the companies developing frontier AI.
The controversy therefore centers less on whether advanced AI should be independently evaluated than on who should perform that evaluation and what standards should determine whether an organization is sufficiently independent.
Additional details about the relationships among Amodei, Anthropic, METR and Effective Altruism figures were reported by the New York Post, which also published the full account of the connections and responses.
Original article
Comments