Hot on the heels of the announcement that its Claude 3.5 Sonnet large language model beat out other leading models, including GPT-4o and Llama-400B, AI startup Anthropic announced Monday that it plans to launch a new program to fund the development of independent, third-party benchmark tests against which to evaluate its upcoming models.
Per a blog post, the company is willing to pay third-party developers to create benchmarks that can “effectively measure advanced capabilities in AI models.”
“Our investment in these evaluations is intended to elevate the entire field of AI safety, providing valuable tools that benefit the whole ecosystem,” Anthropic wrote in a Monday blog post. “Developing high-quality, safety-relevant evaluations remains challenging, and the demand is outpacing the supply.”
The company wants submitted benchmarks to help measure the relative “safety level” of an AI based on a number of factors, including how well it resists attempts to coerce responses that might include cybersecurity; chemical, biological, radiological, and nuclear (CBRN); and misalignment, social manipulation, and other national security risks. Anthropic is also looking for benchmarks to help evaluate models’ advanced capabilities and is willing to fund the “development of tens of thousands of new evaluation questions and end-to-end tasks that would challenge even graduate students,” essentially testing a model’s ability to synthesize knowledge from a variety of sources, its ability to refuse cleverly worded malicious user requests, and its ability to respond in multiple languages.
Anthropic is looking for “sufficiently difficult,” high-volume tasks that can involve as many as “thousands” of testers across a diverse set of test formats that help the company inform its “realistic and safety-relevant” threat modeling efforts. Any interested developers are welcome to submit their proposals to the company, which plans to evaluate them on a rolling basis.
In 1950, Alan Turing asked whether machines can think. Rather than argue over definitions, he turned the question into a party game, where an interrogator trades typed messages with hidden players and tries to work out who is who. Since then, chatbots have gotten so good that they have already fooled humans in modern versions of the test.
So, four researchers from Paris Cité University asked the obvious follow-up question in a new study published in Computers in Human Behavior Reports: can AI at least catch its own kind?
Mozilla is giving Firefox’s Smart Window two new tricks. If you haven’t tried it yet, Smart Window is an optional AI mode in Firefox with a built-in chat that helps you get things done while you browse. With this update, it can fill out online forms using details from your tabs and keep an eye on websites for you.
Tired of typing the same details into every form?
The absolutely absurd idea of putting a touchscreen display on a mouse isn't exactly new. Turtle Beach already did it with its Command Series MC7, which features a 2.25-inch control screen. But apparently, even that wasn't ridiculous enough for Arctic Fox Technologies. Its new Super Mouse looks like someone took the chunky old Atari ST mouse, slapped a touchscreen and scroll wheel on it, and called it an upgrade.
The result is a rather unusual-looking mouse that is now on Kickstarter, starting at $199. And as ridiculous as it looks, some of its features are genuinely intriguing.