The AI Alliance has released a new version of the MERA large language model benchmark

The Artificial Intelligence Alliance has unveiled a new version of the MERA benchmark: it includes a dynamic leaderboard, an updated measurement codebase, a more sophisticated prompt system, improved datasets, API support, and measurements for dozens of new models, including those developed by OpenAI.

MERA (Multimodal Evaluation for Russian -language Architectures) is the largest independent open benchmark for evaluating foundational Russian-language models, jointly developed on the AI Alliance platform by industry researchers—teams from Sber and MTS AI—as well as academic partners Skoltech AI and the National Research University Higher School of Economics (HSE).

The updated version of the benchmark includes 15 core tasks, which form the basis of the ranking, and 8 publicly available datasets.

Since the release of the benchmark’s first version, it has been used by dozens of model developers, who have submitted over 1,000 entries. The improvements to MERA were made possible by user comments and feedback from members of the NLP community. Future developments for MERA will include tasks for evaluating image, audio, and video recognition.

You can read more about all the updates in an article by the project team.