Toloka AI

18/11/2025

For over a decade, Toloka has powered the world's most advanced AI models with high-quality data, combining our core technology with a global network of human experts.

Today, we're making this power accessible to everyone.

Introducing Tendem, the industry's first hybrid AI + Human agent.

Tendem is built for busy professionals who need to delegate tasks and trust the results. It’s not just AI—it's AI combined with expert human-in-the-loop technology to deliver high-quality, verified results.

Delegate your tedious and time-consuming work and get back accurate, ready-to-use results in hours.

No hallucinations. No need to double-check. ✅

This is the reliability of enterprise-grade AI, now accessible to everyone.

Tendem is now open for early access. Join the waitlist: tendem.ai

15/04/2025

We’re excited to share our new multimodal benchmark in Arabic, developed in partnership with . The JEEM evaluation dataset tests VLMs on image captioning and visual question answering skills in MSA (Modern Standard Arabic) and 4 Arabic dialects.

What makes this dataset special?
✔️ Authentic regional nuances and cultural differences captured by annotators in Jordan, Egypt, UAE, and Morocco speaking regional varieties of Arabic. None of the data was translated from another language.
✔️ 2196 regional images across 13 categories of everyday scenes, from food and transport to education and technology.
✔️ This dataset is the first of its kind for testing VLM performance in regional Arabic dialects.

Mohamed Bin Zayed University of Artificial Intelligence in Abu Dhabi is one of the top global universities specializing in AI, CV, ML, and NLP.

We tested 6 top models with JEEM—see the results and insights here: https://bit.ly/3EnqIju

25/03/2025

What can we learn from current research on multilingual models? Join us for an in-depth webinar exploring the latest advancements in multilingual AI systems. https://bit.ly/4c2L8KY

Industry experts will discuss:
✔️ Approaches to address the scarcity of multilingual datasets in post-training
✔️ How sample-efficient learning techniques overcome data scarcity for multilingual NLU
✔️ How translation impacts fine-tuning and evaluation
✔️ Challenges of evaluating generative models without human references in a multilingual setting
✔️ Why VLMs struggle with cultural and linguistic nuances

Whether you're an AI enthusiast or a practitioner, this session will shed light on how multilingual models are transforming AI development.

Register now: https://bit.ly/4c2L8KY

23/01/2025

📣 We’re looking for graduate students and researchers passionate about data-centric and human-centric AI to join the Toloka Research Fellowship Program, a 3–6 month paid opportunity.
Share with your network or submit your application here: https://bit.ly/4av5yeM

What’s in it for you?
✅ Hands-on experience: Work on AI projects with experts.
✅ Funding: Get paid for your services.
✅ Impact: Contribute to impactful publications and open-source projects.

How does it work?
🔹 Duration: 3–6 months.
🔹 Projected workload: From 20 hours/week and up.
🔹 Flexible location: Participate remotely.

You’ll collaborate on innovative research areas, including:
- Developing evaluation metrics for complex tasks.
- Enhancing synthetic data generation with human feedback.
- Improving data quality and its impact on model performance.
- Advancing AI agents' capabilities and collaboration with human experts.
- Tackling safety, bias detection, and mitigation in AI models.

Does this opportunity align with your expertise and interests?
If you're self-motivated, proficient in Python, and passionate about AI, this could be a great match for you.

The Fellowship Research Program at Toloka invites students and researchers to advance data-centric and human-centric AI research. Over a 3 to 6-month period, participants will work on a project under expert mentorship.

09/12/2024

It’s here — the AI Safety milestone we’ve all been waiting for! 🤩 MLCommons released AILuminate, the first AI safety benchmark with broad support from industry and academia.

The AILuminate benchmark produces “safety grades” for the largest LLM models, giving companies and consumers direct access to model safety comparisons for immediate decision-making. The safety grades are based on model responses to prompts across 12 categories of hazards, including crime, child sexual exploitation, hate, and self-harm.

This project is unique in its collaborative, cross-sector approach. The v1.0 AILuminate benchmark tests the most comprehensive set of English language hazards in the world, with additional language coverage to follow.

Toloka experts contributed complex prompts to make this benchmark possible. We are proud to have played a part in bringing this important benchmark to life. Congrats to the MLCommons team and we look forward to future collaborations!

https://mlcommons.org/ailuminate/

The v1.0 AILuminate benchmark from the MLCommons AI Risk & Reliability working group is the first AI risk assessment benchmark developed with broad involvement from leading AI companies, academia, and civil society. It is a significant step towards standard risk assessment for “chat” style syste...

05/12/2024

Introducing U-MATH: A comprehensive LLM benchmark for university-level math
✨We compared small, large, and proprietary models — and the leader is not GPT-4o. ✨

Toloka and Gradarius have joined forces to build the U-MATH benchmark, challenging LLMs with math problems from curriculum at top US universities. Our testing reveals that even top-tier models fall short in practical math skills, but the leader in university math is Gemini 1.5 Pro.

The U-MATH benchmark has 1,100 math problems across 6 university-level math subjects, with 20% of the problems requiring visual comprehension — the only benchmark of its kind in the industry.

For insights into math performance and details on how U-MATH was collected, read our post: https://bit.ly/4f4Cg7g

Want to check your model’s performance? Download from Hugging Face https://huggingface.co/toloka/u-math
Or connect with our team to discuss in-depth evaluation and fine-tuning with our math dataset.

04/12/2024

Are you ready for 2025? Check out Forbes’ top 20 technology challenges to prepare for in the upcoming year. Challenge #2 features our CEO, Olga Megorskaya, highlighting one of the most pressing issues in the AI space: 𝙚𝙣𝙨𝙪𝙧𝙞𝙣𝙜 𝙩𝙝𝙚 𝙨𝙖𝙛𝙚𝙩𝙮 𝙖𝙣𝙙 𝙧𝙚𝙡𝙞𝙖𝙗𝙞𝙡𝙞𝙩𝙮 𝙤𝙛 𝙖𝙙𝙫𝙖𝙣𝙘𝙚𝙙 𝘼𝙄 𝙨𝙮𝙨𝙩𝙚𝙢𝙨.

Olga predicts: “As AI advances in reasoning, autonomy and domain expertise, 2025 will see it tackling complex tasks and driving automation. The challenge will be ensuring safety and reliability as systems grow more complex. My team is dedicated to providing large language model developers with top-quality data to support the safe, responsible and impactful advancement of these technologies.”

Read the full article for 19 more insights into how leaders are shaping the future of tech. https://www.forbes.com/councils/forbestechcouncil/2024/12/03/tech-in-2025-industry-leaders-detail-their-top-challenges/

As they look to 2025, tech leaders see a continued focus on the optimal and ethical use of artificial intelligence, but AI is far from the only topic of conversation.

04/12/2024

Toloka is happy to sponsor NeurIPS this year. Meet the team in Vancouver, Dec 10–15. Let us know if you’re attending!

20/11/2024

Is your AI sensitive to emotion in voice interactions?

At Toloka, we're paving the way for more natural and engaging AI interactions by exploring emotion recognition in voice interactions. Our recent audio annotation project provided high-quality, scalable, and affordable emotional data. By leveraging our global crowd of annotators, we ensure diverse cultural perspectives and exceptional scalability—all without sacrificing quality.

What Toloka offers:
🔹 High-quality emotional annotation: Annotate recordings with detailed emotions and voice characteristics.
🔹 Scalable and affordable solutions: Use our global crowd of diverse annotators to get fast and cost-effective results.
🔹 Cultural diversity: Capture nuances from diverse cultural backgrounds, enriching your model’s emotional understanding.

Don’t let your chatbot or AI agent sound like a robot. We can push AI towards true empathy, from recognizing happiness and excitement to understanding compassion and awe. 🤖💡
Curious about how we make it happen? Dive into our latest blog post to learn more! https://bit.ly/4hPMAmi