ExpertCrowd, Toptal’s platform for training and improving AI models and systems, announced a collaboration with HUD, a Y Combinator-backed startup that provides tools for building AI training data through its DataVendor™ marketplace. Together, the teams released ExpertCrowd-SWE, a software engineering benchmark that evaluates the performance of frontier-grade LLMs and coding agents.
As AI systems take on increasingly complex work, measuring their capabilities requires tasks that reflect professional practice and grading criteria that capture what successful execution actually means. This collaboration brings together ExpertCrowd’s expert-led task development and HUD’s infrastructure for running complex RL environments.
ExpertCrowd and HUD co-developed ExpertCrowd-SWE, bringing together engineers from Toptal’s global network of highly skilled domain experts with HUD’s experience collaborating with frontier research teams. After Toptal’s experts authored tasks and grading rubrics, HUD’s infrastructure hosted task-run environments and verified the benchmarks against internal quality assurance pipelines. HUD and Toptal’s AI research teams, composed of more than a dozen PhDs and researchers with educational or professional experience at Harvard, EPFL, the University of Liège, DeepMind, Qwen, and ByteDance, reviewed and assessed the methodology and results to build one of the most robust software engineering benchmarks in the industry.
Also Read: Geographic Solutions Takes Workforce Innovation Further with Latest VOS Sapphire AI+ Release
“ExpertCrowd puts practitioners at the center of defining the work AI systems should perform and judging how well they perform it. By combining that specialized expertise with HUD’s testing infrastructure, we’re advancing what AI systems can achieve and setting a higher standard for real-world performance,” said Taso Du Val, founder and CEO of Toptal.
ExpertCrowd-SWE is the first release of an ongoing Toptal-HUD collaboration, with additional initiatives in development and further announcements coming soon. Building on this foundation, ExpertCrowd and HUD will bring more specialized human expertise into the tasks, environments, and evaluations that support the next generation of AI.
“We built HUD’s infrastructure to give research teams the tools to create realistic RL training environments, run evaluations across AI models, and inspect model performance. ExpertCrowd put that infrastructure to work, bringing the professional expertise to translate complex, real-world workflows into rigorous tasks, reference outputs, and evaluation frameworks that test the limits of today’s leading AI models,” said Jay Ram, founder and CEO of HUD.
SOURCE: PRNewswire
