AIMSEC Blog
Introducing MARS: Our Insights from Building an Open-Source AI Measurement Harness
The next frontier in AI isn't building more capable models, it's measuring them.
The pace of AI innovation has been unrelenting, and our approaches to testing and evaluation have struggled to keep up. As AI systems become more capable, autonomous, and embedded in high-consequence applications, measurement needs to become proactive. Progress depends not only on what models can do, but on our ability to rigorously evaluate their realized capabilities.
This challenge is at the heart of recent work conducted by the AI Measurement Science & Engineering Cooperative Research Center (AIMSEC), a joint initiative of 好色先生TV and NIST. Together, we are developing the Model-Agnostic Rigorous Sandbox (MARS), a testing and evaluation environment for AI system measurement. MARS brings together AIMSEC's expertise in measurement science with recent work in , including the recently released .
In this post, we introduce the architecture and design principles behind MARS, outline our perspective on the evolving AI measurement landscape, and tease the next phase of our work as we continue to challenge and refine our perspectives.
Introducing the Model-Agnostic Rigorous Sandbox (MARS)
To expand the state-of-the-art in automated language model testing, the research team developed MARS, the Model-Agnostic Rigorous Sandbox. The system enables AI practitioners to perform a detailed evaluation of a model or system under test.
System Design
MARS breaks evaluations down into three components: Corpus, Workflow, and Umpire. Each component is built using a modular, customizable class architecture allowing for adjustments, reviews, and restarts depending on the needs of the AI practitioner. Additionally, the components create distinct output files in an organized directory. Finally, the sandbox produces a log.txt file containing messages produced by each component, providing a record of the entire process.
The Corpus component creates, stores, and maintains a corpus of use cases. These use cases can represent either populations of user personas that might use an AI agent, various tasks the agent might be asked to perform, or a collection of scenarios that may apply to the AI agent. The generator allows an AI practitioner to either insert their own dataset/benchmark/personas or generate their own with their preferred generative model.
Prior work on population-scale agent modeling informed the design of our Corpus component. We were inspired by frameworks such as , , , , and . These frameworks demonstrate how LLM-driven agents can populate modular scalable simulations of social, economic, and digital environments. Although these systems emphasize orchestration and population-level experimentation, they are primarily designed as simulation platforms rather than as standardized, model-agnostic evaluation infrastructures. We extend their applicability by packaging tasks into portable, audit-ready test cases suitable for systematic verification across heterogeneous AI agent systems.
Our implementation of the Corpus component exemplifies how our perspectives on standardizing and contextualizing AI measurement can be done in practice. The customization and tracing features of the Corpus enable AI practitioners and subject matter experts to ensure that the test cases truly represent problems the AI system will need to handle.
The Workflow component is the harness for the system under test within the overall evaluation. It allows practitioners to link API keys or access locally stored models. By housing the workflow within the architecture, MARS supports adjusting system prompts and outlining the tools and capabilities that an agent can access. Existing harnesses shaped the design of the Workflow component. Systems such as and demonstrate the value of structured execution environments for evaluating agent behavior, particularly in interactive or tool-mediated settings such as terminal-based tasks. Similarly, provides a highly extensible framework to rapidly benchmark LLMs across diverse natural language processing (NLP) datasets with standardized loaders, model providers, and metrics. The is another widely utilized framework that inspires the evaluation sandbox's Workflow module. Many of these existing frameworks provide an experience comparable to the Workflow. The novelty of MARS lies in the direct integration of the Corpus and Umpire tooling in support of the central Workflow.
The Umpire component is responsible for expanding the assessment capabilities. Within this module, practitioners can define their own evaluation methods. A common approach we have already implemented is an , where an API calls to an LLM with a rubric prompt that reviews and assesses the outputs of the Workflow, considering what was created in the Corpus.
In developing this portion of the evaluation sandbox, we aim to provide a tool for researchers interested in exploring the validity and effectiveness of LLMs as judges and the evaluations performed by generative models. While the Umpire was built with LLM-as-judge in mind, the modular class structure can support the integration of non-generative evaluation scripts that review the workflow outputs. The Umpire dictates the final output of the system.
The resulting output file offers a complete evaluation connecting individually made corpus files to the outputs from the workflows and attaching the appropriate response evaluations. In this way, the Umpire embodies our perspective on measurement as the forcing function for the AI system. As the ultimate step of an evaluation, the Umpire requires AI practitioners to understand what they are measuring, how it is being measured, and how the results are recorded in a way that is meaningful for future decision making.
Code Overview
To request access to the MARS source code, please fill out the form here.
Within the GitHub repository, AI practitioners and researchers will find detailed documentation guiding them through the set-up of the sandbox. The documentation also includes guided examples for both quick starts and more thorough evaluations. The README provides a detailed overview of the core concepts of MARS, including how evaluation experiments measure the results of tasks performed using models or agents. An evaluation experiment consists of three main stages:
- Generate/retrieve a dataset of tasks.
- Complete the tasks using one or more models or agents.
- Assess the output from the tasks on user-defined metrics.
The user defines the behavior for each stage of the experiment programmatically through the implementation of small classes. An evaluation experiment is conducted by running the les_evaluate tool and providing it a JSON configuration file, in which the implementation details for each stage of the experiment are listed.
Insights into AI Measurement from Designing MARS
The first phase of this collaboration focused on designing and implementing a modular, model-agnostic evaluation sandbox. The platform enables practitioners to define experimental use cases, connect the models they want to evaluate, and analyze outcomes through measurement frameworks appropriate to their objectives. Building the sandbox has forced us to confront a broader question: What does it mean to measure an AI system?
Benchmarking remains the foundation of AI evaluation. At its core, benchmarking compares model outputs against selected tasks and reference responses to produce consistent measurements of capability. The field has rapidly expanded, with dozens of widely adopted benchmark suites, thousands of benchmark-focused research publications, and newer evaluation paradigms such as human preference arenas. While methodologies continue to evolve, benchmarking remains the primary mechanism through which AI systems are measured and compared.
The Established Principles of AI Measurement
Several principles underpin effective AI measurement through benchmarking.
Broad applicability across model implementations
Evaluation datasets and interfaces should be model agnostic so that differences in measured performance reflect differences in model capability rather than artifacts of data formatting or evaluation pipelines.
Construct validity
emphasized this term from psychology as core to effectively evaluating AI systems. An evaluation should measure the capability required for the intended production task rather than general intelligence in the abstract. A legal reasoning model, for example, should not be judged primarily on code generation performance. Meaningful evaluation depends on alignment between the construct being measured and the construct of interest.
As AI systems evolve beyond standalone language models into reasoning systems, tool-using agents, and multi-agent architectures, additional measurement principles become essential.
Comprehensive logging and tracing
Generative systems are inherently stochastic, so understanding outcomes requires visibility into the complete execution process rather than only the final output. Execution traces enable evaluators to identify error propagation, unnecessary computation, inefficient tool usage, and hidden failure modes.
Layered evaluation
Increasing system complexity requires evaluation at multiple levels rather than relying on a single benchmark score. At a minimum, this evaluation includes infrastructure-level measurements such as latency, resource consumption, and efficiency. These measures need to be reported alongside workflow-level measurements that assess whether the system successfully performs its intended task according to domain-specific rubrics. These complementary assessments provide a more complete understanding of system performance than either layer alone.
Throughout this project, we have adopted these established principles because they reflect both the current literature and long-standing measurement methodology. At the same time, our work has led us to identify several areas where AI evaluation must evolve beyond current practice.
Developing Our Perspectives on the Improvement and Expansion of AI Measurement
As AI systems become increasingly sophisticated, benchmark construction must mature alongside them. Collections of domain-specific questions are no longer sufficient to evaluate complex reasoning systems, nor are loosely specified tasks with expected outputs. Benchmark items should become standardized measurement instruments with clearly defined objectives, evaluation criteria, and reproducible construction methodologies that support meaningful comparison across models and over time.
Similarly, benchmark scores should not be interpreted as complete evaluation results. A reported score represents a single measurement rather than a comprehensive statement of capability. Robust conclusions require repeated measurements, statistical analysis, and transparent reporting of evaluation methodology. This approach includes documenting the number of independent model executions used to generate reported scores and accounting for evaluation procedures such as or , which can substantially influence reported performance.
More fundamentally, current AI evaluations remain largely descriptive rather than predictive. Most benchmarks measure how well a model performs on today's tasks, but provide limited evidence about how that model will perform when deployed in real operational environments. They resemble year-end examinations that assess retained knowledge more than standardized assessments designed to predict future performance. Bridging this gap will require evaluation methodologies that better capture operational behavior, long-term reliability, and adaptation to real-world conditions.
This challenge extends beyond evaluating the model itself. AI systems operate within organizational workflows that already possess measurable performance characteristics. Without establishing baseline measurements before deployment, it becomes difficult to determine whether introducing an AI system meaningfully improves outcomes. Although simulation offers one possible approach for estimating deployment performance, accurately reproducing real operational environments remains an open research challenge.
Evaluation must move beyond task completion alone. Modern AI agents should be evaluated as complete systems. While modularity is valuable for developing AI agents, the entire system is more than the sum of its parts. Measurements should integrate task success with observations of reasoning behavior, tool utilization, model selection, computational efficiency, resource consumption, and execution strategy. These complementary measurements provide substantially richer insight than task accuracy in isolation and better reflect the trade-offs encountered in production environments.
Finally, we view measurement as more than an assessment activity: it is a forcing function. Rigorous evaluation forces practitioners to define objectives, clarify requirements, and articulate what success means before systems are built. In this sense, measurement serves as a forcing function that improves system design as much as system assessment. This position can be reframed as navigating the overlap of the problem space with the solution space, moving from one to the other by connecting the applicable use cases. (See for more.) As AI measurement science matures, it has the potential to establish a common methodological foundation that guides problem definition, system development, and evaluation with the same level of rigor.
The remainder of this post demonstrates how these perspectives have emerged through the development of our evaluation sandbox and how they inform our approach to measuring increasingly complex AI systems.
Looking Ahead to Mapping Measurement Methods to AI Agent Requirements and Capabilities
To explore the robustness of the perspectives we developed during the initial project, the AIMSEC team will engage in projects that bring the expertise of measurement scientists into the practice of AI measurement and evaluation.
Upcoming work will further explore the capabilities and requirements of AI agents and map those to specific measurement methods. We hope to present a taxonomy of AI evaluation that can be operationalized in another prototype: an interactive tool to guide practitioners from AI agent design and definition towards rigorous testing and performance benchmarking.
We are truly aiming to see whether measurement can serve as the forcing function to help dictate exactly how an AI system is designed. We are also going to deploy the evaluation sandbox into CMU courses where students will utilize the system to perform their own measurements and evaluations on different AI systems.
Across the upcoming projects, we will continue to share our perspectives on AI measurement and revisit the underlying assumptions to ensure they are grounded in documented engineering approaches.
Acknowledgments
This project has been led by Thomas Serban von Davier and represents the work of a dedicated team of contributors from across AIMSEC. We would like to thank Eric Heim, Fernando Diaz, Lujo Bauer, Tyler Brooks, and Ramayya Krishnan for their contributions to this work.