Building Agents Backwards from Evaluation

Executive Summary Evaluating a security agent begins with defining the work it should perform and the evidence required to consider that work complete and accurate. Domain experts establish reference answers and grading criteria. A successful approach must combine evaluations of complete runs alongside tests of individual competencies to both demonstrate whether a system is useful and help engineers identify what exactly to improve.