Breaking the Black Box: A Case Study in Red-Teaming a Government Education AI

The “black box” assessment represents the most authentic test of an AI system’s security. Unlike white-box testing, where the system prompt and architecture are known, black box red-teaming forces the attacker to rely solely on the system’s outputs to map its internal defense mechanisms. This case study details a recent engagement with “[EduBot]” (pseudonym), a stateless AI assistant deployed by a Government office to answer resident questions regarding education.