Day 1 Discussion

On the first day of the EAAMO metrics makerspace, participants engaged in a breakout-group exercise designed to explore what types of information government agencies or non-profit organizations would need when procuring AI systems. Participants were divided into small groups and asked to review an AI fact from 'City Detect Inc.' for their urban blight detection system. These fact sheets typically included technical descriptions of the system, such as its purpose, performance metrics, and general information about data practices.

The goal of the breakout discussion was to critically evaluate these fact sheets from the perspective of a government or NGO decision-maker responsible for determining whether to adopt or deploy the AI system. Participants were asked to identify information that was missing, unclear, or insufficient for making a well-informed procurement decision. In particular, groups were encouraged to consider what additional metrics, principles, or contextual information might help procurers assess not only the system's technical performance but also its broader operational, governance, and societal implications.

Across the breakout groups, participants noted that existing fact sheets tend to focus heavily on technical metrics, such as model accuracy or system performance, while offering limited insight into other critical aspects for procurement decisions. These include considerations such as the necessity of adopting the system, operational costs and maintenance requirements, accountability structures, and potential impacts on affected communities. The discussions, therefore, emphasized the need for a broader set of evaluation criteria that goes beyond traditional technical documentation.