Hugging Face - Evaluating the Social Impact of Generative AI Systems in Systems and Society

PRINCETON AI ALIGNMENT AND SAFETY SEMINAR SERIES
Generative AI systems across modalities, ranging from text, image, audio, video, and multimodal, have broad social impacts, but there exists no official standard for means of evaluating those impacts and which impacts should be evaluated. We move toward a standard approach in evaluating a generative AI system for any modality, in two overarching categories: what is able to be evaluated in a base system that has no predetermined application and what is able to be evaluated in society. We describe specific social impact categories and how to approach and conduct evaluations in the base technical system, then in people and society. We are currently crafting an evaluation repository for the AI research community and examining methods for evaluating evaluations.