Artificial Intelligence (AI) 101 for evaluators

The seminar will open with an orientation to the current state of play in Artificial Intelligence (AI), provided by Kristy Hornby. Attendees will be talked through common language in the field, as well as Australian and global policy settings regarding this technology. Following this overview, Kristy will talk to the key opportunities and risks for evaluators in engaging with AI, both as part of their own service delivery and as part of their work to evaluate AI-enabled programs.
Gerard Atkinson will then explore the practical applications of large language models (LLMs) in the field. Discover how LLMs and machine learning approaches can be applied to qualitative analysis tasks like transcription, topic coding and thematic analysis, streamlining data processing and enabling deeper synthesis. Gerard will also explore how LLMs can aid in designing and making assessments against program logics and evaluation rubrics, enhancing decision-making processes. We will consider the capabilities and limitations of LLMs and understand best practices for leveraging this technology ethically and effectively in policy and program evaluation.
Session co-facilitator Kristy Hornby is an Associate Director at Grosvenor who established and still runs its Victorian program evaluation practice. She has completed over 310+ evaluation, review and strategy projects to date across more than a decade of practice. She is always keen to experiment with new technology and methods to explore the ‘what if’ and ‘how come’. Qualified with a BA, MBA and an MEvaluation (ongoing) she loves to make sense of complex contexts, connect with new people and have fun in her work.
Session co-facilitator Gerard Atkinson is a Director at ARTD Consultants who leads the Victorian practice and oversees the Learning and Development program for the firm. He has worked with big data and AI approaches for over 20 years, originally as a physicist then as a strategy consultant and evaluator. He has an MBA in Business Analytics focusing on the applications of machine learning to operational data. Gerard has previously presented at AES conferences on big data (2018) and on experimental tests of AI applications in evaluation (2023).
This event was organised by the AES Victoria Regional Network