Graeme Keith - How expert are your experts?

Whenever subject matter experts make quantitative assessments in a business process, we check those assessments after the fact and report back on precision and bias so that our SMEs can improve their future assessments. This is rather more difficult where the assessments are probabilities because the verification is necessarily statistical. Traditional methods can reveal optimism and pessimism in predictions, but they are not so effective at revealing cases where the impact of positive or negative evidence is over- or under-incorporated into an assessment and probabilities are either polarized - i.e. too extreme - or sitting on the fence - i.e .not extreme enough. They also demand substantial numbers of assessments.
I present an alternative approach to exposing systematic bias, which is based on a deeper look at how systematic bias effects probabilities. The tool produces a "bias footprint" which in a single figure shows not only the optimism or pessimism in a sequence of assessments, but also the tendency to polarize or fence-sit, together with a clear sense of the extent to which the data support those conclusions, which is dependent on both the number of assessments and the consistency of the bias.
I will demonstrate the method on Hubbard Decision Research's FrankenSME aggregation methodology, showing how the aggregation method substantially improves predictions by removing systematic biases.
Join this channel to get access to perks:
/ @riskacademy
ONLINE TRAINING: riskacademy.vonza.com/
OFFICIAL BLOG: www.riskacademy.blog
RAW2021 2021.riskawarenessweek.com/ Check out awesome risk management quantitative software www.archerirm.com/insight-ris... Enhance your company's risk management practices and improve the quality of your board's decision making with our advanced risk governance course, which covers everything from traditional risk management elements to advanced risk oversight and includes practical case studies and additional resources to strengthen your board's oversight. courses.dcroi.org/courses/ale...