Session4B:Assessing the Calibration and Performance of Attention-based Spatiotemporal Neural Network

Dr. Nathan Gaw is an Assistant Professor of Data Science in the Department of Operational Sciences at Air Force Institute of Technology, Wright-Patterson AFB, Ohio, USA. His research develops new statistical machine learning algorithms to optimally fuse high-dimensional, heterogeneous, multi-modality data sources to support decision making in military, healthcare and remote sensing. He received his B.S.E. and M.S. in biomedical engineering and a Ph.D. in industrial engineering from Arizona State University (ASU), Tempe, AZ, USA, in 2013, 2014, and 2019, respectively. Dr. Gaw was a Postdoctoral Research Fellow at the ASU-Mayo Clinic Center for Innovative Imaging (AMCII), Tempe, AZ, USA, from 2019-2020, and a Postdoctoral Research Fellow in the School of Industrial and Systems Engineering (ISyE) at Georgia Institute of Technology, Atlanta, GA, USA, from 2020-2021. He is also chair of the INFORMS Data Mining Society, and a member of IISE and IEEE. For additional information, please visit www.nathanbgaw.com.
In the last decade, deep learning models have proven capable of learning complex spatiotemporal relations and producing highly accurate short-term forecasts, known as nowcasts. Various models have been proposed to forecast precipitation associated with storm events hours before they happen. More recently, neural networks have been developed to produce accurate lightning nowcasts, using various types of satellite imagery, past lightning data, and other weather parameters as inputs to their model. Furthermore, the inclusion of attention mechanisms into these spatiotemporal weather prediction models has shown increases in the model’s predictive capabilities.
However, the calibration of these models and other spatiotemporal neural networks is rarely discussed. In general, model calibration addresses how reliable model predictions are, and models are typically calibrated after the model training process using scaling and regression techniques. Recent research suggests that neural networks are poorly calibrated despite being highly accurate, which brings into question how accurate the models are.
This research develops attention-based and non-attention-based deep-learning neural networks that uniquely incorporate reliability measures into the model tuning and training process to investigate the performance and calibration of spatiotemporal deep-learning models. All of the models developed in this research prove capable of producing lightning occurrence nowcasts using common remotely sensed weather modalities, such as radar and satellite imagery. Initial results suggest that the inclusion of attention mechanisms into the model architecture improves the model’s accuracy and predictive capabilities while improving the model’s calibration and reliability.
Session Materials: dataworks.test...