Session 4A: Development, Test, and Evaluation of Small-Scale Artificial Intelligence Models

MAJ David Niblick graduated from the United States Military Academy at West Point in 2010 with a BS in Electrical Engineering. He served in the Engineer Branch as a lieutenant and captain at Ft. Campbell, KY with the 101st Airborne Division (Air Assault) and at Schofield Barracks, HI with the 130th Engineer Brigade. He deployed twice to Afghanistan ('11-'12 and '13-'14) and to the Republic of Korea ('15-'16). After company command, he attended Purdue University and received an MS in Electrical and Computer Engineering with a thesis in computer vision and deep learning. He instructed in the Department of Electrical Engineering and Computer Science at USMA, after which he transferred from the Engineer Branch to Functional Area 49 (Operations Research and Systems Analysis). He currently serves as an Artificial Intelligence Evaluator with Army Test and Evaluation Command at Aberdeen Proving Ground, MD.
As data becomes more commoditized across all echelons of the DoD, developing Artificial Intelligence (AI ) solutions, even at small scales, offer incredible opportunity for advanced data analysis and processing. However, these solutions require intimate knowledge of the data in question, as well as robust Test and Evaluation (T&E) procedures to ensure performance and trustworthiness. This paper presents a case study and recommendations for developing and evaluating small-scale AI solutions. The model automates an acoustic trilateration system. First, the system accurately identifies the precise times of acoustic events across a variable number of sensors using a neural network. It then corresponds the events across the sensors through a heuristic matching process. Finally, using the correspondences and difference of times, the system triangulates a physical location. We find that even a relatively simple dataset requires extensive understanding at all phases of the process. Techniques like data augmentation and data synthesis, which must capture the unique attributes of the real data, were necessary both for improved performance, as well as robust T&E. The T&E metrics and pipeline required unique approaches to account for the AI solution, which lacked traceability and explainability. As leaders leverage the growing availability of AI tools to solve problems within their organizations, strong data analysis skills must remain at the core of process.
Session Materials: dataworks.test...