Evaluating Forecasts - Hindcasting & Visual Evaluation

[@datascience1274]

Great video. I actually have a question. What do we do if we want to forcast into the future and still test the model? Do we forecast with a bigger horizon than 36 since the beginning (which would also include the future part), or maybe after testing the model the way you showed, we should create another training set with all the observations to then forecast into the future?

[@weecology]

Good question. Your second thought is the most standard approach. You would typically train on all of the observations, forecast into the future however many time steps you wanted to be able to forecast, and then evaluate the forecasts for those observations as they became available. That's what we do for our forecasts for the project that these lessons are based on: portal.naturecast.org/

[@datascience1274]

@@weecology ah ok I understand, thanks a lot. I was actually gonna go about this by testing the model first, the way you showed in the video, and only after figuring how the model performs in validation, I would create a training set with all the observations to forecast directly into the future. I may be completely wrong on this. Anyways, I’ve also heard of rolling window and I wonder if you may want to show the coding for that on this same exercise. It’d help immensely. My coding is still not quite there.

[@weecology]

@@datascience1274 Testing first, refitting, and then forecasting is definitely a good approach as well. In the short run that tells you how well you can expect your forecasts to perform on novel data. If you're doing on-going forecasting I do think that continual evaluation is useful because it helps you know if the underlying state of the system shifts and therefore your previous performance evaluation may no longer hold. For more information on forecast evaluation and example code you can check out our: * Paper: www.researchgate.net/profile/Juniper-Simonis/publication/352263925_Evaluating_probabilistic_ecological_forecasts/links/618aec4961f098772079c4d9/Evaluating-probabilistic-ecological-forecasts.pdf * Associated material (including a bunch of the associated code): zenodo.org/record/4663361

[@datascience1274]

@@weecology thank you so much, and I absolutely love this channel!

[@chittibabukapuganti9533]

Superb Lecture

[@weecology]

Thanks!