How AI Upscaling Improves Weather Forecasts

[@carolyncopeland2722]

Given whats happening in Auckland NZ now and the past couple of days this is highly relevant and timely

[@WojciechowskaAnna]

in case of Auckland the rainfall forecast like 1 to 2 weeks ahead would not help. There's a lot of discussion about their drain infrastructure and general urbanisation. This time nobody can't blame forecast, but more infrastructure preparation... and also crisis case preparation...

[@dr.zoidberg8666]

Y'all remember when billionaires were huffing on that copium? "Just move to New Zealand! Climate change can't get you if you're down there."

[@DeathToMockingBirds]

I hope we'll not lose the science and the ingenuity that went behind the models, as dependent on initial conditions as they are. Just as generating a torrent of A.I. art can stop people from going into Art, learning techniques, lighting, composition, perspective, etc., I wish we will keep improving the principles that could let us carry on if global supply chains are not able to manifacture the insanely complicated tech needed to sustain our current system (if for example, fossile fuels came to be costlier and costlier to extract, refine, process, transport, etc.). Let's not surrender everything to A.I., though it's a great tool to use while we can.

[@MeOnStuff]

Modern physics-based weather forecasts are also run on purpose-built supercomputers.

[@DeathToMockingBirds]

@MeOnStuff True. That's not my gripe though.

[@benjaminkindle1841]

Personally, I don't think using AI for art and science will make art and science less popular any more than the onset of recorded music made playing instruments less popular - giving people new tools to use has the potential to compound their interest, and make the field more accessible! Another analogy could be calculators. Someone may justified in worrying that people are worse at basic arithmetic, but I believe the possibilities that the tech unlocks are well worth it. But of course, it's good to have some skepticism and not blindly accept all technology. Your concerns are good points and valid things to keep an eye on!

[@Zappygunshot]

Currently, AI is very good at taking the average out of a huge pile of data, but it cannot innovate on that by itself. Because of that, there are still many aspects of human society and existence that cannot (yet) be replaced by artificial intelligence. It can, however, make those things easier by sorting through ridiculous amounts of data in a much shorter time and without getting depressed doing it, which makes it very useful indeed, but it won't replace everything just yet. If ever.

[@ShaoVideoProduction]

Look mom, Simon is a real youtuber, he got the NordVPN sponsorship!

[@xaviermaster1]

no a real KZbinr gets sponsored by "raid shadow legends!"

[@davidbryden7904]

No, a real youtuber suffers with investment scams here in the replies! 🤪

[@iantaakalla8180]

A real KZbinr takes one video to explain their burnout

[@Treebark1313]

6:35 what the network does is still a very big question mark. training the model is analogous to linear regression, but what those weights do is a much more complicated picture.

[@MRfuzzjr101]

This is such a good video. You explained complicated ideas in very digestible terms, and the interviews were awesome! Keep it up!

[@miguelguerreiro5280]

This is very similar to what I am researching: Species distribution models. Many of the terms are the same, but instead of weather forecast, we do species forecasts.

[@strawberrycheesecake5502]

It was cool to learn about a use for GAN that isn't just bypassing licensing and copyright laws. An area close to where I live got hit by desasterous rain and floods two years ago and while there where warnings, nobody knew how bad it would get. It would be great to get more accurate forecasts.

[@kieral4098]

Love videos like this where you dive into a specific paper. Super helpful for me as I try to pick a graduate project in atmospheric and oceanic sciences!

[@lucyharris2127]

Good luck with your graduate project!

[@likebot.]

It's funny how I knew all my life that it doesn't rain without clouds yet never cottoned onto the idea that a raincloud can be statistically microscopic on the macroscopy of a weather forecast. I'd never understand the brilliance of this neat trick if you didn't put that out there. Really happy that Nord is sponsoring your video. They're Tom Scott approved, a man miserly with his endorsements.

[@100c0c]

Why is it a neat trick?

[@GOATMENTATOR]

Dr Andrew and Dr Lucy are showing us an example of uppers and downers :D

[@Troyseph]

Hard to concentrate on one, and hard to stay interested in the other >

[@personenkenzahl]

thanks for another brilliant video and congrats on getting the sponsorship. thumbs up.

[@ShirleeKnott]

👍

[@gideonwilliams6307]

Thank you! I have loved Chaos Theory for years (brother is a math PhD, I'm an engineer) and I was hoping to see one.

[@elismart13]

as someone who lives in ireland this is somewhat concerning, but luckily it wasn't too bad where I lived

[@pokepress]

I’ve been working on AI-based upscaling a fair bit over the last year, so this is pretty interesting. Great to see another real-world use case.

[@snowboardkid202]

I’m glad I’m studying the PBL, I think we have a bit more work to do on our parameterizations there.. interesting video though, I hope a benefit of these “AI” models will be allowing for more hi-res nwp due to the freed up computational resources.

[@nikolai1714]

TV's predicts what pixels will most likely be next to the pixel it has next to it, in this way it needs much less data to show a picture. That is why some times you can see lags in the colors if the colors are very similar.

[@emmabird9745]

Hi Simon, I've read your book and its ok. It appealed to my "who did that and how" outlook. In this vid you appear to be telling us that when you come up with a new model its a good idea to see what result it would have predicted on past events with past data. Good sense really.

[@PremierCCGuyMMXVI]

I get so mad when people criticize meteorology when meteorologists and atmospheric scientists get short term weather wrong. Our atmosphere is extremely complex and it takes a ton of physics to figure it out. The good news is predictions are getting more accurate and I wish people had more confidence in science. Thank you for this, atmospheric science is extremely interesting to me.

[@DrAndrewSteele]

One thing that’s easier to predict than precipitation: every machine learning researcher watching this video reeling from the sick linear regression burn

[@Jaqaroo]

That was great Simon! Really interesting. I noticed about 5 minutes into the video that there was some plinky piano background music... umm....

[@etiennelamole9565]

5:41 Quick correction, neural network’s whole purpose is to solve NON linear problems, thus perform non linear calculations

[@elise3455]

A key limitation that should be highlighted with pure machine learning models over conceptual/theoretical models is that you can't easily interpret them and they tend to be difficult to control; meaning it's very difficult if you want to, say, see how a change in some parameters (e.g. due to climate change) might affect how the weather behaves in the future (Kaiser et al 2018). Neural nets can definitely be useful for pure operational forecasting, post processing, or for finding optimal coordinates that make your physics easy to model, interpret, and control however (Champion et al 2019).

[@jakubiskra523]

Can't you just run neural net with this changes and see result?

[@pokepress]

Honestly one of the places that would probably appreciate this the most is the national/state Department of Transportation. Having a better idea where precipitation will fall would massively help allocate resources more quickly and effectively.

[@narnigrin]

1:10 had me proper cracked up 😅 ... and with fond memories of Scotland. Where it actually snows sometimes! But not much on the coast, where I was. So ... rain, yeah.

[@SimonEliasen123]

Another reason we don't see widespread machine learning adaptation in all fields yet is the explainability of the model's decision-making. e.g. models far outperform human radiologists, but straight machine-learning models are not adapted in industry, due to a limited understanding of how exactly decisions are derived.

[@SimonEliasen123]

This is especially important in safety-critical applications.

[@someonespotatohmm9513]

@@SimonEliasen123 To add to that, even in non safety-critical applications it can be hard to work with. Because they fail in such , often weird, ways. So even when you succeed to, on average, get better results. The failures are often unpredictable and complete. Making mitigating them difficult, and much more important, because often sub optimal is much better then completely wrong.

[@douglasboyle6544]

It's funny when you got to the sponsor spot and the way you announced it, I was guessing it was either Raid Shadow Legends or Nord VPN

[@tadhgtwo]

Love the video Simon. One serious question, where did you get the jumper? Looks great.

[@hgriff10]

fabulous - the techs living in basement flats in New York City a few weeks ago would have appreciated the forecast as their flat went under water

[@thecountingthot7638]

I'm a bit skeptical these will continue to pan out beyond post-processing algos, I've heard that ML models represent chaos quite poorly (as you say, it's just fancy linear regression), so until that hurdle is overcome I don't see how we can possibly get accurate weather out of this...

[@thamiordragonheart8682]

I think that would be more of a limit on how far into the future they can go than their other capabilities. In the future, I could see near-term high-resolution models being made with machine learning and long-range models still relying on expert knowledge. There's also always the possibility of training convolutional auto-encoders for near-term high-resolution forecasting as a way of learning about the underlying structures to improve handcrafted models. Auto-encoders just turn out to be the best dimensional reduction technique we have and tend to be relatively understandable.

[@lucyharris2127]

@@thamiordragonheart8682 I'm not sure it made the final cut of the video, but Andrew did mention that we tried auto-encoders and we were much less successful than we were using GANs

[@lucyharris2127]

The PanguWeather (Huawei iirc) and GraphCast (DeepMind) papers both represent really strong steps forward in this direction

[@thecountingthot7638]

@@lucyharris2127 Cool I'll check them out! Thanks.

[@thamiordragonheart8682]

@@lucyharris2127 I was thinking of autoencoders more as an analysis technique for understanding processes better than as an effective stand-alone model, though it's at the edge of my data science knowledge and I think the most common autoencoder architectures aren't really meant to be used on images, which makes it harder. I also know just enough about GANs to realize that if you're ok with a bit of a black box there isn't anything much more precise than iterating a generator and classifier against each other.

[@yeetyeet7070]

I really hope that during my lifetime we will still always double-check the weather forecasts with our existing models. Machine learning can make a big poopoo when you least expect it, in a way traditional phyiscs simply can't.

[@jakubiskra523]

Are you basing your words on something specific or are you just expressing your opinion not based on facts?

[@yeetyeet7070]

@@jakubiskra523 elaborate

[@ofentsewenalesego822]

Hello Simon. This is actually quite a great video. Has there been some applications on Hydrology and water resources? And could it be used for predicting other climatic variables especially those with implications in disaster risk management?

[@lucyharris2127]

I think this is a really insightful question. I have now left Oxford university, but before I left I was working on applying this technique to improve hydrological models. The other members of the group are currently working on exactly what you've mentioned here. We all think it is really important that the science is applicable to the real world.

[@ofentsewenalesego822]

@@lucyharris2127 I'd honestly love to learn and try applying it in the country that I reside. Is it possible to be guided until I'm able to navigate through. I'm early career researcher with little experience, so that would be helpful. Please share links to some research articles too.

[@lucyharris2127]

@@ofentsewenalesego822 that sounds like a great project - you will have insights from living in that country to develop the model that no one else will. In terms of guidance, I'm still early in my career myself but I'm always happy to chat and help where I can!

[@ofentsewenalesego822]

@@lucyharris2127 great. Just sent you a connection request on another platform. We can chat from there.

[@yanononopon]

It's funny you're talking about super resolution as it's one of my research subjects atm. I'm doing super resolution of 3D Large Eddy Simulation data using a machine learning algorithm

[@kyle_weather]

This meteorologist appreciates this video!

[@thesilentone4024]

It snowed in las vegas last night and almost every mountain around las vegas is covered in snow.

[@_yonas]

Just for people who want to learn more about the state of current climate models: there is a podcast (Manifold) episode called 'Status and Future of Climate Modelling' with Tim Palmer.

[@toni4729]

I can remember some years ago when my dear old aunt visited us in Australia. It was early September and she heard the weather forecast. "It's not likely to rain before January" Her chin nearly hit the floor, she could believe her ears. It was true that year however.

[@harveytheparaglidingchaser7039]

Really interesting video. Enjoyed this a lot

[@thes7754]

lol congrats on the nord VPN sponsorship

[@WAMTAT]

Amazing video

[@PracticalScicomm]

Great video, love a bit of weather.

[@oskars.5477]

I've been a fan of yours since 2017, love the content

[@JS-pb6gb]

Please can you do a video on Judith curry, she is one of the most outspoken scientists who downplay the affect off climate change

[@kevinpils4716]

Awesome video and awesome topic. Maybe someone with more knowledge of climate modeling can answer me this: I would assume "classical" (i.e. non-ML) climate models also use the probabilistic method with ensembles - how many runs are there in such an ensemble?

[@markrichardson21]

There are always two factors in these cases: resolution and ensemble size. When you like to have a higher resolution which can give you better projections (not a must, but possible), then you have to reduce the ensemble size which gives you better opportunities to estimate the probabilities of your variables. What is the best option depends on your application. In climate models it has increased over time, from originally 3 per model in the mid 2000 to now 10s (because you also want to play with different scenarios). But there are also model simulations out there with multiple hundred ensemble members, also for the full complexity models (but usually in quite low resolution). For simple climate model you might be able to generate thousands or millions of members. The tricky question is how many ensembles should there be to estimate the probability curves alright and whether doing the simulations with different models (multi-model ensemble) offers you more insights than doing the projections with only one model. General number of ensembles nowadays: weather ~40 (some much more, some less), seasonal to decadal predictions are about 10-30 members in operational mode, climate projections are about 5-30 per model and in the order of 200 in multi-model-ensembles. Higher than that might exists in the research field, but are rarely used in operational settings (but might be one day when they proof successful).

[@kevinpils4716]

@@markrichardson21 Thank you!

[@gehwissen3975]

Thank you guys. That's how it looks like when YT Comments works best.

[@lucyharris2127]

Hi Kevin, the ECMWF weather model has 51 ensemble members (50 perturbed and 1 control) and a higher resolution deterministic run

[@mtawali13]

Hi Simon, who would be the best people to link up with to talk about AI and weather prediction??

[@ciaranmcinerney4162]

"It's really just linear regession" 😄 Yessir it is...but don't tell the funder

[@mahirdhali]

Sick thumbnail, man. Loved it.

[@REXilef]

How likely is it, that this video gets uploaded the day before I start my PhD in meteorology, where a large part of my work will be with post Processing 😅

[@benjaminfraeyman]

As a cyclist to work i hate how bad predicting rain is. Like i can dress for the cold... But rain is just annoying as heck. Dont wanna sit in wet clothes all day

[@AjSmit1]

0:01 you can't fool me with a touch of gaussian blur, i've seen the nordvpn logo too many times

[@petereldergill2942]

I've been wondering why we were so delayed in being able to buy Simon's book in Canada (I got a copy at Christmas) and why the soft cover is also delayed here. What's the reasoning? I'm sure there is one I just don't understand why we had to wait for it here. Cheers from Toronto

[@ShirleeKnott]

This comment is for feeding the algorithm monster. 🍪 Seems it likes snacking on comments, replies and likes to both. 👍 Why not spread some happiness to day and share a treat or two and watch the channel grow? 💖 🌩⛈

[@simonabunker]

Does this mean you are going to add machine learning to Claude? And have there been any updates to it?

[@SimonClark]

There have been big updates recently! Check out the twitch channel for a vod where I talk through the new structure. CLAUDE is a real priority for me this year, and we're not far off getting a dry dynamical core together (he says, cursing himself)

[@simonabunker]

@@SimonClark Twitch? I'm old! Any chance of cross posting some of the highlights to KZbin? I don't know what those words mean, but it sounds exciting!

[@ShirleeKnott]

@@simonabunker 👍👍👍👍👍

[@shayan_idk]

im curious whether or not these models are open-source or not, i have no idea how weather models operate tbh. what i mean to say is that i hope its available to access for the whole world for free and doesnt prevent struggling countries from utilizing it, like the vaccine rollout

[@nikolai1714]

I don't see how 8:42 answers your question. Is the answer that we will do?

[@powpowvideos]

Thank you Simon for a wonderful video! I'm strongly considering sending this video to a State Climatologist i've worked with🤣

[@TheCosmicGuy0111]

Wow

[@thejll]

Does NordVPN help me see BBCPlay from outside the UK?

[@frosty_brandon]

More meteorology please!!

[@tamasmatyasgal5629]

its unlucky than we move to black box models from white box ones...

[@Francis_UD]

6:29 butterfly effect isn’t it ? 😅😝

[@advanceringnewholder]

So, what next youtube sponsorship? Raid?

[@mauritsbol4806]

that outro

[@petersilie648]

Never understand how we are not utilising machine Learning to self improve the weather Forecast

[@wellurban]

People have been doing it for years: it just haven’t been obvious since weather forecasts don’t usually specify what models and other processes were involved, yet some sort of statistical/ML model will most likely have been used to generate things like spot temperatures from NWP models, usually in combination with human meteorological expertise. It’s only very recently that deep learning techniques and hardware have started to become competitive with NWP models based on actual physics, and I’m still sceptical about their ability to generalise to conditions that aren’t in their training set, but from the papers cited here it seems that they’re close to wider adoption.

[@KosmitPL]

Oh Simon, you're such a Nord... Nerd, I meant nerd.

[@Junoswoof]

The nord deal!

[@ShubhamBhushanCC]

Shhhh!!! Don't tell them it's just linear regression!!!

[@sitrakaforler8696]

Dam... this is fcking cool !

[@melissamybubbles6139]

Here is a cookie 🍪 for the algorithm. Math and computers were never good subjects for me, so I don't have anything to add.

[@concretesmoothie9202]

Comment for algorithm

[@NeoShameMan]

Linear aggression

[@Mermaidkilla]

not sure if he is actually happy to promote NordVPN

[@byrnemeister2008]

It’s witchcraft I tell yeah!!

[@caittastic]

ooooooh not a fan of not explicitly stating who you are sponsored by at the start of the video

[@tengkualiff]

Ofc he has a smurf army

[@SimonClark]

I don't actually! That was an eavy metal marine haha, check out my painting insta "fiftyshadesofspacewolfgrey" to see my armies

[@jonathandevries2828]

...i wont find out; i use a chrome extension called SponsorBlock...it blocks all those in show sponsors that adblock misses

[@travcollier]

Louis Fry Richardson served in the Friends Ambulance Unit in WWI alongside several other notable folks, including the sci-fi author Olaf Stapledon. en.m.wikipedia.org/wiki/Friends%27_Ambulance_Unit

[@KristieSaunder]

I really hope that during my lifetime we will still always double-check the weather forecasts with our existing models. Machine learning can make a big poopoo when you least expect it, in a way traditional phyiscs simply can't.