Training a Neural Network to operate drones using Genetic Algorithm

[@Alayric]

Good idea, and I like your smoke!

[@PezzzasWork]

Thanks! I think smoke is where I spent the most time :D

[@mendelovitch]

@@PezzzasWork Why do we get hung up on those small sidequests?

[@I_SEE_RED]

@@mendelovitch it’s an easy way to procrastinate the main problem

[@anujanshu2917]

How or where you stimulate this in unity or special software

[@katzen3314]

I love how they seem to move so organically even though it seems like a relatively simple model. I bet there's some really interesting optimisation problems and extra restrictions you could throw at this.

[@katzen3314]

Also thanks for uploading the demo and source code, very fun to play around with!

[@NetHacker100]

I think that the need to center themselves perfectly with the sphere is what makes them not become speed machines. Because when they reach the target they always gotta somehow "dock". And that requires their inertia to be 0 when they reach that point so they have to slow down. If somehow this was changed by making the drones to just need to touch the point at any part and maybe making the orb bigger I would certainly expect that there would be more speedy manoeuvres to just arrive at the target and pass through it. Perhaps even in an elliptical patrolling. Would be certainly interesting to see.

[@00swinter21]

im currently working on the same thing but with more inputs; I will try ours too;

[@eliaswenner7847]

@@00swinter21 Don't forget to post the result on your KZbin channel !

[@feffy380]

Exactly my thoughts. It looks like the target requires pixel perfect precision to count as a success. Careful approach is the only way when the targeting criteria are so unnecessarily strict.

[@christopheroldfield1066]

​@@Wock__ I believe you are right. On one of their videos, there is an actual clock face that counts down on top of the target, like a circular loading bar.

[@UnitSe7en]

The goal is to dock, not to touch the target. Changing the goals to achieve a better outcome does not mean that your model improved. Making them just have to touch the target so they could go really fast does not mean that they are suddenly better. Your thinking is flawed.

[@blmppes9876]

5:28, gen 900: Ok, you guys are too good and I'm tired now. Bye!!!

[@NanoCubeOG]

true

[@tuna3977]

"I have to go now, my planet needs me"

[@Furebel]

I'd love to see a game where your enemies are all neural network trained AI, and the higher the difficulty, the more trained AI variant you will have to face

[@ChunkyWaterisReal]

Give it 10 years

[@kirtil5177]

imagine if the AI is being trained while you play. The better you play the less hard the ai is, but if you slow down the difficulty increases

[@marfitrblx]

@@ChunkyWaterisReal it's already possible now lol

[@ChunkyWaterisReal]

@@marfitrblx AI has been shit since the 64 hush yourself.

[@keyboardegg931]

Or even the player being an AI - I can totally see a 2D game with your cursor being the target point, and the more you play/the more enemies you defeat/etc. the smarter your character gets

[@phantuananh2163]

This channel is a gem

[@dan_obie]

Would be really interesting to add fuel consumption to the mix and watch them optimize their fuel economy

[@dazcarrr]

and give them more fuel for every target they reach as more reward for doing that

[@markoftheland3115]

Very cool stuff, well done! Now make them go through an obstacle course 😁

[@PezzzasWork]

I am working on it ;)

[@marc_frank]

a combination of the ants finding the optimal path and then the drones following that? :)

[@Vofr]

@@PezzzasWork where's the video 🗿

[@GG64du02]

I wrote my autopilot cargo drone for space engineers and still i am impressed by the work

[@raffimolero64]

love this channel. what separates this guy from others is his consistent ability to make his sims look cool.

[@nandorboda8049]

These vids are awsome and very inspiring! It's amazing how a neuron network can adapt for a specific task!!!

[@osman4172]

Great work. I think many people would appreciate seeing background of the work.

[@reaperbs7105]

Props to Gen 300 and 400 for beings underdogs and yet surviving for so long

[@youssefelshahawy8080]

This is one of the coolest implementations i've seen. Nj!

[@xDeltaF1x]

That end result with the live-tracking is so good! I wonder how viable it is to train simple neural networks like this for game enemy AI

[@originalbillyspeed1]

Depends on the game, but on games with a clear goal, it is fairly trivial and will quickly surpass humans.

[@AB-bp9fi]

@@originalbillyspeed1 i guess for different difficulty levels game designer can use agents (enemies) from different generations, for example "easy" = generation 400, medium = generation 500, hard=generation 1000.

[@commenturthegreat2915]

​@@AB-bp9fi I don't think that would work for most applications. When you want to make enemy AI easier or harder, you always have to think of it in relation to the player - for instance, in a stealth game, harder AI could mean it detects you faster - which pushes the player to improve and be more careful. That won't happen if you just made the enemies drunk (which is basically what would happen if you pick bad neural networks) - it just adds randomness which can be annoying to deal with. Maybe it could work better in things like racing games though.

[@williambarnes5023]

I'm now imagining a game cloud coordinating through the internet. The AI uses background CPU while the game is running to simulate and evolve against itself, spits its best results against the player to see how they fare, and takes those results as more data to go back to the cloud with to keep working. The bots will start laughably bad at first, but they'll learn how players act, and make players devise new tactics... You might even get good teammate and wingman AI out of it if you put those AIs on the player's side.

[@MrStealthWarrior]

@@commenturthegreat2915 What about training AI to match the certain level of intelligence? Like if AI detects a player too fast, then it failed the test.

[@the0neskater]

This is one of the coolest projects I've ever seen. Would be awesome to extend to add walls and an environment! Great work.

[@SongStudios]

Dude I love it when they get sooo roofless! So fun to watch!

[@YellingSilently]

The end of play lineup was a cute touch. Nice work!

[@noiky6164]

OMG This is so cool, your video actually change my attitude toward neural network from hate to love.

[@s.m8766]

very nice! I'd love to see the same tests, but with added random disturbances like wind gusts from the side, to see how well they can adapt to that!

[@motbus3]

It would be great to have a remake of this one

[@PezzzasWork]

I am actually working on a follow up :)

[@motbus3]

@@PezzzasWork noice! I will certainly watch it

[@Reverend-dd2lq]

Getting some strong Factorio vibes at 4:57

[@Phiwipuss]

5:56 The drone in the left down corner synchronized with the beat in the music. Perfection.

[@jeremybertoncini6935]

Hello, very interesting work ! Did you think about testing scenarios with obstacles ? It would be also interesting to compare the last trajectories and controls with optimal control algorithms solutions. Cheers.

[@thorbenpultke1350]

Impressive Stuff! Had my hands on GAs too for my Bachelor Thesis but with a 6 DOF 3D acting robotic arm. Kinda addicting when you dive deep down in ML :)!

[@Lengthy_Lemon]

You are amazing. Thank you for sharing your fascinating work.

[@skoll6007]

1:58 that faint Vader "noooooo" put me on the floor for some reason

[@xandon24]

7:25 the music moves to your left and right ear as the drone in the top right moves it's power to it's left and right thruster.

[@jayshukla6724]

7:24 Loved how the Gen-400's legs synced with the music... Btw, How do we decide the size of the hidden layers? Is there some rule or formula for the best size approximation?

[@clairedcaptions]

I’d love to see an algorithm where you simply add the direction from the current target point to the next, and see if it, with only that information learns to steer ahead of time.

[@kovacsattila8993]

I tryed the mouse controlled vesion what you uploaded on github. And i saw that it's easy to confuse the A.I. in that way to lose controll and fall off the map. I think if you crate a small Trainer A.I. for the target control what best interest to confuse the drone and make it fall off the map, it can train the drone to not fall off no matter how the target moves.

[@PezzzasWork]

Yes I did a more robust version that I can upload as well

[@KiemPlant]

Other than giving us almost 20 seconds to read 6 words at 4:39 this was very enjoyable to watch :p

[@ThePizzaGoblin]

I like how it learned to turn off its thrusters to arrest upward motion and to speed up descent.

[@raphulali8937]

i have no idea about how you did it ..but it seems like something fun to learn

[@PezzzasWork]

Machine learning is extremely fun and addictive :)

[@00swinter21]

@@PezzzasWork can confirm

[@dromeosaur1031]

Thanks for the video! It's really inspiring.

[@darkfrei2]

Very nice! Please make more such content, with neural network and drones! :)

[@DeepRafterGaming]

I suggest to add more then just time to the fitness equation. Fe. Energy use, pressicion, stability of flight and adding external forces like wind. with these factors the movement would become smooth like silk. But nice project anyway

[@PezzzasWork]

The current fitness evaluation takes speed, precision and stability into account. I tried to add wind after the training was done and it worked quite well :)

[@DeepRafterGaming]

@@PezzzasWorkahh I see, but the angled engines while hovering still seem very inefficient to me :)

[@PezzzasWork]

@@DeepRafterGaming Yes you're right and I don't really know why they do this. My assumption is that it is a way to reduce power, as if they couldn't go very close to 0 power so it is easier to add angle. This could be avoided by taking energy into account in the fitness function. If I increase gravity, they don't angle the thrusters to gain more power. Here is a windows demo with a config file if you want to try it out github.com/johnBuffer/AutoDrone/releases/tag/v1

[@DeepRafterGaming]

@@PezzzasWork Yeah it's hard to tell why. The fitness function is the most complicated part of any neural network. I would allway advocate for implementing energy use in any neural network because, if you think about it, if the network doesn't have to bother with the used energy it will always come up with unnecessary movement patterns that look jenky. It's more important than speed I'd say ^^

[@jetison333]

@@PezzzasWork if you watch the way generation 5500 flys sideways, it ends up with one thruster almost horizontal and the other almost vertical. They might like tilting the thrusters because its kind of an inbetween state between flying right and left. So when it gets a new target, it can start flying towards the target sooner. That might be part of the reason anyway.

[@ziggyzoggin]

I'm kind of upset that you didn't publish the thing at the end on itch. Its so satisfying to see the drone follow your mouse and I want to play around with it. Great video!

[@PezzzasWork]

You can download the control demo here github.com/johnBuffer/AutoDrone/releases/tag/v1

[@ziggyzoggin]

@@PezzzasWork thank you! :)

[@flight_risk]

somewhat smaller models and policy gradient following might have increased convergence speed. MLPs are differentiable, so you could just backpropagate through them, sampling distance to the target at every frame and accumulating rewards over the trajectory for an unbiased estimate of a policy’s optimality. you could even use a decay term to incentivize the robots to move faster by downweighting rewards acquired later in the trajectory: distance to the target is ideally the same in the end, but according to the gradient of this reward function, faster would be better. the only thing left would be running the simulations in parallel or faster than real-time by simply not fully rendering the state of the environment at every training step

[@EsbenEugen]

The target tracking would be cool for a background

[@ferociousfeind8538]

You could turn the target tracking into a game, try to get the drone to lose control as quickly as possible, using your mouse as the target! Or, just play with it. It looks fun.

[@DogsRNice]

Give the target to another network that tries to learn how to get the drones to crash while the drones learn how not to crash

[@angelo.strand]

@@DogsRNice oh no the ai wars

[@dromedda6810]

gen 400 is like that one kid in your class that cant stand still when waiting in a queue

[@manuelpena3988]

xDDD the "ok..." almost kills me

[@Zygorg]

The memes are fun on this vid

[@darkfrei2]

Which parameters give the drone positive or negative feedback? Is flying time a positive or a negative parameter? An acceleration to the target?

[@alessandrodamato5059]

give a consolation prize to generation 300! It deserves it all Have you ever tried using a neural network on a hardware platform?

[@argmentum22]

Adding a fuel allowance would probably add a more varied result, possibly get those burn hard drones quicker. Also maybe increase your destination bubble a fraction ? This increase the prize rate and hopefully the drones would tighten up the homecoming naturally like the ants do for food routes

[@908animates]

Imagine spending hours and hours trying to get to something and then when you finally get there you just have to go to another one

[@crristox]

What about creating new variables? Like saving fuel or energy consumption, or giving priorities like speed over energy/fuel consumption

[@aiksi5605]

This video felt like it's 30 minutes because I somehow kept falling asleep every ten seconds or so. And it's not boring and no I am not high, idk I guess I just got tired or something

[@veggiet2009]

oooh idea. Space Invaders: Drones Addition. Different levels use different generations of drones as enemies.

[@neut_ro]

I dont understand the sin and cos part in the inputs can someone explain?

[@00swinter21]

basically you take the angle the drone is currently at and get the sinus and cosinus of that angle

[@eyalsegal6730]

Nice work! What mutation/crossover did you use?

[@bobingstern4448]

im more impressed by the smoke, great project though!

[@keltskiy]

This would be a great premise for a game where the character tracks the mouse so instead of controlling the character you're directing it and it gets better as you play through AI learning

[@baconofburger8784]

why not add a fuel limitation (which would refill once they get to a point) forcing them to switch between points as quickly as possible from the beginning

[@chfr]

wouldn't be necessary, they're already rewarded for speed

[@notfirstTHERMAL]

Great video! I've been trying to make a similar recreation of this project in Python but while I get some decent results, I'm struggling with local minima trapping and have failed to get the kind of 'brutal' drones you got at the end of training. Tried having a look at the source code but I'm not too familiar with C++. Just wanna know, what did you use for your fitness function and how did you mutate your networks? A reply would be very much appreciated!

[@Fallout3131]

That drone that got yeeted at 5:30 had me dieing 😂

[@kovacsattila8993]

Why don't you upload a Gen 5500 drone to github? I tryed it but i only founded the Gen 3100 one. I really want to try that one. :D

[@kahwigulum]

Gen 5500 appears to display knowing how to fall rather than turning the thrusters to push itself down.

[@JuanPabloLorenzo.]

Great video! How long have you been training them? Greetings from Uruguay!

[@thesteveremix]

i want to see how chaotic it will be if the drones had collision

[@PezzzasWork]

I will try this, that’s a good idea ;)

[@quietsamurai1998]

Would it be possible to have the drones compete? For example, by simulating the entire population of drones at once, and only rewarding the first drone to reach a target.

[@00swinter21]

That's exactly what genetic algorithms (GA) means

[@quinn840]

Pls make more vids like this I love them

[@241lolololol]

man this is so cool. a bit off topic but how are you rendering the thruster particles and smoke?

[@PezzzasWork]

The smoke is just made out of static sprites and the thruster particles are baked into the flame's texture

[@bluecrystal_7843]

if you had an body orientation/angle input they would have been able to recover from a spin out or even fly upsidedown

[@Algok17]

Very nice result!

[@thetafritz9868]

the target tracking drone would be a really cool and distracting extension, it follows your cursor around where ever you put it lol

[@abeltoth1878]

Really cool project!!! I was wondering what fitness function you used?

[@memento9979]

I like these projects !

[@Vinz_1223]

Now create an additional network which positions the orange dot (target) to navigate around obstacles on its own.

[@artherius535]

400 was such a trooper

[@SoulZeroTwo]

After a few tweaks, I have a feeling this could have real-world use.

[@rishikumarsoni]

Hi Pezzza, I really liked the video and the way you trained it. Can you tell me how can I learn to code to train a model like this ?? I really want to learn how to do this level of coding. pls reply

[@00swinter21]

ask chat gpt. it knows a lot about it i used unity for the physics and did my recreation there and it was even better then the original

[@kolterdyx]

I love this! I'm gonna implement it right now in Python. What genetic algorithm were you using? I'm planning on using Neat

[@CE-ov7of]

how did you get this environment in Python? I want to test policy gradient RL algorithms

[@j_owatson]

@@CE-ov7of not sure if you still need this question answering however i'll give it my shot. My guess is hes implementing the basic algorithm of the envirment in python using pygame and and numpy. Then for the AI my second guess is he'll be using NEAT Python library or custom AI/NN algorithm for the agent and training. That's my guess however if you want any question just reply and i'll do my best to help. Python isn't my strongest language however but i'll try my best.

[@CE-ov7of]

Hey @@j_owatson , unfortunately this is not something I have time/interest for anymore. But I really appreciate your willingness to help! This is what makes the software/tech community great!

[@guillearnautamarit9102]

Wow that's amazing and looks amazing, how did you cross the two neural networks?

[@chinmayghule8272]

That was really cool.

[@ethos8863]

You may have to select more aggressively for speed. They seem a bit slower than what the optimal handmade algorithm could do

[@jenvetcar5319]

great Awesome!!👌👌😀. where did you learn to do this?

[@ravenatorful]

While it was nice for the visual of all the different generations together, I feel like it would have been better to randomize the dot locations so that they have to learn to adapt to a new path every time

[@Success_Unlimited_]

Nice work! Can you propose me material so that I can understand in practice how to build a neural network? Something with examples.

[@PezzzasWork]

That's a good tutorial idea, I will think about it :)

[@Ab-cj6gl]

i wonder do you use frameworks to build your neural network or you build them from scratch?

[@PezzzasWork]

It's from scratch with a naive implementation

[@petersmythe6462]

Would be interesting to have a drone sumo where they can collide and try to shove each other out of a ring.

[@SomeAutomaton]

Ok, now make these drones fight in groups of 5, they can kill other drones in 2 ways one is to ram into enemy drones (killing both of them instantaneously), or shooting them with miniguns (only killing the target if it is hit X amount of times). But every time when they die they respawn, smarter, faster, more accurate, etc.

[@JavierAlbinarrate]

Beginning of the video: LOL!! those squeaks as they fall are really funny End of the video: let's run to buy some food cans before they come for me!!!

[@sulaimantriarjo8097]

how do you tune the weight and bias using GA,? do you intercept the backward process with GA?

[@aesvarash3256]

Can u make a tutorial how to choose the best inputs depend on sample ?

[@linsproul3548]

you should make a game where you control a small ship like asteroids and your goal is to juke out the drones and cause them to crash or see how long you can survive before they hit you or something

[@markvarden3802]

I would love for you to make an eco system like the bibites using those drones

[@av3stube480]

It would be interesting to see which learning algorithm would produce better results, this genetic algorithm or back propagation or something similar.

[@GhostStyle007]

you can't achieve that comparison unless you use super humains to play that game and train the models... since it's not the case, the genetic algorithm will always reach far more better results at some point, it's about time.

[@triplezgames3882]

Am I right, that the final code controlling the drone is really slim and efficient code? It's just some products and sums, and an activation curve, right? Seems pretty amazing for how organic the movement appears

[@00swinter21]

yes exactly. thats the good part about this.

[@rishiniranjan1746]

its really beautiful.... can you please suggest how do I learn all this. What I learn in what seuqence ??

[@adolfocabarcas5795]

where do i star to do games and machine learning like this?

[@MarkusBurrer]

you should place the targets randomly and not in a specific order. And for more challenge, they only have a specific time to reach the target. After the time the target disappears. And finally, the targets are fuel. If they miss too often they run out of fuel. Edit: maybe even add obstacles.

[@PezzzasWork]

In the video the targets are in a specific order to be able to benchmark the different generations, for the training I used random sequences

[@MarkusBurrer]

@@PezzzasWork Ok, that makes sense

[@nobu7198]

the smoke and the flames looks very clean can people play this?

[@tylerbunnell8714]

I would love to see what happens if you give them a finite amount of fuel to manage. Have the fuel decrease quickly/slowly depending on how hard they burn thrusters. Extra bonus for fuel remaining when the task is complete. Death if you run out of fuel.

[@00swinter21]

i think just counting how much fuel is used based on thrusterpower and then rewarding low numbers is better then limiting fuel

[@prakharsethi4910]

Hi! i am new, in 3rd year of undergraduation. can u guide me how have you got this far? and what tools and softwares are u using?

[@cathsaigh2197]

Gen 2600 was a big leap in speed and control.

[@angelodeus8423]

it's cool to see your using dropout, so it learns better

[@alejandromartinez-vp4sx]

Fantastic. Did you published it? Why did you choose GA for learning instead of traditional NN methods, e.g. stochastic gradient descent?

[@spoo77jj78]

"Im a Hovercraft like my Father before me and his before him!"