Reinforcement Learning - My Algorithm vs State of the Art

[@chris-graham]

I think you would be interested in network pruning. This is something that's typically done periodically during training to thin networks. If you examine the weights in your PPO-optimized network, you'll find that many are very small, while others are larger. If some near-zero weights are set to zero, networks will often become more stable after fine-tuning. You'll find that the connections in the network begin to look sparse and very similar to networks generated via. Evolutionary methods. PPO is just an optimizer and will work with whatever network configuration you want. The evolutionary networks shown in the video are all differentiable, so PPO would be able to optimize. That would be an interesting comparison if you'd want to pursue that!

[@nodrance]

i smell a part 4

[@w花b]

​@@nodrance I smell you smelling something

[@Firestorm-tq7fy]

They only sparsen if you use regulation methods like L1

[@chris-graham]

@@nodrance Do you smell it? That smell. The kind of smelly smell. The kind of smelly smell that smells... smelly.

[@bitblit]

@@chris-graham Right you are, Mr. Krabs.

triple pendulum next?

[@dongyulee2095]

Impossible...

[@sumitbiswas164]

How to get the solution for dynamic (n) chain of pendulums? Is it possible now?

[@alxklgn364]

I think that I've read a paper explaining why triple pendulum is a total chaos and impossible to solve. But I would also like to see an attempt.

[@elie_]

@@dongyulee2095 "Source: lol"... kzbin.info/www/bejne/maqxXnaIg9eVaac And all possible instable equilibrium states kzbin.info/www/bejne/f2aqp6qNgNB1o8k

[@elie_]

@@dongyulee2095 "Source: lol" kzbin.info/www/bejne/maqxXnaIg9eVaac (13 years ago) kzbin.info/www/bejne/o5awiJmslpJ9n9E kzbin.info/www/bejne/f2aqp6qNgNB1o8k (even more impressive)

[@Waffle_6]

getting that sort of aid from NVIDIA is super nice. super cool, my school just got an ai accelerator, " AGX Orin" very cool piece of computing and fantastic of AI training and research. also, as someone who is more hardware orientated, it has a super fascinating architecture(shared cpu and gpu global memory!)

[@meronamsamho]

security be damned I want faster training!

[@conorstewart2214]

They definitely are cool but I would not class the AGX orin as an AI accelerator, not in the same way GPUs are. Or at least not just an AI accelerator. The AGX Orin and the whole Jetson lineup is meant for embedding in things, like robots, cars, etc. It is a full system, CPU, RAM, GPU. It is also not very powerful for the cost, at least in terms of raw compute performance. Even a 4060 gets 242 TOPS whereas the AGX Orin only gets 275 TOPS. If you don't need the portability and embeddability of a Jetson system then you are far better just buying GPUs. I can get a 4060 for £250 (yes this is without a CPU and only has 8 GB VRAM) but the AGX orin costs £1992, so just going with desktop PC hardware your money goes much further. For the price of an AGX Orin you could likely build a 4080 or possibly 4090 PC and get much more performance. If RAM is that much of an issue then you should probably look at enterprise or data centre level systems.

[@jibcot8541]

When you are the most valuable company to have ever existed, I guess you can give a bit of money away to teachers and researches, still nice of them I guess.

[@fluffsquirrel]

Thank you so much for this demonstration and adding the links! I didn't know of Isaac Lab and was wondering how it was possible to control the mechanics. Great video!

[@max_me_is]

We got Pezzza's work X Nvidea collab before GTA VI 😭

[@Djellowman]

Shut up

[@harriehausenman8623]

I prefer *this* 😄

[@imanuelbaca2468]

The quality and education of these videos is unmatched please keeping making stuff like this!

[@PezzzasWork]

Thank you :)

[@kubstoff1418]

I've been looking for a subject for my engineering degree and this video might be exactly it! Thank you for the inspiration, your videos are always a blast!

[@briandeanullery]

This is just brilliant. I verbally gasped at those numbers. I am so grateful to be living in a world with this sort of stuff, it's truly amazing!

[@PatrickHoodDaniel]

Oh my god, a video from Pezzza!! I'm so excited!!

[@poketopa1234]

PPO and gradient-based policy learning in general is amazing. I will still say that your struggle to get an evolutionary algorithm to learn this problem led to some really creative and impressive curriculum learning ideas which also apply to PPO :)

[@conorstewart2214]

This is very impressive and makes me want to look into RL for robotics again. I really don't think you can make much comparison about network size though if you only tried one network configuration that you chose randomly. A followup video seeing how small you can make it would be very interesting. It would also be interesting to see you try and take it from simulation to real life.

[@_nemo]

17:06 That's so similar to what the timescales of evolution in nature, and a human learning a skill are like. That's kinda crazy. Really makes it look like the algorithms successfully mimic real counterparts.

[@TheRealZitroX]

And still, some Human doesn't learn at all.

[@raspberryjam]

@@TheRealZitroX mean

[@0osk]

@@TheRealZitroX *some humans don't learn :)

[@realzakariax]

7:50 I love how the base also returns to the middle of the field, so fascinating!

[@drhxa]

Have you considered adding physical parameters from motor torque and motor weight? This would help you get much more realistic sim and difficulty level. Also, realistic response times (based on inference speed + connection latency). Also, you can either have a motor at the base and one at the middle joint or both at the base. You may also consider adding a battery's weight, so you have the voltage required to power those two motors for some period (say 5 min). This will be an awesome challenge and help you connect simulation to reality much more closely, which sounds super exciting. Looking forward to see if you end up working on it!

[@lorem9587]

I like these suggestions. Where are the two motors, though? I thought there was only one, the one driving the carriage.

[@drhxa]

@@lorem9587 oops, haha, you're absolutely right!

[@drhxa]

The hinges have to be free, that's the whole point of the control problem! My bad haha

[@requestfx5585]

Thanks for this high quality video and comparison of those algorithms, very nice. Keep it up

[@nodrance]

I'd love it if you spent more time playing with this. Smaller network, triple pendulum, add random forces to the sim to increase stability, maybe make it target alternate configurations (for example first arm up second arm down or vice versa) and make it chooseable, make it not able to exert as much force. Really push it to the limits and see what it can accomplish

[@FoXMaSteR001]

Aweome :D Try to apply to same method to penspinning learning, the fact the brain can coordinate all fingers to use the momentum of the pen in complexe figures is amazing, the duration you need to learn the tricks is probably linked to the touch sensation instead of the view of the figures, as a pro can perform tricks without watching his hand. With time the brain can adjust the position of the hand and fingers depending of the rotation of the pen to save the figure or to trigger a new one to smoothly, that's very automatic at some point. Using a different pen with another balance lead to faster adaptation once the person is pro, the only way to learn it is to try and error, which looks like this video. The movement tends to optimized with time, once you manage to make a trick with the pen it seems your brain remember what happened, which can help to do it again, when this happen it's like bicycling, you can spend a week to try with no success and suddently reach very high sucess rate in a few hours once you made it once. That's a very weird feeling.

[@FIT7Y]

I would love to see you tackle other kinds of equilibrium positions. Where one of the pendulums is up while the other is down. And maybe even efficiently switching between the different equilibriums. Something like Embedded Control Lab's videos about switching between the different equilibriums for a triple pendulum.

[@optozorax]

I'm solving similiar task: I'm trying to learn AI car to drive, with realistic physics. And I was struggling with learning as you do in previous video, I was inspired by your solution and tried another approach: I started from simple physics (no inertia, no wheels, just rotations + offsets), then gradually interpolated between this simple physics and hard physics. And my NN was able to learn how to drive perfectly. But then I tried energy-based model, basically it's an NN that receives current state, desired action and outputs just a single number - energy. You need to find best action that outputs minimum energy. I iterated over 9 possible actions, and that NN was able to learn how to drive in complex physics without any hacks and very fast. So, what do I think: first try CMA-ES, as a superior zero-order optimization method. I think that NEAT is a trash, and one day I will test it out. Then you should try energy-based model. Then it will be someway fair comparison. Now it's not fair absolutely, and I slightly disappointed with this video.

[@vastabyss6496]

what's the difference between energy and the loss? Also, your method sounds a lot like a DQN if I understand you correctly, and vanilla DQNs are much worse than PPO

[@optozorax]

@@vastabyss6496 energy is minimized during inference (to find best action for an agent), loss is being minimized during traiting. So, to train an energy based model you need to minimize energy in every step of a simulation, while minimizing overall loss. Many minimizations inside a big one.

[@marcserraortega8772]

Thenks a lot for the high quality video! I would love to see more videos related to RL in the future. Keep it up!

[@rcnhsuailsnyfiue2]

Please consider a side quest to balance a double pendulum IRL?! 😱 You could (relatively) easily build a device for this with a single stepper motor, drive belt, and an arduino. Look at X/Y plotters like Axidraw, enthusiasts regularly build these things themselves with off-the-shelf parts. Hook the stepper motor up to your model, and you’ve got a scientific viral video just waiting to happen…

[@firedeveloper]

I would love to see it IRL but that's a serious task. 1. There is a huge gap between model and real hardware. 2. IRL you can't have x,y,z positions without camera. The most viable way would be with accelerometers and definitely rings with contacts for data transfer. Imagine how hard are some projects with a simple PID, this is 100x more difficult.

[@rcnhsuailsnyfiue2]

@@firedeveloper fair point, maybe not “easy”. I just think for a motivated novice it would certainly be achievable. If it were me I would use a rotational angle sensor on each pivot point, they’re very cheap and can be frictionless too. Mount it all on a sliding steel rail, pulled continuously along the long axis by a computer-controlled stepper motor. Then it’s simply a motion control system running in a feedback loop. Because the stepper motor is quantized, you can know the entire state of the system from just the 3 angle sensors.

[@conorstewart2214]

​@@firedeveloper it would need some way of sensing position but if they can make the simulated model as accurate as possible to the real one, including the sensor data the model is fed, they it should be possible for it to work in real life.

[@rsflipflopsn]

@@conorstewart2214 same thought. maybe balancing a double pendulum (so I mean a pendulum with two moveable axis, could be a triple pendulum? sorry I am not that familiar with the nomenclature of these in the field of physics) is possible if you have two really precise sensors at both axis which respond with their positions accordingly and really fast. the bigger challenge (if you do something like that with ML) could even be the response time of the model plus the call to the actuator (?). nevertheless I really like your thought!

[@rcnhsuailsnyfiue2]

@@conorstewart2214 there’s no need to sense position, only the angle of the pendulum. The position is inherently measured by simply knowing the history of commands given to the stepper motor. The same technique is used by 3D printers, as long as they start from the “home corner”, their position will be known to the computer by simple addition/subtraction.

[@sutsuj6437]

Do note that Evolutionary algorithms are usually better than pure RL agents for problems with very sparse rewards (Which is not the case here). For these problems, a hybrid approach might work best.

[@florianvanleeuwen6683]

Randomly seeing my physics lecture building on youtube, nice video :)

[@smokeydude3]

Why not try testing a more compact PPO network?

[@miran248]

Silently hoping for a part 4 and a triple pendulum :)

[@PezzzasWork]

I tried but I couldn't manage to find a good solution (they were very unstable)

[@stephaneduhamel7706]

@@PezzzasWork Maybe you could try distilling the working network and see how small you can make it before it breaks?

[@cagedgandalf3472]

@@PezzzasWork Try compacting only the actor network (and also lowering the learning rate) and keeping the critic network to default. That is what I did, although I use TD3 with auxiliary networks.

[@vincentverbergy9816]

​@@cagedgandalf3472 PPO is not an actor critic network? In general with RL size comes at the cost of computing time and risk for over fitting is not necessarily that big meaning that bigger network size isn't really a drawback given enough compute.

[@waity5856]

It's amazing to see it temorarily give up on balancing when it gets too close to the edge of the rail, so it can try again later in a more favorable position

[@nexttonic6459]

Now you have to add flex to the materials, a small gap to the rollers and the beam. Then add a slack in the bearings...

[@rcnhsuailsnyfiue2]

Don’t forget to account for the acoustic energy of a squeaky pendulum hinge… And a gentle breeze from a robot farting nearby…

[@nexttonic6459]

@@rcnhsuailsnyfiue2 Agreed. Nvidia talks about real physical world, yet farts and acoustic energy probably not accounted for.. though that is a simulation thing I don't think the video maker can effect that.

[@BananaDude508]

just did a school based research paper on machine learning and pendulums using your other videos as reference, this video wouldve been perfect if it was 2 months earlier lol Either way thanks!

[@R.B.]

The next task is transitioning between states, of which there are four positions, both arms down, both arms up, and two positions with one arm up and the other down. After that you can move to three arms, where there are 8 states. At three arms you have a chaotic system, but this has been solved already with physical systems, so it would be interesting for a simulated system.

[@jaiveersingh5538]

Isn't it already a chaotic system with just 2 pendulum arms?

[@StevenJAckerman]

Very nice work. Thank you for sharing.

[@harriehausenman8623]

Beautiful and informative video! 🤗 So satisfying animations. thx 🙏

[@PezzzasWork]

Thanks :)

[@harriehausenman8623]

@@PezzzasWork Wow! that was quick 😄

[@harriehausenman8623]

@@PezzzasWork I only heard of Isaac before, but wasn't aware it's *that* powerful! 😲 I wouldn't mind a follow-up video where you show the things addressed in this comment section. Like how the smaller layer sizes failed (blooper-time!!) and stuff like that.

[@alejandromartinez-vp4sx]

Beautiful as usual.

[@Deniil2000]

15:58 i really like how it knows not to chase the pendulum into the end of the rail, and makes a flip instead

[@dottedboxguy]

well, sure it's only a few minutes of training, but just how much computational power (or just electricity) was used during these few minutes ? i think it's much much more than your simpler approach. it's cool, but it would be interesting to do a test with power usage normalization to do a fair efficiency comparison

[@sirynka]

Still, 8h of cpu time, even in single core mode, would consume around 40w, so 320wh total. A fully utilized 4090 - 450w * 5m = 40wh. Units were edited according to @somedudewillson. Thanks for the explanation.

[@dottedboxguy]

@@sirynka what tells you it's a 4090 though ? as it stands, it seems more like a cloud GPU compute approach within a large GPU bay, which consume a tremendous amount of power, though only pezzza could confirm that

[@PezzzasWork]

I didn’t specify it in the video as I thought the difference in time was large enough. My algorithm consumes around 120wh for around 5 hours when the 4090 consumes around 150wh for 3 minutes. I agree that it would have been a nice addition in the comparison.

[@dottedboxguy]

@@PezzzasWork thanks for the precision ! this is indeed good to know, and does change things around a little as to which solution is better, especially considering the resulting NN depths

[@somdudewillson]

@@sirynka A Watt is a rate of energy transfer - specifically a Joule per second. A 40W CPU does not consume 40 Joules per second per hour, on account of how that doesn't make sense in this context (If the rate of energy usage was changing it _would_ make sense as a unit, however).

[@EricSundquistKC]

That is seriously impressive!

[@Jiorgos3D]

Yay! New Video

[@thomas_c]

Amazing job ! I'm in love with PPO now :) What hardware did you need to train your ai ?

[@PezzzasWork]

Thank you :) I added the PC spec in the description.

[@thor9000]

Super nice video and explanation! Question, how much did you need to tune the reward, and how essential are the rewards with the low weights?

[@louisdupont2126]

Great video man ! Is it possible to share your code you really motivated me to dive deeper into isaac lab !

[@michael_pio]

Great informative video

[@devsquaaa]

Love the content. Please keep it up.

[@Blooper1980]

This was so interesting! 🎉🎉❤❤

[@gpjedy7379]

Sick! Will you do videos on training multi-agent tasks?

[@rafa_br34]

Well done. However, I feel like the video was a bit rushed. Primarily because you didn't test other network sizes, which would have made it more fair for the evolutionary algorithm. It also makes me wonder if the network really "learned" how to balance the pendulum or if it just memorized how to do it in the weights.

[@NaifAlqahtani]

Agreed. This video contained no real information. Just an ad and a benchmark of an algorithm on dissimilar hardware

[@PezzzasWork]

I didn't specify other architectures because I couln't manage to have a satisfactory solution with smaller networks. Since I am not an expert with PPO I prefered to only mentioned that it is probably possible in the video. Regarding the learning I think the fact that the solution was able to recover from any perturbation means that there is no overfitting here.

[@lMINERl]

Love your work im a big fan XD

[@Gabonidaz]

1:08 what are this dashboard? How did you builded? I need to try ...

[@goatknight777]

PPO really is incredible in all ways

[@tyler2854]

Amazing work

[@Xotexgaming]

Beaucoup trop fort !

[@glizzygoat]

the legend is back

[@nikbivation]

Love this!

[@fenokineza6259]

I just watched a 20 min ad, and im happy

[@IsaiahSugar]

would love to see you implement ppo yourself! i think that as a viewer i would learn a lot more from that

[@seanoconnor1984]

Continuous Gray Code optimization is very good. SwitchNet, the one using a fast transform, is a neural network that uses a low number of parameters.

[@GelloMello-j9z]

woah....the graphical interface is so gooood

[@luke.perkin.inventor]

Great video, can you do evolutionary distillation or pruning of the ~65536 parameter ppo model?

[@Rekklessss]

How did you manage to create such a sleek looking dashboard for the model in the beginning of the video? 1:19

[@Thk10188965]

I wonder if you can use PPO to get a solution fast, then evolution to slim it down (by adding some cost per node/connection I assume)

[@AureliusAurelius]

Awesome! Next, can you build a real-world robot (or partner with a hardware KZbinr) and test the algorithms on it?

[@narpwa]

WTF IT LOOKS SOOO CLEAN HOW DID YOU DO VISUALS LIKE THAT ???

[@UonBoat]

Such a smooth live chart system in the initial part of the video. Does this came from certain library or you wrote it yourself? Thanks.

[@PezzzasWork]

Thank you :) It is a tool I wrote myself, I plan on doing a tutorial on the subject.

[@UonBoat]

@@PezzzasWork Sounds cool! I know it's a bit off topic, but I’m looking forward to it whenever it’s out.

[@felixconrad9248]

great video as always, i am not excited to see a video by a lot of youtubers but you are surely one of them

[@PezzzasWork]

Thank you :)

[@Nothingguy562]

Hey I am very starstruck by your work I would be very grateful if u could tell me about how you learnt all of this.What would you recommend to a total beginner. Thanks

[@r.g.thesecond]

11:20 I'm a bit surprised. Is it not possible to use constraints or IK in Blender to also describe the joints, and export them as well?

[@PezzzasWork]

It is certainly possible but I am not very familiar with all these tools, for my use it was simpler to rig the model directly into Isaac Sim

[@marcelob.5300]

Would it be possible to include in the description the hardware specs, please?

[@PezzzasWork]

I added them in the description

[@marcelob.5300]

@@PezzzasWork thanks a lot!

[@sucim]

You might be interested in looking into RLtools / the "Learning to Fly in Seconds" paper!

[@marcelob.5300]

You're great!

[@Aeorthian]

In this simulation you mention you need both the position and the velocity of each joint. Your model does not appear to have any rotary encoders modeled on it unless you have a point mass added to represent it that we can't see? You would have to retrain this if you actually wanted to use this in the real world as it would require rotary encoders to measure the angular velocity/position, no? Also, does your bottom motor have a rotary encoder built into it or does it also lack a rotary encoder? Still a great job with the proof of concept even if it's not actually usable in real life.

[@n.lu.x]

did you by any chance try the OpenAI-ES algorithm from their 2017 paper? its quite simple yet powerful for (larger) neural networks. + you could also run it on the gpu in parallel

[@Build_the_Future]

Can You do more with Issac Lab I always run into problems when using it.

[@MarimeGui]

Did this simulation include limits on acceleration to try to match real motors ?

[@phrozenwun]

For the single pendulum, is it possible to move the inverted "upper" node to any horizontal position as fast as the driven node can move?

[@rishiniranjan1746]

this is crazy Man

[@Bluelightzero]

Is it possible to analyse what these neurons are doing?

[@PezzzasWork]

Probably, I don't know how though :D

[@steve_gatsis]

Us there a comparison on how "demanding" each method was in terms of computational resources and memory? What i mean is; after training, how much does your pc "struggle" to obtain the result it trained upon Do you think something like that matters in the end?

[@PezzzasWork]

This is a tough question. On the one hand, PPO uses a much larger network than the evolutionary approach, but inference is performed on specialized hardware that is far more efficient for mass computation.

[@lefm_]

Yeah i came from home building a small evo AI class in c# using maybe 6 nodes, stumbled upon ml-agents where solutions involves 256, often 512 nodes. It looks like a need for PPO.

[@left_eyebr0w]

You talk about things you like and I like that So am I 🧐

[@biobuu4118]

Very impressive but what would happen if the task was not to balance the poles but to aim let's say 90° angle at their joint ?

[@expired___milk]

Could you use a big network using PPO and then make it smaller using the evolutionary algorithm?

[@kahwigulum]

this is how the borg will learn to modulate their shields against phaser fire

[@cloudzero2049]

Any possibility of comparing TD3 (Twin Delayed Deep Deterministic Policy Gradient) to PPO for this? I'm curious because I am working with TD3. It's a little more complex than PPO from what I understand, and maybe overkill for this project if that holds true, but I was just curious.

[@lucao9059]

how can i like this vid more than once?

[@tofonofo4606]

So cool.

[@FlashTheMusik]

How do you make these awesome dashboards for your visualization?

[@PezzzasWork]

I am using a tool I wrote myself

[@EliasMheart]

I enjoyed your showcase! There was something nagging at me throughout, though... I feel like you are comparing apples and oranges here. Evo and RL are different paradigms. And comparing them on the same task with different amounts of resources... I guess I'm just not sure what you are trying to say.

[@PezzzasWork]

The point of this video is just to compare my homemade approach vs modern reinforcement algorithm. These are different algorithms but in the end what I want is a balanced double pendulum. This video only shows what to expect from the two options regarding training and results.

[@EliasMheart]

@@PezzzasWork Ahh, right! So exploring different approaches, and the comparison is not of the approach in general, but specifically for this task. Gotcha, thanks for the explanation (:

[@yannsadowski8292]

Hi, you say it take you 5 hour with the Evolutionary tech. But with the RTX 4090 or another graphic card ?

[@PezzzasWork]

The evolutionary algorithm isn't GPU accelerated, it is running on the CPU (mutlithreaded). A big advantage of most of RL algorithms is that they are able to run on GPUs very efficiently.

[@galacticlava1475]

Can you please post your code in the description? We’d love to tinker with it.

[@PezzzasWork]

I will make my fork of Isaac Lab public soon and add the model featured in the video

[@galacticlava1475]

@@PezzzasWork Thanks! Your AI content is some of the best on youtube rn. And I really commend people like you who keeps code open source so that we can all learn together.

[@PezzzasWork]

@@galacticlava1475 thank you !

[@LinkLaine]

if you expand the problem to full 3d where you have cart on a 2d surface and the pendulum that can fall in 3d will that algorithm be so effective as in 2d?

[@ErickTakada]

Got a question: what if you add a slower reaction time? Like a human handling the pendulum?

[@Nyihm226]

Maybe look at realistic bounds for velocity or impulse, it looks like it requires a lot of motor acceleration to achieve its control

[@RaaynML]

Seems like you can actually get surprisingly close with an order of magnitude less params if you are willing to train longer

[@mozartantonio1919]

awesome

[@szeredaiakos]

So ... for non-linear, unstable system control .. you got a tool which requires more upfront investment and it is significantly more expensive to run to reduce the initial time investment. In certain applications it may be worth it, in your application it most certainly does not.

[@theshadowx]

Do you plan on making prey vs predator 3?

[@the_real_aaxel]

honey wake up, pezzza made a new video

[@Markus-r6g]

6:57 the "simple task" is the limit of humans because a double makes it impossible for a human to accomplish

[@afdf96]

What about n-pendulum with n > 2? Or n-body problem?

[@ofabiolima]

Please make a prediction of three body problem

[@HettyP]

Hey man what tools are you using for your visualizations?

[@PezzzasWork]

I am using a tool I wrote myself

[@HettyP]

@@PezzzasWork i see thanks!

[@ArnaudMEURET]

The cart motor seems extremely (unrealistically) capable. I wonder how the network would react with a more reasonable responsiveness of the cart.

[@adamp3988]

I'm still saddened by the fact that Nvidia has so thoroughly cornered the market on AI hardware and consequently *software* . It would be better to have some viable competition, or to at least make these things *possible* on AMD cards to prevent vendor lock-in. Understandably, AMD is not a leader in the space for a reason --- that being its generally inferior hardware. But I feel like Nvidia is doing damage to the future of the field by creating an ecosystem of things that *only* work on its hardware while AMD tends to focus more-so on open standards that can be used on any card.