DTC: Deep Tracking Control

We have combined trajectory optimization and reinforcement learning to achieve versatile and robust perceptive legged locomotion.
Published in Science Robotics: www.science.or...
arXiv: doi.org/10.485...
Abstract: Legged locomotion is a complex control problem that requires both accuracy and robustness to cope with real-world challenges. Legged systems have traditionally been controlled using trajectory optimization with inverse dynamics. Such hierarchical model-based methods are appealing due to intuitive cost function tuning, accurate planning, generalization, and most importantly, the insightful understanding gained from more than one decade of extensive research. However, model mismatch and violation of assumptions are common sources of faulty operation. Simulation-based reinforcement learning, on the other hand, results in locomotion policies with unprecedented robustness and recovery skills.
Yet, all learning algorithms struggle with sparse rewards emerging from environments where valid footholds are rare, such as gaps or stepping stones. In this work, we propose a hybrid control architecture that combines the advantages of both worlds to simultaneously achieve greater robustness, foot-placement accuracy, and terrain generalization. Our approach utilizes a model-based planner to roll out a reference motion during training. A deep neural network policy is trained in simulation, aiming to track the optimized footholds. We evaluate the accuracy of our locomotion pipeline on sparse terrains, where pure data-driven methods are prone to fail. Furthermore, we demonstrate superior robustness in the presence of slippery or deformable ground when compared to model-based counterparts. Finally, we show that our proposed tracking controller generalizes across different trajectory optimization methods not seen during training. In conclusion, our work unites the predictive capabilities and optimality guarantees of online planning with the inherent robustness attributed to offline learning.
Authors: Fabian Jenelten, Junzhe He, Farbod Farshidian, and Marco Hutter
Video: Fabian Jenelten
#ANYmal #leggedrobot #robot #robotics #robotdog #AI #reinforcementlearning #rl #rescue #innovation #armasuisse #arche2023 #scienceresearch #stepping

[@TheCrassEnnui_2501]

To say this is impressive would be an understatement. I look forward to what comes next and wonder what the state of the art will be in 10 years.

[@leggedrobotics]

Thanks a lot. We hope that in 10 years these robots can be deployed fully autonomously in hazardous real-world environments to assist humans or even completely remove them from harms way.

[@gloudsdu]

The most natural movement I've ever seen, so good

[@avinashthakur80]

Truly impressive. I had previously tried to implement a walking gait for walking robot, but the MPC controller never worked out of the simulation. Seeing this paper motivates me to try it again.

[@leggedrobotics]

We are very happy to hear that we can inspire people all round the world. No control methods works out of the box. There is always a lot of tuning and engineering work required to successfully bridge the sim2real gap.

[@divinusfilius3771]

This is way more impressive than ANY other demo I've seen lately from anyone. Just don't go greedy and have governments take up all your contracts.

[@BalajiSankar]

Your work on walking in the soft deformable floor when the camera perceives a rigid surface is amazing. I am floored.

[@c016smith52]

Great video and impressive work! Can't wait to see more about swarm planning and coordination in the future, thanks for sharing.

[@leggedrobotics]

These are indeed very exciting research topics. Check our channel for updates about coordinated robot collaboration.

[@deTobs]

Absolutely amazing!

[@AnianBrosig-yv3jn]

pretty amazing work

[@Cian-_-]

that was awesome!

[@Skora420]

Do these robots have some kind of pressure sensors on their limbs? It seems to me that this makes it extremely easier to navigate on soft and uncertain ground. Although the design is not perfect yet, it moves incredibly well on a hard surface and I am sure it will be very useful in helping people.

[@leggedrobotics]

They do not have force sensors on their feet because the repeated impacts will most likely break them. However, we can measure the torque at the actuators, which allows to estimate the interaction force. But you are probably right: if the robot would also have a sense of touch, it would understand its surroundings even better.

[@waynephillips2613]

They are ahead of many other robotics companies including Tesla when it comes to locomotion...but what about autonomy? The robot needs to do this autonomously for this to be impressive. And self charge/self sustain

[@leggedrobotics]

Absolutely agree! Navigation is the next big step that we try to tackle.

[@deepanshu5592]

This is so awesome

[@matthiaskubisch]

chapeau ❤

[@divinusfilius3771]

So with artificial training environments with realistic physics you can literally train this thing fast forwarded, give it thousands of years of experience in days or something?? That's so cool. We're all fked.

[@omnianti0]

it start to look serious atill wonder the autonomy of such battery

[@divinusfilius3771]

Really incredible work. Wow

[@Suushidesu]

It stumbles very "environment aware" looks like an actuall creature :o

[@VK-qh6pr]

Apart from tge very impressive results can you say ehat you have used to simulate/train the trajectory logic?

[@leggedrobotics]

The reference trajectories come from a model-based optimization method. In our case we have used TAMOLS (arxiv.org/abs/2206.14049), but it could be substituted with any other method aswell.

[@NeuroScientician]

Is this open-source project?

[@leggedrobotics]

The TO method, unfortunately, is not open.source because it depends on ANYmal research. We currently considering to open-source the RL part independently.

[@Ree1981]

I feel like this will soon be child's play for anyone, with how fast AI is improving. Model in the slipperiness and sliding footholds in the 3D simulation and train a new AI from the ground up just to see if it works. I believe we've learned from other new AI systems that "human intervention", like how these robots always 'step' rhythmically, even when 'standing still', is something that holds them back in the end. I don't know what type of AI system this bot is using, but it definitely seems like an advanced iteration of a fairly old way of doing things in the AI space. Don't be afraid to step out of your comfort zone and have a completely separate AI try to learn things solely from 3D environments, with little to no 'humans thinks it should be done this way' intervention.

[@leggedrobotics]

In this work, we have combined model-based with model-free control. While this does combine the advantages of both methods, it also merges some of their disadvantages, one of them being the fixed gait. This assumption reduces the number of potential motions but also ensures that the learned behaviors transfer well to reality.

[@Ree1981]

@@leggedrobotics Fair, and I am just speculating. But let me ask, is this the same movement system as other popular 'dogs', like Spot? If no, what are they (Boston Dynamics) doing differently? Because their robots definitely seem... well, more agile to be honest. Haven't seen Spot in a while though.

[@leggedrobotics]

@@Ree1981 Most quadrupedal robots, including Spot, are deployed with a gait pattern called "trot". This type of foot alternation can typically be observed with horses, zebras, and deers when they target intermediate fast locomotion velocities. It is a very stable gait that does not requite large lateral or vertical base motions. Because of these properties, it also emerges with learning-based methods, even when not specifically enforced (in our case, however, we do enforce it). Spot's legs are are much lighter and longer than those of ANYmal. This allows for fast gait cycles and large steps, without moving a large amount of intertia. Think, for example, about a pitbull (ANYmal) and a gazell (Spot). Both animals have specialized locomotion behaviors based on their kinematic structure and dynamic properties.

[@Ree1981]

@@leggedrobotics Ok, so it could be that my opinion was spot-on (pun intended). Human intervention could be slowing your progress. Because let's face it, ANYmal looks slow and clunky by today's standards. Consider a new approach.

[@averagehummus]

Black mirror vibes

[@divinusfilius3771]

I can see IRS ordering 200,000, oh these will find you even under your house. Attach some lasers to the front, you WILL pay your taxes.

[@tingdongyan7502]

black mirror is coming

[@leggedrobotics]

Our robots don't harm humans, animals, or other robots ;)

[@peterlustig2048]

@@leggedrobotics A robot may not injure a human being or, through inaction, allow a human being to come to harm. :)

[@gf2e]

@@peterlustig2048Based on that rule, why doesn’t ANYmal try and grab a cigarette from any smoking human it encounters? :)

[@peterlustig2048]

@@gf2e lol, love it! The second law states: A robot must obey the orders given it by human beings except where such orders would conflict with the First Law. So I guess you still have a vaild point. It is really hard to state what we want robots to do :)