Transformer-based Neural Augmentation of Robot Simulation Representations

Simulation representations of robots have advanced
in recent years. Yet, there remain significant sim-to-real gaps
because of modeling assumptions and hard-to-model behaviors
such as friction. We propose to augment common simulation representations
with a transformer-inspired architecture, by training
a network to predict the true state of robot building blocks
given their simulation state. Because we augment building blocks,
rather than the full simulation state, we make our approach
modular which improves generalizability and robustness.