If you pay attention to the vertical bars on this taiko map, you can confirm that it never actually loses track of its original 4/4 beat. It doesn't just occasionally stop the chaos and return to being 4/4, it returns to the _same_ beat times as if it had never left.