2002 Honda Rancher 350 shifting issue

I was troubleshooting an electric shift issue and it has grown into a bigger mystery. The unit only wanted to go into 1st or neutral with the electric switch.
After a bit of research I read where it could be because the ECU only puts out 5v and it may not be enough on an older engine. I then installed the bypass system to give it a full 12v. I changed out the shift motor and position sensor to no avail. I then removed, cleaned and greased the reduction gears for the shift motor.. That didn't fix it either.
When you use the manual shifter, it would change gears only when you would rock the wheels forward and backward, but it would go through all the gears.
I then drained the oil, loosened up the front diff so that I pull the front drive shaft off, removed the oil cooler lines and the front clutch cover. Upon inspection I did not notice anything broken or looking wrong.
This video shows the issue I'm seeing. The Cam gear does not appear to smoothly transition between the gears. I'm thinking it may now be a transmission issue. It's almost like there is a spring somewhere that is worn out.
Anyone got any ideas?