I made machine with similar Spindle(300W-24V), full Stell frame 150Kg, I can mill 6mm end mill: F500, ae 10mm, ap 1mm with glass finish. You lack stiffness. And also i added bearing(6203-16) on er11 holder to rise stiffness. Click my name to see machine in action. By the way good work.
your second attempt is no highfeed milling, it‘s just shoulder milling with too less ap and propably too much feed. i don‘t know, what your real feedrate was at the end, but if you push it to far, your endmill isn‘t cutting anymore, it’s braking off the material. for real highfeed milling you simply need a different type of cutter! your regular endmill has a tip angle of 180°, that does not create that chip thinning effect, you need for highfeed milling. get yourself a highfeed mill, that has sloped cutting edges at the tip, they have roundabout from 10-20° rake angle - the tip angle then is 140-160°. yor ap is that 0.2-0.5mm, but ae is 0.7-08 x cutterdiameter! the feed per tooth should be roundabout 4/5 x faster, than your actual value. give it a try, dude! let us see the results in another clip!🫵👍