Ahhh yes…. Honestly, I still don’t know all of the special counting numbers in Japanese by heart ;( I can read Natsume Soueseki but I still mess up stuff like にふん、さんぶん Edit: and apparently I messed it up again even in that example lol
It's just because you just read but don't listen Japanese that much. Ippun, nifun, sampun... are grammatically hard to grasp but if you listen to Japanese that how they use them, you'll be used to use them as well.