I am Japanese but have been living overseas for about 30 years.. Sarchan's Japanese is not bad at all. It is good as a foreigner. I have good and bad experiences in my overseas life. Life is no perfect. I think you just need to get used to it. What you experience as a foreigner is not only you are experiencing . I think all the foreigners living outside their home countries experience similar things like what you are talking about. So you have nothing to worry about. . That is how it is and something you can not change and it is just like that . If you still feel uncomfortable for many years you have a choice to move out of Japan if you like. You have choices .