I recommend reading Japanese books in English translation, for example, Before the Coffee Gets Cold by Toshikazu Kawaguchi, The Kamogawa Food Detectives by Hisashi Kashiwai, or There's no such thing as an easy job by Kikuko Tsumura. You don't need to finish a book. It's OK to give up as long as you continue reading other books in English, as you said. I'm bilingual and have been reading in Japanese and English for decades. I read all seven Harry Potter books in English as a beginner. So, it's not impossible.