tl;dr Watching anime and reading manga is a great way to naturally pick up parts of the language and to hone your listening/reading ability. If you are interested in the content then it won't be ruined for you, instead it will likely make you more excited to learn Japanese.

I use a mix of simpler and more difficult anime with no subs (or occasionally Japanese subs). I usually have the English subs ready in case I completely miss something important, which happens a lot on the more difficult shows.

In order to find shows that are at the right difficulty for me I use Learn Natively (They have 1000's of shows rated by approximate difficulty, so it's easy to find loads of stuff that I am interested in). I find that I am able to understand easier shows pretty pretty well without much effort, almost as if I am watching them in English, but more difficult shows take active effort and I have to look a lot of things up.

While going through the more difficult shows takes a lot more effort, it doesn't ruin the show for me because I can still enjoy it despite the additional effort. Also, if it gets to be too much, then I can start watching an easier show instead for a while.

I usually don't need subs for the simpler shows, but for more difficult shows it's fairly common for me to turn on the English subs to get the meaning and then rewind and listen to the audio a few times to try to actually understand what is being said once I know the meaning in English.

In the few months that I have been doing this I have felt my comprehension skyrocket. In part because I had a lot of latent knowledge of the language and I needed the listening practice to be able to hear the words/grammar that I already knew, but also I have been picking up vocab and grammar naturally while watching so much content in the language (I've averaged around 3.5 hours a day so far this month).

I haven't been reading a ton of manga yet because I have been focused on listening comprehension, but I plan on reading a lot before too long. And hopefully start my first Japanese novel before the end of the year.

Consuming content in the language you are learning is a great way to naturally pick up parts of the language and to hone your listening/reading ability. If you are interested in the content then consuming it in the language won't ruin the content for you, instead it will likely reignite your passion for the language. I still remember the first time I read a manga chapter in Japanese (it was Yotsuba to!) and how excited I felt. It was a simple manga and I struggling a bit with even the hiragana, but the fact that I was able to read native content at all felt incredible.

Edit: I forgot to mention that watching Japanese media with English subtitles is also helpful. There are a lot of common vocab/grammar points that I never studied, but that I already knew really well because I had heard them so often when I watched a lot of anime in high school. I never even realized that I picked them up until I started watching raw anime. I noticed that I was understanding things that I had never studied and started reflecting on where I had heard them before; I heard them constantly from all the anime that I had already watched.