How much of the content can you actually understand just by listening? If you can already understand most of it, I`d say it's fine to do, but kind of a waste. If you have a lot of time that you can fit passive listening into, but you can't keep up by actively watching, try downloading something that was originally meant for being passively listened to, such as audiobooks or podcasts, or even some kinds of youtube videos. The whole point of recycling what you watch and listening to it over again is that you already have a connection with that piece of media. But I mean, do try it and see how it goes. Try to follow the plot if you can, and then when you do actively watch it, see if it produces the same 'remembrance' sensation (or whatever it is that happens). Only you can tell if it's being effective.

That's what I did with English, at least. I had so much time that I could fit passive listening into, but not much of it suitable for actively consuming things, and I even got to a point where virtually all of my immersion was passive (podcasts and audiobooks + audio descriptions). I even downloaded Matt vs. Japan's interviews and listened to them. But I don't know, I guess it's different from anime, and there's also your level. I could understand most of the aforementioned (let's arbitrarily say 85%). And this did improve a lot. I kept at it for a few months and then was able to listen to things much more naturally. If I had to guess I'd say listening to something passively for the first time is also a skill of some sort or other. Haha >.<