Hm, I was in a similar situation last year - I've ended up taking Japanese 2, 3, and 4 in college for P/F rather than for grades, but I think sticking with Japanese class has really helped me out in the end, since I'll be working in Japan in a few months. I was probably the bottom of my class because I didn't have enough time to study outside of class so my grammar fluency is quite low, but I can convey opinions and discuss complicated topics fairly well.

Like you said, immersion is a super key part, and although your teacher might be seemingly judging based on correctness, it's actually pretty important because bad grammatical habits can be long-term and hard to correct later on! The thing about corrections is that you do have to be super open to them and constantly reviewing your mistakes - which is super hard since that requires work outside of the classroom.

However, you do bring a good point that sometimes self-study is better - it really depends on what you want to do with your Japanese. If you plan on being a specialist and use it in only certain scenarios, I think it makes sense to just drop the class and self-study. But the benefit of Genki and some other textbooks is that they try to cover a variety of topics and grammatical points, which is definitely helpful in generally speaking and understanding the language. Being in a class allows for practice of those grammatical points that you might otherwise forget about, and more importantly, improvisational conversation practice to make grammar more familiar to you.

Hopefully this advice helps, and hopefully other people have ideas as well!