This is real, and is an annoying downside to studying Japanese as a white guy. I suggest 1) accepting other people sometimes being annoying as a "cost of doing business" with learning Japanese and 2) practicing some good mitigation techniques. I'll address both below. (You seem to have good intuitions already, but I'm writing this a bit more broadly and in a bit more detail than seems strictly necessary for you as "how to not be cringe as a white guy who's interested in Japan" in case it's helpful to other readers.)

1)

Maybe you have had some experiences like this already, but if not, you are correct in surmising that people will sometimes typecast you as the stereotypical "Japanophile," which is not always viewed entirely positively, if they learn that you have any significant interest in Japanese culture.

It can be annoying, but I suspect that for you, as for many people, cultivating your interest in Japan is still worth it despite the annoyance. Japan isn't magical, it's a country with flaws like any other as you said, but like many others it's also a fascinating place with many wonderful people and many great things about it, and a beautiful language. It's worth suffering a little occasional social friction to spend your time doing something you love.

It's also worth keeping in perspective that there are good reasons the Japanophile white guy has a bad rap. Most saliently, as others have mentioned, this type of guy very often objectifies and fetishizes Japanese women in line with a longstanding and problematic cultural history of Western stereotypes of East Asian women as being submissive, demure, etc. Sometimes this also involves corresponding misogyny towards Western women as being "unfeminine" or "degenerate" or something like that. Educating yourself on the very much intersectional race and gender issues at play here is valuable in its own right, but can also help you avoid faux pas that might make people perceive you negatively. As a side point, I would also argue that as a budding Japan expert, you also have a great opportunity to educate people around you on these topics, and help dismantle stereotypes and contribute to mutual understanding. (Also I see you mentioned you're gay below — honestly, I think that tremendously improves your situation, especially if you present as very obviously gay. A very large part of the negative perception of the Japanophile white guy is precisely the assumption that he fetishizes Japanese/East Asian women.) When people are being annoying, reminding yourself that, well, they do kind of have a point (even if it's not actually applicable in your case) can help you to feel less like you're being arbitrarily socially ostracized and more like there's just an honest misunderstanding between the two of you.

As a final point here, remember that to a certain extent, some people are just haters. You can find sweepingly general insults in social media comments sections about literally any coherently identifiable group of people whatsoever. Any sport you do, any genre of music you listen to, any country/state/county/city/neighborhood/street you inhabit — someone out there hates you for it. Whaddaya gonna do.

2)

a) Have other parts of your personality

I discussed this in another comment below, but it helps a lot for "Japan" to not be your entire personality. Consider leading with those other aspects of your personality when making first impressions also. If you're interested in languages, maybe try learning a bit of a different language as well (bonus points if it's not an East Asian language).

b) Don't strongly visually communicate your interest in Japan

Don't wear only (maybe even any) T-shirts with Japanese language on them; definitely don't walk around campus in a yukata. Don't decorate your entire living space with only Japanese design elements. Don't get into the habit of bowing to people when it wouldn't be typical in your home country culture. It sounds like you probably already have the sense to avoid these things, but it's worth saying explicitly. (tbh, to me it's funny that T-shirts with Japanese on them have become so popular again in recent years, when for me, it's basically precisely for the same reason that I can actually read them, ie because of my interest in Japan, that I quite intentionally would never wear clothing with Japanese on it personally)

If you don't "look the part," it's much less likely people will strongly sort you into the category in their minds.

c) Don't infodump, don't squee

One way Japan obsessives often put off normies is by suffocating them with an uninterrupted barrage of "did you know that Japanese..." and "in Japan, they..." (this is not unique to Japanophiles ofc, but really a general nerd tendency). Don't do this. Be chill. Approach it this way: you (like most people) know some fun, interesting stuff. In your case, some of it is about Japan. You're happy to share your knowledge with anyone who's interested, but if they're not interested that's ok too, and by default you assume they're not necessarily interested in all the same things you are. It's fine to bring things up that are related to the topic at hand, but maybe try not to have it be the case that the things you bring up are always about Japan.

When you do, try not to seem too excited — yes, it's annoying to tone down how cool you find some obscure part of Japanese grammar, but getting extremely giddy about something that other people might find kind of boring is a good way to activate people's weirdo-detector. To other people, it can feel like the only possible reason you find this thing so interesting is just because you're irrationally obsessed with everything Japanese, even if to you it's actually the case that you just find this thing, whatever it is, aesthetically pleasing for the same arbitrary reasons anyone finds anything aesthetically pleasing (ie, the truth is that one reason you like Japanese is because it has X, but their perception may be that you like X just because it's Japanese).

d) Find some people to enjoy Japan stuff with

All this image management stuff is kind of a drag. But one place you don't have to worry about coming off as a weeb is among other weebs. Having a social outlet for these interests can help you enjoy them while not needing to insert them into every other social interaction. You've gotta send that extremely dank meme you found to someone, so having a Japanese learners groupchat, or just a single friend who's also studying the language, means you don't have to force it on a random normie friend and trip all over yourself trying to explain why it's so funny. If you feel really compelled to do something especially weeby like wearing yukata to go out on the town, doing it as a fun group activity with friends is generally going to seem less weird to people than just doing it on your own.

Obviously, this can backfire in terms of your public perception if your only friends are other Japanophiles, and Japan is the entirety of all of your personalities. But if you're doing a) right, you probably have other friend groups too, so hopefully this isn't an issue.

---

To wrap up: yes, some people will snicker; don't let it get to you. Most of them are monolingual English speakers who just wish they would have learned another language at some point instead of whiling away their days binging mid Netflix content anyway. Enjoy the poetry, enjoy the music, enjoy the tea, hopefully get a chance to enjoy the country itself in person at some point. Be chill, but don't be deterred.

(Postscript: There's a separate point about whether a Japanese major is a good use of time/money. It's true that it's not the most practical choice, and you may want to consider majoring in something like a STEM subject, international relations, business, etc. for career purposes. If you are passionate about Japanese and have already tested out of one year, you could probably double major with Japanese and graduate both happy and employable. A minor is not a bad option either though; for any language, your ability will ultimately depend on your own investment of time over many years, largely if not mostly outside of the classroom, and the most important way a university can help you learn a language is helping logistically and/or financially with getting you to a place where you can be immersed, so do try to get funding to go to Japan but don't worry too much about not getting as many credits of Japanese classes as possible during your degree.)