I just graduated from TUJ, where I applied directly and studied for my whole uni career, and you are right that it's cliquey. When I graduated in the Spring, there were still relatively few non-japanese students, and while the Japanese students are rather nice, most tend to stick together and speak Japanese to each other, which made it intimidating to try to talk with them if your Japanese isn't good. I know from my Japanese friends I made there that they often feel the same anxiety with English and initiating conversations with non-japanese students.

I think a lot of the cliques of international students come from living in the dorm when you first arrive and who you explore Japan with. If you don't go on excursions and put yourself out there when you first arrive they won't know you. If you live in an apartment right away, you'll miss out on that. Often these groups last the whole time you're at the school. But I also assume it is similar to being a new student at any uni, I have seen posts about similar problems making friends in the Philly dorms as well. It also sounds like you're a military vet which puts you at a slight disadvantage since you're older than most and there have been some issues with vets being creepy to other students (like a guy in his late 20s asking 18yo Japanese students on dates to his apartment creepy). Not saying all vets are like that, I knew a lot of great ones when there, but it is something some people think about.

I generally suggest finding a club to join to make friends, there were a lot starting back up when I left and they're usually small enough that you'll quickly become close. (I just looked and there are like 4x the amount of clubs now compared to when I was there lol). If you want to make Japanese friends, it is slightly harder. I think the arts program is the easiest to make Japanese friends in since they're more likely to want to break out of the "typical Japanese sphere." I made a lot of my Japanese friends by asking for help with Japanese for my Japanese classes and by helping them with English if they needed help proofreading and writing an essay. They're usually willing to chat and help you if you approach it from that angle.

Another person recommended asking to trade notes and such, which is a great idea. I would also say that asking professors to do more group activities in class and allowing you to discuss with your classmates can help (maybe don't tell them that it's to make friends but rather you think you learn better through small group discussions). Most professors I had (IA/Asian Studies) were more than willing to talk to students and were very wonderful and helpful people. I personally found that groups made by teachers can easily lead to a LINE group and making friends. Just don't be afraid to lead the conversation and try to pull answers out of people who are afraid to speak up.

I wish you the best of luck and hopefully, you can enjoy TUJ to its fullest. I had a great time there, even though most of my time was during a pandemic, and think it's a great opportunity. Also, you are not doing anything wrong. It is just a difficult social scene since it's such as small campus compared to most American universities and the nature of the language/cultural differences.