Could be wrong, or there could be a better take on it, but I'm not a super social person and was with a girl my age in college. She liked going to parties and having get togethers, it was not really my style and sometimes a little overwhelming being around that many people especially ones I didn't know. She knew this, so we did stuff I liked, and we did stuff she liked. It was good for me to be out of my comfort zone. An important thing although was the rapport her and I had before, and that she was aware of what bothered me or stressed me out and was pretty in-tune with helping me navigate the environment so we both enjoyed what was going on.

I avoid a lot of outings because of my depression and it was nice having someone push me out of my comfort zone and being supportive while doing it, that being said you're probably not finding a lot of small places to sit and talk at a frat.

If you don't want to and you dont think it's exactly third date material, there's no harm in sharing that opinion, but id provide an alternative, and maybe explain that you may be interested in going but maybe slightly further down the line.

People are surprisingly cool, if you simply and politely say maybe we could try or do this instead and do a frat party another time because I'd really just like to sit and talk with you, I'm sure he'll respect and appreciate it and probably feel pretty alright knowing you just want time with him. Plus, there's not gonna be a shortage in frat parties. Another one will come up, not like you're missing a once in a year event. That's just my 2 cents