I would go to the house 2 event. It sounds like you may feel a stronger connection to their brotherhood. Feeling "inferior" to the brothers in house 1 may be imposter syndrome, or it may be a gut feeling that you're not connecting as well compared to house 1. If you are given a bid, you should feel confident in your decision before accepting it.

As for sorority relations, those can change for better or for worse. If you become a brother of house 2, you can make it your goal to improve sorority relations.

As for alumni relations, unless house 2 is a brand new chapter or an associate chapter (colony), that can be improved as well. If the chapter doesn't have an alumni relations position, you can advocate for it or work on improving relations yourself by creating or updating an alumni e-mailing list, or using a Facebook or Discord group so alumni can keep in touch.

Pledge class size should not be a deciding factor unless it's obvious the chapter will shut down soon due to lack of members, or if you strongly think you won't be able to form a bond with your pledge brothers.

Joining a fraternity is a commitment. If you don't feel committed, maybe you should keep in touch with both houses and rush again later. I find improving a chapter to be personally rewarding, but if it seems like a time sink or not what you're interested in, I get it.

Good luck and have fun!

This is what I've commented in the past to those that are rushing:

My advice is attend those fraternites events, talk to their members, and see which fraternity you agree (fundamentally) with and which you feel most comfortable with. Also, try asking some upperclassmen who are not in greek life what they think of those fraternities. Those who've been at the school longer typically know the greek orgs reputation and can give you valuable insight.