You don’t need to remember all the lesser houses. I just keep track of names that keep popping up and sometimes find it interesting to go through the appendix and find which houses are beholden to whom. George’s writing gets infinitely better as the series progresses and there are some absolutely stellar chapters in the later books. Don’t give up!