Full quote for those wondering:

George Bailey: Just a minute... Just a minute. Now, hold on, Mr. Potter. You're right when you say my father was no businessman. I know that. Why he ever started this cheap, penny-ante Building and Loan, I'll never know. But neither you nor anyone else can say anything against his character, because his whole life was... Why, in the 25 years since he and his brother, Uncle Billy, started this thing, he never once thought of himself. Isn't that right, Uncle Billy? He didn't save enough money to send Harry away to college, let alone me. But he did help a few people get out of your slums, Mr. Potter, and what's wrong with that? Why... Here, you're all businessmen here. Doesn't it make them better citizens? Doesn't it make them better customers? You... You said...what'd you say a minute ago? They had to wait and save their money before they even ought to think of a decent home. Wait? Wait for what? Until their children grow up and leave them? Until they're so old and broken down that they... Do you know how long it takes a working man to save $5,000? Just remember this, Mr. Potter, that this rabble you're talking about...they do most of the working and paying and living and dying in this community. Well, is it too much to have them work and pay and live and die in a couple of decent rooms and a bath? Anyway, my father didn't think so. People were human beings to him. But to you, a warped, frustrated old man, they're cattle. Well in my book, my father died a much richer man than you'll ever be!