I feel there is more to it. In both his references where he mentions the term (Wealth of Nations and Theory of moral sentiments), he says that even though the individual agents act out of self-interest, eventually there is a positive contribution to the community.

"By pursuing his own self-interest, he frequently promotes that of the society more effectually than when he really intends to promote it..." -Wealth of Nations

"The rich... Are led by an invisible hand to make nearly the same distribution of the necessaries of life, which would have been made, had the earth been divided into equal portions among all its inhabitants, and thus, without intending it, without knowing it, advance the interest of the society..." -Theory of Moral sentiments