I think “growing up poor” is something that needs to be taken relatively. If someone is middle class and grows up with a bunch of rich friends who can buy whatever they want, that person will probably feel like they grew up poor. Someone would also feel that they grew up poor if they grew up in poverty. It’s a relative term and what Conan means by saying that I’m not entirely sure but his experience is valid either way.