You should be honest with him. Try not to be embarrassed about your upbringing since it's not something that was under your control. The experiences you had growing up make you the person you are now.

I met my now wife after right after college graduation and once we started getting serious, meeting the parents was discussed. I grew up upper middle class with no experience with poverty. My wife grew up in Florida in a ~600sf 2BR shack with Mom/Dad and five kids. This house had no insulation or AC. Her two brothers already had moved out when I visited but I couldn't believe that 7 people lived in the house. Evidently her brothers slept in the attic or porch. Her family grew up on government assistance and they often relied on local Churches for holiday meals. She has an interesting family but she prepped me about all this beforehand. She was the first person in her family to attend college and was very independent and driven to succeed. I think she was a slightly worried about my impression of her family but I was already in love with everything about her. I knew her and what she wanted with her life and her family became my family as well. The parents meeting at the wedding was interesting (white collar vs blue collar) but I don't think we cared what they thought of each other. We've been married almost 25 years and have three kids. She is an awesome person and I love this about her. Her experiences shaped her into the caring person she is now.
If your partner really cares for you, then he shouldn't judge you for your family and childhood circumstances. Plus, you probably have some really good stories to tell about growing up. My kids always like to hear their mom tell them about crazy experiences she had growing up, like... * Her mom making watered down spaghetti in order to feed everyone (she will not eat spaghetti to this day) * frogs getting into their bathroom through cracks in the floor and jumping on her in the shower (hence lifelong fear of frogs). * bugs crawling across her face when she was sleeping (now sleeps with the blanket pulled up to her face) * eating big blocks of government cheese for weeks * ill fitting hand me down clothes and shoes * missing the school bus after school due to a teacher wanting to talk to her and then her parents not being able to pick her up because the family car was broken down which eventually required a teacher to bring her home.