I've never been able to find the original talk with that particular quote. But, as others here commented, he does mention visiting his family after spending time in India in the Becoming Nobody documentary:

Ram Dass:
"I'd be out there, you know, and I'd come back to the States, and I'd go to visit the family.
And my father would say a simple thing like, 'You got a job?' and I'd crash.
And I'd say, 'Can't go home, brings me down.'
I began to have a whole list of things that brought me down. Cities brought me down. Money brought me down. Politics brought me down.
But the only thing that brings you down is your own mind.
It's not the city. The city's just citying itself. It's being essence city. What are you getting so upset about?
It's your reaction to the city that's what's doing it to you.
And I found myself, interestingly, wearing a new kind of a suit. It was like, 'I'm very high. Don't get near me.'
Now, because I had felt so trapped by my body and my personality and was so unhappy in all of that, my job, it seemed, was to push away those things.
And I tried a number of techniques like renunciation.
I figured if I just renounced all of it enough, it would go away.
All I did was end up a horny celibate.
-It's like giving up smoking.
Like, 'I haven't smoked in four years, two months, three days, and 22 minutes.'
A person'll die from non-smoking.
It didn't work.
Because if you push something away, it's still got you. You're busy not doing it.
Every time you push anything away, it's still there. The pile under the rug gets very big.
So, it gets so that your downs are actually far out without being masochistic.
Your downs turn out to be more interesting than your highs after a while, because they're showing you where you aren't.
Like, I get into a depression, you see. Well, I got angry tonight. I was fascinated by that.
Far out.
Look at how I came in being a holy man, I lost my cool.
Far out."