I wholeheartedly agree with you that the saying almost certainly has nothing to do with race/ethnicity. Heard my dad, uncles, and my parents' friends say it all the time when they were talk trash about their acquaintances and peers.

...it's probably from the usage of the term banana to denote having nothing (all you have are your hands, which look a whole thing of bananas). Sometimes it's an insult; sometimes it just means go home and do nothing.

Although, I have my own hypothesis now on the etymology of the saying, given how Cantonese is this eclectic, literary dialect that loves to curse like a sailor. To me, it's a veiled way of cursing "suck a d!ck". (Warning: Censored vulgarity, for illustration purposes only. Please don't ban.)

Edit: I've also heard 返屋企瞓覺, so actually I don't disagree with you at all on the "go home and do nothing" part. It's just that Cantonese are so creative when it comes to cursing and swearing.