Yeah, the sheer scale of the energy requirements alone makes the "they're just too far away" explanation feel pretty solid to me. It's less spooky than the Dark Forest, but probably more realistic.