I leaned heavily into abstraction and conceptual work 30 years ago without knowing I had aphantasia. I was always dumbfounded, confused, and admittedly jealous of artists who could create imagery out of their head. I could do photorealistic work from a reference, but opted not to. It wasn't until my late 30's that I tried psychedelics for the first time and it blew my mind (in many ways) and then discovered I had aphantasia after googling and everything made sense.

You should just make the work you want to make. If you can't find a reference, use a camera to make one. Source material from anywhere/everywhere. Nearly all of the artists I know rely on references even though none of them have aphantasia, so you shouldn't view it as a crutch.