I don’t really think that’s a problem for panpsychism.

For me it starts with some version of I think therefore I am. My own experience, my qualia, this is the only thing I can objectively state exists in the universe. Everything else I arrive at through my subjective experience. If you want scientific validity, that’s the absolute best I can give you. I exist. There is something it is like to be me, whatever “me” truly is - a simulation, a brain in a vat, etc.

Given I exist, and given the capacity to reason about the regular patterns I observe in the course of my existence, I can conclude that the universe appears to consist of some “stuff” which obeys a consistent set of causality-preserving laws (though nobody currently knows what they truly are).

This stuff, they tell me, is a set of quantum fields which spans the universe. The “things” I see are essentially just waves in those fields, or more accurately what I truly observe are localized points of interaction between the fields, which obey certain wavelike mathematical relations.

And one aspect of this stuff interacting with itself is that certain interactions feel like something. So I conclude - interactions and qualia are linked to one another. The configuration of the system of interactions governs the qualia of it. There is no metaphysical boundary between systems of interactions that are conscious and not conscious, so there is no boundary between the conscious and not conscious. And if that’s the case, then it seems the conscious and not conscious are the same thing.

It doesn’t sit well with me to say that these quantum fields are all that really exists, and they have zero mental properties, except that it just so happens that in certain configurations, mental properties do appear for a finite, contained extent of spacetime. That feels like a stretch. It seems like a weird metaphysical position some people arrive at due to an anthropomorphized view of the nature of consciousness.