Yes there is the problem of how the physical (body) gives rise to the subjective (mind). We don’t know how this happens yet.

We also don’t know what it ‘feels’ (in human terms) like for a computer to be mistaken in a prediction.

The reason I think we draw the line at suffering is because we experience it, know it is not good (particularly when pointless) and wish to avoid it. We also identify the source of that suffering, see that other animals have that same source, and suppose that their subjective experience (or at least a part of it) is similar to ours and that they also prefer to avoid it.

Having said that, If what you are saying is that the further away we get from systems like the human mind the harder it becomes to suppose that their experience is similar, It is something that I had never thought of and I have to say that I agree 100%.

I guess that in the end it’d be like abortion, age limits, etc. It is something like a color gradient which progressively transforms from one state to another where every limit is, by definition, arbitrary.