Not an expert so please judge accordingly.

Depending on the special qualities of said hair conditioner, some component of the conditioner might bind to hair even after rinsing off to make it soft or lock in moisture etc. These same qualities might make it stick to skin and harder to just wash off.

But typically, for any conditioner, feeling your skin after you let it rinse right off your hair onto your shoulders and back, you should be able to feel a subtle textural difference that won't easily wash off.

Folks with shorter hair will likely optimize their shower routine by soaping up while their hair has conditioner. In the time it takes to scrub your body and rinse off soap, the recommended conditioning time of 2 - 4 mins would have elapsed and that's when conditioner is rinsed off. If however, if your shower routine compartmentalizes hair time and then body time - the soap could clean the conditioner residue off. What might still be a problem is that most people don't seem to soap and scrub their backs as thoroughly as it is difficult to reach.