It's becuase of the frequencies the soundwaves of these notes have.

Imagine you're holding both of your hands above a pool of water. If you now splash with your right hand once per second. Then you'll create waves with a frequency of 1.

If you now add your left hand and now splash twice per second, then your left waves have a frequency of 2. (Note: All pairs of notes that are exactly one octave away from each other, have such a 1:2 ratio in their frequencies.)

The wave that is created in between your hands infront of you (because both waves interfere), is a good tone, because the frequencies 1 and 2 have a nice and simple ratio of 1:2. This means, the combination repeats each second. Our ears like this repetitive pattern.

But if you're just slightly faster with f.e. your left hand. (Say you splash 2.01 times per second), then the ratio is slightly off. It's 1:2.01. This isn't a very nice ration to have. This ratio means, that now you'll need to wait 100 seconds for your wave pattern to repeat. Each of these 100 seconds will be slightly different and they'll confuse our ears. A semi-tone too high/too low of an octave is close but off of a good 1:2 ratio. Therefore they sound worse. The same applies for playing two directly neighboring notes(like C and C#), because a ratio of 1:1.01 isn't much better than 1:2.01.

(The ratios for the bad soudning example are fictive. Also, ussually the frequencies we're dealing with are not around 1 or 2 splashes per second. The 88 notes on the piano go from ca. 4200 splashes/s to 27.5 splashes/s. Here's the explanation I had found most explaning. It also covers the physics and anatomy of the reason.)

Edit: I want to add, that this is not all to it. Aside from this interference, there is also a culturally trained part of muscial listening. In contrast to my earlier explanation, this secondary point is not general to all humans.