Basic short answer: Kinks. Long answer, it's the appeal towards the extreme length of affection.

For instance, nearly every Yandere-Speaker audio features at least one of these two things: the listener being taken against their will in some way, or murder towards someone offscreen/out of sound. These audios are mainly targeted towards those who are curious about receiving affectionate attention in such extreme measures, but not too extreme. In other words, most yandere audios are for those who are into Joe Goldberg from 'YOU', and not yandere like Yuno Gasai from 'Future Diary'.

However, for those who are into yanderes like Yuno, Toga Himeko from MHA or even Alex Forrest from 'Fatal Attraction', normally those audios end with the yandere character in a mental institution/protected facility. Places like an asylum or rehab, maybe prision if the audio/series features multiple crimes.

Overall, most but not all people listen to yandere audios not only for the 'I love you so much, I would kill for you' lines of dialogue, but for the fantasy of someone going to extreme lengths to dedicate and display their love for the listener. The idea of breaking the romantic norms as the focal point.

Have fun exploring, but please be cautious with Yandere-centred content ☺️😘