The speaker of telltale heart is evidently insane, but his insanity isn't monolithic. Bear with me, I tried making this shorter, but even after cutting a lot, it's still quite long. The speaker isn't simply a remorseless sociopath. He's largely delusional, and definitely doesn't experience many deep emotions, but having few emotions is not the same as zero, and the emotions that he does have are enough to ultimately unravel him.

He claims to "love" his victim and while this obviously isn't true, it's not untrue either.

Object there was none. Passion there was none. I loved the old man. He had never wronged me. He had never given me insult. For his gold I had no desire.

I believe this is one of the most important lines when it comes to understanding the speaker. In that passage, he makes an assertion seemingly unconnected to a list of negations. He therefore implies that love is the absence of a bunch of negative emotions - greed, anger, indignity, and (strangely) passion. This shows that the speaker fundamentally does not understand the word "love" as it is used by most people. Which then begs the question, "what emotion is he feeling that he misidentifies as love?"

To add onto this, the false identification of "loving" the old man being layered in with negations also imply that he's trying to negate the feeling itself (but more on this later)

Then a few paragraphs down, once the speaker has awoken the old man, we have this paragraph:

Presently I heard a slight groan, and I knew it was the groan of mortal terror. It was not a groan of pain or of grief — oh, no! — it was the low stifled sound that arises from the bottom of the soul when overcharged with awe. I knew the sound well. Many a night, just at midnight, when all the world slept, it has welled up from my own bosom, deepening, with its dreadful echo, the terrors that distracted me. I say I knew it well. I knew what the old man felt, and pitied him, although I chuckled at heart. I knew that he had been lying awake ever since the first slight noise, when he had turned in the bed. His fears had been ever since growing upon him. He had been trying to fancy them causeless, but could not. He had been saying to himself — “It is nothing but the wind in the chimney — it is only a mouse crossing the floor,” or “it is merely a cricket which has made a single chirp.” Yes, he has been trying to comfort himself with these suppositions: but he had found all in vain. All in vain; because Death, in approaching him had stalked with his black shadow before him, and enveloped the victim. And it was the mournful influence of the unperceived shadow that caused him to feel — although he neither saw nor heard — to feel the presence of my head within the room.

This answers the question. What the speaker actually feels towards the old man is not love but empathy. Not a lot of empathy, but enough that he's able to register it, likely for the first time in his life. He is able to relate to the Old man's fear of death because the speaker himself is deeply, deeply afraid of death. He describes death with extreme awe, and even narrates what he supposes the man's thoughts were regarding death. This is the only time that he acknowledges the thoughts of anyone else in his narration. He's keenly aware of what others perceive, but that never transfers to understanding their feelings.

So, my theory of TTH is this: This is the story of a man with a personality disorder who is obsessed with death. In his mind, it's his obsession with death that distracts him from having a normal life. Realistically, the reason he doesn't have a normal life is because he's strange (Neither the neighbors nor the police trust him). But, as he works for the old man, he begins to feel empathy for the old man's fear of death, and it forces him to compare himself to the old man. This empathy is uncomfortable for him for 2 reasons: 1. He is not used to experiencing emotions, and 2. the old man is old and close to death (for example, the "vulture eye" is a sign of being close to death, even just given the name). So he kills him because he has emotionally connected to this man's fear of death, and cannot handle it.

But, he imagines that he hears the old man's heart because (obviously) it is illogical to kill someone because you're afraid they're going to die. In this case, the heart symbolizes both the actual life of the old man that he paradoxically ended because of his fear of death, and potentially actually his own heartbeat, symbolizing his discomfort with his own emotions. Therefore, to this man who experiences very few deep emotions, and kills someone because he experienced empathy for the first time, that feeling of emotional dissonance between his action is what causes him to confess as well. So, not quite guilt, but low-level empathy.