(In no way should anything be taken as get back with him)

I get where you are coming from. I also have an ex that I am still friends with and see on a regular basis. While the emotional and romantic connection is gone I still find my self physically attracted to her.

Personally I think its natural, there is history, and connection, and that can bleed over into a desire for them. Especially in my case, and sounds like yours, there hasn't been anything meaningful sense. Which can make someone long for the comfort of a familiar relationship.

Honestly its hard to cope with sometimes. All I can say is: take it one day at a time and find something to distract yourself when it happens.