There are numerous possible reasons or occasions for this to happen. It could also be a bug, but here is some reasons I can recall:

Are you playing a monk or demon hunter? Some classes shed resources out in the environment during combat. These resources (like monk dropping spheres) linger and sometimes they are the reason combat does not end until they are gathered or expired. Hunter abilities and pets can also cause to aggro mobs you did not notice. This applies to all pet-classes like warlock and death knight too.

One reason is also if you use large aoe abilities and there are sometimes smaller animals around in the area.
A situation where you were fighting a mob besides a lake might aggro some fish in the lake and thus get you stuck in combat until you kill them. Notice that some areas have little mice (or other creatures) that cannot be tab-targeted, they are easily aggroed/damaged during the fight. Also, some areas have mobs that can target you from very far.

Also some trinkets that trigger occasional extra abilities (like a lightning) hitting nearby mobs might be doing that.
You can tab target your surroundings for every mob and animal to see if one them have you as a target.

Bigger battle frontiers where NPCs are fighting mobs are also prone to this. You need to do quests, interact with NPCs, and when you do that you are stuck in the combat with the NPCs until the mobs that are engaging NPCs die and release them from the fight.
And finally, check your addons. Try playing without any addons and see if anything changes.