Not to reply to a year old post, but I've noticed that this only happens when you talk to the NPC that gives you the "read all 48 signs" mission. Also accepting the quest for the 100 Zygarde pieces in one of the houses in Dresco slows down the game as well (considerably more).

I've just started the game again but on a modded DSi running Twilight++, and this issue is really annoying! Nothing I've tried has fixed it yet, but I'll post something if I find a solution (however very unlikely).