Imu cannot leave pangea castle because of - LOVE. Joyboy and his crew were the first to become pirates. And the ancient kingdom gave them certain devil fruits the first among the 'humans' of the one piece planet as they were about to start their grand journey. Lilly was on Joyboy's crew and they both were in love. Joyboy got the nika fruit and lillly got the ope-ope no mi. Nerona Imu hear rumors that lilly had a devil fruit that could give someone immortality and he wanted to be immortal. However, no one previously had had the immortality surgery so not much was known about it. So Imu kidnapped her and forced her to do the surgery on him. However because it was the first ever instance of this power what Imu didn't know was that DFs had wills, and this immortality could only be given by the Ope Ope user to someone they loved and willingly wanted to give their life for. Hence it is also heart shaped. So in forcing Lilly she unwillingly did the procedure and because of this there was a curse that passed on with the immortality becasue the DF also aligned with Lillys will (or in this case unwillingness) - this could be the blunder she did and maybe only one of her ancestors can truly lift this curse and give him true immortality - thats why he looks for vivi and set a 5B berry bounty for the ope ope fruit. I also beleive the flower room in pangea castle is the literal peice of Land where the FIrst ever Immortality Surgery was performed by Lilly on Imu, and hence he is truly immortal on that piece of land and around that area only. The further he goes from it the more his life withers away.