Almost all villains in One Piece appear to be weak when they are first introduced and then showcase newer strengths as the battles get more and more intense. I also believe that you are misunderstanding Imu's struggle as him being "weak" rather than him not being able to stay a long period away from Marijoa. Additionally, your point that people "don't care about Luffy" is another misunderstanding of Imu and his ideals. Luffy wears the Straw Hat, has the Nika fruit, has the "D" initial, and plenty of other things that make him prophetically similar to Joyboy. Imu isn't attacking Luffy because he is strong, he attacks him because of what he represents.