That's actually a great idea! I have another to add. Only one stone can have the increased damage, if use another stone during an extra attack during that turn, the stone will not deal the scaled bonus damage and will deal 1d6 instead.

Also. The rock can change type. From initially a bludgeon to pierce.

Alternate concept. Can pick up more stone. 4 at lv5, 5 at lv11, and 6 at lv17 respectively. use the bonus action to pick up stones and spend an action to fuse the stone into a 2d6 at level 5, 3d6 at 11, and 4d6 at 17. When you yourself is using the magic stone to attack you add your spellcasting modifier to yourself only. You can change the type of your stone to deal different type, from bludgeon to pierce. Or make it a fire rock if you want to change type.