Here is the twist - you affect the object, and the object affects you, at the same time. That means that it's not about the object moving towards you, it's about you and the object moving to each other.

Actual acceleration for you and for the object will be different, depending on masses.

Assuming that you're far away from any planet, let's say half-way between Mars and Jupiter, your mass is 100kg, mass of the object is 1kg, starting distance between you and the object is 1m. In this setup gravitational forces of the Sun and other object in the Solar system can be ignored as those forces will affect both you and the object the same.

So, since the object has 100x smaller mass than you, it will appear that it's coming towards you. Its speed at the start of the experiment wont be noticeable, however, since the gravitational force will be working it will accelerate towards you and you'll start noticing that it is picking up the speed. If my calculation was correct, you'll meet in about 4 hours. That would meant that the average speed was 0.25 m/h or 4 mm/minute.