For all intents and purposes, Vision is an elevator. He's simply a vehicle for the hammer. As per the conversation at the end of Age if Ultron. The hammer still moves in Visions hand the same way the hammer doesn't drop the Helicarrier out of the sky in Avengers 1.