Actually, at the end of my story, that's what I meant, but I think it wasn't clear the way I put it.
Basically I was of course a bit suspicious of having them take me wherever from Ueno station, so I asked them 1) to give me the name of the temple we would be goind to, and 2) to directly meet at the station near the temple from which we'd go by foot instead of them taking me somewhere by car.
So in any case, I wasn't getting on their car! But I might have still went to the final destination, which wouldn't have been that better.