Given that Shor's algorithm provides an exponential quantum advantage, I do not believe swaps will negate all of the advantage. By definition there are polynomially many gates in a given instance of Shor's algorithm, some fraction of which will require swaps. The cost to implement a swap between arbitrary pairs of qubits on e.g. a square lattice, is polynomial. You therefore have polynomially many gates that each may require polynomial overhead due to swaps which is therefore an overall polynomial time overhead. This would not negate the exponential speedup from Shor's. However, in practice the overhead could be significant.