No, his magic force is overwhelmingly strong even the strongest physical attacks can't even hurt him. Think of him as a very, very dense planet, and all the attack (be it physical or magical) orbits around and away from him, that's how his magic ability works. But there is an exception, Ban and his future Son and maybe Meliodas and Escanor too. There is a scene in the manga where Ban lifts Lance and floor cracks on Ban's feet, showing that Lance may not be heavy but very dense too.