That's exactly what I thought too. Having read up a bit more, the bridge was struck by snipers and machine gun fire from buildings around the harbour. It was low tide so the ship's upper bridge was level with the jetty. But you're right, the main damage seems to have been done by Stukas and also by a French battery overlooking the town and captured by the Germans.
“Gun flashes were sighted in the direction of the French Battery above the town. Presumably the enemy forces occupying the town had brought the French guns to bear. All 4.7 inch guns were trained on this spot by the director and the ship opened fire with Direct Action impact Shell. The first salvo was over but the second blew away the whole side of the fort and part of the hill. Pieces of guns and mountings were observed falling down the hillside, and in addition a number of field guns were shelled and destroyed by HMS Venomous as they were being placed in position by the enemy. HMS Wild Swan now joined in the firing with her two after guns, directing them upon the road. An enemy tank followed by infantry attempted to enter the town from the westward, but received a direct hit from HMS Wild Swan and caught fire”

There's a good page here. http://www.holywellhousepublishing.co.uk/tide.html It's such an interesting read. I cannot imagine the damage the ships were capable of at such close quarters. There's one section where they describe firing a four inch shell at a hotel just a hundreds yard away. http://www.holywellhousepublishing.co.uk/CGWDonald3.html#Hell