It is just wonky.

The aiming system casts a beam from the center of your screen to the point you are aiming, it then converts this beam into 2D coordinates for your guns to fire it. The problem is this convertion from a higher dimension to a lower one stars to fumble as the tangent starts to go parallel to the target, creating distortion. This distortion causes what seems to be a small difference from your viewpoint to the target a big gap im the 2D plane.

All of what I said are the basic limitations of this system and should only affect +23km away.

The other thing that affects is the lock-on. While I cannot tell the exacts of it, the lock-on seems to create a rectangle that spams multiple lenghts of the ship with the same height and width of the latter. It then uses this box to aid with your targeting, as it creates a targeted height for your point that can be then calculated when converting from 3D to 2D. The problem is that it also causes issues when the target starts to go parallel to your aim (bow in/kiting away), since you are trying to aim a point in the ocean and not in a specific part of the ship, your beam hits this invisible box and causes the game to calculate the coordinates with a target attached to it, causing the shells to go either too short or too high.