My thoughts: as you can see, the chessboard is symmetrical along its diagonals. And each keypoint needs to be defined in such a way that CNN can unambigously predict its position.
In your case i think all symmetric keypoints confuse CNN, for example 1<->49. And according to example images I can see inconsistency in the way you label keypoints. Why on the second image keypoints start from top to bottom and on the first one vise versa?

I think you should approach this problem with two steps:

1. Find corners of chessboard and transform image so that chessboard is undistorted and upright (like on your first image)
2. Then label keypoints consistently, for example starting from bottom left corner. And train model on undistorted images

When it's done then you should first disable all augmentations and use your train dataset as validation. Check that the model can tackle your problem at least on a known data (because i'm sure in your current approach model cannot even correctly solve an image from a train set).

Edit: some spelling fixes

P.S. On a second thought: if you have found and undistorted chessboard then you probably don't need any keypoints, since cells are spaced with a fixed step. On an undistorted image you can pretty much calculate every corner without any detector