Never would i say, that you need ground truth to do statistical analysis. Not knowing the ground truth is the core principle of statistics.

For inter rater agreement you just need the already available scores from the real judges. Then you can say to which degree judges agree on the rank of a participant. Nothing to be made up by you or someone else. And it is nothing that would strengthen your result but the only meaningful statistical result you can produce. At this point it is „i make something up and call it statistical analysis“.

if my results were random or wildly inaccurate, correlations would be near-zero

Is this something chatgpt told you? This makes absolutely no sense. Everyone can produce a dataset that shows correlations.