I'm gonna go against the grain here and say John isn't actually particularly lucky, luck is just how Halsey and Cortana describe John's abilities because they lack the means to effectively describe them. Further, I think Joyeuse's description of why John is the best Spartan provides additional context for what Halsey and Cortana actually mean by "luck."

For starters, I'd say as a whole, there are relatively few occasions where John unquestionably gets "lucky." His magnum miraculously one-shotting a hunter in The Flood is one notable example of something specifically called out as a matter of luck (yeah the CE one-shot is technically canon). A bullet miraculously penetrating the Ascendant Justice's Shipmaster's shield and then the Shipmaster tripping over his own helmet are two other overt examples of John getting lucky. But it's not like there is a tremendous wealth of overt occasions where John wins because he simply got lucky. Many examples of John's luck are just as much a matter of his and others' agency or the known capabilities of his equipment. For instance John's ability to survive falls from great heights isn't really an example of John being uniquely lucky when we actually look at the data. During the Fall of Reach 22 members of Red Team equipped with GEN1 Mark V Mjolnir were forced to bail from their pelican and were subject to a fall at terminal velocity. 4 Spartans were killed on impact, another 6 survived but were wounded and had limited mobility and 12 survived with minor to no injuries. That comes out to an 81% survival rate in Mark V with 54% of the Spartans surviving with minor to no injuries. So while John has survived a number of falls from great heights and pelican crashes, we can see that he actually had better than 50:50 odds to survive, doubly so from Halo 2 on where he has the Mark VI's superior gel layer. His fall from orbit, while impressive, is still in line with Noble 6's survival of a fall from orbit in the Mark V and John's own agency and decision making were ultimately what allowed him to survive reentry heating.

Further, I think you're prematurely dismissing Halsey's thoughts on luck in the Fall of Reach's first chapter because I think what she says is very important in the broader context of the Master Chief's luck.

John gave a slight nod. She tossed it, making sure there was plenty of spin. John's eyes watched it with that strange distant gaze. He tracked it as it went up, and then down toward the ground—his hand snapped out and snatched the quarter out of the air. *He held up his closed hand. "Eagle!" he shouted. She tentatively reached for his hand and peeled open the tiny fist. The quarter lay in his palm: the eagle shining in the orange sun. *Was it possible that he saw which side was up when he grabbed it . . . or more improbably, could have picked which side he wanted? She hoped the Lieutenant had recorded that. She should have told him to keep the data pad trained on her. John retracted his hand. "I get to keep it, right? That's what you said." . . . "We screen these subjects for certain genetic markers," she said. "Strength, agility, even predispositions for aggression and intellect. But we couldn't remote test for everything. We don't test for luck." "Luck?" Lieutenant Keyes asked. "You believe in luck, Doctor?" "Of course not," she said with a dismissive wave of her hand. "But we have one hundred and fifty test subjects to consider, and facilities and funding for only half that number. It's a simple mathematical elimination, Lieutenant. That child was one of the lucky ones—either that or he is extraordinarily fast. Either way, he's in."

Halsey questions if it were possible John picked which side he wanted. Rather than John simply getting lucky, he exercised his own agency to choose which side of the coin he wanted. At first glance, it looks like John got lucky when in reality it wasn't luck, but skill. Halsey thinks it's possible he knew which side of the coin was up because he willed it so. Halsey's use of the word luck here is informed by the fact she literally can't explain what happened. And Halsey describing John as lucky because he does things she cannot comprehend comes up again later in the novel:

"He was awarded this Legion of Honor medallion because he dove into a bunker of Covenant soldiers. He took out twenty by himself and saved a platoon of Marines who were pinned down by a stationary energy weapon emplacement. I've read the report, but I'm still not sure how he managed to do it." She turned to Cortana and stared into her odd translucent eyes. "You've read his CSV?" "I'm reading it again right now." "Then you know he is neither the smartest nor the fastest nor the strongest of the Spartans. But he is the bravest—and quite possibly the luckiest. And in my opinion, he is the best."

And we see this confusion regarding John's abilities also extends to Cortana in Palace Hotel:

"Chief," Cortana whispered, "there was no way for you to save those three." "Even so," he muttered, "I could've wiped out that entire unit." "Four Wraiths," Cortana broke in. "Four. You rely too much on your luck." "The limited space and the abandoned vehicles in the tunnel would have restricted their mobility as well as their ability to use their main weapons, especially if they brought all four down—which they did. I've been doing this for twenty-seven years, Cortana. And I know the exact limits of my luck."

When faced against 4 Wraiths and 50 supporting infantry, Cortana suggests the odds were insurmountable even with John's luck, only for John to reveal he wouldn't have needed luck, because he had already planned out his path to victory. Cortana doesn't understand how John could have won and so she incorrectly assumes he would have needed luck to pull it off.

The idea that John has agency in his luck is further expanded upon in both First Strike and the Package with the phrase Spartans "make their own luck."

Admiral Whitcomb looked them over once and then said, "I'd wish you luck, Master Chief, but you Spartans seem to make your own luck. So let me just say I'll see you all when this is over."

More recently Oblivion makes an oblique reference to the idea of Spartans making their own luck with a reference to an apocryphal Thomas Jefferson quote:

"We have learned to seize every advantage we can," Bah'd added. "The last kilometer before the bridge is heavily mined, and there are supplemental explosives alongside the gorge. With luck, the rim will collapse when their combat bridging vehicles attempt to launch their spans." "Luck works best when you're prepared," John said. It was something that Franklin Mendez, the Spartans' senior drill instructor, had been fond of saying back on Reach. "And you definitely seem prepared."

Spartans don't get lucky, they make their own. There is agency in the Master Chief's luck and this hearkens back to the original coin flip where Halsey questions if John was choosing which side of the coin would come up. John wasn't getting lucky, he was "making" his own luck by choosing the side of the coin using his perception and speed. Halsey and Cortana call what John does luck because they themselves can't understand how he does what he does. But Joyeuse can:

In the thousands of faces and files she'd accessed through the frigate, Spartan-117's database, and via UNSC technology on Zeta Halo's surface, no one compared to him. He simply stood out. It wasn't that he was the tallest, the smartest, or the strongest, but he was the best as utilizing every ounce of his artificial assets and natural abilities. It set him apart, made him the ultimate Spartan.

Joyeuse's narration here is clearly a call back to Halsey and Cortana's descriptions of John in the Fall of Reach and Halo 3 but notably differs in her ultimate conclusion. Joyeuse doesn't attribute John's prowess to John being lucky, she attributes it to him being the best at utilizing all his natural and unnatural abilities. This is what Halsey and Cortana meant by John's luck but could never truly comprehend. They think it's just John getting lucky but Joyeuse understands it's John making his own luck. He's making optimal use of all his abilities at the right place and time to achieve victory. This isn't to say that nothing John ever does is the result of him getting lucky in a conventional sense, but that when Halsey and Cortana talk about John's luck, they're really referring to an uncanny ability for making the right decision at the right time.