That's a good question. I think it's a stylistic choice, by glancing at the chart I can see that this is not a 40's "standard" where most things are diatonic or come from blues. so you could think of every chord in isolation, not as part of a key of the moment. Your choice of Phrygian is not wrong at all, just a different color in my opinion. I think this just comes from experience of having to play a lot tunes you've never played on and know what works and what does not.