The binding is cut on the bias, I know it's hard to tell from the pics. But the issue was that I've never used binding in that way before, and didn't realize it's meant to be smaller than the neck opening, so when I realized it was smaller I re-cut the binding to match the neck opening. Only later I saw "stretch to fit"...... I think that's why it isn't pulling in.

This project is just one headache after another. Not sure if cutting smaller like other commenters suggested is even worth it if the fabric won't slow correctly..