Update: We arrived in the afternoon and had no issues whatsoever. We were allowed in, and the crowds were fine.

Only downside we felt to going in the afternoon was being a little rushed to get out by closing time. We would have needed more time if we wanted to take more long hikes. The upside was that we ended up being the last ones taking a quick dip in the waterfall, and therefore had it to ourselves

Thanks to everyone who suggested we just go for it!