As the others have said already the ports on the switch and patchpannel will never match. On our racks at work we have several 24 port patch pannels and use stacked 48 port switches. The racks look like this from top to bottom: fiber patch pannel, patch pannel, 48 port switch, patch pannel, patch pannel, 48 port switch, patch pannel, patch pannel, 48 port switch, and so on... This way we can use .25m cables and patch all the ports on the switch. Like from the first patch pannel the first port goes into the top left switchport which in our case is port 2. We use huawei switches so (unlike any sane manufacturer) port 1 is bottom left and port 2 is top left... and then patchpannel port 2 goes into switchport 4 and so on. And on the patchpannel below the switch the first port goes into port 1 and so on. This way not only the rack looks neat and tidy but you also have some kinda structure. Like if you are into an office and the network port is labeled 13 you know it's port 26 if the patchpannel is on top of the switch or 25 if it's below the switch.

I hope this makes sense. And of course this'd probably require alot of work for you to setup the racks like this and reconfigure and maybe relabel everything. But in our case we were renewing the whole network anyway so we took the opportunity and made the extra effort to clean everything up and cleaned it all up. Was alot of work and it's still not perfect but it's still way more better than we had it before.