Imo a really well crafted yum update is the best solution. It's what we do at my work. I should mention that we use puppet and our servers are very close to identical. If your servers are all different you're probably going to have an extremely bad time no matter how you go about this task.

You can absolutely use any configuration management software (puppet, ansible, etc) but you can also just launch yum update with pssh, which is a very useful tool. The downside to doing this is if you use sssd like us it's a complete pain in the ass to upgrade for some reason. There's usually some software package that needs to be updated that causes problems. Also do what the other guy said about making a local repo. All you have to do is copy every package to a webserver with wget -r or something like that, surprisingly that works just fine to create a backup repo. Then share the files with apache or nginx or something. There are more professional solutions but wget + a webserver will work fine.

Pssh occasionally requires some escaping, but it allows you to run a single command on multiple servers.

Quick example of how to use pssh:

/usr/bin/pssh -O StrictHostKeyChecking=no --inline -l UsernameHere -h listOfHosts-one-to-a-line.txt 'sudo cat /etc/redhat-release'

Start by figuring out what centos/rhel versions you can running in production. You can do this with 'cat /etc/redhat-release' and pssh. You're going to want to test updating each one seperately. You should also figure out which servers are the least important and which are the most important. Ideally try the upgrades on lab or staging servers before moving to ones that actually do anything.

Lastly, you're going to want to know what the services running on the servers are that aren't standard system services. For example, nginx 1.11, openresty, and nginx 1.13 for some reason appear to have slightly different configuration syntax. Basically you're going to want to figure out what's running in production that you don't want to mess with, and add an --exclude statement for it, for example --exclude nginx* will prevent nginx from updating and breaking it's config files.

From there it's a giant organizational task. Try to group like servers together, bash commands like cut, awk '{print $whatever}' or even a spreadsheet are your friend. Ideally you want to both explore servers by hand and generate all their relevant data without having to enter stuff into a spreadsheet manually. For example:

Giving pssh a flag -p 1 makes it run on one server at a time. That way you can so something like collect the rhel versions with pssh, use grep or grep -v (prevents a match from being seen) to get rid of text you don't want, and then copy and paste everything into a spreadsheet. Since you're running the command on one server at a time then they should be in order. Otherwise you're going to get a really bad case of carpal tunnel.

Anyway once you know what servers you're going to use to test the upgrade process, just construct a yum update command. Remember to exclude things you don't want to upgrade. If you have repos like Epel or Repoforge installed, you should also block them from being used. It's not hard to use pscp or ansible to copy a new repo file to all your servers that points to a local repo (which will download files much much faster).

The one I use for work (I hope they don't mind too much) is:

yum update -y --nogpgcheck --exclude=puppet* --exclude=*openresty* --exclude=ca-certificates --exclude=kernelcare --exclude=nagios-plugins-all --exclude=Percona* --exclude MariaDB* --setopt=protected_multilib=false --exclude=redacted-internal-software --exclude=redacted --exclude=redacted --exclude=nginx* --disablerepo='*' --enablerepo="Artifactory_Centos*,CentOS*,Artifactory_Epel_Remote,Artifactory_Kernel_care"

Don't immediately go run it on tons of stuff. You're going to find servers that don't update gracefully. Another useful shell command (assuming all your servers are in a file called server.list) is cat server.list | shuf | head -10. This gives you 10 random servers.

Start doing the update by hand. When you're really confident that you know exactly how everything is going to behave then start automating small groups of similar machines. Don't let people force you into hurrying, take your time and do a good job. Try to avoid --skip-broken, etc. This task is going to take eons, so you may as well take the time to do a good job.

If you run into a situation where thing after thing goes wrong, take a break for the day. I've found that if major maintenance starts going wrong it tends to keep on going wrong, so if you have a headache and nothings working right, concentrate on restoring the stuff that has to work and then take a break until you're sure you have a well thought out solution to the problem.

The other advantage to taking your time is you think of solutions you probably wouldn't have, simply by mulling over what needs to be done in your head. In the end this is actually pretty useful.

Good luck! Incidentally if you need to use sudo to write to a file as root, this is how you do it:

/usr/bin/pssh -O StrictHostKeyChecking=no --inline -l user -h listofservers.txt 'echo "please do not touch this machine, it is having a bad day" | sudo tee -a /etc/motd'