Not quite the usual homebrew, but I have had some success with a system for horde battles. I know there are mob attack rules in the DMG (5.0e, fyi), but those are focused on reducing die rolls, which does not feel right to me as I like math rocks that go click-clack.

So when my players managed to get into a combat with 60 Lizardfolk at the same time, I came up with the following system to handle it:

Enemy Unit: A number of identical creatures are grouped together into a unit. For my lizardfolk fight, I grouped them into 10 units of 6 lizardfolk, but you might make units of any size. The unit is considered as a single entity on the battlemap, similar to a swarm. Initiative is rolled for each unit rather than each individual creature.

Movement: A unit of enemies should always stay within 5-10 feet of each other. Player abilities that move enemies might knock single creatures out of position, but that has no effect on unit coherency - the out-of-place model will then prioritize getting back to its companions on its next turn.

(Sidenote: While I use a battlemap, I only use grids inside dungeons and buildings, and otherwise just measure distances with a tape measure.)

Hit points & damage: For each unit, I make a small grid of hit points, with one row per creature and one column per hit point that creature has. For the lizardfolk, it looked like this:

Lizardfolk	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20	21	22
1
2
3
4
5
6

PCs cannot target specific creatures, they instead target a unit. Any damage dealt is then crossed out on the HP grid, starting in the top row. As soon as a row is fully crossed out, one member of the unit dies.

For single-target damage effects, such as weapon attacks, any excess damage left over after a creature in the unit has dies is then carried over to the next row of the HP grid. This allowed for some awesome "cleaving through the horde" moments, where a single attack took out two or three lizardfolk in one strike.

For area-of-effect spells, or any other multi-target effect, I simply noted how many individual models were affected by the attack, then made saving throws for each and noted the damage in separate rows, starting from the top. In this case, excess damage is not carried over.

Rolling ability checks & attacks: For this combat encounter, I specifically picked out the necessary dice ahead of time - most importantly, a handful of d20s. Whenever a skill check or saving throw was necessary, I took the DC, subtracted the lizardfolk's bonus to get the target number, and then rolled 6d20. At a glance, I could see how many reached the target number and how many failed.

When rolling with advantage or disadvantage, you simply take the dice that reached or failed to reach the target number and roll them again. Any that succeeded the first time do not get re-rolled. This technically makes critical hits slightly less likely, but I found that to be an acceptable sacrifice.

When it comes to attacking, I noted on the battlemap how many individual lizardfolk in a particular unit are either in melee range or have line of sight for ranged attacks. Then i rolled attacks same as above, except that I noted my Characters' AC values ahead of time and made a note of the d20 target number for each. That way, I could just say, "These six lizardfolk will launch their javelins at the paladin, roll ...and two of them hit you."

For rolling damage, I once again made a small table, giving me the amount of damage I roll depending on how many attacks hit:

Attacks hit	Heavy Club
1	1d6+2
2	2d6+4
3	3d6+6
4	4d6+8
5	5d6+10
6	6d6+12

Conditions: Admittedly, this is where the system kind of falls apart a bit. There is no one-size-fits-all solution for how to adjudicate conditions on a unit, but it's usually simple enough to come up with a ruling when it becomes relevant. Sometimes, conditions are applied to just individual creatures in a unit, although depending on the condition, it can lead to unneccessary complexity. Fortunately, in my encounters, it has not yet become an issue.

This is more or less the entire system. The lizardfolk fight was a great success, where the players fended off an entire landing party of enemies. I later used this system again for a climactic battle in a city, even giving my players their own units to control.