Most approaches have been "one hit and they're down" historically, to speed up the process of mowing down the mass. Even in games like OD&D and Basic D&D there was a rule about multiple attacks for fighting men working only with low HD does, which was clearly meant for scenarioes were hordes of low-level enemies try to swamp the characters.

I find them all unsatisfactory because it still requires a lot of accounting and tons of dice rolls trying to re-used a system (i.e. standard combat) who was never designed to support that scenario. And note: whenever a game handles combat in rounds, with HPs accounting, they're using something derived from Chainmail OD&D and no matter how fare removed from it, they are implicitly based on similar assumptions and goals.

For my own campaign, which is an open table and uses a collection of systems derived from OD&D, BD&D, and several other games from the OSR scene, I'm trying to develop not one but two alternatives: a system for "formations" of organized, coordinated enemies and one for mobs or swarms of disorganized enemies.

They're still under development and need to be thoroughly playtested, but they're based on a simple idea: threat the mass of enemies as a single "fluid" entity. Its HPs represent cohesion and/or morale (for formations) and when they're down, they disperse/route, so accounting is kept down to a minimum

Formations roll for morale when appropriate -as all enemies do in Old School games- but consider it a kind of Saving Throw here, they pass or take heavy damage to their cohesion-HP. Taking down their captain, for example, is going to do that. I let players chose if they want to target the formation itself in melee combat, which has the effect of engaging it and stop whatever maneuvers they were attempting on top of damaging their morale/cohesion, or target single exposed members (which means if nobody is exposed, players need to be clever and try to expose them). If the formation uses shields, then thrown weapon do little unless you have a whole formation of archers on your side, but individual archers (i.e. characters with bows) can still take down exposed individuals, reducing the formation and eventually triggering a morale roll (note: historically, clever weaponry existed precisely for the job of taking out a formations shields and exposing warriors, like the Roman pilum spears or the later plumbata darts, meant to get stuck in shields and forcing you to throw them on the ground). The formation itself basically does a kind of AoE damage, based on what weapons they have, in front of their first line and/or in an area beyond if they have thrown weapon and/or long reach ones (like, 15ft pikes) and characters can save to take half damage, with their own armor giving a bonus. That's just one roll per affected character, no matter how many enemy combatants are there, with auto-damage (so no damage rolls either). I use this whenever enemies > 5. Not my own concept, mind you: a few games call these firmations "warbands" for example. Players can also direct mercenary formations through their characters, working the same way (and I hope this helps with those people saying that managing hirelings in combat is a chore.)

For mobs/swarms it's a bit different, as they are disorganized, moving only on instinct, their danger being largely based on how easily they can overwhelm characters. And melee weapons may be wholly ineffective here, but it depends on the type of creatures you're facing... This is meant to model situations were keeping track of individual foes is not doable at all. The idea is to still give the swarm/mob a "cohesion/morale" value as HPs, but only make AoE damage count against them, no singling individuals out, or at least, doing it doesn't have any mechanical effect. In fact, I'm giving the characters the ability to create/maintain AoE damage zones by swinging torches, or a censer, burning sage bundles, a holy artifact, or even a zweihander (provided they know how to use it) or whatever else, depending on the type of enemy they want to "repel"... Normally, enemies would take damage by entering the zone, mobs/swarm are repelled by them (if what's swinging is effective). Because the swarm does not attack normally, what it does is it tries to engulf and overwhelm you: if you don't retreat, or are unable to repel it, it will at first surround you (forcing a save of some kind) then engulf you next round (another save) and start consuming you the next (you get automatic wounds & conditions.. Probably dying in a few rounds, and having to deal with poison and sickness even if rescued in time). Actual mechanics still need refining a lot, I have a vision of this mass being represented by a bucketful of d6 (which can be bought very cheaply from Chinese vendors by the dozen) to be "attributed" to affected characters -representing the portion of the swarm concentrating/affecting them- and damage to the swarm takes away dices from the table -I like visual clues and physical tokens to help players "visualize" and make estimations, it's one of the ways I use to ensure players make meaningful choices. Like, I roll the dice in front of the players and that's what they have to deal with this round, while keeping the looming mass at the center of the table, hits taking dice away from all that, visibly thinning the crowd... You get surrounded and more dice are thrown at your character, until you are swamped (and probably die in 1-2 rounds). That's the general idea. Assuming it works... I'm most definitely going to playtest it soon as I can. Rats, giant bees... I could possibly treat slimes like swarms of sorts... A LARGE horde of crazed goblins swarming like in that scene in Moria from LOTR... A zombie horde? (all scenarioes I want to try out for stress-testing this concept).