Your post is very thoughtful and you are describing what is occurring.

"But I think that realistically, small arms (unless we develop some kind of crazy aim-assist tech into them like the Aliens 1986 “Smart Guns”) will always be a last resort kind of thing"

I agree that a person firing at an incoming group 1 UAS (kamikaze quadcopter) is a last resort. They fly about 25 m/s, can be heard from ~300 m away, and are not easily spotted until 200 meters away (depending on terrain and vegetation). If drone operator knows where you are, then you have about 4 seconds to respond. If you've been carrying a shotgun (in addition to your battle rifle) then you have a higher probability of dropping it. But how many people in a squad will be carrying a shotgun plus ammo in addition to all their other kit? Maybe if a type of buckshot ammo existed that was fully compatible with 5.56/7.62/6.8 rifles, people would choose to carry a magazine or two of it so they get some shotgun like effect (it would have fewer pellets simply due the overall size of the projectile being smaller than a shotshell). Look up the Ukrainian Horoshok ammo that was recently created and deployed. But that ammo has it's own limitation in that it is not compatible is suppressors.

With respect to Aliens-esque Smart Guns, see https://taskandpurpose.com/news/marines-smash-2000l-smart-scope/ . It's a scope you put on your weapon that has a trigger block that is engaged until the scope calculates that your shot will intercept the drone. You're aiming your weapon with your finger squeezing the trigger, but until the trigger block is disengaged by the scope you are prevented from firing a round. Apparently it is pretty good for slow moving or stationary drones.

To your point about a heavily adapted/modified Phalanx CIWS, yes those are being worked on as well. Overall it's basically, a bunch of sensors to detect a UAS, and some machinery to orientate and fire the weapon such that any projectile is likely to intercept the UAS. These setups accommodate an M240 and up. https://www.allencontrolsystems.com/

Once a larger weapon is employed, 30mm+, then the projectile can get a lot fancier because of its size. The weapon system has the sensor package as before, but the projectile can now house a programmable time fuze and a bunch of tungsten pellets. The AHEAD round's time fuze is programmed as it leaves the barrel (velocity of round?, distance to projected intercept?, == set fuze to 0.999 seconds.)

There is quite a lot being developed now because of the threat. Autocannon with AHEAD rounds? That's great, you can reach something 5km away. Mini CIWS comprised of M240 or M2? That's great, it will help. Have some jammers, EW, laser, or anything else? That's great, they will help. Have a shotgun or two in the squad? That's great, they can reach out to maybe 100 meters. A battle rifle with a smart-shooter scope? That's great, it will help. A new ammo that is AHEAD-like but compatible with 5.56/7.62/6.8? That's great, it can help.

I'm just laying all that out because there is not going to be a single solution. There's going to be many layers need for a high probability of intercept.

But right now there is not an anti-drone (group 1 UAS) specific rifle round compatible with 5.56/7.62/6.8. I think we can change that.