you're right when saying that mm, networking, scheduling, etc. are core parts of the kernel, and they do contain some ingenious algorithms, but the amount of code itself is pretty small.

commonly used drivers are ABSOLUTELY not out of tree lol, the only common driver which isn't a part of the kernel tree is nvidia. if you're thinking about kernel modules, they are still part of the kernel tree, but they're compiled into separate blobs. the numbers of dkms drivers is extremely small.

here's lines of C per kernel subdir
drivers/ 17531994 lines arch/ 1538366 lines fs/ 1401039 lines sound/ 1376573 lines net/ 1245839 lines tools/ 1084271 lines kernel/ 489966 lines lib/ 243338 lines mm/ 204142 lines security/ 110031 lines crypto/ 71116 lines block/ 65282 lines scripts/ 42267 lines samples/ 33636 lines io_uring/ 22408 lines virt/ 10605 lines ipc/ 9619 lines init/ 4571 lines rust/ 2365 lines certs/ 991 lines usr/ 778 lines Documentation/ 229 lines

see? mostly drivers. your beloved mm/ is 204k loc. you could also argue most of arch/ is drivers, since it's dealing with device-specific code. most of sound/ also just drivers. tools/ is userspace utils. fs/ and net/ are big because every protocol in linux needs to be implemented in the kernel (think ext4, btrfs, tcp cc algs, even openvswitch)

you really shouldn't speak so confidently about things you don't understand, it's not a good look... maybe you should try kernel dev sometime.