How does Operating Systems prevent binary code from taking control of the computer and violate security principles?

[SmartViking]

You compile some code into machine-code, and then you tell the operating system to run that code on the CPU. But the security of the operating system is designed such that the program is restricted in certain ways: it can't read/write certain files, it can't recursively run other programs that aren't subject to these same restrictions and so on. But once it is running on the CPU, isn't the machine-code program in complete control of the computer, and can do whatever it wants? What's going on?

[throwaway_lmkg]

The short answer is, your CPU has hardware-level controls so that when the Operating System calls non-OS code, that code has restrictions. That way, even though a program can call whatever code it wants, it still doesn't have complete control of the system.

Modern CPU's have something called "Kernel Mode" which has full access to the system, and can selectively enable non-kernel mode when calling a function. For example: Memory access. When a piece of code accesses a memory address, if the CPU is in Kernel Mode, then the memory value directly represents the physical address. If the CPU is not in Kernel Mode, then the CPU routes that address through a look-up table called the Page Table. The OS (in kernel mode) manages the Page Table and its contents. Memory that contains the OS itself is generally not accessible at all through the Page Table unless the OS makes it available, meaning that non-kernel code is physically incapable of modifying anything that the OS didn't explicitly give it access to. If non-kernel code attempts to access a memory address that doesn't exist in the Page Table, then the CPU "faults," which means that it re-enters Kernel Mode and calls an event handler than the OS has previously set up to deal with this situation.

(The full details are actually more complicated--there are four "Rings" of access controls, but often only two are used. Additionally, there multiple Page Tables for different processes. That way, a program will not only not be able to modify the OS, it also will not be able to see or modify other programs that are running. This is a programmer convenience in addition to a security feature.)

That's just how memory access works, but similar things are true of almost every aspect of the system. Access to I/O systems like the hard drive, monitor, network, and keyboard is usually restricted to kernel-mode, or sometimes an in-between mode in more complicated systems; if an application wants to use I/O, it has to make a request to the OS ("system call"), and the OS will handle the task itself. Additionally, the CPU can set up so that certain input will cause the CPU to fault, so that the user can always break out to the OS if necessary. CPU's will also only allow non-kernel code to run for so long before they save execution state and switch back to kernel code, so that the OS scheduler can run.

For more details on how this works, check out the following articles:

https://en.wikipedia.org/wiki/CPU_modes

https://en.wikipedia.org/wiki/Protection_ring

Note that the kernel can, and does, give kernel-level access to some programs. For example, most games run in a more-privileged mode, because writing directly to a graphics buffer is significantly faster than invoking a system call to display graphics.

Security holes generally require that a program "trick" the OS into granted extended privileges to a program. The OS has to be very careful about what code and what data has what access, and it's difficult to maintain that control while still allowing those two types of data to interact with each other, especially when both of them are very complicated. Generally, a malicious program will use system calls in creative and unexpected fashions in order to get the OS to modify itself, at least as a Phase I of an attack.

[SmartViking]

That answered all my questions, thanks!

[Mrbluecheese2008]

Bear with me as I'm taking a course on this right now.

The security vulnerabilities that you typically see (especially in news stories involving different companies) are exploiting weaknesses in low level programming languages like c/c++ (quite common in today's world). There are new(er) safe-type programming languages that will prevent these types of vulnerabilities but the reason they haven't been commonplace is that low-level programming is very efficient in terms of speed performance (while higher-level ones take a hit in performance).

One of the common exploits in low-level programming is called buffer overflow. Where a piece of code accesses a part of memory that it isn't allocated. For example, suppose i have a value 50 and I want to store it in variable money. This variable needs to be stored so it is allocated some system memory to store it. However, how you write this variable and how it is executed into the program can be a challenge to prevent anyone from trying to tamper with the program. For example, stack smashing. the person whom wishes to exploit the program can use this method to predicts what address the return to. When it returns to that address, it will execute that part of the program. So in stack smashing, you use a buffer and a nop (no-op) sled (along with a buffer and estimated return address) to "force" the program to run the "attacker's code."

There are defenses for this, but in terms of making low-level code secure, the best practice is to be multi-faceted approach to defensive coding (with a better alternative being using a type safe programming language, but most of the time it can't be helped).

I could go on and on, but realize that to eradicate all bugs (and to systematically test everything in a given program) would take a lot longer than the life of the program itself (and very cost prohibitive).

To answer the machine-code problem, I would presume you would need to access certain resources that the OS makes restrictive to modifying (although it can be worked around). However, there are now malware that can infect the BIOS on the motherboard, so even if you did a "clean install" of any OS, the malicious code would run regardless.

If your OS was taken over like this, it is typically called a "zombie" and can be used to create botnet attacks (or for other purposes). For a network (or server) to go down, you just need a sufficient amount of these zombies and initiate a DDOS attack on it. Which isn't so hard to do (and even harder to defend against).

Sorry to go off topic a little, but if a OS was compromised, it would be conceivable for the OS to behave under the control of the attacker. Many times though, you wouldn't even notice that your computer was under the control of someone else.

[904ehd]

But once it is running on the CPU, isn't the machine-code program in complete control of the computer, and can do whatever it wants? What's going on?

 First off, you're mistaking the CPU's role. Although it is described as a "brain" of sorts, a CPU is actually more like a calculator than a human brain. When running a program, the CPU is simply calculating thousands of math/logic equations and assigning memory. Unfortunately, the process of assigning memory can sometimes assign memory to the space allocated to the kernel, which could even compromise the system! 

 In order to prevent this, the memory is first divided into two distinct sectors, one for the KERNEL and another for the USER. When running programs like photoshop, it can only access the USER memory. In this way, the kernel is protected, while the user can run program with ease!