How Does the OS Provide Protection?
Part 3 of the Operating System Fundamentals series.
We've said that processes are isolated—one can't read another's memory, and a buggy app can't bring down the whole system. But how?
The OS is just software. It runs on the same CPU as your programs. So what stops a malicious process from just... ignoring the rules?
The problem with software-only protection
Imagine the OS tries to enforce isolation purely in software. It checks every memory access, validates every instruction. But here's the issue: your program runs on the same CPU. If protection is just code, what stops an attacker from jumping over it? Or overwriting it?
You can't build a secure system if the enforcer can be bypassed by the very code it's trying to contain.
Hardware must help
The solution: the hardware itself enforces boundaries. The CPU has built-in mechanisms that the OS configures, and then the hardware checks every operation. User code can't bypass these checks because the checks happen in silicon, not software.
There are three things the hardware must provide:
1. Privileged modes
The CPU has (at least) two modes:
- User mode — restricted. Your programs run here. Certain instructions are forbidden.
- Kernel mode — privileged. The OS runs here. Full access to everything.
A bit in the CPU tracks which mode you're in. Some instructions—halting the CPU, accessing I/O directly, modifying memory mappings—only work in kernel mode. If user code tries to execute them, the CPU traps to the OS.
This is enforced in hardware. No amount of clever programming can flip that mode bit directly.
2. Controlled mode transfer
If user code can't enter kernel mode directly, how does it ask the OS for help?
The hardware provides controlled entry points. User code can trigger a transition to kernel mode, but only to locations the OS has predefined. You can't jump to arbitrary kernel code—you land where the OS decided you should land.
Three events cause this transition:
- System calls — intentional requests for OS services
- Exceptions — something went wrong (divide by zero, bad memory access)
- Interrupts — external events (timer tick, keyboard, disk)
In all cases, the CPU saves the user's state, switches to kernel mode, and jumps to a handler. The OS validates everything before acting.
3. Memory protection (MMU)
Even with privileged modes, we need to isolate processes from each other. The Memory Management Unit (MMU) does this.
The MMU sits between the CPU and memory. Every memory access goes through it. The OS configures the MMU with rules: which addresses this process can access, which are read-only, which are off-limits entirely.
When user code tries to access forbidden memory, the MMU raises an exception. The OS catches it and typically kills the offending process.
This is how each process gets its own address space. The MMU translates addresses so that process A's address 0x1000 maps to different physical memory than process B's 0x1000. We'll explore this more when we cover virtual memory.
Why this matters
These three mechanisms—privileged modes, controlled transfer, and memory protection—are the foundation of OS security. Without hardware support, protection would be impossible.
The OS isn't trusting programs to behave. It can't trust them. The hardware ensures that even malicious code is confined to its sandbox—unless it finds a bug in the kernel itself.
Next: User-to-Kernel Transitions—how interrupts, exceptions, and system calls work in practice.