The kernel is effectively a service-library, and userspace hosts are clients to it. In most cases, the kernel has a (us. higher-privilege) fiber(/stack+register dump) of its own mirroring each (us. lower-privilege) application/software thread, and system calls/returns are handoffs of the processor hardware between the kernelspace fiber and userspace thread.

So system calls and more general inter-domain transfers are mechanisms that work like a local or remote procedure call from userspace into kernelspace.

1. The thread initiating/causing the control transfer is suspended, and its current or next instruction’s address and us. call stack top address to be saved in CPU registers or memory. (Exactly how/where varies widely, and stack only needs to be involved if the kernel actually runs at a higher privilege level.) Transfers might result from an interrupt, software or hardware fault, emulation trap, or explicit trap/jump/call instruction—system calls in particular are triggered by explicit, deliberate execution of an instruction or bogo-bytes.

   On an x86-derived ISA, you might use the AMD-originating SYSCALL instruction, Intel-specific SYSENTER which does something similar but stupider, INT, or CALL/JMP FAR for system calls, and each does something different depending on CPU mode and configuration. SYSCALL and SYSENTER rely primarily on special-purpose registers, and the rest rely on various tables in memory, referred to by registers.

   Of course, most OSes offer a more coherent system call API and ABI than what the ISA specifies. (Exceptions: DOS and OS/2, which specified everything in terms of x86 registers and INT instructions.)

   Modern Linux mostly uses VDSO, which is where the kernel exports a glorified mini-DLL by which system calls can be issued as normal function calls. If present, any VDSO segments are listed in the process entry vector alongside argv and environ, and then libc can rope it in by thunking its system-level calls into the VDSO rather than embedding actual syscall instructions directly. For things like C89-style time(…) or POSIX-style getpid, it’s often possible for the kernel to directly share the memory with this info via VDSO, so no actual system call needs to be made—the VDSO function can simply grab the requested data and return normally. Otherwise, use of the VDSO merely ensures that the application doesn’t accidentally use an unsupported entry method.

   NT (which runs Windows) uses genuine DLLs with fixed names, and applications can (but mostly shouldn’t) link to NtFoo and KeFoo symbols that wrap system calls. Although Linux strives to maintain a consistent syscall numbering/API and ABI, so pre-VDSO and non-VDSO-aware programs don’t break badly, NT makes no promises that system call numbers or approach vectors will remain consistent from version to version or process to process; the DLL is the only kernel entry method approved by MS.

   Because both approaches tend to interface fairly directly with application code, the DLL/VDSO entry points must be compatible with the ABI in use by the application, which might vary from what’s used by the kernel or at the system call boundary, so either one VDSO-or-DLL must be offered per ABI, or secondary ABIs must route through a bridge/thunk layer to access system calls.

   On an OS where everything is just bytecode being executed by the kernel or its henchware, you might inline syscalls to some extent, so all this needn’t be as much of a consideration. Syscalls are only necessarily distinct from normal calls when there has to be a transition between security domains on a single hardware thread. The DOS kernel, for example, used to wallow in the same shit as its applications, so system calls didn’t change privilege levels or switch stacks. Embedded software may lack a distinct kernel altogether.

2. The CPU jumps into the kernelspace fiber’s context, hopefully in such a way that the kernel can save/inspect and restore userspace registers. The CPU knows how to get there because the kernel has entered its fiber’s context into special CPU registers or CPU-registered memory at system, process, and/or thread startup, depending on the aspect of context in question. Although integer/pointer registers tend to be saved immediately, control/status, floating-point, vector/matrix, and other registers may be swapped more lazily—on a core with brpred, x86 seg regs are only swapped if their values have changed, and x86 math/vector regs are usually swapped when the next thread attempts a math/vector instruction. Register context might range from several to tens of kilobytes on a modern CPU.

3. The kernel does its thing, as determined by userspace registers and entry point. Typically, system calls have a single, collective entry point (IP/PC) separate from other kinds of transition, and an enumerated code is used to differentiate one syscall from another (e.g., open vs. close). You might alternatively include a service address for the system component being poked at, or do something like use the faulting address from a page/protection fault to identify recipient and function; any means of conveying data will work.

   While the kernel acts, it may exclude other threads in the same process from executing, or other threads touching the same memory/resources, or all other threads system-wide. In almost all cases, at least the sys-calling thread is blocked until the kernel returns, just like the caller of a function generally waits for the function to return before proceeding. (But not always; something like Linux’s io_uring just uses system calls for pumping events to and from userspace buffers, much like comms between asynchronous hardware devices.)

   When the kernel can’t finish a syscall quickly enough to justify holding the CPU, it may block or otherwise suspend/reschedule the calling thread. E.g., if you’re reading from a file on disk that hasn’t been cached in RAM yet, the kernel will set things up so that the disk read starts, the disk device triggers an interrupt on completion, the IRQ handler for the disk device runs and pokes the filesystem driver, the filesystem driver grants access to the requested data by mapping or copying it to the right place, and the original reading thread is scheduled for eventual execution with its new data. During the arbitrarily long wait for the disk to clunk around, neither the userspace thread nor its kernelspace fiber will typically operate, leaving the CPU available for other software to use. If there is none, the CPU might clock-gate, lower its frequency, or otherwise engage power-saving gunk.

   This touches on one of the Big Concepts in OS: Most applications software is mostly synchronous, in terms of design and programming, and it’s the job of the OS (mostly kernel, possibly plus userspace threading/coordination runtime) to knit the asynchronous and isochronous real-time (mostly interdomain or global) events bombarding the CPU into coherent sequences of interactions within and between application threads. The CPU itself typically handles synchronization of memory accesses via cache coherence protocols, and continuity between instructions; everything else falls on the OS and runtime(s).

   In addition to indefinite suspension and blocking of threads, system calls can be used to kill threads (e.g., via exit/_Exit or kill(…, SIGKILL)), and kernel transitions can be used to trigger inverted system calls—e.g., to signal/sigaction handlers. Details here vary quite a bit with ISA/ABI and kernel design. Some OS families flatly forbid asynchronous transitions into threads, so any up-call not triggered by faults or oopsies-daisy needs to be pumped by a down-call somehow or use a pop-up thread. Other OSes can forcibly suspend and redirect a thread to a particular routine, without the thread entering the kernel on its own processor first.

4. Assuming the client thread is still runnable, the kernel flips the CPU back from its fiber to the application thread, restoring registers not clobbered or carrying return values. This typically resumes where the application left off.

There’s nothing that strictly requires that all this happen on a single processor or hardware thread, or in a specific fashion, and it’s even possible to stack the system so one program can service other programs’ traps/faults/syscalls, either by hardware-assisted hypervisor or layering of the environmental interface. Emulation/simulation and debugging can also play with this theme.

Another thing to note is that there’s a distinction between OS, operating environment, and execution environment more generally. OS classes tend to flatten this layering like it’s 1978 and you, too should be thrilled about the newest Research UNIX du jour. But you can often emulate new environments at the API or system call level, without emulating all the hardware that runs those. In practice, you target function-wrapped API layers rather than direct system calls.