How does Linux retain control of the CPU on a single-core machine?

The kernel gains control quite frequently in normal operations: whenever a process calls a system call, and whenever an interrupt occurs. Interrupts happen when hardware wants the CPU’s attention, or when the CPU wants the kernel’s attention, and one particular piece of hardware can be programmed to request attention periodically (the timer). Thus the kernel can ensure that, as long as the system doesn’t lock up so hard that interrupts are no longer generated, it will be invoked periodically.

As a result,

if that process is doing some extremely long running computation without yielding

isn’t a concern: Linux is a preemptive multitasking operating system, i.e. it multitasks without requiring running programs’ cooperation.

When it comes to killing processes, the kernel is involved anyway. If a process wants to kill another process, it has to call the kernel to do so, so the kernel is in control. If the kernel decides to kill a process (e.g. the OOM killer, or because the process tried to do something it’s not allowed to do, such as accessing unmapped memory), it’s also in control.

Note that the kernel can be configured to not control a subset of a system’s CPUs itself (using the deprecated isolcpus kernel parameter), or to not schedule tasks on certains CPUs itself (using cpusets without load balancing, which are fully integrated in cgroup v1 and cgroup v2); but at least one CPU in the system must always be fully managed by the kernel. It can also be configured to reduce the number of timer interrupts which are generated, depending on what a given CPU is being used for.

There’s also not much distinction between single-CPU (single-core, etc.) systems and multi-CPU systems, the same concerns apply to both as far as kernel control is concerned: each CPU needs to call into the kernel periodically if it is to be used for multitasking under kernel control.

Linux and most modern OSes use preemptive multitasking which means the kernel has complete control over the time each process is allowed to run and will preempt a process if it runs for too long, unlike cooperative multitasking where a process will pass control whenever it likes

Basically in preemptive multitasking the kernel will be fired periodically from a timer, and whenever the kernel is in control (when the timer interrupt happens or a system call is invoked) the kernel will save the current process' context and then switch to the next process' context. That's called a context switch where the whole state of the process including all thread information, register values... are saved and restored to make the process continue to run from the exact point it was preempted without even knowing it hasn't run continuously. Therefore lots of processes will appear to run simultaneously in a single CPU core, even though in reality only 1 process is being run at any moment. The kernel is also just a special process that does all the process and resource handling. It's not run from a separate core just to monitor other processes

See also What does it mean to say “linux kernel is preemptive”?