How does a debugger work in Linux?

There is a system call named ptrace. It takes 4 parameters: the operation, the PID of the target process, an address in the target process memory, and a data pointer. The way the last 2 parameters are used is dependent on the operation.

For example you can attach/detach your debugger to a process:

ptrace(PTRACE_ATTACH, pid, 0, 0);
...
ptrace(PTRACE_DETACH, pid, 0, 0);

Single step execution:

ptrace(PTRACE_ATTACH, pid, 0, 0);
int status;
waitpid(pid, &status, WSTOPPED);
while (...) {
    ptrace(PTRACE_SINGLESTEP, pid, 0, 0);
    // give the user a chance to do something
}
ptrace(PTRACE_DETACH, pid, 0, 0);

You can also read/write the memory of the target process with PTRACE_PEEKDATA and PTRACE_POKEDATA. If you want to see a real example check out gdb.