Retq instruction, where does it return

ret is how you spell pop rip on x86: a stack pop and an indirect branch to that value. https://www.felixcloutier.com/x86/ret documents exactly what it does and doesn't do.

It's effectively pop %tmp / jmp *%tmp where tmp is an internal temporary register.

ret depends only on RSP.

Using RBP as a frame pointer is a totally optional software convention that modern compilers don't even do when optimization is enabled.


After studying assembly code, here are my thoughts, let's look at a sample:

fun:
push %rbp
mov %rsp,%rbp
...
...
pop %rbp
retq

main:
...
...
callq  "address" <fun>
...
...

We can see there is a instruction before retq. The pop %rbp (sometimes it is a leave instruction but they are similar) instruction will

  1. save the content of current stack pointer %rsp to base stack pointer %rbp.
  2. move the %rsp pointer to previous address on stack.

For example: before pop command, the %rsp pointed to 0x0000 0000 0000 00D0. After the pop command it points to 0x0000 0000 0000 00D8 (assume the stack grows from high address to low address).

After the pop command, now %rsp points to a new address and retq takes this address as return address.