How does a shell execute a program?

Well, the exact sequence may vary, as there might be a shell alias or function that first gets expanded/interpreted before the actual program gets executed, and then differences for a qualified filename (/usr/libexec/foo) versus something that will be looked for through all the directories of the PATH environment variable (just foo). Also, the details of the execution may complicate matters, as foo | bar | zot requires more work for the shell (some number of fork(2), dup(2), and, of course, pipe(2), among other system calls), while something like exec foo is much less work as the shell merely replaces itself with the new program (i.e., it doesn't fork). Also important are process groups (especially the foreground process group, all PIDs of which eat SIGINT when someone starts mashing on Ctrl+C, sessions, and whether the job is going to be run in the background, monitored (foo &) or background, ignored (foo & disown). I/O redirection details will also change things, e.g., if standard input is closed by the shell (foo <&-), or whether a file is opened as stdin (foo < blah).

strace or similar will be informative about the specific system calls made along this process, and there should be man pages for each of those calls. Suitable system level reading would be any number of chapters from Stevens's "Advanced Programming in the UNIX Environment" while a shell book (e.g., "From Bash to Z Shell") will cover the shell side of things in more detail.