Best practices for job/process tracking and multi-tasking

If the issue is you're finding it difficult to keep track of what you're working on as you bounce from task to task you might want to take the time to look at using a tool such as tmux and/or screen. These are virtual terminal window servers and allow you to setup a terminal within them and name it. This allows you to put some context on a terminal and keep multiple terminals going without cluttering up your desktop.

I typically use screen so I'm more familiar with it's work flow but in general I setup a screen session like so:

screen -S appX

I then connect to it like so:

screen -r appX

Then within the space of appX you can setup different tabs/windows for work related to appX. I might setup a windows called compiling while another might be log, where I tail the log file for the app. I can then either use key combinations Ctrl+A+Ctrl+A to move from one tab/window to another or you can split your terminal up so that one of them is in the upper 1/2 of the terminal, while the other is in the lower 1/2.

If you're running similar commands many times, put the similar parts in a shell script or in a makefile. Give the script or the target meaningful names. That way, when you look at what your various terminals are doing, the command line (plus possibly the host name and the current directory) will tell clearly you all you need to know.

Set your terminal's title to show the current directory and running command from $PROMPT_COMMAND (bash) or preexec (zsh). That way you can tell what's going on directly on your window list, without having to switch to the window. When the command terminates, set the title to the current directory and the exit status of the last command ($?).

Log all commands that produce output, and make sure the log includes an indication of success or failure (if it doesn't happen naturally, finish with echo $?). This logging should be included in your wrapper script (see above). That way, you can look back and see what happened before. Log the dates of start and completion as well if you find them useful.

To answer the questions directly:

• "What is running?" ps afux, pstree and similar can give you a good idea of what's running at the moment. If you name your scripts in a reasonable way it should be easy to get an overview from this. You can also name your terminal window using printf "\033]0;Window title\007"
• "Why is it running? What was I trying to accomplish when I started the script?" You can redirect the command output to log files named according to the issue you're handling (optionally using tee if you want to see it while running as well).
• "What completed running and when? What was the exit code (pass/fail)?" By logging the time and echo $? output at the end of the tasks to the same files (or even better, making it a standard part of the script) you can get a very good idea of what happened at the end.