Does watch only monitor the visible output?

I found this thread titled: Bug#225549: have watch monitor stderr. That thread is from 2008, but it looks like older versions don't support the watching of anything other than STDOUT.

So we're limited to just STDOUT. As for visible there is a lot of language in the info watch and man watch that make me think your observation/assumption is correct.

excerpt

   watch runs command repeatedly, displaying its output (the first screen‐
   full).   This  allows you to watch the program output change over time.
   By default, the program is run every 2 seconds; use -n or --interval to
   specify a different interval.

Also this bit under BUGS:

BUGS
       Upon  terminal resize, the screen will not be correctly repainted until
       the next scheduled update.  All --differences highlighting is  lost  on
       that update as well.

If I had to guess I'd think they were storing the visible bits in a buffer between runs, and then analyzing just those characters.

EDIT #1

I debugged this further using strace and you can see watch reading the output from the ls command so it's internally dropping the change.

before I delete the m file

$ strace -o w.log watch -g 'ls -1'
read(3, "a\nb\nc\nd\ne\nf\ng\nh\ni\nj\nk\nl\nm\nn\nw.lo"..., 4096) = 34
close(3)                                = 0
--- SIGCHLD (Child exited) @ 0 (0) ---
munmap(0x7f4da83af000, 4096)            = 0
wait4(31011, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 31011
rt_sigaction(SIGTSTP, {SIG_IGN, [], SA_RESTORER|SA_RESTART, 0x7f4da79b94a0}, {0x7f4da7f81ee0, [], SA_RESTORER|SA_RESTART, 0x7f4da79b94a0}, 8) = 0
write(1, "\33[H\33[2JEvery 2.0s: ls -1\33[1;140H"..., 119) = 119
rt_sigaction(SIGTSTP, {0x7f4da7f81ee0, [], SA_RESTORER|SA_RESTART, 0x7f4da79b94a0}, NULL, 8) = 0
nanosleep({2, 0}, NULL)                 = 0
stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=3519, ...}) = 0
pipe([3, 4])                            = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f4da839f9d0) = 31014
close(4)                                = 0
fcntl(3, F_GETFL)                       = 0 (flags O_RDONLY)
fstat(3, {st_mode=S_IFIFO|0600, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f4da83af000
lseek(3, 0, SEEK_CUR)                   = -1 ESPIPE (Illegal seek)
read(3, "a\nb\nc\nd\ne\nf\ng\nh\ni\nj\nk\nl\nm\nn\nw.lo"..., 4096) = 34
close(3)                                = 0
munmap(0x7f4da83af000, 4096)            = 0
--- SIGCHLD (Child exited) @ 0 (0) ---

after the m file is deleted

--- SIGCHLD (Child exited) @ 0 (0) ---
rt_sigaction(SIGTSTP, {SIG_IGN, [], SA_RESTORER|SA_RESTART, 0x7f4da79b94a0}, {0x7f4da7f81ee0, [], SA_RESTORER|SA_RESTART, 0x7f4da79b94a0}, 8) = 0
poll([{fd=0, events=POLLIN}], 1, 0)     = 0 (Timeout)
poll([{fd=0, events=POLLIN}], 1, 0)     = 0 (Timeout)
write(1, "\33[1;158H8\33[11;163H", 18)  = 18
rt_sigaction(SIGTSTP, {0x7f4da7f81ee0, [], SA_RESTORER|SA_RESTART, 0x7f4da79b94a0}, NULL, 8) = 0
nanosleep({2, 0}, NULL)                 = 0
stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=3519, ...}) = 0
pipe([3, 4])                            = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f4da839f9d0) = 31028
close(4)                                = 0
fcntl(3, F_GETFL)                       = 0 (flags O_RDONLY)
fstat(3, {st_mode=S_IFIFO|0600, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f4da83af000
lseek(3, 0, SEEK_CUR)                   = -1 ESPIPE (Illegal seek)
read(3, "a\nb\nc\nd\ne\nf\ng\nh\ni\nj\nk\nl\nn\nw.log\n", 4096) = 32
close(3)                                = 0
--- SIGCHLD (Child exited) @ 0 (0) ---
munmap(0x7f4da83af000, 4096)            = 0
wait4(31028, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 31028

I expect it to exit if a file is added or removed

I'm pretty sure you're after inotify-tools.

My manpage for watch, from procps-ng, says

watch runs command repeatedly, displaying its output and errors (the first screenfull).