Can I scroll the file up or down in an inactive window in Vim?

You could add a couple of custom mappings:

:nmap ,d <C-W>W<C-D><C-W>W
:nmap ,u <C-W>W<C-U><C-W>W

and then use ,d and ,u to scroll down and up in the other window.