On-the-fly monitoring HTTP requests on a network interface?

Try tcpflow:

tcpflow -p -c -i eth0 port 80 | grep -oE '(GET|POST|HEAD) .* HTTP/1.[01]|Host: .*'

Output is like this:

GET /search?q=stack+exchange&btnI=I%27m+Feeling+Lucky HTTP/1.1
Host: www.google.com

You can obviously add additional HTTP methods to the grep statement, and use sed to combine the two lines into a full URL.

You can use httpry or Justniffer to do that.

httpry is available e.g. via the Fedora package repository.

Example call:

# httpry -i em1

(where em1 denotes an network interface name)

Example output:

2013-09-30 21:35:20    192.168.0.1     198.252.206.16    >    POST    unix.stackexchange.com    /posts/6281/editor-heartbeat/edit    HTTP/1.1
2013-09-30 21:35:20    198.252.206.16  192.168.0.1       < HTTP/1.1   200    OK
2013-09-30 21:35:49    192.168.0.1     198.252.206.16    >    POST    unix.stackexchange.com    /posts/validate-body                 HTTP/1.1
2013-09-30 21:35:49    198.252.206.16  192.168.0.1       < HTTP/1.1   200    OK
2013-09-30 21:33:33    192.168.0.1      92.197.129.26    >    GET     cdn4.spiegel.de    /images/image-551203-breitwandaufmacher-fgoe.jpg    HTTP/1.1

(output is a little bit shortened)

I was looking for something similar, with the added requirement that it should work for https too.

pcap based tools like tcpflow httpry urlsnarf and other tcpdump kung fu work well for http, but for secure requests you're out of luck.

I came up with urldump, which is a small wrapper around mitmproxy.
iptables is used to redirect traffic to the proxy, so it works transparently.

$ sudo urldump   
http://docs.mitmproxy.org/en/stable/certinstall.html
http://docs.mitmproxy.org/en/stable/_static/js/modernizr.min.js
https://media.readthedocs.org/css/sphinx_rtd_theme.css
https://media.readthedocs.org/css/readthedocs-doc-embed.css
https://media.readthedocs.org/javascript/readthedocs-doc-embed.js
...

See README for more info.