wget --spidedr code example

Example: wget --spidedr

wget --spider --debug -e robots=off -r -p http://jcode.me 2>&1 \
        | egrep -A 1 '(^HEAD|^Referer:|^Remote file does not)' > ~/wget.log