Can the command completion for `cd` be modified to only show directories and ignore files?

Just add

complete -d cd

in your ~/.bashrc (or other bash configuration file).

This should be happening automatically on a typical install on many distros.
If it is not, you're probably missing the bash-completion package:

• Ubuntu: sudo apt-get install bash-completion
• Arch: sudo pacman -S bash-completion