How do I rename the extension for a bunch of files?

rename 's/\.html$/\.txt/' *.html

does exactly what you want.

For an better solution (with only bash functionality, as opposed to external calls), see one of the other answers.

---

The following would do and does not require the system to have the rename program (although you would most often have this on a system):

for file in *.html; do
    mv "$file" "$(basename "$file" .html).txt"
done

EDIT: As pointed out in the comments, this does not work for filenames with spaces in them without proper quoting (now added above). When working purely on your own files that you know do not have spaces in the filenames this will work but whenever you write something that may be reused at a later time, do not skip proper quoting.

If using bash, there's no need for external commands like sed, basename, rename, expr, etc.

for file in *.html
do
  mv "$file" "${file%.html}.txt"
done