Are solder fumes bad for me?

Solder fumes aren't very good for you. Some people can become sensitized to flux fumes, especially from the older rosin flux used in cored solder, and get breathing problems:

Controlling health risks from rosin (colophony) based solder fluxes

The no-clean flux isn't as bad.

I once felt quite ill after assembling about 30 boards that I had to do myself as my distributor wanted them very quickly.

Breathing out whilst you are soldering each joint helps a lot, if you don't have fume extraction.

At the very least, put a fan near your work area to blow the fumes away from your face.

My first reaction to your question was, yes the fumes are going to harm you. My second was to conduct some quick research.

Amazingly, as I quote next, Lead-Free Solder inhalation can lead to pneumonia and a 24-48h flu.

INHALATION: Inhalation of dusts and fumes may cause irritation of the upper respiratory tract. Inhalation of freshly formed metal oxides may cause metal fume fever, a brief (24-48 H) flu-like illness. Inhalation of tin oxide may cause pneumonia.

According to the MSDS from Weller on some of their non-lead solder.

You can guarantee this must be a frequent event or the company would not be disclosing this in such a cavalier manner.