How does sending referrer HTTP headers protect against CSRF attacks?
To understand this, one first needs to understand what a cross-site request forgery is.
A Cross-Site Request Forgery is when the attacker has some script or embedded media on a website they control which makes any visitors to that website request a resource from a different site. That request appears to that other site in the context of the user. I could, for example, add an image with the url https://othersite.example.com/delete_my_account.php?really=true to my website. When you visit my site and are currently logged into your account on othersite.example.com, your browser will request that URL, which would cause othersite.example.com to delete your account.
But there is a catch: When othersite.example.com can read your referrer, it can see that this request is caused by someone interacting with a different site. There is no good reason why a website would have a deep-link to an URL which performs an action. When the other site reject requests without a referrer, that attack becomes impossible.
Regarding referrer spoofing: Keep in mind that the client which does the request isn't the attacker. The attacker is a website operator. The worst that operator can do is have your web browser execute Javascript in the context of their website. While there are several ways to make requests to other websites using Javascript, none of those APIs allows to fake the referrer.
Couldn't the attacker just spoof the referrer header
No, they can't.
It is obviously not possible when submitting a form (or the various GET methods such as image tags or url in CSS), and it is not possible via XMLHttpRequest either, as per the documentation.
That being said, I would suggest using a token instead, as checking the referer may be bypassed if an open redirect vulnerability exists and the application doesn't follow restful, or if it allows downgrade to GET and contains open redirect vulnerabilities, HTML injection vulnerabilities, or CSS injection vulnerabilities. It may also cause usability problems (as in your case).