Is checking the Referer and Origin headers enough to prevent CSRF, provided that requests with neither are rejected?
Expanding on the answers of @Sjoerd and @lindon.
Origin vs Referer vs CSRF token
Most likely, the reason OWASP recommends also using a CSRF token, is that at the time when this recommendation was made - a significant portion of browsers did not yet support the Origin header. This is no longer the case, but people are chimpanzees.
In order to preserve privacy, any browser request can decide to omit the Referer header. So it is probably best to only check the Origin header. (In case you want to allow for users to preserve their privacy)
The Origin header is null in some cases.
Note that all of these requests are GET requests, which means they should not have any side effects.
As long as you make sure the malicious website sending the requests with your browser cannot read the responses, you should be fine. This can be ensured using proper CORS headers. (Do not use Access-Control-Allow-Origin: *!)
To prevent "click-jacking", set the header X-Frame-Options: DENY. This will tell your browser that it is not allowed to display any part of your website in an iframe.
The "new" approach
Setting Cookie properties SameSite=lax or SameSite=strict will prevent CSRF attacks. This is a quite new feature though, and cannot be used alone, simply for the reason that not all common browsers support it yet. You can track support HERE.
When the browsers do, people will likely still recommend checking Origin/Referer/CSRF tokens. If they do - without giving a good reason, it is likely because they are chimps.
Yes, this is secure.
However, the referer header is not exactly mandatory, so there may be browsers or proxies than don't send a referer header. This would mean that these clients can't access your web site.
With the introduction of referrer policy it is possible to remove the referer header from a forged request. So to protect against CSRF it is necessary to block any requests that are missing a referer (and origin) header.
Edit: This paper has some numbers on what portion of clients omit a referer header.