Googlebot causes an invalid Cross Origin Request (COR) on Rails 4.1

As per "CSRF protection from remote tags " from the rails guide:

In the case of tests, where you also doing the client, change from:

get :index, format: :js

To:

xhr :get, :index, format: :js

http://edgeguides.rubyonrails.org/upgrading_ruby_on_rails.html#csrf-protection-from-remote-script-tags

In the case you want to make this route skip csrf check, white list the route using something like:

protect_from_forgery :except => :create

Googlebot is using the format "*/*" (http://apidock.com/rails/Mime) and the application renders the js since it's the only thing available. Since it's remote, it correctly causes an Invalid COR.

This was reproducible using:

curl -H "Accept: */*" https://www.example.com/users/123/flag

The fix is to have an html fallback resource for the spider to crawl:

respond_to do |format|
  format.html { render template: 'users/flag' }
  format.js { render template: 'users/flag', layout: "some_layout" }
end