Files I should have on every website

As far as I know, no bots or apps request sitemap.xml without being told it should be there. Most sites probably don't have it, and of the sites that do, many use gzip, and many call the file something else or put the sitemaps in a subfolder.

Here are all the ones I know of:

• favicon.ico
  Gives your pages an icon in tabs, bookmarks, etc.
• robots.txt
  Useful if you need to block any pages from search engines, but can be blank to allow access to everything. It can also contain the location for your sitemap.
• Webmaster Tools verification files
  The easiest method of verification (saves having useless meta tags in your HTML code). There's google[hash].html and BingSiteAuth.xml. I have one for Yahoo in my root too but that's probably no longer needed.
• crossdomain.xml
  This file is requested by Adobe Flash any time Flash needs to connect to your site. You will probably never see this file in your logs on a small site, but I had a bunch of requests for it on one site. You can just create a blank file since the default is to block access to external sites.
• .htaccess (Apache)
  Obviously not requested by browsers/bots but vital for most sites.

Finally, regarding the icons that the iPhone insists on requesting: there are several sizes that may get requested. My iPhone 3G (iOS 5) requests the 57x57 version and I see 72x72 and 114x114 (most common) in my error logs. So if you want to eliminate all 404s from your logs, mind-boggingly you would need to provide all of these:

• /apple-touch-icon-57x57-precomposed.png
• /apple-touch-icon-57x57.png
• /apple-touch-icon-72x72-precomposed.png
• /apple-touch-icon-72x72.png
• /apple-touch-icon-114x114-precomposed.png
• /apple-touch-icon-114x114.png
• /apple-touch-icon-precomposed.png
• /apple-touch-icon.png

I think you probably have it.

Even apple touch icon is not entirely nescessary, since if you choose not to have it (and don't link to it) then iOS will just use a thumbnail of the page (personally, I add them in though).

I'd say you only really NEED favicon.ico and robots.txt, sitemap.xml is optional too.

Another one that I've noticed is browserconfig.xml which is some bastardisation requested by IE. Bloody rubbish filling up the logs.