Drupal - How do I preserve custom files in my Drupal configuration across minor core upgrades?

You can build a site totally with git to help manage this process. The process is basically

Git clone into your DOCROOT:

git clone --branch 7.x http://git.drupal.org/project/drupal.git /var/www/mysite/docroot

Rename the remote:

git remote rename origin drupal

Checkout the latest tag, and make a site branch for your site:

git checkout 7.26
git branch -b mysite

Now, you can build your site and customize robots.txt and .htaccess to your needs.

When there is a core update, you can:

git fetch drupal

and then use the normal git methods for merging your changes in, eg:

git merge 7.27

or you can preview what will happen, check for conflicts, etc.

If you do this, you need to make sure you don't do a plain:

drush pm-update drupal

as this will then mess things up. I do not think that drush pm-update --lock=drupal works to prevent this.

One way to solve this, is as follows:

1. Create a set of patches that patches any altered files (i.e. .htaccess, robots.txt, and other files where you're have added customization).
2. After downloading the new minor release of the core, apply the patches. Check if there are any conflicts.
3. If there are no conflicts, great - you're done! If there are conflicts, resolve them. Iterate until there are no more conflicts.

You can script this, using drush for doing the upgrade and git for managing patches and checking for conflicts. This usually makes upgrading quick and painless, even if you've have customized .htaccess, robots.txt, and other files.