How to use my own version of a standard package

I would fix the packaged version of Nautilus, which might seem daunting at first but is easy enough — although it doesn’t survive package upgrades so it does require some discipline. (See Wouter’s answer for details.)

The simplest approach in your situation is to add a diversion:

sudo dpkg-divert --divert /usr/bin/nautilus.original --rename /usr/bin/nautilus

This will instruct dpkg to rename /usr/bin/nautilus to /usr/bin/nautilus.original whenever a package tries to install it. Then you can add your own symlink, and it will remain untouched even when the Nautilus package is upgraded. To remove it, run

sudo dpkg-divert --rename --remove /usr/bin/nautilus

You can apply the same technique for any other file you need to replace in a similar fashion, apart from some configuration files which aren’t handled correctly when diverted.

The best long-term solution is to contact the nautilus developers and work with them on a patch to the official nautilus code, so that once the updated version of nautilus hits your system, you won't need to deal with this anymore.

Assuming you've done that, until that change trickles down to your system, you'll still need to deal with the issue of wanting to use your own patched version of nautilus.

The easiest way is to produce a package of nautilus with your patch already applied, and to install it in such a way that it won't be upgraded automatically.

On Debian (which according to the tags on this question you're using), the easiest way to accomplish that is:

apt-get install devscripts
apt-get build-dep nautilus # note, this also pulls in build-essential
apt-get source nautilus
cd nautilus-*
# apply your patch
dch -i
# Set the version number of the new changelog entry to something like x.y.z-a.local,
# where "x.y.z-a" is the version of the package as it existed prior to your patch,
# add a description of the change after the '*', then save and exit the editor
debuild -uc -us -i && sudo debi --upgrade
echo "nautilus hold" | sudo dpkg --set-selections

This builds a new package with your patch, installs it, and then marks it in dpkg so that it will not be auto-updated anymore.

The next time an updated version of the package will be released, apt will notify you that an updated version exists, but is being held back. You then need to do the above again.