Can GIT, Mercurial, SVN, or other version control tools work well when project tree has binary files?

In general, version control systems work better with text files. The whole merge/conflict concept is really based around source code. However, SVN works pretty well for binary files. (We use it to version CAD drawings.)

I will point out that the file locking (svn:needs-lock) are pretty much a must-have when there are multiple people working on a common binary file. Without file locking, it is possible for 2 people to work on a binary file at once. Someone commits their changes first. Guess what happens to the person that didn't commit. All of that binary/unmergable work they did is effectively lost. File-locking serializes work on the file. You do lose the "concurrent" access capabilities of a version control system, but you still have the benefits of a commit log, rolling back to a previous version, etc.

The TortoieSVN client is smart enough to use MS Word's built in merge tool to diff a doc/docx file. It also has configuration options to let you specify alternate diff tools based on file extension, which is pretty cool. (It's a shame no one has made a diff tool for our CAD package).

Current-generation DVCSes like Git or Hg tend to suck with binary files though. They don't have any sort of mechanism for file locking.

There exist binary diff tools, however they don't help much, since the change in one pixel of an image, or a change of one character in a Word document, does not correspond to change of one byte in the file, due to compression. Thus "nice" handling of such binary data is impossible.

If you want to commit such documents, consider committing uncompressed variants - RTF instead of DOC, TeX instead of PDF, etc. If the version control system employs compression to compress its internal repository, then this method should work rather well. For instance, in Git,

Newly added objects are stored in their entirety using zlib compression.

EDIT: I just wanted to note that even RTF is horrible, but not as horrible as DOC. If you can switch to TXT or TeX for your documents, that would be best.

See the mercurial wiki page about Binary files. Your main problem is that even minor changes in files such as doc and others will triger large changes in the file structure (partly because it's zipped).

Therefore, I don't believe you'll find any nice way to handle these files in a version control system.