Binary File Handling
Mercurial generally makes no assumptions about file contents. Thus, most things in Mercurial work fine with any type of file.
The exceptions are commands like diff, export, and annotate, that work well on files intended to be read by humans, and merge, where processing binary files makes very little sense at all.
The question naturally arises, what is a binary file anyway? It turns out there's really no good answer to this question, so Mercurial uses the same heuristic that programs like diff(1) use. The test is simply if there are any NUL characters in the first 1K or so of a file.
For diff, export, and annotate, this will get things right almost all of the time and it will not attempt to process files it thinks are binary. If necessary, you can force these commands to treat files as text with -a.
Merging is another matter. The actual merging of individual files in Mercurial is handled entirely by external programs and Mercurial doesn't pretend to tell these programs what files they can and cannot merge.
The example merge script hgmerge currently makes no attempt to do anything special for various file types, but it could easily be extended to do so. But precisely what you would want to do with these files will depend on the specific file type and your project needs.