I encountered this issue in a repository which was created on Linux,
containing a file and a symlink pointing to this file, amongst other things.

It was cloned on Windows and because the symlink became an editable file,
the Windows user (using an older Mercurial, I think 1.7.x) changed this file
(in this case png image, so probably containing NUL bytes) and committed it.

Now this new revision can't be checked out on the Linux side (using
Mercurial 1.9 or 1.8.4), yielding a traceback similar to the other reports:

$ hg co -C tip
** unknown exception encountered, please report by visiting
**  http://mercurial.selenic.com/wiki/BugTracker
** Python 2.5.2 (r252:60911, Jan 24 2010, 14:53:14) [GCC 4.3.2]
** Mercurial Distributed SCM (version 1.9+20-29d324c2bc93)
** Extensions loaded: foo, mq, patchbomb, convert, transplant, fetch, churn,
keyword, nearest, inout, rebase, relink
Traceback (most recent call last):
  File "/home/thomas/bin/hg", line 38, in <module>
    mercurial.dispatch.run()
  File "/home/thomas/hg/repos/tah/mercurial/dispatch.py", line 27, in run
    sys.exit(dispatch(request(sys.argv[1:])))
  File "/home/thomas/hg/repos/tah/mercurial/dispatch.py", line 64, in dispatch
    return _runcatch(req)
  File "/home/thomas/hg/repos/tah/mercurial/dispatch.py", line 87, in _runcatch
    return _dispatch(req)
  File "/home/thomas/hg/repos/tah/mercurial/dispatch.py", line 675, in _dispatch
    cmdpats, cmdoptions)
  File "/home/thomas/hg/repos/tah/mercurial/dispatch.py", line 454, in
runcommand
    ret = _runcommand(ui, options, cmd, d)
  File "/home/thomas/hg/repos/tah/mercurial/dispatch.py", line 729, in
_runcommand
    return checkargs()
  File "/home/thomas/hg/repos/tah/mercurial/dispatch.py", line 683, in checkargs
    return cmdfunc()
  File "/home/thomas/hg/repos/tah/mercurial/dispatch.py", line 672, in <lambda>
    d = lambda: util.checksignature(func)(ui, *args, **cmdoptions)
  File "/home/thomas/hg/repos/tah/mercurial/util.py", line 385, in check
    return func(*args, **kwargs)
  File "/home/thomas/hg/repos/tah/mercurial/extensions.py", line 137, in wrap
    util.checksignature(origfn), *args, **kwargs)
  File "/home/thomas/hg/repos/tah/mercurial/util.py", line 385, in check
    return func(*args, **kwargs)
  File "/home/thomas/hg/repos/tah/hgext/mq.py", line 3218, in mqcommand
    return orig(ui, repo, *args, **kwargs)
  File "/home/thomas/hg/repos/tah/mercurial/util.py", line 385, in check
    return func(*args, **kwargs)
  File "/home/thomas/hg/repos/tah/mercurial/commands.py", line 5122, in update
    ret = hg.clean(repo, rev)
  File "/home/thomas/hg/repos/tah/mercurial/hg.py", line 406, in clean
    stats = mergemod.update(repo, node, False, True, None)
  File "/home/thomas/hg/repos/tah/mercurial/merge.py", line 554, in update
    stats = applyupdates(repo, action, wc, p2, pa, overwrite)
  File "/home/thomas/hg/repos/tah/mercurial/merge.py", line 351, in applyupdates
    repo.wwrite(f, t, flags)
  File "/home/thomas/hg/repos/tah/mercurial/localrepo.py", line 659, in wwrite
    self.wopener.symlink(data, filename)
  File "/home/thomas/hg/repos/tah/mercurial/scmutil.py", line 251, in symlink
    os.symlink(src, linkname)
TypeError: symlink() argument 1 must be (encoded string without NULL bytes),
not str


I think a current Mercurial should fallback to extracting a plain file in
this case, i.e. the same that would happen if a Windows clients does the
checkout.
Unlike the Windows client, it should print a warning and note in the
dirstate that this now is a file instead of a symlink. Perhaps the warning
should indicate that the files should be examined and committed if applicable.

I can reproduce the same (or at least very similar) backtrace when I do an
hg fetch in a local repository after having changed the default path from
one remote repository to another (which originally came from the same svn,
but with a later conversion of the whole repo, don't ask!).

pulling from ssh://hg@somewhere.com/somerepo
searching for changes
adding changesets
adding manifests
adding file changes
added 3718 changesets with 1245 changes to 2702 files (+1 heads)
updating to 7445:fdb03cb47d59
92 files updated, 0 files merged, 141 files removed, 0 files unresolved
merging with 3727:06161d6baaa7
merging afile.cpp
merging afile.h
warning: conflicts during merge.
merging afile.h failed!
merging amakefile.mak
merging anothermakefile.mak
merging yetanothermakefile.mak
merging fromscratch/Make.q.mac
merging fromscratch/Make.w32
merging fromscratch/checkout.mak
merging fromscratch/checkout.q.mak
** unknown exception encountered, please report by visiting
**  http://mercurial.selenic.com/wiki/BugTracker
** Python 2.6.6 (r266:84292, Sep 24 2010, 15:31:10) [GCC 4.2.1 (Apple Inc.
build 5664)]
** Mercurial Distributed SCM (version 1.7.1)
** Extensions loaded: fetch
Traceback (most recent call last):
  File "/opt/local/bin/hg", line 38, in <module>
    mercurial.dispatch.run()
  File
"/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/mercurial/dispatch.py",
line 16, in run
    sys.exit(dispatch(sys.argv[1:]))
  File
"/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/mercurial/dispatch.py",
line 36, in dispatch
    return _runcatch(u, args)
  File
"/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/mercurial/dispatch.py",
line 58, in _runcatch
    return _dispatch(ui, args)
  File
"/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/mercurial/dispatch.py",
line 590, in _dispatch
    cmdpats, cmdoptions)
  File
"/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/mercurial/dispatch.py",
line 401, in runcommand
    ret = _runcommand(ui, options, cmd, d)
  File
"/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/mercurial/dispatch.py",
line 641, in _runcommand
    return checkargs()
  File
"/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/mercurial/dispatch.py",
line 595, in checkargs
    return cmdfunc()
  File
"/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/mercurial/dispatch.py",
line 588, in <lambda>
    d = lambda: util.checksignature(func)(ui, *args, **cmdoptions)
  File
"/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/mercurial/util.py",
line 427, in check
    return func(*args, **kwargs)
  File
"/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/hgext/fetch.py",
line 122, in fetch
    err = hg.merge(repo, secondparent, remind=False)
  File
"/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/mercurial/hg.py",
line 402, in merge
    stats = mergemod.update(repo, node, True, force, False)
  File
"/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/mercurial/merge.py",
line 532, in update
    stats = applyupdates(repo, action, wc, p2, pa)
  File
"/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/mercurial/merge.py",
line 323, in applyupdates
    r = ms.resolve(fd, wctx, mctx)
  File
"/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/mercurial/merge.py",
line 69, in resolve
    self._repo.wwrite(dfile, f.read(), flags)
  File
"/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/mercurial/localrepo.py",
line 625, in wwrite
    self.wopener.symlink(data, filename)
  File
"/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/mercurial/util.py",
line 936, in symlink
    os.symlink(src, linkname)
TypeError: symlink() argument 1 must be encoded string without NULL bytes,
not str

Degrading to bug. We should probably refuse to commit things that claim to
be symlinks and are either very long and/or contain NULs.

The initial problem was a commit of symbolic links from windows in a working
repo created on linux and accessed via CIFS/SMB.
(This problem is already described in Issue 1888!)

Most of the links were pointing to text files, where the content where used
as symlink target (creating an invalid link or aborting with "File name too
long")
This is part 1 of the bug.
If possible, do a check for valid file names and allow an update with
ignoring this problem.

The second part is more critical, as it provokes an exception:
One of the links were pointing to a binary file and using the file content
as target of the symbolic link was resulting in an exception in the python
code. This is more critical, because of the exception no indication to the
corrupted file/symlink is printed, which might help to fix the issue (by
committing changes to a local repo and merging)

Log of the exception:

pulling subrepo drivers
requesting all changes
adding changesets
adding manifests
adding file changes
added 16 changesets with 272 changes to 185 files
** unknown exception encountered, details follow
** report bug details to http://mercurial.selenic.com/bts/
** or mercurial@selenic.com
** Mercurial Distributed SCM (version 1.4.3)
** Extensions loaded:
Traceback (most recent call last):
  File "/opt/mercurial_1.4/mercurial_1.4.3/bin/hg", line 27, in <module>
    mercurial.dispatch.run()
  File
"/opt/mercurial_1.4/mercurial_1.4.3/linux30_64/lib/python2.6/site-packages/mercurial/dispatch.py",
line 16, in run
    sys.exit(dispatch(sys.argv[1:]))
  File
"/opt/mercurial_1.4/mercurial_1.4.3/linux30_64/lib/python2.6/site-packages/mercurial/dispatch.py",
line 30, in dispatch
    return _runcatch(u, args)
  File
"/opt/mercurial_1.4/mercurial_1.4.3/linux30_64/lib/python2.6/site-packages/mercurial/dispatch.py",
line 46, in _runcatch
    return _dispatch(ui, args)
  File
"/opt/mercurial_1.4/mercurial_1.4.3/linux30_64/lib/python2.6/site-packages/mercurial/dispatch.py",
line 454, in _dispatch
    return runcommand(lui, repo, cmd, fullargs, ui, options, d)
  File
"/opt/mercurial_1.4/mercurial_1.4.3/linux30_64/lib/python2.6/site-packages/mercurial/dispatch.py",
line 324, in runcommand
    ret = _runcommand(ui, options, cmd, d)
  File
"/opt/mercurial_1.4/mercurial_1.4.3/linux30_64/lib/python2.6/site-packages/mercurial/dispatch.py",
line 505, in _runcommand
    return checkargs()
  File
"/opt/mercurial_1.4/mercurial_1.4.3/linux30_64/lib/python2.6/site-packages/mercurial/dispatch.py",
line 459, in checkargs
    return cmdfunc()
  File
"/opt/mercurial_1.4/mercurial_1.4.3/linux30_64/lib/python2.6/site-packages/mercurial/dispatch.py",
line 453, in <lambda>
    d = lambda: util.checksignature(func)(ui, *args, **cmdoptions)
  File
"/opt/mercurial_1.4/mercurial_1.4.3/linux30_64/lib/python2.6/site-packages/mercurial/util.py",
line 386, in check
    return func(*args, **kwargs)
  File
"/opt/mercurial_1.4/mercurial_1.4.3/linux30_64/lib/python2.6/site-packages/mercurial/commands.py",
line 649, in clone
    update=opts.get('updaterev') or not opts.get('noupdate'))
  File
"/opt/mercurial_1.4/mercurial_1.4.3/linux30_64/lib/python2.6/site-packages/mercurial/hg.py",
line 325, in clone
    _update(dest_repo, uprev)
  File
"/opt/mercurial_1.4/mercurial_1.4.3/linux30_64/lib/python2.6/site-packages/mercurial/hg.py",
line 339, in update
    stats = _merge.update(repo, node, False, False, None)
  File
"/opt/mercurial_1.4/mercurial_1.4.3/linux30_64/lib/python2.6/site-packages/mercurial/merge.py",
line 501, in update
    stats = applyupdates(repo, action, wc, p2)
  File
"/opt/mercurial_1.4/mercurial_1.4.3/linux30_64/lib/python2.6/site-packages/mercurial/merge.py",
line 318, in applyupdates
    subrepo.submerge(repo, wctx, mctx, wctx)
  File
"/opt/mercurial_1.4/mercurial_1.4.3/linux30_64/lib/python2.6/site-packages/mercurial/subrepo.py",
line 108, in submerge
    wctx.sub(s).get(r)
  File
"/opt/mercurial_1.4/mercurial_1.4.3/linux30_64/lib/python2.6/site-packages/mercurial/subrepo.py",
line 206, in get
    hg.clean(self._repo, revision, False)
  File
"/opt/mercurial_1.4/mercurial_1.4.3/linux30_64/lib/python2.6/site-packages/mercurial/hg.py",
line 350, in clean
    stats = _merge.update(repo, node, False, True, None)
  File
"/opt/mercurial_1.4/mercurial_1.4.3/linux30_64/lib/python2.6/site-packages/mercurial/merge.py",
line 501, in update
    stats = applyupdates(repo, action, wc, p2)
  File
"/opt/mercurial_1.4/mercurial_1.4.3/linux30_64/lib/python2.6/site-packages/mercurial/merge.py",
line 315, in applyupdates
    repo.wwrite(f, t, flags)
  File
"/opt/mercurial_1.4/mercurial_1.4.3/linux30_64/lib/python2.6/site-packages/mercurial/localrepo.py",
line 543, in wwrite
    self.wopener.symlink(data, filename)
  File
"/opt/mercurial_1.4/mercurial_1.4.3/linux30_64/lib/python2.6/site-packages/mercurial/util.py",
line 847, in symlink
    os.symlink(src, linkname)
TypeError: symlink() argument 1 must be encoded string without NULL bytes,
not str