Issue1788

Title Mac OS dot file problem
Priority bug Status chatting
Superseder Nosy List MikeHoward, danchr, dhaun, djc, durin42, kiilerix, man8, mg, pmezard, smarks, tonfa, wjv
Assigned To Topics

Created on 2009-08-10.20:52:20 by MikeHoward, last changed 2011-05-03.09:00:34 by wjv.

Messages
msg16134 (view) Author: wjv Date: 2011-05-03.09:00:34
I see!  Yes, you are quite correct -- in my "fixed" (via the convert 
extension) the revlog of the file in question would no longer fall prey to 
Time Machine's vagaries.  Thanks for the clarification, and apologies for the 
extra noise I caused.
msg16133 (view) Author: mg Date: 2011-05-03.08:10:52
Johann: even if you use a new version of Mercurial, old repositories wont
upgrade themselves automatically.

The on-disk format of a repository is determined when you create it. So a
new repository created with 'hg init' and a post-1.7 Mercurial will use the
dotencode format (look in .hg/requires to confirm this).

Clones are a little more tricky: 'hg clone foo bar' will create a bar clone
with the same on-disk format as foo because of hardlinks. You need to use
'hg clone --pull foo bar' to make bar a new format clone with no hardlinks
back to foo.
msg16130 (view) Author: wjv Date: 2011-05-02.19:49:10
I can confirm the issue with Mercurial 1.8.2 under OS X 10.6.7.

I have recently restored a hard drive from a Time Machine backup, and a 
repository that was located on said drive was corrupted after the restore.

It appears that the one particular file's revlog was just completely absent 
after the restore.  The file in question had a name starting with a dot, 
followed by a capital letter (say, ".Foo").  Hence, its revlog had a name 
starting in ._* (say, "._foo").

It appears that currently Time Machine DOES NOT BACK UP files starting in ._* 
at all!  I searched back through my time machine backups, but this particular 
revlog was never present.  (I have yet to create such a file on purpose and 
see what Time Machine does.)

One would assume that Time Machine views a file starting in ._* as an 
AppleDouble [1][2] and hence simply neglects to back it up.

For the record, I managed to recover by using the convert extension as 
described at [3].  (Fortunately it was only a private repository.)

This is clearly an Apple / Time Machine issue, not a Mercurial one.  But I 
thought I'd respond here, as it's probably more likely to get the Mercurial 
folks to come up with a workaround than it is to get Apple to change their 
ways...

[1] http://en.wikipedia.org/wiki/AppleSingle_and_AppleDouble_formats
[2] http://support.apple.com/kb/TA20578
[3] http://mercurial.selenic.com/wiki/RepositoryCorruption
msg15624 (view) Author: dhaun Date: 2011-03-13.19:17:28
I was running hg 1.7.5 before the disk died but the repositories may have
been created with earlier versions.
msg15623 (view) Author: durin42 Date: 2011-03-13.18:59:30
I'm fairly certain dotencode fixes this. It was released as part of hg 1.7.
msg15622 (view) Author: dhaun Date: 2011-03-13.18:52:03
Just ran into this issue after having to replace a failed hard drive. At
first it looked like Time Machine restored everything just fine, but a hg
verify shows that all my local repositories are now damaged. It would be
really nice if this could be addressed in some future Mercurial version ...
msg12110 (view) Author: tonfa Date: 2010-03-20.21:34:51
I'm wrong, the decodefunction doesn't really matter, sorry.
msg12109 (view) Author: tonfa Date: 2010-03-20.21:10:13
We should have done something like that in the first place:

diff --git a/mercurial/store.py b/mercurial/store.py
--- a/mercurial/store.py
+++ b/mercurial/store.py
@@ -51,7 +51,15 @@
                 except KeyError:
                     pass
             else:
-                raise KeyError
+                if s[i] != '~':
+                    raise KeyError
+
+                try:
+                    val = int(s[i + 1: i + 3], 10)
+                except ValueError:
+                    raise KeyError
+                yield chr(val)
+                i += 3
     return (lambda s: "".join([cmap[c] for c in encodedir(s)]),
             lambda s: decodedir("".join(list(decode(s)))))
 

That would allow us to add new patterns to encode
msg12036 (view) Author: danchr Date: 2010-03-12.22:16:09
In order to fix this bug, it would seem so. On the Mercurial list, Adrian 
Buehlmann mentioned a similar Windows bug, filed as issue1713.

I don't know much about how fncache actually works, but it seems to me that it 
would be preferable to avoid ever creating files whose names begin with a dot 
or a space. While ‘.[^_]*’ should be safe, such files are normally hidden in 
shells, and that seems inappropriate for the store.
msg11845 (view) Author: djc Date: 2010-02-22.19:49:10
Ugh, we have to rev the store version again?
msg10312 (view) Author: danchr Date: 2009-08-11.07:19:22
I'd consider this a bug in Mercurial; dot-underscore files are generally 
treated specially on Mac OS X, and the ‘fncache’ repository storage shouldn't 
generate those.

I took the liberty of changing the priority to »bug«.
msg10311 (view) Author: MikeHoward Date: 2009-08-11.00:51:10
That makes a lot of sense and is consistent with what I've seen. It probably
makes sense for mercurial to automatically ignore 'resource fork' files
since they don't contain user (directly) created data. It would make the
repository more robust.

I have a call in to Apple to try to find out what they are actually backing
up via Time Machine - where I don't see any dot files of any kind.
msg10309 (view) Author: smarks Date: 2009-08-10.23:45:13
The Mac stores the "resource fork" of files in a file prefixed with "._", on
filesystems that don't support resource forks natively (e.g. SMB). See

http://en.wikipedia.org/wiki/Resource_fork

In general dot-files should be OK, however, there are probably a bunch of
special cases in the Mac code where the "._" prefix is treated specially.
msg10308 (view) Author: MikeHoward Date: 2009-08-10.20:52:19
This is a low frequency event, but a real pain in the butt.

At least one of my hg repositories was corrupted after purchasing a new
macbook and using Apple's automatic copy-your-old-system-to-your-new-one.

Specifically, about 6 files beginning with ._ are missing. These are mostly
named ._ds_store.i and ._bridgesort. [they probably shouldn't have been
there to begin with, but I was new to mercurial]. They are in a backup I
made using 'cp'. [All other dot files seem to be OK - so it's obviously
something in Apple's strange mind]

I tried restoring them by copying from the backup copy on the external disk,
but 'hg verify' is still upset.

Suggest considering an alternate naming convention which avoids leading dot's.

I've a call into Apple to try to find out what is going on from their point
of view.
History
Date User Action Args
2011-05-03 09:00:34wjvsetnosy: tonfa, pmezard, smarks, kiilerix, mg, djc, durin42, danchr, MikeHoward, man8, dhaun, wjv
messages: + msg16134
2011-05-03 08:10:52mgsetnosy: tonfa, pmezard, smarks, kiilerix, mg, djc, durin42, danchr, MikeHoward, man8, dhaun, wjv
messages: + msg16133
2011-05-02 19:49:10wjvsetnosy: + wjv
messages: + msg16130
2011-03-13 19:17:28dhaunsetnosy: tonfa, pmezard, smarks, kiilerix, mg, djc, durin42, danchr, MikeHoward, man8, dhaun
messages: + msg15624
2011-03-13 18:59:30durin42setnosy: tonfa, pmezard, smarks, kiilerix, mg, djc, durin42, danchr, MikeHoward, man8, dhaun
messages: + msg15623
2011-03-13 18:52:03dhaunsetnosy: + dhaun
messages: + msg15622
2010-07-22 08:36:53mgsetnosy: + mg
2010-07-22 00:06:22kiilerixsetnosy: + kiilerix
2010-05-28 16:40:19man8setnosy: + man8
2010-03-20 21:34:51tonfasetnosy: tonfa, pmezard, smarks, djc, durin42, danchr, MikeHoward
messages: + msg12110
2010-03-20 21:10:13tonfasetnosy: tonfa, pmezard, smarks, djc, durin42, danchr, MikeHoward
messages: + msg12109
2010-03-20 20:52:58tonfasetnosy: + tonfa
2010-03-20 20:52:19durin42setnosy: + durin42
2010-03-12 22:16:09danchrsetnosy: pmezard, smarks, djc, danchr, MikeHoward
messages: + msg12036
2010-02-22 19:49:10djcsetnosy: + djc
messages: + msg11845
2009-08-23 10:17:30pmezardsetnosy: + pmezard
2009-08-11 07:19:22danchrsetpriority: wish -> bug
nosy: + danchr
messages: + msg10312
2009-08-11 00:51:11MikeHowardsetnosy: smarks, MikeHoward
messages: + msg10311
2009-08-10 23:45:13smarkssetstatus: unread -> chatting
nosy: + smarks
messages: + msg10309
2009-08-10 20:52:20MikeHowardcreate