Issue1876

Title Push traffic doubled over authenticated HTTP/S causes inefficiency or failure
Priority bug Status resolved
Superseder Nosy List BradOlson, abuehl, durin42, kiilerix, mpm, tonfa
Assigned To durin42 Topics 1.2.1, 1.3.1, http_proto, performance, push, ssh_proto

Created on 2009-10-17.15:46:29 by BradOlson, last changed 2011-09-12.22:00:07 by mpm.

Files
File name Uploaded Type Edit Remove
unnamed BradOlson, 2010-03-29.11:19:52 text/html
Messages
msg16683 (view) Author: durin42 Date: 2011-07-03.15:13:19
For those following this issue:

Mercurial 1.9 includes my ground-up rewrite of httplib, which you can enable
with ui.usehttp2. When usehttp2 is enabled and you're talking to a 1.9-based
http(s) server, 100-continue is properly used and should avoid the
double-send of the bundle data in (nearly) all cases. If you find defects
(either in the new continue support or regressions caused by usehttp2),
please report them and assign the bug to me.

Thanks!
msg13975 (view) Author: mpm Date: 2010-10-11.21:37:24
Degrading to bug.
msg12186 (view) Author: BradOlson Date: 2010-03-29.11:19:52
Augie,

Thanks for all your work on this. I'll take a look at the queues and talk to
my lead about my assisting with testing.

On Sun, Mar 28, 2010 at 8:53 PM, Augie Fackler
<bugs@mercurial.selenic.com>wrote:

>
> Augie Fackler <durin42@gmail.com> added the comment:
>
> I just posted a status update to the hg list. Linking here for sanity.
> http://article.gmane.org/gmane.comp.version-control.mercurial.devel/29897
>
> ____________________________________________________
> Mercurial issue tracker <bugs@mercurial.selenic.com>
> <http://mercurial.selenic.com/bts/issue1876>
> ____________________________________________________
>
msg12179 (view) Author: durin42 Date: 2010-03-29.00:53:31
I just posted a status update to the hg list. Linking here for sanity. 
http://article.gmane.org/gmane.comp.version-control.mercurial.devel/29897
msg11355 (view) Author: mpm Date: 2010-01-01.22:45:45
Interesting. Bear in mind that we'll probably be wanting to revamp our SSL
support to not suck.
msg11353 (view) Author: durin42 Date: 2010-01-01.22:28:23
The cleanest best path appears to be to fix (that is, rewrite most of) 
httplib. I'm planning on taking 20% time for Q1 to work on that.

The other option is to monkeypatch httplib heavily, which is fragile at best. 
I spent 3 days or so on it, and came to the conclusion that we'd be better off 
making a clean async-safe httplib replacement instead.
msg11351 (view) Author: mpm Date: 2010-01-01.22:25:00
What's the status with this one?
msg10827 (view) Author: durin42 Date: 2009-10-22.16:21:17
I've got the beginnings of a client-side fix for this as well, it seems to be 
working at a basic level.
msg10825 (view) Author: durin42 Date: 2009-10-21.20:20:20
hgweb actually has code to handle this (look for req.drain() in the code). 
It's inelegant, but server implementations can do that as a stopgap until hg 
is fixed.

I'm going to try and spend 1 day a week on this until it gets fixed.
msg10799 (view) Author: durin42 Date: 2009-10-17.21:48:22
I don't know that it got discussed in irc. I submitted a preliminary version 
of the patch to the maillist, but I want to improve it some (catch other 
errors, like "ssl required") before resubmitting it again. Now that I've 
gotten single-dir clones done for hgsubversion, I'll be revisiting the patch.
msg10798 (view) Author: BradOlson Date: 2009-10-17.21:46:33
Thanks tonfa, sorry I missed this one on irc. Is that conversation stored
anywhere?

Don't think Augie has a full fix for this yet.

The small request to trigger auth is clever. It would alleviate failures due
to a server closing the pipe on an unauthenticated body. But my read on the
code says the next request would also have to re-authenticate. So a smaller
bundles would stand a better chance of not failing, but still take twice the
traffic on many servers.
msg10795 (view) Author: tonfa Date: 2009-10-17.17:22:20
Putting Augie in the nosy list, since he worked on a solution to this 
(sending an almost empty bundle to trigger the auth).
msg10794 (view) Author: tonfa Date: 2009-10-17.17:14:49
Is it the same effort that the one discussed on irc some time ago?
msg10793 (view) Author: BradOlson Date: 2009-10-17.15:46:29
The behavior of mercurial.url, mercurial.keepalive, and urllib2 have every
request sent twice to http servers requiring authentication.

This behavior can cause failures or huge (>8mb documented) re-transmissions
when the 'hg push' sends the 'unbundle' command.

Details and fix suggestions at bitbucket.org/bradobro/liquidhg/wiki/TheFixes.
History
Date User Action Args
2011-09-12 22:00:07mpmsetstatus: testing -> resolved
nosy: mpm, tonfa, kiilerix, abuehl, durin42, BradOlson
2011-07-03 15:13:19durin42setstatus: in-progress -> testing
nosy: mpm, tonfa, kiilerix, abuehl, durin42, BradOlson
messages: + msg16683
2011-03-17 18:04:02kiilerixsetnosy: + kiilerix
2010-10-11 21:37:24mpmsetpriority: urgent -> bug
nosy: mpm, tonfa, abuehl, durin42, BradOlson
messages: + msg13975
2010-03-29 11:19:52BradOlsonsetfiles: + unnamed
nosy: mpm, tonfa, abuehl, durin42, BradOlson
messages: + msg12186
2010-03-29 00:53:31durin42setnosy: mpm, tonfa, abuehl, durin42, BradOlson
messages: + msg12179
2010-01-01 22:45:45mpmsetnosy: mpm, tonfa, abuehl, durin42, BradOlson
messages: + msg11355
2010-01-01 22:28:23durin42setnosy: mpm, tonfa, abuehl, durin42, BradOlson
messages: + msg11353
2010-01-01 22:25:00mpmsetnosy: + mpm
messages: + msg11351
2009-10-22 16:22:38durin42setstatus: chatting -> in-progress
nosy: tonfa, abuehl, durin42, BradOlson
assignedto: durin42
2009-10-22 16:21:17durin42setnosy: tonfa, abuehl, durin42, BradOlson
messages: + msg10827
2009-10-21 20:20:20durin42setnosy: tonfa, abuehl, durin42, BradOlson
messages: + msg10825
2009-10-17 21:48:23durin42setnosy: tonfa, abuehl, durin42, BradOlson
messages: + msg10799
2009-10-17 21:46:33BradOlsonsetnosy: tonfa, abuehl, durin42, BradOlson
messages: + msg10798
2009-10-17 17:22:21tonfasetnosy: + durin42
messages: + msg10795
2009-10-17 17:14:49tonfasetstatus: unread -> chatting
nosy: tonfa, abuehl, BradOlson
messages: + msg10794
2009-10-17 17:14:20tonfasetnosy: + tonfa
2009-10-17 15:55:03abuehlsetnosy: + abuehl
2009-10-17 15:46:29BradOlsoncreate