For test purposes I wanted to clone the last commit only by using the --depth option as in
/work/tmp/ # git clone --depth=10 https://gerrit.wikimedia.org/r/p/mediawiki/core.git
and got this error message:
Initialized empty Git repository in /work/tmp/core/.git/ *error: RPC failed; result=22, HTTP code = 500*
My git client version is 1.7.1
Any idea why ?
On Thu, 12 Apr 2012 23:32:03 -0700, Thomas Gries mail@tgries.de wrote:
For test purposes I wanted to clone the last commit only by using the --depth option as in
/work/tmp/ # git clone --depth=10 https://gerrit.wikimedia.org/r/p/mediawiki/core.git
and got this error message:
Initialized empty Git repository in /work/tmp/core/.git/ *error: RPC failed; result=22, HTTP code = 500*
My git client version is 1.7.1
Any idea why ?
I already tried that myself awhile ago. Judging by the fact that I get that on Gerrit but it works when --depth from my mediawiki-core on GitHub; Gerrit's faux SSH git hack doesn't implement whatever is necessary for git to do --depth.
By the way, using --depth only saves you disk space (which is a trivially small amount, which linked repositories can cut back on as well). When you use --depth git still needs to go through the whole history. So whether you are using everything or try to limit by --depth it still takes the same amount of time to do an initial clone.
On 13/04/12 10:29, Daniel Friesen wrote:
By the way, using --depth only saves you disk space (which is a trivially small amount,
I disagree, 18M vs 341M is quite noticeable.
which linked repositories can cut back on as well).
Only if you're cloning on the same filesystem. We're talking about a network operation here.
When you use --depth git still needs to go through the whole history. So whether you are using everything or try to limit by --depth it still takes the same amount of time to do an initial clone.
You're not counting the bandwidth needs. Transfering the whole amount is probably the slowest operation, not the CPU time needed to traverse the tree.
On Fri, 13 Apr 2012 09:37:47 -0700, Platonides Platonides@gmail.com wrote:
On 13/04/12 10:29, Daniel Friesen wrote:
By the way, using --depth only saves you disk space (which is a trivially small amount,
I disagree, 18M vs 341M is quite noticeable.
which linked repositories can cut back on as well).
Only if you're cloning on the same filesystem. We're talking about a network operation here.
When you use --depth git still needs to go through the whole history. So whether you are using everything or try to limit by --depth it still takes the same amount of time to do an initial clone.
You're not counting the bandwidth needs. Transfering the whole amount is probably the slowest operation, not the CPU time needed to traverse the tree.
I was a little wrong about time, it does save a little bit of time:
Daniels-MacBook-Air:test daniel$ time git clone https://dantman@github.com/dantman/mediawiki-core.git a Cloning into 'a'... remote: Counting objects: 398899, done. remote: Compressing objects: 100% (52956/52956), done. remote: Total 398899 (delta 345088), reused 398821 (delta 345010) Receiving objects: 100% (398899/398899), 109.91 MiB | 558 KiB/s, done. Resolving deltas: 100% (345088/345088), done.
real 4m49.148s user 1m51.726s sys 0m7.310s
Daniels-MacBook-Air:test daniel$ time git clone --depth=5 https://dantman@github.com/dantman/mediawiki-core.git b Cloning into 'b'... remote: Counting objects: 32122, done. remote: Compressing objects: 100% (18059/18059), done. remote: Total 32122 (delta 26125), reused 18294 (delta 13859) Receiving objects: 100% (32122/32122), 48.25 MiB | 648 KiB/s, done. Resolving deltas: 100% (26125/26125), done.
real 2m35.719s user 0m13.701s sys 0m2.041s
However I don't get where you are getting those numbers on space, they look mistaken or biased. This is the disk usage of a plain clone and a --depth=5 clone: Daniels-MacBook-Air:test daniel$ du -hs * 189M a 117M b
By linked repositories I'm talking about the situation where you have multiple copies of the repo sitting around. 200MB is nothing when you have only one repo, it's not even worth typing --depth and the potential downsides you could end up with. But in the case of multiple copies of the same repo you can save a HUGE amount of space by making a --bare clone of the remote with the whole history. And then cloning your other repos from that one. You kill the extra network ops of course. But by using a filesystem path you also trigger --local which makes git hardlink objects. If you think you can use it safely there is also --shared which will completely skip hardlinking or copying of objects and use them all directly from the parent repo.
On 14/04/12 08:42, Daniel Friesen wrote:
I was a little wrong about time, it does save a little bit of time:
Daniels-MacBook-Air:test daniel$ time git clone https://dantman@github.com/dantman/mediawiki-core.git a Cloning into 'a'... remote: Counting objects: 398899, done. remote: Compressing objects: 100% (52956/52956), done. remote: Total 398899 (delta 345088), reused 398821 (delta 345010) Receiving objects: 100% (398899/398899), 109.91 MiB | 558 KiB/s, done. Resolving deltas: 100% (345088/345088), done.
real 4m49.148s user 1m51.726s sys 0m7.310s
Daniels-MacBook-Air:test daniel$ time git clone --depth=5 https://dantman@github.com/dantman/mediawiki-core.git b Cloning into 'b'... remote: Counting objects: 32122, done. remote: Compressing objects: 100% (18059/18059), done. remote: Total 32122 (delta 26125), reused 18294 (delta 13859) Receiving objects: 100% (32122/32122), 48.25 MiB | 648 KiB/s, done. Resolving deltas: 100% (26125/26125), done.
real 2m35.719s user 0m13.701s sys 0m2.041s
However I don't get where you are getting those numbers on space, they look mistaken or biased. This is the disk usage of a plain clone and a --depth=5 clone: Daniels-MacBook-Air:test daniel$ du -hs * 189M a 117M b
$ time git clone file://mediawiki/core core-full Cloning into 'core-full'... remote: Counting objects: 356100, done. remote: Compressing objects: 100% (60989/60989), done. Receiving objects: 100% (356100/356100), 121.81 MiB | 12.30 MiB/s, done. remote: Total 356100 (delta 301263), reused 348780 (delta 294195) Resolving deltas: 100% (301263/301263), done.
real 1m43.218s user 1m38.904s sys 0m3.693s
$ time git clone --depth 1 file:///home/repos/git/mw/mediawiki/core core-shallow Cloning into 'core-shallow'... remote: Counting objects: 2670, done. remote: Compressing objects: 100% (2504/2504), done. remote: Total 2670 (delta 273), reused 1372 (delta 119) Receiving objects: 100% (2670/2670), 16.55 MiB | 4.30 MiB/s, done. Resolving deltas: 100% (273/273), done.
real 0m7.022s user 0m7.216s sys 0m0.493s
$ du -sh core-full/.git core-shallow/.git 132M core-full/.git 18M core-shallow/.git
Using --depth 5 vs --depth 1 doesn't make much difference: 19M core-shallow5/.git
Also counting the checkout adds a constant amount: $ du -sh core-full core-shallow core-shallow5 199M core-full 84M core-shallow 85M core-shallow5
I don't know why your depthed clone was so big. Could be related to https using the dumb protocol, but still, it's a difference too big.
So I tried with your parameters:
$ time git clone https://github.com/dantman/mediawiki-core.git a Cloning into 'a'... remote: Counting objects: 398899, done. remote: Compressing objects: 100% (52956/52956), done. remote: Total 398899 (delta 345088), reused 398821 (delta 345010) Receiving objects: 100% (398899/398899), 109.91 MiB | 164 KiB/s, done. Resolving deltas: 100% (345088/345088), done.
real 7m54.988s user 1m5.466s sys 0m4.456s
$ time git clone --depth 5 https://github.com/dantman/mediawiki-core.git b Cloning into 'b'... remote: Counting objects: 2705, done. remote: Compressing objects: 100% (2487/2487), done. remote: Total 2705 (delta 264), reused 1628 (delta 171) Receiving objects: 100% (2705/2705), 17.22 MiB | 294 KiB/s, done. Resolving deltas: 100% (264/264), done.
real 1m6.691s user 0m1.603s sys 0m0.573s
$ du -sh a b 191M a 88M b
$ du -sh a/.git b/.git 122M a/.git 18M b/.git
Note that the .git folder in the shallow case is just 18M and I only needed to download 17M from the net (why were them 48.25M for you?). For reference, the github clone was at 76dd5fe50aa8f3ed3f436ce737ab339f85e0c63e and mediawiki/core at 76881580cd9c30681aa65b228e0757c0515bbaa5 (288 commits more) when I performed these tests.
https://bugzilla.wikimedia.org/show_bug.cgi?id=35949 I think, this issue should be solved as soon as possible to allow everyone to download the latest core version in a compact and bandwidth-efficient way.
If you read my comments on that bug you will see I believe it is an upstream issue and is probably fixed in 2.3.
-Chad On Apr 14, 2012 12:31 PM, "Thomas Gries" mail@tgries.de wrote:
https://bugzilla.wikimedia.org/show_bug.cgi?id=35949 I think, this issue should be solved as soon as possible to allow everyone to download the latest core version in a compact and bandwidth-efficient way.
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Am 14.04.2012 20:17, schrieb Chad:
If you read my comments on that bug you will see I believe it is an upstream issue and is probably fixed in 2.3.
-Chad
Yes, but see https://code.google.com/p/gerrit/issues/detail?id=769 . The questions are: who is upgrading the software, and when ?
Comment by y Oct 27, 2011 "has gerrit upgraded to new JGit?" "Comment by x Apr 12 (2 days ago) jgit 1.1 has been around for a long time now -- any reason not to upgrade?"
Yes, but if you read the release notes for 2.3 you will see they've upgraded to jgit 1.1.
I'd like to upgrade our gerrit install in a week or two.
-Chad On Apr 15, 2012 3:34 AM, "Thomas Gries" mail@tgries.de wrote:
Am 14.04.2012 20:17, schrieb Chad:
If you read my comments on that bug you will see I believe it is an upstream issue and is probably fixed in 2.3.
-Chad
Yes, but see https://code.google.com/p/gerrit/issues/detail?id=769 . The questions are: who is upgrading the software, and when ?
Comment by y Oct 27, 2011 "has gerrit upgraded to new JGit?" "Comment by x Apr 12 (2 days ago) jgit 1.1 has been around for a long time now -- any reason not to upgrade?"
On Fri, Apr 13, 2012 at 2:32 AM, Thomas Gries mail@tgries.de wrote:
For test purposes I wanted to clone the last commit only by using the --depth option as in
/work/tmp/ # git clone --depth=10 https://gerrit.wikimedia.org/r/p/mediawiki/core.git
and got this error message:
Initialized empty Git repository in /work/tmp/core/.git/ *error: RPC failed; result=22, HTTP code = 500*
My git client version is 1.7.1
Any idea why ?
Perhaps related to issue 769?
-Chad
now filed as https://bugzilla.wikimedia.org/show_bug.cgi?id=35949 *Bug 35949* https://bugzilla.wikimedia.org/show_bug.cgi?id=35949 -Shallow clones fail with errors: git clone --depth 1 fails with error: RPC failed; result=22, HTTP code = 500
wikitech-l@lists.wikimedia.org