On 31 May 2014 16:08, Chad <innocentkiller(a)gmail.com> wrote: But it's not an issue /per se/. The issue is slow clones/slow pulls, not so much the 270MB on your hard drive. The slow clones/pulls can be improved by re-packing the git repository on the server side -- this helped significantly for the pywikibot repositories. I'm not sure if this has been attempted for mw/core yet.

On 31/05/14 15:03, Merlijn van Deen wrote: Currently the slow clones can be almost completely avoided by cloning from the github mirror, if you have a reasonably fast connection. But considering how most folks are likely to want older history for blames and whatnot sooner or later anyway, the longer download time for the bulk of the data will still come up when they have to download the rest, but this way it would be with a few extra steps. What /does/ github do, do we know? Would that be useful/applicable? Is phabricator similar? -I

On Sat, May 31, 2014 at 9:38 AM, Chad <innocentkiller(a)gmail.com> wrote: See https://bugzilla.wikimedia.org/show_bug.cgi?id=57430 for discussion on this in the context of mw-vagrant.

On Sat, May 31, 2014 at 12:30 PM, Tim Landscheidt <tim(a)tim-landscheidt.de> wrote: Right. That's less of a cloning problem as it's a problem on the remote and slows down operations on *that* repo. -Chad

On 05/31/2014 12:38 PM, Chad wrote: Chad, thanks for bringing this up. I'm especially grateful that you named the target repo size at 30 to 35 MB -- it makes this more concrete. I think some more numbers/facts might help us make the right tradeoffs here to think about ways to contain the repo size going forward -- like: * Antoine said "My repacked copy of core is 270MB" and we have some disagreement about whether that's okay. Is there a max size we want to avoid getting to? * A bigger repo is obviously slower to download in full; is it also slower to search or otherwise work with? How much slower? Most of our developers are probably not on SSDs yet. -- Sumana Harihareswara Senior Technical Writer Wikimedia Foundation

On Mon, Jun 9, 2014 at 11:55 AM, Sumana Harihareswara <sumanah(a)wikimedia.org git is largely insensitive to repo size. It doesn't traverse the entire history unless it needs to. So for most developer tasks, the only constraint is disk size. (There might be a mild size dependency in 'git fetch' -- but I think that's more related to the number of branches in your tree and the tree structure, not simply to number of commits in the history.) However, there are several steps in our current development *pipeline* that do things like naive 'git clone', whose speed depends linearly on the size of the repo. ("Naive" as it's a "simple matter of software" to use a cached local repo or pack to speed things up.) I believe currently both "git review" (submitting patches to gerrit) and the time it takes for jenkins to run tests have steps of this sort; "git review" on mediawiki/core is noticibly much slower than on parsoid (for example). Traded off again this -- if history is truncated, it will be much slower/more complicated for developers to do meaningful history searches, as has been mentioned. I'd expect that most hard-core developers would end up having to download the complete history anyway, as Bartosz suggests. In summary, it seems to me that the reasonable forward path at the moment is some combination of (a) better documenting the use of shallow clones for newbie/infrequent contributors to reduce the initial developer roadblock (including verifying that this works with gerrit, etc), and (b) spending more effort optimizing the 'git clone' step in jenkins/gerrit (we already do some of this), and (c) paying attention to how phabricator uses git, to ensure that the repo size does not become an issue in the future. --scott