Nightly shallow clones of mediawiki/core

List overview All Threads
Download

newer

older

Meetup: Lua meets Wikipedia

[Language Engineering] Office hour...

Ori Livneh

10 Mar 2013 10 Mar '13

10:15 p.m.

Hello, I'm in the process of re-working mediawiki-vagrant, which is a set of scripts for provisioning a virtual machine for MediaWiki development. I'm struggling to identify the best way of fetching mediawiki/core. An ideal solution would have the following attributes: - Fast. - Includes .git metadata, to facilitate contribution of patches. - Viable on slow network connections. - Does not require a Gerrit account (to help newcomers get started quickly) What I tried: - A shallow (--depth=0) git-clone over HTTPS took around half an hour and required transferring 272MB, with 200MB taken up by .git/objects/pack. - The nightlies on integration.mediawiki.org are small (18MB) and easy to retrieve, but the most recent one is from December, and they don't contain any .git metadata. - The snapshots Krinkle maintains on the toolserver are both small and up-to-date, but they too do not contain any .git metadata. - The snapshot link on http://www.mediawiki.org/wiki/Download (https://gerrit.wikimedia.org/r/gitweb?p=mediawiki/core.git;a=snapshot;h=ref…) just didn't work. It hangs for a while and then spits out HTML. - Getting a snapshot from GitHub would probably work, but I am loathe to depend on it. Does anyone have any suggestions? -- Ori Livneh

Show replies by date

Yuri Astrakhan

10 Mar 10 Mar

10:33 p.m.

Hi Ori, I use your vagrant VM all the time. Thanks!!! If we have a good vagrant setup, getting new devs on-board might be that much easier. I keep all the notes of the things I needed to do to your VM at http://www.mediawiki.org/wiki/User:Yurik/Installing_Linux_virtual_box_under… Most of it is not related to Windows, and I would love to have some (most) of it automated: * reconfigure network to be "hostonly" instead of "bridged" - major security issue * install phpmyadmin & sqlite * samba (very useful on the windows box, possibly others) * configure git tools (script?) * DEBUG <== that's a big one, setting it up for easy nobrainer debugging would be a great help. Including CLI -- i'm having hard time debugging unit tests. An initial shell script * auto-generated the SSH key and place it into the right dir * git submodule update --init Thanks! On Sun, Mar 10, 2013 at 5:15 PM, Ori Livneh <ori(a)wikimedia.org> wrote:

...

Hello, I'm in the process of re-working mediawiki-vagrant, which is a set of scripts for provisioning a virtual machine for MediaWiki development. I'm struggling to identify the best way of fetching mediawiki/core. An ideal solution would have the following attributes: - Fast. - Includes .git metadata, to facilitate contribution of patches. - Viable on slow network connections. - Does not require a Gerrit account (to help newcomers get started quickly) What I tried: - A shallow (--depth=0) git-clone over HTTPS took around half an hour and required transferring 272MB, with 200MB taken up by .git/objects/pack. - The nightlies on integration.mediawiki.org are small (18MB) and easy to retrieve, but the most recent one is from December, and they don't contain any .git metadata. - The snapshots Krinkle maintains on the toolserver are both small and up-to-date, but they too do not contain any .git metadata. - The snapshot link on http://www.mediawiki.org/wiki/Download ( https://gerrit.wikimedia.org/r/gitweb?p=mediawiki/core.git;a=snapshot;h=ref…) just didn't work. It hangs for a while and then spits out HTML. - Getting a snapshot from GitHub would probably work, but I am loathe to depend on it. Does anyone have any suggestions? -- Ori Livneh _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Matthew Flaschen

11:57 p.m.

On 03/10/2013 05:33 PM, Yuri Astrakhan wrote:

...

* install phpmyadmin & sqlite

PHPMyAdmin also has major security issues. It isn't allowed on Wikimedia Labs and probably shouldn't be used here. Why does SQLite need to be installed exactly?

...

* DEBUG <== that's a big one, setting it up for easy nobrainer debugging would be a great help. Including CLI -- i'm having hard time debugging unit tests.

PHP Unit tests, or QUnit? Matt Flaschen

Yuri Astrakhan

11 Mar 11 Mar

12:17 a.m.

Answered Inline. Also, I apologize as I think my email was slightly off-topic to Ori's question. On Sun, Mar 10, 2013 at 6:57 PM, Matthew Flaschen <mflaschen(a)wikimedia.org>wrote;wrote:

...

PHPMyAdmin also has major security issues. It isn't allowed on Wikimedia Labs and probably shouldn't be used here. Why does SQLite need to be installed exactly?

Matthew, we are talking about a developer's virtual machine that has no network connection to anything except the developer's machine itself, and used purely for development. MySql could be accessed through the network if I have the local tools, but if we are talking about the lowest barrier of entry, the novice could follow a few steps to get VM and edit code with anything including notepad, while the VM would have most of the tools to examine and experiment with MediaWiki. BTW, samba would also be a massive security hole - I set mine up to share / with no password root access (no network, no issues) SQLite is needed because the dev server test with SQLite, and I personally got burnt by their difference (unit tests passed in mysql but not in sqlite due to a bug in the testing framework). This unit test<https://gerrit.wikimedia.org/r/#/c/48098/> still fails.

...

* DEBUG <== that's a big one, setting it up for easy nobrainer debugging would be a great help. Including CLI -- i'm having hard time debugging

unit

tests.

PHP Unit tests, or QUnit?

I think both would be good (I only used phpunit so far).

Matthew Flaschen

12:56 a.m.

On 03/10/2013 07:17 PM, Yuri Astrakhan wrote:

...

Matthew, we are talking about a developer's virtual machine that has no network connection to anything except the developer's machine itself, and used purely for development.

As you said yourself, it's currently bridged. Doesn't that mean it is in fact accessible to the whole LAN, until that's changed?

...

PHP Unit tests, or QUnit?

I think both would be good (I only used phpunit so far).

QUnit can be debugged easily at http://127.0.0.1:8080/wiki/Special:JavaScriptTest . Matt Flaschen

Yuri Astrakhan

1:04 a.m.

On Sun, Mar 10, 2013 at 7:56 PM, Matthew Flaschen <mflaschen(a)wikimedia.org>wrote;wrote:

...

As you said yourself, it's currently bridged. Doesn't that mean it is in fact accessible to the whole LAN, until that's changed?

Of course - that's why I put the hostonly as the first requirement, and was the first thing I changed locally.

...

QUnit can be debugged easily at http://127.0.0.1:8080/wiki/Special:JavaScriptTest . Thanks, good to know! Still, it's the phpunit that I have had issues with

- the debugger would not reliably attach, it wouldn't break anywhere except on entry point, etc. But I think this should be a separate discussion.

Platonides

12 Mar 12 Mar

12:13 a.m.

On 11/03/13 00:17, Yuri Astrakhan wrote:

...

Answered Inline. Also, I apologize as I think my email was slightly off-topic to Ori's question. On Sun, Mar 10, 2013 at 6:57 PM, Matthew Flaschen <mflaschen(a)wikimedia.org>wrote;wrote:

PHPMyAdmin also has major security issues. It isn't allowed on Wikimedia Labs and probably shouldn't be used here. Why does SQLite need to be installed exactly?

While popular, I don't think PHPMyAdmin is the best tool for managing the tables, either.

Chad

10 Mar 10 Mar

10:39 p.m.

Hi, I've been thinking about this for the last week or so because it's becoming incredibly clear to me that core isn't scaling. It's already taking up over 4GB on the Gerrit box, and this is the primary reason core operations are slow. A couple of things I can think of to help the situation. - We can repack core on manganese. This should provide a bit of relief, but won't help long term. Core would have to be read-only for about an hour or two. -We can rewrite history (git-filter-branch) to remove some mistakes that exploded the repo size. Binaries later removed, things accidentally checked into ./extensions, etc. This could potentially greatly reduce object sizes and allow for tighter repacks. Major issue with history rewriting is everyone will have to reclone (all sha1s would change). I've not tested my theory yet. I'm open to any other ideas that could help core. -Chad On Mar 10, 2013 5:15 PM, "Ori Livneh" <ori(a)wikimedia.org> wrote:

...

Hello, I'm in the process of re-working mediawiki-vagrant, which is a set of scripts for provisioning a virtual machine for MediaWiki development. I'm struggling to identify the best way of fetching mediawiki/core. An ideal solution would have the following attributes: - Fast. - Includes .git metadata, to facilitate contribution of patches. - Viable on slow network connections. - Does not require a Gerrit account (to help newcomers get started quickly) What I tried: - A shallow (--depth=0) git-clone over HTTPS took around half an hour and required transferring 272MB, with 200MB taken up by .git/objects/pack. - The nightlies on integration.mediawiki.org are small (18MB) and easy to retrieve, but the most recent one is from December, and they don't contain any .git metadata. - The snapshots Krinkle maintains on the toolserver are both small and up-to-date, but they too do not contain any .git metadata. - The snapshot link on http://www.mediawiki.org/wiki/Download ( https://gerrit.wikimedia.org/r/gitweb?p=mediawiki/core.git;a=snapshot;h=ref…) just didn't work. It hangs for a while and then spits out HTML. - Getting a snapshot from GitHub would probably work, but I am loathe to depend on it. Does anyone have any suggestions? -- Ori Livneh _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Tyler Romeo

10:48 p.m.

The git-archive command is supposed to fulfill this functionality, but it seems it does not work anonymously, i.e., over HTTPS. I think it only works with ssh:// and with git://. Otherwise, GitHub's snapshots seem to be the only solution. *--* *Tyler Romeo* Stevens Institute of Technology, Class of 2015 Major in Computer Science www.whizkidztech.com | tylerromeo(a)gmail.com

Bartosz Dziewoński

11:03 p.m.

A shallow clone certainly shouldn't be as large as a normal one. Something's borked. On Sun, 10 Mar 2013 22:39:44 +0100, Chad <innocentkiller(a)gmail.com> wrote:

...

- We can repack core on manganese. This should provide a bit of relief, but won't help long term. Core would have to be read-only for about an hour or two.

This should be just done already, and repeated from time to time. The repo is growing in size much quicker than it should (full clone now is over 250 MB, in September it was under 180 MB IIRC).

...

-We can rewrite history (git-filter-branch) to remove some mistakes that exploded the repo size. Binaries later removed, things accidentally checked into ./extensions, etc. This could potentially greatly reduce object sizes and allow for tighter repacks. Major issue with history rewriting is everyone will have to reclone (all sha1s would change). I've not tested my theory yet.

Ugh, the nuclear option. Let's try repacking first, can we? :) This is probably going to make gerrit a little unhappy when it's database thing gets inconsistent with the repo. -- Matma Rex

Kevin Israel

11:32 p.m.

On 03/10/2013 06:03 PM, Bartosz Dziewoński wrote:

...

A shallow clone certainly shouldn't be as large as a normal one. Something's borked.

--depth 0 is what's broken. --depth 1 works fine. $ git clone --depth 1 https://gerrit.wikimedia.org/r/p/mediawiki/core.git core-shallow Cloning into 'core-shallow'... remote: Counting objects: 2815, done remote: Finding sources: 100% (2815/2815) remote: Getting sizes: 100% (2665/2665) remote: Compressing objects: 61% (1631/2660) remote: Total 2815 (delta 317), reused 1182 (delta 147) Receiving objects: 100% (2815/2815), 17.87 MiB | 1.16 MiB/s, done. Resolving deltas: 100% (342/342), done. -- Wikipedia user PleaseStand http://en.wikipedia.org/wiki/User:PleaseStand

Rob Lanphier

12 Mar 12 Mar

2:51 a.m.

On Sun, Mar 10, 2013 at 3:32 PM, Kevin Israel <pleasestand(a)live.com> wrote:

...

On 03/10/2013 06:03 PM, Bartosz Dziewoński wrote:

A shallow clone certainly shouldn't be as large as a normal one. Something's borked.

--depth 0 is what's broken. --depth 1 works fine. $ git clone --depth 1

[...]

...

Receiving objects: 100% (2815/2815), 17.87 MiB | 1.16 MiB/s, done.

Yup, I'm seeing more or less the same thing. Importantly: $ du -sh .git 19M .git I was able to do the clone in 50 seconds over HTTPS. Most of that time was spent in data transfer (which would be the same for a snapshot). Ori, have you tried this with --depth 1? Rob

Thomas Gries

3:50 a.m.

Am 12.03.2013 02:51, schrieb Rob Lanphier:

...

I was able to do the clone in 50 seconds over HTTPS.... Ori, have you tried this with --depth 1? Rob

I added - already some weeks ago - this comment to our main download _talk_ page https://www.mediawiki.org/wiki/Talk:Download#Discussion:_Whether_or_not_we_… "I was unsure, whether it makes sense to add in section Download#Development releases <https://www.mediawiki.org/wiki/Download#Development_releases> the following: By using the depth parameter as shown in the following, you can download a "shallow" clone which saves bytes, bandwidth and time, [ but can lead to problems if you are developer ] " git clone --depth 1 https://gerrit.wikimedia.org/r/p/mediawiki/core.git If my Url is correct, then someone of you can simply add the command to section https://www.mediawiki.org/wiki/Download#Development_releases of the _article_ page. At least, I wanted to suggest this, because it is the correct place. Thomas

Ori Livneh

8 a.m.

On Monday, March 11, 2013 at 6:51 PM, Rob Lanphier wrote:

...

On Sun, Mar 10, 2013 at 3:32 PM, Kevin Israel <pleasestand(a)live.com (mailto:pleasestand@live.com)> wrote:

On 03/10/2013 06:03 PM, Bartosz Dziewoński wrote:

A shallow clone certainly shouldn't be as large as a normal one. Something's borked.

--depth 0 is what's broken. --depth 1 works fine. $ git clone --depth 1

[...]

Receiving objects: 100% (2815/2815), 17.87 MiB | 1.16 MiB/s, done.

Rob, thanks for checking. I tried it yesterday and again just now, and in both cases it took around 15 minutes: vagrant@precise32:~$ time git clone --depth 1 https://gerrit.wikimedia.org/r/p/mediawiki/core.git Cloning into 'core'... remote: Counting objects: 46297, done remote: Finding sources: 100% (46297/46297) remote: Getting sizes: 100% (25843/25843) remote: Compressing objects: 76% (19864/25833) remote: Total 46297 (delta 33063), reused 26399 (delta 20010) Receiving objects: 100% (46297/46297), 102.66 MiB | 194 KiB/s, done. Resolving deltas: 100% (37898/37898), done. real 15m14.500s user 0m27.562s sys 0m13.421s The output of 'git config --list' is blank; this is vanilla git. 'Compressing objects' took the longest. For comparison: vagrant@precise32:~$ time wget -q https://github.com/wikimedia/mediawiki-core/archive/master.zip && unzip -x -q master.zip real 1m15.592s user 0m0.184s sys 0m3.480s -- Ori Livneh

Rob Lanphier

4:51 p.m.

Hi Ori, I'm at the office now, but (heh) that may be a tougher test than my home connection. More below.... On Tue, Mar 12, 2013 at 12:00 AM, Ori Livneh <ori(a)wikimedia.org> wrote:

...

I'm on Ubuntu 12.10 (Quantal), so that might have something to do with it. $ git --version git version 1.7.10.4 $ time git clone --depth 1 https://gerrit.wikimedia.org/r/p/mediawiki/core.git core-shallow Cloning into 'core-shallow'... remote: Counting objects: 3456, done remote: Finding sources: 100% (3456/3456) remote: Getting sizes: 100% (3074/3074) remote: Compressing objects: 63% (1958/3069) remote: Total 3456 (delta 690), reused 1496 (delta 379) Receiving objects: 100% (3456/3456), 18.71 MiB | 1.49 MiB/s, done. Resolving deltas: 100% (816/816), done. real 0m34.507s user 0m4.252s sys 0m0.940s I wasn't able to find anything in the Git release notes to indicate anything different on the client side, though I didn't look too hard. However, it appears as though JGit server-side support for shallow clones is relatively recent: https://bugs.eclipse.org/bugs/show_bug.cgi?id=394543 ...characterized here as "coming along bit by bit": http://dev.eclipse.org/mhonarc/lists/jgit-dev/msg01976.html ...which would imply that maybe the JGit server side stuff hasn't been tested with a lot of client versions. Rob

Ori Livneh

9:36 p.m.

On Tuesday, March 12, 2013 at 8:51 AM, Rob Lanphier wrote:

...

...which would imply that maybe the JGit server side stuff hasn't been tested with a lot of client versions.

Rob, thanks again for investigating. I filed a bug in Bugzilla so we can track this: https://bugzilla.wikimedia.org/show_bug.cgi?id=46041 Could you capture the output of the following command: GIT_CURL_VERBOSE=1 GIT_TRACE=true git clone --verbose --depth 1 https://gerrit.wikimedia.org/r/p/mediawiki/core.git &>~/core-git-clone.log And upload it as an attachment? I already uploaded mine. This way we can compare. -- Ori Livneh

Ori Livneh

10:31 p.m.

On Tuesday, March 12, 2013 at 1:36 PM, Ori Livneh wrote:

...

I filed a bug in Bugzilla so we can track this: https://bugzilla.wikimedia.org/show_bug.cgi?id=46041

Platonides figured it out. Incredibly enough, in the half-sub-sub-version delta from 1.7.9.5 to 1.7.10, the behavior of git-clone was fixed so that it only fetches a single branch rather than all. The workaround for git <1.7.10 is: mkdir core; cd core git init git fetch --depth=1 https://gerrit.wikimedia.org/r/p/mediawiki/core.git master:refs/remotes/origin/master -- Ori Livneh

Chad

10:48 p.m.

On Tue, Mar 12, 2013 at 5:31 PM, Ori Livneh <ori(a)wikimedia.org> wrote:

...

On Tuesday, March 12, 2013 at 1:36 PM, Ori Livneh wrote:

I filed a bug in Bugzilla so we can track this: https://bugzilla.wikimedia.org/show_bug.cgi?id=46041

This totally doesn't surprise me at all. Version numbers in Git don't reflect any sort of reality in terms of features or things to look forward to--if memory serves, a series of really bad performance regressions (and fixes) were introduced in the course of just 1.6.x.y. -Chad

Željko Filipin

13 Mar 13 Mar

4:02 p.m.

On Tue, Mar 12, 2013 at 10:48 PM, Chad <innocentkiller(a)gmail.com> wrote:

...

Version numbers in Git don't reflect any sort of reality in terms of features or things to look forward

Sometimes I mention Semantic Versioning[1] to people that write code for a living and get surprised that they did not hear about it. Željko -- [1] http://semver.org/

Brad Jorsch

4:43 p.m.

On Wed, Mar 13, 2013 at 11:02 AM, Željko Filipin <zfilipin(a)wikimedia.org>wrote;wrote:

...

On Tue, Mar 12, 2013 at 10:48 PM, Chad <innocentkiller(a)gmail.com> wrote:

Version numbers in Git don't reflect any sort of reality in terms of features or things to look forward

Sometimes I mention Semantic Versioning[1] to people that write code for a living and get surprised that they did not hear about it. [1] http://semver.org/

Maybe that's because it's a buzzwordy new name for a very old idea? Although as described there, it tends to break if you have multiple different APIs that aren't necessarily in sync, or if the API is only a minor portion of the product. And marketing may complain if your competitor is at version "10" and you're at "2" (despite that being 2.98.917).

Matthew Flaschen

11 Mar 11 Mar

12:12 a.m.

On 03/10/2013 06:03 PM, Bartosz Dziewoński wrote:

...

Ugh, the nuclear option. Let's try repacking first, can we? :) This is probably going to make gerrit a little unhappy when it's database thing gets inconsistent with the repo.

Yeah, I agree we should be extremely reluctant to resort to break git history on a project as important as MW core. It would cause a lot of disruption. Matt Flaschen

Platonides

10 Mar 10 Mar

11:43 p.m.

On 10/03/13 22:39, Chad wrote:

...

4GB?? My not specially wellpacked clone takes 191M (and that includes all the changes references), plus other 74 MB for the checkout.

...

Core would have to be read-only for about an hour or two.

Having core read-only occasionally should not be a problem with jenkins collaboration. Instead of merging make it say "this change has been approved and is waiting for the epoch for merging", and have it actually merge everything when it reopens.

Matthew Flaschen

11 Mar 11 Mar

12:10 a.m.

On 03/10/2013 06:43 PM, Platonides wrote:

...

On 10/03/13 22:39, Chad wrote:

4GB?? My not specially wellpacked clone takes 191M (and that includes all the changes references), plus other 74 MB for the checkout.

Yeah, and one of my git dirs is 812 MB, including a clone of core *and* many extension submodules. I don't know what that 4 GB includes. Either it's hardly packed at all, or it includes a lot more than a regular clone (all non-merged changes to core?). Matt Flaschen

Tim Landscheidt

12:58 a.m.

(anonymous) wrote:

...

[...]

...

A couple of things I can think of to help the situation. - We can repack core on manganese. This should provide a bit of relief, but won't help long term. Core would have to be read-only for about an hour or two.

...

[...]

Is this read-only to prevent the hiccup caused by the overeager garbage collection some months ago? Tim

Ori Livneh

12 Mar 12 Mar

12:09 a.m.

On Sun, Mar 10, 2013 at 2:39 PM, Chad <innocentkiller(a)gmail.com> wrote:

...

I'm open to any other ideas that could help core.

I think it would be very useful to provide regular snapshots of core with a shallow '.git' included. Git-cloning from GitHub is faster than Gerrit but is still quite slow, and their snapshots, too, exclude '.git'. --- Ori Livneh ori(a)wikimedia.org

Mark A. Hershberger

1:31 a.m.

On 03/11/2013 07:09 PM, Ori Livneh wrote:

...

You can get a nightly snapshot of mediawiki from: https://toolserver.org/~krinkle/mwSnapshots/#!/mediawiki-core/master There is not .git, but maybe it could be added. -- http://hexmode.com/ Many that live deserve death. And some that die deserve life. Can you give it to them? Then do not be too eager to deal out death in judgment. For even the very wise cannot see all ends. -- J.R.R. Tolkien, The Fellowship of the Ring

Tyler Romeo

1:47 a.m.

On Mon, Mar 11, 2013 at 8:31 PM, Mark A. Hershberger <mah(a)everybody.org>wrote;wrote:

...

You can get a nightly snapshot of mediawiki from: https://toolserver.org/~krinkle/mwSnapshots/#!/mediawiki-core/master There is not .git, but maybe it could be added.

This is spectacular. Simple and easy. *--* *Tyler Romeo* Stevens Institute of Technology, Class of 2015 Major in Computer Science www.whizkidztech.com | tylerromeo(a)gmail.com

Tim Starling

10 Mar 10 Mar

11:25 p.m.

On 11/03/13 08:15, Ori Livneh wrote:

...

- Getting a snapshot from GitHub would probably work, but I am loathe to depend on it.

We now depend on GitHub for ExtensionDistributor. I don't think it's such a bad thing. External services can be a nasty trap when they have the only copy of your data and it's difficult to get it out, but for a git mirror, it's no big deal. We can always make our own non-jgit mirror of our Git repository if GitHub stops existing. -- Tim Starling

4074

days inactive

4077

days old

wikitech-l@lists.wikimedia.org

Manage subscription

27 comments

15 participants

tags (0)

participants (15)

Bartosz Dziewoński
Brad Jorsch
Chad
Kevin Israel
Mark A. Hershberger
Matthew Flaschen
Ori Livneh
Platonides
Rob Lanphier
Thomas Gries
Tim Landscheidt
Tim Starling
Tyler Romeo
Yuri Astrakhan
Željko Filipin