Hi all,
tl;dr: I've cleaned up the mediawiki/core repo, and performance for fetch/clone operations should be noticeably faster.
So, due to some recent upgrades in Gerrit, we've now got GC support in JGit as a result. Gerrit supports this functionality, which will greatly reduce the size of our repositories on-disk. It also makes use of an improved bitmap algorithm for fetches & clones, making them wayyy faster. I've now run this on mediawiki/ core, and the repo went from 3.0G down to ~620M on disk. You should now notice all read actions (over https & ssh) to be way faster--a fresh clone should now only be limited by your bandwidth, not how fast Gerrit can serve the repo from disk.
If you notice *any* problems with mediawiki/core, please let me know immediately. If everything looks good, we'll set this up as a cron to run weekly or something for all repositories.
-Chad
On Wed, Apr 3, 2013 at 1:55 AM, Chad innocentkiller@gmail.com wrote:
I've now run this on mediawiki/core, and the repo went from 3.0G down to ~620M on disk.
I copy+pasted this wrong. This is actually 323M on disk. To give you an idea of the kind of savings we're looking at for fetch & clone, here's my test clone I just attempted:
$ time git clone https://gerrit.wikimedia.org/r/p/mediawiki/core Cloning into 'core'... remote: Counting objects: 13011, done remote: Finding sources: 100% (184/184) remote: Getting sizes: 100% (51/51) remote: Compressing objects: 11% (6/51) remote: Total 419954 (delta 58), reused 419948 (delta 57) Receiving objects: 100% (419954/419954), 230.35 MiB | 1.85 MiB/s, done. Resolving deltas: 100% (346577/346577), done.
real 3m41.556s user 1m7.721s sys 0m8.240s
-Chad
On Wed, Apr 3, 2013 at 2:02 AM, Chad innocentkiller@gmail.com wrote:
This is actually 323M on disk.
Wow. Bravo. That is truly an extraordinary improvement.
*-- * *Tyler Romeo* Stevens Institute of Technology, Class of 2015 Major in Computer Science www.whizkidztech.com | tylerromeo@gmail.com
On 04/03/2013 02:02 AM, Chad wrote:
On Wed, Apr 3, 2013 at 1:55 AM, Chad innocentkiller@gmail.com wrote:
I've now run this on mediawiki/core, and the repo went from 3.0G down to ~620M on disk.
I copy+pasted this wrong. This is actually 323M on disk. To give you an idea of the kind of savings we're looking at for fetch & clone, here's my test clone I just attempted:
That's great. It's really good to hear you could fix the performance issues without resorting to the drastic step of history rewriting.
Matt Flaschen
Hi,
On Wed, Apr 03, 2013 at 02:02:19AM -0400, Chad wrote:
[ cloning core in 3 minutes ]
Yay \o/ Chad, you rock!
Thanks, Christian
On 04/03/2013 02:02 AM, Chad wrote:
On Wed, Apr 3, 2013 at 1:55 AM, Chad innocentkiller@gmail.com wrote:
I've now run this on mediawiki/core, and the repo went from 3.0G down to ~620M on disk.
I copy+pasted this wrong. This is actually 323M on disk. To give you an idea of the kind of savings we're looking at for fetch & clone, here's my test clone I just attempted:
$ time git clone https://gerrit.wikimedia.org/r/p/mediawiki/core Cloning into 'core'... remote: Counting objects: 13011, done remote: Finding sources: 100% (184/184) remote: Getting sizes: 100% (51/51) remote: Compressing objects: 11% (6/51) remote: Total 419954 (delta 58), reused 419948 (delta 57) Receiving objects: 100% (419954/419954), 230.35 MiB | 1.85 MiB/s, done. Resolving deltas: 100% (346577/346577), done.
real 3m41.556s user 1m7.721s sys 0m8.240s
-Chad
Chad, when I sing your praises, what meter do you want that in? 4/4 ok? :D
On Wednesday, April 3, 2013 at 11:09 AM, Andrew Bogott wrote:
On 4/3/13 12:55 AM, Chad wrote:
Hi all,
tl;dr: I've cleaned up the mediawiki/core repo, and performance for fetch/clone operations should be noticeably faster.
Fast enough that non-interactive clones don't time out anymore! \o/
Yeah, confirmed. Very different experience. Thanks Chad.
-- Ori Livneh
On 04/03/2013 01:55 AM, Chad wrote:
Hi all,
tl;dr: I've cleaned up the mediawiki/core repo, and performance for fetch/clone operations should be noticeably faster.
This also seems to have fixed https://bugzilla.wikimedia.org/show_bug.cgi?id=46399 (the "Included in" feature in Gerrit was previously very slow), which is great.
Thanks again,
Matt Flaschen
On Wed, Apr 3, 2013 at 1:55 AM, Chad innocentkiller@gmail.com wrote:
Hi all,
tl;dr: I've cleaned up the mediawiki/core repo, and performance for fetch/clone operations should be noticeably faster.
I've now done this for all other repositories, so everyone should see the benefits. I've got a change under review to add a cron to perform this weekly.
-Chad
Hey,
I've now done this for all other repositories, so everyone should
see the benefits. I've got a change under review to add a cron to perform this weekly.
Thanks Chad! You win one free interwebs!
Cheers
-- Jeroen De Dauw http://www.bn2vs.com Don't panic. Don't be evil. --
wikitech-l@lists.wikimedia.org