-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hello all!
Pywikibot finished GIT migration. Now a question has anybody ever tryed to clone and use the e.g. the pywikibot/compat repo on the TS? If I do, I get and .git folder of about 200MB which nearly fills my home quota. (this is the plain clone - nothing done to/with it yet!!)
Yes I could do 'git gc' I could also switch to using SVN from github (but needs to handle all externals manually)...
So... what was acutally the reason for switching to GIT? Despite the fact that I have to enter about 1 million different commands now (in SVN I needed 5 ... maybe 10 - 1 command per action, in GIT I need about 5-10 command PER action) I am dealing 75% of my time with GIT issues (that SVN did not have)...
In fact I am not able to to USE or further develop the GIT repo since the migration - and I am SERIOUSLY thinking about setting up an own copy on sf.net again and work there... (then we would also have the bug-tracker back in the same place) At the moment I see just 2 drawbacks:
1.) not hosted by mediawiki (really a drawback?) 2.) no review (this is the one and only advantage of git till now, and in fact it's git-review which causes issues too)
The list of advantages on the other side, is that long that I will not even start making one, think of it yourself and please give me some feedback!
At least I do not see if there will ever be a point in time when GIT will start to work as well as SVN did in the past. I do also not see why I have to enter 5 commands just to make a simple commit. And that will never change...
Thanks for your thoughts and Greetings DrTrigon
O Sábado, 10 de Agosto de 2013 11:26:13 Dr. Trigon escribiu:
So... what was acutally the reason for switching to GIT?
http://blog.teamtreehouse.com/why-you-should-switch-from-subversion-to-git
On Sat, Aug 10, 2013 at 5:26 PM, Dr. Trigon dr.trigon@surfeu.ch wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hello all!
Pywikibot finished GIT migration. Now a question has anybody ever tryed to clone and use the e.g. the pywikibot/compat repo on the TS? If I do, I get and .git folder of about 200MB which nearly fills my home quota. (this is the plain clone - nothing done to/with it yet!!)
If you read the instructions ( https://www.mediawiki.org/wiki/Manual:Pywikipediabot/Gerrit), it recommends you only clone with a --depth of 3 (or anything smaller), and it will take up very little space.
Yes I could do 'git gc' I could also switch to using SVN from github (but needs to handle all externals manually)...
So... what was acutally the reason for switching to GIT? Despite the fact that I have to enter about 1 million different commands now (in SVN I needed 5 ... maybe 10 - 1 command per action, in GIT I need about 5-10 command PER action) I am dealing 75% of my time with GIT issues (that SVN did not have)...
I don't follow. git commit filename.py, git review -R, and then hit +2.
In fact I am not able to to USE or further develop the GIT repo since the migration - and I am SERIOUSLY thinking about setting up an own copy on sf.net again and work there... (then we would also have the bug-tracker back in the same place) At the moment I see just 2 drawbacks:
I don't know why you would ever do that, sf.net pretty much sucks.
1.) not hosted by mediawiki (really a drawback?) 2.) no review (this is the one and only advantage of git till now, and in fact it's git-review which causes issues too)
The list of advantages on the other side, is that long that I will not even start making one, think of it yourself and please give me some feedback!
At least I do not see if there will ever be a point in time when GIT will start to work as well as SVN did in the past. I do also not see why I have to enter 5 commands just to make a simple commit. And that will never change...
Thanks for your thoughts and Greetings DrTrigon -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.13 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iEYEARECAAYFAlIGBzQACgkQAXWvBxzBrDC40gCeNt8b7KoBLV9BDNDojjM01Upp ZsEAoIQIJ7W1FGMNysOE7LIp4qi25pyW =mvrX -----END PGP SIGNATURE-----
--Legoktm
On 10 August 2013 11:26, Dr. Trigon dr.trigon@surfeu.ch wrote:
Pywikibot finished GIT migration. Now a question has anybody ever tryed to clone and use the e.g. the pywikibot/compat repo on the TS? If I do, I get and .git folder of about 200MB which nearly fills my home quota. (this is the plain clone - nothing done to/with it yet!!)
Running git gc --aggressive --prune brings it back from 178M to 14M. As a comparison: an SVN checkout of trunk is 34M, of which 15M are externals.
I've CC'ed Chad, who possibly knows whether there is anything we can do to improve this on the server side -- is there any way to transfer this optimized repository instead of the 200M one?
Merlijn
On Sat, Aug 10, 2013 at 11:28 PM, Merlijn van Deen valhallasw@arctus.nlwrote:
On 10 August 2013 11:26, Dr. Trigon dr.trigon@surfeu.ch wrote:
Pywikibot finished GIT migration. Now a question has anybody ever tryed to clone and use the e.g. the pywikibot/compat repo on the TS? If I do, I get and .git folder of about 200MB which nearly fills my home quota. (this is the plain clone - nothing done to/with it yet!!)
Running git gc --aggressive --prune brings it back from 178M to 14M. As a comparison: an SVN checkout of trunk is 34M, of which 15M are externals.
I've CC'ed Chad, who possibly knows whether there is anything we can do to improve this on the server side -- is there any way to transfer this optimized repository instead of the 200M one?
I asked about a solution of the 200M everywhere. The best option is cloning shallowly
Merlijn
Pywikipedia-l mailing list Pywikipedia-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikipedia-l
On 15 August 2013 12:27, Amir Ladsgroup ladsgroup@gmail.com wrote:
I asked about a solution of the 200M everywhere. The best option is cloning shallowly
No, it's not. After running git gc, I can re-clone the existing repository by only transferring 8MB:
valhallasw@lisilwen:~/src/pywikibot-compat$ git daemon --reuseaddr --base-path=. --export-all --verbose & valhallasw@lisilwen:~/src/pywikibot-compat$ cd .. valhallasw@lisilwen:~/src$ git clone git://localhost/ pwb2 Cloning into 'pwb2'... [6373] Connection from 127.0.0.1:36357 [6373] Extended attributes (16 bytes) exist <host=localhost> [6373] Request upload-pack for '/' remote: Counting objects: 37384, done. remote: Compressing objects: 100% (10563/10563), done. [6370] [6373] Disconnected2525/37384) remote: Total 37384 (delta 26576), reused 37384 (delta 26576) Receiving objects: 100% (37384/37384), 7.96 MiB, done. Resolving deltas: 100% (26576/26576), done.
So it should be possible to solve this issue on the Wikimedia end: downloading 7MB sounds very reasonable to me.
Merlijn
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 15.08.2013 16:02, Merlijn van Deen wrote:
On 15 August 2013 12:27, Amir Ladsgroup <ladsgroup@gmail.com mailto:ladsgroup@gmail.com> wrote:
I asked about a solution of the 200M everywhere. The best option is cloning shallowly
No, it's not. After running git gc, I can re-clone the existing repository by only transferring 8MB:
valhallasw@lisilwen:~/src/pywikibot-compat$ git daemon --reuseaddr --base-path=. --export-all --verbose & valhallasw@lisilwen:~/src/pywikibot-compat$ cd .. valhallasw@lisilwen:~/src$ git clone git://localhost/ pwb2 Cloning into 'pwb2'... [6373] Connection from 127.0.0.1:36357 http://127.0.0.1:36357 [6373] Extended attributes (16 bytes) exist <host=localhost> [6373] Request upload-pack for '/' remote: Counting objects: 37384, done. remote: Compressing objects: 100% (10563/10563), done. [6370] [6373] Disconnected2525/37384) remote: Total 37384 (delta 26576), reused 37384 (delta 26576) Receiving objects: 100% (37384/37384), 7.96 MiB, done. Resolving deltas: 100% (26576/26576), done.
So it should be possible to solve this issue on the Wikimedia end: downloading 7MB sounds very reasonable to me.
+1
I had a deeper look of this, consider e.g. compat. First cloning everything (including externals) and then garbage collecting:
$ git clone --recursive https://gerrit.wikimedia.org/r/pywikibot/compat.git [target-dir] $ cd [target-dir] $ git gc --aggressive --prune $ cd i18n/ $ git gc --aggressive --prune $ cd ../externals/spelling/ $ git gc --aggressive --prune
results in ~24MB size. But after cloning it is ~210MB so you need that much of free disk space and it is slow. An alternative, not needing that much free space and beeing a lot faster, is shallow cloning:
$ git clone --depth 1 --recursive https://gerrit.wikimedia.org/r/pywikibot/compat.git [target-dir]
and results in a repo size of ~36MB. The problem here is in order to shallow clone the submodules also you need either git 1.8.4 [1] or to follow a quite complex sequence of steps.
[1] http://stackoverflow.com/questions/6238590/set-git-submodule-to-shallow-clon...
Now forgetting about externals and looking at compat only results in:
$ git clone https://gerrit.wikimedia.org/r/pywikibot/compat.git [target-dir] $ cd [target-dir] $ git gc --aggressive --prune
gives a size of ~15MB as mentioned by Merljin already, and:
$ git clone --depth 1 https://gerrit.wikimedia.org/r/pywikibot/compat.git [target-dir]
one of only ~6MB so it is possible to reduce it even more, but only if not considering externals.
Recognize also that garbage collecting works better when working with the externals, otherwise shallow cloning gives smaller sizes.
Sizes around 20MB for everything - including externals - are ok for me, but I would prefer not having to do garbage collection (gc). This would solve issues with small quotas (disk sizes) and reduce the possible sources of other problems (e.g. reducing the number of command to enter ;). Therefore, solving this on the server side would be a good thing!
Greetings DrTrigon
pywikipedia-l@lists.wikimedia.org