[Pywikipedia-l] SVN reaching end of life - migration plan needed - pywikibot

List overview All Threads
Download

newer

[Pywikipedia-l] SVN reaching end of life - migration plan needed

older

[Pywikipedia-l] bots for Semantic...

[Pywikipedia-l] Sourceforge ticket...

Chad Horohoe

15 May 2013 15 May '13

1:37 a.m.

Hi all, The time has finally come upon us--I'm finally moving forward with shutting down SVN and making it a read-only service. As Pywikipedia is the only consumer of SVN anymore, I wanted to reach out to the community to find out what everyone wants to do. As I see it, there's three courses of action that Pywikipedia can go in: 1) Move to Gerrit 2) Move to Git elsewhere (Github, Google Code, etc) 3) Move to some other SVN service I'm more than willing to help with any of these choices--the first two would involve a conversion of the history to Git, along with importing it to the destination of choice. Staying with SVN is also potentially possible, I'm more than happy to provide full SVN dumps if someone's wanting to setup that service elsewhere. What are people's thoughts? I've not come up with a firm date yet, but coming to consensus sooner rather than later would be nice. Thanks! -Chad H.

Attachments:

attachment.htm (text/html — 1.3 KB)

Show replies by date

Merlijn van Deen

15 May 15 May

10:19 a.m.

Hi Chad, In December, I think the consensus was to move to Gerrit [1] - basically with the goal to keep it as close as possible to mediawiki, hopefully also easing contributions (to mw for pwb developers and vice versa). I have put some initial work into the migration (see https://github.com/pywikibot/svn2git ), but I have not had the time to smooth out the wrinkles. Will you be at the Hackathon next week? I think this would be a good topic to discuss there. Merlijn [1] http://lists.wikimedia.org/pipermail/pywikipedia-l/2012-December/007657.html On 15 May 2013 01:37, Chad Horohoe <chorohoe(a)wikimedia.org> wrote:

...

Chad Horohoe

1:33 p.m.

On Wed, May 15, 2013 at 4:19 AM, Merlijn van Deen <valhallasw(a)arctus.nl>wrote;wrote:

...

Ooh, you found my tool :) Cleaning up those rule files should be pretty trivial, then we can convert all of the wanted history pretty quickly. I've gone ahead and drafted a page on mw.org based on the rules you wrote along with a survey of the existing SVN layout. https://www.mediawiki.org/wiki/Git/Conversion/pywikipedia

...

Will you be at the Hackathon next week? I think this would be a good topic to discuss there.

I'm afraid I won't be making it this year. -Chad

Hazard-SJ

22 May 22 May

3:44 a.m.

I'm guessing we could have pywikipedia/trunk and pywikipedia/rewrite in Gerrit. Hazard-SJ ________________________________ From: Chad Horohoe <chorohoe(a)wikimedia.org> To: Merlijn van Deen <valhallasw(a)arctus.nl> Cc: Pywikipedia discussion list <pywikipedia-l(a)lists.wikimedia.org> Sent: Wednesday, May 15, 2013 6:33 AM Subject: Re: [Pywikipedia-l] SVN reaching end of life - migration plan needed On Wed, May 15, 2013 at 4:19 AM, Merlijn van Deen <valhallasw(a)arctus.nl> wrote: Hi Chad,

...

topic to discuss there.

I'm afraid I won't be making it this year. -Chad _______________________________________________ Pywikipedia-l mailing list Pywikipedia-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikipedia-l

Chad Horohoe

1 Jun 1 Jun

2:10 p.m.

On Wed, May 15, 2013 at 4:19 AM, Merlijn van Deen <valhallasw(a)arctus.nl>wrote;wrote:

...

Hi all, The time has finally come upon us--I'm finally moving forward with

shutting

down SVN and making it a read-only service. As Pywikipedia is the only consumer of SVN anymore, I wanted to reach out to the community to find out what everyone wants to do. As I see it, there's three courses of

action

that Pywikipedia can go in: 1) Move to Gerrit 2) Move to Git elsewhere (Github, Google Code, etc) 3) Move to some other SVN service I'm more than willing to help with any of these choices--the first two

would

involve a conversion of the history to Git, along with importing it to

the

destination of choice. Staying with SVN is also potentially possible, I'm more than happy to provide full SVN dumps if someone's wanting to setup that service elsewhere. What are people's thoughts? I've not come up with a firm date yet, but coming to consensus sooner rather than later would be nice. Thanks!

Hazard-SJ

2 Jun 2 Jun

3:39 a.m.

Looks fine by me, and the sooner, the better (or so I want to think :P). Hazard-SJ ________________________________ From: Chad Horohoe <chorohoe(a)wikimedia.org> To: Merlijn van Deen <valhallasw(a)arctus.nl> Cc: Pywikipedia discussion list <pywikipedia-l(a)lists.wikimedia.org> Sent: Saturday, June 1, 2013 7:10 AM Subject: Re: [Pywikipedia-l] SVN reaching end of life - migration plan needed On Wed, May 15, 2013 at 4:19 AM, Merlijn van Deen <valhallasw(a)arctus.nl> wrote: Hi Chad,

...

In December, I think the consensus was to move to Gerrit [1] - basically with the goal to keep it as close as possible to mediawiki, hopefully also easing contributions (to mw for pwb developers and vice versa). I have put some initial work into the migration (see https://github.com/pywikibot/svn2git ), but I have not had the time to smooth out the wrinkles. Will you be at the Hackathon next week? I think this would be a good topic to discuss there. Merlijn [1] http://lists.wikimedia.org/pipermail/pywikipedia-l/2012-December/007657.html On 15 May 2013 01:37, Chad Horohoe <chorohoe(a)wikimedia.org> wrote:

So, this can be done easily, I'm just wondering when we should go ahead with things? It'll only take a few hours, so it's just a matter of picking a date that people are ok with. Also, if anyone has anything they want to add to the page on mw.org, I'd welcome the input. https://www.mediawiki.org/wiki/Git/Conversion/pywikipedia -Chad _______________________________________________ Pywikipedia-l mailing list Pywikipedia-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikipedia-l

legoktm

11:32 a.m.

One thing I didn't notice on the plan is that right now we have a lot of externals set up, some of which are also dependent on the Toolserver. Should we be converting these to git submodules? Or what? Also, I added a comment on the page about giving all current commiters +2 since that wasn't explicitly stated, and drafted a list on the talk page of some things we should do/update after the migration (update docs, update nightlies source, etc) -- Legoktm On Sat, Jun 1, 2013 at 5:10 AM, Chad Horohoe <chorohoe(a)wikimedia.org> wrote:

...

On Wed, May 15, 2013 at 4:19 AM, Merlijn van Deen <valhallasw(a)arctus.nl>wrote;wrote:

Hi all, The time has finally come upon us--I'm finally moving forward with

shutting

action

would

involve a conversion of the history to Git, along with importing it to

the

destination of choice. Staying with SVN is also potentially possible,

I'm

more than happy to provide full SVN dumps if someone's wanting to setup that service elsewhere. What are people's thoughts? I've not come up with a firm date yet, but coming to consensus sooner rather than later would be nice. Thanks!

Chad Horohoe

2:20 p.m.

On Sun, Jun 2, 2013 at 5:32 AM, legoktm <legoktm.wikipedia(a)gmail.com> wrote:

...

If they're available via Git too, then yes, we can switch them over to Git submodules once everything's migrated. For stuff coming from SVN, we'll have to find another way (but it's not impossible).

...

Also, I added a comment on the page about giving all current commiters +2 since that wasn't explicitly stated, and drafted a list on the talk page of some things we should do/update after the migration (update docs, update nightlies source, etc)

As I said on-wiki, that sounds like a reasonable list of people to start with for having +2 access on the repo. The group will be self-managing, so you guys can add new people to it as you see fit. The docs/nightlies stuff is more in your guys' court, but it's good to keep a list of such things, sure. -Chad

Amir Ladsgroup

4 Jun 4 Jun

2:30 p.m.

I agree with "sooner is better", I will help in updating documentations and everything I can Tell me what I can do :) Best On 6/2/13, Chad Horohoe <chorohoe(a)wikimedia.org> wrote:

...

On Sun, Jun 2, 2013 at 5:32 AM, legoktm <legoktm.wikipedia(a)gmail.com> wrote:

If they're available via Git too, then yes, we can switch them over to Git submodules once everything's migrated. For stuff coming from SVN, we'll have to find another way (but it's not impossible).

-- Amir

Chad Horohoe

5 Jun 5 Jun

4:49 p.m.

Everyone seems to be ok with doing this sooner rather than later, so how about we pencil in the date of June 14th (next Friday)? That gives us enough time to figure out the parts we don't know as well as get notifications out to the relevant places. Things I think we need to figure out: 1) Exact repo structure. I think we've gotten a good discussion started on the wiki, but I'd like to see more input (and please, feel free to edit it mercilessly, this was all written based on my naïve understanding of how you guys work) 2) Who gets initial permissions on the repo(s). I've seen suggestions for a select list, all committers in 2013...any of these is fine by me. Remember this is permission for +2'ing and merging, all registered users will be able to submit changes for review (just like everywhere else in Gerrit) 3) What to do with submodules. A concise list of all the current externals will make this easier. Thanks everyone! -Chad On Tue, Jun 4, 2013 at 8:30 AM, Amir Ladsgroup <ladsgroup(a)gmail.com> wrote:

...

I agree with "sooner is better", I will help in updating documentations and everything I can Tell me what I can do :) Best On 6/2/13, Chad Horohoe <chorohoe(a)wikimedia.org> wrote:

On Sun, Jun 2, 2013 at 5:32 AM, legoktm <legoktm.wikipedia(a)gmail.com> wrote:

If they're available via Git too, then yes, we can switch them over to

Git

submodules once everything's migrated. For stuff coming from SVN, we'll have to find another way (but it's not impossible). > Also, I added a comment on the page about giving all current commiters

since that wasn't explicitly stated, and drafted a list on the talk page of some things we should do/update after the migration (update docs, update nightlies source, etc)

As I said on-wiki, that sounds like a reasonable list of people to start with for having +2 access on the repo. The group will be self-managing, so you guys can

add

new people to it as you see fit. The docs/nightlies stuff is more in your

guys'

court, but it's good to keep a list of such things, sure. -Chad

-- Amir _______________________________________________ Pywikipedia-l mailing list Pywikipedia-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikipedia-l

Amir Ladsgroup

9:16 p.m.

This is list of trunk branch externals: --'i18n' --'externals/spelling' --'externals/simplejson' --'externals/pycolorname' --'externals/opencv' On 6/5/13, Chad Horohoe <chorohoe(a)wikimedia.org> wrote:

...

I agree with "sooner is better", I will help in updating documentations and everything I can Tell me what I can do :) Best On 6/2/13, Chad Horohoe <chorohoe(a)wikimedia.org> wrote:

On Sun, Jun 2, 2013 at 5:32 AM, legoktm <legoktm.wikipedia(a)gmail.com> wrote:

If they're available via Git too, then yes, we can switch them over to

Git

submodules once everything's migrated. For stuff coming from SVN, we'll have to find another way (but it's not impossible). > Also, I added a comment on the page about giving all current commiters

since that wasn't explicitly stated, and drafted a list on the talk page of some things we should do/update after the migration (update docs, update nightlies source, etc)

As I said on-wiki, that sounds like a reasonable list of people to start with for having +2 access on the repo. The group will be self-managing, so you guys can

add

new people to it as you see fit. The docs/nightlies stuff is more in your

guys'

court, but it's good to keep a list of such things, sure. -Chad

-- Amir _______________________________________________ Pywikipedia-l mailing list Pywikipedia-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikipedia-l

-- Amir

Maarten Dammers

10:38 p.m.

Hi Chad, Op 5-6-2013 16:49, Chad Horohoe schreef:

...

Everyone seems to be ok with doing this sooner rather than later, so how about we pencil in the date of June 14th (next Friday)?

Hazard-SJ

11:14 p.m.

I agree about having an overlapping test period. Also, we've already agreed to move to Bugzilla, but it was decided that we would focus on moving to Gerrit first. Hazard-SJ ________________________________ From: Maarten Dammers <maarten(a)mdammers.nl> To: pywikipedia-l(a)lists.wikimedia.org Sent: Wednesday, June 5, 2013 3:38 PM Subject: Re: [Pywikipedia-l] SVN reaching end of life - migration plan needed Hi Chad, Op 5-6-2013 16:49, Chad Horohoe schreef:

...

Everyone seems to be ok with doing this sooner rather than later, so how about we pencil in the date of June 14th (next Friday)?

Hold your horses please. I appreciate you putting effort in this, but I don't want to rush this. We should at least have: * Good plan * A timeline * An overlapping test period Also all our bugs are still at Sourceforge. If we're going to move, it would be good to move that to bugzilla so we have the integration. We should really think things through and not make hasty decisions. Maarten _______________________________________________ Pywikipedia-l mailing list Pywikipedia-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikipedia-l

Amir Ladsgroup

6 Jun 6 Jun

4:48 a.m.

On Thu, Jun 6, 2013 at 1:08 AM, Maarten Dammers <maarten(a)mdammers.nl> wrote:

...

Hi Chad, Op 5-6-2013 16:49, Chad Horohoe schreef: Everyone seems to be ok with doing this sooner rather than later, so

how about we pencil in the date of June 14th (next Friday)?

<www.mediawiki.org/wiki/Manual:Pywikipediabot/Migrating_to_bugzilla>I like to packing up these migrations but It seems other devs don't Maarten

...

______________________________**_________________ Pywikipedia-l mailing list Pywikipedia-l(a)lists.wikimedia.**org <Pywikipedia-l(a)lists.wikimedia.org> https://lists.wikimedia.org/**mailman/listinfo/pywikipedia-l<https://lis…

-- Amir

Hazard-SJ

5 Jun 5 Jun

11:27 p.m.

1. The repo structure looks fine (to me, at least) 2. That suggestion also seems fine. (Just saying that I've been trying to get access for some time now, but I'm probably no closer than when I just started :P) 3. As for the externals, someone suggested using git submodules, but since they wouldn't be included in the clones (and checkouts, for that matter), I've just done a bit of reading and subtree merge could be considered. I also found this. Hazard-SJ ________________________________ From: Chad Horohoe <chorohoe(a)wikimedia.org> To: Pywikipedia discussion list <pywikipedia-l(a)lists.wikimedia.org> Sent: Wednesday, June 5, 2013 9:49 AM Subject: Re: [Pywikipedia-l] SVN reaching end of life - migration plan needed Everyone seems to be ok with doing this sooner rather than later, so how about we pencil in the date of June 14th (next Friday)? That gives us enough time to figure out the parts we don't know as well as get notifications out to the relevant places. Things I think we need to figure out: 1) Exact repo structure. I think we've gotten a good discussion started on the wiki, but I'd like to see more input (and please, feel free to edit it mercilessly, this was all written based on my naïve understanding of how you guys work) 2) Who gets initial permissions on the repo(s). I've seen suggestions for a select list, all committers in 2013...any of these is fine by me. Remember this is permission for +2'ing and merging, all registered users will be able to submit changes for review (just like everywhere else in Gerrit) 3) What to do with submodules. A concise list of all the current externals will make this easier. Thanks everyone! -Chad On Tue, Jun 4, 2013 at 8:30 AM, Amir Ladsgroup <ladsgroup(a)gmail.com> wrote: I agree with "sooner is better", I will help in updating

...

documentations and everything I can Tell me what I can do :) Best On 6/2/13, Chad Horohoe <chorohoe(a)wikimedia.org> wrote:

On Sun, Jun 2, 2013 at 5:32 AM, legoktm <legoktm.wikipedia(a)gmail.com> wrote:

If they're available via Git too, then yes, we can switch them over to Git submodules once everything's migrated. For stuff coming from SVN, we'll have to find another way (but it's not impossible).

-- Amir _______________________________________________ Pywikipedia-l mailing list Pywikipedia-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikipedia-l

_______________________________________________ Pywikipedia-l mailing list Pywikipedia-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikipedia-l

Merlijn van Deen

7 Jun 7 Jun

10:29 p.m.

Hi Chad, As we discussed on IRC, my suggestion is the following: On 5 June 2013 16:49, Chad Horohoe <chorohoe(a)wikimedia.org> wrote:

...

Let's get the conversion working, but without allowing patches in Gerrit. This allows us to do the migration in a somewhat slower pace, updating e.g. the nightly generation and making it easier to check if the conversion works as expected. This also makes it easier to get feedback on the conversion, hopefully. As a timeline, I would suggest to have continuous updates from svn to git, roughly weekly updates to the conversion code and a transition period of maybe 6 weeks, starting next friday? I think it is realistic as a time frame to update most things that depend on svn. Things I think we need to

...

1) Exact repo structure. I think we've gotten a good discussion started on the wiki, but I'd like to see more input (and please, feel free to edit it mercilessly, this was all written based on my naïve understanding of how you guys work)

I have added a new proposed structure on the wiki. Basically, I'd like to formally separate trunk and rewrite, and to rename them to something like 'compat' and 'core'. After discussing it with you and translatewiki people, I think having i18n as submodule is the easiest option.

...

3) What to do with submodules. A concise list of all the current externals will make this easier.

I think we should not have the same method with external libraries svn:external'ed in. I'd be more interested in including a pip-style requirements.txt, with an optional 'external libraries' zip package. Merlijn

Chad Horohoe

8 Jun 8 Jun

3:18 a.m.

On Fri, Jun 7, 2013 at 4:29 PM, Merlijn van Deen <valhallasw(a)arctus.nl>wrote;wrote:

...

Hi Chad, As we discussed on IRC, my suggestion is the following: On 5 June 2013 16:49, Chad Horohoe <chorohoe(a)wikimedia.org> wrote:

As a timeline, I would suggest to have continuous updates from svn to

...

git, roughly weekly updates to the conversion code and a transition period of maybe 6 weeks, starting next friday? I think it is realistic as a time frame to update most things that depend on svn.

This sounds reasonable to me.

...

Things I think we need to

1) Exact repo structure. I think we've gotten a good discussion started on the wiki, but I'd like to see more input (and please, feel free to

edit

it mercilessly, this was all written based on my naïve understanding of how you guys work)

This also looks good.

...

3) What to do with submodules. A concise list of all the current

externals

will make this easier.

Up to you guys. Depending on the size of the dependency and ease of packaging, some things can just be copy+pasted in (with licenses!) as needed. I think we're all in agreement on 2013 committers as being the initial group to have +2. I've updated the wiki to reflect this. -Chad

Merlijn van Deen

19 Jun 19 Jun

11:14 p.m.

The conversion is now up and running! https://git.wikimedia.org/project/pywikibot Gerrit mirrors SVN (hourly), so please do not submit any patches you actually want merged. Test patches are, of course, welcome. Merlijn

3962

days inactive

3998

days old

pywikibot@lists.wikimedia.org

Manage subscription

17 comments

6 participants

tags (0)

participants (6)

Amir Ladsgroup
Chad Horohoe
Hazard-SJ
legoktm
Maarten Dammers
Merlijn van Deen