Hey all!
We had set a deadline (Sept 30th) to ask WMDE for support to migrate your tools to Tool Labs. As very few of you made use of this until now, Johannes still has time to help you. So we are still accepting support requests. Please think about whether you'll have the time and the know-how to migrate on your own. Your tools shall have left the toolserver until June 30th, 2014. This deadline is serious.
If you would like Johannes to give you a hand, please tell us until November 15th 2013. * Write to me or to directly to johannes.kroll@wikimedia.de. * Give some details about your tool: Did you try to migrate already but something fails to work? What works? What doesn't? What exactly do you want Johannes to do?
Best, Silke
Hi Silke,
I'm afraid you're effort won't have the intended outcome. What I expect will happen: * Some tool authors will move in the next couple of months * Some tool authors will move at the last moment * A lot of tools willl just be abandoned. * You'll get a lot of upset people when you shut down the Toolserver
Don't say I didn't warn you,
Maarten
Op 7-10-2013 14:03, Silke Meyer schreef:
Hey all!
We had set a deadline (Sept 30th) to ask WMDE for support to migrate your tools to Tool Labs. As very few of you made use of this until now, Johannes still has time to help you. So we are still accepting support requests. Please think about whether you'll have the time and the know-how to migrate on your own. Your tools shall have left the toolserver until June 30th, 2014. This deadline is serious.
If you would like Johannes to give you a hand, please tell us until November 15th 2013.
- Write to me or to directly to johannes.kroll@wikimedia.de.
- Give some details about your tool: Did you try to migrate already but
something fails to work? What works? What doesn't? What exactly do you want Johannes to do?
Best, Silke
Maybe this just needs a little more planning. * Look at the server logs and find the most used tools on toolserver * Contact those tool authors directly * Make copies of the source of those tools publicly available (they should all be open source, but some directories are not accessibly for other users from shell IIRC) * Keep a Foundation-readable snapshot of the toolserver home directories, user-store, DB dumps etc. * Forcibly port the most important, unmaintained tools to Labs I have already started with one or two, though I'd appreciate it if I don't have to do it all by myself ;-)
On Mon, Oct 7, 2013 at 8:31 PM, Maarten Dammers maarten@mdammers.nl wrote:
Hi Silke,
I'm afraid you're effort won't have the intended outcome. What I expect will happen:
- Some tool authors will move in the next couple of months
- Some tool authors will move at the last moment
- A lot of tools willl just be abandoned.
- You'll get a lot of upset people when you shut down the Toolserver
Don't say I didn't warn you,
Maarten
Op 7-10-2013 14:03, Silke Meyer schreef:
Hey all!
We had set a deadline (Sept 30th) to ask WMDE for support to migrate your tools to Tool Labs. As very few of you made use of this until now, Johannes still has time to help you. So we are still accepting support requests. Please think about whether you'll have the time and the know-how to migrate on your own. Your tools shall have left the toolserver until June 30th, 2014. This deadline is serious.
If you would like Johannes to give you a hand, please tell us until November 15th 2013.
- Write to me or to directly to johannes.kroll@wikimedia.de.
- Give some details about your tool: Did you try to migrate already but
something fails to work? What works? What doesn't? What exactly do you want Johannes to do?
Best, Silke
______________________________**_________________ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.**orgToolserver-l@lists.wikimedia.org ) https://lists.wikimedia.org/**mailman/listinfo/toolserver-lhttps://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/** view/Mailing_list_etiquettehttps://wiki.toolserver.org/view/Mailing_list_etiquette
On 10/07/2013 06:27 PM, Magnus Manske wrote:
- Forcibly port the most important, unmaintained tools to Labs
Sadly, that's only possible when the code is unambiguously licensed as Open Source, which is often not the case (much of the code running on the Toolserver has no licensing information at all).
-- Marc
Don't we have to state the license for all our code to get a password renewal? I remember setting that at some point...
In any case, having a list of most accessed tools would be a start. At the very least we could determine if the licensing is a major issue.
On Mon, Oct 7, 2013 at 11:34 PM, Marc A. Pelletier marc@uberbox.org wrote:
On 10/07/2013 06:27 PM, Magnus Manske wrote:
- Forcibly port the most important, unmaintained tools to Labs
Sadly, that's only possible when the code is unambiguously licensed as Open Source, which is often not the case (much of the code running on the Toolserver has no licensing information at all).
-- Marc
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Hello, Am 08.10.2013 00:42, schrieb Magnus Manske:
Don't we have to state the license for all our code to get a password renewal? I remember setting that at some point...
yes, but you could and can chooses "none" or "all rights reserved" or "Freeware" or anything else. We do not force people to license their stuff as open source (and when I read something like "we force them to move", then I’m glad we didn’t).
Sincerely, DaB.
On Tue, Oct 8, 2013 at 7:47 PM, DaB. ts@dabpunkt.eu wrote:
We do not force people to license their stuff as open source (and when I read something like "we force them to move", then I’m glad we didn’t).
Because then we could just make a copy of the tool in that nasty
open-source way of Wikipedia, instead of having it die with the toolserver?
Hello Magnus, Am 09.10.2013 00:45, schrieb Magnus Manske:
Because then we could just make a copy of the tool in that nasty open-source way of Wikipedia, instead of having it die with the toolserver?
if somebody choose to move his/her tool to Labs: It’s their thing. But why forcing people? If somebody chose to not move his/her stuff and let the tools die: It’s their thing too.
Sincerely, DaB.
On 08/10/13 00:34, Marc A. Pelletier wrote:
On 10/07/2013 06:27 PM, Magnus Manske wrote:
- Forcibly port the most important, unmaintained tools to Labs
Sadly, that's only possible when the code is unambiguously licensed as Open Source, which is often not the case (much of the code running on the Toolserver has no licensing information at all).
-- Marc
Then the default license applies.
Hi Magnus and all!
- Make copies of the source of those tools publicly available (they should
all be open source, but some directories are not accessibly for other users from shell IIRC)
- Keep a Foundation-readable snapshot of the toolserver home directories,
user-store, DB dumps etc.
I found this this suggestion rather surprising and I'd like to discuss it with the TS admins first. Not sure if we can just do that. I'll get back to you.
Cheers, Silke
Hi again,
- Make copies of the source of those tools publicly available (they should
all be open source, but some directories are not accessibly for other users from shell IIRC)
- Keep a Foundation-readable snapshot of the toolserver home directories,
user-store, DB dumps etc.
I found this this suggestion rather surprising and I'd like to discuss it with the TS admins first. Not sure if we can just do that. I'll get back to you.
ok, I consulted Nosy, DaB. and Mette about this proposal and we will not make the home directories accessible to others. They can contain all sorts of private data, user names, e-mail addresses, ssh keys, who knows what else - maybe even gpg keys, so handing the complete package over is no option. I wouldn't want to "intrude" into the people's stuff. And we'd get in trouble with the law here, as the German data protection act is binding for us.
It's the slower way to contact all maintainers (which will happen repeatedly) but hey, we've got plenty of time left.
Cheers, Silke
On 10/10/2013 12:32 PM, Silke Meyer wrote:
ok, I consulted Nosy, DaB. and Mette about this proposal and we will not make the home directories accessible to others. They can contain all sorts of private data, user names, e-mail addresses, ssh keys, who knows what else - maybe even gpg keys, so handing the complete package over is no option. I wouldn't want to "intrude" into the people's stuff. And we'd get in trouble with the law here, as the German data protection act is binding for us.
I agree with this as well, for those very reasons. This is the reason why Tool Labs insists that any "real" tool is run from multi-maintainer project equivalents, and with clear licensing.
I would not allow user's own home directories to be turned over to a third party under any circumstances, this is why we want no code running there we might want to salvage in the future. And, likewise, I'm pretty sure the foundation wouldn't /want/ to be given custody over data that may or may not be private from contributors who have not given explicit leave to do so.
-- Marc
On Thu, Oct 10, 2013 at 5:58 PM, Marc A. Pelletier marc@uberbox.org wrote:
On 10/10/2013 12:32 PM, Silke Meyer wrote:
ok, I consulted Nosy, DaB. and Mette about this proposal and we will not make the home directories accessible to others. They can contain all sorts of private data, user names, e-mail addresses, ssh keys, who knows what else - maybe even gpg keys, so handing the complete package over is no option. I wouldn't want to "intrude" into the people's stuff. And we'd get in trouble with the law here, as the German data protection act is binding for us.
I agree with this as well, for those very reasons. This is the reason why Tool Labs insists that any "real" tool is run from multi-maintainer project equivalents, and with clear licensing.
I would not allow user's own home directories to be turned over to a third party under any circumstances, this is why we want no code running there we might want to salvage in the future. And, likewise, I'm pretty sure the foundation wouldn't /want/ to be given custody over data that may or may not be private from contributors who have not given explicit leave to do so.
To be clear, I was not suggesting to just dump all home directories on
bittorrent ;-)
public_html and cgi-bin should contain most code, and for small tools, could be checked quickly by hand for personal/secret data.
Also, of course I prefer people moving their own tools! Just in the important-tool-will-die-with-toolserver scenario...
Magnus
Hi Magnus!
Also, of course I prefer people moving their own tools! Just in the important-tool-will-die-with-toolserver scenario...
This question is being discussed in our office these days. I'll keep the list updated.
Cheers, Silke
Hello,
reading this I feel forced to publish my short Top50 usage of yesterday URLs from Toolserver.
Its just a quick hack on my console but I guess it is more clear output then probably a webalizer could offer. I just counted the lines in my access log from yesterday of how often a user dir was called. This is my command line: zcat access.log.1.gz|cut -d / -f 4,5|cut -d ? -f 1|sort |uniq -c|sort -rn
4410444 tiles/hikebike 3410590 ~cmarqu/hill 3030592 ~dschwen/wma 1568706 tiles/osm 651872 tiles/osm-labels-ru 452436 (should be toolserver.org) 242462 ~dapete/ime 191536 ~kolossos/openlayers 172239 ~para/geoip.fcgi HTTP 146808 ~kolossos/geoworld 117295 ~dschwen/iip 114200 tiles/bw-mapnik 112167 ~para/region.php 104377 tiles/osm-no-labels 99926 ~apper/pd 85028 ~osm/libs 63087 ~apper/sc 61585 ~dispenser/temp 59166 ~dapete/random 53725 ~dispenser/cgi-bin 50095 ~para/GeoCommons 48399 ~geohack/geohack.php 46208 ~lvova/cgi-bin 43747 images/wikimedia-toolserver-button.png HTTP 30192 ~tparis/autoedits 26460 ~geohack/siteicon.png HTTP 24401 ~master/osmjson 23565 ~dispenser/ghct 22924 ~cmarqu/opentiles.com 22199 ~mzmcbride/redirector 22089 ~daniel/potd 21928 geohack/geohack.php 20984 ~para/geoip.fcgi 20635 ~chm/whois.php HTTP 16343 ~quentinv57/sulinfo 15717 ~krinkle/I18N 11955 ~daniel/WikiSense 11533 ~soxred93/images 11007 ~authoritycontrol/redirect 9610 ~dapete/rss 8677 favicon.ico HTTP/1.1" 200 1150 - 8492 %7Egeohack/geohack.php 7913 tsthumb/tsthumb 7707 tiles/osm-labels-de 7460 ~rhaworth/os 7363 ~dispenser/resources 6789 ~para/cgi-bin 6561 tiles/osm-labels-en 6543 ~dapete/wikinews-rss 6205 ~dapete/ImageMapEdit
Interesting.
Most of this seems to be related to OSM/geo-something. Not sure how much could actually be ported to Labs at the moment.
There are some tools from e.g. apper that would probably move quite well.
Also, there is geohack, which I had moved to Labs quite a while ago ( http://tools.wmflabs.org/geohack/ ). Not sure why it's not redirected/URLs on Wikipedia haven't changed yet.
On Wed, Oct 9, 2013 at 1:26 PM, Marlen Caemmerer < marlen.caemmerer@wikimedia.de> wrote:
Hello,
reading this I feel forced to publish my short Top50 usage of yesterday URLs from Toolserver.
Its just a quick hack on my console but I guess it is more clear output then probably a webalizer could offer. I just counted the lines in my access log from yesterday of how often a user dir was called. This is my command line: zcat access.log.1.gz|cut -d / -f 4,5|cut -d ? -f 1|sort |uniq -c|sort -rn
4410444 tiles/hikebike 3410590 ~cmarqu/hill 3030592 ~dschwen/wma 1568706 tiles/osm 651872 tiles/osm-labels-ru 452436 (should be toolserver.org) 242462 ~dapete/ime 191536 ~kolossos/openlayers 172239 ~para/geoip.fcgi HTTP 146808 ~kolossos/geoworld 117295 ~dschwen/iip 114200 tiles/bw-mapnik 112167 ~para/region.php 104377 tiles/osm-no-labels 99926 ~apper/pd 85028 ~osm/libs 63087 ~apper/sc 61585 ~dispenser/temp 59166 ~dapete/random 53725 ~dispenser/cgi-bin 50095 ~para/GeoCommons 48399 ~geohack/geohack.php 46208 ~lvova/cgi-bin 43747 images/wikimedia-toolserver-**button.png HTTP 30192 ~tparis/autoedits 26460 ~geohack/siteicon.png HTTP 24401 ~master/osmjson 23565 ~dispenser/ghct 22924 ~cmarqu/opentiles.com 22199 ~mzmcbride/redirector 22089 ~daniel/potd 21928 geohack/geohack.php 20984 ~para/geoip.fcgi 20635 ~chm/whois.php HTTP 16343 ~quentinv57/sulinfo 15717 ~krinkle/I18N 11955 ~daniel/WikiSense 11533 ~soxred93/images 11007 ~authoritycontrol/redirect 9610 ~dapete/rss 8677 favicon.ico HTTP/1.1" 200 1150 - 8492 %7Egeohack/geohack.php 7913 tsthumb/tsthumb 7707 tiles/osm-labels-de 7460 ~rhaworth/os 7363 ~dispenser/resources 6789 ~para/cgi-bin 6561 tiles/osm-labels-en 6543 ~dapete/wikinews-rss 6205 ~dapete/ImageMapEdit
______________________________**_________________ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.**orgToolserver-l@lists.wikimedia.org ) https://lists.wikimedia.org/**mailman/listinfo/toolserver-lhttps://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/** view/Mailing_list_etiquettehttps://wiki.toolserver.org/view/Mailing_list_etiquette
Hi,
Am 09.10.2013, 14:55 Uhr, schrieb Magnus Manske magnusmanske@googlemail.com:
There are some tools from e.g. apper that would probably move quite well.
I just started moving some simple tools a few days ago, but the availability seems very bad to me. Sometimes the tools work, sometimes not - at the moment nothing works for me, not only my tools but also e.g. http://tools.wmflabs.org/add-information/ or http://tools.wmflabs.org/catfood/
It was told that the toolserver will die one year after "Tool Labs has all necessary features to replace the Toolserver.". One feature is availability - and for now it's impossible for me to move tools if they only work 50% of the day. So either the availability will be at nearly 100% or the shutdown will be delayed.
Regards, Christian / APPER
It is annoying, but to be fair, it's "only" been this bad since yesterday or so. Usually it's OK.
On Thu, Oct 10, 2013 at 11:50 AM, Christian Thiele apper@apper.de wrote:
Hi,
Am 09.10.2013, 14:55 Uhr, schrieb Magnus Manske < magnusmanske@googlemail.com>:
There are some tools from e.g. apper that would probably move quite well.
I just started moving some simple tools a few days ago, but the availability seems very bad to me. Sometimes the tools work, sometimes not
- at the moment nothing works for me, not only my tools but also e.g.
http://tools.wmflabs.org/add-**information/http://tools.wmflabs.org/add-information/or http://tools.wmflabs.org/**catfood/ http://tools.wmflabs.org/catfood/
It was told that the toolserver will die one year after "Tool Labs has all necessary features to replace the Toolserver.". One feature is availability
- and for now it's impossible for me to move tools if they only work 50% of
the day. So either the availability will be at nearly 100% or the shutdown will be delayed.
Regards, Christian / APPER
______________________________**_________________ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.**orgToolserver-l@lists.wikimedia.org ) https://lists.wikimedia.org/**mailman/listinfo/toolserver-lhttps://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/** view/Mailing_list_etiquettehttps://wiki.toolserver.org/view/Mailing_list_etiquette
On Wed, 9 Oct 2013, Magnus Manske wrote:
Most of this seems to be related to OSM/geo-something. Not sure how much could actually be ported to Labs at the moment.
True. Part of it is that we have to look at the caching options etc of the tiles but the demand is undeniable. I guess the OSM stuff needs the Wikipedia DBs and the OSM DBs most of the time.
This means its quite important that labs can provide a postgres instance that servers OSM and tile rendering + some TBs of storage. Its starts to be important that Labs has these features.
Also, there is geohack, which I had moved to Labs quite a while ago ( http://tools.wmflabs.org/geohack/ ). Not sure why it's not redirected/URLs on Wikipedia haven't changed yet.
Had a quick look - to me it looks like the htaccess file does not have to regarding redirects. Another probably more sustainable way would be to change the links pointing there.
Cheers nosy
Hi Marlen,
Op 9-10-2013 14:26, Marlen Caemmerer schreef:
Hello,
reading this I feel forced to publish my short Top50 usage of yesterday URLs from Toolserver.
Its just a quick hack on my console but I guess it is more clear output then probably a webalizer could offer. I just counted the lines in my access log from yesterday of how often a user dir was called. This is my command line: zcat access.log.1.gz|cut -d / -f 4,5|cut -d ? -f 1|sort |uniq -c|sort -rn
Nice list. These statistics provide a good starting point on where to focus effort on. Some thoughts: * Can you make such statistics on a regular basis? Maybe weekly with a bigger sample * Can you publish the list somewhere on a wiki? * Can you make this list longer? * Is it possible to also make aggregated statistics per username? * Is it possible to also make these statistics only focused on multi maintainer projects? These should be easier to move
I ran into two things I think could be structurally fixed: * I need a place to store my code (now: Toolserver svn) * I need a place to track bugs (now:Jira)
Wouldn't it be nice that when you create a new project that you automatically get your own git repository (/toollabs/<your tool>/? at https://git.wikimedia.org/) and your own component under https://bugzilla.wikimedia.org/enter_bug.cgi?product=Tool%20Labs%20tools ? I'm sure someone with puppet skills could set this up.
Maarten
+1 on Maarten and also Magnus on public_html etc. I also assume all SVN repositories will be exported, JIRA & tswiki etc. archived, license information in LDAP preserved somewhere? The roadmap contains no information whatsoever.
Nemo
Hello, Hello all, Am 13.10.2013 11:51, schrieb Federico Leva (Nemo):
+1 on Maarten and also Magnus on public_html etc.
the cgi-dirs could contain passwords, so: no.
I also assume all SVN repositories will be exported,
If somebody ask we will give the owner a svn-dump. Until now nobody asked AFAIK.
JIRA & tswiki etc. archived,
That’s unclear at the moment.
license information in LDAP preserved somewhere?
The LDAP with all data within will be deleted before the TS is shut down for data privacy reasons. Maybe a root will do a username/license-extract before.
The roadmap contains no information whatsoever.
I’m sure Silke will handle this.
Nemo
Sincerely, DaB.
Then I officially ask a svn dump of all SVN repos. Importing them on sourceforge looks rather easy: http://sourceforge.net/p/forge/community-docs/Subversion/ (But I'm sure you can easily get them imported in gerrit too if you ask a repo when it's time.)
Nemo
On 13 October 2013 20:29, Federico Leva (Nemo) nemowiki@gmail.com wrote:
Then I officially ask a svn dump of all SVN repos. Importing them on sourceforge looks rather easy: http://sourceforge.net/p/** forge/community-docs/**Subversion/http://sourceforge.net/p/forge/community-docs/Subversion/
You can also mirror them without any admin intervention:
svnadmin create svnroot svnsync init file://$PWD/svnroot http://svn.wikimedia.org/svnroot/pywikipedia
then svnsync sync file://$PWD/svnroot
to update. Of course, change svnroot and the svn repo path to wherever it is located.
Yes, I thought that was a typo. Why only the owner? Doesn't seem to make any sense. I can do screenscraping of your SVN, but looks rather tedious.
Nemo
Nice, turns out any user can migrate a SVN repo history to Git, so it doesn't matter that much if dumps are not provided. http://git-scm.com/book/en/Git-and-Other-Systems-Migrating-to-Git
Nemo
toolserver-l@lists.wikimedia.org