Toolserver-l

toolserver-l@lists.wikimedia.org

1612 discussions

sql-s5 (dewiki) will be read-only tomorrow
by DaB. 21 Feb '13

21 Feb '13

Hello all, s5 (dewiki) is the next cluster for re-setup. I already got a fresh dump from the WMF and imported it too on a server. The next step is to move the user- databases over. For this I have to switch s5 into read-only. The read-only- phase will start at Tomorrow, Thursday after 13:00 UTC. s5 carries a few very big user-databases, so I think the movement will take more an 1 day. Below is a list of the biggest user-databases, maybe their owners can take a look if some clean-up is possible? Unfortunately s5 can not stay at the new place because the server belongs to OSM. So a second read-only-phase will happen in near future (that one should be a bit faster because I will just rsync files around instead of importing). Sincerely, DaB. 1.2G u_lvova_golem_s2_zh 1.3G u_hroest 1.3G u_rdf_bots 1.6G u_multichill_commons_categories_p 1.7G u_lvova_golem_s5_de 1.8G u_mauro742 2.7G u_dschwen 20G p_merlbot_dewiki_lists 51G p_globalusage_weekly_p -- Userpage: [[:w:de:User:DaB.]] — PGP: 0x2d3ee2d42b255885

3 4

Wikidata-moving
by DaB. 21 Feb '13

21 Feb '13

Hello all, it looks like the WMF-techs are moving wikidata to cluster s5 at the moment (without announcement of course). I'm quite sure that this will break our wikidata-replication; so after my sleep I will look how the situation will be and than probably re-import wikidata everywhere (wikidata is still quite small at the moment so that's still possible). So expect the wikidata replag to rise during the next hours. Sincerely, DaB. -- Userpage: [[:w:de:User:DaB.]] — PGP: 0x2d3ee2d42b255885

2 1

Re: [Toolserver-l] Visibility of tools
by Tim Landscheidt 20 Feb '13

20 Feb '13

Silke Meyer <silke.meyer(a)wikimedia.de> wrote: > I was thinking about the list of tools a bit more... In my opinion, this > collection of tools should be more visible to the world! There is so > much work and knowledge in there and partially you really have to dig to > find them... > There is this plan to create/improve Labs/Tool Labs so that tools can be > shared with others more easily. So one very important thing is to *know* > about existing tools. > How about dragging these tools into the spotlight? (The ones on the > toolserver as well as the ones already living in Labs.) The other day I > heard the proposal from a community member to have featured tools in a > visible place, e.g. a "tool of the month" or even of the week. I like > the idea! It could shortly present a tool, what it does, where to find > it and the person(s) behind. > What do you think of it? The first place that came to my mind where The > Signpost and/or German language Kurier where we could suggest this > topic. I'll be afk for the next few days and I'd be glad to read some > opinions when back. :) I've suggested a while back (https://jira.toolserver.org/browse/TS-1271) to import the old Toolserver blog/comments to blog.wikimedia.de and then post status and ongoing project reports there. It has a different audience, probably more (potential) donors :-), but showing them what "some server in Amsterdam" is used for isn't a bad idea IMHO. amaranth needs to be fixed before that, though :-). Tim

4 3

New Rule: SGE-constraint for bots
by DaB. 20 Feb '13

20 Feb '13

Hello all, today I discovered (thanks to the mailing-list) that a few users have run several bot-instances in parallel on willow. I'm sure that these people did it by mistake, but it is annoying nevertheless and it is easy to fix: Use SGE. The problem is that I wrote several of eMails about "use SGE!" already and somehow it did not work as good as it should (if you converted you stuff already: thank you and you can stop to read here ;-)). I understand that we all are busy with our lives and Wikipedia and that we all love to "do it right…later", but as you know that resources of the toolserver are limited. So I hereby declare the following new rule: All bots have to run by SGE. A bot is every program or script that makes changes at a Wikimedia project. It does not matter if the bot runs periodically or continuous. The only exclusions are a.) interactive bots, b.) bots that can't run by SGE yet and c.) if you start a bot by hand for testing (no screen, no cron, no while). The rule will become active at Sunday, 10. February 2013. Exception b is almost NEVER the case, if it runs on a shell it is VERY likely that it can run by SGE. Some time ago I wrote a simple SGE-how-to at [1]. Maybe you all can take a look and correct things and make things more clear. In very most cases the using of SGE IS easy. Sincerely, DaB. [1] https://wiki.toolserver.org/view/SGE_for_beginners -- Userpage: [[:w:de:User:DaB.]] — PGP: 2B255885

5 10

List of tools on the toolserver - Help needed
by Silke Meyer 19 Feb '13

19 Feb '13

Hey all! I am impressed! Wondering what tools are actually running on the toolserver I started a list. It is long! You can admire it here: http://www.mediawiki.org/wiki/Toolserver/List_of_Tools For sure, it is incomplete as I used the toolserver wiki and jira to find information. In the light of building Tool Labs to become a platform the tools can run on, we need more information about the tools. So here is what I ask all tool developers for: * Please check if your tools are on the list (correctly). * Please fill in your software dependencies, data dependencies, use patterns (are they running continously? or webservices? Batch runs? etc.), the license. * If you have not given your software an explicit license, please note that only free software can migrate to Labs. Consider putting your stuff under a free license. * If you have already migrated your bot or tool to Labs and it is in the list, please say so in the last column "status". The same if a mentioned tool is abandoned and shall not move. Thanks for your help! Cheers, Silke -- Silke Meyer Systemadministratorin und Projektassistenz Wikidata Wikimedia Deutschland e.V. | Obentrautstr. 72 | 10963 Berlin Tel. (030) 219 158 260 http://wikimedia.de Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e.V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/681/51985.

6 8

Block mail from dysgo.org?
by Christopher David Howie 17 Feb '13

17 Feb '13

We've been receiving messages from this domain at unblock(a)toolserver.org and they appear to be related to this: <http://www.reddit.com/r/WTF/comments/l44r1/i_just_got_this_email_at_work_i_…>. Viral advertising for some film. In reality, it's a message with a crapload of images attached serving no purpose for us. Can we just block this whole domain from sending mail to toolserver accounts? It's a nuisance, and the messages are quite large. -- Chris Howie http://www.chrishowie.com http://en.wikipedia.org/wiki/User:Crazycomputers If you correspond with me on a regular basis, please read this document: http://www.chrishowie.com/email-preferences/ PGP fingerprint: 2B7A B280 8B12 21CC 260A DF65 6FCE 505A CF83 38F5 ------------------------------------------------------------------------ IMPORTANT INFORMATION/DISCLAIMER This document should be read only by those persons to whom it is addressed. If you have received this message it was obviously addressed to you and therefore you can read it. Additionally, by sending an email to ANY of my addresses or to ANY mailing lists to which I am subscribed, whether intentionally or accidentally, you are agreeing that I am "the intended recipient," and that I may do whatever I wish with the contents of any message received from you, unless a pre-existing agreement prohibits me from so doing. This overrides any disclaimer or statement of confidentiality that may be included on your message.

3 3

Coren joining WMF Ops department as Tools Labs contractor
by Sumana Harihareswara 16 Feb '13

16 Feb '13

Welcome, Coren (Marc-Andre Pelletier); I look forward to working with you to help build Tools Labs and help those who'd like to move bots and webtools to the Wikimedia Labs infrastructure. You can look at his TODO list here: https://www.mediawiki.org/wiki/Wikimedia_Labs/TODO and his profile here: https://meta.wikimedia.org/wiki/User:MPelletier_%28WMF%29 Coren, thank you for agreeing to join us and help work on this. best, Sumana Harihareswara Engineering Community Manager Wikimedia Foundation -------- Original Message -------- Subject: [Engineering] Engineering Announcement: Marc-Andre Pelletier joins TechOps Date: Fri, 15 Feb 2013 23:25:09 -0800 From: Ct Woo <ctwoo(a)wikimedia.org> To: Wikimedia developers <wikitech-l(a)lists.wikimedia.org>, Development and Operations Engineers <engineering(a)lists.wikimedia.org> All, I am pleased to announce Marc-Andre will be joining us as aTechnical Operations Engineer (contractor) starting this February 25, 2013. His primary focus will be to build up the Wikimedia Labs infrastructure and to assist the community developers to migrate their tools to that infrastructure, especially those residing on the Toolserver today. Marc-Andre is an active wikipedian and he is better known as 'Coren' on English Wikipedia and has been a volunteer editor since 2006 where he has served as administrator and arbitrator. He also always kept himself involved with the technical and procedural aspects of automated editing (so-called bots), having written and operated a copyright-violation detection bot for several years. Marc has been a Unix system administrator and occasional computer science instructor for 20+ years, in fields ranging from telecommunication to game development. He studied IT at École de Technologie Supérieure (Canada). Please join me to welcome him and you can find him on IRC (freenode.net) using the nick 'Coren'. Thanks. CT Woo

1 0

Cron errors at hawthorn for qcronsub
by Krinkle 15 Feb '13

15 Feb '13

I get a bunch of these every few weeks. Got them again today/yesterday. What's up? -- Krinkle Begin forwarded message: > From: root(a)toolserver.org (Cron Daemon) > Subject: Cron <krinkle@hawthorn> qcronsub -N dbbot_wm -m n -j y -b y -l arch='*' -l h_rt=INFINITY -l virtual_free=90M "$HOME/bots/dbbot-wm-start.sh" > Date: February 15, 2013 8:55:08 AM GMT+01:00 > To: krinkle(a)toolserver.org > > error: JSV stderr: Traceback (most recent call last): > error: JSV stderr: File "/sge/GE/bin/sol-amd64/qjobtest", line 108, in <module> > error: JSV stderr: dom = minidom.parse(child_stdout) > error: JSV stderr: File "/opt/ts/python/2.7/lib/python2.7/site-packages/_xmlplus/dom/minidom.py", line 1915, in parse > error: JSV stderr: return expatbuilder.parse(file) > error: JSV stderr: File "/opt/ts/python/2.7/lib/python2.7/site-packages/_xmlplus/dom/expatbuilder.py", line 930, in parse > error: JSV stderr: result = builder.parseFile(file) > error: JSV stderr: File "/opt/ts/python/2.7/lib/python2.7/site-packages/_xmlplus/dom/expatbuilder.py", line 207, in parseFile > error: JSV stderr: parser.Parse(buffer, 0) > error: JSV stderr: xml.parsers.expat.ExpatError: syntax error: line 1, column 0 > Unable to run job: JSV stderr: Traceback (most recent call last): > JSV stderr: File "/sge/GE/bin/sol-amd64/qjobtest", line 108, in <module> > JSV stderr: dom = minidom.parse(child_stdout) > JSV stderr: File "/opt/ts/python/2.7/lib/python2.7/site-packages/_xmlplus/dom/minidom.py", line 1915, in parse > JSV stderr: return expatbuilder.parse(file) > JSV stderr: File "/opt/ts/python/2.7/lib/python2.7/site-packages/_xmlplus/dom/expatbuilder.py", line 930, in parse > JSV stderr: result = builder.parseFile(file) > JSV stderr: File "/opt/ts/python/2.7/lib/python2.7/site-packages/_xmlplus/dom/expatbuilder.py", line 207, in parseFile > JSV stderr: parser.Parse(buffer, 0) > JSV stderr: xml.parsers.expat.ExpatError: syntax error: line 1, column 0 > JSV stderr is - xml.parsers.expat.ExpatError: syntax error: line 1, column 0. > Exiting. Begin forwarded message: > From: root(a)toolserver.org (Cron Daemon) > Subject: Cron <krinkle@hawthorn> qcronsub -N dbbot_wm -m n -j y -b y -l arch='*' -l h_rt=INFINITY -l virtual_free=90M "$HOME/bots/dbbot-wm-start.sh" > Date: February 15, 2013 1:45:09 AM GMT+01:00 > To: krinkle(a)toolserver.org > > error: JSV stderr: Traceback (most recent call last): > error: JSV stderr: File "/sge/GE/bin/sol-amd64/qjobtest", line 108, in <module> > error: JSV stderr: dom = minidom.parse(child_stdout) > error: JSV stderr: File "/opt/ts/python/2.7/lib/python2.7/site-packages/_xmlplus/dom/minidom.py", line 1915, in parse > error: JSV stderr: return expatbuilder.parse(file) > error: JSV stderr: File "/opt/ts/python/2.7/lib/python2.7/site-packages/_xmlplus/dom/expatbuilder.py", line 930, in parse > error: JSV stderr: result = builder.parseFile(file) > error: JSV stderr: File "/opt/ts/python/2.7/lib/python2.7/site-packages/_xmlplus/dom/expatbuilder.py", line 207, in parseFile > error: JSV stderr: parser.Parse(buffer, 0) > error: JSV stderr: xml.parsers.expat.ExpatError: syntax error: line 1, column 0 > Unable to run job: JSV stderr: Traceback (most recent call last): > JSV stderr: File "/sge/GE/bin/sol-amd64/qjobtest", line 108, in <module> > JSV stderr: dom = minidom.parse(child_stdout) > JSV stderr: File "/opt/ts/python/2.7/lib/python2.7/site-packages/_xmlplus/dom/minidom.py", line 1915, in parse > JSV stderr: return expatbuilder.parse(file) > JSV stderr: File "/opt/ts/python/2.7/lib/python2.7/site-packages/_xmlplus/dom/expatbuilder.py", line 930, in parse > JSV stderr: result = builder.parseFile(file) > JSV stderr: File "/opt/ts/python/2.7/lib/python2.7/site-packages/_xmlplus/dom/expatbuilder.py", line 207, in parseFile > JSV stderr: parser.Parse(buffer, 0) > JSV stderr: xml.parsers.expat.ExpatError: syntax error: line 1, column 0 > JSV stderr is - xml.parsers.expat.ExpatError: syntax error: line 1, column 0. > Exiting. Begin forwarded message: > From: root(a)toolserver.org (Cron Daemon) > Subject: Cron <krinkle@hawthorn> qcronsub -N dbbot_wm -m n -j y -b y -l arch='*' -l h_rt=INFINITY -l virtual_free=90M "$HOME/bots/dbbot-wm-start.sh" > Date: February 14, 2013 8:30:06 PM GMT+01:00 > To: krinkle(a)toolserver.org > > error: JSV stderr: Traceback (most recent call last): > error: JSV stderr: File "/sge/GE/bin/sol-amd64/qjobtest", line 108, in <module> > error: JSV stderr: dom = minidom.parse(child_stdout) > error: JSV stderr: File "/opt/ts/python/2.7/lib/python2.7/site-packages/_xmlplus/dom/minidom.py", line 1915, in parse > Unable to run job: JSV stderr: Traceback (most recent call last): > JSV stderr: File "/sge/GE/bin/sol-amd64/qjobtest", line 108, in <module> > JSV stderr: dom = minidom.parse(child_stdout) > JSV stderr: File "/opt/ts/python/2.7/lib/python2.7/site-packages/_xmlplus/dom/minidom.py", line 1915, in parse > JSV stderr is - File "/opt/ts/python/2.7/lib/python2.7/site-packages/_xmlplus/dom/minidom.py", line 1915, in parse. > Exiting.

2 1

Cron on submit
by Dr. Trigon 13 Feb '13

13 Feb '13

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello! There were 2 messages here during Janury reporting problems with cron. I am now noticing issues with my cronjobs too. By looking at [1] you are able to see that the strange behaviour started somewhen week 2 and 3 (mid January). Do we have again cron (the server) running out of memory or what is the issue here? DaB can you may be give some hints here? Or someone else? [1] http://munin.toolserver.org/Login/hawthorn/cron_jobs_sh.html Thanks a lot and greetings! DrTrigon -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.13 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlEO31oACgkQAXWvBxzBrDDGugCg1jb5AWvJJfNlBJjebcfOA2cr tPAAoNtSSs6auQFkp1unbFpEv+Zi07Zu =u2tk -----END PGP SIGNATURE-----

6 7

Postmortem: Partial Toolserver-outage
by DaB. 12 Feb '13

12 Feb '13

Hello all, great parts of the toolserver-cluster were down or very slow in the last few hours. AFAIS it was a problem with the user-store or rosemary (where the user- store is physically connected). I rebooted rosemary, but the reboot showed problems with its IPv6-address. I tried to fix that what caused several other reboots. Rosemary is now up and running but the user-store is not available (looks like Nosy just mounted it without updating the fstab-file). So I was forced to remove the user-store everywhere (beside on willow because it need a reboot to do that and a reboot is scheduled already later for today). I will try if I can find the partition for user-store and mount it but I have not much hope (there are way to many devices to try) – just to be clear: There is no data lost. Also away will be munin, because its data is also mounted on that host. I fear that we have to wait for Nosy to recover before we get the user-store back. tl;dr: TS had problems, user-store is away. Sincerely, DaB. -- Userpage: [[:w:de:User:DaB.]] — PGP: 0x2d3ee2d42b255885

2 4

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

Toolserver-l