Toolserver-l June 2009

toolserver-l@lists.wikimedia.org

28 participants
16 discussions

New hardware ordered
by Daniel Kinzler 28 Jun '11

28 Jun '11

Hi all I'm happy to let you know that new hardware has been ordered by Wikimedia Deutschland and will arrive probably in about two weeks. We will get two new systems: * A more powerful web server, to replace hemlock: Sun Fire X4150, 2x Quad-Core Xeon, 8GB RAM, 2x73GB SAS HDD. The current web server only has two cores. * Another database server, to be used for S1 (english wikipedia), so S1 and S3 no longer have to share a server: Sun Fire X4250, 2x Quad-Core Xeon, 32GB RAM, 16x146GB SAS RAID. This should improve performance and give us some head space for growth. Once the new servers arrive, S3 will be re-imported too, so we will have live data again. Any ideas for names? To stay with the nightshade theme, how about Jurubeba and Erubia? Or perhaps we go the "witches' weed" way, with Datura and Mandrake? Henbane is taken, i think. Amanita sounds nice, too :) A third server has been ordered, which will also be installed in Amsterdam, but will not be part of the toolserver cluster. It's a storage server (X4540, 24TB RAID) that will keep a live backup of all media files. Cheers, Daniel

11 17

Re: [Toolserver-l] Betawiki translations
by Luxo 20 Mar '10

20 Mar '10

>> hi all >> >> many of our tools uses messages from betawiki because they are >> translated in many languages. >> But this messages must be synchronized and I think it's not very >> economically if every user/tool do that self. Because of that I >> propose >> to create one database for all users which will be sync periodically. > >A database with one table per user? Sounds good. > >> What do you think? Are any other users interested for such a db? > >I would be interested. > >> Greetings, >> Luxo > >Pietrodn >powerpdn(a)gmail.com I thought not one table per user, but rather one table for all messages. On betawiki are a lot messages e.g. http://translatewiki.net/wiki/MediaWiki:Abusefilter-revert-title/en. So I would make a table something like + ----------+------------------------+-----------------------------------------------------+ | LANG | MESSAGE | TEXT | + ----------+------------------------+-----------------------------------------------------+ | en | Abusefilter-revert-title | Revert all changes by filter $1 | + ----------+------------------------+-----------------------------------------------------+ | de | Abusefilter-revert-title | Alle Änderungen durch Filter $1 rückgängig machen | + ----------+------------------------+-----------------------------------------------------+ | fr | Abusefilter-revert-title | Révoquer toutes les modifications par le filtre $1 | + ----------+------------------------+-----------------------------------------------------+ the messages which will be synced should be one a list which every ts-user can edit, so that it's easy for every user to add the messages he requires. --Luxo

7 23

s2 server status
by River Tarnell 05 Jul '09

05 Jul '09

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 hi, the new s2 server (daphne) is online, and should have caught up with replication later today. however, it is using a new MySQL build (version 5.1.35), which contains a critical InnoDB bug causing server crashes when innodb_locks_unsafe_for_binlog is enabled. since we need this option to support long queries, we won't switch servers until a patch is available. (this is a new bug, unrelated to the previous one.) i hope this will happen later this week. when we switch, user databases currently on the s2 server will _not_ be moved, and the 'sql' alias will continue to point at the old server (zedler). therefore, if you have created tables on the old server which you need to join to s2 databases, you will need to move these to the new server yourself. do not move normal user databases, they should remain on 'sql' as before. - river. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (HP-UX) iEYEARECAAYFAko+R7MACgkQIXd7fCuc5vIV9gCdGvqGabLQCienTQOB3Lk5ClyP l3EAoIZebLgBV8X6zeUIspNZNn1C8mlE =uYxB -----END PGP SIGNATURE-----

2 3

Little Project Idea
by K. Peachey 28 Jun '09

28 Jun '09

Heres a little side project idea for anyone interested: I often see people wanting dumps of one or two articles from a Wikipedia wiki (most commonly en) and most commonly have to resort to screen scraping via third party tools which isn't very nice and some times even blocked (at the squid level) although there is the API, it isn't exactly the most easier system for anyone to use. What might be nice is a little tool where people can enter a few article names in a box and click a button and have it produce the static html dumps of the desired article(/s).

7 10

[stable] server migration
by River Tarnell 21 Jun '09

21 Jun '09

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 hi, given the current low uptake of stable server usage, we have decided we cannot afford to dedicate an entire server to it, which ends up being mostly idle. therefore we will be migrating the stable server to a zone (virtual machine) on hyacinth, which is also the new /home server. there should be no change in either reliability or performance as a result. unlike the previous move, no user intervention will be required this time. we are currently copying the existing /users and /projects directories to the new server. at switchover time, an incremental rsync will be done to copy any changes since then. you can log into the new server now, using the hostname "newstable.toolserver.org"; however, i recommend not changing any files, as they will be overwritten when the next copy is done. if you would prefer to edit files on the new server, let me know and i will exclude your project from the automatic copy. (it may take a while until you can log in, since not all .ssh directories have been copied yet.) i *do* recommend doing a quick sanity check of your software on the new server; for example, that any software or libraries it needs are available and work correctly. downtime for the final migration will be on Monday, 6th June, in line with the maintenance schedule for stable. we might begin the downtime slightly earlier than usual (5AM UTC) to allow time for the file copies. i would like to change to ZWS at the same time as the migration; if your project has no yet confirmed that it works with ZWS, *please* consider trying to do so before the migration, as it will make things much easier for us. this applies to: * wma * wmfgcbot if you are unable to complete the web server migration yourself, please ask for help, either on IRC or the mailing list. the move is accompanied by a more recent version of the TS package repository (/opt/ts), which includes newer versions of Perl, Python and other software. this is documented in more detail at: <https://wiki.toolserver.org/view/Solaris_software> for projects using third-party Python modules currently installed in /opt/ts/lib/python2.4/lib/site-packages, these will still be available after the move. however, we recommend changing to the Toolserver Python. after the move is complete, the existing stable server (willow) will be repurposed into a general login server. more details on this will follow later. lastly, we are still interested in more stable projects. stable server projects can have multiple maintainers, which reduces the workload on one maintainer, and means tools don't die when their maintainers leave. we are currently considering disabling the unmaintained web tools of expired user accounts, meaning these tools will stop working. stable tools will only be disabled if *all* maintainers leave, and no new maintainer can be found. additionally, the stable Toolserver has a well-defined maintenance schedule to avoid downtime during peak times, and has much less unplanned downtime than the normal Toolserver. if you would like to move a project to stable, open a request in the TS project in JIRA, describing the project and listing the initial maintainer(s). - river. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (HP-UX) iEYEARECAAYFAko8YXMACgkQIXd7fCuc5vKkogCgkUtkExNsFNbczDCq9d4IbT6v S0sAmgLHns9mqnkzDwMjkICWpVnNJsI1 =11m+ -----END PGP SIGNATURE-----

7 9

communication with wiki communities
by Daniel Kinzler 17 Jun '09

17 Jun '09

Hi all during wikimedia germany's brithday party yesterday, I was asked to look into improving the communication between toolserver folks and the wiki communities using the toolserver. We have become an important part of their infrastructure, and when things break, they notice - but often don't know who to ask about it. So, I would like to propose a few things to improve this situation. 1) There should be a short description of the toolserver, with the relevant links, in the project namespace of each wiki that us the toolserver significantly. That page should be maintained by the local community of course, but I think it would be ideal if a toolserver user who is also a member of that community would help with that, or even start the page. I will do this for the german wikipedia soon. 2) we, the toolserver admins, should use the toolserver blog more to communicate what's going on with the TS. I guess several of the messages I posted here lately should have gone to the blog too. Maybe blog posts could be forwarded to toolserver-l automatically? posting the same thing to several places is annoying... 3) we should incurage people to describe their tools on the toolserver wiki, kind of like the description pages for extension on mediawiki.org. Perhaps we should have a guideline for this? The idea is to promote some more information about who we are, what we do, and how the toolserver works. And we should also help people to find the right person to contact when things break. What do you think? -- daniel

13 27

Re: [Toolserver-l] communication with wiki communities /Support the wiki
by Tim Alder 15 Jun '09

15 Jun '09

Hello, I also believe that it would a big benefit if we would have a better overview over existing tools in the toolserver wiki. So I created today some pages for my own tools there. Because I had already project pages for this I only link to them, which wasn't hard work. The benefit is that we can begin to categories tools by topics. I hope other developer see also that it is necessary and do the same. The documentation can avoid that we do things duplicated and you can find an alternative tool easier. It would be also good if we can show what we have if we want more money for better hardware... In the wiki it would be perhaps good if we would have a "tool" namespace. So you can better search for tools or showing all tools. Or should we use the toolserver wiki as semanticwiki-playground ? Greetings Kolossos

3 2

new server status; downtime notification
by River Tarnell 15 Jun '09

15 Jun '09

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 hi, the two new Toolservers, hyacinth (/home server) and daphne (s2 database) have been installed. on Monday, June 15, between 5-7AM UTC, i will move /home from hemlock to hyacinth; this will involve some downtime as the entire filesystem has to be copied. the total downtime should be less than 2 hours. more information will be forthcoming later regarding the new s2 database, and the OpenStreetMap Toolserver, which was also installed today. - river. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (HP-UX) iEYEARECAAYFAko0FGwACgkQIXd7fCuc5vIQ8QCdGjAp23Ug0Aa/X+0633PsnUYG hT4AmQHOnuLToyrUBFTHtPCMLUnH+daQ =Ch82 -----END PGP SIGNATURE-----

1 1

disk usage
by River Tarnell 15 Jun '09

15 Jun '09

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 hi, as usual, the following list contains users whose home directories are using more than 1GB of disk space. please consider checking your home directory and deleting any files you no longer need. space is listed in kilobytes. 1044294 mzmcbride 1077935 leon 1138620 vvv 1231049 tawker 1257963 autocracy 1461143 erenrich 1520995 danny_b 1714427 purodha 1898741 fmaunier 2388170 misza13 2498892 darkdadaah 2620823 flacus 2929883 skyluke 3352872 stwalkerster 4541646 henna 5029182 drh08 5070470 tparscal 5637043 gmaxwell 6229967 kolossos 7113430 werdna 9724094 dschwen 10780027 cbm 15499050 bryan 15866279 emijrp 21621695 voj 33116919 sk 56239790 daniel 117855613 soxred93 - river. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (HP-UX) iEYEARECAAYFAko2joUACgkQIXd7fCuc5vLVRACfWRMPy/fW4sUApfCS84V0yEgW Og4AoLJNcVkPjCy8F1RE9KWNVMG8i2G3 =2gev -----END PGP SIGNATURE-----

1 0

Problem
by Stefan Kühn 13 Jun '09

13 Jun '09

Hello all, I need help. I have a perl programme (for Check Wikipedia). This can scan a dump of a language very fast. 2200 pages per minute is no problem. I will daily scan with the same script the page text of the live Wikipedia. Not all pages, but maybe 20000 per day per language. Normally this need only 10 minutes with the dump, but with the live Wikipedia this need many time. I use the Wikipedia-API to get the text of an article and so my script can only scan 120 pages per minute. So this scan need at the moment in enwiki 300 minutes or in dewiki 134 minutes. The most time my script is waiting. This is a problem because there is a high CPU usage. I need a faster way to get the text from the live Wikipedia. So I can reduce the CPU usage. Maybe someone know a faster way! Or have an other idea. Thanks for every help, Stefan (sk) More Info: http://de.wikipedia.org/wiki/Benutzer:Stefan_Kühn/Check_Wikipedia http://en.wikipedia.org/wiki/Wikipedia:WikiProject_Check_Wikipedia http://toolserver.org/~sk/checkwiki/checkwiki.pl

10 12

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

Toolserver-l June 2009