Toolserver-l August 2011

toolserver-l@lists.wikimedia.org

31 participants
21 discussions

Wiktionary table of contents
by Lars Aronsson 06 Sep '11

06 Sep '11

Has anybody compiled a list of all entries in Wikitionary? I know there are XML dumps for page titles, e.g. http://download.wikimedia.org/enwiktionary/20110827/enwiktionary-20110827-a… But I was thinking on a dictionary entry level, e.g. the word "snigel" is available on en.wiktionary as a Norwegian Nynorsk entry, and on fr.wiktionary and pl.wiktionary as a Swedish entry. This could be expressed as a table with 3 columns: Entry Site Language snigel en nn snigel fr sv snigel pl sv An additional table could indicate the date when each Site's XML dump was used to update this large table of entries. Is anybody doing this already? -- Lars Aronsson (lars(a)aronsson.se) Aronsson Datateknik - http://aronsson.se

3 2

Index on image table by MIME type?
by Ilmari Karonen 30 Aug '11

30 Aug '11

While tweaking my MIMEStatBot[1] to handle random Query Killer attacks better, I got to thinking... Would it be practical to set up a local index on the toolserver for (img_major_mime, img_minor_mime, img_media_type) on the image table for commonswiki (and maybe others too)? Such an index would hopefully cut the runtime of my weekly queries from hours to seconds or less. It might also enable other tools to make better use of the MIME type info stored in the database. Of course, populating and maintaining such an index would consume some time and storage space. But I wonder if it might not be worth the tradeoff. What do the TS admins think? [1] http://commons.wikimedia.org/wiki/User:MIMEStatBot -- Ilmari Karonen

2 3

Fisheye is working again
by DaB. 30 Aug '11

30 Aug '11

Hello all, today we repaired Fisheye (our viewer for Subversion). If you see that a repository there is not updating for more then 1h, then please (re-)open a Bugreport at JIRA. Sincerly, DaB. -- Userpage: [[:w:de:User:DaB.]] — PGP: 2B255885

2 1

z-dat-s4-a (sql-s4) crashed
by DaB. 28 Aug '11

28 Aug '11

Hello all, mysql on z-dat-s4-a (that's the host that serves sql-s4 (aka commons) at the moment) crashed and does not recover as far as I see. So I switched sql-s4-rr and sql-s4-user to ther hosts which have also a copy of commons. The problem is now that the host that now serves sql-s4-user have no user-databases at the moment. I switched the server read/write so you can create databases there if you like. For details see [1]. Sincerly, DaB. [1] https://jira.toolserver.org/browse/MNT-1074 -- Userpage: [[:w:de:User:DaB.]] — PGP: 2B255885

2 5

Query killer heuristics
by Mashiah Davidson 27 Aug '11

27 Aug '11

> the linked wiki-page was updated with the information about the > query-killer > more than 1 year ago. > > Sincerly, > DaB. > > On my view the query killer is indeed a very nice function to ballance between toolserver's force shared to user tools and its (toolserver's) stable operation and overall performance. I just thought once it previously worked in a less restricting way, there was a good reason for higher restrictions just introduced. Let's take an example of killer's behaviour, which is partially could be considered helpfull and partially questionable: > a MySQL-query of yours was killed because you didn't mark it as SLOW_OK and it have run for 3612 seconds which was longer than allowed. I see this as a very helpfull part, which recommends to think of the query optimization or at least on the magic word. Next, > The replication lag at kill-time was 0s. and the query itself starts with > DELETE l > FROM l, > a2cr > WHERE l_to=a2cr_to Each table is local to a user database and used exclusively by a single (TRANSACTION ISOLATION LEVEL READ UNCOMMITTED) thread. This query should not cause any other queries wating for its completion, and as far as replag is zero at the execution time it does not cause any toolserver's resource being unavailable or limited in requested capacity to other users. Previously the query killer interrupted queries just in case of replag upper some thresold, which is a bit less restrictive condition, but seems not yet the very proper one. It killed all the queries worked for more then allowed and (looked like) did not undertake any smarter analysis on which queries are the most influencing or causing the replag increased. I assume once the query killer started looking for replag more regularely it could be in a position to mark queries working for long and not causing more or less stable replag increase as more or less safe and wait longer prior to kill them. mashiah

1 0

Re: [Toolserver-l] S4-user-databases (was: Re: z-dat-s4-a (sql-s4) crashed)
by Jan Luca 27 Aug '11

27 Aug '11

Hi, Have you try "mysql -h sql-s4-user" to get a mysql shell? Best regards, Jan Von: toolserver-l-bounces(a)lists.wikimedia.org [mailto:toolserver-l-bounces@lists.wikimedia.org] Im Auftrag von Krinkle Gesendet: Samstag, 27. August 2011 03:43 An: toolserver-l(a)lists.wikimedia.org Betreff: Re: [Toolserver-l] S4-user-databases (was: Re: z-dat-s4-a (sql-s4) crashed) I do indeed want to restore these as my s4/commons-related tools are currently error-ing out with "Table 'u_krinkle.foobarcommons' doesn't exist" etc. How do I restore them ? "$ sql u_krinkle" brings me to the u_krinkle of sql.toolserver.org, not of s4-user -- Krinkle 2011/8/26 DaB. <WP(a)daniel.baur4.info> Hello all, it took a while, back I finaly managed to extract your user-databases of the crashed s4-server. You can find the dumps at /mnt/user-store/s4-user-dbs.backup/. Please take a look if you still need the data and -if yes: import them into the temporary s4-user-server and delete the dump-file afterwarts. -if no: just delete the dump-file. The dump-files will vanish in 2 weeks from now. Sincerly, DaB. -- Userpage: [[:w:de:User:DaB.]] - PGP: 2B255885 _______________________________________________ Toolserver-l mailing list (Toolserver-l(a)lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette

1 0

help me with my first cronsub
by Ted Timmons 27 Aug '11

27 Aug '11

Here's the error: Your "cron" job on willow cronsub $HOME/newpagefetch.sh produced the following output: qsub: ERROR! argument to -N option must not contain / Here's the cron entry: $ crontab -l 0 3 * * * cronsub $HOME/newpagefetch.sh Here's the shell script: $ cat newpagefetch.sh #!/bin/sh #$ -sl newpagefetch #$ -m bae #$ -j y #$ -o $HOME/newpagefetch.log java -jar $HOME/NewPageFetcherApplication.jar What am I doing wrong?

2 1

Re: [Toolserver-l] Toolserver-l Digest, Vol 70, Issue 10
by Ja Ga 26 Aug '11

26 Aug '11

Does anyone know if there's a trick to get /* SLOW_OK */ into a stored procedure? When I create a stored procedure with a /* SLOW_OK */ comment in it, the stored procedure is saved with the comments stripped out. - Jason --- On Tue, 8/16/11, toolserver-l-request(a)lists.wikimedia.org <toolserver-l-request(a)lists.wikimedia.org> wrote: From: toolserver-l-request(a)lists.wikimedia.org <toolserver-l-request(a)lists.wikimedia.org> Subject: Toolserver-l Digest, Vol 70, Issue 10 To: toolserver-l(a)lists.wikimedia.org Date: Tuesday, August 16, 2011, 5:00 AM Send Toolserver-l mailing list submissions to toolserver-l(a)lists.wikimedia.org To subscribe or unsubscribe via the World Wide Web, visit https://lists.wikimedia.org/mailman/listinfo/toolserver-l or, via email, send a message with subject or body 'help' to toolserver-l-request(a)lists.wikimedia.org You can reach the person managing the list at toolserver-l-owner(a)lists.wikimedia.org When replying, please edit your Subject line so it is more specific than "Re: Contents of Toolserver-l digest..." Today's Topics: 1. Re: [Toolserver-announce] Query-Killer is back in action (Huib Laurens) 2. Re: Query-Killer is back in action (Maarten Dammers) 3. Re: Query-Killer is back in action (DaB.) 4. Re: Query-Killer is back in action (K. Peachey) ---------------------------------------------------------------------- Message: 1 Date: Mon, 15 Aug 2011 17:43:15 +0200 From: Huib Laurens <sterkebak(a)gmail.com> Subject: Re: [Toolserver-l] [Toolserver-announce] Query-Killer is back in action To: toolserver-l(a)lists.wikimedia.org Message-ID: <CAGkfB_qcKj=yfB9hTn+oeW1GTJcOo-_uPznmXX1fTrbDB8iCfQ(a)mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1 Is this script somewhere aivable? I would like to use it also outside the toolserver., 2011/8/15, Magnus Manske <magnusmanske(a)googlemail.com>: > On Mon, Aug 15, 2011 at 1:30 AM, DeltaQuad Wikipedia > <deltaquadwiki(a)gmail.com> wrote: >> Quick question, is "Unable to run job: got no response from JSV script >> "/sge62/default/common/jsv.sh" " The killer? becuase ya, I got a lot of >> them, came home after about 5 hours to about 8 of them, and my API queries >> are quick...I don't get (unless it's the server) what i'm doing wrong. > > It's not 'the killer', but it might be related, because I got those as > well, first time ever: > > Unable to run job: got no response from JSV script > "/sge62/default/common/jsv.sh". > > Magnus > > _______________________________________________ > Toolserver-l mailing list (Toolserver-l(a)lists.wikimedia.org) > https://lists.wikimedia.org/mailman/listinfo/toolserver-l > Posting guidelines for this list: > https://wiki.toolserver.org/view/Mailing_list_etiquette > -- Verzonden vanaf mijn mobiele apparaat Kind regards, Huib Laurens WickedWay.nl Webhosting the wicked way. ------------------------------ Message: 2 Date: Mon, 15 Aug 2011 23:03:52 +0200 From: Maarten Dammers <maarten(a)mdammers.nl> Subject: Re: [Toolserver-l] Query-Killer is back in action To: toolserver-l(a)lists.wikimedia.org Message-ID: <4E4989B8.6080600(a)mdammers.nl> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Op 14-8-2011 18:42, DaB. schreef: > <knip> > > [1] > https://wiki.toolserver.org/view/Database_access#Slow_queries_and_the_query… Ah, you are the one who killed the categorization bot. Thanks for announcing that beforehand (not!). It's very annoying that you suddenly just decide to deploy a new toy the screws up our tools. Maarten ------------------------------ Message: 3 Date: Mon, 15 Aug 2011 23:14:56 +0200 From: "DaB." <WP(a)daniel.baur4.info> Subject: Re: [Toolserver-l] Query-Killer is back in action To: toolserver-l(a)lists.wikimedia.org Message-ID: <201108152315.03221.WP(a)daniel.baur4.info> Content-Type: text/plain; charset="utf-8" Hello, At Monday 15 August 2011 23:12:28 DaB. wrote: > Thanks for announcing that beforehand (not!) the linked wiki-page was updated with the information about the query-killer more than 1 year ago. Sincerly, DaB. -- Userpage: [[:w:de:User:DaB.]] ? PGP: 2B255885

2 1

z-dat-s7-a (sql-s7) also crashed
by DaB. 25 Aug '11

25 Aug '11

Hello all, z-dat-s7-a also crashed arround morning, but the reason looks different to me. We have no other host which carries s7 so I can't switch over. I managed to start s7 again, but I will leave it read-only (and without replication) because I fear the further writes will crash it totaly. I'm dumping the data of s7 at the moment so I can re-import it later (I hope that will fix it). The reimport will cause a downtime of unknown lenght – I will send a email before I start. Sincerly, DaB. -- Userpage: [[:w:de:User:DaB.]] — PGP: 2B255885

1 2

Re: [Toolserver-l] MySQL-CLI and comments
by Ja Ga 23 Aug '11

23 Aug '11

Is this problem with comments outside queries in a SQL file exclusive to /* Comment */ style comments, or are # # Comment # style comments a problem as well? Thanks, Jason --- On Tue, 8/23/11, toolserver-l-request(a)lists.wikimedia.org <toolserver-l-request(a)lists.wikimedia.org> wrote: the following message is only interesting for people who uses the MySQL-CLI (the "mysql"-command on the command-line) or the "sql"-script. It doesn't matter if you use php, perl, phyton, java or other languages. The problem is that the mysql-cli strips out comments by default (see bugreport [1]). That is bad for us, because it strips all SLOW_OK or LIMIT: too; so if you add a SLOW_OK at the mysql-cli it will stript out and the query-killer will kill it sooner. To prevent this, I changed the mysql-config in a way that comments will NOT be stripted yesterday [2] . In the last hour a user told me, that this has a problematic side-effect: if you use /* foobar */ in a mysql-file OUTSIDE of a query, the file will fail; the user was angry because I changed the default without telling. I still think that the mysql-default (to strip comments) is stupid (because in my eyes the user doesn't expect that comments he/she inserts will be removed). So here is my plan: I will revert my changes for now (so the mysql-cli will strip comments again). And I hereby announce that I'm going to change the default at wednesday, the 31. August, in a way that comments will NOT the striped out. You what that means for you: -If you don't use comments at the mysql-cli: nothing. -If you use comments at the mysql-cli: You have to add the following lines to ,your .my.cnf-file (you can find this hidden file in your home ? please notice the "." at the start): --snip-- [mysql] comments --snap-- -If you need the default of the mysql-cli after the 31. August, you have to add the following lines to your .my.cnf: --snip-- [mysql] skip-comments --snap-- My revert will not affect the "sql"-script (our recommend way to uise myql on the command-line): It will still not strip the comments. Please feel free to discuss this and ask for details on the mailinglist. Sincerly, DaB.

2 1

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

Toolserver-l August 2011