Has anybody compiled a list of all entries in Wikitionary? I know
there are XML dumps for page titles, e.g.
http://download.wikimedia.org/enwiktionary/20110827/enwiktionary-20110827-a…
But I was thinking on a dictionary entry level, e.g. the word
"snigel" is available on en.wiktionary as a Norwegian Nynorsk
entry, and on fr.wiktionary and pl.wiktionary as a Swedish entry.
This could be expressed as a table with 3 columns:
Entry Site Language
snigel en nn
snigel fr sv
snigel pl sv
An additional table could indicate the date when each Site's
XML dump was used to update this large table of entries.
Is anybody doing this already?
--
Lars Aronsson (lars(a)aronsson.se)
Aronsson Datateknik - http://aronsson.se
While tweaking my MIMEStatBot[1] to handle random Query Killer attacks
better, I got to thinking...
Would it be practical to set up a local index on the toolserver for
(img_major_mime, img_minor_mime, img_media_type) on the image table for
commonswiki (and maybe others too)?
Such an index would hopefully cut the runtime of my weekly queries from
hours to seconds or less. It might also enable other tools to make
better use of the MIME type info stored in the database.
Of course, populating and maintaining such an index would consume some
time and storage space. But I wonder if it might not be worth the
tradeoff. What do the TS admins think?
[1] http://commons.wikimedia.org/wiki/User:MIMEStatBot
--
Ilmari Karonen
Hello all,
today we repaired Fisheye (our viewer for Subversion). If you see that a
repository there is not updating for more then 1h, then please (re-)open a
Bugreport at JIRA.
Sincerly,
DaB.
--
Userpage: [[:w:de:User:DaB.]] — PGP: 2B255885
Hello all,
mysql on z-dat-s4-a (that's the host that serves sql-s4 (aka commons) at the
moment) crashed and does not recover as far as I see. So I switched sql-s4-rr
and sql-s4-user to ther hosts which have also a copy of commons. The problem
is now that the host that now serves sql-s4-user have no user-databases at the
moment. I switched the server read/write so you can create databases there if
you like.
For details see [1].
Sincerly,
DaB.
[1] https://jira.toolserver.org/browse/MNT-1074
--
Userpage: [[:w:de:User:DaB.]] — PGP: 2B255885
> the linked wiki-page was updated with the information about the
> query-killer
> more than 1 year ago.
>
> Sincerly,
> DaB.
>
>
On my view the query killer is indeed a very nice function to ballance
between toolserver's force shared to user tools and its (toolserver's)
stable operation and overall performance. I just thought once it previously
worked in a less restricting way, there was a good reason for higher
restrictions just introduced.
Let's take an example of killer's behaviour, which is partially could be
considered helpfull and partially questionable:
> a MySQL-query of yours was killed because you didn't mark it as SLOW_OK
and it have run for 3612 seconds which was longer than allowed.
I see this as a very helpfull part, which recommends to think of the query
optimization or at least on the magic word.
Next,
> The replication lag at kill-time was 0s.
and the query itself starts with
> DELETE l
> FROM l,
> a2cr
> WHERE l_to=a2cr_to
Each table is local to a user database and used exclusively by a single
(TRANSACTION ISOLATION LEVEL READ UNCOMMITTED) thread. This query should not
cause any other queries wating for its completion, and as far as replag is
zero at the execution time it does not cause any toolserver's resource being
unavailable or limited in requested capacity to other users.
Previously the query killer interrupted queries just in case of replag upper
some thresold, which is a bit less restrictive condition, but seems not yet
the very proper one. It killed all the queries worked for more then allowed
and (looked like) did not undertake any smarter analysis on which queries
are the most influencing or causing the replag increased.
I assume once the query killer started looking for replag more regularely it
could be in a position to mark queries working for long and not causing more
or less stable replag increase as more or less safe and wait longer prior to
kill them.
mashiah
Hi,
Have you try "mysql -h sql-s4-user" to get a mysql shell?
Best regards,
Jan
Von: toolserver-l-bounces(a)lists.wikimedia.org
[mailto:toolserver-l-bounces@lists.wikimedia.org] Im Auftrag von Krinkle
Gesendet: Samstag, 27. August 2011 03:43
An: toolserver-l(a)lists.wikimedia.org
Betreff: Re: [Toolserver-l] S4-user-databases (was: Re: z-dat-s4-a (sql-s4)
crashed)
I do indeed want to restore these as my s4/commons-related tools are
currently error-ing out with "Table 'u_krinkle.foobarcommons' doesn't exist"
etc.
How do I restore them ? "$ sql u_krinkle" brings me to the u_krinkle of
sql.toolserver.org, not of s4-user
--
Krinkle
2011/8/26 DaB. <WP(a)daniel.baur4.info>
Hello all,
it took a while, back I finaly managed to extract your user-databases of the
crashed s4-server.
You can find the dumps at /mnt/user-store/s4-user-dbs.backup/. Please take a
look if you still need the data
and
-if yes: import them into the temporary s4-user-server and delete the
dump-file
afterwarts.
-if no: just delete the dump-file.
The dump-files will vanish in 2 weeks from now.
Sincerly,
DaB.
--
Userpage: [[:w:de:User:DaB.]] - PGP: 2B255885
_______________________________________________
Toolserver-l mailing list (Toolserver-l(a)lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list:
https://wiki.toolserver.org/view/Mailing_list_etiquette
Here's the error:
Your "cron" job on willow
cronsub $HOME/newpagefetch.sh
produced the following output:
qsub: ERROR! argument to -N option must not contain /
Here's the cron entry:
$ crontab -l
0 3 * * * cronsub $HOME/newpagefetch.sh
Here's the shell script:
$ cat newpagefetch.sh
#!/bin/sh
#$ -sl newpagefetch
#$ -m bae
#$ -j y
#$ -o $HOME/newpagefetch.log
java -jar $HOME/NewPageFetcherApplication.jar
What am I doing wrong?
Does anyone know if there's a trick to get /* SLOW_OK */ into a stored procedure? When I create a stored procedure with a /* SLOW_OK */ comment in it, the stored procedure is saved with the comments stripped out.
- Jason
--- On Tue, 8/16/11, toolserver-l-request(a)lists.wikimedia.org <toolserver-l-request(a)lists.wikimedia.org> wrote:
From: toolserver-l-request(a)lists.wikimedia.org <toolserver-l-request(a)lists.wikimedia.org>
Subject: Toolserver-l Digest, Vol 70, Issue 10
To: toolserver-l(a)lists.wikimedia.org
Date: Tuesday, August 16, 2011, 5:00 AM
Send Toolserver-l mailing list submissions to
toolserver-l(a)lists.wikimedia.org
To subscribe or unsubscribe via the World Wide Web, visit
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
or, via email, send a message with subject or body 'help' to
toolserver-l-request(a)lists.wikimedia.org
You can reach the person managing the list at
toolserver-l-owner(a)lists.wikimedia.org
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Toolserver-l digest..."
Today's Topics:
1. Re: [Toolserver-announce] Query-Killer is back in action
(Huib Laurens)
2. Re: Query-Killer is back in action (Maarten Dammers)
3. Re: Query-Killer is back in action (DaB.)
4. Re: Query-Killer is back in action (K. Peachey)
----------------------------------------------------------------------
Message: 1
Date: Mon, 15 Aug 2011 17:43:15 +0200
From: Huib Laurens <sterkebak(a)gmail.com>
Subject: Re: [Toolserver-l] [Toolserver-announce] Query-Killer is back
in action
To: toolserver-l(a)lists.wikimedia.org
Message-ID:
<CAGkfB_qcKj=yfB9hTn+oeW1GTJcOo-_uPznmXX1fTrbDB8iCfQ(a)mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1
Is this script somewhere aivable? I would like to use it also outside
the toolserver.,
2011/8/15, Magnus Manske <magnusmanske(a)googlemail.com>:
> On Mon, Aug 15, 2011 at 1:30 AM, DeltaQuad Wikipedia
> <deltaquadwiki(a)gmail.com> wrote:
>> Quick question, is "Unable to run job: got no response from JSV script
>> "/sge62/default/common/jsv.sh" " The killer? becuase ya, I got a lot of
>> them, came home after about 5 hours to about 8 of them, and my API queries
>> are quick...I don't get (unless it's the server) what i'm doing wrong.
>
> It's not 'the killer', but it might be related, because I got those as
> well, first time ever:
>
> Unable to run job: got no response from JSV script
> "/sge62/default/common/jsv.sh".
>
> Magnus
>
> _______________________________________________
> Toolserver-l mailing list (Toolserver-l(a)lists.wikimedia.org)
> https://lists.wikimedia.org/mailman/listinfo/toolserver-l
> Posting guidelines for this list:
> https://wiki.toolserver.org/view/Mailing_list_etiquette
>
--
Verzonden vanaf mijn mobiele apparaat
Kind regards,
Huib Laurens
WickedWay.nl
Webhosting the wicked way.
------------------------------
Message: 2
Date: Mon, 15 Aug 2011 23:03:52 +0200
From: Maarten Dammers <maarten(a)mdammers.nl>
Subject: Re: [Toolserver-l] Query-Killer is back in action
To: toolserver-l(a)lists.wikimedia.org
Message-ID: <4E4989B8.6080600(a)mdammers.nl>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Op 14-8-2011 18:42, DaB. schreef:
> <knip>
>
> [1]
> https://wiki.toolserver.org/view/Database_access#Slow_queries_and_the_query…
Ah, you are the one who killed the categorization bot. Thanks for
announcing that beforehand (not!). It's very annoying that you suddenly
just decide to deploy a new toy the screws up our tools.
Maarten
------------------------------
Message: 3
Date: Mon, 15 Aug 2011 23:14:56 +0200
From: "DaB." <WP(a)daniel.baur4.info>
Subject: Re: [Toolserver-l] Query-Killer is back in action
To: toolserver-l(a)lists.wikimedia.org
Message-ID: <201108152315.03221.WP(a)daniel.baur4.info>
Content-Type: text/plain; charset="utf-8"
Hello,
At Monday 15 August 2011 23:12:28 DaB. wrote:
> Thanks for announcing that beforehand (not!)
the linked wiki-page was updated with the information about the query-killer
more than 1 year ago.
Sincerly,
DaB.
--
Userpage: [[:w:de:User:DaB.]] ? PGP: 2B255885
Hello all,
z-dat-s7-a also crashed arround morning, but the reason looks different to me.
We have no other host which carries s7 so I can't switch over. I managed to
start s7 again, but I will leave it read-only (and without replication)
because I fear the further writes will crash it totaly.
I'm dumping the data of s7 at the moment so I can re-import it later (I hope
that will fix it). The reimport will cause a downtime of unknown lenght – I
will send a email before I start.
Sincerly,
DaB.
--
Userpage: [[:w:de:User:DaB.]] — PGP: 2B255885
Is this problem with comments outside queries in a SQL file exclusive to /* Comment */ style comments, or are
#
# Comment
#
style comments a problem as well?
Thanks,
Jason
--- On Tue, 8/23/11, toolserver-l-request(a)lists.wikimedia.org <toolserver-l-request(a)lists.wikimedia.org> wrote:
the following message is only interesting for people who uses the MySQL-CLI
(the "mysql"-command on the command-line) or the "sql"-script. It doesn't
matter if you use php, perl, phyton, java or other languages.
The problem is that the mysql-cli strips out comments by default (see
bugreport [1]). That is bad for us, because it strips all SLOW_OK or LIMIT:
too; so if you add a SLOW_OK at the mysql-cli it will stript out and the
query-killer will kill it sooner. To prevent this, I changed the mysql-config
in a way that comments will NOT be stripted yesterday [2] .
In the last hour a user told me, that this has a problematic side-effect: if
you use /* foobar */ in a mysql-file OUTSIDE of a query, the file will fail; the
user was angry because I changed the default without telling.
I still think that the mysql-default (to strip comments) is stupid (because in
my eyes the user doesn't expect that comments he/she inserts will be removed).
So here is my plan: I will revert my changes for now (so the mysql-cli will
strip comments again). And I hereby announce that I'm going to change the
default at wednesday, the 31. August, in a way that comments will NOT the
striped out.
You what that means for you:
-If you don't use comments at the mysql-cli: nothing.
-If you use comments at the mysql-cli: You have to add the following lines to
,your .my.cnf-file (you can find this hidden file in your home ? please notice
the "." at the start):
--snip--
[mysql]
comments
--snap--
-If you need the default of the mysql-cli after the 31. August, you have to
add the following lines to your .my.cnf:
--snip--
[mysql]
skip-comments
--snap--
My revert will not affect the "sql"-script (our recommend way to uise myql on
the command-line): It will still not strip the comments.
Please feel free to discuss this and ask for details on the mailinglist.
Sincerly,
DaB.