Toolserver-l

toolserver-l@lists.wikimedia.org

1612 discussions

[Wikipedia-Tag Dresden] CPU
by Conrad Nutschan 14 Aug '06

14 Aug '06

Heyho, I'm searching for very cpu complex jobs we can solve as a little attraction on Wikipedia-Tag in Dresden on our university cluster. What jobs can be done extern and who is interessted in helping by this project? Systemdata: http://tu-dresden.de/die_tu_dresden/zentrale_einrichtungen/zih/dienste/rech… Maybe somebody can help or have information, greetings from Dresden, [[de:Benutzer:Conny]]... Details WPT: http://de.wikipedia.org/wiki/Wikipedia_Diskussion:Wikipedia-Tag_Dresden

3 4

new root user
by River Tarnell 14 Aug '06

14 Aug '06

Daniel Kinzler (Duesentrieb) now has Unix/MySQL root access. - river.

1 0

Re: [Toolserver-l] [Commons-l] Stats on deletion
by Daniel Kinzler 10 Aug '06

10 Aug '06

(This mail is in response to a thread on Commons-l; I'm cross posting it to the toolserver list, because it seems relevant there. Ultimately, this should probably be discussed with the WMF and its local associates) Please note that publishing "intelligence" obtained from analyzing person-related data may be considered a violation of that person's privacy, even if the analyzed data is publicly available - at least under German law and, afaik, EU guidelines. Data mining can expose things about a person that are not easily found out by looking straight at the raw data - this is often problematic, especially since the results can be quite misleading, as per the nature of the methods used. If this is stupid or not is besides the point. We have been asked explicitly by the German Wikimedia e.V. not do make any analysis of user data available on the toolserver, so we won't (although i find it a bit hard to draw a line). If it would be legal for the US based foundation to do it, is a different question. A different question still is if it would be wise and desirable. To quote Wau Holland and the "hacker ethics" of the Chaos Computer Club: "utilize public data, protect private data". Information wants to be free - but so do people. In the end, the latter are more important. I personally feel that any analysis that exposes information that is not *relevant* to activity on the project should be strictly opt-in. An example would be a breakup of user activity be time of days or day of the week. I'm not sure about things like the number of untagged images a user has uploaded, for example - that does seem relevant to me. If it's legal or wise to expose such an analysis is an open question to me (actually, I'd like to have some input on this, since my tools can give that statistic). On the other hand, I believe that admins should expect to be subject to "public oversight". This is in my opinion an important part of an informal "watch the powerful" mechanism. We already have the ability to see a list of "admin action" an admin has performed. I'm a bit unsure about consolidating that log data into a statistics of deletions per week or whatever - I think we should ask ourselves how useful that would really be. In any case it should be made more obvious to people what "data trails" they leave when working on Wikimedia projects, as a "normal" user and as an admin. General statistics about admin activity - i.e. sum of all admins, not per person, would be quite interesting, though, and unproblematic. Regards, Daniel -- Homepage: http://brightbyte.de

2 6

Tools and privacy
by Jakob Voss 07 Aug '06

07 Aug '06

Hi folks, I just wanted to let you know that we had a meeting of the board of Wikimedia Germany. At least until we have a clear answer from our lawyers we don't want tools that aggregate personal data because of privacy. But opt-in is all-right of course. Maybe you could spread more that there are tools you can opt-in for and that other people cannot get nice stats of what you have done in a Wiki if you don't opt-in. Greetings, Jakob

4 5

jawiki data corruption?
by interiot＠68k.org 28 Jul '06

28 Jul '06

DaB., Do you use the same replication script for enwiki and jawiki? A jawiki user is reporting data corruption issues with a jawiki pageID... > SELECT * FROM jawiki_p.page WHERE page_id=588373; page_namespace=0 page_title=IntelMac http://ja.wikipedia.org/w/query.php?what=revisions&pageids=588373&rvlimit=1… ==> <title>Wikipedia:?????</title> If this is a data corruption issue that was recently fixed, would it be possible to replay jawiki to fix the corrupted data? If so, do you have an estimate when that might be run? -Dave

1 0

Don't spider my tools!
by Daniel Kinzler 27 Jul '06

27 Jul '06

Hi all I just got about 7000 hits for CatScan in the last 60 minutes - originating from zedler itself. This cause replication lag to rise nearly linearly over that time. So, whoever is doing that: STOP IT! Spidering tools is generally a bad idea, and if you have access to the database yourself, it's plain silly. In any case, before abusing the web interface, simply ask me to provide the data in an easy to process manner. Regards, -- Daniel -- Homepage: http://brightbyte.de

2 3

PHP weirdness
by Magnus Manske 22 Jul '06

22 Jul '06

Hi, it seems the current PHP version on the toolserver has a weird bug, according to several online sources. Google for: php "Cannot use string offset as an array" At least one of my scripts is suffering from this. Can someone please upgrade to the latest PHP? I hope that will fix it. In the meantime, I'm trying to code around it, with little success. Magnus

2 1

transaction isolation level set to READ UNCOMMITTED
by Daniel Kinzler 20 Jul '06

20 Jul '06

Hi all Today, we had another replag problem, caused by Aka's user stats script. Replag kept rising even after the script was stopped (not killed). Apparently, the reason lies with the transaction mode used: a long running query may lock (parts of) the tables it uses, denying any updates the replication daemon wants to make. As far as I know, this should not happen if the long running query uses READ UNCOMMITTED (dirty read). Using it should not cause serious problems in our context, save a bit of resources and allow replication to continue while reading from tables. Details on transaction isolation levels can be found here: <http://dev.mysql.com/doc/refman/5.0/en/innodb-transaction-isolation.html> DaB. has set the global default isolation level to READ UNCOMMITTED for now. If you are experiencing problems or need data that is guarantied to be consistent under all circumstances, *and* your query does not take long (no more than, say, a minute or so), you can change the isolation level like this: SET SESSION TRANSACTION ISOLATION LEVEL REPEATABLE READ; REPEATABLE READ is MySQL's "normal" default, i.e. what you have been using until now if you did not set the isolation level explicitly. There are other options, look at the documentation link above. I hope this actually does solve the problem, and does not cause you too much trouble. Regards -- Daniel -- Homepage: http://brightbyte.de

10 20

Killing long running queries
by Daniel Kinzler 16 Jul '06

16 Jul '06

Hi all The toolserver is currently bogged down, again, by several massive, long running queries by kmartin and edwardspec (guys, think what you are doing before launching a query - and if you kill one, make sure you *really* kill it). As often, there's no root user online to take care of the issue (rob, please come back...). So, I have been thinking: How about giving all users (or at least a group of "senior" users) a way to kill long running queries (if not whitelisted)? It has been policy for some time now that anything running more than an hour unannounced can be killed - so why not allow more people to do it? I think this could be implemented as a stored procedure - or two, actually. One for listing queries that are eligible for killing (long running, not whitelisted), and one for actually killing it per ID. Would that be possible permission-wise? I.e. elevate permissions in a stored procedure? Alternatively, this could be invoked from the command line using a small executable with setuid flag. Any such action should be logged, of course, and ideally an email should be sent to the affected user automatically. So, what do you think? Alternatively, we would have to make sure to have a root around 24/7 - or perhaps install an automated email alert if replag rises above an hour or so? Can you think of a smart way to resolve the issue? -- Daniel PS: I'm aware that long running queries are not the only thing that may bog down zedler. But it's something that happens frequently, and is quite easy to detect. So I propose to start there. The next step would be memory hogs, i guess. -- Homepage: http://brightbyte.de

3 4

Apache logs
by River Tarnell 16 Jul '06

16 Jul '06

apache logfiles (/var/log/apache/access) have now been modified to not include the remote client's IP (it will show 127.0.0.1 instead). real logs are available in /var/log/apache/private/access for people in the 'privlogs' group. - river.

3 2

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

Toolserver-l