Wikitech-l April 2008

wikitech-l@lists.wikimedia.org

100 participants
99 discussions

Re: [Wikitech-l] So... status of category intersections?
by Aerik Sylvan 24 Apr '08

24 Apr '08

On Wed, 23 Apr 2008, Robert Stojnic wrote: > > Roan Kattouw wrote: > > >Brion Vibber schreef: > > > >>Should check whether Robert's already hacked some of this stuff into the > >>lucene server or what changes it would require. > >> > >> > >If I understand correctly, Lucene shouldn't really care what it stores, > >as long as it's text and it's searchable. Storing "Living_people > >Articles_needing_cleanup" would work just fine, right? We do need to > >think about case-sensitivity, though. > > > > Let me briefly repeat what I said earlier about my experience with this > category > intersection thingy. Adding categories to lucene index is easy *IF* they > are inside > the article, e.g. try this: > > > http://en.wikipedia.org/w/index.php?title=Special%3ASearch&search=%2Bincate… > > This will give you category intersection of "Living People" and "English > comedy writers" > in fraction of the second. > Hey Robert, That is really cool - but it seems to be doing a text match on the whole article, not just the categories... ? > > What I found that the hard part is keeping the index updated. If we want > a fancy category > intersection system discussed here before we need to have an index that > is frequently updated, > that will be integrated with the job queue, that will understand > templates etc.. > That is always the hurdle with Lucene, right? It doesn't do updates, just delete, re-add, and then optimize (and I'd guess optimizing can get resource-intensive on a big index). > > Lucene is not that good with very frequent updates. The usual setting is > to have an indexer, > make snapshots of the index at regular intervals and then rsync it onto > searchers. The whole > process takes time, although for a category-only index it will probably > be fast. I assume there > would be at least few tens of minutes lag anyhow. Our current lucene > framework could > easily be used for index distribution and such. > > What remains unsolved, however, is keeping the index updated with the > latest changes > on the site. If one changes a template with a category in it, the thing > goes on the job queue. > I assume there would need to be some kind of hook that will either log > the change somewhere > or send data to lucene somehow. This is the part of the backend that > needs thinking and solving. > Well... this isn't a complete plan, just some thoughts (and maybe they're naive, but I'll give it a shot anyway). I'm thinking of a new table that holds the pageid and a text field that holds the category strings, leaving the underscores in place. This gets updated via hook at the same time an edit triggers an update to the categorylinks table (not familiar enough with the code to have that data at hand) - this will handle categories in templates etc (leverage the logic that already deals with this). Okay, so once you build the table, the updates to that table aren't too bad. In core, this would be a MyISAM table with a fulltext index. For larger wikis, this is an innodb table. Then the question (the one I think you're raising) is at what point do we refresh or update the lucene index from that table? I'm not sure of the best answer to that question. Is it feasible to do delete/add every time a category is changed, and then optimize once a day or something? (probably not, eh?) What are we doing for the main search index? rebuild daily or so? In an initial implementation, why not follow the same type of schedule? Alternately, perhaps do an update and optimize once an hour? Guess it depends on how much time/resources it takes to update and optimize the index... But certainly using the same schedule as the main index is a safe and conservative plan? Best Regards, Aerik -- http://www.wikidweb.com - the Wiki Directory of the Web http://tagthis.info - Hosted Tagging for your website!

3 2

Re: [Wikitech-l] raising the upload limit
by Birgitte SB 24 Apr '08

24 Apr '08

Also archive.org will not work with the ProofeadPage extention. Birgitte SB > Message: 10 > Date: Thu, 24 Apr 2008 11:17:04 +0200 > From: Husky <huskyr(a)gmail.com> > Subject: Re: [Wikitech-l] raising the upload limit > To: "Wikimedia developers" > <wikitech-l(a)lists.wikimedia.org> > Message-ID: > <85f46e6f0804240217of23f254y470b023a79bd208f(a)mail.gmail.com> > Content-Type: text/plain; charset=UTF-8 > > On Wed, Apr 23, 2008 at 9:41 PM, David Gerard > <dgerard(a)gmail.com> wrote: > > archive.org is better than nothing for now, or do > they have onerous > > requirements? > I haven't looked into archive.org (yet). My feeling was > that > Wikimedia-related files should be hosted on Wikimedia > servers, not on > an external site (even if that site is very > 'friendly' towards our > type of content by allowing CC-licenses to be added to > files). > However, until Tim's proposal with the drop-box is > really built > archive.org is of course a good (temporary) solution. > > -- Hay / Husky > > > > ------------------------------ > > Message: 11 > Date: Thu, 24 Apr 2008 06:14:21 -0500 > From: "Tim Laqua" <t.laqua(a)gmail.com> > Subject: Re: [Wikitech-l] A blank wiki where only the > article appears > To: "'Wikimedia developers'" > <wikitech-l(a)lists.wikimedia.org> > Message-ID: > <48106baa.2de2220a.20cb.426e(a)mx.google.com> > Content-Type: text/plain; charset="us-ascii" > > You could use mod_rewrite to just rewrite every incoming > request to exactly > what you want them to see - completely ignoring any > arguments, query > strings, etc. > > -tl > > > -----Original Message----- > > From: wikitech-l-bounces(a)lists.wikimedia.org > [mailto:wikitech-l- > > bounces(a)lists.wikimedia.org] On Behalf Of Jim Hu > > Sent: Thursday, April 24, 2008 12:45 AM > > To: Wikimedia developers > > Subject: Re: [Wikitech-l] A blank wiki where only the > article appears > > > > It might be easier to page-scrape into an alternate > site using > > action=render. At least if you don't want sneaky > users figuring out > > how to switch skins. Or can one restrict skin use? > > > > Jim > > > > On Apr 21, 2008, at 2:32 PM, Michael Daly wrote: > > > > > Leander Tal wrote: > > >> Hello, i need a blank wiki where only the > text of the article > > >> appears for > > >> the viewer. No header, footer, navigation, > interaction, search, > > >> toolbox, > > >> languages, or categories should appear. Only > the admin, or logged > > >> in persons > > >> should have the standard navigation, and the > ability to edit, and > > >> discuss > > >> the page. How to realize this request? > Thanks, Leander Taler > > > > > > You can do that by creating a skin with the > unwanted items not used. > > > Have the admins and logged-in users select a > conventional skin and > > > have > > > the bland skin used for others. > > > > > > Mike > > > > > > > > > > > > > > > _______________________________________________ > > > Wikitech-l mailing list > > > Wikitech-l(a)lists.wikimedia.org > > > > https://lists.wikimedia.org/mailman/listinfo/wikitech-l > > > > ===================================== > > Jim Hu > > Associate Professor > > Dept. of Biochemistry and Biophysics > > 2128 TAMU > > Texas A&M Univ. > > College Station, TX 77843-2128 > > 979-862-4054 > > > > > > _______________________________________________ > > Wikitech-l mailing list > > Wikitech-l(a)lists.wikimedia.org > > > https://lists.wikimedia.org/mailman/listinfo/wikitech-l > > > > > ------------------------------ > > _______________________________________________ > Wikitech-l mailing list > Wikitech-l(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/wikitech-l > > > End of Wikitech-l Digest, Vol 57, Issue 37 > ****************************************** ____________________________________________________________________________________ Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it now. http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ

1 0

A blank wiki where only the article appears
by Leander Tal 24 Apr '08

24 Apr '08

Hello, i need a blank wiki where only the text of the article appears for the viewer. No header, footer, navigation, interaction, search, toolbox, languages, or categories should appear. Only the admin, or logged in persons should have the standard navigation, and the ability to edit, and discuss the page. How to realize this request? Thanks, Leander Taler

5 5

raising the upload limit
by Graham Pearce 24 Apr '08

24 Apr '08

I'd like the upload limit to be raised to at least 40MB. I'd like to upload some free classical music from Pandora Music that is more than 20MB and cannot be split easily without a loss in quality. For example: http://music.ibiblio.org/pub/multimedia/pandora/vorbis/chamber_orchestra/MA… http://music.ibiblio.org/pub/multimedia/pandora/vorbis/piano/Goldstein/Vari… I am aware of the message in October 2007 to this list: https://lists.wikimedia.org/mailman/htdig/wikitech-l/2007-October/034344.ht… Has much changed since then? Graham

7 10

Re: [Wikitech-l] Wikitech-l Digest, Vol 57, Issue 36
by Samuel Wantman 24 Apr '08

24 Apr '08

> I don't have suggestions > for an interface, but why not just start with a SpecialPage and see what > happens? Once the functionality is there, suggestions for how to > better use > it will come out of the woodwork. Several of us at Wikipedia did some work on an interface for incorporating category intersections a while back. It can be found at shortcut [[WP:CI]] or full url: http://en.wikipedia.org/wiki/Wikipedia:Category_intersection --Samuel Wantman [[en:User:Sam]]

1 0

Add SPF to wikimedia.org
by Platonides 23 Apr '08

23 Apr '08

I'm seeing a number of spam bounces at OTRS, which means that some spammers are using the queue mails as source addresses, at least at the SMTP level. Restricting outgoing servers with SPF would protect servers using SPF from receiving that spam. Which IMHO is much more important than handling the backscatters :)

4 3

userOptions script failing - help!
by Andreas Rindler 22 Apr '08

22 Apr '08

Hi, we are trying to change the default skin for all users using the userOptions script and we get an error. Any idea how to resolve the error or change the skin user option for users with another method? MW version: 1.11 Php: 5.1.6 Here is the command and the error: [openm@machine maintenance]$ php userOptions.php -cli skin --old monobook --new mike2 PHP Warning: array_shift() [<a href='function.array-shift'>function.array-shift</a>]: The argument should be an array in /chroot/home/openmeth/openmethodology.org/html/mike2/w/maintenance/commandLine.inc on line 39 PHP Warning: reset() [<a href='function.reset'>function.reset</a>]: Passed variable is not an array or object in /chroot/home/openmeth/openmethodology.org/html/mike2/w/maintenance/commandLine.inc on line 49 X-Powered-By: PHP/5.1.6 Content-type: text/html Thanks, Andi

2 2

Enabled! HttpOnly cookies and search suggestions
by Brion Vibber 22 Apr '08

22 Apr '08

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Thanks to Werdna, Tim, and Robert for work on getting these things ready. :) Rather than repeat myself, lemme just link to my writeups: Support for HttpOnly cookies should provide an extra security barrier for session-stealing via XSS; the required PHP upgrades probably help too. ;) http://leuksman.com/log/2008/04/21/httponly-cookies/ And I've enabled the AJAX search suggestion drop-down, using the basic prefix matching: http://leuksman.com/log/2008/04/21/suggestion-search-drop-down/ I'm still a little worried that the search hits might cause extra load. So far I don't even see a blip on Ganglia, though. :) If things bog down while I'm away, turn off $wgEnableMWSuggest! - -- brion vibber (brion @ wikimedia.org) -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkgNO0kACgkQwRnhpk1wk44GmQCgozYkDf9gWZlpw0+O2MIOKh5f 0SQAoMXM+appFeeDQ0XPe9sslUphhoVa =l7/2 -----END PGP SIGNATURE-----

1 0

10 weeks, 17 changes since last Interwiki update
by Andrew Cates 21 Apr '08

21 Apr '08

Be nice to get another update of the database please Last one was 10 Feb Thanks BozMo

2 1

Language pack for MediaWiki
by Gerard Meijssen 21 Apr '08

21 Apr '08

Hoi, For the MediaWiki 1.12 stable release, Betawiki has issued a language pack. This language pack includes all the localisations that were added after the release of MediaWiki 1.12. Several languages have been substantially improved in the month and a half since the original release of MediaWiki 1.12. We now have 6.9% more messages ... The language pack is for the moment available here<http://translatewiki.net/mediawiki/>. We are looking how we can support your language better and make the best MediaWiki localisation available in an optimal way. As always, you are welcome to help support your language by localising at Betawiki. Thanks, GerardM

1 0

← Newer
1
2
3
4
5
6
7
8
9
10
Older →

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

Wikitech-l April 2008