Wikitech-l January 2004

wikitech-l@lists.wikimedia.org

92 participants
148 discussions

Takedown notice for Easter Bradford page
by Poor, Edmund W 16 Jan '04

16 Jan '04

Jimbo forwarded to Wikien-l what looks like an "official" takedown notice for the article about Easter Bradford -- a man who has been, from time to time, a contributor to Wikipedia. See: http://mail.wikipedia.org/pipermail/wikien-l/2004-January/009529.html I'm not sure what we should do. If I still had ssh software on my PC, I would probably delete the pages from the database (just to be safe). On the other hand, it might be a good idea to back them up them somewhere, accessible only to the head developer, in case the notice is either (a) faked; or (b) not something we have to comply with, even if it's real. Cautiously, Ed Poor

6 5

[phil@ipal.net: Re: Who do I contact when webmaster@wikipedia.org does not work?]
by Jimmy Wales 15 Jan '04

15 Jan '04

This is something we should discuss. Right now, by refusing access to people who don't send a user agent, we may be messing some people up who are coming from behind a proxy? That's what I understand him to be saying. Ideally, when we get our capacity up, we'll let the robots run free. ----- Forwarded message from Phil Howard <phil(a)ipal.net> ----- From: Phil Howard <phil(a)ipal.net> Date: Wed, 14 Jan 2004 12:41:16 -0600 To: Jimmy Wales <jwales(a)bomis.com> Subject: Re: Who do I contact when webmaster(a)wikipedia.org does not work? On Wed, Jan 14, 2004 at 06:56:02AM -0800, Jimmy Wales wrote: | I am told that this is because you don't send a user-agent string. | We block those requests because they are usually from badly-written | spiders that clobber our site. Ah. That would be the case. RFCs require that if a web cacher sends a user-agent string, each received document must be cached separately on the basis of that entire string so that a new request for the very same document from a different browser does not get the one previously requested. Thus if 20 different users go to a given web site from 20 different browsers, the cache has to access the origin server all 20 times, and no cache benefit is obtained, even for the image files. Given that browsers not only put major version numbers, but also minor version numbers, and in many cases even build numbers/dates, in the user agent string, that makes it appear to be a lot of different browsers for the purpose of caching. Here's what my broswer does send: User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.5a) Gecko/20030728 Mozilla Firebird/0.6.1 I think that's excessive, but that is what is going on. I think you need to reconsider the issue with user-agent. There are a few commercial web caches that do this to enhance performance for businesses that are on slower network links. And I don't see any way to make it make an exception for certain sites. Maybe I can hack the code in Squid (which is what we use here) to make it add a generic user-agent string in place of the deleted one. If you decide you cannot change your action for lack of user-agent, then clearly the step to be taken is to fake one that can't be interpreted as any specific browser. User-Agent: generic cache | But I notice that the screenshot is of mozilla firebird. May I | share that link with the developers? Sure. If by developers you mean the mozilla developers, be sure to tell them it's not a mozilla issue; it's a caching issue. BTW, I had heard that some spiders specifically omitted user-agent so that they could be sure to cache something generic for every browser. Maybe they will be going to the fake user-agent string, too. So why not at least put up a page that says why the site cannot be accessed? Just don't put any links in that page for any spiders to continue following so they cache/index only that one, and leave. Surely they are not accessing the same page over and over and over because it didn't have an error status. | | | Jimmy Wales wrote: | | > wow that must have been a very temporary thing, let me know if you see | > it again. | > | > Phil Howard wrote: | > | > > On Tue, Jan 13, 2004 at 04:26:55AM -0800, Jimmy Wales wrote: | > > | > > | Phil Howard wrote: | > > | > Who do I contact when webmaster(a)wikipedia.org does not work? | > > | | > > | Me. Hi, what's up? | > > | > > webmaster(a)wikipedia.org did not work, which is what I first tried when I | > > could not access the web site. | > > | > > I get a 403 permission denied. Here is a screenshot: | > > http://phil.ipal.org/wikipedia.png | > > | > > Here's the bounce from the original message I sent: | > > | > > ============================================================================= | > > [-- Attachment #1: Notification --] | > > [-- Type: text/plain, Encoding: 7bit, Size: 0.5K --] | > > | > > This is the Postfix program at host vega.ipal.net. | > > | > > I'm sorry to have to inform you that the message returned | > > below could not be delivered to one or more destinations. | > > | > > For further assistance, please send mail to <postmaster> | > > | > > If you do so, please include this problem report. You can | > > delete your own text from the message returned below. | > > | > > The Postfix program | > > | > > <webmaster(a)wikipedia.org>: host mail.wikipedia.org[130.94.122.197] said: 550 | > > <webmaster(a)wikipedia.org>: User unknown in local recipient table (in reply | > > to RCPT TO command) | > > | > > [-- Attachment #2: Delivery error report --] | > > [-- Type: message/delivery-status, Encoding: 7bit, Size: 0.3K --] | > > | > > Reporting-MTA: dns; vega.ipal.net | > > Arrival-Date: Mon, 12 Jan 2004 23:31:44 -0600 (CST) | > > | > > Final-Recipient: rfc822; webmaster(a)wikipedia.org | > > Action: failed | > > Status: 5.0.0 | > > Diagnostic-Code: X-Postfix; host mail.wikipedia.org[130.94.122.197] said: 550 | > > <webmaster(a)wikipedia.org>: User unknown in local recipient table (in reply | > > to RCPT TO command) | > > | > > [-- Attachment #3: Undelivered Message --] | > > [-- Type: message/rfc822, Encoding: 7bit, Size: 0.8K --] | > > | > > Date: Mon, 12 Jan 2004 23:31:44 -0600 | > > From: Phil Howard <phil(a)ipal.net> | > > To: webmaster(a)wikipedia.org | > > Subject: still not working | > > User-Agent: Mutt/1.4.1i | > > | > > Here's the screenshot: http://phil.ipal.org/wikipedia.png | > > | > > -- | > > ----------------------------------------------------------------------------- | > > | Phil Howard KA9WGN | http://linuxhomepage.com/ http://ham.org/ | | > > | (first name) at ipal.net | http://phil.ipal.org/ http://ka9wgn.ham.org/ | | > > ----------------------------------------------------------------------------- | > > ============================================================================= | > > | > > -- | > > ----------------------------------------------------------------------------- | > > | Phil Howard KA9WGN | http://linuxhomepage.com/ http://ham.org/ | | > > | (first name) at ipal.net | http://phil.ipal.org/ http://ka9wgn.ham.org/ | | > > ----------------------------------------------------------------------------- | > > -- ----------------------------------------------------------------------------- | Phil Howard KA9WGN | http://linuxhomepage.com/ http://ham.org/ | | (first name) at ipal.net | http://phil.ipal.org/ http://ka9wgn.ham.org/ | ----------------------------------------------------------------------------- ----- End forwarded message -----

3 2

re: new system ideas
by Graham Cox 15 Jan '04

15 Jan '04

> > Dual 64 bit Opteron mainboards are available with 8 DIMM sockets > capable > > of taing 16Gb. eg: > > http://www.tyan.com/products/html/thunderk8w_spec.html > > Right, we have one of those (though broken at the moment). Right now, > our entire db should fit comfortably in 4 gig of Ram, which we have. > > But I totally agree with you about the importance of avoiding the > hard drive as much as possible. > > --Jimbo Just to throw my 2 cents-worth into the debate - any particular reason for running on Intel? An Xserve dual G5 gives better price/performance and is (possibly) much easier to administer. It's probably also likely to be more reliable. I'm not an Apple apologist, but I think these machines should be at least given fair consideration. If not, why not? --Graham

3 3

Problem with en?
by Jimmy Wales 14 Jan '04

14 Jan '04

This guy is having trouble connecting. Although I can use the website fine with a number of browsers, I am able to replicate this exact sequence using telnet. What's the issue here? This fellow apparently can't see the website. ============================================================================= phil@altair:/home/phil 96> telnet en.wikipedia.org 80 Trying 130.94.122.199... Connected to en.wikipedia.org. Escape character is '^]'. GET / HTTP/1.1 Host: en.wikipedia.org HTTP/1.1 403 Forbidden Date: Tue, 13 Jan 2004 21:03:19 GMT Server: Apache/1.3.28 (Unix) mod_throttle/3.1.2 PHP/4.3.2 Connection: close Transfer-Encoding: chunked Content-Type: text/html; charset=iso-8859-1 10f <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <HTML><HEAD> <TITLE>403 Forbidden</TITLE> </HEAD><BODY> <H1>Forbidden</H1> You don't have permission to access / on this server.<P> <HR> <ADDRESS>Apache/1.3.28 Server at en.wikipedia.org Port 80</ADDRESS> </BODY></HTML> 0 Connection closed by foreign host. phil@altair:/home/phil 97> ============================================================================= On Tue, Jan 13, 2004 at 09:13:50AM -0800, Jimmy Wales wrote: | wow that must have been a very temporary thing, let me know if you see | it again. | | Phil Howard wrote: | | > On Tue, Jan 13, 2004 at 04:26:55AM -0800, Jimmy Wales wrote: | > | > | Phil Howard wrote: | > | > Who do I contact when webmaster(a)wikipedia.org does not work? | > | | > | Me. Hi, what's up? | > | > webmaster(a)wikipedia.org did not work, which is what I first tried when I | > could not access the web site. | > | > I get a 403 permission denied. Here is a screenshot: | > http://phil.ipal.org/wikipedia.png | > | > Here's the bounce from the original message I sent: | > | > ============================================================================= | > [-- Attachment #1: Notification --] | > [-- Type: text/plain, Encoding: 7bit, Size: 0.5K --] | > | > This is the Postfix program at host vega.ipal.net. | > | > I'm sorry to have to inform you that the message returned | > below could not be delivered to one or more destinations. | > | > For further assistance, please send mail to <postmaster> | > | > If you do so, please include this problem report. You can | > delete your own text from the message returned below. | > | > The Postfix program | > | > <webmaster(a)wikipedia.org>: host mail.wikipedia.org[130.94.122.197] said: 550 | > <webmaster(a)wikipedia.org>: User unknown in local recipient table (in reply | > to RCPT TO command) | > | > [-- Attachment #2: Delivery error report --] | > [-- Type: message/delivery-status, Encoding: 7bit, Size: 0.3K --] | > | > Reporting-MTA: dns; vega.ipal.net | > Arrival-Date: Mon, 12 Jan 2004 23:31:44 -0600 (CST) | > | > Final-Recipient: rfc822; webmaster(a)wikipedia.org | > Action: failed | > Status: 5.0.0 | > Diagnostic-Code: X-Postfix; host mail.wikipedia.org[130.94.122.197] said: 550 | > <webmaster(a)wikipedia.org>: User unknown in local recipient table (in reply | > to RCPT TO command) | > | > [-- Attachment #3: Undelivered Message --] | > [-- Type: message/rfc822, Encoding: 7bit, Size: 0.8K --] | > | > Date: Mon, 12 Jan 2004 23:31:44 -0600 | > From: Phil Howard <phil(a)ipal.net> | > To: webmaster(a)wikipedia.org | > Subject: still not working | > User-Agent: Mutt/1.4.1i | > | > Here's the screenshot: http://phil.ipal.org/wikipedia.png | > | > -- | > ----------------------------------------------------------------------------- | > | Phil Howard KA9WGN | http://linuxhomepage.com/ http://ham.org/ | | > | (first name) at ipal.net | http://phil.ipal.org/ http://ka9wgn.ham.org/ | | > ----------------------------------------------------------------------------- | > ============================================================================= | > | > -- | > ----------------------------------------------------------------------------- | > | Phil Howard KA9WGN | http://linuxhomepage.com/ http://ham.org/ | | > | (first name) at ipal.net | http://phil.ipal.org/ http://ka9wgn.ham.org/ | | > ----------------------------------------------------------------------------- | > -- ----------------------------------------------------------------------------- | Phil Howard KA9WGN | http://linuxhomepage.com/ http://ham.org/ | | (first name) at ipal.net | http://phil.ipal.org/ http://ka9wgn.ham.org/ | ----------------------------------------------------------------------------- ----- End forwarded message -----

3 4

Secondary servers
by Brion Vibber 14 Jan '04

14 Jan '04

We really, really need to move the database back to a machine with a decent hard drive. The wikis are very sluggish, and a fair chunk of it's from waiting on the database. Ursula's sitting around with a 90% idle CPU, but everything's blocked on disk I/O to the point it's got a load average of about 16. At any given time from 8-20 processes are blocked and waiting. Operations that hit a lot of rows like history and watchlist are particularly badly hit since they don't play as well with caching. If Geoffrin's not going to be up soon, and Pliny's still emitting spurious disk errors, what are our options out of the available machines? -- brion vibber (brion @ pobox.com)

15 52

RE: [Wikitech-l] The order
by Poor, Edmund W 14 Jan '04

14 Jan '04

At that price, they should send a technician to install it, test it, and hang around for a few days to make sure it "burns in" all right. Ed Poor

2 1

The order
by Jimmy Wales 14 Jan '04

14 Jan '04

8 SM-1151SATA, P4 2.6, 2x1GB RAM, 80GB SATA RD, Redhat 9 preload, 1.44Mb floppy, rail kits. The 2x1Gb was more expensive than 4x512Mb, but gives us more flexibility for growth and also for shuffling the memory around to specialize the machines. For machines that aren't doing anything too intensive (mail server), we can buy cheap 2x256=512Mb and donate the extra 2gb to the squid(s), if that seems helpful. And there will be slots available to just buy more memory outright if that seems like a good idea. 8 machines gives us some oomph, no matter what. 1 SM-2280S, Dual AMD Opteron 246, 4GB memory, SCSI RAID 5, 3x36, logical capacity 72GB, redundant power supply, RH9 preload, 1.44mB floppy,, rail kits. This machine is very similar to our big dog geoffrin. The total cost is $19,308, plus tax and shipping. --Jimbo

2 1

ETA on the order
by Jimmy Wales 14 Jan '04

14 Jan '04

Build in approximately 5 business days, and then shipping will take approximately 5 more business days. So that's two weeks in dog years, give or take. Let's say January 27th.

1 0

Wikipedia for the Zaurus
by Markus Baumeister 13 Jan '04

13 Jan '04

Hello, I have modified Alfio Puglisi's wiki2static script and added a few of my own to generate a static compressed version of wikipedia which should be viewable on any UNIX-PDA featureing a webserver and a sh-compatible shell. Compression works quite nicely, I currently have the German and the English content (including media and TeX images) taking up up ca. 600MB on a microdrive. Since TomeRaider is not available for Linux-based PDAs this should fill the last non-Wikipedia hole. See http://www.retsiemuab.de/wiki2zaurus . Two questions: 1) Can some people (preferably with UNIX PDAs and desktops) check if the descriptions on the above site are understandable and work for them also? 2) Should I put a link on a wikipedia page and if yes on which one? "Wikipedia:Database download" has the links to the wiki2static script but I would rather not be listed on a page about how to get database dumps. Without necessary bandwidth for offering dumps I'd prefer to distribute only scripts. Markus

2 1

RE: [Wikitech-l] Secondary servers
by Poor, Edmund W 13 Jan '04

13 Jan '04

Well, which is cheaper or faster? 40 GB of RAM, or a 40 GB hard drive? Surely we can afford either one. Ed Poor

3 2

← Newer
1
...
7
8
9
10
11
12
13
14
15
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

Wikitech-l January 2004