Hi Steve, Because we need to identify malicious behavior. I don't know. Probably. We always told people to specify User-Agent, just the check was broken. No, you clearly miss very simple idea that with such user-agent you clearly identify yourself as malicious, whereas when you don't specify, you're either malicious or ignorant. Do note, we're good at detecting spoofed user-agents too, so if your bots disguise as MSIE or Firefox or any other regular browser, your behavior is seen as malicious. We do not like malicious behavior. Domas

Hi! Thanks! I'm relatively new to this all operations game, so I'm obsessed about graphs and whack-a-mole :) Cheers, Domas

William, Thanks for insights. But no. We don't use UA as first step of analysis, it was helpful tertiary tool, that put these people into "ignorant or malicious" category. If they'd have spoofed their UAs, we'd block the IPs and inform upstreams, as fully malicious behavior. If they had nice UA, we might have attempted to contact them or have isolated their workload until the issue is fixed ;-) Domas

William, I can't say, that we entirely eliminated it - we transform it a bit, I guess. Oh, I'm observing all the input. The decision made wasn't entirely "oh we must do it", and of course, there could be other courses of action taken, like cherry-picking IPs to ban, or combine subnet-wide bans with URL-based restrictions. All of that needs work, and if WMF is willing to spend resources on implementing such restrictions, it can sure work on it - none of my choices are binding, all I do is usually to keep the site up in good shape, without wasting too much money ;-) Domas

On 02/16/2010 06:57 PM, Steve Summit wrote: If you assume every problem is caused by actively malicious intelligent agents, then there are no complete solutions (or very, very few). Luckily the combination of actively malicious and intelligent isn't so common, so you can get away with making the problem just slightly harder than people are willing to deal with to get whatever kicks they get. Given the lack of of any evidence, I assert that most of the percentage of people who a) notice a problem, b) care, c) know how to fix it; probably deserve to be using the resources anyway. Besides anyone who doesn't deserve but still fixes the problem will likely be able to, and want to, circumvent other measures. Conrad

Hello, Check the CPU drop in Monday: http://ganglia.wikimedia.org/pmtpa/graph.php?g=cpu_report&z=medium&… Network drop on API: http://ganglia.wikimedia.org/pmtpa/graph.php?g=network_report&z=medium&… etc. You can sure assume, that we need to come up with something to "defend a new policy". Yes, we will ban all IPs participating in this. Random strings are easy to identify, fixed strings are easy to verify. Going where it always goes, proper operations of the website. Been there, done that. Domas

Anthony, Your insight is entirely bogus here. It isn't useless. It clearly shows that the user is acting malicious by having automated software that disguises under common user agent. I don't know about UA policies but... Various websites have various techniques to deal with such problems. On the other hand, no other major website has such scarcity of hardware and/or human resources as Wikipedia, at such exposure and API complexity provided. BR, Domas

Hi! You probably don't get scale of wikipedia or scale of the behavior we had to deal with, if you think that it isn't possible to notice behavior patterns of it :-) Acts like a duck, quacks like a duck :) Perhaps. Depends. Probably indefinitely. Dunno, probably not. Would avoid it. Anyway, you probably are missing one important point. We're trying to make Wikipedia's service better. It doesn't always have definite answers, and one has to balance and search for decent solutions. Probably everything looks easier from your armchair. I'd love to have that view! :) Domas

Anthony I'm only going to say this once, either shut the fuck up or put your money where your mouth is and develop and propose a valid replacement, and stop wining when others take actions that are deemed necessary for the betterment of wikimedia related projects. UserAgents are not a big deal, Domas is doing a good job at addressing problems, when you have a massive drain on the servers for no apparent reason and no method to even attempt to identify the source (except IPs) and a majority of these are probably wannabe live mirrors full of spam, or broken automated algorithms some action is better than nothing especially because most of those idiots wont know how to fix their broken programs. John On Tue, Feb 16, 2010 at 9:00 PM, Anthony <wikimail(a)inbox.org> wrote:

On Tue, Feb 16, 2010 at 9:47 PM, John Vandenberg <jayvdb(a)gmail.com> wrote: I don't think so. I believe his point was to complain about the position that he is in. My response was that he was in that position by choice, and that if he'd love to be in my position, it's a really easy for thing for him to accomplish. The graphs provided in this thread clearly show that the solution had It showed that there was quite a bit of bathwater thrown out. And at least one very large baby (Google translation), which was temporarily resurrected. We still don't know how many other, smaller, babies were thrown out, and likely never will. In any case, I don't see how your comment follows from mine.

On Wed, Feb 17, 2010 at 6:54 AM, Domas Mituzas <midom.lists(a)gmail.com>wrote;wrote: I know you are. That's why I started out by trying to figure out who your boss is.

Hi, On Tue, Feb 16, 2010 at 8:31 PM, Domas Mituzas <midom.lists(a)gmail.com> wrote: Yeah, ban no/broken-UA clients for these things that do cause CPU load, but leave article reading unharmed. Normal readers with Privoxy or other privacy filters (you know, people DO still use them, even if their percentage is small!) can at least READ, then. Good luck fighting a dynamic bot herder (though I do ask me, with the spam blacklist and the captchas for URLs, what the hell can a botnet master achieve by hitting Wikipedia?!). The point is, what should bot writers do: 1) no UA at all, that's the typical newbie mistake who just supplies GET /w/index.php?action=edit, which works with his localhost wiki and every other wegs. 2) default UA of the programming language (PHPs thingy, cURL, Python, some bots may even use wget and bash scripting, it's not THAT difficult to write a Wikibot in bashscript!) 3) own UA (stuff like "HDBot v1.1 (http://xyz.tld)"quot;, which I couldn't use some longer time ago) 4) spoof a browser UA (bad, as the site cant differ between bot and browser) To avoid the ban, only 3 and 4 are possible, as the default UAs are blocked for most cases. But as 3 not really works, or at least is hard to troubleshoot, it leaves only 4, which you do not want. Please write some doc that answers this once and for all. Marco PS: Oh, and please, please make the 403 msg something that people can figure out what's wrong, it takes AGES if you are a newbie to scripting. -- VMSoft GbR Nabburger Str. 15 81737 München Geschäftsführer: Marco Schuster, Volker Hemmert http://vmsoft-gbr.de

Steve, RFC2616: 14.43 User-Agent The User-Agent request-header field contains information about the user agent originating the request. This is for statistical purposes, the tracing of protocol violations, and automated recognition of user agents for the sake of tailoring responses to avoid particular user agent limitations. User agents SHOULD include this field with requests. The field can contain multiple product tokens (section 3.8) and comments identifying the agent and any subproducts which form a significant part of the user agent. By convention, the product tokens are listed in order of their significance for identifying the application. User-Agent = "User-Agent" ":" 1*( product | comment ) Example: User-Agent: CERN-LineMode/2.15 libwww/2.17b3 RFC2119: 3. SHOULD This word, or the adjective "RECOMMENDED", mean that there may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course. I guess you just found one more implication to carefully weight before not specifying U-A. Domas

Steve, someone is sending us this User-Agent, is that you?:)) User-Agent: Mozilla 5.0 (Compatible; with Safari, with Opera, Chrome, Netscape, MSIE etc. You get the idea. It's compatible with everything!) Let me tell you a story. Once upon a time, there was a browser named SeaMonkey. It was of noble inheritance, as it was a direct descendant of the famous Netscape of old. Oh, it could have been so proud, this browser, it could have stood up, radiant and tall and strong. But no, that was not to be. Many websites closed their doors for this browser, saying, We don't know who you are, go away! And poor SeaMonkey would have no other choice than to go in disguise, to make the websites believe it was some other browser. And you who read this, you are one of those websites that are putting SeaMonkey to shame. Listen. All browsers support HTML. If you just send the same HTML, all browsers will accept it. Only in some borderline cases will the page you send to the client need to be tailored to a specific client. In the majority of cases, the standard HTML you send to, say, Opera, can be sent to every other browser as well; K-Meleon, Galeon, etc. There is no need to scan the user agent string for keywords like Firefox, Konqueror, Midori or whatever. Just send the standard HTML, OK? But even if you do scan the user agent string, even if you do insist on sending different stuff to different browsers, you should look for distinguishing signs of the rendering engines: Gecko, Webkit, KHTML, Trident, Presto, and so on, not for different browser names. SeaMonkey works the same way as Firefox, Netscape and Flock; they all have Gecko/yyyymmdd in the user agent string. Similarly, Google Chrome works the same way as Safari, Midori and SRWare Iron; they can be identified by the word AppleWebKit in the string. And so on. Distinguishing browsers by name is not only overkill, but it even can backfire in cases such as this, when a misrepresentation is made. Anyway, even more important than that, above all, what you never ever should to is send fatal errors! The real error, incidentally, is not in the content part of the page you send, but in the header: putting info in the header that identifies the page as XHTML, while sending the content of the page as HTML, which will trigger the fatal error message. Errors like that will make you look silly, or worse, they make you look like you're doing it on purpose; sending bad stuff to some browsers, while sending perfectly OK looking pages to others. You're not REALLY doing it on purpose, are you? Trying to make people think that their browsers aren't good enough, that for instance MS Internet Explorer is a better browser than the ones they're using now, because MSIE can display the site while their own browsers can't? No, let's just give you the benefit of the doubt; you're not doing it on purpose. Blame your content management system if you want. Still, it's nothing I can help from here; you will have to make the change to your site to make sure to not saddle some browsers like SeaMonkey or Kazehakaze with your errors. So please, would you consider having a look? Thanks in advance! Cheers, Domas

On 02/15/2010 07:25 PM, Domas Mituzas wrote: Ok. I'm going to take that as "no". In the future, I think it would be better to let people know in advance about non-urgent changes that may break things for them. Would you care to share the results with us? In the future, I'd suggest giving basic info like that as part of an announcement. That's another good thing to share as part of a change announcement: motivation for the change. Last I looked, there were a lot of poorly maintained proxies out there, some of which mangle headers. It seemed reasonable to me that some of those are on low-rent ISPs in poor countries. If you have already done the work to prove that no legitimate users anywhere in the world are impacted by this change, then perhaps you could save us further discussion and just explain that? Thanks, William

50% of load, which at that time was using ~20% of search server CPU load. It also cut our API node traffic into half (some CPU too), and got our average response times for API way nicer: http://www.nedworks.org/~mark/reqstats/svctimestats-daily.png ;-) Also it removed some pressure on API squids, which were misbehaving yesterday, and caused API outage. Also, currently 20% of cluster CPU is being spent on generating atom feeds for people who never really subscribed to them. We don't know why, yet, though :) Domas

On Tue, Feb 16, 2010 at 10:31 AM, Domas Mituzas <midom.lists(a)gmail.com>wrote;wrote: Cool. Who's your boss, and who's your boss's boss? Sorry, I couldn't find you in the org chart or I'd just have looked that up myself.

On Tue, Feb 16, 2010 at 10:39 AM, Domas Mituzas <midom.lists(a)gmail.com>wrote;wrote: Really? Were you doing this work as a contractor, or as a volunteer? Someone's gotta be in charge of the contractors and/or the volunteers, no? No idea. For ages you've been able to just go onto the Wikimedia servers and change whatever you feel like, and answer to nobody? You must be misunderstanding my question or something.

Hi! Volunteer. Dunno, Cary maybe? :) On the other hand, even if they are in charge, it doesn't mean that they are my bosses :-) Kind of. Isn't that a good enough motivation? :-) Though of course, I tend to consult with tech team members, and they're free to overturn anything I change, especially if they come up with better solutions (and they usually do!). And indeed, I guess WMF owns the ultimate power of terminating my access :) Cheers, Domas

On 16 February 2010 16:13, Anthony <wikimail(a)inbox.org> wrote: That should be the next book about Wiki[mp]edia. I had *never* thought that volunteer sysadmins would be feasible until I saw it happening here. Wikimedia: Things Are Different Here(tm). - d.

Ariel Glenn wrote: Well, sorry, no, it's not quite like that. A few of us -- though I fear an inconsequential minority -- are concerned that this is a destabilizing change, being made in a hurry, by a top-10 website, with consequences that aren't easy to predict and (apparently) haven't even been thought about. The more of us who "just" go along with it, the less the consequences will be thought and talked about, and the more they'll be further hidden, and made inevitable.

Robert Rohde wrote: When the new code blocks requests with missing User Agent strings (which is, oddly, not all of the time), it is with a 403 Forbidden response and the very simple message Please provide a User-Agent header (No <html> tags, no nothing.)

Robert, our error message system became overkill, with all those nice designs and multiple languages. in certain cases serving that message too certain requests caused gigabits of bandwidth. it is also not practical to update it with policies, because, um, it has all those nice designs and multiple languages. we may actually end up having better error messages at some point in the future. I agree. We're not sending this response to missing UAs, as this response is being sent by Squid ACLs, and the UA check is done at the MW side. Domas

On Tue, Feb 16, 2010 at 11:32 AM, Ariel T. Glenn &lt;ariel(a)wikimedia.org&gt;wrote;wrote: It's not a big deal for anyone aware of the problem. Adding "User-Agent: Janna" to my scripts is no big deal, nor is adding a randomized UA from a list of common UAs in privoxy (I'm pretty sure there's a plugin for that). I do wonder how many people are going to wind up getting strange errors that they don't know how to fix due to this, though. Is it at all feasible to throttle such traffic rather than blocking it completely?

daniel wrote: Thanks, but FWIW, the very first sentence: Wikimedia sites require a HTTP User-Agent header for all requests. is false. (As near as I can tell, the header is required only for those requests that include an "action=" modifier.)