On 06.04.2011 09:15, Alex Brollo wrote: wow, really? that seems a lot... there are two ways: a) the old style "render" action, like this: <http://en.wikipedia.org/wiki/Foo?action=render> b) the api "parse" action, like this: <http://en.wikipedia.org/w/api.php?action=parse&page=Foo&redirects=1&format=xml> To learn more about the web API, have a look at <http://www.mediawiki.org/wiki/API> No, quite to the contrary. The full page HTML is heavily cached. If you pull the full page (without being logged in), it's quite likely that the page will be served from a front tier reverse proxy (squid or varnish). API requests and render actions however always go through to the actual Apache servers and cause more load. However, as long as you don't make several requests at once, you are not putting any serious strain on the servers. Wikimedia servers more than a hundret thousand requests per second. One more is not so terrible... It is indeed frequently used for that. cheers, daniel

Hi Alex That's because the render call returns just the HTML, while the API call includes some meta-info in the XML wrapper. No. I'm saying that non-logged-in views of full pages are what causes the least server load. I'm not saying that this is what you should use. For one thing, it wasts bandwidth and causes additional work on your side (tripping the skin cruft). I would recommend to use action=render if you need just the plain old html, or the API if you need a bit more control, e.g. over whether templates are resolved or not, how redirects are handled, etc. If your bot is logged in when fetching the pages would only matter if you requested full page html. Which, as I said, isn't the best option for what you are doing. So, log in or not, it doesn't matter. But do use a distinctive and descriptive User Agent string for your bot, ideally containing some contact info <http://meta.wikimedia.org/wiki/User-Agent_policy>. Note that as soon as the bot does any editing, it really should be logged in, and, depending on the wiki's rules, have a bot flag, or have some specific info on its user page. That'S always a good idea :) -- daniel

On Wed, Apr 6, 2011 at 3:15 AM, Alex Brollo &lt;alex.brollo(a)gmail.com&gt; wrote: It could save a huge amount of bandwidth. This could be a big deal for mobile devices, in particular, but it could also reduce TCP round-trips and make things noticeably snappier for everyone. I recently read about an interesting technique on Steve Souders' blog, which he got by analyzing Google and Bing: http://www.stevesouders.com/blog/2011/03/28/storager-case-study-bing-google/ The gist is that when the page loads, assuming script is enabled, you store static pieces of the page in localStorage (available on the large majority of browsers, including IE8) and set a cookie. Then if the cookie is present on a subsequent request, the server doesn't send the repetitive static parts, and instead sends a <script> that inserts the desired contents synchronously from localStorage. I guess this breaks horribly if you request a page with script enabled and then request another page with script disabled, but otherwise the basic idea seems powerful and reliable, if a bit tricky to get right. Of course, it would at least double the amount of space pages take in Squid cache, so for us it might not be a great idea. Still interesting, and worth keeping in mind.