Issue with loading specific page

List overview All Threads
Download

newer

older

HELLO

Re: [Mediawiki-api] Redirects *to*...

Anand Ramanathan

22 Nov 2010 22 Nov '10

10:17 p.m.

Hi, I am not sure this is the right place for this issue, so please redirect me if there is a better forum. I have been loading the wikipedia page for India<http://en.wikipedia.org/wiki/India>programmatically as part of testing my project. It was working fine, until recently. Now, most of the time (but not always), loading it gives me garbled content. Other pages on wikipedia work fine both programmatically and from the browser. This particular page also loads fine from my browser (safari on mac) - but when I load it programmatically (using curl or open-uri from mac-os x), the content is garbled. I am using a user agent, but don't know what changed. I tried closing the bar on top which appeared recently (Jimmy Wales message), but even without that bar, it seems to be garbled. Any suggestions? Thanks Anand

Attachments:

attachment.htm (text/html — 1.0 KB)

Show replies by date

Betacommand

22 Nov 22 Nov

11:26 p.m.

Can you define "garbled" On Mon, Nov 22, 2010 at 5:17 PM, Anand Ramanathan <rcanand(a)gmail.com> wrote:

...

Anand Ramanathan

23 Nov 23 Nov

1:07 a.m.

It is intermittent - sometimes I see english characters, as expected. Sometimes, i see some junk characters - right now getting normal english characters, so cant show a sample of the junk. But this is definitely happening with this page - over the last 3 days off and on... Will send the garbled example if and when it happens again. Thanks Anand On Mon, Nov 22, 2010 at 3:26 PM, Betacommand <Betacommand(a)gmail.com> wrote:

...

Can you define "garbled" On Mon, Nov 22, 2010 at 5:17 PM, Anand Ramanathan <rcanand(a)gmail.com>wrote;wrote:

_______________________________________________ Mediawiki-api mailing list Mediawiki-api(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-api

Andrew Dunbar

7:16 a.m.

On 23 November 2010 03:07, Anand Ramanathan <rcanand(a)gmail.com> wrote:

...

You might be getting gzipped content sometimes. I had this issue with certain Perl libraries. Normally you should get the content unzipped. What programming language/framework etc have you been using? Andrew Dunbar (hippietrail)

...

Thanks Anand On Mon, Nov 22, 2010 at 3:26 PM, Betacommand <Betacommand(a)gmail.com> wrote:

Can you define "garbled" On Mon, Nov 22, 2010 at 5:17 PM, Anand Ramanathan <rcanand(a)gmail.com> wrote:

Hi, I am not sure this is the right place for this issue, so please redirect me if there is a better forum. I have been loading the wikipedia page for India programmatically as part of testing my project. It was working fine, until recently. Now, most of the time (but not always), loading it gives me garbled content. Other pages on wikipedia work fine both programmatically and from the browser. This particular page also loads fine from my browser (safari on mac) - but when I load it programmatically (using curl or open-uri from mac-os x), the content is garbled. I am using a user agent, but don't know what changed. I tried closing the bar on top which appeared recently (Jimmy Wales message), but even without that bar, it seems to be garbled. Any suggestions? Thanks Anand _______________________________________________ Mediawiki-api mailing list Mediawiki-api(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-api

_______________________________________________ Mediawiki-api mailing list Mediawiki-api(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-api

Anand Ramanathan

8:06 a.m.

I tried curl from terminal on Mac, also tried open-uri from a ruby on rails app. On Mon, Nov 22, 2010 at 11:16 PM, Andrew Dunbar <hippytrail(a)gmail.com>wrote;wrote:

...

On 23 November 2010 03:07, Anand Ramanathan <rcanand(a)gmail.com> wrote:

Thanks Anand On Mon, Nov 22, 2010 at 3:26 PM, Betacommand <Betacommand(a)gmail.com>

wrote:

> > Can you define "garbled" > > On Mon, Nov 22, 2010 at 5:17 PM, Anand Ramanathan <rcanand(a)gmail.com> > wrote: >> >> Hi, >> I am not sure this is the right place for this issue, so please

redirect

>> me if there is a better forum. >> I have been loading the wikipedia page for India programmatically as

part

>> of testing my project. It was working fine, until recently. Now, most

of the

time (but not always), loading it gives me garbled content. Other pages on wikipedia work fine both programmatically and from the browser. This particular page also loads fine from my browser (safari on mac) - but when I load it programmatically (using curl or open-uri from mac-os x), the content is garbled. I am using a user agent, but don't know what changed. I tried closing the bar on top which appeared recently (Jimmy Wales message), but even without that bar, it seems to be garbled. Any suggestions? Thanks Anand _______________________________________________ Mediawiki-api mailing list Mediawiki-api(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-api

_______________________________________________ Mediawiki-api mailing list Mediawiki-api(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-api

Anand Ramanathan

8:08 a.m.

forgot to mention: versions : ruby 1.8.7, rails 3.0 - but within the same environment, it is intermittent... currently have been getting 'clean' content for last 12 hours or so, but had bad content before... On Tue, Nov 23, 2010 at 12:06 AM, Anand Ramanathan <rcanand(a)gmail.com>wrote;wrote:

...

I tried curl from terminal on Mac, also tried open-uri from a ruby on rails app. On Mon, Nov 22, 2010 at 11:16 PM, Andrew Dunbar <hippytrail(a)gmail.com>wrote;wrote:

On 23 November 2010 03:07, Anand Ramanathan <rcanand(a)gmail.com> wrote:

Thanks Anand On Mon, Nov 22, 2010 at 3:26 PM, Betacommand <Betacommand(a)gmail.com>

wrote:

> > Can you define "garbled" > > On Mon, Nov 22, 2010 at 5:17 PM, Anand Ramanathan <rcanand(a)gmail.com> > wrote: >> >> Hi, >> I am not sure this is the right place for this issue, so please

redirect

>> me if there is a better forum. >> I have been loading the wikipedia page for India programmatically as

part

>> of testing my project. It was working fine, until recently. Now, most

of the

>> time (but not always), loading it gives me garbled content. >> Other pages on wikipedia work fine both programmatically and from the >> browser. >> This particular page also loads fine from my browser (safari on mac) - >> but when I load it programmatically (using curl or open-uri from

mac-os

> x), the content is garbled. > I am using a user agent, but don't know what changed. > I tried closing the bar on top which appeared recently (Jimmy Wales > message), but even without that bar, it seems to be garbled. > Any suggestions? > Thanks > Anand > _______________________________________________ > Mediawiki-api mailing list > Mediawiki-api(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/mediawiki-api > _______________________________________________ Mediawiki-api mailing list Mediawiki-api(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-api

_______________________________________________ Mediawiki-api mailing list Mediawiki-api(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-api

Andrew Dunbar

8:14 a.m.

On 23 November 2010 10:08, Anand Ramanathan <rcanand(a)gmail.com> wrote:

...

Try getting it with the full HTTP headers too. Keep a copy of them from when it works to compare against when it doesn't work. Andrew Dunbar (hippietrail)

...

On Tue, Nov 23, 2010 at 12:06 AM, Anand Ramanathan <rcanand(a)gmail.com> wrote:

I tried curl from terminal on Mac, also tried open-uri from a ruby on rails app. On Mon, Nov 22, 2010 at 11:16 PM, Andrew Dunbar <hippytrail(a)gmail.com> wrote:

On 23 November 2010 03:07, Anand Ramanathan <rcanand(a)gmail.com> wrote:

Thanks Anand On Mon, Nov 22, 2010 at 3:26 PM, Betacommand <Betacommand(a)gmail.com> wrote: > > Can you define "garbled" > > On Mon, Nov 22, 2010 at 5:17 PM, Anand Ramanathan <rcanand(a)gmail.com> > wrote: >> >> Hi, >> I am not sure this is the right place for this issue, so please >> redirect >> me if there is a better forum. >> I have been loading the wikipedia page for India programmatically as >> part >> of testing my project. It was working fine, until recently. Now, most >> of the >> time (but not always), loading it gives me garbled content. >> Other pages on wikipedia work fine both programmatically and from the >> browser. >> This particular page also loads fine from my browser (safari on mac) >> - >> but when I load it programmatically (using curl or open-uri from >> mac-os >> x), the content is garbled. >> I am using a user agent, but don't know what changed. >> I tried closing the bar on top which appeared recently (Jimmy Wales >> message), but even without that bar, it seems to be garbled. >> Any suggestions? >> Thanks >> Anand >> _______________________________________________ >> Mediawiki-api mailing list >> Mediawiki-api(a)lists.wikimedia.org >> https://lists.wikimedia.org/mailman/listinfo/mediawiki-api >> > > > _______________________________________________ > Mediawiki-api mailing list > Mediawiki-api(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/mediawiki-api > _______________________________________________ Mediawiki-api mailing list Mediawiki-api(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-api

_______________________________________________ Mediawiki-api mailing list Mediawiki-api(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-api

Anand C Ramanathan

8:27 a.m.

Ok, will do. Thanks! Sent from my iPad On Nov 23, 2010, at 12:14 AM, Andrew Dunbar <hippytrail(a)gmail.com> wrote:

...

On 23 November 2010 10:08, Anand Ramanathan <rcanand(a)gmail.com> wrote:

Try getting it with the full HTTP headers too. Keep a copy of them from when it works to compare against when it doesn't work. Andrew Dunbar (hippietrail)

On Tue, Nov 23, 2010 at 12:06 AM, Anand Ramanathan <rcanand(a)gmail.com> wrote:

I tried curl from terminal on Mac, also tried open-uri from a ruby on rails app. On Mon, Nov 22, 2010 at 11:16 PM, Andrew Dunbar <hippytrail(a)gmail.com> wrote:

On 23 November 2010 03:07, Anand Ramanathan <rcanand(a)gmail.com> wrote: > It is intermittent - sometimes I see english characters, as expected. > Sometimes, i see some junk characters - right now getting normal > english > characters, so cant show a sample of the junk. But this is definitely > happening with this page - over the last 3 days off and on... > Will send the garbled example if and when it happens again. You might be getting gzipped content sometimes. I had this issue with certain Perl libraries. Normally you should get the content unzipped. What programming language/framework etc have you been using? Andrew Dunbar (hippietrail) > Thanks > Anand > > On Mon, Nov 22, 2010 at 3:26 PM, Betacommand <Betacommand(a)gmail.com> > wrote: >> >> Can you define "garbled" >> >> On Mon, Nov 22, 2010 at 5:17 PM, Anand Ramanathan <rcanand(a)gmail.com> >> wrote: >>> >>> Hi, >>> I am not sure this is the right place for this issue, so please >>> redirect >>> me if there is a better forum. >>> I have been loading the wikipedia page for India programmatically as >>> part >>> of testing my project. It was working fine, until recently. Now, most >>> of the >>> time (but not always), loading it gives me garbled content. >>> Other pages on wikipedia work fine both programmatically and from the >>> browser. >>> This particular page also loads fine from my browser (safari on mac) >>> - >>> but when I load it programmatically (using curl or open-uri from >>> mac-os >>> x), the content is garbled. >>> I am using a user agent, but don't know what changed. >>> I tried closing the bar on top which appeared recently (Jimmy Wales >>> message), but even without that bar, it seems to be garbled. >>> Any suggestions? >>> Thanks >>> Anand >>> _______________________________________________ >>> Mediawiki-api mailing list >>> Mediawiki-api(a)lists.wikimedia.org >>> https://lists.wikimedia.org/mailman/listinfo/mediawiki-api >>> >> >> >> _______________________________________________ >> Mediawiki-api mailing list >> Mediawiki-api(a)lists.wikimedia.org >> https://lists.wikimedia.org/mailman/listinfo/mediawiki-api >> > > > _______________________________________________ > Mediawiki-api mailing list > Mediawiki-api(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/mediawiki-api > > _______________________________________________ Mediawiki-api mailing list Mediawiki-api(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-api

_______________________________________________ Mediawiki-api mailing list Mediawiki-api(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-api

Anand Ramanathan

25 Nov 25 Nov

8:02 a.m.

OK, I got it again: Here is my curl output (headers + first few characters) for the garbled India wikipedia page (and the proper China wikipedia page for comparison below that): *$ curl -i http://en.wikipedia.org/wiki/India* HTTP/1.0 200 OK Date: Thu, 25 Nov 2010 07:40:49 GMT Server: Apache Cache-Control: private, s-maxage=0, max-age=0, must-revalidate Content-Language: en Vary: Accept-Encoding,Cookie Last-Modified: Wed, 24 Nov 2010 21:34:20 GMT Content-Encoding: gzip Content-Length: 115091 Content-Type: text/html; charset=UTF-8 X-Cache: HIT from sq66.wikimedia.org X-Cache-Lookup: HIT from sq66.wikimedia.org:3128 Age: 1125 X-Cache: HIT from sq66.wikimedia.org X-Cache-Lookup: HIT from sq66.wikimedia.org:80 Connection: close ??َ$K? ?=W~??ߊ??n涸??????X?????ՙ5 WS???=?V? ??"????M 9???EDTTU???#"#?ع??????r????O???7'?q?ę???9????'?Vgw???????????y~???????T?Y??I,???'????Y?/???^__w???$??}?}??e???v???r?S???yg-?􏎎??-'???Vo9~?>܊????0??E???8????"??;N> ??? ?,'QB????????<ȅ????(«?['I?q?9_.?-g?O???}???x?Lf"͂??w?O;?[??^??eX??d??n,??íi?ȓ?h? ǋ???_?/???(?/?4?n????'?d? ?-g?z5??{???ִ޾????>?Y??"w?w/???4D:?m99L??N??.?x̵~?M?p????qGe??F?t?[?pK?s7??տ;I?4???ڬ'i?? &?|??M????v??N&v???nc-?<Hc??????y???C?s?1?0???8????g?`?????????????|?_|??7V?'?A}0?;a܁S?P??????Y?H?Ç`??d}????fpL?3}????a??3?6??\?B?\&??w??Q?/_Ā? ?k8?qÚ?ØF???ID?'??w?(?t?)BX?????6:????? ?~??@??l:?Y?p)?ń>?߾3x?fs?^L?????ý^??????T\?c??'?????5?5?v?$??؟??9?1?c???|O??S???IE@!<????}?i?f?ĕ?R ??D? ?|??8`k?'4΅?gk!Zx?z?]??I?L?HRo??????|????)+/̺0???d1 ??9?8???86_?N?ȿ??+???????Y\?'?Yv?j?q???(8????v??O5g?"??(z????-E????QзiS?,{?W??n??*H[ 4??D?I ?G?(?l!&??? *$ curl -i http://en.wikipedia.org/wiki/China* HTTP/1.0 200 OK Date: Thu, 25 Nov 2010 07:40:59 GMT Server: Apache Cache-Control: private, s-maxage=0, max-age=0, must-revalidate Content-Language: en Vary: Accept-Encoding,Cookie Last-Modified: Tue, 23 Nov 2010 05:09:28 GMT Content-Length: 254063 Content-Type: text/html; charset=UTF-8 Age: 1230 X-Cache: HIT from sq71.wikimedia.org X-Cache-Lookup: HIT from sq71.wikimedia.org:3128 X-Cache: MISS from sq40.wikimedia.org X-Cache-Lookup: MISS from sq40.wikimedia.org:80 Connection: close <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" " http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org/1999/xhtml" lang="en" dir="ltr"> <head> <title>China - Wikipedia, the free encyclopedia</title> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /> <meta http-equiv="Content-Style-Type" content="text/css" /> <meta name="generator" content="MediaWiki 1.16wmf4" /> <link rel="apple-touch-icon" href=" http://en.wikipedia.org/apple-touch-icon.png" /> <link rel="shortcut icon" href="/favicon.ico" /> On Tue, Nov 23, 2010 at 12:27 AM, Anand C Ramanathan <rcanand(a)gmail.com>wrote;wrote:

...

Ok, will do. Thanks! Sent from my iPad On Nov 23, 2010, at 12:14 AM, Andrew Dunbar <hippytrail(a)gmail.com> wrote:

On 23 November 2010 10:08, Anand Ramanathan <rcanand(a)gmail.com> wrote: > forgot to mention: versions : ruby 1.8.7, rails 3.0 - but within the

same

environment, it is intermittent... currently have been getting 'clean' content for last 12 hours or so, but had bad content before...

Try getting it with the full HTTP headers too. Keep a copy of them from when it works to compare against when it doesn't work. Andrew Dunbar (hippietrail) > On Tue, Nov 23, 2010 at 12:06 AM, Anand Ramanathan <rcanand(a)gmail.com> > wrote: >> >> I tried curl from terminal on Mac, also tried open-uri from a ruby on >> rails app. >> >> On Mon, Nov 22, 2010 at 11:16 PM, Andrew Dunbar <hippytrail(a)gmail.com> >> wrote: >>> >>> On 23 November 2010 03:07, Anand Ramanathan <rcanand(a)gmail.com>

wrote:

>>>> It is intermittent - sometimes I see english characters, as expected. >>>> Sometimes, i see some junk characters - right now getting normal >>>> english >>>> characters, so cant show a sample of the junk. But this is definitely >>>> happening with this page - over the last 3 days off and on... >>>> Will send the garbled example if and when it happens again. >>> >>> You might be getting gzipped content sometimes. I had this issue with >>> certain Perl libraries. Normally you should get the content unzipped. >>> What programming language/framework etc have you been using? >>> >>> Andrew Dunbar (hippietrail) >>> >>>> Thanks >>>> Anand >>>> >>>> On Mon, Nov 22, 2010 at 3:26 PM, Betacommand <Betacommand(a)gmail.com> >>>> wrote: >>>>> >>>>> Can you define "garbled" >>>>> >>>>> On Mon, Nov 22, 2010 at 5:17 PM, Anand Ramanathan <

rcanand(a)gmail.com>

>>>>> wrote: >>>>>> >>>>>> Hi, >>>>>> I am not sure this is the right place for this issue, so please >>>>>> redirect >>>>>> me if there is a better forum. >>>>>> I have been loading the wikipedia page for India programmatically

>>>>>> part >>>>>> of testing my project. It was working fine, until recently. Now,

most

>>>>>> of the >>>>>> time (but not always), loading it gives me garbled content. >>>>>> Other pages on wikipedia work fine both programmatically and from

the

>>>>>> browser. >>>>>> This particular page also loads fine from my browser (safari on

mac)

>>>> - >>>> but when I load it programmatically (using curl or open-uri from >>>> mac-os >>>> x), the content is garbled. >>>> I am using a user agent, but don't know what changed. >>>> I tried closing the bar on top which appeared recently (Jimmy Wales >>>> message), but even without that bar, it seems to be garbled. >>>> Any suggestions? >>>> Thanks >>>> Anand >>>> _______________________________________________ >>>> Mediawiki-api mailing list >>>> Mediawiki-api(a)lists.wikimedia.org >>>> https://lists.wikimedia.org/mailman/listinfo/mediawiki-api >>>> >>> >>> >>> _______________________________________________ >>> Mediawiki-api mailing list >>> Mediawiki-api(a)lists.wikimedia.org >>> https://lists.wikimedia.org/mailman/listinfo/mediawiki-api >>> >> >> >> _______________________________________________ >> Mediawiki-api mailing list >> Mediawiki-api(a)lists.wikimedia.org >> https://lists.wikimedia.org/mailman/listinfo/mediawiki-api >> >> > > _______________________________________________ > Mediawiki-api mailing list > Mediawiki-api(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/mediawiki-api

_______________________________________________ Mediawiki-api mailing list Mediawiki-api(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-api

Bryan Tong Minh

8:12 a.m.

On Thu, Nov 25, 2010 at 9:02 AM, Anand Ramanathan <rcanand(a)gmail.com> wrote:

...

OK, I got it again: Here is my curl output (headers + first few characters) for the garbled India wikipedia page (and the proper China wikipedia page for comparison below that):

Can you verify that the first two characters are 0x1f and 0x8b respectively? Looks like gzip.

Anand Ramanathan

8:41 a.m.

Yes, confirmed that they are. It is gzip - what is the best way to deal with this? Is this a bug that is tracked, or is this something worth handling in client code (checking if gzip and manually unzipping)? Thanks Anand On Thu, Nov 25, 2010 at 12:12 AM, Bryan Tong Minh <bryan.tongminh(a)gmail.com>wrote;wrote:

...

On Thu, Nov 25, 2010 at 9:02 AM, Anand Ramanathan <rcanand(a)gmail.com> wrote:

OK, I got it again: Here is my curl output (headers + first few

characters)

for the garbled India wikipedia page (and the proper China wikipedia page for comparison below that):

Can you verify that the first two characters are 0x1f and 0x8b respectively? Looks like gzip. _______________________________________________ Mediawiki-api mailing list Mediawiki-api(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-api

Bryan Tong Minh

10:01 a.m.

On Thu, Nov 25, 2010 at 9:41 AM, Anand Ramanathan <rcanand(a)gmail.com> wrote:

...

Do you send the Accept-Encoding: gzip header? If not, this is a bug that should be reported in bugzilla. Bryan

Andrew Dunbar

12:23 p.m.

On 25 November 2010 19:41, Anand Ramanathan <rcanand(a)gmail.com> wrote:

...

Curl can definitely handle gzipped responses. Here's something about it from a very quick Google search: http://curl.haxx.se/mail/curlphp-2004-01/0043.html Andrew Dunbar (hippietrail)

...

On Thu, Nov 25, 2010 at 12:12 AM, Bryan Tong Minh <bryan.tongminh(a)gmail.com> wrote:

On Thu, Nov 25, 2010 at 9:02 AM, Anand Ramanathan <rcanand(a)gmail.com> wrote:

OK, I got it again: Here is my curl output (headers + first few characters) for the garbled India wikipedia page (and the proper China wikipedia page for comparison below that):

_______________________________________________ Mediawiki-api mailing list Mediawiki-api(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-api

Anand Ramanathan

7:19 p.m.

Bryan: No, I didnt set the Accept-Encoding header explicitly - I found the following related issue on bugzilla: 7098<https://bugzilla.wikimedia.org/show_bug.cgi?id=7098> Andrew: Yes, thanks. I see that curl can support this, and so can open-uri. I wanted to clarify if I should be handling this in the client: As per http 1.1 (section 14.3<http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html>)l>), for non-browser user agents, if no Accept-Encoding is explicitly set, the response should be the document itself if the server supports returning the document itself (identity). However, if the server is unable to return the document itself, it is preferable to return gzip or compressed content. I think this issue is happening whenever I hit a cache node that has the gzip, but not the identity cached. From a server standpoint, it seems like the right behavior. So, it is up to the client, which needs to do one of the following: a) Set Accept-Encoding to make gzip not-acceptable, and identity as acceptable. In this case, a cache node containing only gzip encoded document will miss, and eventually a node that contains the identity will return it. (This is a leap of faith, as I cannot target such a cache node explicitly. If a node has both gzip and identity content, and is responding with gzip for a request with no explicit Accept-Encoding set, then it violates the spec and is a bug. Can anyone comment on this?) b) Set Accept-Encoding to accept gzip or identity (or leave it unset), and on the client, if Content-Encoding is gzip, unzip it explicitly. I am fine with either of these approaches. Is this an accurate assessment of the issue and options? Thanks Anand On Thu, Nov 25, 2010 at 4:23 AM, Andrew Dunbar <hippytrail(a)gmail.com> wrote:

...

On 25 November 2010 19:41, Anand Ramanathan <rcanand(a)gmail.com> wrote:

Yes, confirmed that they are. It is gzip - what is the best way to deal

with

this? Is this a bug that is tracked, or is this something worth handling

client code (checking if gzip and manually unzipping)? Thanks Anand

Curl can definitely handle gzipped responses. Here's something about it from a very quick Google search: http://curl.haxx.se/mail/curlphp-2004-01/0043.html Andrew Dunbar (hippietrail)

On Thu, Nov 25, 2010 at 12:12 AM, Bryan Tong Minh <

bryan.tongminh(a)gmail.com>

wrote:

On Thu, Nov 25, 2010 at 9:02 AM, Anand Ramanathan <rcanand(a)gmail.com> wrote:

OK, I got it again: Here is my curl output (headers + first few characters) for the garbled India wikipedia page (and the proper China wikipedia page for comparison below that):

_______________________________________________ Mediawiki-api mailing list Mediawiki-api(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-api

Bryan Tong Minh

26 Nov 26 Nov

7:54 a.m.

I am forwarding your request to wikitech-l, in the hope that there are more people on there who can comment on this issue. For those who did not follow the entire thread: the user does not send an Accept-Encoding: gzip header, but nevertheless gets a gzipped response. On Thu, Nov 25, 2010 at 8:19 PM, Anand Ramanathan <rcanand(a)gmail.com> wrote:

...

Bryan: No, I didnt set the Accept-Encoding header explicitly - I found the following related issue on bugzilla: 7098 Andrew: Yes, thanks. I see that curl can support this, and so can open-uri. I wanted to clarify if I should be handling this in the client: As per http 1.1 (section 14.3), for non-browser user agents, if no Accept-Encoding is explicitly set, the response should be the document itself if the server supports returning the document itself (identity). However, if the server is unable to return the document itself, it is preferable to return gzip or compressed content. I think this issue is happening whenever I hit a cache node that has the gzip, but not the identity cached. From a server standpoint, it seems like the right behavior. So, it is up to the client, which needs to do one of the following: a) Set Accept-Encoding to make gzip not-acceptable, and identity as acceptable. In this case, a cache node containing only gzip encoded document will miss, and eventually a node that contains the identity will return it. (This is a leap of faith, as I cannot target such a cache node explicitly. If a node has both gzip and identity content, and is responding with gzip for a request with no explicit Accept-Encoding set, then it violates the spec and is a bug. Can anyone comment on this?) b) Set Accept-Encoding to accept gzip or identity (or leave it unset), and on the client, if Content-Encoding is gzip, unzip it explicitly. I am fine with either of these approaches. Is this an accurate assessment of the issue and options? Thanks Anand On Thu, Nov 25, 2010 at 4:23 AM, Andrew Dunbar <hippytrail(a)gmail.com> wrote:

On 25 November 2010 19:41, Anand Ramanathan <rcanand(a)gmail.com> wrote:

Curl can definitely handle gzipped responses. Here's something about it from a very quick Google search: http://curl.haxx.se/mail/curlphp-2004-01/0043.html Andrew Dunbar (hippietrail)

On Thu, Nov 25, 2010 at 12:12 AM, Bryan Tong Minh <bryan.tongminh(a)gmail.com> wrote:

On Thu, Nov 25, 2010 at 9:02 AM, Anand Ramanathan <rcanand(a)gmail.com> wrote: > OK, I got it again: Here is my curl output (headers + first few > characters) > for the garbled India wikipedia page (and the proper China wikipedia > page > for comparison below that): Can you verify that the first two characters are 0x1f and 0x8b respectively? Looks like gzip. _______________________________________________ Mediawiki-api mailing list Mediawiki-api(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-api

_______________________________________________ Mediawiki-api mailing list Mediawiki-api(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-api

Platonides

3 p.m.

Bryan Tong Minh wrote:

...

Bug https://bugzilla.wikimedia.org/show_bug.cgi?id=15149 / https://bugzilla.wikimedia.org/show_bug.cgi?id=7098 again :S Seems worth taking a look at sq66. Earlier I was getting that wrong gzipped output from sq66 (whereas eg. sq65 didn't). Now sq66.wikimedia.org is having a cache MISS.

4900

days inactive

4904

days old

mediawiki-api@lists.wikimedia.org

Manage subscription

15 comments

6 participants

tags (0)

participants (6)

Anand C Ramanathan
Anand Ramanathan
Andrew Dunbar
Betacommand
Bryan Tong Minh
Platonides