Hi,
I am not sure this is the right place for this issue, so please redirect me if there is a better forum.
I have been loading the wikipedia page for Indiahttp://en.wikipedia.org/wiki/Indiaprogrammatically as part of testing my project. It was working fine, until recently. Now, most of the time (but not always), loading it gives me garbled content. Other pages on wikipedia work fine both programmatically and from the browser. This particular page also loads fine from my browser (safari on mac) -
but when I load it programmatically (using curl or open-uri from mac-os x), the content is garbled.
I am using a user agent, but don't know what changed.
I tried closing the bar on top which appeared recently (Jimmy Wales message), but even without that bar, it seems to be garbled.
Any suggestions?
Thanks Anand
Can you define "garbled"
On Mon, Nov 22, 2010 at 5:17 PM, Anand Ramanathan rcanand@gmail.com wrote:
Hi,
I am not sure this is the right place for this issue, so please redirect me if there is a better forum.
I have been loading the wikipedia page for Indiahttp://en.wikipedia.org/wiki/Indiaprogrammatically as part of testing my project. It was working fine, until recently. Now, most of the time (but not always), loading it gives me garbled content. Other pages on wikipedia work fine both programmatically and from the browser. This particular page also loads fine from my browser (safari on mac) -
but when I load it programmatically (using curl or open-uri from mac-os x), the content is garbled.
I am using a user agent, but don't know what changed.
I tried closing the bar on top which appeared recently (Jimmy Wales message), but even without that bar, it seems to be garbled.
Any suggestions?
Thanks Anand
Mediawiki-api mailing list Mediawiki-api@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-api
It is intermittent - sometimes I see english characters, as expected. Sometimes, i see some junk characters - right now getting normal english characters, so cant show a sample of the junk. But this is definitely happening with this page - over the last 3 days off and on...
Will send the garbled example if and when it happens again.
Thanks Anand
On Mon, Nov 22, 2010 at 3:26 PM, Betacommand Betacommand@gmail.com wrote:
Can you define "garbled"
On Mon, Nov 22, 2010 at 5:17 PM, Anand Ramanathan rcanand@gmail.comwrote:
Hi,
I am not sure this is the right place for this issue, so please redirect me if there is a better forum.
I have been loading the wikipedia page for Indiahttp://en.wikipedia.org/wiki/Indiaprogrammatically as part of testing my project. It was working fine, until recently. Now, most of the time (but not always), loading it gives me garbled content. Other pages on wikipedia work fine both programmatically and from the browser. This particular page also loads fine from my browser (safari on mac) -
but when I load it programmatically (using curl or open-uri from mac-os x), the content is garbled.
I am using a user agent, but don't know what changed.
I tried closing the bar on top which appeared recently (Jimmy Wales message), but even without that bar, it seems to be garbled.
Any suggestions?
Thanks Anand
Mediawiki-api mailing list Mediawiki-api@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-api
Mediawiki-api mailing list Mediawiki-api@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-api
On 23 November 2010 03:07, Anand Ramanathan rcanand@gmail.com wrote:
It is intermittent - sometimes I see english characters, as expected. Sometimes, i see some junk characters - right now getting normal english characters, so cant show a sample of the junk. But this is definitely happening with this page - over the last 3 days off and on... Will send the garbled example if and when it happens again.
You might be getting gzipped content sometimes. I had this issue with certain Perl libraries. Normally you should get the content unzipped. What programming language/framework etc have you been using?
Andrew Dunbar (hippietrail)
Thanks Anand
On Mon, Nov 22, 2010 at 3:26 PM, Betacommand Betacommand@gmail.com wrote:
Can you define "garbled"
On Mon, Nov 22, 2010 at 5:17 PM, Anand Ramanathan rcanand@gmail.com wrote:
Hi, I am not sure this is the right place for this issue, so please redirect me if there is a better forum. I have been loading the wikipedia page for India programmatically as part of testing my project. It was working fine, until recently. Now, most of the time (but not always), loading it gives me garbled content. Other pages on wikipedia work fine both programmatically and from the browser. This particular page also loads fine from my browser (safari on mac) - but when I load it programmatically (using curl or open-uri from mac-os x), the content is garbled. I am using a user agent, but don't know what changed. I tried closing the bar on top which appeared recently (Jimmy Wales message), but even without that bar, it seems to be garbled. Any suggestions? Thanks Anand _______________________________________________ Mediawiki-api mailing list Mediawiki-api@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-api
Mediawiki-api mailing list Mediawiki-api@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-api
Mediawiki-api mailing list Mediawiki-api@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-api
I tried curl from terminal on Mac, also tried open-uri from a ruby on rails app.
On Mon, Nov 22, 2010 at 11:16 PM, Andrew Dunbar hippytrail@gmail.comwrote:
On 23 November 2010 03:07, Anand Ramanathan rcanand@gmail.com wrote:
It is intermittent - sometimes I see english characters, as expected. Sometimes, i see some junk characters - right now getting normal english characters, so cant show a sample of the junk. But this is definitely happening with this page - over the last 3 days off and on... Will send the garbled example if and when it happens again.
You might be getting gzipped content sometimes. I had this issue with certain Perl libraries. Normally you should get the content unzipped. What programming language/framework etc have you been using?
Andrew Dunbar (hippietrail)
Thanks Anand
On Mon, Nov 22, 2010 at 3:26 PM, Betacommand Betacommand@gmail.com
wrote:
Can you define "garbled"
On Mon, Nov 22, 2010 at 5:17 PM, Anand Ramanathan rcanand@gmail.com wrote:
Hi, I am not sure this is the right place for this issue, so please
redirect
me if there is a better forum. I have been loading the wikipedia page for India programmatically as
part
of testing my project. It was working fine, until recently. Now, most
of the
time (but not always), loading it gives me garbled content. Other pages on wikipedia work fine both programmatically and from the browser. This particular page also loads fine from my browser (safari on mac) - but when I load it programmatically (using curl or open-uri from mac-os x), the content is garbled. I am using a user agent, but don't know what changed. I tried closing the bar on top which appeared recently (Jimmy Wales message), but even without that bar, it seems to be garbled. Any suggestions? Thanks Anand _______________________________________________ Mediawiki-api mailing list Mediawiki-api@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-api
Mediawiki-api mailing list Mediawiki-api@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-api
Mediawiki-api mailing list Mediawiki-api@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-api
Mediawiki-api mailing list Mediawiki-api@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-api
forgot to mention: versions : ruby 1.8.7, rails 3.0 - but within the same environment, it is intermittent... currently have been getting 'clean' content for last 12 hours or so, but had bad content before...
On Tue, Nov 23, 2010 at 12:06 AM, Anand Ramanathan rcanand@gmail.comwrote:
I tried curl from terminal on Mac, also tried open-uri from a ruby on rails app.
On Mon, Nov 22, 2010 at 11:16 PM, Andrew Dunbar hippytrail@gmail.comwrote:
On 23 November 2010 03:07, Anand Ramanathan rcanand@gmail.com wrote:
It is intermittent - sometimes I see english characters, as expected. Sometimes, i see some junk characters - right now getting normal english characters, so cant show a sample of the junk. But this is definitely happening with this page - over the last 3 days off and on... Will send the garbled example if and when it happens again.
You might be getting gzipped content sometimes. I had this issue with certain Perl libraries. Normally you should get the content unzipped. What programming language/framework etc have you been using?
Andrew Dunbar (hippietrail)
Thanks Anand
On Mon, Nov 22, 2010 at 3:26 PM, Betacommand Betacommand@gmail.com
wrote:
Can you define "garbled"
On Mon, Nov 22, 2010 at 5:17 PM, Anand Ramanathan rcanand@gmail.com wrote:
Hi, I am not sure this is the right place for this issue, so please
redirect
me if there is a better forum. I have been loading the wikipedia page for India programmatically as
part
of testing my project. It was working fine, until recently. Now, most
of the
time (but not always), loading it gives me garbled content. Other pages on wikipedia work fine both programmatically and from the browser. This particular page also loads fine from my browser (safari on mac) - but when I load it programmatically (using curl or open-uri from
mac-os
x), the content is garbled. I am using a user agent, but don't know what changed. I tried closing the bar on top which appeared recently (Jimmy Wales message), but even without that bar, it seems to be garbled. Any suggestions? Thanks Anand _______________________________________________ Mediawiki-api mailing list Mediawiki-api@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-api
Mediawiki-api mailing list Mediawiki-api@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-api
Mediawiki-api mailing list Mediawiki-api@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-api
Mediawiki-api mailing list Mediawiki-api@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-api
On 23 November 2010 10:08, Anand Ramanathan rcanand@gmail.com wrote:
forgot to mention: versions : ruby 1.8.7, rails 3.0 - but within the same environment, it is intermittent... currently have been getting 'clean' content for last 12 hours or so, but had bad content before...
Try getting it with the full HTTP headers too. Keep a copy of them from when it works to compare against when it doesn't work.
Andrew Dunbar (hippietrail)
On Tue, Nov 23, 2010 at 12:06 AM, Anand Ramanathan rcanand@gmail.com wrote:
I tried curl from terminal on Mac, also tried open-uri from a ruby on rails app.
On Mon, Nov 22, 2010 at 11:16 PM, Andrew Dunbar hippytrail@gmail.com wrote:
On 23 November 2010 03:07, Anand Ramanathan rcanand@gmail.com wrote:
It is intermittent - sometimes I see english characters, as expected. Sometimes, i see some junk characters - right now getting normal english characters, so cant show a sample of the junk. But this is definitely happening with this page - over the last 3 days off and on... Will send the garbled example if and when it happens again.
You might be getting gzipped content sometimes. I had this issue with certain Perl libraries. Normally you should get the content unzipped. What programming language/framework etc have you been using?
Andrew Dunbar (hippietrail)
Thanks Anand
On Mon, Nov 22, 2010 at 3:26 PM, Betacommand Betacommand@gmail.com wrote:
Can you define "garbled"
On Mon, Nov 22, 2010 at 5:17 PM, Anand Ramanathan rcanand@gmail.com wrote:
Hi, I am not sure this is the right place for this issue, so please redirect me if there is a better forum. I have been loading the wikipedia page for India programmatically as part of testing my project. It was working fine, until recently. Now, most of the time (but not always), loading it gives me garbled content. Other pages on wikipedia work fine both programmatically and from the browser. This particular page also loads fine from my browser (safari on mac)
but when I load it programmatically (using curl or open-uri from mac-os x), the content is garbled. I am using a user agent, but don't know what changed. I tried closing the bar on top which appeared recently (Jimmy Wales message), but even without that bar, it seems to be garbled. Any suggestions? Thanks Anand _______________________________________________ Mediawiki-api mailing list Mediawiki-api@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-api
Mediawiki-api mailing list Mediawiki-api@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-api
Mediawiki-api mailing list Mediawiki-api@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-api
Mediawiki-api mailing list Mediawiki-api@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-api
Mediawiki-api mailing list Mediawiki-api@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-api
Ok, will do. Thanks!
Sent from my iPad
On Nov 23, 2010, at 12:14 AM, Andrew Dunbar hippytrail@gmail.com wrote:
On 23 November 2010 10:08, Anand Ramanathan rcanand@gmail.com wrote:
forgot to mention: versions : ruby 1.8.7, rails 3.0 - but within the same environment, it is intermittent... currently have been getting 'clean' content for last 12 hours or so, but had bad content before...
Try getting it with the full HTTP headers too. Keep a copy of them from when it works to compare against when it doesn't work.
Andrew Dunbar (hippietrail)
On Tue, Nov 23, 2010 at 12:06 AM, Anand Ramanathan rcanand@gmail.com wrote:
I tried curl from terminal on Mac, also tried open-uri from a ruby on rails app.
On Mon, Nov 22, 2010 at 11:16 PM, Andrew Dunbar hippytrail@gmail.com wrote:
On 23 November 2010 03:07, Anand Ramanathan rcanand@gmail.com wrote:
It is intermittent - sometimes I see english characters, as expected. Sometimes, i see some junk characters - right now getting normal english characters, so cant show a sample of the junk. But this is definitely happening with this page - over the last 3 days off and on... Will send the garbled example if and when it happens again.
You might be getting gzipped content sometimes. I had this issue with certain Perl libraries. Normally you should get the content unzipped. What programming language/framework etc have you been using?
Andrew Dunbar (hippietrail)
Thanks Anand
On Mon, Nov 22, 2010 at 3:26 PM, Betacommand Betacommand@gmail.com wrote:
Can you define "garbled"
On Mon, Nov 22, 2010 at 5:17 PM, Anand Ramanathan rcanand@gmail.com wrote: > > Hi, > I am not sure this is the right place for this issue, so please > redirect > me if there is a better forum. > I have been loading the wikipedia page for India programmatically as > part > of testing my project. It was working fine, until recently. Now, most > of the > time (but not always), loading it gives me garbled content. > Other pages on wikipedia work fine both programmatically and from the > browser. > This particular page also loads fine from my browser (safari on mac) > - > but when I load it programmatically (using curl or open-uri from > mac-os > x), the content is garbled. > I am using a user agent, but don't know what changed. > I tried closing the bar on top which appeared recently (Jimmy Wales > message), but even without that bar, it seems to be garbled. > Any suggestions? > Thanks > Anand > _______________________________________________ > Mediawiki-api mailing list > Mediawiki-api@lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/mediawiki-api >
Mediawiki-api mailing list Mediawiki-api@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-api
Mediawiki-api mailing list Mediawiki-api@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-api
Mediawiki-api mailing list Mediawiki-api@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-api
Mediawiki-api mailing list Mediawiki-api@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-api
Mediawiki-api mailing list Mediawiki-api@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-api
OK, I got it again: Here is my curl output (headers + first few characters) for the garbled India wikipedia page (and the proper China wikipedia page for comparison below that):
*$ curl -i http://en.wikipedia.org/wiki/India* HTTP/1.0 200 OK Date: Thu, 25 Nov 2010 07:40:49 GMT Server: Apache Cache-Control: private, s-maxage=0, max-age=0, must-revalidate Content-Language: en Vary: Accept-Encoding,Cookie Last-Modified: Wed, 24 Nov 2010 21:34:20 GMT Content-Encoding: gzip Content-Length: 115091 Content-Type: text/html; charset=UTF-8 X-Cache: HIT from sq66.wikimedia.org X-Cache-Lookup: HIT from sq66.wikimedia.org:3128 Age: 1125 X-Cache: HIT from sq66.wikimedia.org X-Cache-Lookup: HIT from sq66.wikimedia.org:80 Connection: close
??َ$K? ?=W~??ߊ??n涸??????X?????ՙ5 WS???=?V? ??"????M 9???EDTTU???#"#?ع??????r????O???7'?q?ę???9????'?Vgw???????????y~???????T?Y??I,???'????Y?/???^__w???$??}?}??e???v???r?S???yg-???-'???Vo9~?>܊????0??E???8????"??;N> ??? ?,'QB????????<ȅ????(«?['I?q?9_.?-g?O???}???x?Lf"͂??w?O;?[??^??eX??d??n,??íi?ȓ?h? Nj???_?/???(?/?4?n????'?d? ?-g?z5??{???ִ????>?Y??"w?w/???4D:?m99L??N??.?x̵~?M?p????qGe??F?t?[?pK?s7??տ;I?4???ڬ'i?? &?|??M????v??N&v???nc-?<Hc??????y???C?s?1?0???8????g?`?????????????|?_|??7V?'?A}0?;a܁S?P??????Y?H?Ç`??d}????fpL?3}????a??3?6???B?&??w??Q?/_Ā?
?k8?qÚ?ØF???ID?'??w?(?t?)BX?????6:????? ?~??@??l:?Y?p)?ń>?߾3x?fs?^L?????ý^??????T?c??'?????5?5?v?$??؟??9?1?c???|O??S???IE@!<????}?i?f?ĕ?R ??D? ?|??8`k?'4΅?gk!Zx?z?]??I?L?HRo??????|????)+/̺0???d1
??9?8???86_?N?ȿ??+???????Y?'?Yv?j?q???(8????v??O5g?"??(z????-E????QзiS?,{?W??n??*H[ 4??D?I ?G?(?l!&???
*$ curl -i http://en.wikipedia.org/wiki/China* HTTP/1.0 200 OK Date: Thu, 25 Nov 2010 07:40:59 GMT Server: Apache Cache-Control: private, s-maxage=0, max-age=0, must-revalidate Content-Language: en Vary: Accept-Encoding,Cookie Last-Modified: Tue, 23 Nov 2010 05:09:28 GMT Content-Length: 254063 Content-Type: text/html; charset=UTF-8 Age: 1230 X-Cache: HIT from sq71.wikimedia.org X-Cache-Lookup: HIT from sq71.wikimedia.org:3128 X-Cache: MISS from sq40.wikimedia.org X-Cache-Lookup: MISS from sq40.wikimedia.org:80 Connection: close
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" " http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org/1999/xhtml" lang="en" dir="ltr"> <head> <title>China - Wikipedia, the free encyclopedia</title> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /> <meta http-equiv="Content-Style-Type" content="text/css" /> <meta name="generator" content="MediaWiki 1.16wmf4" /> <link rel="apple-touch-icon" href=" http://en.wikipedia.org/apple-touch-icon.png" /> <link rel="shortcut icon" href="/favicon.ico" />
On Tue, Nov 23, 2010 at 12:27 AM, Anand C Ramanathan rcanand@gmail.comwrote:
Ok, will do. Thanks!
Sent from my iPad
On Nov 23, 2010, at 12:14 AM, Andrew Dunbar hippytrail@gmail.com wrote:
On 23 November 2010 10:08, Anand Ramanathan rcanand@gmail.com wrote:
forgot to mention: versions : ruby 1.8.7, rails 3.0 - but within the
same
environment, it is intermittent... currently have been getting 'clean' content for last 12 hours or so, but had bad content before...
Try getting it with the full HTTP headers too. Keep a copy of them from when it works to compare against when it doesn't work.
Andrew Dunbar (hippietrail)
On Tue, Nov 23, 2010 at 12:06 AM, Anand Ramanathan rcanand@gmail.com wrote:
I tried curl from terminal on Mac, also tried open-uri from a ruby on rails app.
On Mon, Nov 22, 2010 at 11:16 PM, Andrew Dunbar hippytrail@gmail.com wrote:
On 23 November 2010 03:07, Anand Ramanathan rcanand@gmail.com
wrote:
It is intermittent - sometimes I see english characters, as expected. Sometimes, i see some junk characters - right now getting normal english characters, so cant show a sample of the junk. But this is definitely happening with this page - over the last 3 days off and on... Will send the garbled example if and when it happens again.
You might be getting gzipped content sometimes. I had this issue with certain Perl libraries. Normally you should get the content unzipped. What programming language/framework etc have you been using?
Andrew Dunbar (hippietrail)
Thanks Anand
On Mon, Nov 22, 2010 at 3:26 PM, Betacommand Betacommand@gmail.com wrote: > > Can you define "garbled" > > On Mon, Nov 22, 2010 at 5:17 PM, Anand Ramanathan <
rcanand@gmail.com>
> wrote: >> >> Hi, >> I am not sure this is the right place for this issue, so please >> redirect >> me if there is a better forum. >> I have been loading the wikipedia page for India programmatically
as
>> part >> of testing my project. It was working fine, until recently. Now,
most
>> of the >> time (but not always), loading it gives me garbled content. >> Other pages on wikipedia work fine both programmatically and from
the
>> browser. >> This particular page also loads fine from my browser (safari on
mac)
>> - >> but when I load it programmatically (using curl or open-uri from >> mac-os >> x), the content is garbled. >> I am using a user agent, but don't know what changed. >> I tried closing the bar on top which appeared recently (Jimmy Wales >> message), but even without that bar, it seems to be garbled. >> Any suggestions? >> Thanks >> Anand >> _______________________________________________ >> Mediawiki-api mailing list >> Mediawiki-api@lists.wikimedia.org >> https://lists.wikimedia.org/mailman/listinfo/mediawiki-api >> > > > _______________________________________________ > Mediawiki-api mailing list > Mediawiki-api@lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/mediawiki-api >
Mediawiki-api mailing list Mediawiki-api@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-api
Mediawiki-api mailing list Mediawiki-api@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-api
Mediawiki-api mailing list Mediawiki-api@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-api
Mediawiki-api mailing list Mediawiki-api@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-api
On Thu, Nov 25, 2010 at 9:02 AM, Anand Ramanathan rcanand@gmail.com wrote:
OK, I got it again: Here is my curl output (headers + first few characters) for the garbled India wikipedia page (and the proper China wikipedia page for comparison below that):
Can you verify that the first two characters are 0x1f and 0x8b respectively? Looks like gzip.
Yes, confirmed that they are. It is gzip - what is the best way to deal with this? Is this a bug that is tracked, or is this something worth handling in client code (checking if gzip and manually unzipping)?
Thanks Anand
On Thu, Nov 25, 2010 at 12:12 AM, Bryan Tong Minh bryan.tongminh@gmail.comwrote:
On Thu, Nov 25, 2010 at 9:02 AM, Anand Ramanathan rcanand@gmail.com wrote:
OK, I got it again: Here is my curl output (headers + first few
characters)
for the garbled India wikipedia page (and the proper China wikipedia page for comparison below that):
Can you verify that the first two characters are 0x1f and 0x8b respectively? Looks like gzip.
Mediawiki-api mailing list Mediawiki-api@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-api
On Thu, Nov 25, 2010 at 9:41 AM, Anand Ramanathan rcanand@gmail.com wrote:
Yes, confirmed that they are. It is gzip - what is the best way to deal with this? Is this a bug that is tracked, or is this something worth handling in client code (checking if gzip and manually unzipping)?
Do you send the Accept-Encoding: gzip header? If not, this is a bug that should be reported in bugzilla.
Bryan
On 25 November 2010 19:41, Anand Ramanathan rcanand@gmail.com wrote:
Yes, confirmed that they are. It is gzip - what is the best way to deal with this? Is this a bug that is tracked, or is this something worth handling in client code (checking if gzip and manually unzipping)? Thanks Anand
Curl can definitely handle gzipped responses. Here's something about it from a very quick Google search: http://curl.haxx.se/mail/curlphp-2004-01/0043.html
Andrew Dunbar (hippietrail)
On Thu, Nov 25, 2010 at 12:12 AM, Bryan Tong Minh bryan.tongminh@gmail.com wrote:
On Thu, Nov 25, 2010 at 9:02 AM, Anand Ramanathan rcanand@gmail.com wrote:
OK, I got it again: Here is my curl output (headers + first few characters) for the garbled India wikipedia page (and the proper China wikipedia page for comparison below that):
Can you verify that the first two characters are 0x1f and 0x8b respectively? Looks like gzip.
Mediawiki-api mailing list Mediawiki-api@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-api
Mediawiki-api mailing list Mediawiki-api@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-api
Bryan: No, I didnt set the Accept-Encoding header explicitly - I found the following related issue on bugzilla: 7098https://bugzilla.wikimedia.org/show_bug.cgi?id=7098
Andrew: Yes, thanks. I see that curl can support this, and so can open-uri.
I wanted to clarify if I should be handling this in the client:
As per http 1.1 (section 14.3http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html), for non-browser user agents, if no Accept-Encoding is explicitly set, the response should be the document itself if the server supports returning the document itself (identity). However, if the server is unable to return the document itself, it is preferable to return gzip or compressed content.
I think this issue is happening whenever I hit a cache node that has the gzip, but not the identity cached. From a server standpoint, it seems like the right behavior. So, it is up to the client, which needs to do one of the following:
a) Set Accept-Encoding to make gzip not-acceptable, and identity as acceptable. In this case, a cache node containing only gzip encoded document will miss, and eventually a node that contains the identity will return it. (This is a leap of faith, as I cannot target such a cache node explicitly. If a node has both gzip and identity content, and is responding with gzip for a request with no explicit Accept-Encoding set, then it violates the spec and is a bug. Can anyone comment on this?) b) Set Accept-Encoding to accept gzip or identity (or leave it unset), and on the client, if Content-Encoding is gzip, unzip it explicitly.
I am fine with either of these approaches. Is this an accurate assessment of the issue and options?
Thanks Anand
On Thu, Nov 25, 2010 at 4:23 AM, Andrew Dunbar hippytrail@gmail.com wrote:
On 25 November 2010 19:41, Anand Ramanathan rcanand@gmail.com wrote:
Yes, confirmed that they are. It is gzip - what is the best way to deal
with
this? Is this a bug that is tracked, or is this something worth handling
in
client code (checking if gzip and manually unzipping)? Thanks Anand
Curl can definitely handle gzipped responses. Here's something about it from a very quick Google search: http://curl.haxx.se/mail/curlphp-2004-01/0043.html
Andrew Dunbar (hippietrail)
On Thu, Nov 25, 2010 at 12:12 AM, Bryan Tong Minh <
bryan.tongminh@gmail.com>
wrote:
On Thu, Nov 25, 2010 at 9:02 AM, Anand Ramanathan rcanand@gmail.com wrote:
OK, I got it again: Here is my curl output (headers + first few characters) for the garbled India wikipedia page (and the proper China wikipedia page for comparison below that):
Can you verify that the first two characters are 0x1f and 0x8b respectively? Looks like gzip.
Mediawiki-api mailing list Mediawiki-api@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-api
Mediawiki-api mailing list Mediawiki-api@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-api
Mediawiki-api mailing list Mediawiki-api@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-api
I am forwarding your request to wikitech-l, in the hope that there are more people on there who can comment on this issue.
For those who did not follow the entire thread: the user does not send an Accept-Encoding: gzip header, but nevertheless gets a gzipped response.
On Thu, Nov 25, 2010 at 8:19 PM, Anand Ramanathan rcanand@gmail.com wrote:
Bryan: No, I didnt set the Accept-Encoding header explicitly - I found the following related issue on bugzilla: 7098
Andrew: Yes, thanks. I see that curl can support this, and so can open-uri.
I wanted to clarify if I should be handling this in the client: As per http 1.1 (section 14.3), for non-browser user agents, if no Accept-Encoding is explicitly set, the response should be the document itself if the server supports returning the document itself (identity). However, if the server is unable to return the document itself, it is preferable to return gzip or compressed content. I think this issue is happening whenever I hit a cache node that has the gzip, but not the identity cached. From a server standpoint, it seems like the right behavior. So, it is up to the client, which needs to do one of the following: a) Set Accept-Encoding to make gzip not-acceptable, and identity as acceptable. In this case, a cache node containing only gzip encoded document will miss, and eventually a node that contains the identity will return it. (This is a leap of faith, as I cannot target such a cache node explicitly. If a node has both gzip and identity content, and is responding with gzip for a request with no explicit Accept-Encoding set, then it violates the spec and is a bug. Can anyone comment on this?) b) Set Accept-Encoding to accept gzip or identity (or leave it unset), and on the client, if Content-Encoding is gzip, unzip it explicitly. I am fine with either of these approaches. Is this an accurate assessment of the issue and options? Thanks Anand
On Thu, Nov 25, 2010 at 4:23 AM, Andrew Dunbar hippytrail@gmail.com wrote:
On 25 November 2010 19:41, Anand Ramanathan rcanand@gmail.com wrote:
Yes, confirmed that they are. It is gzip - what is the best way to deal with this? Is this a bug that is tracked, or is this something worth handling in client code (checking if gzip and manually unzipping)? Thanks Anand
Curl can definitely handle gzipped responses. Here's something about it from a very quick Google search: http://curl.haxx.se/mail/curlphp-2004-01/0043.html
Andrew Dunbar (hippietrail)
On Thu, Nov 25, 2010 at 12:12 AM, Bryan Tong Minh bryan.tongminh@gmail.com wrote:
On Thu, Nov 25, 2010 at 9:02 AM, Anand Ramanathan rcanand@gmail.com wrote:
OK, I got it again: Here is my curl output (headers + first few characters) for the garbled India wikipedia page (and the proper China wikipedia page for comparison below that):
Can you verify that the first two characters are 0x1f and 0x8b respectively? Looks like gzip.
Mediawiki-api mailing list Mediawiki-api@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-api
Mediawiki-api mailing list Mediawiki-api@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-api
Mediawiki-api mailing list Mediawiki-api@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-api
Mediawiki-api mailing list Mediawiki-api@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-api
Bryan Tong Minh wrote:
I am forwarding your request to wikitech-l, in the hope that there are more people on there who can comment on this issue.
For those who did not follow the entire thread: the user does not send an Accept-Encoding: gzip header, but nevertheless gets a gzipped response.
Bug https://bugzilla.wikimedia.org/show_bug.cgi?id=15149 / https://bugzilla.wikimedia.org/show_bug.cgi?id=7098 again :S
Seems worth taking a look at sq66. Earlier I was getting that wrong gzipped output from sq66 (whereas eg. sq65 didn't). Now sq66.wikimedia.org is having a cache MISS.
mediawiki-api@lists.wikimedia.org