There appears to be some sort of bug affecting a subset of pages that started happening yesterday night. Certain <img src=> tags on Wikipedia's API are being served with an invalid URL, missing the scheme in front:
<img src="//bits.wikimedia.org/skins-1.17/common/images/magnify-clip.png">
This is seen on http://en.wikipedia.org/w/api.php?action=parse&page=Cereal
However certain other pages, such as http://en.wikipedia.org/w/api.php?action=parse&page=Actinide appear to be fine.
Please let me know if this e-mail should go to some Wikipedia list instead.
Thanks,
Robert
On Tue, Sep 27, 2011 at 09:40:20AM -0700, Robert Chin wrote:
There appears to be some sort of bug affecting a subset of pages that started happening yesterday night. Certain <img src=> tags on Wikipedia's API are being served with an invalid URL, missing the scheme in front:
<img src="//bits.wikimedia.org/skins-1.17/common/images/magnify-clip.png">
This is seen on http://en.wikipedia.org/w/api.php?action=parse&page=Cereal
However certain other pages, such as http://en.wikipedia.org/w/api.php?action=parse&page=Actinide appear to be fine.
Please let me know if this e-mail should go to some Wikipedia list instead.
This is due to the change to protocol-releative URIs. See [1] for details of the feature, and for the announcement on how this affects the API.
[1] https://secure.wikimedia.org/wikipedia/mediawiki/wiki/MediaWiki_1.18#Protoco... [2] http://lists.wikimedia.org/pipermail/mediawiki-api-announce/2011-July/000023...
On Tue, Sep 27, 2011 at 6:40 PM, Robert Chin robert.chin@gmail.com wrote:
There appears to be some sort of bug affecting a subset of pages that started happening yesterday night. Certain <img src=> tags on Wikipedia's API are being served with an invalid URL, missing the scheme in front:
<img src="//bits.wikimedia.org/skins-1.17/common/images/magnify-clip.png">
These URLs are not invalid, they're protocol-relative. They should work in every browser. They're officially called 'network-path references' and specified in RFC 3986 section 4.2 [1]. If your client can't deal with these URLs, that's because it doesn't comply with that RFC.
That said, I should probably have announced that this was happening yesterday. Per Brad, there was a generic "this is coming" announcement, but I should've announced the exact time in advance. My apologies for that. Also, per my earlier announcement, the API will output http URLs rather than protocol-relative URLs in almost all cases.
Roan Kattouw (Catrope)
On Tue, Sep 27, 2011 at 10:37 AM, Roan Kattouw roan.kattouw@gmail.com wrote:
These URLs are not invalid, they're protocol-relative. They should work in every browser. They're officially called 'network-path references' and specified in RFC 3986 section 4.2 [1]. If your client can't deal with these URLs, that's because it doesn't comply with that RFC.
That said, I should probably have announced that this was happening yesterday. Per Brad, there was a generic "this is coming" announcement, but I should've announced the exact time in advance. My apologies for that. Also, per my earlier announcement, the API will output http URLs rather than protocol-relative URLs in almost all cases.
Ouch, I guess I missed this e-mail since it didn't say "BREAKING CHANGE" like normal.
Is it possible to roll back this change for the main servers for about two weeks? It actually breaks all of the mobile clients iOS I have tested: Wikipanion, Articles, iWiki and probably almost others. They have to set their base URL to the local disk (file:///) in order to be able to load local resources off of the hard drive (stylesheets, images, etc.) due to security limitations.
Thanks,
Robert
On Tue, Sep 27, 2011 at 11:59 PM, Robert Chin robert.chin@gmail.com wrote:
Ouch, I guess I missed this e-mail since it didn't say "BREAKING CHANGE" like normal.
Hmm, I guess I should've put that in, my apologies. I made it so most things weren't breaking, only URLs embedded in HTML changed.
Is it possible to roll back this change for the main servers for about two weeks?
I'd really rather not. The change is necessary to make the impending deployment of HTTPS possible, and it was announced well in advance on the announcements list. Admittedly it didn't say "BREAKING CHANGE" like it should have, but still, I don't send announcements to mediawiki-api-announce if they're not important.
It actually breaks all of the mobile clients iOS I have tested: Wikipanion, Articles, iWiki and probably almost others. They have to set their base URL to the local disk (file:///) in order to be able to load local resources off of the hard drive (stylesheets, images, etc.) due to security limitations.
Ouch. That's a painful way to break, but also a really nasty hack. Don't they have to do something special to make /wiki/Some_page work with that hack, anyway?
Roan Kattouw (Catrope)
On Tue, Sep 27, 2011 at 4:14 PM, Roan Kattouw roan.kattouw@gmail.com wrote:
On Tue, Sep 27, 2011 at 11:59 PM, Robert Chin robert.chin@gmail.com wrote:
Is it possible to roll back this change for the main servers for about two weeks?
I'd really rather not. The change is necessary to make the impending deployment of HTTPS possible, and it was announced well in advance on the announcements list. Admittedly it didn't say "BREAKING CHANGE" like it should have, but still, I don't send announcements to mediawiki-api-announce if they're not important.
How about deferring by just one week? I realize the problem is entirely the fault of developers such as myself not realizing the scope of your previous e-mail, but it would be really great to get at least some time for myself and others to remedy this situation on our side. Those of us on iOS unfortunately have to deal with waiting for Apple to approve application updates and are thus unable to roll out updates to customers in under a week. It would be really great if you could allow us some flexibility in this area though -- could you maybe put in a temporary workaround in action=parse to resolve the protocol relative links?
It actually breaks all of the mobile clients iOS I have tested: Wikipanion, Articles, iWiki and probably almost others. They
Ouch. That's a painful way to break, but also a really nasty hack. Don't they have to do something special to make /wiki/Some_page work with that hack, anyway?
It's actually not a hack -- it's the only way to load local resources. iOS enforces clients to declare their base URL and only if their base URL is a local URL then web pages are allowed to load local resources. This prevents remote web pages from ever being allowed to read local resources at all, effectively sandboxing web pages. Loading links is ok, as on iOS a callback is made to the client application any time a user clicks on a link, thus allowing the link itself to be appropriately processed. Obviously if clients are using the API to fetch pages they will have to parse these links anyway rather than simply following them as normal links.
Robert
On Wed, Sep 28, 2011 at 1:36 AM, Robert Chin robert.chin@gmail.com wrote:
How about deferring by just one week?
This is not very feasible. We will have lots of HTML with protocol-relative URLs in it in our caches at this point, so at best you'll get an inconsistent mix between absolute and protocol-relative URLs. The majority will probably shift quickly but the minority will keep breaking you. It would also push back the HTTPS deployment that relies on this, and that would also be undesirable.
I realize the problem is entirely the fault of developers such as myself not realizing the scope of your previous e-mail, but it would be really great to get at least some time for myself and others to remedy this situation on our side. Those of us on iOS unfortunately have to deal with waiting for Apple to approve application updates and are thus unable to roll out updates to customers in under a week.
Yeah that sucks for you, I sympathize.
It would be really great if you could allow us some flexibility in this area though -- could you maybe put in a temporary workaround in action=parse to resolve the protocol relative links?
That might work. I'll look into that but it'll have to wait tomorrow, because it's almost 2am and I need sleep.
It's actually not a hack -- it's the only way to load local resources. iOS enforces clients to declare their base URL and only if their base URL is a local URL then web pages are allowed to load local resources. This prevents remote web pages from ever being allowed to read local resources at all, effectively sandboxing web pages.
Hmm, interesting. The idea of sandboxing is good, but the implementation sounds like it has issues.
Loading links is ok, as on iOS a callback is made to the client application any time a user clicks on a link, thus allowing the link itself to be appropriately processed.
Right. That hook is pretty much an indication that the whole thing is a hack :)
But my criticizing iOS architecture isn't really useful :) ; it's not like we can change it, we'll have to suck it up and live with it.
Obviously if clients are using the API to fetch pages they will have to parse these links anyway rather than simply following them as normal links.
Right, so clients already use this hook to rewrite URLs, they'll just have to rewrite them differently? Sounds like that shouldn't be too hard.
Roan Kattouw (Catrope)
On Tue, Sep 27, 2011 at 4:54 PM, Roan Kattouw roan.kattouw@gmail.com wrote:
On Wed, Sep 28, 2011 at 1:36 AM, Robert Chin robert.chin@gmail.com wrote:
It would be really great if you could allow us some flexibility in this area though -- could you maybe put in a temporary workaround in action=parse to resolve the protocol relative links?
That might work. I'll look into that but it'll have to wait tomorrow, because it's almost 2am and I need sleep.
Thanks in advance for looking into this! I would be more than happy to look and this and hand you a patch if it helps this temporary solution get in.
Right, so clients already use this hook to rewrite URLs, they'll just have to rewrite them differently? Sounds like that shouldn't be too hard.
Unfortunately not -- these callback only happen when users click on links (and not when every resource on the page is loaded). I expect that most developers will parse and modify or run a regular expression on the page returned from api.php, or will write custom javascript to rewrite image URLs.
Robert
On Wed, Sep 28, 2011 at 1:54 AM, Roan Kattouw roan.kattouw@gmail.com wrote:
It would be really great if you could allow us some flexibility in this area though -- could you maybe put in a temporary workaround in action=parse to resolve the protocol relative links?
That might work. I'll look into that but it'll have to wait tomorrow, because it's almost 2am and I need sleep.
I have now deployed a temporary hack that does a few things differently in action=parse:
1) Links that would normally be internal links (/wiki/Main_Page) are now fully-qualified URLs (http://en.wikipedia.org/wiki/Main_Page). This is similar to action=render's behavior, except that that generates protocol-relative URLs (//en.wikipedia.org). This sounds weird and unnecessary, and at first I thought it was, but if I don't do this, some URLs like //en.wikipedia.org/w/index.php?title=Talk:Foo&action=edit are generated. 2) Paths to images, resources, etc etc are all fully-qualified
On http://en.wikipedia.org/w/api.php?action=parse&page=Cereal , this made searching for '"//' (double quote slash slash) turn up nothing, which satisfied me. So your clients should all be "fixed" now, in that we should be working around their bugs completely.
Because of the cache splitting involved, this is causing a load increase on our API cluster. So please let me know when the iOS clients are fixed and I can turn this hack back off. I am not leaving this up for more than two weeks, though, so I will turn it off on October 12th if I don't hear from you before then.
Roan
On Wed, Sep 28, 2011 at 9:58 AM, Roan Kattouw roan.kattouw@gmail.com wrote:
Because of the cache splitting involved, this is causing a load increase on our API cluster. So please let me know when the iOS clients are fixed and I can turn this hack back off. I am not leaving this up for more than two weeks, though, so I will turn it off on October 12th if I don't hear from you before then.
Wouldn't be a bad idea to add an addendum to the blog post on protocol-relativeness for people who might not be on the list.
-Chad
On Wed, Sep 28, 2011 at 4:02 PM, Chad innocentkiller@gmail.com wrote:
Wouldn't be a bad idea to add an addendum to the blog post on protocol-relativeness for people who might not be on the list.
Good idea, I'll get on that. Right now I'm informing our operations people of the increased server load.
Roan Kattouw (Catrope)
On Wed, Sep 28, 2011 at 4:03 PM, Roan Kattouw roan.kattouw@gmail.com wrote:
On Wed, Sep 28, 2011 at 4:02 PM, Chad innocentkiller@gmail.com wrote:
Wouldn't be a bad idea to add an addendum to the blog post on protocol-relativeness for people who might not be on the list.
Good idea, I'll get on that. Right now I'm informing our operations people of the increased server load.
Done, see http://blog.wikimedia.org/2011/09/27/protocol-relative-urls-enabled-on-all-w...
Roan
On Wed, Sep 28, 2011 at 6:58 AM, Roan Kattouw roan.kattouw@gmail.com wrote:
On Wed, Sep 28, 2011 at 1:54 AM, Roan Kattouw roan.kattouw@gmail.com wrote: Because of the cache splitting involved, this is causing a load increase on our API cluster. So please let me know when the iOS clients are fixed and I can turn this hack back off. I am not leaving this up for more than two weeks, though, so I will turn it off on October 12th if I don't hear from you before then.
Thanks, I will try to contact as many as possible to make sure they are aware of the changes. An October 12 deadline should be more than enough time.
Robert
mediawiki-api@lists.wikimedia.org