tl;dr: Let's add class="skintimestamp-YYYYMMDD" to <body> in all skins, so we can make HTML+CSS changes without breaking the sites.
[Resending as it doesn't seem to have gotten through the first time.]
We need HTML versioning information needed in skins to be able to cleanly make incompatible CSS changes.
In the Wikimedia setup generated page HTML is cached for ~30 days regardless of skin HTML changes, while CSS&JS is purged at most a few minutes after a change is deployed.
This is a highly suboptimal situation, as this means that in every change that modifies both the HTML and the CSS, the CSS must be backwards-compatible with old HTML. This requires a lot of care and additional awkward testing and causes major issues when not done carefully enough (e.g. bug 42452), and sometimes just isn't possible at all unless some transitional hacks are inserted (e.g. bug 46947).
Luckily this isn't an issue for most third-party wikis, as `php update.php` after upgrade purges the cache entirely.
----
I'm proposing adding another class to <body>, "skintimestamp-YYYYMMDD", where YYYYMMDD is the year, month and day of the time that given skin's HTML was last modified in an incompatible way. Day should be enough granularity to avoid conflicts while keeping the class name short enough.
This can be easily done using the addToBodyAttributes() method of the Skin class. The timestamp would be updated manually by whoever is making those changes, and the class could be used in the CSS to only apply new styles to newly generated HTML. Older styles could be simply left intact, and then removed after enough time has passed.
If multiple incompatible changes are ever done in overlapping time periods, the successive ones would include updates to the "old new" styles to use both the new and old class.
Thoughts?
[Bug for this is https://bugzilla.wikimedia.org/show_bug.cgi?id=46956 , but let's keep the discussion here, please.]
On 04/07/2013 07:10 AM, Bartosz Dziewoński wrote:
tl;dr: Let's add class="skintimestamp-YYYYMMDD" to <body> in all skins, so we can make HTML+CSS changes without breaking the sites.
[Resending as it doesn't seem to have gotten through the first time.]
We need HTML versioning information needed in skins to be able to cleanly make incompatible CSS changes.
In the Wikimedia setup generated page HTML is cached for ~30 days regardless of skin HTML changes, while CSS&JS is purged at most a few minutes after a change is deployed.
This is a highly suboptimal situation, as this means that in every change that modifies both the HTML and the CSS, the CSS must be backwards-compatible with old HTML. This requires a lot of care and additional awkward testing and causes major issues when not done carefully enough (e.g. bug 42452), and sometimes just isn't possible at all unless some transitional hacks are inserted (e.g. bug 46947).
Luckily this isn't an issue for most third-party wikis, as `php update.php` after upgrade purges the cache entirely.
I'm proposing adding another class to <body>, "skintimestamp-YYYYMMDD", where YYYYMMDD is the year, month and day of the time that given skin's HTML was last modified in an incompatible way. Day should be enough granularity to avoid conflicts while keeping the class name short enough.
This can be easily done using the addToBodyAttributes() method of the Skin class. The timestamp would be updated manually by whoever is making those changes, and the class could be used in the CSS to only apply new styles to newly generated HTML. Older styles could be simply left intact, and then removed after enough time has passed.
If multiple incompatible changes are ever done in overlapping time periods, the successive ones would include updates to the "old new" styles to use both the new and old class.
Thoughts?
[Bug for this is https://bugzilla.wikimedia.org/show_bug.cgi?id=46956 , but let's keep the discussion here, please.]
Thanks, Bartosz, for clearly and collegially stating this problem and your proposed solution! The problem sounds like a real problem. I don't see any particular blockers in your proposed solution, but I do want to watch out for one implementation detail. "The timestamp would be updated manually by whoever is making those changes" -- could we introduce some clever automation to propose an update every time a relevant file is touched or a particular automated test breaks? Otherwise we have to thoroughly educate authors and reviewers to always update this, which sounds tedious.
Thanks again.
(Sorry for late reply, I was traveling and only had scarce Internet connection.)
On Thu, 11 Apr 2013 23:25:06 +0200, Sumana Harihareswara sumanah@wikimedia.org wrote:
"The timestamp would be updated manually by whoever is making those changes" -- could we introduce some clever automation to propose an update every time a relevant file is touched or a particular automated test breaks? Otherwise we have to thoroughly educate authors and reviewers to always update this, which sounds tedious.
I'm afraid that not really. These issues aren't really detectable in any way other than just pushing the code to a test wiki and seeing if the layout looks good, and the transitive nature of the only makes it harder to automate the checking. Maybe the Selenium tests could be useful here, but I wouldn't hold my breath.
Also, there is approximately one person wanting to make such changes right now (myself), and approximately five other people willing to review any skin changes (all of them already knowledgeable about the matter or able to figure it out by themselves), so this isn't really an issue for the review process, either.
On Sun, 07 Apr 2013 04:10:23 -0700, Bartosz Dziewoński matma.rex@gmail.com wrote:
Thoughts?
This sounds like a really ugly and hacky way to handle this issue. This is something that's mostly a WMF issue but this solution would involve all general updates being accompanied by a hack just for WMF. It also wouldn't scale well to extensions. I also have a felling that it would fall apart quickly if we make say 3 or more incompatible css changes within a month. We'd have pages with three different timestamps. And the conflicts would re-appear on some pages.
It sounds more like an issue with how smart ResourceLoader is and our current concept of what should and shouldn't be purged.
We already have other issues with how smart RL is. Or rather we have issues with the fact that on the client side RL is smart about handling dependencies and everything. But on the server side RL is completely stupid. All it does is serve precisely what it's asked to serve out. Then leaves absolutely everything else up to the client side scripting. There is absolutely no server side handling of dependencies. This leads skins that can't require JS to serve out their critical stylesheets having no concept of dependencies (which is killing my ability to fix my mistake with things like commonElement.css and turn them into modules and implement remoteSkinPath support), hacks like addModuleStyles which is buggy and bypasses parts of RL, and bugs like css being double loaded because it was added server side with addModuleStyles and then a script loaded a second copy.
One of the issues in general seems to be that we need to make ResourceLoader smarter. It needs to understand dependencies on the server side, give more information to the client about what it's already loaded to avoid double loading, and give it some level of understanding of versioning on the server side.
Right now our stylesheets loaded by the server contain no versioning timestamp (for cache breaking). Our HTML is cached for up to 30 days. And we try to clear css asap. From what I understand my guess is that we don't include the timestamp in the url because of how long the HTML is cached. In order to use the cache breaking timestamp the HTML of every page would have to be purged in order for the cache breaker to work. Since on old pages the cache wouldn't be broken and the client would use old css for some pages. So we use non-versioned CSS without long expires so we can purge that separately and have everyone use the css updates.
But from the issue we're having with incompatible css and html we might be wrong about that. Rather than trying to keep the css up to date all the time perhaps it's actually correct to serve old css on pages with old html instead of serving the up to date css. Even if it means that pages will be a little inconsistent with modifications to the UI. Some pages with old stuff and others with new.
For that we would start including timestamps for all css. And we'd make it so that the old css stays around at those urls so that on old pages the old css is loaded. To do that we could increase the time the server caches timestamped css. But more flexibly we might consider making RL smart about how it handles those timestamps. Do something smart with them instead of making them simple cache breakers/holders. Write a maintenance script; When you run it it creates a new snapshot containing the css of all the RL modules. The script could also be used to clear out old snapshots. RL would then be smart about serving things out of load.php. load.php would use the timestamp to serve out old css that was cached when you ran the script. That way if we make some incompatible css changes then we run the script before deployment. RL creates a snapshot. And after the deployment RL keeps serving the old snapshotted css for the old timestamps.
(Sorry for late reply, I was traveling and only had scarce Internet connection.)
On Fri, 12 Apr 2013 11:21:27 +0200, Daniel Friesen daniel@nadir-seen-fire.com wrote:
This sounds like a really ugly and hacky way to handle this issue.
A little, sadly. But it will *work*, unlike what we have right now.
This is something that's mostly a WMF issue but this solution would involve all general updates being accompanied by a hack just for WMF.
It would be trivial to remove these five lines or so in release versions. (Assuming, of course, we do want to remove them, as they won't be causing any trouble.)
It also wouldn't scale well to extensions.
There is already a hook for this that extensions are free to use. (Although I don't know of any that does.)
I also have a felling that it would fall apart quickly if we make say 3 or more incompatible css changes within a month. We'd have pages with three different timestamps. And the conflicts would re-appear on some pages.
Yes, it wouldn't be pretty, but you're exaggerating. The latter changes would just have to be made more carefully. It would still be better than status quo.
It sounds more like an issue with how smart ResourceLoader is and our current concept of what should and shouldn't be purged.
(... wall of text ...)
But from the issue we're having with incompatible css and html we might be wrong about that. Rather than trying to keep the css up to date all the time perhaps it's actually correct to serve old css on pages with old html instead of serving the up to date css. Even if it means that pages will be a little inconsistent with modifications to the UI. Some pages with old stuff and others with new.
For that we would start including timestamps for all css. And we'd make it so that the old css stays around at those urls so that on old pages the old css is loaded. To do that we could increase the time the server caches timestamped css. (...)
This is all true. It also seems quite hard to implement - and thus unlikely to be implemented - both on the MW core side and on the WMF engineering side. If you want to, feel free to; but please don't block my practical - while slightly hacky - solutions in favor of vaporware. I'm trying to make the skins better *right now*, and what I'm proposing would make it way easier than it is right now, both for the code author and for the reviewers.
I could implement what I'm proposing even more hackily in every incompatible change, sure, but this would be much nicer.
wikitech-l@lists.wikimedia.org