It occurred to me that we can't avoid server-based rewriting of <img> tags
on subdomains of
, meaning we *probably* would need to
have two versions, not one, of the Wikipedia Zero HTML: one for
subdomains. But I
still believe we can avoid the dozens or more versions of HTML per language
article as is currently the side effect of Wikipedia Zero HTML delivery.
The only way I can think of getting back to the ideal of one version of
Wikipedia Zero HTML per language article is to assume that users on
would *always* have JavaScript turned on and we could
rewrite the DOM client-side to change interstitial hyperlinks for images
into actual images. But I'm not comfortable making that assumption yet, and
wouldn't want those users to have to click twice to see an image when they
could have seen the thumbnail in the article in the first place.
I updated the RFC to reflect the <img> tag thought.
On Mon, Jun 10, 2013 at 5:09 PM, Adam Baso <abaso(a)wikimedia.org> wrote:
I've posted a basic plan on ESI and JavaScript
stuff for reduction of
Varnish cache fragmentation.
https://www.mediawiki.org/wiki/Requests_for_comment/Zero_Architecture#Cache…
Comments welcome at
https://www.mediawiki.org/wiki/Talk:Requests_for_comment/Zero_Architecturea… on this
thread.
By the way, thanks to a couple of nice people, I arrived at the following
for individual and digest emails in Gmail for the same mailing list: (1)
receive the individual wikitech-l emails at my main email address, and
filter them to a folder for later retrieval, then (2) sign up for the
digest with an email address of the format <username>+
arbitrarytext(a)wikimedia.org, as that's a legal alias for the same email
address. So I get the digest email in my inbox, and can go to the folder
with the individual messages if I want to reply to the latest message on
the thread easily.
Here's a copy-and-paste of the wikitext in the RFC section regarding this
cache and JavaScript stuff:
== Cache fragmentation improvement ==
''See the Wikitech-l emails with '''ZERO Architecture'''
in the subject
line at [
http://lists.wikimedia.org/pipermail/wikitech-l/2013-May/thread.html] and
[
http://lists.wikimedia.org/pipermail/wikitech-l/2013-June/thread.html]
for more background.''
The Wikipedia Zero (Partners) software engineering team believes it can
achieve performance and usability enhancements for the Wikipedia Zero
experience with Edge Side Includes (ESI) and JavaScript DOM manipulation.
=== The problem ===
Wikipedia Zero webpages are served to users on mobile devices with
participating mobile carriers. The number of Wikipedia Zero cached pages,
in excess of non-Wikipedia Zero mobile-formatted pages, is roughly:
<pre>cached_pages = 0
foreach carrier c:
cached_pages += c.one_or_two_subdomains_from_m_or_zero_subdomains *
c.num_languages_supported</pre>
Carriers support Zero-rating of <language>.zero.wikipedia.org, <language>.
m.wikipedia.org, or both. They also support up to ten customized free
languages, otherwise they support all languages.
The amplification of cached pages means more hits at the origin servers
than wanted, meaning slower loading pages for Wikipedia Zero users.
Furthermore, the current page caching scheme employed via Wikipedia Zero
introduces a challenge to differentiating such Wikipedia Zero cached pages
from non-Wikipedia Zero cached pages - a problem when the Wikipedia Zero
team wants to purge the cache to modify aspects of the Wikipedia Zero
experience without impacting other aspects of the Wikipedia
mobile-formatted experience.
=== Detail ===
==== Coarse grained page elements ====
Wikipedia Zero pages are visually composed of several elements. For
simplicity, here's a coarse grained breakdown excluding general
mobile-formatted navigational and other JavaScript or CSS pieces:
Special:ZeroRatedMobileAccess:
* Partner banners
* Listed high-priority languages ("showLangs" and
"langNameOverrides")
* Language dropdown list composed of hyperlinks that differ based on
Zero-rated supported languages ("whitelistedLangs")
Articles:
* Partner banners
* Article body
* Read in Another Language section
* General purpose footer (Legal, Privacy, and so on)
Partner banners are different depending on language (the language code in
the domain name dictates the site which yields the language code) and the
carrier (X-CS header).
Listed high-priority languages are different depending on the carrier
(X-CS header yields "showLangs" and "langNameOverrides") and the
subdomain
(X-Subdomain of M or ZERO).
Language dropdown lists are different depending on the carrier (X-CS
header yields "whitelistedLangs") and the subdomain (X-Subdomain of M or
ZERO).
Article bodies are different depending on the URL (the entire URL, which
includes the language code in the domain name), the carrier (X-CS header
yields "whitelistedLangs"), and the subdomain (X-Subdomain of M or ZERO).
Read in Another Language sections are different depending on the URL (the
entire URL, which includes the language code in the domain name), the
carrier (X-CS header yields "whitelistedLangs"), and the subdomain
(X-Subdomain of M or ZERO).
General purpose footers (Legal, Privacy, and so on) are different
depending on language (specifically, the language code in the domain name).
==== Cache duration ====
Currently, banners and footers can in effect end up being cached for up to
31 days on articles that have not been flushed from the Varnish cache. This
is good for content that never changes, but is also inconvenient when
things like banner wording actually need to be updated.
We don't see this as a problem if we can make cache flushes less painful.
==== Simplest to hardest ====
We recommend starting with the simplest possible conversion to ESI, and
then, depending on the learning, converting more and more stuff to ESI in
order of ascending complexity and risk. In the case of the actual articles,
ESI may not be a good option, at least not yet, but use of a central
redirector controller and JavaScript may serve as an equally good option.
Footers vary on the language alone. An ESI fragment of the following form
would be easiest: <nowiki>http://<language>.
wikipedia.org/wiki/Special:ZeroRatedMobileAccess?generatefooter=please</…
(or if it is easier due to existing hook architecture, <nowiki>http://
<language>.
m.wikipedia.org/wiki/Special:ZeroRatedMobileAccess?generatefooter=please<…)i>).
The backing code would only need to identify the language of the request in
order to generate a suitable footer.
===== Footers =====
Footers can retain their 31 day cache lifetime. In the case that a footer
needs to be flushed from cache, it will be easiest to issue a cache purge
for all objects matching
/wiki/Special:ZeroRatedMobileAccess?generatefooter=please and let the
banners naturally be re-populated as inbound requests are made.
===== Language dropdown lists =====
Language dropdown lists vary on X-Subdomain and X-CS.
Special:ZeroRatedMobileAccess will already have access to both headers, and
therefore all information necessary for the ESI fragment is already present.
<nowiki>
http://en.wikipedia.org/wiki/Special:ZeroRatedMobileAccess?generatelangdrop…
</nowiki>
Headers...
X-CS: <carrier>
X-Subdomain: <M or ZERO>
Dropdowns can retain their 31 day cache lifetime. In the case that a
dropdown needs to be flushed from cache, it will be easiest to issue a
cache purge for all objects matching
/wiki/Special:ZeroRatedMobileAccess?generatelangdropdown=please and let the
dropdown footers naturally be re-populated as inbound requests are made.
As you'll see below, we may be able to actually simplify to one dropdown
list, but this may still be a good place for use of ESI.
===== Listed high priority languages =====
Listed high priority languages are similar to language dropdowns:
<nowiki>
http://en.wikipedia.org/wiki/Special:ZeroRatedMobileAccess?generateshowedla…
</nowiki>
Headers...
X-CS: <carrier>
X-Subdomain: <M or ZERO>
Listed high priority languages can retain their 31 day cache lifetime. In
the case that a high priority languages listing needs to be flushed from
cache, it will be easiest to issue a cache purge for all objects matching
/wiki/Special:ZeroRatedMobileAccess?generateshowedlangs=please and let the
high priority languages listsings naturally be re-populated as inbound
requests are made.
===== Partner banners =====
Partner banners vary on language and X-CS. Banners should be requested in
an ESI fragment with a URL that includes the language code and which
includes an X-CS header.
ZeroRatedMobileAccess-generated pages already have access to the X-CS
header, and the <language> will by definition also be known because of the
enclosing page, and therefore all information necessary for the ESI
fragment is already present.
<nowiki>http://<language>.
wikipedia.org/wiki/Special:ZeroRatedMobileAccess?generatebanner=please
</nowiki>
Header...
X-CS: <carrier>
Banners can retain their 31 day cache lifetime. In the case that a banner
needs to be flushed from cache, it will be easiest to issue a cache purge
for all objects matching
/wiki/Special:ZeroRatedMobileAccess?generatebanner=please and let the
banners naturally be re-populated as inbound requests are made.
===== Articles and Read in Another Language Sections =====
Currently, articles and their corresponding Read in Another Language
sections are rewritten to support users upgrading their experience to (1)
be able to see an image while browsing on a
zero.wikipedia.org domain,
(2) switch from the text and icons-only view of a zero.wikipedia.orgsubdomain to the
image-rich
m.wikipedia.org subdomain experience (which may or may not be zero-rated
at a carrier), and (3) to visit external sites.
The rules for the server's rewriting of <nowiki><img></nowiki>s
and
hyperlinks in articles and Read in Another Language sections are based upon
whether the carrier supports
zero.wikipedia.org,
m.wikipedia.org, or
both, as well as the array of languages supported freely by the carrier. We
believe we can reduce the cache fragmentation by transitioning to one
canonical mobile article (and Read in Another Language section) per
language, relying upon JavaScript to rewrite hyperlinks for upgrade
transitions. But this does not address non-JavaScript, insufficient
JavaScript, or JavaScript-disabled browsing and the corresponding cache
fragmentation that would still be present with the current server code.
We believe the simplest solution to reducing cache fragmentation for
article and Read in Another Language content is to convert all article and
Read in Another Language links that do not point directly to a same-origin
article (specifically, hyperlinks that are non-redirects are strictly
relative paths) served to Wikipedia Zero-participating UAs to one
consistent redirect URL.
For example, instead of a relative link to
?renderZeroRatedRedirect=true&returnto=http%3A%2F%2Fwww.mlb.com%2F to
visit an external site, such a hyperlink would instead be set as
/wiki/Special:ZeroRatedMobileAccess?redirect=http%3A%2F%2Fwww.mlb.com%2F.
And a relative link such as
"/w/index.php?title=Major_League_Baseball&renderZeroRatedBanner=true&renderZeroRatedRedirect=true&returnto=%2F%
2Fda.m.wikipedia.org%2Fwiki%2FMajor_League_Baseball would instead be
written as /wiki/Special:ZeroRatedMobileAccess?redirect=%2F%
2Fda.m.wikipedia.org%2Fwiki%2FMajor_League_Baseball. The redirect
endpoint would examine the content of the '''redirect'''
parameter, and
depending on carrier support would render an appropriate dialogue for
upgrading (or for links accessed outside of Wikipedia Zero networks, simply
continuing) or immediately redirect the user to the same-origin resource.
In case of the need to flush the redirect cache for Wikipedia Zero,
requests of the form /wiki/Special:ZeroRatedMobileAccess?redirect could be
flushed, and the cache could be repopulated as clicks are accumulated.
Although the total number of cached objects could remain the same, the
ability to flush the cache would be dramatically improved and the size of
those cached redirect objects would be slight compared to full pages.
For user agents that support JavaScript-driven link rewriting, such user
agents could rewrite such URLs at page runtime by:
# Making a request to an endpoint to obtain JSON data providing carrier
preferences on free languages, M versus ZERO support, and so forth. This
endpoint could be something to the effect of <nowiki>
http://en.wikipedia.org/wiki/Special:ZeroRatedMobileAccess?generateprefs=pl…ki>.
The X-CS header augmentation from Varnish should be present for such
requests, making the lookup straightforward.
# Using the received preferences to walk the DOM and rewrite hyperlinks
and set onClick actions for appropriate dialogs.
# Taking this concept to its logical extreme, it's actually conceivable
that for UAs sufficiently supporting JavaScript, the article and Read in
Another Language page content could actually be identical to the
non-Wikipedia Zero mobile-formatted experience, further reducing the
mobile-formatted cache. This said, the extra work involved in trying to
shoehorn Wikipedia Zero for JS UAs into the exact same experience as
general mobile users may be a later stage change involving more complicated
use of ESI fragments. And to be clear, it doesn't make sense to try to
shoehorn all of the mobile-formatted experience into Wikipedia Zero (i.e.,
hyperlinks should not have redirect parameters in them outside of the
Wikipedia Zero experience).
Hyperlinks on the same domain don't require redirect behavior through URL
rewriting. By standardizing all hyperlinks, the cache size can be shrunk
down to one object per article in a given language on the Wikipedia Zero
experience.
==== Order of work and dependencies ====
The ESI fragment generation code and the server-based redirector endpoint
code (/wiki/Special:ZeroRatedMobileAccess?redirect=<path> can be worked on
concurrently.
JavaScript rewriting of hyperlinks depends upon (1) the comprehensive
change to redirect hyperlinks in HTML and (2) a well-functioning redirector
endpoint.
On Fri, May 31, 2013 at 11:25 AM, Adam Baso <abaso(a)wikimedia.org> wrote:
Sorry to reply on a thread that will probably not
sort nicely on the
mailman web interface or threading mail clients. Anybody know of an
easy way to reply to digest email in Gmail such that mailman will
retain threading?
I'll be working on the conversion of Wikipedia Zero banners to ESI.
There will be good lessons here I think for looking at dynamic loading
in other parts of the interface.
Arthur, to address your questions in the parent to Mark's reply:
"is there any reason you'd need serve different HTML than what is
already being served by MobileFrontend?"
Currently, some languages are not whitelisted at carriers, meaning
that users may get billed when they hit <language>.m.wikipedia.org or
<language>.zero.wikipedia.org. Thus, a number of <a href>s are
rewritten to include interstitials if the link is going to result in a
charge. By the way, we see some shortcomings in the existing rewrites
that need to be corrected (e.g., some URLs don't have interstitials,
but should), but that's a separate bug.
My thinking is that we start intercepting all clicks by JavaScript if
it's a Javascripty browser, or via a default interceptor at the URL on
that <language>.(m|zero).wikipedia.org's language's corresponding
subdomain otherwise. In either case, if the destination link is on a
non-whitelisted Wikipedia domain for the carrier or if the link is
external of Wikipedia, the user should land at an interstitial page
hosted on the same whitelisted subdomain from whence the user came.
"Out of curiosity, is there WAP support in Zero? I noticed some
comments like '# WAP' in the varnish acls for Zero, so I presume so.
Is the Zero WAP experience different than the MobileFrontend WAP
experience?"
No special WAP considerations in ZeroRatedMobileAccess above and
beyond MobileFrontend, as I recall. The "# WAP" comments is just for
us to remember in case a support case comes up with that particular
carrier.
We'll want to keep in mind impacts for USSD/SMS support. I think
Jeremy had some good conversations at the Wikimedia Hackathon in
Amsterdam that will help him to refine how his middleware receives and
transforms content.
Mark Bergsma mark at
wikimedia.org
Fri May 31 09:44:48 UTC 2013
Previous message: [Wikitech-l] ZERO architecture
Next message: [Wikitech-l] ZERO architecture
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
________________________________
* feature
phones -- HTML only, the banner is inserted by the ESI
** for carriers with free images
** for carriers without free images
What about including ESI tags for banners for smart devices as well as
feature phones, then either use ESI to insert the banner for both device
types or, alternatively, for smart devices don't let Varnish populate
the
ESI chunk and instead use JS to replace the ESI
tags with the banner?
That
way we can still serve the same HTML for smart
phones and feature phones
with images (one less thing for which to vary the cache).
I think the verdict is still out on whether it's better to use ESI for
Banners in Varnish or use JS for that client-side. I guess we'll have
to test and see.
Are there carrier-specific things that would
result in different HTML
for
devices that do not support JS, or can you get
away with providing the
same
non-js experience for Zero as MobileFrontend
(aside from the
banner, presumably handled by ESI)? If not currently, do you think its
feasible to do that (eg make carrier-variable links get handled via
special
pages so we can always rely on the same URIs)?
Again, it would be nice
if
we could just rely on the same HTML to further
reduce cache variance. It
would be cool if MobileFrontend and Zero shared buckets and they were
limited to:
* HTML + images
* HTML - images
* WAP
That would be nice.
Since we improved MobileFrontend to no longer
vary the cache on
X-Device,
I've been surprised to not see a significant
increase in our cache hit
ratio (which warrants further investigation but that's another email).
Are
there ways we can do a deeper analysis of the
state of the varnish
cache to
determine just how fragmented it is, why, and how
much of a problem it
actually is? I believe I've asked this before and was met with a
response
of 'not really' - but maybe things have
changed now, or others on this
list
have different insight. I think we've mostly
approached the issue with a
lot more assumption than informed analysis, and if possible I think it
would be good to change that.
Yeah, we should look into that. We've already flagged a few possible
culprits, and we're also working on the migration of the desktop wiki
cluster from Squid to Varnish, which has some of the same issues with
variance (sessions, XVO, cookies, Accept-Language...) as
MobileFrontend does. After we've finished migrating that and confirmed
that it's working well, we want to unify those clusters'
configurations a bit more, and that by itself should give us
additional opportunity to compare some strategies there.
We've since also figured out that the way we've calculate cache
efficiency with Varnish is not exactly ideal; unlike Squid, cache
purges are done as HTTP requests to Varnish. Therefore in Varnish,
those cache lookups are calculated into the cache hit rate, which
isn't very helpful. To make things worse, the few hundreds of purges a
second vs actual client traffic matter a lot more on the mobile
cluster (with much less traffic but a big content set) than it does
for our other clusters. So until we can factor that out in the Varnish
counters (might be possible in Varnish 4.0), we'll have to look at
other metrics.
More useful therefore is to check the actual backend fetches
("backend_req"), and these appear to have gone down some. Annoyingly,
every time we restart a Varnish instance we get a spike in the Ganglia
graphs, making the long-term graphs pretty much unusable. To fix that
we'll either need to patch Ganglia itself or move to some other stats
engine (statsd?). So we have a bit of work to do there on the Ops
front.
Note that we're about to replace all Varnish caches in eqiad by
(fewer) newer, much bigger boxes, and we've decided to also upgrade
the 4 mobile boxes with those same specs. And we're also doing that in
our new west coast caching data center as well as esams. This will
increase the mobile cache size a lot, and will hopefully help by
throwing resources at the problem.
--
Mark Bergsma <mark at wikimedia.org>
Lead Operations Architect
Wikimedia Foundation