In the course of examining Opera Mini bugs, I've been talking with Jared
and Maryana and others about different types of browser feature support,
and I needed to run some queries to get a handle on the facts. Results
follow.
Prologue: I ran some queries to gauge <noscript> (no JavaScript support at
all) usage, versus usage of browsers with low-end JavaScript
("ResourceLoader-impaired", or for short, "RLI") precluding them from
things like mobile editing, on the mobile web Wikipedias. I thought I'd
share this information with you.
Caveat Emptor: I am not a Ph.D in maths/stats. And I haven't had the
queries peer reviewed. But I've tried to analyze each OR case for
User-Agent in the queries below to ensure that I actually see results that
are sensible.
Results:
On zero-rated networks accessing mobile web Wikipedia:* About 1.8% of
Wikipedia Zero banners served are to <noscript> browsers. The other 98.2%
of banners are served to browsers with at least some form of JavaScript
support.* About 36% of Wikipedia Zero pageviews (using a dumbed down
definition) are on RLI browsers.
On non-zero-rated networks accessing mobile web Wikipedia:* The percentage
of users on <noscript> browsers is unknown (AFAIK) but if we implemented a
<noscript> tracking pixel akin to the GIF banner on Wikipedia Zero, I
suspect we'd find it's 1.8% or less.* About 6% of overall Wikipedia mobile
web pageviews (using a dumbed down definition) are on RLI browsers.
Even gorier details. SQL/HQL at the end:
1. When browsers request GIF Wikipedia Zero banners, the URL has
zcmd=img-banner in it. When they request JavaScript banners, the URL has
zcmd=js-banner in it. Both banners have a cache lifetime of 300 seconds, so
it's roughly apples to apples when we look at their usage.
2. The simplest definition of a pageview I've found useful is to look at
HTTP 200 (non-cached) HTML responses and JSON responses for well defined
URL paths.
3. Our ResourceLoader bootstrapping code at
https://git.wikimedia.org/blob/mediawiki%2Fcore.git/09c55592a73544cc21e002f…
, which puts things like Opera Mini in the RLI bucket, also puts Google
Glass in the RLI bucket. That doesn't mean Google Glass is incapable of
running high end JavaScript, it just reflects that the screen is really
small. Just wanted to point that out. Expansion into wearables and IoT will
be a theme I'm sure in due time.
4. The other approximately 62% or approximately 92% of user agents for
zero-rated networks and non-zero-rated networks, respectively, in the
non-RLI buckets may in fact lack good JavaScript facilities and we just
don't know any better (actually, we know for a fact at least some have bad
JS support), but these are rough figures, after all.
5. Some of the traffic is definitely from bots and the like, but I didn't
want to overcomplicate these queries any further. On the upside, In the
overall scheme of non-zero-rated mobile web Wikipedia access, bot traffic
such as GoogleBot kind of gets washed out because of the large number of
pageviews. Bot traffic on Wikipedia Zero networks seems to be fairly low
grade at the moment.
6. The queries. These queries could be optimized, but I erred on the side
of clarity, if you can call it that, for people familiar with SQL/HQL. If
we were to explore this more, I might also be interested in tuning the
non-Opera Mini browsers that are still Opera but are at too low of a
version; these non-OM Opera browsers might be slightly undercounted. It may
be the case that the particular two OR cases for the existing queries is
getting those particular rows with sufficient accuracy as is.
Single day requests for GIF Banners on zero-rated networks:74,807 (1.8% of
total)
select count(*)from webrequestwhere uri_host like '%.wikipedia.org'and
uri_query like '%zcmd=img-banner'and year = 2014 and month = 11 and day =
19and webrequest_source = 'mobile'and x_analytics like '%zero=%';
Single day requests for JavaScript Banners on zero-rated networks:4,100,385
(98.2% of total)
select count(*)from webrequestwhere uri_host like '%.wikipedia.org'and
uri_query like '%zcmd=js-banner'and year = 2014 and month = 11 and day =
19and webrequest_source = 'mobile'and x_analytics like '%zero=%';
Single day zero-rated "dumbed down" pageviews on Wikipedia mobile web,
expressly ResourceLoader-impaired UAs:1,099,607 (36% of total)
select count(*)from webrequestwhere year = 2014and month = 11and day =
19and webrequest_source = 'mobile'and uri_host like '%.wikipedia.org'and
http_status = '200' and(content_type like '%text/html%' or content_type
like '%application/json%')and (uri_path like '/wiki/%' or uri_path like
'/w/index.php' or(uri_path like '/w/api.php%' and uri_query like
'%action=mobileview%'))and (coalesce(regexp_extract(user_agent, '(MSIE
*[0-7][^\\d])', 1), '') != ''or coalesce(regexp_extract(user_agent,
'(Firefox\\/ *[0-2][^\\d])', 1), '') != ''or(user_agent like
'%Opera/%'and
user_agent not like '%Version/%'and coalesce(regexp_extract(user_agent,
'(Opera\\/[0-9][^\\d])', 1), '') != '')or(user_agent like
'%Opera/%'and
user_agent like '%Version/%'and coalesce(regexp_extract(user_agent,
'(Version\\/(11|10|[0-9])[^\\d])', 1), '') != '')or
coalesce(regexp_extract(user_agent, '( Opera *[0-9][^\\d])', 1), '') !=
''or coalesce(regexp_extract(user_agent,
'(BlackBerry[^\\/]*\\/[1-5]\\.)',
1), '') != ''or coalesce(regexp_extract(user_agent,
'(webOS\\/1\\.[0-4])',
1), '') != ''or lower(user_agent) like '%playstation%'or
user_agent like
'%SymbianOS%'or user_agent like '%Series60%'or user_agent like
'%NetFront%'or user_agent like '%Opera Mini%'or user_agent like
'%S40OviBrowser%'or (user_agent like '%Glass%' and user_agent like
'%Android%'))and x_analytics like '%zero=%';
Single day zero-rated "dumbed down" pageviews on Wikipedia mobile web, all
UAs:3,047,700 (100% of total)
select count(*)from webrequestwhere year = 2014and month = 11and day =
19and webrequest_source = 'mobile'and uri_host like '%.wikipedia.org'and
http_status = '200' and(content_type like '%text/html%' or content_type
like '%application/json%')and (uri_path like '/wiki/%' or uri_path like
'/w/index.php' or(uri_path like '/w/api.php%' and uri_query like
'%action=mobileview%'))and x_analytics like '%zero=%';
Single day "dumbed down" pageviews on Wikipedia mobile web (includes both
zero-rated and non-zero-rated), expressly ResourceLoader-impaired
UAs:12,152,389 (6% of total)
select count(*)from webrequestwhere year = 2014and month = 11and day =
19and webrequest_source = 'mobile'and uri_host like '%.wikipedia.org'and
http_status = '200' and(content_type like '%text/html%' or content_type
like '%application/json%')and (uri_path like '/wiki/%' or uri_path like
'/w/index.php' or(uri_path like '/w/api.php%' and uri_query like
'%action=mobileview%'))and (coalesce(regexp_extract(user_agent, '(MSIE
*[0-7][^\\d])', 1), '') != ''or coalesce(regexp_extract(user_agent,
'(Firefox\\/ *[0-2][^\\d])', 1), '') != ''or(user_agent like
'%Opera/%'and
user_agent not like '%Version/%'and coalesce(regexp_extract(user_agent,
'(Opera\\/[0-9][^\\d])', 1), '') != '')or(user_agent like
'%Opera/%'and
user_agent like '%Version/%'and coalesce(regexp_extract(user_agent,
'(Version\\/(11|10|[0-9])[^\\d])', 1), '') != '')or
coalesce(regexp_extract(user_agent, '( Opera *[0-9][^\\d])', 1), '') !=
''or coalesce(regexp_extract(user_agent,
'(BlackBerry[^\\/]*\\/[1-5]\\.)',
1), '') != ''or coalesce(regexp_extract(user_agent,
'(webOS\\/1\\.[0-4])',
1), '') != ''or lower(user_agent) like '%playstation%'or
user_agent like
'%SymbianOS%'or user_agent like '%Series60%'or user_agent like
'%NetFront%'or user_agent like '%Opera Mini%'or user_agent like
'%S40OviBrowser%'or (user_agent like '%Glass%' and user_agent like
'%Android%'));
Single day "dumbed down" pageviews on Wikipedia mobile web (includes both
zero-rated and non-zero-rated), all UAs:199,764,072 (100% of total)
select count(*)from webrequestwhere year = 2014and month = 11and day =
19and webrequest_source = 'mobile'and uri_host like '%.wikipedia.org'and
http_status = '200' and(content_type like '%text/html%' or content_type
like '%application/json%')and (uri_path like '/wiki/%' or uri_path like
'/w/index.php' or(uri_path like '/w/api.php%' and uri_query like
'%action=mobileview%'));