In the course of examining Opera Mini bugs, I've been talking with Jared and Maryana and others about different types of browser feature support, and I needed to run some queries to get a handle on the facts. Results follow.
Prologue: I ran some queries to gauge <noscript> (no JavaScript support at all) usage, versus usage of browsers with low-end JavaScript ("ResourceLoader-impaired", or for short, "RLI") precluding them from things like mobile editing, on the mobile web Wikipedias. I thought I'd share this information with you.
Caveat Emptor: I am not a Ph.D in maths/stats. And I haven't had the queries peer reviewed. But I've tried to analyze each OR case for User-Agent in the queries below to ensure that I actually see results that are sensible.
Results:
On zero-rated networks accessing mobile web Wikipedia:* About 1.8% of Wikipedia Zero banners served are to <noscript> browsers. The other 98.2% of banners are served to browsers with at least some form of JavaScript support.* About 36% of Wikipedia Zero pageviews (using a dumbed down definition) are on RLI browsers.
On non-zero-rated networks accessing mobile web Wikipedia:* The percentage of users on <noscript> browsers is unknown (AFAIK) but if we implemented a <noscript> tracking pixel akin to the GIF banner on Wikipedia Zero, I suspect we'd find it's 1.8% or less.* About 6% of overall Wikipedia mobile web pageviews (using a dumbed down definition) are on RLI browsers.
Even gorier details. SQL/HQL at the end:
1. When browsers request GIF Wikipedia Zero banners, the URL has zcmd=img-banner in it. When they request JavaScript banners, the URL has zcmd=js-banner in it. Both banners have a cache lifetime of 300 seconds, so it's roughly apples to apples when we look at their usage.
2. The simplest definition of a pageview I've found useful is to look at HTTP 200 (non-cached) HTML responses and JSON responses for well defined URL paths.
3. Our ResourceLoader bootstrapping code at https://git.wikimedia.org/blob/mediawiki%2Fcore.git/09c55592a73544cc21e002f229686d17806686ba/resources%2Fsrc%2Fstartup.js#L40 , which puts things like Opera Mini in the RLI bucket, also puts Google Glass in the RLI bucket. That doesn't mean Google Glass is incapable of running high end JavaScript, it just reflects that the screen is really small. Just wanted to point that out. Expansion into wearables and IoT will be a theme I'm sure in due time.
4. The other approximately 62% or approximately 92% of user agents for zero-rated networks and non-zero-rated networks, respectively, in the non-RLI buckets may in fact lack good JavaScript facilities and we just don't know any better (actually, we know for a fact at least some have bad JS support), but these are rough figures, after all.
5. Some of the traffic is definitely from bots and the like, but I didn't want to overcomplicate these queries any further. On the upside, In the overall scheme of non-zero-rated mobile web Wikipedia access, bot traffic such as GoogleBot kind of gets washed out because of the large number of pageviews. Bot traffic on Wikipedia Zero networks seems to be fairly low grade at the moment.
6. The queries. These queries could be optimized, but I erred on the side of clarity, if you can call it that, for people familiar with SQL/HQL. If we were to explore this more, I might also be interested in tuning the non-Opera Mini browsers that are still Opera but are at too low of a version; these non-OM Opera browsers might be slightly undercounted. It may be the case that the particular two OR cases for the existing queries is getting those particular rows with sufficient accuracy as is.
Single day requests for GIF Banners on zero-rated networks:74,807 (1.8% of total)
select count(*)from webrequestwhere uri_host like '%.wikipedia.org'and uri_query like '%zcmd=img-banner'and year = 2014 and month = 11 and day = 19and webrequest_source = 'mobile'and x_analytics like '%zero=%';
Single day requests for JavaScript Banners on zero-rated networks:4,100,385 (98.2% of total)
select count(*)from webrequestwhere uri_host like '%.wikipedia.org'and uri_query like '%zcmd=js-banner'and year = 2014 and month = 11 and day = 19and webrequest_source = 'mobile'and x_analytics like '%zero=%';
Single day zero-rated "dumbed down" pageviews on Wikipedia mobile web, expressly ResourceLoader-impaired UAs:1,099,607 (36% of total)
select count(*)from webrequestwhere year = 2014and month = 11and day = 19and webrequest_source = 'mobile'and uri_host like '%.wikipedia.org'and http_status = '200' and(content_type like '%text/html%' or content_type like '%application/json%')and (uri_path like '/wiki/%' or uri_path like '/w/index.php' or(uri_path like '/w/api.php%' and uri_query like '%action=mobileview%'))and (coalesce(regexp_extract(user_agent, '(MSIE *[0-7][^\\d])', 1), '') != ''or coalesce(regexp_extract(user_agent, '(Firefox\\/ *[0-2][^\\d])', 1), '') != ''or(user_agent like '%Opera/%'and user_agent not like '%Version/%'and coalesce(regexp_extract(user_agent, '(Opera\\/[0-9][^\\d])', 1), '') != '')or(user_agent like '%Opera/%'and user_agent like '%Version/%'and coalesce(regexp_extract(user_agent, '(Version\\/(11|10|[0-9])[^\\d])', 1), '') != '')or coalesce(regexp_extract(user_agent, '( Opera *[0-9][^\\d])', 1), '') != ''or coalesce(regexp_extract(user_agent, '(BlackBerry[^\\/]*\\/[1-5]\\.)', 1), '') != ''or coalesce(regexp_extract(user_agent, '(webOS\\/1\\.[0-4])', 1), '') != ''or lower(user_agent) like '%playstation%'or user_agent like '%SymbianOS%'or user_agent like '%Series60%'or user_agent like '%NetFront%'or user_agent like '%Opera Mini%'or user_agent like '%S40OviBrowser%'or (user_agent like '%Glass%' and user_agent like '%Android%'))and x_analytics like '%zero=%';
Single day zero-rated "dumbed down" pageviews on Wikipedia mobile web, all UAs:3,047,700 (100% of total)
select count(*)from webrequestwhere year = 2014and month = 11and day = 19and webrequest_source = 'mobile'and uri_host like '%.wikipedia.org'and http_status = '200' and(content_type like '%text/html%' or content_type like '%application/json%')and (uri_path like '/wiki/%' or uri_path like '/w/index.php' or(uri_path like '/w/api.php%' and uri_query like '%action=mobileview%'))and x_analytics like '%zero=%';
Single day "dumbed down" pageviews on Wikipedia mobile web (includes both zero-rated and non-zero-rated), expressly ResourceLoader-impaired UAs:12,152,389 (6% of total)
select count(*)from webrequestwhere year = 2014and month = 11and day = 19and webrequest_source = 'mobile'and uri_host like '%.wikipedia.org'and http_status = '200' and(content_type like '%text/html%' or content_type like '%application/json%')and (uri_path like '/wiki/%' or uri_path like '/w/index.php' or(uri_path like '/w/api.php%' and uri_query like '%action=mobileview%'))and (coalesce(regexp_extract(user_agent, '(MSIE *[0-7][^\\d])', 1), '') != ''or coalesce(regexp_extract(user_agent, '(Firefox\\/ *[0-2][^\\d])', 1), '') != ''or(user_agent like '%Opera/%'and user_agent not like '%Version/%'and coalesce(regexp_extract(user_agent, '(Opera\\/[0-9][^\\d])', 1), '') != '')or(user_agent like '%Opera/%'and user_agent like '%Version/%'and coalesce(regexp_extract(user_agent, '(Version\\/(11|10|[0-9])[^\\d])', 1), '') != '')or coalesce(regexp_extract(user_agent, '( Opera *[0-9][^\\d])', 1), '') != ''or coalesce(regexp_extract(user_agent, '(BlackBerry[^\\/]*\\/[1-5]\\.)', 1), '') != ''or coalesce(regexp_extract(user_agent, '(webOS\\/1\\.[0-4])', 1), '') != ''or lower(user_agent) like '%playstation%'or user_agent like '%SymbianOS%'or user_agent like '%Series60%'or user_agent like '%NetFront%'or user_agent like '%Opera Mini%'or user_agent like '%S40OviBrowser%'or (user_agent like '%Glass%' and user_agent like '%Android%'));
Single day "dumbed down" pageviews on Wikipedia mobile web (includes both zero-rated and non-zero-rated), all UAs:199,764,072 (100% of total)
select count(*)from webrequestwhere year = 2014and month = 11and day = 19and webrequest_source = 'mobile'and uri_host like '%.wikipedia.org'and http_status = '200' and(content_type like '%text/html%' or content_type like '%application/json%')and (uri_path like '/wiki/%' or uri_path like '/w/index.php' or(uri_path like '/w/api.php%' and uri_query like '%action=mobileview%'));