Wikitech-l August 2016

wikitech-l@lists.wikimedia.org

78 participants
100 discussions

Wiki Conference India 2016 (Chandigarh hackathon)
by Santosh Shingare 12 Aug '16

12 Aug '16

Hello all, I am happy to announce, chandigarh hackathon WCI2016, close with 7 application, which is useful to all community. *1)* *WikiSpeak with native language :* (Hindi) (*Web Application* + *Android Mobile application*). *2)* *Edit Tamil Wiktionary **(Tamil community request)** :* - (*Android application*) *3)* *Audio file upload to WikiData (Punjabi community request) :* (*Android application*) *4) Wikipedia articles on google map :* find wikipedia articles on google map (with latitude and longitude) as per city. *5) OCR (Native Application) *: Convert scanned book copy to Indian language text with google doc (Tested for Hindi and Malayalam). *6) Communication platform[WebRTC] (Web Application)* : Community used this to talk or conference (Audio/video web conferencing application) *7) Notification (Event based)* : showing popup on event eg (If recent changes happen, It will show popup which article was updated) Thank you all participant. who participate in hackathon and deliver a big success *Note : I will share code, links and other details very shortly.* Thank you *Santosh Shingare.* *User:cherishsantosh*

1 0

Language switching improvements on mobile web
by Jon Katz 12 Aug '16

12 Aug '16

Hey Everyone, A big shout out to the reading web team (and Adam B), who rolled out a number of subtle, but important changes to the mobile web sites of all our projects earlier this week. Through a series of UX improvements, the team made it easier for people to switch between languages in an article on mobile web. To all you monolinguists out there, this is a very common and vital ability for people who speak more than one language. In addition to being a big improvement for our multi-lingual users, the development of this feature followed many of the best practices we aspire to at the foundation: - language switching on mobile initially discovered as a problem to be solved through quantitative research - tied to reading team's early strategic objective of better serving global readers - development stages included: - community consultation <https://www.mediawiki.org/wiki/Reading/Web/Projects/Improve_in-article_lang…> - iteration based on results - a/b testing and analysis in beta - qualitative research with live users <https://docs.google.com/a/wikimedia.org/presentation/d/1gbqV0fDHUDJjDZgYNDY…> - iteration based on results - collaboration with the language team (they are simultaneously rolling out compact language links <https://www.mediawiki.org/wiki/Universal_Language_Selector/Design/Interlang…> out of beta on desktop) - iteration based on collaboration Am I missing anything? More on the project here <https://www.mediawiki.org/wiki/Reading/Web/Projects/Improve_in-article_lang…> and a screenshot of the new language switching button is below Best, J

1 0

Schema migration for 'image' and 'oldimage' tables
by Krinkle 11 Aug '16

11 Aug '16

TL;DR: Participate on T589 and help decide what the upcoming schema change should entail, and how we'll migrate existing data. Hey all, Couple weeks ago we dedicated an IRC office hour to https://phabricator.wikimedia.org/T589 (RFC: image and oldimage tables). Updated draft at: https://www.mediawiki.org/wiki/Requests_for_comment/image_and_oldimage_tabl… We clarified scope and purpose of this particular RFC. Other issues are still important but considered orthogonal, and to be dealt with parallelly (or at a later time). Revised problem statement: 1. File revisions should have unique identifiers (better than "current file title + upload timestamp". (Subject to race conditions, hard to index/query, etc.) 2. Uploading new file revisions must not involve rows moving across tables, or rows being replaced. Participants agreed with the revised problem statement, it makes sense not to merely add primary keys to the existing tables ("Proposal 1" on the RFC draft), as that wouldn't adequately solve the Problem 2. The second proposal was to separate information about image revision from the image entity itself. Similar to the page/revisions tables. This was generally accepted as a good idea, but details are still to be determined. The general idea is that all revision-specific information (except for a pointer to the current revision) would no longer live in the 'image' table. Instead, information about all (for both current and past revisions) would live in the same table (instead of being moved around from one table to another when it's no longer the current one). Details at: https://www.mediawiki.org/wiki/Requests_for_comment/image_and_oldimage_tabl… Some open questions I'd like to see discussed on Phabricator (or here on wikitech): 1. Which fields do we keep in the 'image' table (img_id, img_name, img_latest, anything else?). All fields currently being queried from both tables, will probably only stay in the image revision table. But there are a few fields that we intentionally only want to query about current versions. For example 'img_sha1'. For duplicate-detection, we need to only consider the latest revisions. Doing this by keeping img_sha1 means uploading a new revision will involve updating two fields instead of one (img_latest and img_sha1). This isn't unprecedented as we do this for page as well (WikiPage::updateRevisionOn; page_latest, page_touched, page_is_redirect, page_len). Are there other fields we need to keep besides img_sha1? Or should we can solve the img_sha1 use case in a different manner? 2. img_metadata This field is a blob of serialised PHP (typically representing the Exif data of an image). Tim (correct me if I got it wrong) mentioned we could potentially make migration easier by changing img_metadata to be stored in a separate table and change the img_metadata field (in the image revision table) to instead be a pointer to a primary key. This could potentially be done separately later, but if it helps migration, we should consider doing it now. How will this interact with file deletion? Will it be difficult to garbage collect this? Do we need to? (We don't seem to do it for the 'text' table / external store; is it worth moving this an external store?) 3. Migration If we rename both tables (image/oldimage -> file/filerevision), we'd have the ability to run migration in the background without interfering with the live site, and without requiring a long read-only period and/or duplicate and additional code complexity to be developed. Is there a way we can do the migration without creating two new tables? Using the oldimage table as import destination for current rows isn't straight forward as existing scan queries would need to skip the current rows somehow while in the midst of this migration. Seems possible, but is it worth the complexity? (We'd need extra code that knows about that migration field, and how long do we keep that code? Also complicates migration for third-parties using update.php). Is creating the new tables separately viable for the scale of Wikimedia Commons? (and dropping the old ones once finished). Is this a concern from a DBA perspective with regards to storage space? (We'd temporarily need about twice the space for these tables). So far I understood that it wouldn't be a problem per se, but that there are also other options we can explore for Wikimedia. For example we could use a separate set of slaves and alter those while depooled (essentially using entirely separate set of db slaves instead of a separate table within each slave). Do we create the new table(s) separately and switch over once it's caught up? This would require doing multiple passes as we depool slaves one by one (we've done that before at Wikimedia). Switch-over could be done by migrating before the software upgrade, with a very short read-only period after the last pass is finished. It wouldn't require maintaining multiple code paths, which is attractive. Other ideas? -- Timo

2 1

CI is having issues, hang with us
by Chad Horohoe 11 Aug '16

11 Aug '16

Hi, Nodepool isn't happy, so our CI is having trouble spinning up new instances for testing. We're well aware and working on it. Since a couple of people have already asked us what they can do to help...best thing you can do is avoid pushing changes for review (if they're not urgent) for a bit. And *definitely* don't run "recheck" on existing changes for right now. Thanks for understanding! -Chad

3 3

2016-08-10 Scrum of Scrums meeting notes
by Grace Gellerman 11 Aug '16

11 Aug '16

https://www.mediawiki.org/wiki/Scrum_of_scrums/2016-08-10 = 2016-08-10 = == Product == === Reading === ==== Reading Web ==== * Current sprint: https://phabricator.wikimedia.org/project/view/2115/ * Next sprint : https://phabricator.wikimedia.org/project/view/2126/ * New language switcher button deployed this morning (about 5 minutes ago, as pages are purged naturally changes will be visible) * Lazy loaded images for mobile web Wikipedias shipping next sprint in 3 deployments: all small wikis, all medium wikis, all large wikis * Adding a contribution tab to the hamburger menu next sprint * Enhancing watchstar next sprint, primarily for non-js users ==== iOS native app ==== * 5.0.6 Passed Regression last week ** Holding while we diagnose a memory crash ** Working with beta testers to verify fixes https://phabricator.wikimedia.org/tag/ios-app-v5.0.6-hotfix/ * 5.1 is in Development ** Major features are iPad and Find in Page ** Expectd to be feature complete end of week ** Expected to go to beta next week https://phabricator.wikimedia.org/project/board/1736/query/open/ * 5.2 Scheduled to begin development next week ** iOS 10 release ** Major features are Widgets and Feed improvements ** Expected to go to beta in early september ==== Android native app ==== * Current sprint: https://phabricator.wikimedia.org/project/view/2091/ * Next sprint: https://phabricator.wikimedia.org/project/board/2142/ * Working on navigation overhaul ==== Mobile Content Service ==== * Working on fixing links with + signs ( https://phabricator.wikimedia.org/T136223) ==== Reading Infrastructure ==== * Gergo working on pywikibot issue, Brad is OOO ===Community Tech=== * Requested new hardware for cross-wiki watchlists ( https://phabricator.wikimedia.org/T142538 ) * Will be switching Macedoian and/or Swedish Wikipedias to numeric sorting next week for wider testing * Working on Programs Dashboard (https://outreachdashboard.wmflabs.org/) * Working on IABot improvements/stability == Editing == === VisualEditor === * Blocked: ** We're waiting on Design Research with T141069 so we can work with them on T141068 for the new wikitext editor work. * Blocking: ** Parsing team are waiting for our response on "native" Parsoid <gallery> implementation. Thalia will look at it. * Updates: ** Now live for logged-in users on Arabic-script Wikipedias; logged-out users planned for next week, then Indic script Wikipedias ** Lots of the team in town this week. Working on a few things. Come talk to us if you're also in town and want to say hi. ** HTML diffs for partial page saving (so faster saves); needs input from Services team on https://github.com/edg2s/restbase/tree/sections branch of RESTbase; PR forthcoming ** New wikitext editor work continues; https://phabricator.wikimedia.org/T142138 and so on === Parsing === * Blocked: ** https://phabricator.wikimedia.org/T141591 -- Security review needed ** https://gerrit.wikimedia.org/r/#/c/264026/ -- Parsoid-native implementation of <gallery>; need review and feedback from VE team * Blocking: ** Shadow Namespaces: Please review my comments https://phabricator.wikimedia.org/T91162#2408633 -- I think this is a RFC discussion matter, but happy to read through it and review it. :) * Updates: ** Parsoid cluster upgraded to node v4 & Jessie (thanks ops) - deployments resumed yesterday ** Next step: migrate deployment process to scap3 -- to be scheduled with Services ** https://gerrit.wikimedia.org/r/#/c/303912/ Implement magic word based opt-out of global user pages -- bikeshed on __NOGLOBAL__ or __NOFOREIGN__ ... === Multimedia === * Blocked: ** Is there an update on thumbor production status? (Performance?) It's still blocking ImageTweaks deployment. * Blocking: None of which we're aware. * Updates: ** Continuing our work on FileAnnotations; currently pending on === Language === * Blocked: * Blocking: * Updates: ** ContentTranslation dumps are available at, https://dumps.wikimedia.org/other/contenttranslation/ ** CX MT card and templates getting new changes this and next week. ** Apertium -> Jessie work in progress. Few packages left to upload. === Collaboration === * Blocked * Blocking ** We are working on rewriting the Flow caching for the data center migration - https://phabricator.wikimedia.org/T120009 * Updates ** Finishing the ptwikibooks LiquidThreads -> Flow conversion, which had some issues - https://phabricator.wikimedia.org/T119509 ** Working on allowing dynamic actions in the Echo secondary links (e.g. unwatching a Flow board through AJAX) - https://phabricator.wikimedia.org/T132975 == Technology == === Analytics === * refinery deployment migrated to scap3, tested and everything works well * sqooping all mediawiki databases out to hdfs, from dbstore1002, so far seems to not impact that machine much * pagecounts-raw and pagecounts-all-sites dataset updates have been stopped, old files remain, upgrade to the new pageviews or pagecounts-ez datasets as per: https://dumps.wikimedia.org/other/analytics/ * eventlogging kafka pipeline upgraded to fix broker restart bug * follow-up on pageview spike: Windows update caused a problem with the TLS handshake in Chrome 41 === Architecture === ** **https://www.mediawiki.org/wiki/Architecture_committee/Status* <https://www.mediawiki.org/wiki/Architecture_committee/Status> ** Last week: ArchCom meetings 2016W31: 2016-08-03 (Wed) *** 1pm PDT (20 UTC) Planning meeting: [[Phab:E250]] *** 2pm PDT (21 UTC) IRC #wikimedia-office [[Phab:*E251]]* ***** [[Phab:T128351]]: Notifications in core* ** This week: ArchCom meetings 2016W32: 2016-08-10 (Wed) *** 1pm PDT (20 UTC) Planning meeting: [[Phab:E258]] *** 2pm PDT (21 UTC) IRC #wikimedia-office [[Phab:*E259]]* ***** [[Wikitext]] *(and whether Wikimedia should invest in a spec) === Services === * Parsoid move to Jessie and node 4 complete * Electron PDF rendering service discussion: *https://phabricator.wikimedia.org/T142226* <https://phabricator.wikimedia.org/T142226> * Working on new kafka driver for Change-Prop - deploy as soon as kafka upgrade to 0.9 === Technical Operations === * '''Blocked''' ** None * '''Blocking''' ** None * Updates ** Parsoid clusters upgraded to jessie throughout the week ** Worked with research to get ORES workers their own dedicated cluster. Hardware estimations done ** xkey meeting held. ** Review of ContentTranslation apertium packages ongoing ** labs got openstack liberty upgrade === Discovery === * No blockers * Working on BM25 implementation * Working on multi-wiki indexes * Question mark removal deployed * Bugfixes for cross-language search deployed * Portal stats updated + fixes for small languages === Interactive Team === * Working on deploying <mapframe> and <maplink> to all non-Wikipedia sites * Working on replacing GeoHack with <maplink> + info screen on all Wikipedias * Working on deploying Tabular data on Commons. Already synced up with Wikidata. === Security === * MediaWiki 1.27.1 will be released this week * Notifications for OAuth consumer proposals are being deployed this week: ** https://phabricator.wikimedia.org/T61772 ** https://phabricator.wikimedia.org/T62528 * Planning for captcha improvements continues ** https://phabricator.wikimedia.org/T125132 ** https://phabricator.wikimedia.org/T141490 ** First step will be re-generating images using current script == Wikidata == * No blockers. * New features on query.wikidata.org (live preview of example queries, map layers). * Working on first feedback we got from publishing the Commons MediaInfo prototype. * Layout tweaks to the default Wikidata UI. == Fundraising Tech == * No blockers * Deploying and monitoring first activemq replacement changes * Minor Central Notice deploy with tonight's SWAT * Testing background contact de-duping script for CiviCRM == Performance == * No blockers * Ori still out * Still doing heavy refactoring in ResourceLoader/Outputpage for the critical rendering path * Still working on transaction bugfixes and improvements for multi-DC * WebpageTest now supports Opera Mini & UC Mini * Thumbor plugins test suite complete, production work to resume next week * Team offsite to be in NY area end of November

1 0

Fwd: Technical Collaboration quarterly goals for October - December 2016
by Quim Gil 10 Aug '16

10 Aug '16

Hi, We are starting to discuss the Technical Collaboration <https://meta.wikimedia.org/wiki/Technical_Collaboration> team plans for October - December. If you or your communities have proposals about goals or other tasks this team should plan for, please share them at https://phabricator.wikimedia.org/T142355 For more information about our quarterly goals, see https://meta.wikimedia. org/wiki/Technical_Collaboration/Goals -- Quim Gil Engineering Community Manager @ Wikimedia Foundation http://www.mediawiki.org/wiki/User:Qgil

1 0

Reading team updates from last quarter and planning for next quarter
by Moushira Elamrawy 09 Aug '16

09 Aug '16

Good day everyone, Planning for the next quarter is coming soon, if you have relevant ideas that the reading team <https://www.mediawiki.org/wiki/Reading> can consider to work on for the upcoming 3 months, please add them to the page here <https://www.mediawiki.org/wiki/Reading/Quarterly_planning/FY2016-2017/Q2>[0]. The page has archives for quarterly planning from previous months, and you can review ideas from last quarter here <https://www.mediawiki.org/wiki/Reading/Quarterly_planning/FY2016-2017/Q1> [1] There is also a summary for Reading team updates during the last quarte <https://www.mediawiki.org/wiki/Reading_team_updates_April_2016_-_July_2016>r [2], major highlights include, a) Lazy loading of images <https://www.mediawiki.org/wiki/Reading/Web/Projects/Performance/Lazy_loadin…>,[3] b) new readers <https://meta.wikimedia.org/wiki/New_Readers>,[4] c) and the move of the language switching button on mobile. Hope that was helpful, looking forward to your ideas and input! Regards, Moushira [0] https://www.mediawiki.org/wiki/Reading/Quarterly_planning/FY2016-2017/Q2 [1] https://www.mediawiki.org/wiki/Reading/Quarterly_planning/FY2016-2017/Q1 [2] https://www.mediawiki.org/wiki/Reading_team_updates_April_2016_-_July_2016 [3] https://www.mediawiki.org/wiki/Reading/Web/Projects/Performance/Lazy_loadin… [4] https://meta.wikimedia.org/wiki/New_Readers

1 0

Semantic MediaWiki Conference Fall 2016: Call for Contributions
by Karsten Hoffmeyer 09 Aug '16

09 Aug '16

Dear users, developers and all people interested in semantic wikis, We are very happy to announce that early bird registration to the 13th Semantic MediaWiki Conference is now open! Important facts reminder: * Dates: September 28th to September 30th 2016 (Wednesday to Friday) * Location: German Institute for International Educational Research (DIPF), Schloßstraße 29, Frankfurt am Main, Germany. * Conference page: https://www.semantic-mediawiki.org/wiki/SMWCon_Fall_2016 * Participants: Everybody interested in semantic wikis, especially in Semantic MediaWiki, e.g. users, developers, consultants, business representatives and researchers. We welcome new contributions from you: * We encourage contributions about applications and development of semantic wikis; for a list of topics, see [0]. * Please propose regular talks, posters or workshops on the conference website. We will do our best to consider your proposal in the conference program. An interesting variety of contributions has already been proposed, see [1]. * Presentations will generally be video and audio recorded and made available for others after the conference. News on participation and tutorials: * You can now officially register for the conference [2] and benefit from early bird fees until September 11, 2016. Organization: * German Institute for International Educational Research (DIPF) [3] and Open Semantic Data Association e. V. [4] have become the official organisers of SMWCon Fall 2016 * Thanks go to Wikimedia Germany [5] for supporting SMWCon Fall 2016 If you have questions you can contact Sabine Melnicki, Kendra Sticht and Christoph Schindler (Program Chairs), Karsten Hoffmeyer (General Chair) or Lia Veja (Local Chair) per e-mail (Cc). We will be happy to see you in Frankfurt! Sabine Melnicki, Kendra Sticht and Christoph Schindler (Program Board) [0] <https://www.semantic-mediawiki.org/wiki/SMWCon_Fall_2016/Announcement> [1] <https://www.semantic-mediawiki.org/wiki/SMWCon_Fall_2016#Program_proposals> [2] <https://ti.to/smwconfall2016/frankfurt> [3] <http://www.dipf.de/en/dipf-news> [4] <http://www.opensemanticdata.org/> [5] <https://www.wikimedia.de/>

1 0

2016W32 ArchCom-RFC meeting: Wikitext
by Rob Lanphier 09 Aug '16

09 Aug '16

Hi everyone, This week's office hour: Wikitext! This discussion is intended to be a continuation of the "Loosing the history of our projects to bitrot." thread. Coren stated the work in front of us very well at the start of it. > You know, this is actually quite troublesome: as the platform evolves > the older data becomes increasingly hard to use at all - making it > effectively lost even if we kept the bits around. This is a rather > widespread issue in computing as a rule; but I now find myself distressed > at its unavoidable effect on what we've always intended to be a permanent > contribution to humanity. The thread he started had pretty robust participation on a really important, which seemed to us in ArchCom worth continuing the discussion in one of our weekly office hours. So, after checking with Subbu (in my list message) that's what ended up as the top candidate. Subbu did some work to structure the conversation ([A Spec For Wikitext][1]) and I did some cleanup of the [Wikitext page on mw.org][2] as a possible hub for information on this topic, with [Talk:Wikitext][3] providing a durable conversation venue. Please join us to discuss this further Wednesday UTC Time: Wednesday 21 UTC (2pm PDT, 23 CEST) Place: #wikimedia-office Phab event: [E259][4] Rob p.s. as always, ArchCom status updates continue on [ArchCom/Status][5] [1]: https://www.mediawiki.org/wiki/Parsing/Notes/A_Spec_For_Wikitext [2]: https://www.mediawiki.org/wiki/Wikitext [3]: https://www.mediawiki.org/wiki/Talk:Wikitext [4]: https://phabricator.wikimedia.org/E259 [5]: https://www.mediawiki.org/wiki/Architecture_committee/Status

1 0

Readership metrics for the timespan until July 31, 2016
by Tilman Bayer 08 Aug '16

08 Aug '16

Hi all, this resumes the usual look <https://www.mediawiki.org/wiki/Reading/Readership_metrics_reports> at our most important readership metrics. This edition of the report introduces a new year-over-year comparison chart to better understand which traffic changes are seasonal, and highlights a recent browser-related issue that has, suddenly and probably artificially, increased our global usage by 11%. We also take a first look of how the ratio of pageviews per device has been developing over time, and point out some other events that had an impact on core metrics. As laid out earlier <https://lists.wikimedia.org/pipermail/mobile-l/2015-September/009773.html>, the main purpose is to raise awareness about how these are developing, call out the impact of any unusual events, and facilitate thinking about core metrics in general. As always; feedback and discussion welcome. Week-over-week and month-over-month changes are now being recorded on the Product page <https://www.mediawiki.org/wiki/Wikimedia_Product#Reading> at MediaWiki.org. This edition of the report covers a timespan of 14 weeks. Some other recent items of interest, in case they didn’t already catch your attention: - The WMF Reading team published its quarterly review presentation for Q4 2015-16 (April-June), which includes lots of traffic and usage data <https://commons.wikimedia.org/w/index.php?title=File:Reading_Quarterly_Revi…> . - I gave an update on long-term traffic trends at the Foundation’s metrics meeting in May, see slides <https://commons.wikimedia.org/w/index.php?title=File%3AWMF_Metrics_%26_Acti…> (and will again later this month). Now to the usual data. (All numbers below are averages for April 25-July 31, 2016 unless otherwise noted.) Pageviews Total: 511 million/day (-5.0% from the previous report) Context (April 2015-July 2016): <https://commons.wikimedia.org/wiki/File:Wikimedia_daily_pageviews,_all_vs._…> See also the Vital Signs dashboard <https://analytics.wikimedia.org/dashboards/vital-signs/#projects=all/metric…> (Small caveat: Android app pageviews in April and early May were undercounted by up to 1-2 million/dayl due to a bug <https://phabricator.wikimedia.org/T132965>.) Overall pageviews developed relatively uneventfully during the timespan of this report until around July 20, when the desktop traffic to the main pages of several large Wikipedias (including English, Russian and Dutch) suddenly increased drastically. The investigation <https://phabricator.wikimedia.org/T141506> is still ongoing. At this point though it seems likely that this is not due an undetected spider or such, but caused by browsers of real users behaving in an unusual way, namely an old version of Chrome on Windows. Overall pageviews for all projects appear to have increased around 11% due to this. To improve our understanding of which traffic movements are seasonal and which may indicate lasting changes, I have pieced together a chart overlaying the total pageview numbers back to May 2013 (the earliest time for which we have data according to the current pageview definition): <https://commons.wikimedia.org/wiki/File:Wikimedia_pageviews_year-over-year_…> For example, we talked before about how the large year-over-year decrease seen in earlier months (e.g. -11.9% from April 2015 to April 2016) could probably be attributed to two one-time events in May/June 2015: the switch to HTTPS-only connections and the block of the Chinese Wikipedia in China. At the time this was based on more indirect evidence, in particular changes in the ratio of pageviews from the Global North. Now that a year has passed, we can directly compare data from after these events year-over-year. It confirms that conclusion (in the chart, compare the red line against the other three on the left hand side). In July 2016 there actually was a 1.1% increase compared to July 2015. Of course this is partly due to the aforementioned main pages anomaly, which is also visible clearly in this chart. Desktop: 54.1% (previous report: 55.2%) Mobile web: 44.6% (previous report: 43.6%) Apps: 1.3% (previous report: 1.2%) (small caveat: app percentage is a bit too low due to the aforementioned Android-related bug in April/May) Context (April 2015-July 2016): <https://commons.wikimedia.org/wiki/File:Wikimedia_daily_pageviews,_mobile_p…> After largely stagnating earlier this year following the christmas bump, the mobile percentage started to rise again towards parity around May (reaching 48.6% in the week until July 11). However, the recent anomalously high main pages traffic described above was entirely on desktop and has thus, perhaps temporarily, brought this percentage down again (to 42.7% in the week until July 31). As a reminder, mobile already has a solid majority in terms of unique devices, cf. below. Global North ratio: 75.5% of total pageviews (previous report: 76.5%) Context (last seven months): After a notable decrease (or, increase in the Global South ratio) at the beginning of the year, this number was relatively steady over the timespan of this report. NB: We are currently rethinking this metric and might replace it with a different country selection constructed as part of the work on the New Readers project. Unique devices See the announcement blog post <https://blog.wikimedia.org/2016/03/30/unique-devices-dataset/> from March for background and details on this recently introduced metric. These estimated numbers are provided for all Wikimedia language projects (separately for the desktop and mobile web version). Because of the instrumentation method, there is no global metric for all projects and all languages, but following some recent discussions it is now planned <https://phabricator.wikimedia.org/T138027> to extend it to a cross-language global metric per project at least. For now, we track the daily numbers of the English Wikipedia in this report. Daily unique devices estimate for English Wikipedia: - Mobile web: 32.2 million (previous report: 33.2 million) - Desktop: 22.3 million (previous report: 25.7 million) - Total: 54.5 million (previous report: 58.9 million) - Average mobile web ratio: 59.3% (previous report: 56.5%) Context (January-August 2016): See also the new Vital Signs dashboard <https://analytics.wikimedia.org/dashboards/vital-signs/#projects=enwiki/met…> As mentioned in the previous report, there was a drop in early to mid March which hasn’t seen an explanation so far. This metric hasn’t existed long enough yet to get a good sense of what yearly seasonalities may exist, but it looks a bit like a decreasing trend in desktop devices during these seven months. Interestingly though, pageviews did not decrease as much. Or to put it differently, pageviews per device rose during this time (even if one disregards the aforementioned main page increase since the end of July): <https://commons.wikimedia.org/wiki/File:Daily_views_per_unique_device_for_E…> New app installations Android:19.7k/day (-37.7% from the previous report) Daily installs per device, from Google Play Context (last six months): As described in the previous report, in March a Google bug had accidentally increased visitor number to the app’s Play Store listing enormously (spike cut off in this chart). So this it’s not surprising that the average install rate over the timespan of this report is much lower. Even disregarding this factor though, the install rates have been decreasing over the last few months and are matched by uninstalls on most days (except for a bump during two weeks in July whose reason is unknown to me). Combined with devices going inactive etc. this has meant that the app’s install base has been shrinking, from from 15.5 million on March 31 to 15.1 million today. On July 28, an updated version of the app with several exciting new features was launched, with a blog post <https://blog.wikimedia.org/2016/07/28/wikipedia-android-redesign/> that had very high pageview numbers and was picked up by media outlets in various countries. Unfortunately this public attention has not translated into a lot of downloads - we estimate around 3000 (a few hours’ worth at the normal rate). In comparison, the relaunch of the OS app in March brought roughly 50k additional installs (see previous report). iOS: 3.18 k/day (-37.2% from the previous report) Download numbers from App Annie Context (last six months): As mentioned, the iOS app’s huge relaunch fell into the timespan of the previous report, so the decrease here is not surprising. App user retention Android: 16.7% (previous report: 15.4%) (Ratio of app installs opened again 7 days after installation, among all installed one week before a date that falls within this report. 1:100 sample) Context (last six months): As remarked in earlier reports, this data is a bit too noisy for drawing conclusions about whether retention changed significantly between different releases. But we can at least rule out the existence of major shifts during this timespan. iOS: N/A <https://phabricator.wikimedia.org/T126693> Unique app users Android: 1.145 million / day (-5.0% from the previous report timespan, reconstructed <https://phabricator.wikimedia.org/T132965>) Context (last six months): Continuing the slight but steady decline that had been interrupted by the aforementioned Google bug windfall in March. iOS: N/A <https://phabricator.wikimedia.org/T130432> While we don’t have total active user numbers available for the iOS app any more, let’s mention an interesting increase that happened in the app’s pageviews back in May, by around 20%. (Because of the app’s small share in overall traffic, it did not affect the global pageviews discussed above in a notable way.) After some investigation, it turned out this coincided with the release of version 5.0.3 of the app, see the yellow slice on the right: Josh and I are planning to look further into possible causes; at the moment it’s still possible that this is either an artefact or due to real improvements in the app. ---- For reference, the queries and source links used are listed below (access is needed for each). Unless otherwise noted, all content of this report is © Wikimedia Foundation and released under the CC BY-SA 3.0 <https://creativecommons.org/licenses/by-sa/3.0/> license. Most of the above charts are available on Commons, too <https://commons.wikimedia.org/w/index.php?title=Special:ListFiles&offset=20…>; together with PDF versions of this report. SELECT year, month, day, CONCAT(year,"-",LPAD(month,2,"0"),"-",LPAD(day,2,"0")) as date, sum(IF(access_method <> 'desktop', view_count, null)) AS mobileviews, SUM(view_count) AS allviews FROM wmf.projectview_hourly WHERE year>0 AND agent_type = 'user' GROUP BY year, month, day ORDER BY year, month, day LIMIT 1000; SELECT LEFT(timestamp, 10) AS date, sum(IF(access_method <> 'desktop', pageviews, null)) AS mobileviews, SUM(pageviews) AS allviews FROM staging.pageviews05 WHERE is_spider = FALSE AND is_automata = FALSE GROUP BY date; SELECT access_method, SUM(view_count)/(7*14) FROM wmf.projectview_hourly WHERE agent_type = 'user' AND CONCAT(year,"-",LPAD(month,2,"0"),"-",LPAD(day,2,"0")) BETWEEN "2016-04-25" AND "2016-07-31" GROUP BY access_method; SELECT year, month, day, CONCAT(year,"-",LPAD(month,2,"0"),"-",LPAD(day,2,"0")), SUM(view_count) AS all, SUM(IF (FIND_IN_SET(country_code, 'AD,AL,AT,AX,BA,BE,BG,CH,CY,CZ,DE,DK,EE,ES,FI,FO,FR,FX,GB,GG,GI,GL,GR,HR,HU,IE,IL,IM,IS,IT,JE,LI,LU,LV,MC,MD,ME,MK,MT,NL,NO,PL,PT,RO,RS,RU,SE,SI,SJ,SK,SM,TR,VA,AU,CA,HK,MO,NZ,JP,SG,KR,TW,US') > 0, view_count, 0)) AS Global_North_views FROM wmf.projectview_hourly WHERE year > 0 AND agent_type='user' GROUP BY year, month, day ORDER BY year, month, day LIMIT 1000; https://console.developers.google.com/storage/browser/pubsite_prod_rev_0281… (“overview”) https://www.appannie.com/dashboard/252257/item/324715238/downloads/ (select “Total”) SELECT LEFT(timestamp, 8) AS date, SUM(IF(event_appInstallAgeDays = 0, 1, 0)) AS day0_active, SUM(IF(event_appInstallAgeDays = 7, 1, 0)) AS day7_active FROM log.MobileWikiAppDailyStats_12637385 WHERE userAgent LIKE '%-r-%' AND userAgent NOT LIKE '%Googlebot%' GROUP BY date ORDER BY DATE; (with the retention rate calculated as day7_active divided by day0_active from seven days earlier, of course) SELECT CONCAT(year,"-",LPAD(month,2,"0"),"-",LPAD(day,2,"0")) as date, unique_count AS Android_DAU FROM wmf.mobile_apps_uniques_daily WHERE platform = 'Android'; SELECT year, month, day, CONCAT(year,'-',LPAD(month,2,'0'),'-',LPAD(day,2,'0')) AS date, CONCAT(SPLIT(user_agent_map['wmf_app_version'], '\\.')[0], COALESCE( CONCAT('.', SPLIT(user_agent_map['wmf_app_version'], '\\.')[1]), ''), COALESCE( CONCAT('.', SPLIT(user_agent_map['wmf_app_version'], '\\.')[2]), '')) AS app_version, SUM(view_count) AS views FROM wmf.pageview_hourly WHERE year >0 AND access_method = 'mobile app' AND user_agent_map['os_family'] = 'iOS' AND agent_type = 'user' GROUP BY year, month, day, user_agent_map['wmf_app_version'] ORDER BY year, month, day, app_version LIMIT 20000; -- Tilman Bayer Senior Analyst Wikimedia Foundation IRC (Freenode): HaeB

1 0

← Newer
1
2
3
4
5
6
7
8
9
10
Older →

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

Wikitech-l August 2016