Just a note that I still disagree with WMF's response to this RFC, more for procedural and wikilegal reasons than analytic ones. I won't rehash that debate here, and could support having more statistics collected, but it seems to me that the principled thing to do is for WMF to respect the RfC process first and work on further analytics second. I wish people success in gathering analytics that the community could consider in a future RfC.

[...], but it seems to me that the principled thing to do is for WMF to respect the RfC process first and work on further analytics second.

What a farce that we have a vote and then ignore clear outcomes as Agreed

p.s. Sorry for the off-topic message, Dan. On-topic, I agree with Erik Z's approach of A/B testing which viewer is default, it seems like good science.

Could we sample only logged-in users?

just events daily events) I assume we are talking about logging stuff for logged in users and not logged in users. If that is the case this strategy will dwarf editor data with reader data, in the RFC most concerns seem to be about editor workflow. Could we sample only logged-in users? That way we will get a sample of data that better represents editor's interaction with media viewer and it will be a lot easier to quantify our results. we will present it to community members to >discuss its impact on the RfC, and agree on acceptance criteria (e.g. keep Media Viewer enabled by default unless >a 51% majority of users disables the feature on Enwiki by September 30?). Perhaps if we want the analytics to impact the RFC we should work with the community to come up with a survey strategy they find aceptable? On Tue, Jul 15, 2014 at 11:59 PM, Fabrice Florin <fflorin(a)wikimedia.org> wrote: _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

Disclaimer: I haven't been following the discussion Full disclosure: 1) I personally like the new feature 2) I value community opinion a lot Just my 2 cents on methodology: It seems to me that if we could present half of a target population with old and half with new settings from the outset (e.g. by focussing on new users only), then the outcome would be more convincing. In the current proposal if 40% would choose to switch , is that because 60% like the new settings better, or because they can't be bothered at that very moment to explore and make up their mind, or for any other reason cling to what they are offered initially. Erik From: analytics-bounces(a)lists.wikimedia.org [mailto:analytics-bounces@lists.wikimedia.org] On Behalf Of Gilles Dubuc Sent: Wednesday, July 16, 2014 14:00 To: A mailing list for the Analytics Team at WMF and everybody who has an interest in Wikipedia and analytics. Subject: Re: [Analytics] Media Viewer User Preference Data Could we sample only logged-in users? I think the idea is to measure both separately, because the default might end up being different for logged-in users and for anons, depending on the preference ratio for each. On Wed, Jul 16, 2014 at 7:29 AM, Nuria Ruiz <nuria(a)wikimedia.org> wrote: I assume we are talking about logging stuff for logged in users and not logged in users. If that is the case this strategy will dwarf editor data with reader data, in the RFC most concerns seem to be about editor workflow. Could we sample only logged-in users? That way we will get a sample of data that better represents editor's interaction with media viewer and it will be a lot easier to quantify our results. Perhaps if we want the analytics to impact the RFC we should work with the community to come up with a survey strategy they find aceptable? On Tue, Jul 15, 2014 at 11:59 PM, Fabrice Florin <fflorin(a)wikimedia.org> wrote: Dear Analytics Team, As discussed with Toby, Dario and Aaron, we would be grateful for your guidance to collect user preference data for Media Viewer (1), which is being challenged in RfCs on English (2) and German (3) Wikipedias, as well as on Wikimedia Commons (4). The English RfC closed last week, requesting that we disable Media Viewer right away. The multimedia team, in consultation with community and legal teams, is not comfortable disabling the feature on the basis of this small RfC, which we believe does not accurately represent the views of the millions of users whom we serve. To address this issue, we would like to conduct a wider outreach to collect more accurate data about user preferences than either this RfC or our optional surveys can provide. We propose to develop a more prominent viewing options panel (5) that would make it very easy to switch quickly to your preferred viewing mode, as shown in this prototype (6). Right after launch of this new feature, all users would be shown this prominent panel and asked to select their favorite viewing option. Once we have collected and analyzed that data over a period of a month or two, we would be able to make more informed decisions with our community on whether or not to keep the feature enabled -- based on actual responses from all users, rather than speculation. With your help, we have already developed a basic dashboard that tracks Media Viewer opt-in/out events (8), based on clicks on the less prominent ‘Disable’ feature at the bottom of the metadata panel. We propose to modify this dashboard to support this new initiative, as outlined in this ticket (7), with these changes: a) aim to track total users who enable/disable Media Viewer, rather than just events b) switch to a 3-state preference setting: enabled / disabled / default c) try to measure the total number of users in each group (instead of daily events) Also note that most users would start in the default state, showing them Media Viewer, since it is enabled by default — but after 10 or so image views, the viewing options panel would appear automatically, asking them to either enable or disable the feature. After they have made their selection, the panel would remain accessible (but much less prominent) and they can still switch state (but can't go back to default). We could use your guidance on the right way of logging total logged-out users, so we can use them as a basis for percentages: should we log enabled/disabled state on some specified event/action: page load? thumbnail click? first site visit of the day? — or do you recommend another method? We would want to collect this data for a month or two, so we don’t only capture the immediate responses from the most active users, but also those of less frequent visitors. We would also appreciate your comments on the specific tickets (5) and (6), with any recommendations for improvement. Keep in mind that we would like to respond quickly to our community’s concerns and have limited resources, so we would prefer to get this work done in the next week or two. And it will take a couple months to develop, release and collect enough data -- so even if we start tomorrow, we may not have conclusive data to share until September. So time is of the essence. :) Gergo is spearheading this project, and may have more technical questions for you. But I wanted to give an overview from a product perspective, before we dive in to the implementation details. Once we figure out a practical way to implement this request together, we will present it to community members to discuss its impact on the RfC, and agree on acceptance criteria (e.g. keep Media Viewer enabled by default unless a 51% majority of users disables the feature on Enwiki by September 30?). Thanks in advance for your advice. We look forward to working with you soon on this important project, which could impact other features now in development. All the best, Fabrice (1) https://www.mediawiki.org/wiki/Multimedia/About_Media_Viewer (2) https://en.wikipedia.org/wiki/Wikipedia:Media_Viewer/June_2014_RfC (3) https://de.wikipedia.org/wiki/Wikipedia:Meinungsbilder/Medienbetrachter (4) https://commons.wikimedia.org/wiki/Commons:Requests_for_comment/Media_Viewe… (5) https://wikimedia.mingle.thoughtworks.com/projects/multimedia/cards/787 (6) http://pauginer.github.io/prototypes/media-viewer/desk/disabling-settings/i… (7) https://wikimedia.mingle.thoughtworks.com/projects/multimedia/cards/793 _______________________________ Fabrice Florin Product Manager, Multimedia Wikimedia Foundation https://www.mediawiki.org/wiki/User:Fabrice_Florin_(WMF) _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

On Thu, Jul 17, 2014 at 4:24 AM, Andrew Gray <andrew.gray(a)dunelm.org.uk> wrote: Regarding help pages ... It is safe to say that a decent segment of the editing community wants to keep the old interface, and the old interface is needed to edit the page, and I think it is safe to assume that the MediaViewer isnt going away - at worst it will go back to being a Beta Feature that is enabled for a large segment of the userbase. Therefore, the help pages need to cover both. This is a cost of introducing a feature that isnt a complete replacement of prior functionality - and has a high maintenance cost that the devs and community must bear. The solution to that is to hold the feature in Beta until it is an acceptable replacement for the existing functionality. If this A/B testing goes ahead, and I think it should, the relevant help pages should have a note at the top that explains that testing is occurring and their user pref may have been preset to either one. -- John Vandenberg _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics