Dear Analytics Team,
As discussed with Toby, Dario and Aaron, we would be grateful for your guidance to collect user preference data for Media Viewer (1), which is being challenged in RfCs on English (2) and German (3) Wikipedias, as well as on Wikimedia Commons (4).
The English RfC closed last week, requesting that we disable Media Viewer right away. The multimedia team, in consultation with community and legal teams, is not comfortable disabling the feature on the basis of this small RfC, which we believe does not accurately represent the views of the millions of users whom we serve.
To address this issue, we would like to conduct a wider outreach to collect more accurate data about user preferences than either this RfC or our optional surveys can provide. We propose to develop a more prominent viewing options panel (5) that would make it very easy to switch quickly to your preferred viewing mode, as shown in this prototype (6). Right after launch of this new feature, all users would be shown this prominent panel and asked to select their favorite viewing option. Once we have collected and analyzed that data over a period of a month or two, we would be able to make more informed decisions with our community on whether or not to keep the feature enabled -- based on actual responses from all users, rather than speculation.
With your help, we have already developed a basic dashboard that tracks Media Viewer opt-in/out events (8), based on clicks on the less prominent ‘Disable’ feature at the bottom of the metadata panel. We propose to modify this dashboard to support this new initiative, as outlined in this ticket (7), with these changes: a) aim to track total users who enable/disable Media Viewer, rather than just events b) switch to a 3-state preference setting: enabled / disabled / default c) try to measure the total number of users in each group (instead of daily events)
Also note that most users would start in the default state, showing them Media Viewer, since it is enabled by default — but after 10 or so image views, the viewing options panel would appear automatically, asking them to either enable or disable the feature. After they have made their selection, the panel would remain accessible (but much less prominent) and they can still switch state (but can't go back to default).
We could use your guidance on the right way of logging total logged-out users, so we can use them as a basis for percentages: should we log enabled/disabled state on some specified event/action: page load? thumbnail click? first site visit of the day? — or do you recommend another method? We would want to collect this data for a month or two, so we don’t only capture the immediate responses from the most active users, but also those of less frequent visitors.
We would also appreciate your comments on the specific tickets (5) and (6), with any recommendations for improvement. Keep in mind that we would like to respond quickly to our community’s concerns and have limited resources, so we would prefer to get this work done in the next week or two. And it will take a couple months to develop, release and collect enough data -- so even if we start tomorrow, we may not have conclusive data to share until September. So time is of the essence. :)
Gergo is spearheading this project, and may have more technical questions for you. But I wanted to give an overview from a product perspective, before we dive in to the implementation details. Once we figure out a practical way to implement this request together, we will present it to community members to discuss its impact on the RfC, and agree on acceptance criteria (e.g. keep Media Viewer enabled by default unless a 51% majority of users disables the feature on Enwiki by September 30?).
Thanks in advance for your advice. We look forward to working with you soon on this important project, which could impact other features now in development.
All the best,
Fabrice
(1) https://www.mediawiki.org/wiki/Multimedia/About_Media_Viewer (2) https://en.wikipedia.org/wiki/Wikipedia:Media_Viewer/June_2014_RfC (3) https://de.wikipedia.org/wiki/Wikipedia:Meinungsbilder/Medienbetrachter (4) https://commons.wikimedia.org/wiki/Commons:Requests_for_comment/Media_Viewer... (5) https://wikimedia.mingle.thoughtworks.com/projects/multimedia/cards/787 (6) http://pauginer.github.io/prototypes/media-viewer/desk/disabling-settings/in... (7) https://wikimedia.mingle.thoughtworks.com/projects/multimedia/cards/793
_______________________________
Fabrice Florin Product Manager, Multimedia Wikimedia Foundation
Just a note that I still disagree with WMF's response to this RFC, more for procedural and wikilegal reasons than analytic ones. I won't rehash that debate here, and could support having more statistics collected, but it seems to me that the principled thing to do is for WMF to respect the RfC process first and work on further analytics second. I wish people success in gathering analytics that the community could consider in a future RfC.
Pine
On Tue, Jul 15, 2014 at 2:59 PM, Fabrice Florin fflorin@wikimedia.org wrote:
Dear Analytics Team,
As discussed with Toby, Dario and Aaron, we would be grateful for your guidance to collect user preference data for Media Viewer (1), which is being challenged in RfCs on English (2) and German (3) Wikipedias, as well as on Wikimedia Commons (4).
The English RfC closed last week, requesting that we disable Media Viewer right away. The multimedia team, in consultation with community and legal teams, is not comfortable disabling the feature on the basis of this small RfC, which we believe does not accurately represent the views of the millions of users whom we serve.
To address this issue, we would like to conduct a wider outreach to collect more accurate data about user preferences than either this RfC or our optional surveys can provide. We propose to develop a more prominent viewing options panel (5) that would make it very easy to switch quickly to your preferred viewing mode, as shown in this prototype (6). Right after launch of this new feature, all users would be shown this prominent panel and asked to select their favorite viewing option. Once we have collected and analyzed that data over a period of a month or two, we would be able to make more informed decisions with our community on whether or not to keep the feature enabled -- based on actual responses from all users, rather than speculation.
With your help, we have already developed a basic dashboard that tracks Media Viewer opt-in/out events (8), based on clicks on the less prominent ‘Disable’ feature at the bottom of the metadata panel. We propose to modify this dashboard to support this new initiative, as outlined in this ticket (7), with these changes: a) aim to track total users who enable/disable Media Viewer, rather than just events b) switch to a 3-state preference setting: enabled / disabled / default c) try to measure the total number of users in each group (instead of daily events)
Also note that most users would start in the default state, showing them Media Viewer, since it is enabled by default — but after 10 or so image views, the viewing options panel would appear automatically, asking them to either enable or disable the feature. After they have made their selection, the panel would remain accessible (but much less prominent) and they can still switch state (but can't go back to default).
We could use your guidance on the right way of logging total logged-out users, so we can use them as a basis for percentages: should we log enabled/disabled state on some specified event/action: page load? thumbnail click? first site visit of the day? — or do you recommend another method? We would want to collect this data for a month or two, so we don’t only capture the immediate responses from the most active users, but also those of less frequent visitors.
We would also appreciate your comments on the specific tickets (5) and (6), with any recommendations for improvement. Keep in mind that we would like to respond quickly to our community’s concerns and have limited resources, so we would prefer to get this work done in the next week or two. And it will take a couple months to develop, release and collect enough data -- so even if we start tomorrow, we may not have conclusive data to share until September. So time is of the essence. :)
Gergo is spearheading this project, and may have more technical questions for you. But I wanted to give an overview from a product perspective, before we dive in to the implementation details. Once we figure out a practical way to implement this request together, we will present it to community members to discuss its impact on the RfC, and agree on acceptance criteria (e.g. keep Media Viewer enabled by default unless a 51% majority of users disables the feature on Enwiki by September 30?).
Thanks in advance for your advice. We look forward to working with you soon on this important project, which could impact other features now in development.
All the best,
Fabrice
(1) https://www.mediawiki.org/wiki/Multimedia/About_Media_Viewer (2) https://en.wikipedia.org/wiki/Wikipedia:Media_Viewer/June_2014_RfC (3) https://de.wikipedia.org/wiki/Wikipedia:Meinungsbilder/Medienbetrachter (4) https://commons.wikimedia.org/wiki/Commons:Requests_for_comment/Media_Viewer... (5) https://wikimedia.mingle.thoughtworks.com/projects/multimedia/cards/787 (6) http://pauginer.github.io/prototypes/media-viewer/desk/disabling-settings/in... (7) https://wikimedia.mingle.thoughtworks.com/projects/multimedia/cards/793
Fabrice Florin Product Manager, Multimedia Wikimedia Foundation
https://www.mediawiki.org/wiki/User:Fabrice_Florin_(WMF)
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
On 15 July 2014 22:58, Pine W wiki.pine@gmail.com wrote:
Just a note that I still disagree with WMF's response to this RFC, more for procedural and wikilegal reasons than analytic ones. I won't rehash that debate here, and could support having more statistics collected, but it seems to me that the principled thing to do is for WMF to respect the RfC process first and work on further analytics second. I wish people success in gathering analytics that the community could consider in a future RfC.
That discussion is very far outside the scope of this mailing list.
Dan
The use of analytics staff resources is something that I take an interest in, and was expressing my qualified support for doing.
Pine
On Tue, Jul 15, 2014 at 11:00 PM, Dan Garry dgarry@wikimedia.org wrote:
On 15 July 2014 22:58, Pine W wiki.pine@gmail.com wrote:
Just a note that I still disagree with WMF's response to this RFC, more for procedural and wikilegal reasons than analytic ones. I won't rehash that debate here, and could support having more statistics collected, but it seems to me that the principled thing to do is for WMF to respect the RfC process first and work on further analytics second. I wish people success in gathering analytics that the community could consider in a future RfC.
That discussion is very far outside the scope of this mailing list.
Dan
-- Dan Garry Associate Product Manager for Platform and Mobile Apps Wikimedia Foundation
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Hi Pine,
[ responding off-list, as others called out that it is beyond the list's scope ]
On Tue, Jul 15, 2014 at 10:58:31PM -0700, Pine W wrote:
[...], but it seems to me that the principled thing to do is for WMF to respect the RfC process first and work on further analytics second.
Full ACK.
What a farce that we have a vote and then ignore clear outcomes as 1:4, and even 1:13 if we do not like them :-(
Have fun, Christian
The issue of the RfC is way outside the scope of this list. Please, let's keep things focussed.
Dan
On Wednesday, 16 July 2014, Dan Andreescu dandreescu@wikimedia.org wrote:
What a farce that we have a vote and then ignore clear outcomes as
1:4, and even 1:13 if we do not like them :-(
Agreed
Sorry, one word is clearly unclear and I should've expanded, as it turns out I misunderstood the context of Christian's comment.
I think it's farcical to establish the opinion of the entire community based on a small RFC. I think Fabrice's proposal to gather the opinion of a much larger audience is great and I look forward to supporting it. Most importantly, I think in the future we should lead with such an approach, and I call on Analytics folks to encourage / support that.
p.s. Sorry for the off-topic message, Dan. On-topic, I agree with Erik Z's approach of A/B testing which viewer is default, it seems like good science.
On Wed, Jul 16, 2014 at 2:03 PM, Dan Garry dgarry@wikimedia.org wrote:
The issue of the RfC is way outside the scope of this list. Please, let's keep things focussed.
Dan
On Wednesday, 16 July 2014, Dan Andreescu dandreescu@wikimedia.org wrote:
What a farce that we have a vote and then ignore clear outcomes as
1:4, and even 1:13 if we do not like them :-(
Agreed
-- Dan Garry Associate Product Manager for Platform and Mobile Apps Wikimedia Foundation
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
On 16 July 2014 11:47, Dan Andreescu dandreescu@wikimedia.org wrote:
p.s. Sorry for the off-topic message, Dan. On-topic, I agree with Erik Z's approach of A/B testing which viewer is default, it seems like good science.
No worries. I just don't want to see this bloat into a cross-list discussion. Just trying to keep things centralised. :-)
Dan
a) aim to track total users who enable/disable Media Viewer, rather than
just events
b) switch to a 3-state preference setting: enabled / disabled / default c) try to measure the total number of users in each group (instead of
daily events)
I assume we are talking about logging stuff for logged in users and not logged in users. If that is the case this strategy will dwarf editor data with reader data, in the RFC most concerns seem to be about editor workflow. Could we sample only logged-in users? That way we will get a sample of data that better represents editor's interaction with media viewer and it will be a lot easier to quantify our results.
Once we figure out a practical way to implement this request together, we
will present it to community members to >discuss its impact on the RfC, and agree on acceptance criteria (e.g. keep Media Viewer enabled by default unless >a 51% majority of users disables the feature on Enwiki by September 30?).
Perhaps if we want the analytics to impact the RFC we should work with the community to come up with a survey strategy they find aceptable?
On Tue, Jul 15, 2014 at 11:59 PM, Fabrice Florin fflorin@wikimedia.org wrote:
Dear Analytics Team,
As discussed with Toby, Dario and Aaron, we would be grateful for your guidance to collect user preference data for Media Viewer (1), which is being challenged in RfCs on English (2) and German (3) Wikipedias, as well as on Wikimedia Commons (4).
The English RfC closed last week, requesting that we disable Media Viewer right away. The multimedia team, in consultation with community and legal teams, is not comfortable disabling the feature on the basis of this small RfC, which we believe does not accurately represent the views of the millions of users whom we serve.
To address this issue, we would like to conduct a wider outreach to collect more accurate data about user preferences than either this RfC or our optional surveys can provide. We propose to develop a more prominent viewing options panel (5) that would make it very easy to switch quickly to your preferred viewing mode, as shown in this prototype (6). Right after launch of this new feature, all users would be shown this prominent panel and asked to select their favorite viewing option. Once we have collected and analyzed that data over a period of a month or two, we would be able to make more informed decisions with our community on whether or not to keep the feature enabled -- based on actual responses from all users, rather than speculation.
With your help, we have already developed a basic dashboard that tracks Media Viewer opt-in/out events (8), based on clicks on the less prominent ‘Disable’ feature at the bottom of the metadata panel. We propose to modify this dashboard to support this new initiative, as outlined in this ticket (7), with these changes: a) aim to track total users who enable/disable Media Viewer, rather than just events b) switch to a 3-state preference setting: enabled / disabled / default c) try to measure the total number of users in each group (instead of daily events)
Also note that most users would start in the default state, showing them Media Viewer, since it is enabled by default — but after 10 or so image views, the viewing options panel would appear automatically, asking them to either enable or disable the feature. After they have made their selection, the panel would remain accessible (but much less prominent) and they can still switch state (but can't go back to default).
We could use your guidance on the right way of logging total logged-out users, so we can use them as a basis for percentages: should we log enabled/disabled state on some specified event/action: page load? thumbnail click? first site visit of the day? — or do you recommend another method? We would want to collect this data for a month or two, so we don’t only capture the immediate responses from the most active users, but also those of less frequent visitors.
We would also appreciate your comments on the specific tickets (5) and (6), with any recommendations for improvement. Keep in mind that we would like to respond quickly to our community’s concerns and have limited resources, so we would prefer to get this work done in the next week or two. And it will take a couple months to develop, release and collect enough data -- so even if we start tomorrow, we may not have conclusive data to share until September. So time is of the essence. :)
Gergo is spearheading this project, and may have more technical questions for you. But I wanted to give an overview from a product perspective, before we dive in to the implementation details. Once we figure out a practical way to implement this request together, we will present it to community members to discuss its impact on the RfC, and agree on acceptance criteria (e.g. keep Media Viewer enabled by default unless a 51% majority of users disables the feature on Enwiki by September 30?).
Thanks in advance for your advice. We look forward to working with you soon on this important project, which could impact other features now in development.
All the best,
Fabrice
(1) https://www.mediawiki.org/wiki/Multimedia/About_Media_Viewer (2) https://en.wikipedia.org/wiki/Wikipedia:Media_Viewer/June_2014_RfC (3) https://de.wikipedia.org/wiki/Wikipedia:Meinungsbilder/Medienbetrachter (4) https://commons.wikimedia.org/wiki/Commons:Requests_for_comment/Media_Viewer... (5) https://wikimedia.mingle.thoughtworks.com/projects/multimedia/cards/787 (6) http://pauginer.github.io/prototypes/media-viewer/desk/disabling-settings/in... (7) https://wikimedia.mingle.thoughtworks.com/projects/multimedia/cards/793
Fabrice Florin Product Manager, Multimedia Wikimedia Foundation
https://www.mediawiki.org/wiki/User:Fabrice_Florin_(WMF)
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Could we sample only logged-in users?
I think the idea is to measure both separately, because the default might end up being different for logged-in users and for anons, depending on the preference ratio for each.
On Wed, Jul 16, 2014 at 7:29 AM, Nuria Ruiz nuria@wikimedia.org wrote:
a) aim to track total users who enable/disable Media Viewer, rather than
just events
b) switch to a 3-state preference setting: enabled / disabled / default c) try to measure the total number of users in each group (instead of
daily events)
I assume we are talking about logging stuff for logged in users and not logged in users. If that is the case this strategy will dwarf editor data with reader data, in the RFC most concerns seem to be about editor workflow. Could we sample only logged-in users? That way we will get a sample of data that better represents editor's interaction with media viewer and it will be a lot easier to quantify our results.
Once we figure out a practical way to implement this request together,
we will present it to community members to >discuss its impact on the RfC, and agree on acceptance criteria (e.g. keep Media Viewer enabled by default unless >a 51% majority of users disables the feature on Enwiki by September 30?).
Perhaps if we want the analytics to impact the RFC we should work with the community to come up with a survey strategy they find aceptable?
On Tue, Jul 15, 2014 at 11:59 PM, Fabrice Florin fflorin@wikimedia.org wrote:
Dear Analytics Team,
As discussed with Toby, Dario and Aaron, we would be grateful for your guidance to collect user preference data for Media Viewer (1), which is being challenged in RfCs on English (2) and German (3) Wikipedias, as well as on Wikimedia Commons (4).
The English RfC closed last week, requesting that we disable Media Viewer right away. The multimedia team, in consultation with community and legal teams, is not comfortable disabling the feature on the basis of this small RfC, which we believe does not accurately represent the views of the millions of users whom we serve.
To address this issue, we would like to conduct a wider outreach to collect more accurate data about user preferences than either this RfC or our optional surveys can provide. We propose to develop a more prominent viewing options panel (5) that would make it very easy to switch quickly to your preferred viewing mode, as shown in this prototype (6). Right after launch of this new feature, all users would be shown this prominent panel and asked to select their favorite viewing option. Once we have collected and analyzed that data over a period of a month or two, we would be able to make more informed decisions with our community on whether or not to keep the feature enabled -- based on actual responses from all users, rather than speculation.
With your help, we have already developed a basic dashboard that tracks Media Viewer opt-in/out events (8), based on clicks on the less prominent ‘Disable’ feature at the bottom of the metadata panel. We propose to modify this dashboard to support this new initiative, as outlined in this ticket (7), with these changes: a) aim to track total users who enable/disable Media Viewer, rather than just events b) switch to a 3-state preference setting: enabled / disabled / default c) try to measure the total number of users in each group (instead of daily events)
Also note that most users would start in the default state, showing them Media Viewer, since it is enabled by default — but after 10 or so image views, the viewing options panel would appear automatically, asking them to either enable or disable the feature. After they have made their selection, the panel would remain accessible (but much less prominent) and they can still switch state (but can't go back to default).
We could use your guidance on the right way of logging total logged-out users, so we can use them as a basis for percentages: should we log enabled/disabled state on some specified event/action: page load? thumbnail click? first site visit of the day? — or do you recommend another method? We would want to collect this data for a month or two, so we don’t only capture the immediate responses from the most active users, but also those of less frequent visitors.
We would also appreciate your comments on the specific tickets (5) and (6), with any recommendations for improvement. Keep in mind that we would like to respond quickly to our community’s concerns and have limited resources, so we would prefer to get this work done in the next week or two. And it will take a couple months to develop, release and collect enough data -- so even if we start tomorrow, we may not have conclusive data to share until September. So time is of the essence. :)
Gergo is spearheading this project, and may have more technical questions for you. But I wanted to give an overview from a product perspective, before we dive in to the implementation details. Once we figure out a practical way to implement this request together, we will present it to community members to discuss its impact on the RfC, and agree on acceptance criteria (e.g. keep Media Viewer enabled by default unless a 51% majority of users disables the feature on Enwiki by September 30?).
Thanks in advance for your advice. We look forward to working with you soon on this important project, which could impact other features now in development.
All the best,
Fabrice
(1) https://www.mediawiki.org/wiki/Multimedia/About_Media_Viewer (2) https://en.wikipedia.org/wiki/Wikipedia:Media_Viewer/June_2014_RfC (3) https://de.wikipedia.org/wiki/Wikipedia:Meinungsbilder/Medienbetrachter (4) https://commons.wikimedia.org/wiki/Commons:Requests_for_comment/Media_Viewer... (5) https://wikimedia.mingle.thoughtworks.com/projects/multimedia/cards/787 (6) http://pauginer.github.io/prototypes/media-viewer/desk/disabling-settings/in... (7) https://wikimedia.mingle.thoughtworks.com/projects/multimedia/cards/793
Fabrice Florin Product Manager, Multimedia Wikimedia Foundation
https://www.mediawiki.org/wiki/User:Fabrice_Florin_(WMF)
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Disclaimer: I haven't been following the discussion
Full disclosure: 1) I personally like the new feature 2) I value community opinion a lot
Just my 2 cents on methodology:
It seems to me that if we could present half of a target population with old and half with new settings from the outset (e.g. by focussing on new users only), then the outcome would be more convincing.
In the current proposal if 40% would choose to switch , is that because 60% like the new settings better, or because they can't be bothered at that very moment to explore and make up their mind, or for any other reason cling to what they are offered initially.
Erik
From: analytics-bounces@lists.wikimedia.org [mailto:analytics-bounces@lists.wikimedia.org] On Behalf Of Gilles Dubuc Sent: Wednesday, July 16, 2014 14:00 To: A mailing list for the Analytics Team at WMF and everybody who has an interest in Wikipedia and analytics. Subject: Re: [Analytics] Media Viewer User Preference Data
Could we sample only logged-in users?
I think the idea is to measure both separately, because the default might end up being different for logged-in users and for anons, depending on the preference ratio for each.
On Wed, Jul 16, 2014 at 7:29 AM, Nuria Ruiz nuria@wikimedia.org wrote:
a) aim to track total users who enable/disable Media Viewer, rather than just events
b) switch to a 3-state preference setting: enabled / disabled / default
c) try to measure the total number of users in each group (instead of daily events)
I assume we are talking about logging stuff for logged in users and not logged in users.
If that is the case this strategy will dwarf editor data with reader data, in the RFC most concerns seem to be about editor workflow. Could we sample only logged-in users? That way we will get a sample of data that better represents editor's interaction with media viewer and it will be a lot easier to quantify our results.
Once we figure out a practical way to implement this request together, we will present it to community members to >discuss its impact on the RfC, and agree on acceptance criteria (e.g. keep Media Viewer enabled by default unless >a 51% majority of users disables the feature on Enwiki by September 30?).
Perhaps if we want the analytics to impact the RFC we should work with the community to come up with a survey strategy they find aceptable?
On Tue, Jul 15, 2014 at 11:59 PM, Fabrice Florin fflorin@wikimedia.org wrote:
Dear Analytics Team,
As discussed with Toby, Dario and Aaron, we would be grateful for your guidance to collect user preference data for Media Viewer (1), which is being challenged in RfCs on English (2) and German (3) Wikipedias, as well as on Wikimedia Commons (4).
The English RfC closed last week, requesting that we disable Media Viewer right away. The multimedia team, in consultation with community and legal teams, is not comfortable disabling the feature on the basis of this small RfC, which we believe does not accurately represent the views of the millions of users whom we serve.
To address this issue, we would like to conduct a wider outreach to collect more accurate data about user preferences than either this RfC or our optional surveys can provide. We propose to develop a more prominent viewing options panel (5) that would make it very easy to switch quickly to your preferred viewing mode, as shown in this prototype (6). Right after launch of this new feature, all users would be shown this prominent panel and asked to select their favorite viewing option. Once we have collected and analyzed that data over a period of a month or two, we would be able to make more informed decisions with our community on whether or not to keep the feature enabled -- based on actual responses from all users, rather than speculation.
With your help, we have already developed a basic dashboard that tracks Media Viewer opt-in/out events (8), based on clicks on the less prominent ‘Disable’ feature at the bottom of the metadata panel. We propose to modify this dashboard to support this new initiative, as outlined in this ticket (7), with these changes:
a) aim to track total users who enable/disable Media Viewer, rather than just events
b) switch to a 3-state preference setting: enabled / disabled / default
c) try to measure the total number of users in each group (instead of daily events)
Also note that most users would start in the default state, showing them Media Viewer, since it is enabled by default — but after 10 or so image views, the viewing options panel would appear automatically, asking them to either enable or disable the feature. After they have made their selection, the panel would remain accessible (but much less prominent) and they can still switch state (but can't go back to default).
We could use your guidance on the right way of logging total logged-out users, so we can use them as a basis for percentages: should we log enabled/disabled state on some specified event/action: page load? thumbnail click? first site visit of the day? — or do you recommend another method? We would want to collect this data for a month or two, so we don’t only capture the immediate responses from the most active users, but also those of less frequent visitors.
We would also appreciate your comments on the specific tickets (5) and (6), with any recommendations for improvement. Keep in mind that we would like to respond quickly to our community’s concerns and have limited resources, so we would prefer to get this work done in the next week or two. And it will take a couple months to develop, release and collect enough data -- so even if we start tomorrow, we may not have conclusive data to share until September. So time is of the essence. :)
Gergo is spearheading this project, and may have more technical questions for you. But I wanted to give an overview from a product perspective, before we dive in to the implementation details. Once we figure out a practical way to implement this request together, we will present it to community members to discuss its impact on the RfC, and agree on acceptance criteria (e.g. keep Media Viewer enabled by default unless a 51% majority of users disables the feature on Enwiki by September 30?).
Thanks in advance for your advice. We look forward to working with you soon on this important project, which could impact other features now in development.
All the best,
Fabrice
(1) https://www.mediawiki.org/wiki/Multimedia/About_Media_Viewer
(2) https://en.wikipedia.org/wiki/Wikipedia:Media_Viewer/June_2014_RfC
(3) https://de.wikipedia.org/wiki/Wikipedia:Meinungsbilder/Medienbetrachter
(4) https://commons.wikimedia.org/wiki/Commons:Requests_for_comment/Media_Viewer...
(5) https://wikimedia.mingle.thoughtworks.com/projects/multimedia/cards/787
(6) http://pauginer.github.io/prototypes/media-viewer/desk/disabling-settings/in...
(7) https://wikimedia.mingle.thoughtworks.com/projects/multimedia/cards/793
_______________________________
Fabrice Florin
Product Manager, Multimedia
Wikimedia Foundation
https://www.mediawiki.org/wiki/User:Fabrice_Florin_(WMF)
_______________________________________________ Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
_______________________________________________ Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
A/B testing for major interface changes isn't such a good idea, I think. It's great for small things like login fields that people aren't likely to notice, but once you get into major interface elements you're going to get quite a few people confused or frustrated.
"Why is your Wikipedia different to mine?", "Why does this not look like the help page says?", etc.
I'm not sure the side-effects would be worth the data.
Andrew.
On 16 July 2014 13:01, Erik Zachte ezachte@wikimedia.org wrote:
Disclaimer: I haven't been following the discussion
Full disclosure: 1) I personally like the new feature 2) I value community opinion a lot
Just my 2 cents on methodology:
It seems to me that if we could present half of a target population with old and half with new settings from the outset (e.g. by focussing on new users only), then the outcome would be more convincing.
In the current proposal if 40% would choose to switch , is that because 60% like the new settings better, or because they can't be bothered at that very moment to explore and make up their mind, or for any other reason cling to what they are offered initially.
Erik
From: analytics-bounces@lists.wikimedia.org [mailto:analytics-bounces@lists.wikimedia.org] On Behalf Of Gilles Dubuc Sent: Wednesday, July 16, 2014 14:00 To: A mailing list for the Analytics Team at WMF and everybody who has an interest in Wikipedia and analytics. Subject: Re: [Analytics] Media Viewer User Preference Data
Could we sample only logged-in users?
I think the idea is to measure both separately, because the default might end up being different for logged-in users and for anons, depending on the preference ratio for each.
On Wed, Jul 16, 2014 at 7:29 AM, Nuria Ruiz nuria@wikimedia.org wrote:
a) aim to track total users who enable/disable Media Viewer, rather than just events
b) switch to a 3-state preference setting: enabled / disabled / default
c) try to measure the total number of users in each group (instead of daily events)
I assume we are talking about logging stuff for logged in users and not logged in users.
If that is the case this strategy will dwarf editor data with reader data, in the RFC most concerns seem to be about editor workflow. Could we sample only logged-in users? That way we will get a sample of data that better represents editor's interaction with media viewer and it will be a lot easier to quantify our results.
Once we figure out a practical way to implement this request together, we will present it to community members to >discuss its impact on the RfC, and agree on acceptance criteria (e.g. keep Media Viewer enabled by default unless >a 51% majority of users disables the feature on Enwiki by September 30?).
Perhaps if we want the analytics to impact the RFC we should work with the community to come up with a survey strategy they find aceptable?
On Tue, Jul 15, 2014 at 11:59 PM, Fabrice Florin fflorin@wikimedia.org wrote:
Dear Analytics Team,
As discussed with Toby, Dario and Aaron, we would be grateful for your guidance to collect user preference data for Media Viewer (1), which is being challenged in RfCs on English (2) and German (3) Wikipedias, as well as on Wikimedia Commons (4).
The English RfC closed last week, requesting that we disable Media Viewer right away. The multimedia team, in consultation with community and legal teams, is not comfortable disabling the feature on the basis of this small RfC, which we believe does not accurately represent the views of the millions of users whom we serve.
To address this issue, we would like to conduct a wider outreach to collect more accurate data about user preferences than either this RfC or our optional surveys can provide. We propose to develop a more prominent viewing options panel (5) that would make it very easy to switch quickly to your preferred viewing mode, as shown in this prototype (6). Right after launch of this new feature, all users would be shown this prominent panel and asked to select their favorite viewing option. Once we have collected and analyzed that data over a period of a month or two, we would be able to make more informed decisions with our community on whether or not to keep the feature enabled -- based on actual responses from all users, rather than speculation.
With your help, we have already developed a basic dashboard that tracks Media Viewer opt-in/out events (8), based on clicks on the less prominent ‘Disable’ feature at the bottom of the metadata panel. We propose to modify this dashboard to support this new initiative, as outlined in this ticket (7), with these changes:
a) aim to track total users who enable/disable Media Viewer, rather than just events
b) switch to a 3-state preference setting: enabled / disabled / default
c) try to measure the total number of users in each group (instead of daily events)
Also note that most users would start in the default state, showing them Media Viewer, since it is enabled by default — but after 10 or so image views, the viewing options panel would appear automatically, asking them to either enable or disable the feature. After they have made their selection, the panel would remain accessible (but much less prominent) and they can still switch state (but can't go back to default).
We could use your guidance on the right way of logging total logged-out users, so we can use them as a basis for percentages: should we log enabled/disabled state on some specified event/action: page load? thumbnail click? first site visit of the day? — or do you recommend another method? We would want to collect this data for a month or two, so we don’t only capture the immediate responses from the most active users, but also those of less frequent visitors.
We would also appreciate your comments on the specific tickets (5) and (6), with any recommendations for improvement. Keep in mind that we would like to respond quickly to our community’s concerns and have limited resources, so we would prefer to get this work done in the next week or two. And it will take a couple months to develop, release and collect enough data -- so even if we start tomorrow, we may not have conclusive data to share until September. So time is of the essence. :)
Gergo is spearheading this project, and may have more technical questions for you. But I wanted to give an overview from a product perspective, before we dive in to the implementation details. Once we figure out a practical way to implement this request together, we will present it to community members to discuss its impact on the RfC, and agree on acceptance criteria (e.g. keep Media Viewer enabled by default unless a 51% majority of users disables the feature on Enwiki by September 30?).
Thanks in advance for your advice. We look forward to working with you soon on this important project, which could impact other features now in development.
All the best,
Fabrice
(1) https://www.mediawiki.org/wiki/Multimedia/About_Media_Viewer
(2) https://en.wikipedia.org/wiki/Wikipedia:Media_Viewer/June_2014_RfC
(3) https://de.wikipedia.org/wiki/Wikipedia:Meinungsbilder/Medienbetrachter
(4) https://commons.wikimedia.org/wiki/Commons:Requests_for_comment/Media_Viewer...
(5) https://wikimedia.mingle.thoughtworks.com/projects/multimedia/cards/787
(6) http://pauginer.github.io/prototypes/media-viewer/desk/disabling-settings/in...
(7) https://wikimedia.mingle.thoughtworks.com/projects/multimedia/cards/793
Fabrice Florin
Product Manager, Multimedia
Wikimedia Foundation
https://www.mediawiki.org/wiki/User:Fabrice_Florin_(WMF)
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
On Thu, Jul 17, 2014 at 4:24 AM, Andrew Gray andrew.gray@dunelm.org.uk wrote:
A/B testing for major interface changes isn't such a good idea, I think. It's great for small things like login fields that people aren't likely to notice, but once you get into major interface elements you're going to get quite a few people confused or frustrated.
"Why is your Wikipedia different to mine?", "Why does this not look like the help page says?", etc.
Regarding help pages ...
It is safe to say that a decent segment of the editing community wants to keep the old interface, and the old interface is needed to edit the page, and I think it is safe to assume that the MediaViewer isnt going away - at worst it will go back to being a Beta Feature that is enabled for a large segment of the userbase.
Therefore, the help pages need to cover both. This is a cost of introducing a feature that isnt a complete replacement of prior functionality - and has a high maintenance cost that the devs and community must bear. The solution to that is to hold the feature in Beta until it is an acceptable replacement for the existing functionality.
If this A/B testing goes ahead, and I think it should, the relevant help pages should have a note at the top that explains that testing is occurring and their user pref may have been preset to either one.
Happy to see methodology being discussed. When WMF has worked out a method that it thinks makes sense, please discuss with the community at https://en.wikipedia.org/wiki/Wikipedia_talk:Media_Viewer/June_2014_RfC. You may also wish to engage with Commons and German Wikipedia.
Thanks,
Pine
On Wed, Jul 16, 2014 at 4:34 PM, John Mark Vandenberg jayvdb@gmail.com wrote:
On Thu, Jul 17, 2014 at 4:24 AM, Andrew Gray andrew.gray@dunelm.org.uk wrote:
A/B testing for major interface changes isn't such a good idea, I think. It's great for small things like login fields that people aren't likely to notice, but once you get into major interface elements you're going to get quite a few people confused or frustrated.
"Why is your Wikipedia different to mine?", "Why does this not look like the help page says?", etc.
Regarding help pages ...
It is safe to say that a decent segment of the editing community wants to keep the old interface, and the old interface is needed to edit the page, and I think it is safe to assume that the MediaViewer isnt going away - at worst it will go back to being a Beta Feature that is enabled for a large segment of the userbase.
Therefore, the help pages need to cover both. This is a cost of introducing a feature that isnt a complete replacement of prior functionality - and has a high maintenance cost that the devs and community must bear. The solution to that is to hold the feature in Beta until it is an acceptable replacement for the existing functionality.
If this A/B testing goes ahead, and I think it should, the relevant help pages should have a note at the top that explains that testing is occurring and their user pref may have been preset to either one.
-- John Vandenberg
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
On Wed, Jul 16, 2014 at 5:01 AM, Erik Zachte ezachte@wikimedia.org wrote:
It seems to me that if we could present half of a target population with old and half with new settings from the outset (e.g. by focussing on new users only), then the outcome would be more convincing.
Disclaimer: I have not been following the discussion in other threads. I am solely commenting on methodology here.
+1 for splitting the population and focusing on those who have not seen Media Viewer by default yet. Depending on the nature of the concerns, we may want to choose the two populations differently though. For example, if the concern is that very active editors have experienced difficulty in their work flow, then the sampling should take into account the frequency of edits by editors. In that case, one possible split is this: group I-> editors who register in August 2014 and do more than n (to be specified) edits in August and September 2014; group II -> all other editors who register in August 2014.
The above is just to address editors' concerns. We need a different split mechanism for readers.
Leila
In the current proposal if 40% would choose to switch , is that because 60% like the new settings better, or because they can't be bothered at that very moment to explore and make up their mind, or for any other reason cling to what they are offered initially.
Erik
*From:* analytics-bounces@lists.wikimedia.org [mailto: analytics-bounces@lists.wikimedia.org] *On Behalf Of *Gilles Dubuc *Sent:* Wednesday, July 16, 2014 14:00 *To:* A mailing list for the Analytics Team at WMF and everybody who has an interest in Wikipedia and analytics. *Subject:* Re: [Analytics] Media Viewer User Preference Data
Could we sample only logged-in users?
I think the idea is to measure both separately, because the default might end up being different for logged-in users and for anons, depending on the preference ratio for each.
On Wed, Jul 16, 2014 at 7:29 AM, Nuria Ruiz nuria@wikimedia.org wrote:
a) aim to track total users who enable/disable Media Viewer, rather than
just events
b) switch to a 3-state preference setting: enabled / disabled / default
c) try to measure the total number of users in each group (instead of
daily events)
I assume we are talking about logging stuff for logged in users and not logged in users.
If that is the case this strategy will dwarf editor data with reader data, in the RFC most concerns seem to be about editor workflow. Could we sample only logged-in users? That way we will get a sample of data that better represents editor's interaction with media viewer and it will be a lot easier to quantify our results.
Once we figure out a practical way to implement this request together,
we will present it to community members to >discuss its impact on the RfC, and agree on acceptance criteria (e.g. keep Media Viewer enabled by default unless >a 51% majority of users disables the feature on Enwiki by September 30?).
Perhaps if we want the analytics to impact the RFC we should work with the community to come up with a survey strategy they find aceptable?
On Tue, Jul 15, 2014 at 11:59 PM, Fabrice Florin fflorin@wikimedia.org wrote:
Dear Analytics Team,
As discussed with Toby, Dario and Aaron, we would be grateful for your guidance to collect user preference data for Media Viewer (1), which is being challenged in RfCs on English (2) and German (3) Wikipedias, as well as on Wikimedia Commons (4).
The English RfC closed last week, requesting that we disable Media Viewer right away. The multimedia team, in consultation with community and legal teams, is not comfortable disabling the feature on the basis of this small RfC, which we believe does not accurately represent the views of the millions of users whom we serve.
To address this issue, we would like to conduct a wider outreach to collect more accurate data about user preferences than either this RfC or our optional surveys can provide. We propose to develop a more prominent viewing options panel (5) that would make it very easy to switch quickly to your preferred viewing mode, as shown in this prototype (6). Right after launch of this new feature, all users would be shown this prominent panel and asked to select their favorite viewing option. Once we have collected and analyzed that data over a period of a month or two, we would be able to make more informed decisions with our community on whether or not to keep the feature enabled -- based on actual responses from all users, rather than speculation.
With your help, we have already developed a basic dashboard that tracks Media Viewer opt-in/out events (8), based on clicks on the less prominent ‘Disable’ feature at the bottom of the metadata panel. We propose to modify this dashboard to support this new initiative, as outlined in this ticket (7), with these changes:
a) aim to track total users who enable/disable Media Viewer, rather than just events
b) switch to a 3-state preference setting: enabled / disabled / default
c) try to measure the total number of users in each group (instead of daily events)
Also note that most users would start in the default state, showing them Media Viewer, since it is enabled by default — but after 10 or so image views, the viewing options panel would appear automatically, asking them to either enable or disable the feature. After they have made their selection, the panel would remain accessible (but much less prominent) and they can still switch state (but can't go back to default).
We could use your guidance on the right way of logging total logged-out users, so we can use them as a basis for percentages: should we log enabled/disabled state on some specified event/action: page load? thumbnail click? first site visit of the day? — or do you recommend another method? We would want to collect this data for a month or two, so we don’t only capture the immediate responses from the most active users, but also those of less frequent visitors.
We would also appreciate your comments on the specific tickets (5) and (6), with any recommendations for improvement. Keep in mind that we would like to respond quickly to our community’s concerns and have limited resources, so we would prefer to get this work done in the next week or two. And it will take a couple months to develop, release and collect enough data -- so even if we start tomorrow, we may not have conclusive data to share until September. So time is of the essence. :)
Gergo is spearheading this project, and may have more technical questions for you. But I wanted to give an overview from a product perspective, before we dive in to the implementation details. Once we figure out a practical way to implement this request together, we will present it to community members to discuss its impact on the RfC, and agree on acceptance criteria (e.g. keep Media Viewer enabled by default unless a 51% majority of users disables the feature on Enwiki by September 30?).
Thanks in advance for your advice. We look forward to working with you soon on this important project, which could impact other features now in development.
All the best,
Fabrice
(1) https://www.mediawiki.org/wiki/Multimedia/About_Media_Viewer
(2) https://en.wikipedia.org/wiki/Wikipedia:Media_Viewer/June_2014_RfC
(3) https://de.wikipedia.org/wiki/Wikipedia:Meinungsbilder/Medienbetrachter
(4) https://commons.wikimedia.org/wiki/Commons:Requests_for_comment/Media_Viewer...
(5) https://wikimedia.mingle.thoughtworks.com/projects/multimedia/cards/787
(6) http://pauginer.github.io/prototypes/media-viewer/desk/disabling-settings/in...
(7) https://wikimedia.mingle.thoughtworks.com/projects/multimedia/cards/793
Fabrice Florin
Product Manager, Multimedia
Wikimedia Foundation
https://www.mediawiki.org/wiki/User:Fabrice_Florin_(WMF)
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics