Hi,
as most on this list will be aware, on the mobile web version of Wikipedia, all top-level sections below the lead section are currently shown collapsed on initial view. Users can tap on a section heading to show the content, and to collapse it again. To examine the tradeoffs of this solution and inform future product decisions, we ran an experiment where 0.05% of mobile web users were shown all pages with every section expanded on initial load, instrumented alongside a control group of 0.05% that kept seeing the standard view where all sections all initially collapsed.
A high-level summary of results is now available at https://meta.wikimedia.org/wiki/Research:Collapsed_vs_uncollapsed_section_vi... . In particular:
* Readers in the test group (sections expanded) tend to stay longer on the page * Readers in the test group tend to spend more time reading, and less time navigating * Readers in the test group tend to scroll more sections into view than readers in the control group open * Readers in the test group tend to stay shorter on the page than readers using the Android Wikipedia app (which offers a TOC for easier navigation, something not yet available in the mobile web test group)
Comments and questions are welcome, feel free to use the talk page for them too.
Note that this experiment only measured some aspects, and that the results don't yet allow the unambiguous conclusion that it would be better to switch to the uncollapsed view. That said, they certainly suggest that such a change should be considered. It is being planned to examine this question further with some user testing sessions.
(As an experiment, I've taken the opportunity to write this up this analysis as a page in the research namespace on Meta, instead of on Phabricator or in form of an email as done on other occasions. Feedback on the format is welcome too.)
Great research, thanks for sharing!
I'm looking forward to further diving into more subjective nuance, like the usefulness of each model for different reading use cases (quick fact checking vs exploratory learning for example).
At some point I saw POC mocks of a mix between expanded and collapsed, where the section was collapsed, but it showed a small summary of the section below the title, like a teaser. It would be very interesting to test that kind of design too and see how it fares with the other two.
On Tue, Jun 21, 2016 at 10:21 AM, Tilman Bayer tbayer@wikimedia.org wrote:
Hi,
as most on this list will be aware, on the mobile web version of Wikipedia, all top-level sections below the lead section are currently shown collapsed on initial view. Users can tap on a section heading to show the content, and to collapse it again. To examine the tradeoffs of this solution and inform future product decisions, we ran an experiment where 0.05% of mobile web users were shown all pages with every section expanded on initial load, instrumented alongside a control group of 0.05% that kept seeing the standard view where all sections all initially collapsed.
A high-level summary of results is now available at
https://meta.wikimedia.org/wiki/Research:Collapsed_vs_uncollapsed_section_vi... . In particular:
- Readers in the test group (sections expanded) tend to stay longer on the
page
- Readers in the test group tend to spend more time reading, and less
time navigating
- Readers in the test group tend to scroll more sections into view
than readers in the control group open
- Readers in the test group tend to stay shorter on the page than
readers using the Android Wikipedia app (which offers a TOC for easier navigation, something not yet available in the mobile web test group)
Comments and questions are welcome, feel free to use the talk page for them too.
Note that this experiment only measured some aspects, and that the results don't yet allow the unambiguous conclusion that it would be better to switch to the uncollapsed view. That said, they certainly suggest that such a change should be considered. It is being planned to examine this question further with some user testing sessions.
(As an experiment, I've taken the opportunity to write this up this analysis as a page in the research namespace on Meta, instead of on Phabricator or in form of an email as done on other occasions. Feedback on the format is welcome too.) -- Tilman Bayer Senior Analyst Wikimedia Foundation IRC (Freenode): HaeB
The chosen metrics are interesting ones as the sensitivity is high for this experiment though they aren't inherently positive or negative, hence the mentioned ambiguity. Do you track metrics which rather closely track user satisfaction? Perhaps a metric like distinct daily page views per user, or days active per week. Breaking down the metrics by page length (short/med/long) could give interesting results.
The semi-collapsed sections mentioned by Joaquin Oltra Hernandez sounds quite useful. Perhaps the sections could be auto-collapsed or semi-collapsed for longer pages but short pages could remain fully expanded.
--justin
On Tue, Jun 21, 2016 at 1:45 AM, Joaquin Oltra Hernandez < jhernandez@wikimedia.org> wrote:
Great research, thanks for sharing!
I'm looking forward to further diving into more subjective nuance, like the usefulness of each model for different reading use cases (quick fact checking vs exploratory learning for example).
At some point I saw POC mocks of a mix between expanded and collapsed, where the section was collapsed, but it showed a small summary of the section below the title, like a teaser. It would be very interesting to test that kind of design too and see how it fares with the other two.
On Tue, Jun 21, 2016 at 10:21 AM, Tilman Bayer tbayer@wikimedia.org wrote:
Hi,
as most on this list will be aware, on the mobile web version of Wikipedia, all top-level sections below the lead section are currently shown collapsed on initial view. Users can tap on a section heading to show the content, and to collapse it again. To examine the tradeoffs of this solution and inform future product decisions, we ran an experiment where 0.05% of mobile web users were shown all pages with every section expanded on initial load, instrumented alongside a control group of 0.05% that kept seeing the standard view where all sections all initially collapsed.
A high-level summary of results is now available at
https://meta.wikimedia.org/wiki/Research:Collapsed_vs_uncollapsed_section_vi... . In particular:
- Readers in the test group (sections expanded) tend to stay longer on
the page
- Readers in the test group tend to spend more time reading, and less
time navigating
- Readers in the test group tend to scroll more sections into view
than readers in the control group open
- Readers in the test group tend to stay shorter on the page than
readers using the Android Wikipedia app (which offers a TOC for easier navigation, something not yet available in the mobile web test group)
Comments and questions are welcome, feel free to use the talk page for them too.
Note that this experiment only measured some aspects, and that the results don't yet allow the unambiguous conclusion that it would be better to switch to the uncollapsed view. That said, they certainly suggest that such a change should be considered. It is being planned to examine this question further with some user testing sessions.
(As an experiment, I've taken the opportunity to write this up this analysis as a page in the research namespace on Meta, instead of on Phabricator or in form of an email as done on other occasions. Feedback on the format is welcome too.) -- Tilman Bayer Senior Analyst Wikimedia Foundation IRC (Freenode): HaeB
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
On Tue, Jun 21, 2016 at 3:17 AM, Justin Ormont justin.ormont@gmail.com wrote:
The chosen metrics are interesting ones as the sensitivity is high for this experiment though they aren't inherently positive or negative, hence the mentioned ambiguity.
Yes, so as mentioned in the writeup, readers in the test group could of course be spending more time on the page simply because they need longer to navigate to the desired part. But the second result addresses this particular concern.
Do you track metrics which rather closely track user satisfaction? Perhaps a metric like distinct daily page views per user, or days active per week.
The experiment was designed to also measure the number of pages viewed per browser session. Sadly, it turned out afterwards during data analysis that that part of the instrumentation had been buggy (different page views by the same reader in the same session were sometimes in the sampled group, sometimes not), so we don't have this data. We took a lesson from this and have since been testing other new instrumentations more thoroughly before deployment.
Breaking down the metrics by page length (short/med/long) could give interesting results.
Yes, that's doable in principle, although it would require some extra effort to join the EventLogging table with a list of page lengths. The schema is here BTW in case you want to suggest concrete queries: https://meta.wikimedia.org/wiki/Schema:MobileWebSectionUsage
The semi-collapsed sections mentioned by Joaquin Oltra Hernandez sounds quite useful. Perhaps the sections could be auto-collapsed or semi-collapsed for longer pages but short pages could remain fully expanded.
--justin
On Tue, Jun 21, 2016 at 1:45 AM, Joaquin Oltra Hernandez jhernandez@wikimedia.org wrote:
Great research, thanks for sharing!
I'm looking forward to further diving into more subjective nuance, like the usefulness of each model for different reading use cases (quick fact checking vs exploratory learning for example).
At some point I saw POC mocks of a mix between expanded and collapsed, where the section was collapsed, but it showed a small summary of the section below the title, like a teaser. It would be very interesting to test that kind of design too and see how it fares with the other two.
On Tue, Jun 21, 2016 at 10:21 AM, Tilman Bayer tbayer@wikimedia.org wrote:
Hi,
as most on this list will be aware, on the mobile web version of Wikipedia, all top-level sections below the lead section are currently shown collapsed on initial view. Users can tap on a section heading to show the content, and to collapse it again. To examine the tradeoffs of this solution and inform future product decisions, we ran an experiment where 0.05% of mobile web users were shown all pages with every section expanded on initial load, instrumented alongside a control group of 0.05% that kept seeing the standard view where all sections all initially collapsed.
A high-level summary of results is now available at
https://meta.wikimedia.org/wiki/Research:Collapsed_vs_uncollapsed_section_vi... . In particular:
- Readers in the test group (sections expanded) tend to stay longer on
the page
- Readers in the test group tend to spend more time reading, and less
time navigating
- Readers in the test group tend to scroll more sections into view
than readers in the control group open
- Readers in the test group tend to stay shorter on the page than
readers using the Android Wikipedia app (which offers a TOC for easier navigation, something not yet available in the mobile web test group)
Comments and questions are welcome, feel free to use the talk page for them too.
Note that this experiment only measured some aspects, and that the results don't yet allow the unambiguous conclusion that it would be better to switch to the uncollapsed view. That said, they certainly suggest that such a change should be considered. It is being planned to examine this question further with some user testing sessions.
(As an experiment, I've taken the opportunity to write this up this analysis as a page in the research namespace on Meta, instead of on Phabricator or in form of an email as done on other occasions. Feedback on the format is welcome too.) -- Tilman Bayer Senior Analyst Wikimedia Foundation IRC (Freenode): HaeB
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Certainly, I hear more often from people I talk to that one of their big gripes with the mobile site is inability to use find in page when sections are collapsed. I know at least 3 people who open all sections just to use this function. Maybe I'm missing something, but based on your results it seems we should be moving towards expanded sections by default as soon as possible. There certainly doesn't seem anything negative here. If not what would you suggest as next steps before doing that.
- Readers in the test group (sections expanded) tend to stay longer on
the page This seems like a good result to me.
* Readers in the test group tend to spend more time reading, and less time navigating
Again this seems a good result for sections expanded. I'm not entirely
sure what you mean by less time navigating though - is this less navigating to another page or to another section (the latter seems a given if sections are already expanded)
* Readers in the test group tend to scroll more sections into view than readers in the control group open
again this seems to be a good result.
* Readers in the test group tend to stay shorter on the page than readers using the Android Wikipedia app (which offers a TOC for easier navigation, something not yet available in the mobile web test group)
This doesn't seem like a fair comparison unless you compare readers outside the test group to the Android app and talk about the delta.
On Tue, Jun 21, 2016 at 1:38 PM, Tilman Bayer tbayer@wikimedia.org wrote:
On Tue, Jun 21, 2016 at 3:17 AM, Justin Ormont justin.ormont@gmail.com wrote:
The chosen metrics are interesting ones as the sensitivity is high for
this
experiment though they aren't inherently positive or negative, hence the mentioned ambiguity.
Yes, so as mentioned in the writeup, readers in the test group could of course be spending more time on the page simply because they need longer to navigate to the desired part. But the second result addresses this particular concern.
Do you track metrics which rather closely track user satisfaction? Perhaps a metric like distinct daily page views per user,
or
days active per week.
The experiment was designed to also measure the number of pages viewed per browser session. Sadly, it turned out afterwards during data analysis that that part of the instrumentation had been buggy (different page views by the same reader in the same session were sometimes in the sampled group, sometimes not), so we don't have this data. We took a lesson from this and have since been testing other new instrumentations more thoroughly before deployment.
Breaking down the metrics by page length (short/med/long) could give interesting results.
Yes, that's doable in principle, although it would require some extra effort to join the EventLogging table with a list of page lengths. The schema is here BTW in case you want to suggest concrete queries: https://meta.wikimedia.org/wiki/Schema:MobileWebSectionUsage
The semi-collapsed sections mentioned by Joaquin Oltra Hernandez sounds quite useful. Perhaps the sections could be auto-collapsed or
semi-collapsed
for longer pages but short pages could remain fully expanded.
--justin
On Tue, Jun 21, 2016 at 1:45 AM, Joaquin Oltra Hernandez jhernandez@wikimedia.org wrote:
Great research, thanks for sharing!
I'm looking forward to further diving into more subjective nuance, like the usefulness of each model for different reading use cases (quick fact checking vs exploratory learning for example).
At some point I saw POC mocks of a mix between expanded and collapsed, where the section was collapsed, but it showed a small summary of the section below the title, like a teaser. It would be very interesting to
test
that kind of design too and see how it fares with the other two.
On Tue, Jun 21, 2016 at 10:21 AM, Tilman Bayer tbayer@wikimedia.org wrote:
Hi,
as most on this list will be aware, on the mobile web version of Wikipedia, all top-level sections below the lead section are currently shown collapsed on initial view. Users can tap on a section heading to show the content, and to collapse it again. To examine the tradeoffs of this solution and inform future product decisions, we ran an experiment where 0.05% of mobile web users were shown all pages with every section expanded on initial load, instrumented alongside a control group of 0.05% that kept seeing the standard view where all sections all initially collapsed.
A high-level summary of results is now available at
https://meta.wikimedia.org/wiki/Research:Collapsed_vs_uncollapsed_section_vi...
. In particular:
- Readers in the test group (sections expanded) tend to stay longer on
the page
- Readers in the test group tend to spend more time reading, and less
time navigating
- Readers in the test group tend to scroll more sections into view
than readers in the control group open
- Readers in the test group tend to stay shorter on the page than
readers using the Android Wikipedia app (which offers a TOC for easier navigation, something not yet available in the mobile web test group)
Comments and questions are welcome, feel free to use the talk page for them too.
Note that this experiment only measured some aspects, and that the results don't yet allow the unambiguous conclusion that it would be better to switch to the uncollapsed view. That said, they certainly suggest that such a change should be considered. It is being planned to examine this question further with some user testing sessions.
(As an experiment, I've taken the opportunity to write this up this analysis as a page in the research namespace on Meta, instead of on Phabricator or in form of an email as done on other occasions. Feedback on the format is welcome too.) -- Tilman Bayer Senior Analyst Wikimedia Foundation IRC (Freenode): HaeB
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- Tilman Bayer Senior Analyst Wikimedia Foundation IRC (Freenode): HaeB
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
On Mon, Jul 4, 2016 at 11:58 AM, Jon Robson jrobson@wikimedia.org wrote:
Certainly, I hear more often from people I talk to that one of their big gripes with the mobile site is inability to use find in page when sections are collapsed. I know at least 3 people who open all sections just to use this function. Maybe I'm missing something, but based on your results it seems we should be moving towards expanded sections by default as soon as possible. There certainly doesn't seem anything negative here. If not what would you suggest as next steps before doing that.
- Readers in the test group (sections expanded) tend to stay longer on the
page
This seems like a good result to me.
- Readers in the test group tend to spend more time reading, and less
time navigating
Again this seems a good result for sections expanded. I'm not entirely sure what you mean by less time navigating though - is this less navigating to another page or to another section (the latter seems a given if sections are already expanded)
See the aforementioned page on Meta for definitions, or the linked Phabricator task for the nitty-gritty details including the exact queries used. The posting here was basically just the TLDR of the summary ;)
- Readers in the test group tend to scroll more sections into view
than readers in the control group open
again this seems to be a good result.
- Readers in the test group tend to stay shorter on the page than
readers using the Android Wikipedia app (which offers a TOC for easier navigation, something not yet available in the mobile web test group)
This doesn't seem like a fair comparison unless you compare readers outside the test group to the Android app and talk about the delta.
Again, see the Meta page for a discussion of limitations of this comparisons. And I would say that the test condition was clearly still more comparable to the Android app reading experience (because both are showing the full content initially) than the control group.
On Tue, Jun 21, 2016 at 1:38 PM, Tilman Bayer tbayer@wikimedia.org wrote:
On Tue, Jun 21, 2016 at 3:17 AM, Justin Ormont justin.ormont@gmail.com wrote:
The chosen metrics are interesting ones as the sensitivity is high for this experiment though they aren't inherently positive or negative, hence the mentioned ambiguity.
Yes, so as mentioned in the writeup, readers in the test group could of course be spending more time on the page simply because they need longer to navigate to the desired part. But the second result addresses this particular concern.
Do you track metrics which rather closely track user satisfaction? Perhaps a metric like distinct daily page views per user, or days active per week.
The experiment was designed to also measure the number of pages viewed per browser session. Sadly, it turned out afterwards during data analysis that that part of the instrumentation had been buggy (different page views by the same reader in the same session were sometimes in the sampled group, sometimes not), so we don't have this data. We took a lesson from this and have since been testing other new instrumentations more thoroughly before deployment.
Breaking down the metrics by page length (short/med/long) could give interesting results.
Yes, that's doable in principle, although it would require some extra effort to join the EventLogging table with a list of page lengths. The schema is here BTW in case you want to suggest concrete queries: https://meta.wikimedia.org/wiki/Schema:MobileWebSectionUsage
The semi-collapsed sections mentioned by Joaquin Oltra Hernandez sounds quite useful. Perhaps the sections could be auto-collapsed or semi-collapsed for longer pages but short pages could remain fully expanded.
--justin
On Tue, Jun 21, 2016 at 1:45 AM, Joaquin Oltra Hernandez jhernandez@wikimedia.org wrote:
Great research, thanks for sharing!
I'm looking forward to further diving into more subjective nuance, like the usefulness of each model for different reading use cases (quick fact checking vs exploratory learning for example).
At some point I saw POC mocks of a mix between expanded and collapsed, where the section was collapsed, but it showed a small summary of the section below the title, like a teaser. It would be very interesting to test that kind of design too and see how it fares with the other two.
On Tue, Jun 21, 2016 at 10:21 AM, Tilman Bayer tbayer@wikimedia.org wrote:
Hi,
as most on this list will be aware, on the mobile web version of Wikipedia, all top-level sections below the lead section are currently shown collapsed on initial view. Users can tap on a section heading to show the content, and to collapse it again. To examine the tradeoffs of this solution and inform future product decisions, we ran an experiment where 0.05% of mobile web users were shown all pages with every section expanded on initial load, instrumented alongside a control group of 0.05% that kept seeing the standard view where all sections all initially collapsed.
A high-level summary of results is now available at
https://meta.wikimedia.org/wiki/Research:Collapsed_vs_uncollapsed_section_vi... . In particular:
- Readers in the test group (sections expanded) tend to stay longer on
the page
- Readers in the test group tend to spend more time reading, and less
time navigating
- Readers in the test group tend to scroll more sections into view
than readers in the control group open
- Readers in the test group tend to stay shorter on the page than
readers using the Android Wikipedia app (which offers a TOC for easier navigation, something not yet available in the mobile web test group)
Comments and questions are welcome, feel free to use the talk page for them too.
Note that this experiment only measured some aspects, and that the results don't yet allow the unambiguous conclusion that it would be better to switch to the uncollapsed view. That said, they certainly suggest that such a change should be considered. It is being planned to examine this question further with some user testing sessions.
(As an experiment, I've taken the opportunity to write this up this analysis as a page in the research namespace on Meta, instead of on Phabricator or in form of an email as done on other occasions. Feedback on the format is welcome too.) -- Tilman Bayer Senior Analyst Wikimedia Foundation IRC (Freenode): HaeB
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- Tilman Bayer Senior Analyst Wikimedia Foundation IRC (Freenode): HaeB
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l