The chosen metrics are interesting ones as the sensitivity is high for this experiment though they aren't inherently positive or negative, hence the mentioned ambiguity. Do you track metrics which rather closely track user satisfaction? Perhaps a metric like distinct daily page views per user, or days active per week. Breaking down the metrics by page length (short/med/long) could give interesting results.

The semi-collapsed sections mentioned by Joaquin Oltra Hernandez sounds quite useful. Perhaps the sections could be auto-collapsed or semi-collapsed for longer pages but short pages could remain fully expanded.

--justin

On Tue, Jun 21, 2016 at 1:45 AM, Joaquin Oltra Hernandez <jhernandez@wikimedia.org> wrote:
Great research, thanks for sharing!

I'm looking forward to further diving into more subjective nuance, like the usefulness of each model for different reading use cases (quick fact checking vs exploratory learning for example).

At some point I saw POC mocks of a mix between expanded and collapsed, where the section was collapsed, but it showed a small summary of the section below the title, like a teaser. It would be very interesting to test that kind of design too and see how it fares with the other two.



On Tue, Jun 21, 2016 at 10:21 AM, Tilman Bayer <tbayer@wikimedia.org> wrote:
Hi,

as most on this list will be aware, on the mobile web version of
Wikipedia, all top-level sections below the lead section are currently
shown collapsed on initial view. Users can tap on a section heading to
show the content, and to collapse it again.
To examine the tradeoffs of this solution and inform future product
decisions, we ran an experiment where 0.05% of mobile web users were
shown all pages with every section expanded on initial load,
instrumented alongside a control group of 0.05% that kept seeing the
standard view where all sections all initially collapsed.

A high-level summary of results is now available at
https://meta.wikimedia.org/wiki/Research:Collapsed_vs_uncollapsed_section_view_on_mobile_web
 . In particular:

* Readers in the test group (sections expanded) tend to stay longer on the page
* Readers in the test group tend to spend more time reading, and less
time navigating
* Readers in the test group tend to scroll more sections into view
than readers in the control group open
* Readers in the test group tend to stay shorter on the page than
readers using the Android Wikipedia app (which offers a TOC for easier
navigation, something not yet available in the mobile web test group)

Comments and questions are welcome, feel free to use the talk page for them too.

Note that this experiment only measured some aspects, and that the
results don't yet allow the unambiguous conclusion that it would be
better to switch to the uncollapsed view. That said, they certainly
suggest that such a change should be considered. It is being planned
to examine this question further with some user testing sessions.

(As an experiment, I've taken the opportunity to write this up this
analysis as a page in the research namespace on Meta, instead of on
Phabricator or in form of an email as done on other occasions.
Feedback on the format is welcome too.)
--
Tilman Bayer
Senior Analyst
Wikimedia Foundation
IRC (Freenode): HaeB



_______________________________________________
Mobile-l mailing list
Mobile-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mobile-l