The chosen metrics are interesting ones as the sensitivity is high for this experiment though they aren't inherently positive or negative, hence the mentioned ambiguity. Do you track metrics which rather closely track user satisfaction? Perhaps a metric like distinct daily page views per user, or days active per week. Breaking down the metrics by page length (short/med/long) could give interesting results. The semi-collapsed sections mentioned by Joaquin Oltra Hernandez sounds quite useful. Perhaps the sections could be auto-collapsed or semi-collapsed for longer pages but short pages could remain fully expanded. --justin On Tue, Jun 21, 2016 at 1:45 AM, Joaquin Oltra Hernandez < jhernandez(a)wikimedia.org&gt; wrote:

Certainly, I hear more often from people I talk to that one of their big gripes with the mobile site is inability to use find in page when sections are collapsed. I know at least 3 people who open all sections just to use this function. Maybe I'm missing something, but based on your results it seems we should be moving towards expanded sections by default as soon as possible. There certainly doesn't seem anything negative here. If not what would you suggest as next steps before doing that. the page This seems like a good result to me. * Readers in the test group tend to spend more time reading, and less time navigating sure what you mean by less time navigating though - is this less navigating to another page or to another section (the latter seems a given if sections are already expanded) * Readers in the test group tend to scroll more sections into view than readers in the control group open again this seems to be a good result. * Readers in the test group tend to stay shorter on the page than readers using the Android Wikipedia app (which offers a TOC for easier navigation, something not yet available in the mobile web test group) This doesn't seem like a fair comparison unless you compare readers outside the test group to the Android app and talk about the delta. On Tue, Jun 21, 2016 at 1:38 PM, Tilman Bayer &lt;tbayer(a)wikimedia.org&gt; wrote: > On Tue, Jun 21, 2016 at 3:17 AM, Justin Ormont &lt;justin.ormont(a)gmail.com&gt; > wrote: > > The chosen metrics are interesting ones as the sensitivity is high for > this > > experiment though they aren't inherently positive or negative, hence the > > mentioned ambiguity. > Yes, so as mentioned in the writeup, readers in the test group could > of course be spending more time on the page simply because they need > longer to navigate to the desired part. But the second result > addresses this particular concern. > > Do you track metrics which rather closely track user > > satisfaction? Perhaps a metric like distinct daily page views per user, > or > > days active per week. > The experiment was designed to also measure the number of pages viewed > per browser session. Sadly, it turned out afterwards during data > analysis that that part of the instrumentation had been buggy > (different page views by the same reader in the same session were > sometimes in the sampled group, sometimes not), so we don't have this > data. We took a lesson from this and have since been testing other new > instrumentations more thoroughly before deployment. > > > > Breaking down the metrics by page length > > (short/med/long) could give interesting results. > Yes, that's doable in principle, although it would require some extra > effort to join the EventLogging table with a list of page lengths. The > schema is here BTW in case you want to suggest concrete queries: > https://meta.wikimedia.org/wiki/Schema:MobileWebSectionUsage > > > > > The semi-collapsed sections mentioned by Joaquin Oltra Hernandez sounds > > quite useful. Perhaps the sections could be auto-collapsed or > semi-collapsed > > for longer pages but short pages could remain fully expanded. > > > > --justin > > > > On Tue, Jun 21, 2016 at 1:45 AM, Joaquin Oltra Hernandez > > &lt;jhernandez(a)wikimedia.org&gt; wrote: > >> > >> Great research, thanks for sharing! > >> > >> I'm looking forward to further diving into more subjective nuance, like > >> the usefulness of each model for different reading use cases (quick fact > >> checking vs exploratory learning for example). > >> > >> At some point I saw POC mocks of a mix between expanded and collapsed, > >> where the section was collapsed, but it showed a small summary of the > >> section below the title, like a teaser. It would be very interesting to > test > >> that kind of design too and see how it fares with the other two. > >> > >> > >> > >> On Tue, Jun 21, 2016 at 10:21 AM, Tilman Bayer &lt;tbayer(a)wikimedia.org&gt; > >> wrote: > >>> > >>> Hi, > >>> > >>> as most on this list will be aware, on the mobile web version of > >>> Wikipedia, all top-level sections below the lead section are currently > >>> shown collapsed on initial view. Users can tap on a section heading to > >>> show the content, and to collapse it again. > >>> To examine the tradeoffs of this solution and inform future product > >>> decisions, we ran an experiment where 0.05% of mobile web users were > >>> shown all pages with every section expanded on initial load, > >>> instrumented alongside a control group of 0.05% that kept seeing the > >>> standard view where all sections all initially collapsed. > >>> > >>> A high-level summary of results is now available at > >>> > >>> > https://meta.wikimedia.org/wiki/Research:Collapsed_vs_uncollapsed_section_v… > >>> . In particular: > >>> > >>> >>> the page > >>> * Readers in the test group tend to spend more time reading, and less > >>> time navigating > >>> * Readers in the test group tend to scroll more sections into view > >>> than readers in the control group open > >>> * Readers in the test group tend to stay shorter on the page than > >>> readers using the Android Wikipedia app (which offers a TOC for easier > >>> navigation, something not yet available in the mobile web test group) > >>> > >>> Comments and questions are welcome, feel free to use the talk page for > >>> them too. > >>> > >>> Note that this experiment only measured some aspects, and that the > >>> results don't yet allow the unambiguous conclusion that it would be > >>> better to switch to the uncollapsed view. That said, they certainly > >>> suggest that such a change should be considered. It is being planned > >>> to examine this question further with some user testing sessions. > >>> > >>> (As an experiment, I've taken the opportunity to write this up this > >>> analysis as a page in the research namespace on Meta, instead of on > >>> Phabricator or in form of an email as done on other occasions. > >>> Feedback on the format is welcome too.) > >>> -- > >>> Tilman Bayer > >>> Senior Analyst > >>> Wikimedia Foundation > >>> IRC (Freenode): HaeB > >>> > >> > >> > >> _______________________________________________ > >> Mobile-l mailing list > >> Mobile-l(a)lists.wikimedia.org > >> https://lists.wikimedia.org/mailman/listinfo/mobile-l > >> > > > > > > _______________________________________________ > > Mobile-l mailing list > > Mobile-l(a)lists.wikimedia.org > > https://lists.wikimedia.org/mailman/listinfo/mobile-l > > > > > > -- > Tilman Bayer > Senior Analyst > Wikimedia Foundation > IRC (Freenode): HaeB > > _______________________________________________ > Mobile-l mailing list > Mobile-l(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/mobile-l >