[Analytics] How best to accurately record page interactions in Page Previews

Hi Leila, On Wed, Jan 17, 2018 at 10:46 AM, Leila Zia <leila(a)wikimedia.org> wrote: This is a good question, sorry that it got buried earlier. (It's kind of orthogonal though to the technical instrumentation questions that have been the focus of attention: as indicated by the capital X in Sam's initial post, we can still decide to fine-tune that threshold right now, it's just a parameter change.) This kind of threshold necessarily needs to be set somewhat arbitrarily, in the sense that there will always be either cases where some content was already read/perceived in a preview card shown for a shorter time, or cases where a reader needed a longer time to consume any content from the card. We picked a time by which we can be reasonably certain that at least some readers can consume content (read some words, perceive an image). It's not the result of an exact calculation to find the provably best limit. But we did have look at the frequency of the different user actions over time during the first seconds after they start to hover over a link. In case you're interested, I recently updated those charts with better quality data from our latest two tests, e.g: https://phabricator.wikimedia.org/F12940888 https://phabricator.wikimedia.org/F13134460 (a zoomed-in look at the same histogram) The following is just eyeballing and thinking aloud, but one way to view this histogram is as the sum of several distributions associated with different user intentions: 1. Most of the time when our instrumentation registered the cursor moving over a link, the user was just on their way to a different part of the screen (with no intention of either clicking that link or viewing the preview). That's mostly the huge yellow spike on the left - "dwelledButAbandoned" meaning that the cursor left the link without either clicking it or causing a preview to show. The feature involves a 500ms delay before the preview card begins to display, so that we don't bother that group too much. (Only the right tail end of that distribution, folks moving the cursor very slowly, will be affected, where things morph from yellow into purple.) 2. Then there are users who want to click the link without viewing the preview, forming all of the green part left of 500ms and an unknown portion to the right of it (after the card starts to show, some of these "open" actions will instead happen after the user intentionally viewed the card, case 3.). 3. And there are users who intentionally view a preview. The little bump in the purple part ("dismissed" meaning that the preview was shown and then closed by moving the cursor away) at about 1100ms indicates that the distribution for that user group also peaks somewhere there, maybe a few 100ms to the right. That would mean that our 1000ms threshold (i.e. only counting the part of the histogram right of 1500ms = 500ms + 1000ms as seen previews) is actually right of that distribution's peak. I.e. that the threshold is in some sense quite conservative. Like I said, this is all of course still a bit handwavy; it involves some assumptions about the form of these distributions, as well as disregarding some other information for now that can give a fuller picture (in particular the analogous histogram for link interaction behavior without page previews being active, which we also have from our A/B tests). We did an analysis of that user behavior, but not regarding the timing question; rather, it was about finding out how much of the reduction in pageviews comes from reduced usage of the back button. I'm not sure how directly we can compare the action of loading an entire new page and then going back (two clicks that also involve moving the mouse cursor to an entirely different part of the screen - the back button - inbetween) with the action of hovering over a link and then moving the cursor away for a small distance to close the preview; it seems to me that the latter involves much less friction - which is kind of the whole point of the previews feature ;) As indicated, we already picked a value for the threshold that we are quite comfortable with. But if you are still interested in this question and have some spare time, I'm more than happy to chat about it further off-list. -- Tilman Bayer Senior Analyst Wikimedia Foundation IRC (Freenode): HaeB