http://searchdata.wmflabs.org/ - boop! This was my Friday. Previously we were playing around with them and testing what we needed with a static snapshot; these dashboards will now update once a day with new information. It has turned up some bugs ("is the mobile schema just not running?") and there are more metrics to add. But for the time being, is progress :) -- Oliver Keyes Research Analyst Wikimedia Foundation _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

wrote:

Awesome. -m. On Fri, May 22, 2015 at 5:35 PM, Oliver Keyes <okeyes(a)wikimedia.org> wrote: _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

This is a very exciting preview of things to come. Where are the data coming from? Am I just confused, or does "6 search sessions per day" seem low? On Fri, May 22, 2015 at 2:35 PM, Oliver Keyes <okeyes(a)wikimedia.org> wrote: _______________________________________________ Wikimedia-search-private mailing list Wikimedia-search-private(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikimedia-search-private

This is a very exciting preview of things to come. Where are the data coming from? Am I just confused, or does "6 search sessions per day" seem low? On Fri, May 22, 2015 at 2:35 PM, Oliver Keyes <okeyes(a)wikimedia.org> wrote: _______________________________________________ Wikimedia-search-private mailing list Wikimedia-search-private(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikimedia-search-private

On Tue, May 26, 2015 at 4:37 PM, Oliver Keyes <okeyes(a)wikimedia.org>

This was brought up as a next step during a number of discussions between the team before I left. Let's focus on what it would take and work with the team to do it. I don't want to present data we have no confidence in but we need to start showcasing the stories that were learning and pushing search forward for our users. Thank you for your work on this Oliver --tomasz _______________________________________________ Wikimedia-search-private mailing list Wikimedia-search-private(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikimedia-search-private

SELECT count(*) FROM Search_11670541 WHERE timestamp >= '20150526' AND

SELECT count(DISTINCT event_userSessionToken) FROM Search_11670541 WHERE

SELECT count(DISTINCT event_userSessionToken) FROM Search_11670541 WHERE

Thanks Tomasz; great feedback! In order: * yeah, top percentiles were a heavily-requested thing so I built it in from the get-go. Similarly, mean/median so we have some ability to avoid distorting the results when the distribution changes. * The 3 days data thing is a known - https://phabricator.wikimedia.org/T100056 - and is next on my to-do list for bugfixes :). * Glad you like the interface! It's actually functional on mobile, too :D. * Sample rate is crucial, yep. I'm reaching out to the authors of the relevant EL schemas to find out how each was handled. * Sessions < results opened makes sense in the event that users didn't find what they wanted and went back to try again, but I'm not sure how "session" is calculated; this is again something we lack transparency around :(. Dan? You're the apps wizard. In supporting this: probably nothing at the moment although Nik/Kevin chipping in on the relevant phabricator ticket (https://phabricator.wikimedia.org/T99762 ) to validate how much of a PITA the idea of a unified schema and the associated implementations are, would be good. I'm sort of shocked to hear "we're supposed to be presenting this data at the next metrics meeting": in the future if there are instances where data is going to be up for public scrutiny, would it be possible to explicitly associate time for that? My goal is to get us to the point where our data is reliable all, or at least, most of the time, and for a fragment of one person's time over two weeks, I think progress on that is pretty fantastic. But prepping data for that kind of event does change the priorities and what tasks should be worked on. If we want to present data, generally speaking, let's discuss what we can show off. If we want to present the dashboards I'll put my all into making the data at least something where we know the deficiencies, if not something where we consider the deficiencies tolerable. On 26 May 2015 at 19:24, Tomasz Finc &lt;tfinc(a)wikimedia.org&gt; wrote: wrote: wrote: -- Oliver Keyes Research Analyst Wikimedia Foundation _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

+Corey Floyd, who recently added code for this stuff on iOS (slated for forthcoming 4.1.4 iOS Wikipedia app release). On Wed, May 27, 2015 at 12:33 PM, Adam Baso &lt;abaso(a)wikimedia.org&gt; wrote: > > Thought I'd step in here. People who know the mechanics of the relevant > logging are as follows: > > Android: Dmitry Brant > Desktop: Bahodir (Baha) Mansurov > Mobile Web: Sam Smith & Baha > > CC'ing them. > > As I understand, Dan Garry's been talking with Baha already on the

> piece. > > It looks like on Chrome/42 UAs the clickthrough rate for a given suggest > search form interaction is about 40%. [1] > > A couple of patches are pending (JS for emitting event on new EL schema) > that will make it possible to figure out how often form submission

> form interaction (ENTER/RETURN, tapping the magnifying class) occurs as > well. > > Total form interactions (keys on userSessionToken...maybe a better name > could be used) minus clickthroughs (click-result) minus form submission > (submit-form in the pending patches on the new EL schema) would be a

> proxy of abandonment, I think. > > I had heard sendBeacon capable UAs were likely to have greater success > emitting the click-result (i.e., when user clicks on a suggestion from

> form on the search panel on desktop) event via mw.track, so it may make > sense to confine queries for such analysis on desktop to known

> browsers [2] to increase the odds of high fidelity data just in case

> are outlier browsers that manage to somehow emit click-result events

> means other than sendBeacon (it seems there may be some of these,

> non-forged UAs). > > -Adam > > [1] > > > SELECT count(*) FROM Search_11670541 WHERE timestamp >= '20150526' AND > > timestamp < '20150527' AND event_action = 'click-result' AND wiki =

> > and userAgent LIKE '%Chrome/42%'; > +----------+ > | count(*) | > +----------+ > | 112 | > +----------+ > 1 row in set (2.96 sec) > > > SELECT count(DISTINCT event_userSessionToken) FROM Search_11670541

> > timestamp >= '20150526' AND timestamp < '20150527' AND event_action = > > 'click-result' AND wiki = 'enwiki' and userAgent LIKE '%Chrome/42%'; > +----------------------------------------+ > | count(DISTINCT event_userSessionToken) | > +----------------------------------------+ > | 112 | > +----------------------------------------+ > 1 row in set (38.06 sec) > > > > SELECT count(DISTINCT event_userSessionToken) FROM Search_11670541

> > timestamp > '20150526' AND timestamp < '20150527' AND wiki = 'enwiki'

> > userAgent LIKE '%Chrome/42%'; > +----------------------------------------+ > | count(DISTINCT event_userSessionToken) | > +----------------------------------------+ > | 286 | > +----------------------------------------+ > 1 row in set (7.26 sec) > > > [2] https://developer.mozilla.org/it/docs/Web/API/Navigator/sendBeacon > > On Tue, May 26, 2015 at 4:37 PM, Oliver Keyes &lt;okeyes(a)wikimedia.org&gt; > wrote: >> >> Thanks Tomasz; great feedback! In order: >> >> * yeah, top percentiles were a heavily-requested thing so I built it >> in from the get-go. Similarly, mean/median so we have some ability to >> avoid distorting the results when the distribution changes. >> * The 3 days data thing is a known - >> https://phabricator.wikimedia.org/T100056 - and is next on my to-do >> list for bugfixes :). >> * Glad you like the interface! It's actually functional on mobile, too >> :D. >> * Sample rate is crucial, yep. I'm reaching out to the authors of the >> relevant EL schemas to find out how each was handled. >> * Sessions < results opened makes sense in the event that users didn't >> find what they wanted and went back to try again, but I'm not sure how >> "session" is calculated; this is again something we lack transparency >> around :(. Dan? You're the apps wizard. >> >> In supporting this: probably nothing at the moment although Nik/Kevin >> chipping in on the relevant phabricator ticket >> (https://phabricator.wikimedia.org/T99762 ) to validate how much of a >> PITA the idea of a unified schema and the associated implementations >> are, would be good. >> >> I'm sort of shocked to hear "we're supposed to be presenting this data >> at the next metrics meeting": in the future if there are instances >> where data is going to be up for public scrutiny, would it be possible >> to explicitly associate time for that? My goal is to get us to the >> point where our data is reliable all, or at least, most of the time, >> and for a fragment of one person's time over two weeks, I think >> progress on that is pretty fantastic. But prepping data for that kind >> of event does change the priorities and what tasks should be worked >> on. >> >> If we want to present data, generally speaking, let's discuss what we >> can show off. If we want to present the dashboards I'll put my all >> into making the data at least something where we know the >> deficiencies, if not something where we consider the deficiencies >> tolerable. >> >> On 26 May 2015 at 19:24, Tomasz Finc &lt;tfinc(a)wikimedia.org&gt; wrote: >> > Thanks Oliver >> > >> > Early observations >> > >> > * Really happy to see top percentiles in load graphs >> > * Mobile Web has only three days data >> > * Interface is simple and easy to use >> > * We need to know the sample rate >> > * Apps have fewer sessions than results page opened >> > >> > Speaking over IRC it's clear that we don't have confidence in this >> > data. We need to fix this and fix it quickly so that we can

>> > plan our work. We're supposed to be presenting this data at the next >> > metrics meeting and we're not a point where I feel comfortable

>> > our data let alone next steps. >> > >> > Oliver & Dan, what can the team do to support you guys on this? I

>> > you guys to own this and know that were here to support you. >> > >> > Should I be adding new feature requests and bugs to >> > https://phabricator.wikimedia.org/tag/search-data-analytics/ ? >> > >> > --tomasz >> > >> > On Tue, May 26, 2015 at 11:04 AM, James Douglas >> > &lt;jdouglas(a)wikimedia.org&gt; wrote: >> >> This is a very exciting preview of things to come. >> >> >> >> Where are the data coming from? Am I just confused, or does "6

>> >> wrote: >> >>> >> >>> http://searchdata.wmflabs.org/ - boop! This was my Friday.

>> >>> we were playing around with them and testing what we needed with a >> >>> static snapshot; these dashboards will now update once a day with

>> >>> information. >> >>> >> >>> It has turned up some bugs ("is the mobile schema just not

>> >>> and there are more metrics to add. But for the time being, is >> >>> progress >> >>> :) >> >>> >> >>> -- >> >>> Oliver Keyes >> >>> Research Analyst >> >>> Wikimedia Foundation >> >>> >> >>> _______________________________________________ >> >>> Wikimedia-search-private mailing list >> >>> Wikimedia-search-private(a)lists.wikimedia.org >> >>>

>> >> >> >> >> >> >> >> _______________________________________________ >> >> Wikimedia-search-private mailing list >> >> Wikimedia-search-private(a)lists.wikimedia.org >> >>

>> >> >> > >> > _______________________________________________ >> > Wikimedia-search-private mailing list >> > Wikimedia-search-private(a)lists.wikimedia.org >> >

_______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics