[Wikimedia-search] Smoothing in dashboard(s) - Discovery

Trey Jones

10 Sep 10 Sep

4:15 p.m.

Smooooooooth! It looks great! Quick question—what's the period of the moving average? Is it a week, or more? (A week makes sense, but it looks like more, based on how much data drops out at the beginning.) If it's not a week, can I suggest a week? It may further smooth out some of the daily bumps to only have one Monday, for example, in any given average, since the daily pattern is so extreme. On Thu, Sep 10, 2015 at 12:00 PM, Mikhail Popov <mpopov(a)wikimedia.org> wrote:

...

Hi all, You can now apply various smoothers to the data on the Search Metrics dashboard (e.g. http://searchdata.wmflabs.org/metrics/#failure_rate). Smoothing can be applied globally or on a per-plot basis. Hopefully yinz will find this new feature useful! Cheers~ -- *Mikhail Popov* // Data Analyst, Discovery <https://www.mediawiki.org/wiki/Wikimedia_Discovery> https://wikimediafoundation.org/ *Imagine a world in which every single human being can freely share in the **sum of all knowledge. That's our commitment.* Donate <https://donate.wikimedia.org/>. _______________________________________________ Wikimedia-search mailing list Wikimedia-search(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikimedia-search

Reply

Mikhail Popov

4:28 p.m.

Thanks! :D The moving average is computed over 17 days. That seems like an arbitrary choice and it largely is but I played around with that number and 17 yielded what seemed to me the best results. I tried 7 and such and wasn't satisfied with how those curves looked. On Thu, Sep 10, 2015 at 9:15 AM, Trey Jones <tjones(a)wikimedia.org> wrote:

...

Smooooooooth! It looks great! Quick question—what's the period of the moving average? Is it a week, or more? (A week makes sense, but it looks like more, based on how much data drops out at the beginning.) If it's not a week, can I suggest a week? It may further smooth out some of the daily bumps to only have one Monday, for example, in any given average, since the daily pattern is so extreme. On Thu, Sep 10, 2015 at 12:00 PM, Mikhail Popov <mpopov(a)wikimedia.org> wrote:

Hi all, You can now apply various smoothers to the data on the Search Metrics dashboard (e.g. http://searchdata.wmflabs.org/metrics/#failure_rate). Smoothing can be applied globally or on a per-plot basis. Hopefully yinz will find this new feature useful! Cheers~ -- *Mikhail Popov* // Data Analyst, Discovery <https://www.mediawiki.org/wiki/Wikimedia_Discovery> https://wikimediafoundation.org/ *Imagine a world in which every single human being can freely share in the **sum of all knowledge. That's our commitment.* Donate <https://donate.wikimedia.org/>. _______________________________________________ Wikimedia-search mailing list Wikimedia-search(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikimedia-search

_______________________________________________ Wikimedia-search mailing list Wikimedia-search(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikimedia-search

-- -- *Mikhail Popov* // Data Scientist, Discovery <https://www.mediawiki.org/wiki/Wikimedia_Discovery> https://wikimediafoundation.org/ *Imagine a world in which every single human being can freely share in the **sum of all knowledge. That's our commitment.* Donate <https://donate.wikimedia.org/>.

Reply

Mikhail Popov

4:53 p.m.

I'll look into adding something for moving average! And thanks! I'm glad y'all like the feature. And I hope the compliments are directed at the Analysis team, as this was a team effort between Oliver and me :) On Thu, Sep 10, 2015 at 9:41 AM, Trey Jones <tjones(a)wikimedia.org> wrote:

...

On Thu, Sep 10, 2015 at 12:28 PM, Mikhail Popov <mpopov(a)wikimedia.org> wrote:

The moving average is computed over 17 days. That seems like an arbitrary choice and it largely is but I played around with that number and 17 yielded what seemed to me the best results. I tried 7 and such and wasn't satisfied with how those curves looked.

Interesting. I'm curious how 14 worked, since that always gives you 2 of each day of the week, instead of 3 of some and 2 of the rest. In any event, is there an easy way to document the moving average length on the page (would it work in the dropdown?), so people will know what they are looking at? Again, this is very cool. All of the smoothed version are so much more comprehensible than the raw plot. Thanks for adding this feature. _______________________________________________ Wikimedia-search mailing list Wikimedia-search(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikimedia-search

-- -- *Mikhail Popov* // Data Scientist, Discovery <https://www.mediawiki.org/wiki/Wikimedia_Discovery> https://wikimediafoundation.org/ *Imagine a world in which every single human being can freely share in the **sum of all knowledge. That's our commitment.* Donate <https://donate.wikimedia.org/>.

Reply