moving to mobile-l, and cc Search & Discovery.
---------- Forwarded message ---------- From: Dmitry Brant dbrant@wikimedia.org Date: Wed, Jul 29, 2015 at 3:38 PM Subject: "Morelike" suggestions - the results are in! To: Internal communication for WMF Reading team < reading-wmf@lists.wikimedia.org>
Hi all,
For the last few weeks, we've had an A/B test in the Android app where we measure user engagement with the "read more" suggestions that we show at the bottom of each article. We display three suggestions for further reading, based on either (A) a plain full-text search query based on the title of the current article, or (B) a query using the "morelike" feature in CirrusSearch.
And the winner is... (perhaps not entirely surprisingly) "morelike"! Users who saw suggestions based on "morelike" were over 20% more likely to click on one of the suggestions.
Here's a quick analysis and chart of the data from the last 10 days:
*https://docs.google.com/spreadsheets/d/1BFsrAcPgexQyNVemmJ3k3IX5rtPvJ_5vdYOy... https://docs.google.com/spreadsheets/d/1BFsrAcPgexQyNVemmJ3k3IX5rtPvJ_5vdYOyGgS5R6Y/edit?usp=sharing*
-Dmitry
This is fantastic work to see!
For future work around this stuff, can I ask that you keep us in the loop? When you build a feature like this, users get more things - and search gets more queries, some that succeed and some but fail. But for this email at the tail-end of the test we wouldn't know this was happening, and it has the potential to mess with some of our core KPIs.
On 29 July 2015 at 16:01, Dmitry Brant dbrant@wikimedia.org wrote:
moving to mobile-l, and cc Search & Discovery.
---------- Forwarded message ---------- From: Dmitry Brant dbrant@wikimedia.org Date: Wed, Jul 29, 2015 at 3:38 PM Subject: "Morelike" suggestions - the results are in! To: Internal communication for WMF Reading team reading-wmf@lists.wikimedia.org
Hi all,
For the last few weeks, we've had an A/B test in the Android app where we measure user engagement with the "read more" suggestions that we show at the bottom of each article. We display three suggestions for further reading, based on either (A) a plain full-text search query based on the title of the current article, or (B) a query using the "morelike" feature in CirrusSearch.
And the winner is... (perhaps not entirely surprisingly) "morelike"! Users who saw suggestions based on "morelike" were over 20% more likely to click on one of the suggestions.
Here's a quick analysis and chart of the data from the last 10 days:
https://docs.google.com/spreadsheets/d/1BFsrAcPgexQyNVemmJ3k3IX5rtPvJ_5vdYOy...
-Dmitry
Wikimedia-search mailing list Wikimedia-search@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikimedia-search
Will surely take note for next time. Thanks! Now that we've got the results from this test, we will soon update the app to use *only* morelike suggestions, which means you'll be getting twice as many morelike queries from the app as before. (We'll let you know when that's released)
On Wed, Jul 29, 2015 at 4:17 PM, Oliver Keyes okeyes@wikimedia.org wrote:
This is fantastic work to see!
For future work around this stuff, can I ask that you keep us in the loop? When you build a feature like this, users get more things - and search gets more queries, some that succeed and some but fail. But for this email at the tail-end of the test we wouldn't know this was happening, and it has the potential to mess with some of our core KPIs.
On 29 July 2015 at 16:01, Dmitry Brant dbrant@wikimedia.org wrote:
moving to mobile-l, and cc Search & Discovery.
---------- Forwarded message ---------- From: Dmitry Brant dbrant@wikimedia.org Date: Wed, Jul 29, 2015 at 3:38 PM Subject: "Morelike" suggestions - the results are in! To: Internal communication for WMF Reading team reading-wmf@lists.wikimedia.org
Hi all,
For the last few weeks, we've had an A/B test in the Android app where we measure user engagement with the "read more" suggestions that we show at
the
bottom of each article. We display three suggestions for further reading, based on either (A) a plain full-text search query based on the title of
the
current article, or (B) a query using the "morelike" feature in CirrusSearch.
And the winner is... (perhaps not entirely surprisingly) "morelike"!
Users
who saw suggestions based on "morelike" were over 20% more likely to
click
on one of the suggestions.
Here's a quick analysis and chart of the data from the last 10 days:
https://docs.google.com/spreadsheets/d/1BFsrAcPgexQyNVemmJ3k3IX5rtPvJ_5vdYOy...
-Dmitry
Wikimedia-search mailing list Wikimedia-search@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikimedia-search
-- Oliver Keyes Research Analyst Wikimedia Foundation
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
I'm late to this, but great news!
I just want to commend those involved for a great example of cross-pollination between the web team and the app team. This is all the more impressive, given that the backend service was designed for A by the growth team(?), web is using it for B, and the app is using it for C. Communication! -J
On Wed, Jul 29, 2015 at 2:23 PM, Dmitry Brant dbrant@wikimedia.org wrote:
Will surely take note for next time. Thanks! Now that we've got the results from this test, we will soon update the app to use *only* morelike suggestions, which means you'll be getting twice as many morelike queries from the app as before. (We'll let you know when that's released)
On Wed, Jul 29, 2015 at 4:17 PM, Oliver Keyes okeyes@wikimedia.org wrote:
This is fantastic work to see!
For future work around this stuff, can I ask that you keep us in the loop? When you build a feature like this, users get more things - and search gets more queries, some that succeed and some but fail. But for this email at the tail-end of the test we wouldn't know this was happening, and it has the potential to mess with some of our core KPIs.
On 29 July 2015 at 16:01, Dmitry Brant dbrant@wikimedia.org wrote:
moving to mobile-l, and cc Search & Discovery.
---------- Forwarded message ---------- From: Dmitry Brant dbrant@wikimedia.org Date: Wed, Jul 29, 2015 at 3:38 PM Subject: "Morelike" suggestions - the results are in! To: Internal communication for WMF Reading team reading-wmf@lists.wikimedia.org
Hi all,
For the last few weeks, we've had an A/B test in the Android app where
we
measure user engagement with the "read more" suggestions that we show
at the
bottom of each article. We display three suggestions for further
reading,
based on either (A) a plain full-text search query based on the title
of the
current article, or (B) a query using the "morelike" feature in CirrusSearch.
And the winner is... (perhaps not entirely surprisingly) "morelike"!
Users
who saw suggestions based on "morelike" were over 20% more likely to
click
on one of the suggestions.
Here's a quick analysis and chart of the data from the last 10 days:
https://docs.google.com/spreadsheets/d/1BFsrAcPgexQyNVemmJ3k3IX5rtPvJ_5vdYOy...
-Dmitry
Wikimedia-search mailing list Wikimedia-search@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikimedia-search
-- Oliver Keyes Research Analyst Wikimedia Foundation
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
On 4 August 2015 at 14:22, Jon Katz jkatz@wikimedia.org wrote:
given that the backend service was designed for A by the growth team(?)
*cough* Discovery!
;-)
Dan