Hello all,
As mentioned previously, the current version of the Android app contains an A/B test where it presents "read more" suggestions to the user, based on (a) the standard "morelike" query, or (b) the new "opening_text" query.
Here are the results from the last ~10 days of the test[0]: - The clickthrough rate using the default morelike query is (and has been) around 15%. - With the new opening_text query, the clickthrough rate decreases to about 12%:
[image: Inline image 1]
Therefore, it seems that the new query has a nontrivial negative effect on CTR :( We'll plan on removing this test in the next release of the app, but we'll be happy to plug in a different or updated query, if it will be of further use to Discovery.
[0] https://docs.google.com/a/wikimedia.org/spreadsheets/d/1BFsrAcPgexQyNVemmJ3k... (queries embedded as comments in the headers)
ouch, that is not at all the result we were hoping for. Just goes to show why we have to test these things and not just take a few examples that perform badly in one set and look to do a better job with some different options. Thanks for putting this together!
On Thu, Feb 25, 2016 at 3:06 PM, Dmitry Brant dbrant@wikimedia.org wrote:
Hello all,
As mentioned previously, the current version of the Android app contains an A/B test where it presents "read more" suggestions to the user, based on (a) the standard "morelike" query, or (b) the new "opening_text" query.
Here are the results from the last ~10 days of the test[0]:
- The clickthrough rate using the default morelike query is (and has been)
around 15%.
- With the new opening_text query, the clickthrough rate decreases to
about 12%:
[image: Inline image 1]
Therefore, it seems that the new query has a nontrivial negative effect on CTR :( We'll plan on removing this test in the next release of the app, but we'll be happy to plug in a different or updated query, if it will be of further use to Discovery.
[0] https://docs.google.com/a/wikimedia.org/spreadsheets/d/1BFsrAcPgexQyNVemmJ3k... (queries embedded as comments in the headers)
-- Dmitry Brant Senior Software Engineer / Product Owner (Android) Wikimedia Foundation https://www.mediawiki.org/wiki/Wikimedia_mobile_engineering
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Thanks!
yes this is not exactly what we expected :( I guess it was too good to be true: reduce latency and improve quality at the same time :) On our side I'd say that perf was the main issue, Erik added a cache at the backend-end level which seems to have a good impact. Morelike queries are still routed to the new datacenter in dallas to reduce stress on eqiad. We could maybe try to reroute them to eqiad and see if caching is sufficient? If it's the case I'd say that we don't need to run any A/B test.
Random questions: Is it possible to analyze the correlation between the chosen article and the presence of an image? Rescoring options are slightly different for enwiki, is it possible to have the detail for e.g. enwiki/frwiki/dewiki?
If it does not require huge effort on your side I'd say that you could run another A/B test by disabling boostLinks, you just have to add cirrusBoostLinks=no to api URL.
Thank you
Le 26/02/2016 00:33, Erik Bernhardson a écrit :
ouch, that is not at all the result we were hoping for. Just goes to show why we have to test these things and not just take a few examples that perform badly in one set and look to do a better job with some different options. Thanks for putting this together!
On Thu, Feb 25, 2016 at 3:06 PM, Dmitry Brant <dbrant@wikimedia.org mailto:dbrant@wikimedia.org> wrote:
Hello all, As mentioned previously, the current version of the Android app contains an A/B test where it presents "read more" suggestions to the user, based on (a) the standard "morelike" query, or (b) the new "opening_text" query. Here are the results from the last ~10 days of the test[0]: - The clickthrough rate using the default morelike query is (and has been) around 15%. - With the new opening_text query, the clickthrough rate decreases to about 12%: Inline image 1 Therefore, it seems that the new query has a nontrivial negative effect on CTR :( We'll plan on removing this test in the next release of the app, but we'll be happy to plug in a different or updated query, if it will be of further use to Discovery. [0] https://docs.google.com/a/wikimedia.org/spreadsheets/d/1BFsrAcPgexQyNVemmJ3k3IX5rtPvJ_5vdYOyGgS5R6Y/edit?usp=sharing (queries embedded as comments in the headers) -- Dmitry Brant Senior Software Engineer / Product Owner (Android) Wikimedia Foundation https://www.mediawiki.org/wiki/Wikimedia_mobile_engineering _______________________________________________ Mobile-l mailing list Mobile-l@lists.wikimedia.org <mailto:Mobile-l@lists.wikimedia.org> https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
+ JMo
On Fri, Feb 26, 2016 at 4:19 AM David Causse dcausse@wikimedia.org wrote:
Thanks!
yes this is not exactly what we expected :( I guess it was too good to be true: reduce latency and improve quality at the same time :) On our side I'd say that perf was the main issue, Erik added a cache at the backend-end level which seems to have a good impact. Morelike queries are still routed to the new datacenter in dallas to reduce stress on eqiad. We could maybe try to reroute them to eqiad and see if caching is sufficient? If it's the case I'd say that we don't need to run any A/B test.
Random questions: Is it possible to analyze the correlation between the chosen article and the presence of an image? Rescoring options are slightly different for enwiki, is it possible to have the detail for e.g. enwiki/frwiki/dewiki?
If it does not require huge effort on your side I'd say that you could run another A/B test by disabling boostLinks, you just have to add cirrusBoostLinks=no to api URL.
Thank you
Le 26/02/2016 00:33, Erik Bernhardson a écrit :
ouch, that is not at all the result we were hoping for. Just goes to show why we have to test these things and not just take a few examples that perform badly in one set and look to do a better job with some different options. Thanks for putting this together!
On Thu, Feb 25, 2016 at 3:06 PM, Dmitry Brant dbrant@wikimedia.org wrote:
Hello all,
As mentioned previously, the current version of the Android app contains an A/B test where it presents "read more" suggestions to the user, based on (a) the standard "morelike" query, or (b) the new "opening_text" query.
Here are the results from the last ~10 days of the test[0]:
- The clickthrough rate using the default morelike query is (and has
been) around 15%.
- With the new opening_text query, the clickthrough rate decreases to
about 12%:
Therefore, it seems that the new query has a nontrivial negative effect on CTR :( We'll plan on removing this test in the next release of the app, but we'll be happy to plug in a different or updated query, if it will be of further use to Discovery.
[0] https://docs.google.com/a/wikimedia.org/spreadsheets/d/1BFsrAcPgexQyNVemmJ3k... (queries embedded as comments in the headers)
-- Dmitry Brant Senior Software Engineer / Product Owner (Android) Wikimedia Foundation https://www.mediawiki.org/wiki/Wikimedia_mobile_engineering
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing listMobile-l@lists.wikimedia.orghttps://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
