This is great. Do you have any categories tracked that could be interesting to break the position click-rates down by? eg: navigational vs. explorative queries, SAT clicks (satisfied user's query intent) vs. DSAT clicks (not satisfied), requery rate (how many times a user reformulates a new query in a session), time-to-first-click, search session duration, user's country/default language/# edits, length of query (# of query tokens), # of query results, popular vs. uncommon queries, high scoring SERP vs. low scoring SERP (or a proxy like Max BM25F of the top result), speller was click vs. not clicked, category of page clicked on, popular pages vs. rarely visited pages, etc. This experiment running on Special:Search is nice as that page doesn't automatically redirect when the query exactly matches a page. You can measure the positional importance by setting up an A/B test where you flip position 2 & 3. Also, a slowdown experiment would tell you the impact of latency, and help focus engineering efforts towards precision, or latency improvements. --justin On Tue, Feb 9, 2016 at 9:53 AM, Erik Bernhardson < ebernhardson(a)wikimedia.org&gt; wrote: _______________________________________________ discovery mailing list discovery(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/discovery

Good hits on page two: There's a few cases where good results could exist only on page two. One case is when incorrectly searching for a homophone or other misspelling. Eg: "their red hot" instead of "they're red hot" (expected result <https://en.wikipedia.org/wiki/They%27re_Red_Hot> -- wikipedia <https://en.wikipedia.org/w/index.php?search=their+red+hot&title=Special%3ASearch> (pos 22), google <https://www.google.com/search?q=their+red+hot&oq=their+red+hot> (pos 1), bing <https://www.bing.com/search?q=their+red+hot> (pos 2), ddg <https://duckduckgo.com/?q=their+red+hot> (pos 2)). Another case is when you get an exact string match on incorrect pages, but only non-exact string match on the correct page. Eg: "Cities in the San Francisco Bay Area" (expected result <https://en.wikipedia.org/wiki/List_of_cities_and_towns_in_the_San_Francisco_Bay_Area> -- wikipedia <https://en.wikipedia.org/w/index.php?title=Special:Search&search=Cities+in+the+San+Francisco+Bay+Area> (pos 122), google <https://www.google.com/search?q=Cities+in+the+San+Francisco+Bay+Area> (pos 1), bing <https://www.bing.com/search?q=Cities+in+the+San+Francisco+Bay+Area> (pos 1), ddg <https://duckduckgo.com/?q=Cities+in+the+San+Francisco+Bay+Area> (pos 1)). This style occurs mostly for a navigation query (only one correct result). For explorative queries, odds are one of the relevant results will be on page 1. There's a couple less direct cases, for instance if/once you integrate a popularity score, freshness score, importance score, page query score, or personalization (eg. ranking by physical distance from user or user's interests), you'll find some examples where incorrect results are non-helpfully boosted. Investigating queries which lead to clicks on page two may find interesting things popping out. -- Knowing the SAT/DSAT-click-rate-vs.-position will tell you if good clicks often occur beyond position 10. Then running an experiment of 10 SERP results vs. 20 SERP results may give interesting insights when watching a session-success-rate metric (and maybe a time-to-success metric). Aka, checking if a click on position 11+ is almost ever useful, or just leads to a requery or abandonment. If you run result size experiments, you can normalize for the query latency effects by generating 20 and displaying 10. The need of scrolling can cause a faster fall off of the click rates listed. On my web browser, as it's currently sized, there are only three results above the fold (my open advanced facet block takes a lot of space, scrolling required for result 4+). Knowing how-much/if the click rate drops for results below the fold will also help optimize the number of results to display, snippet length, and UI design. Could instrument number of results above the fold. -- Side note: possible bug, I can't find the page "List of New York University alumni <https://en.wikipedia.org/wiki/List_of_New_York_University_alumni>" when querying "New York University alumni <https://en.wikipedia.org/w/index.php?search=New+York+University+alumni&title=Special%3ASearch&go=Go>" (screenshot <https://imgur.com/SymW9tv>). --justin On Wed, Feb 10, 2016 at 12:04 PM, billinghurst &lt;billinghurstwiki(a)gmail.com _______________________________________________ discovery mailing list discovery(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/discovery

Thank you for all your suggestions, very inspiring! [response inline] Le 10/02/2016 21:33, Justin Ormont a écrit : Good hits on page two: There's a few cases where good results could exist only on page two. One case is when incorrectly searching for a homophone or other misspelling. Eg: "their red hot" instead of "they're red hot" (expected result <https://en.wikipedia.org/wiki/They%27re_Red_Hot> -- wikipedia <https://en.wikipedia.org/w/index.php?search=their+red+hot&title=Special%3ASearch> (pos 22), google <https://www.google.com/search?q=their+red+hot&oq=their+red+hot> (pos 1), bing <https://www.bing.com/search?q=their+red+hot> (pos 2), ddg <https://duckduckgo.com/?q=their+red+hot> (pos 2)). Indeed, we do a pretty bad job for this kind of queries. But I still don't know how to address that correctly. We don't use any synonym resources yet. This is usually addressed by the list of curated redirects, in this example we're able to catch only "theyre red hot" but we fail for their/there/.... Another case is when you get an exact string match on incorrect pages, but only non-exact string match on the correct page. Eg: "Cities in the San Francisco Bay Area" (expected result <https://en.wikipedia.org/wiki/List_of_cities_and_towns_in_the_San_Francisco_Bay_Area> -- wikipedia <https://en.wikipedia.org/w/index.php?title=Special:Search&search=Cities+in+the+San+Francisco+Bay+Area> (pos 122), google <https://www.google.com/search?q=Cities+in+the+San+Francisco+Bay+Area> (pos 1), bing <https://www.bing.com/search?q=Cities+in+the+San+Francisco+Bay+Area> (pos 1), ddg <https://duckduckgo.com/?q=Cities+in+the+San+Francisco+Bay+Area> (pos 1)). This style occurs mostly for a navigation query (only one correct result). For explorative queries, odds are one of the relevant results will be on page 1. There's a couple less direct cases, for instance if/once you integrate a popularity score, freshness score, importance score, page query score, or personalization (eg. ranking by physical distance from user or user's interests), you'll find some examples where incorrect results are non-helpfully boosted. You're completely right and this is exactly the case here. We always rescore the top 8000 documents (per node) with the number of incoming links (which is far from ideal). By disabling all the top-N rescoring features the expected result is now #2: https://en.wikipedia.org/w/index.php?search=Cities+in+the+San+Francisco+Bay… We don't do anything smart here, it's always the same plan whatever the query is... Investigating queries which lead to clicks on page two may find interesting things popping out. -- Knowing the SAT/DSAT-click-rate-vs.-position will tell you if good clicks often occur beyond position 10. Then running an experiment of 10 SERP results vs. 20 SERP results may give interesting insights when watching a session-success-rate metric (and maybe a time-to-success metric). Aka, checking if a click on position 11+ is almost ever useful, or just leads to a requery or abandonment. If you run result size experiments, you can normalize for the query latency effects by generating 20 and displaying 10. The need of scrolling can cause a faster fall off of the click rates listed. On my web browser, as it's currently sized, there are only three results above the fold (my open advanced facet block takes a lot of space, scrolling required for result 4+). Knowing how-much/if the click rate drops for results below the fold will also help optimize the number of results to display, snippet length, and UI design. Could instrument number of results above the fold. -- Side note: possible bug, I can't find the page "List of New York University alumni <https://en.wikipedia.org/wiki/List_of_New_York_University_alumni>" when querying "New York University alumni <https://en.wikipedia.org/w/index.php?search=New+York+University+alumni&title=Special%3ASearch&go=Go>" (screenshot <https://imgur.com/SymW9tv>). Yes... I usually find the params to tweak the query and push good results near the top but not here... I'll have to dig into more details to see what's going on, the best I can do is a rank around pos 120 :( Thank you very much for your help! -- David _______________________________________________ discovery mailing list discovery(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/discovery

Thanks for this list of possible metrics, lots of things to think about! We don't yet collect enough data for all of these, but some of them we do. A few potentially interesting tables: Number of searches per session: +------------------+--------------+---------+ | session_searches | num_sessions | percent | +------------------+--------------+---------+ | 1 | 59850 | 69.35% | | 2 | 16564 | 19.19% | | 3 | 5481 | 6.35% | | 4 | 2176 | 2.52% | | 5 | 1001 | 1.16% | | 6 | 531 | 0.62% | | 7 | 307 | 0.36% | | 8 | 204 | 0.24% | | 9 | 115 | 0.13% | | 10 | 75 | 0.09% | +------------------+--------------+---------+ Click through position by number of searches in session: +----------------+-------+-------+-------+-------+-------+-------+ | event_position | one | two | three | four | five | six | +----------------+-------+-------+-------+-------+-------+-------+ | 1 | 62.99 | 68.97 | 63.63 | 62.88 | 61.16 | 59.43 | | 2 | 15.42 | 12.06 | 13.78 | 15.07 | 12.43 | 12.74 | | 3 | 7.29 | 5.77 | 6.5 | 5.98 | 8.98 | 8.43 | | 4 | 4.04 | 3.6 | 4.68 | 4.01 | 5.48 | 4.22 | | 5 | 2.48 | 2.45 | 2.54 | 2.4 | 2.51 | 3.24 | | 6 | 1.68 | 1.24 | 1.71 | 1.53 | 1.77 | 1.18 | | 7 | 1.24 | 1.12 | 1.49 | 1.65 | 1.8 | 2.53 | | 8 | 0.75 | 0.82 | 1.04 | 0.81 | 0.56 | 1.42 | | 9 | 0.74 | 0.53 | 0.87 | 1.44 | 0.91 | 0.93 | | 10 | 0.42 | 0.57 | 0.35 | 0.79 | 0.36 | 1.44 | | 11 | 0.4 | 0.52 | 0.63 | 0.72 | 0.35 | 1.46 | | 12 | 0.36 | 0.28 | 0.42 | 0.6 | 0.61 | 0.91 | | 13 | 0.33 | 0.19 | 0.3 | 0.33 | 0.32 | 0.38 | | 14 | 0.29 | 0.38 | 0.2 | 0.53 | 0.6 | 0.13 | | 15 | 0.26 | 0.28 | 0.21 | 0.37 | 0.92 | 0.4 | | 16 | 0.26 | 0.18 | 0.2 | 0.23 | 0.1 | 0 | | 17 | 0.24 | 0.11 | 0.31 | 0.03 | 0.11 | 0.27 | | 18 | 0.21 | 0.26 | 0.21 | 0.17 | 0.11 | 0.09 | | 19 | 0.21 | 0.31 | 0.23 | 0.29 | 0.43 | 0.09 | | 20 | 0.28 | 0.25 | 0.6 | 0.19 | 0.28 | 0.62 | | 21 | 0.01 | 0.06 | 0.01 | 0 | 0.03 | 0 | | 22 | 0.03 | 0.01 | 0.04 | 0 | 0.04 | 0.02 | | 23 | 0.01 | 0.02 | 0 | 0 | 0.06 | 0 | | 24 | 0.02 | 0.01 | 0.01 | 0 | 0.04 | 0.07 | | 25 | 0.02 | 0.01 | 0.03 | 0 | 0.04 | 0 | +----------------+-------+-------+-------+-------+-------+-------+ Click through position vs duration of search session in seconds. Column's are named as min_max +----------------+-------+-------+-------+-------+--------+---------+---------+---------+---------+----------+-----------+ | event_position | 0_1 | 2_15 | 16_30 | 31_60 | 61_120 | 121_180 | 181_240 | 241_300 | 301_600 | 601_1200 | 1201_1800 | +----------------+-------+-------+-------+-------+--------+---------+---------+---------+---------+----------+-----------+ | 1 | 92.52 | 72.07 | 58.59 | 58 | 56.79 | 57.1 | 59.29 | 58.82 | 58.17 | 54.94 | 60.23 | | 2 | 4.43 | 14.34 | 14.29 | 13.72 | 15.22 | 15.14 | 13.2 | 16.3 | 15.08 | 14.8 | 11.13 | | 3 | 1.62 | 5.83 | 8.6 | 7.85 | 7.1 | 8.17 | 7.1 | 4.63 | 7.86 | 7.69 | 7.24 | | 4 | 0.05 | 3.16 | 5.1 | 4.8 | 4.73 | 4.52 | 5.61 | 4.43 | 4.75 | 4.83 | 4.79 | | 5 | 0.5 | 1.59 | 3.7 | 3.5 | 3.81 | 2.27 | 2.64 | 2.57 | 2.9 | 2.78 | 2.74 | | 6 | 0.34 | 0.97 | 2.06 | 2.09 | 2.17 | 1.59 | 2.15 | 3.48 | 1.81 | 2.57 | 0.86 | | 7 | 0 | 0.67 | 1.41 | 1.84 | 1.63 | 2.26 | 1.64 | 1.5 | 2.03 | 2.12 | 2.16 | | 8 | 0 | 0.27 | 0.9 | 1.52 | 1.32 | 1.17 | 1.39 | 1.1 | 1.28 | 1.46 | 1.28 | | 9 | 0 | 0.23 | 1.15 | 1.16 | 1.04 | 1.02 | 0.47 | 1.77 | 1.13 | 1.41 | 1.26 | | 10 | 0 | 0.14 | 0.55 | 0.96 | 0.99 | 0.93 | 0.76 | 0.75 | 0.45 | 1.17 | 0.86 | | 11 | 0 | 0.17 | 0.41 | 0.76 | 0.66 | 0.64 | 0.53 | 1.23 | 0.73 | 1.31 | 0.99 | | 12 | 0.03 | 0.08 | 0.58 | 0.59 | 0.74 | 0.65 | 0.55 | 0.19 | 0.45 | 0.92 | 1.32 | | 13 | 0 | 0.08 | 0.49 | 0.39 | 0.67 | 0.4 | 0.28 | 0.83 | 0.41 | 0.24 | 0.86 | | 14 | 0 | 0.05 | 0.35 | 0.52 | 0.74 | 0.72 | 0.57 | 0.51 | 0.37 | 0.77 | 0.86 | | 15 | 0.16 | 0.05 | 0.41 | 0.41 | 0.37 | 0.35 | 0.7 | 0.16 | 0.59 | 0.54 | 0.79 | | 16 | 0.34 | 0.09 | 0.36 | 0.21 | 0.53 | 0.31 | 0.35 | 0.35 | 0.23 | 0.43 | 0.46 | | 17 | 0 | 0.07 | 0.17 | 0.32 | 0.18 | 0.56 | 0.52 | 0.31 | 0.31 | 0.24 | 0.46 | | 18 | 0 | 0.04 | 0.22 | 0.58 | 0.36 | 0.33 | 0.67 | 0.28 | 0.25 | 0.53 | 0.75 | | 19 | 0 | 0.05 | 0.2 | 0.29 | 0.42 | 0.81 | 0.75 | 0.32 | 0.4 | 0.4 | 0.22 | | 20 | 0 | 0.02 | 0.44 | 0.48 | 0.48 | 0.44 | 0.74 | 0.36 | 0.64 | 0.53 | 0.31 | | 21 | 0 | 0 | 0.01 | 0 | 0.03 | 0 | 0 | 0.04 | 0.06 | 0.17 | 0.09 | | 22 | 0 | 0.02 | 0 | 0 | 0 | 0.12 | 0.01 | 0.01 | 0.05 | 0.03 | 0.24 | | 23 | 0 | 0 | 0 | 0 | 0 | 0.03 | 0.05 | 0 | 0.01 | 0.07 | 0.04 | | 24 | 0 | 0 | 0 | 0 | 0 | 0.13 | 0 | 0.03 | 0.03 | 0.03 | 0 | | 25 | 0 | 0 | 0 | 0.02 | 0.01 | 0.33 | 0 | 0.03 | 0.01 | 0.04 | 0.02 | +----------------+-------+-------+-------+-------+--------+---------+---------+---------+---------+----------+-----------+ Number of searches performed vs number of clicks on search results per session: +----------+--------+--------+-------+-------+-------+ | searches | zero | one | two | 3_5 | gt5 | +----------+--------+--------+-------+-------+-------+ | 0 | 0.00% | 0.08% | 0.00% | 0.00% | 0.00% | | 1 | 40.85% | 19.77% | 0.27% | 0.05% | 0.01% | | 2 | 10.56% | 7.86% | 1.46% | 0.06% | 0.00% | | 3 | 4.75% | 2.55% | 0.98% | 0.27% | 0.00% | | 4 | 1.91% | 1.25% | 0.51% | 0.29% | 0.00% | | 5 | 1.14% | 0.55% | 0.30% | 0.23% | 0.01% | | 6 | 0.61% | 0.31% | 0.17% | 0.18% | 0.02% | | 7 | 0.37% | 0.19% | 0.11% | 0.16% | 0.02% | | 8 | 0.24% | 0.10% | 0.08% | 0.10% | 0.02% | | 9 | 0.14% | 0.06% | 0.06% | 0.07% | 0.02% | | 10 | 0.11% | 0.06% | 0.03% | 0.04% | 0.02% | | 11 | 0.07% | 0.03% | 0.01% | 0.04% | 0.03% | | 12 | 0.06% | 0.02% | 0.01% | 0.02% | 0.02% | | 13 | 0.03% | 0.01% | 0.01% | 0.02% | 0.01% | | 14 | 0.03% | 0.01% | 0.02% | 0.02% | 0.01% | +----------+--------+--------+-------+-------+-------+ This isn't nearly all the things you mentioned, but a few that we have collected data for already. Hopefully these and a few other metrics can become part of how we measure what effect changes to search make. On Wed, Feb 10, 2016 at 3:01 AM, Justin Ormont &lt;justin.ormont(a)gmail.com&gt; wrote: