Thanks for this list of possible metrics, lots of things to think about! We don't yet collect enough data for all of these, but some of them we do. A few potentially interesting tables:

Number of searches per session:
+------------------+--------------+---------+
| session_searches | num_sessions | percent |
+------------------+--------------+---------+
|                1 |        59850 | 69.35%  |
|                2 |        16564 | 19.19%  |
|                3 |         5481 | 6.35%   |
|                4 |         2176 | 2.52%   |
|                5 |         1001 | 1.16%   |
|                6 |          531 | 0.62%   |
|                7 |          307 | 0.36%   |
|                8 |          204 | 0.24%   |
|                9 |          115 | 0.13%   |
|               10 |           75 | 0.09%   |
+------------------+--------------+---------+

Click through position by number of searches in session:
+----------------+-------+-------+-------+-------+-------+-------+
| event_position | one   | two   | three | four  | five  | six   |
+----------------+-------+-------+-------+-------+-------+-------+
|              1 | 62.99 | 68.97 | 63.63 | 62.88 | 61.16 | 59.43 |
|              2 | 15.42 | 12.06 | 13.78 | 15.07 | 12.43 | 12.74 |
|              3 |  7.29 |  5.77 |   6.5 |  5.98 |  8.98 |  8.43 |
|              4 |  4.04 |   3.6 |  4.68 |  4.01 |  5.48 |  4.22 |
|              5 |  2.48 |  2.45 |  2.54 |   2.4 |  2.51 |  3.24 |
|              6 |  1.68 |  1.24 |  1.71 |  1.53 |  1.77 |  1.18 |
|              7 |  1.24 |  1.12 |  1.49 |  1.65 |   1.8 |  2.53 |
|              8 |  0.75 |  0.82 |  1.04 |  0.81 |  0.56 |  1.42 |
|              9 |  0.74 |  0.53 |  0.87 |  1.44 |  0.91 |  0.93 |
|             10 |  0.42 |  0.57 |  0.35 |  0.79 |  0.36 |  1.44 |
|             11 |   0.4 |  0.52 |  0.63 |  0.72 |  0.35 |  1.46 |
|             12 |  0.36 |  0.28 |  0.42 |   0.6 |  0.61 |  0.91 |
|             13 |  0.33 |  0.19 |   0.3 |  0.33 |  0.32 |  0.38 |
|             14 |  0.29 |  0.38 |   0.2 |  0.53 |   0.6 |  0.13 |
|             15 |  0.26 |  0.28 |  0.21 |  0.37 |  0.92 |   0.4 |
|             16 |  0.26 |  0.18 |   0.2 |  0.23 |   0.1 |     0 |
|             17 |  0.24 |  0.11 |  0.31 |  0.03 |  0.11 |  0.27 |
|             18 |  0.21 |  0.26 |  0.21 |  0.17 |  0.11 |  0.09 |
|             19 |  0.21 |  0.31 |  0.23 |  0.29 |  0.43 |  0.09 |
|             20 |  0.28 |  0.25 |   0.6 |  0.19 |  0.28 |  0.62 |
|             21 |  0.01 |  0.06 |  0.01 |     0 |  0.03 |     0 |
|             22 |  0.03 |  0.01 |  0.04 |     0 |  0.04 |  0.02 |
|             23 |  0.01 |  0.02 |     0 |     0 |  0.06 |     0 |
|             24 |  0.02 |  0.01 |  0.01 |     0 |  0.04 |  0.07 |
|             25 |  0.02 |  0.01 |  0.03 |     0 |  0.04 |     0 |
+----------------+-------+-------+-------+-------+-------+-------+

Click through position vs duration of search session in seconds. Column's are named as min_max
+----------------+-------+-------+-------+-------+--------+---------+---------+---------+---------+----------+-----------+
| event_position | 0_1   | 2_15  | 16_30 | 31_60 | 61_120 | 121_180 | 181_240 | 241_300 | 301_600 | 601_1200 | 1201_1800 |
+----------------+-------+-------+-------+-------+--------+---------+---------+---------+---------+----------+-----------+
|              1 | 92.52 | 72.07 | 58.59 |    58 |  56.79 |    57.1 |   59.29 |   58.82 |   58.17 |    54.94 |     60.23 |
|              2 |  4.43 | 14.34 | 14.29 | 13.72 |  15.22 |   15.14 |    13.2 |    16.3 |   15.08 |     14.8 |     11.13 |
|              3 |  1.62 |  5.83 |   8.6 |  7.85 |    7.1 |    8.17 |     7.1 |    4.63 |    7.86 |     7.69 |      7.24 |
|              4 |  0.05 |  3.16 |   5.1 |   4.8 |   4.73 |    4.52 |    5.61 |    4.43 |    4.75 |     4.83 |      4.79 |
|              5 |   0.5 |  1.59 |   3.7 |   3.5 |   3.81 |    2.27 |    2.64 |    2.57 |     2.9 |     2.78 |      2.74 |
|              6 |  0.34 |  0.97 |  2.06 |  2.09 |   2.17 |    1.59 |    2.15 |    3.48 |    1.81 |     2.57 |      0.86 |
|              7 |     0 |  0.67 |  1.41 |  1.84 |   1.63 |    2.26 |    1.64 |     1.5 |    2.03 |     2.12 |      2.16 |
|              8 |     0 |  0.27 |   0.9 |  1.52 |   1.32 |    1.17 |    1.39 |     1.1 |    1.28 |     1.46 |      1.28 |
|              9 |     0 |  0.23 |  1.15 |  1.16 |   1.04 |    1.02 |    0.47 |    1.77 |    1.13 |     1.41 |      1.26 |
|             10 |     0 |  0.14 |  0.55 |  0.96 |   0.99 |    0.93 |    0.76 |    0.75 |    0.45 |     1.17 |      0.86 |
|             11 |     0 |  0.17 |  0.41 |  0.76 |   0.66 |    0.64 |    0.53 |    1.23 |    0.73 |     1.31 |      0.99 |
|             12 |  0.03 |  0.08 |  0.58 |  0.59 |   0.74 |    0.65 |    0.55 |    0.19 |    0.45 |     0.92 |      1.32 |
|             13 |     0 |  0.08 |  0.49 |  0.39 |   0.67 |     0.4 |    0.28 |    0.83 |    0.41 |     0.24 |      0.86 |
|             14 |     0 |  0.05 |  0.35 |  0.52 |   0.74 |    0.72 |    0.57 |    0.51 |    0.37 |     0.77 |      0.86 |
|             15 |  0.16 |  0.05 |  0.41 |  0.41 |   0.37 |    0.35 |     0.7 |    0.16 |    0.59 |     0.54 |      0.79 |
|             16 |  0.34 |  0.09 |  0.36 |  0.21 |   0.53 |    0.31 |    0.35 |    0.35 |    0.23 |     0.43 |      0.46 |
|             17 |     0 |  0.07 |  0.17 |  0.32 |   0.18 |    0.56 |    0.52 |    0.31 |    0.31 |     0.24 |      0.46 |
|             18 |     0 |  0.04 |  0.22 |  0.58 |   0.36 |    0.33 |    0.67 |    0.28 |    0.25 |     0.53 |      0.75 |
|             19 |     0 |  0.05 |   0.2 |  0.29 |   0.42 |    0.81 |    0.75 |    0.32 |     0.4 |      0.4 |      0.22 |
|             20 |     0 |  0.02 |  0.44 |  0.48 |   0.48 |    0.44 |    0.74 |    0.36 |    0.64 |     0.53 |      0.31 |
|             21 |     0 |     0 |  0.01 |     0 |   0.03 |       0 |       0 |    0.04 |    0.06 |     0.17 |      0.09 |
|             22 |     0 |  0.02 |     0 |     0 |      0 |    0.12 |    0.01 |    0.01 |    0.05 |     0.03 |      0.24 |
|             23 |     0 |     0 |     0 |     0 |      0 |    0.03 |    0.05 |       0 |    0.01 |     0.07 |      0.04 |
|             24 |     0 |     0 |     0 |     0 |      0 |    0.13 |       0 |    0.03 |    0.03 |     0.03 |         0 |
|             25 |     0 |     0 |     0 |  0.02 |   0.01 |    0.33 |       0 |    0.03 |    0.01 |     0.04 |      0.02 |
+----------------+-------+-------+-------+-------+--------+---------+---------+---------+---------+----------+-----------+

Number of searches performed vs number of clicks on search results per session:
+----------+--------+--------+-------+-------+-------+
| searches | zero   | one    | two   | 3_5   | gt5   |
+----------+--------+--------+-------+-------+-------+
|        0 | 0.00%  | 0.08%  | 0.00% | 0.00% | 0.00% |
|        1 | 40.85% | 19.77% | 0.27% | 0.05% | 0.01% |
|        2 | 10.56% | 7.86%  | 1.46% | 0.06% | 0.00% |
|        3 | 4.75%  | 2.55%  | 0.98% | 0.27% | 0.00% |
|        4 | 1.91%  | 1.25%  | 0.51% | 0.29% | 0.00% |
|        5 | 1.14%  | 0.55%  | 0.30% | 0.23% | 0.01% |
|        6 | 0.61%  | 0.31%  | 0.17% | 0.18% | 0.02% |
|        7 | 0.37%  | 0.19%  | 0.11% | 0.16% | 0.02% |
|        8 | 0.24%  | 0.10%  | 0.08% | 0.10% | 0.02% |
|        9 | 0.14%  | 0.06%  | 0.06% | 0.07% | 0.02% |
|       10 | 0.11%  | 0.06%  | 0.03% | 0.04% | 0.02% |
|       11 | 0.07%  | 0.03%  | 0.01% | 0.04% | 0.03% |
|       12 | 0.06%  | 0.02%  | 0.01% | 0.02% | 0.02% |
|       13 | 0.03%  | 0.01%  | 0.01% | 0.02% | 0.01% |
|       14 | 0.03%  | 0.01%  | 0.02% | 0.02% | 0.01% |
+----------+--------+--------+-------+-------+-------+

This isn't nearly all the things you mentioned, but a few that we have collected data for already. Hopefully these and a few other metrics can become part of how we measure what effect changes to search make.


On Wed, Feb 10, 2016 at 3:01 AM, Justin Ormont <justin.ormont@gmail.com> wrote:
This is great. Do you have any categories tracked that could be interesting to break the position click-rates down by? eg: navigational vs. explorative queries, SAT clicks (satisfied user's query intent) vs. DSAT clicks (not satisfied), requery rate (how many times a user reformulates a new query in a session), time-to-first-click, search session duration, user's country/default language/# edits, length of query (# of query tokens), # of query results, popular vs. uncommon queries, high scoring SERP vs. low scoring SERP (or a proxy like Max BM25F of the top result), speller was click vs. not clicked, category of page clicked on, popular pages vs. rarely visited pages, etc. 

This experiment running on Special:Search is nice as that page doesn't automatically redirect when the query exactly matches a page.

You can measure the positional importance by setting up an A/B test where you flip position 2 & 3. Also, a slowdown experiment would tell you the impact of latency, and help focus engineering efforts towards precision, or latency improvements.

--justin


On Tue, Feb 9, 2016 at 9:53 AM, Erik Bernhardson <ebernhardson@wikimedia.org> wrote:
2 |  34214 | 14.26%



_______________________________________________
discovery mailing list
discovery@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/discovery