Short answer, no, this data is not available publicy such you can compute the dataset yourself as it is Private data. Thanks, Nuria On Mon, Mar 5, 2018 at 11:31 AM, Georg Sorst <g.sorst(a)findologic.com> wrote:

Sorry for the delayed response, I've been out the last week. Responses inline. On Mon, Mar 12, 2018 at 1:27 AM, Georg Sorst <g.sorst(a)findologic.com> wrote: https://github.com/wikimedia/search-mjolnir. This is a pyspark application that starts with the logged click data and transforms it into trained models. The models are currently trained using xgboost, but we are considering lightgbm as a replacement. Collecting click data is done separately with some processing of web request logs to match up search requests with their clicks. And can you elaborate on how the aggregated search queries are PII? used to learn a ranking function needs to be provided the original query string. That string is then not aggregated, it is passed straight through from the users keyboard to the output data. We unfortunately don't have the kind of search volume, and don't keep long enough records (only 90 days) , to place arbitrary limits for minimum unique sessions issuing a query, and still have data that is representative of the whole. For example on english wikipedia, which is by far the most popular, only 60% of search sessions involve a query that was issued more than 10 times in the last 90 days. And 10 times is *way* too low for public release (I'm not sure where a reasonable cutoff might be, but its certainly not 10). Thank you! We use a DBN (chapelle, 2009) to transform click stream data into labeled search result data, and then LambdaMART for the final ranking model. Link to mjolnir which does the training linked above. arbitrary information and if you want to build a ranking function you have to have those original queries to do feature collection.

On Mar 15, 2018 6:41 AM, "Georg Sorst" &lt;g.sorst(a)findologic.com&gt; wrote: Hi Erik, Erik Bernhardson &lt;ebernhardson(a)wikimedia.org&gt; schrieb am Mi., 14. März 2018 um 15:34 Uhr: Great stuff, thank you! Just for my understanding (not a Machine Learning expert yet :) ): I would need (query -> document) pairs such as ("machine learning" -> https://en.wikipedia.org/wiki/Machine_learning) and how often each of these pairs has ocurred, right? Even if this pair has only occured once, how is this PII? Or do I need more than just (query -> document)? You only need the (query, document) pairs, the PII is the query string itself. To start at the beginning of the pipeline with clicks you don't want click counts, but search sessions. Each search session here is a query, a list of matched documents in the order presented, and timestamped clicks to those results. From that data it can be aggregated to click counts if you want, but I think that would lose very important position bias information from the sessions. We consider all submitted search queries to potentially be PII. In the past when we have released lists of search queries we have multiple people under NDA review and remove PII which typically filters around 10% of the queries. This means any kind of phone number, serial number, or non-notable address. We remove searches for any specific URL, and names of non-notable companies and non-notable people (those that don't have wiki articles and aren't mentioned prominently in any other article). Some time ago, before any current members of search platform were involved, there was a release of unfiltered search queries. This had to be taken down almost immediately after the community reported PII leakage with specific examples from the release. Thank you so much, this is all very enlightening! Georg -- *Georg M. Sorst I CTO* FINDOLOGIC GmbH Jakob-Haringer-Str. 5a | 5020 Salzburg I T.: +43 662 456708 E.: g.sorst(a)findologic.com www.findologic.com Folgen Sie uns auf: XING <https://www.xing.com/profile/Georg_Sorst>facebook <https://www.facebook.com/Findologic> Twitter <https://twitter.com/findologic> Wir sehen uns auf der *SHOPTALK* von 18. bis 21.03 in *Las Vegas*! Hier &lt;beratung(a)findologic.com?subject=SHOPTALK%20Las%20Vegas&gt; Termin vereinbaren! Wir sehen uns auf der *SOM* am 18.04. & 19.04. in *Halle 7 Stand G.17 in Zürich*! Hier &lt;beratung(a)findologic.com?subject=SOM%20Z%C3%BCrich&gt; Termin vereinbaren! Wir sehen uns auf dem *SHOPWARE Community Day* am 18.05.* in Duisburg*! Hier &lt;beratung(a)findologic.com?subject=Shopware%20Community%20Day&gt; Termin vereinbaren! Wir sehen uns auf der *OXID Commons* am 14.06. *in Freiburg*! Hier &lt;beratung(a)findologic.com?subject=OXID%20Commons&gt; Termin vereinbaren! Hier <http://www.findologic.com> geht es zu unserer *Homepage*! _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics