Magnus Manske wrote: We used to do it, and the plan was to make it public, however, there are privacy issues apparently and no-one knows if we can or cannot publish them, and in what format etc.. So since it was filling up the disk and was not used, I have disabled it until a solution and storage space is found. Cheers, r.

On Thu, Jan 14, 2010 at 3:27 PM, Nikola Smolenski <smolensk(a)eunet.rs> wrote: I guess people searching for their own name, or the like. Suggestion : * log search and SHA1 IP hash (anonymous!) * search queries are logged in a standardized fashion (for grouping), e.g. lowercase, single spaces, no leading/trailing spaces, special chars converted to spaces, etc. * display searches per week (?) that have been searched for at least 10 times from at least 5 different IP hashes (to avoid people searching their own name 100 times...) Magnus

On Thu, Jan 14, 2010 at 4:47 PM, Magnus Manske <magnusmanske(a)googlemail.com> wrote: There are only 2 billion unique addresses and they can all be found in half an hour probably. Bryan

On Thu, Jan 14, 2010 at 11:01 AM, David Gerard <dgerard(a)gmail.com> wrote: Magnus was not suggesting disclosing the IP hash, as far as I can tell. He demonstrating an abundance of caution in suggesting only logging that. (er, well, yea, if he was suggesting disclosing that... we shouldn't do that. Even if we add a secret to the hash, it's risky and allows interesting correlation attacks) Here is what I would suggest disclosing: #start_datetime end_datetime hits search_string 2010-01-01-0:0:4 2010-01-13-23-59-50 39284 naked people 2010-01-01-0:0:4 2010-01-13-23-59-50 23950 hot grits ... 2010-01-01-0:0:4 2010-01-13-23-59-50 5 autoerotic quantum chromodynamics Which has first been filtered by: * Canonicalization of strings (at least ascii case folding) * Excluding strings over some length * Excluding searches which did not come from at least 5 distinct IPs during the reporting interval There will be useful information excluded by this process, e.g. gads of misspellings which came from only two to four unique IPs... but the output would still be *far* more useful no information at all.

2010/1/14 Gregory Maxwell <gmaxwell(a)gmail.com>om>: my $0.02 I expect some fun here, since error encodings will hit things like &, ñ, ó. 2010-01-01-0:0:4 2010-01-13-23-59-50 de.wikipedia 25093 Bondage amp; Disziplin Pokémon 2010-01-01-0:0:4 2010-01-13-23-59-50 de.wikipedia 25093 Bondage %33amp; Disziplin Pokémon .... on the other part, all these errors will be browser/proxy bugs, and not mediawiki bugs. I think. Anyway, if the "special characters" are replaced by spaces, there will be less weird shit, and more misterious space holes. -- -- Fin del Mensaje.

On Thu, Jan 14, 2010 at 12:22 PM, Conrad Irwin <conrad.irwin(a)googlemail.com> wrote: The logs are taken from the Squids, long before MediaWiki touches them, so they shouldn't be normalized at all. Some people might search for their own name more than five times in a week, possibly together with other embarrassing or incriminating search terms.

Whoops, that should have been 14, can't do maths any more; sorry for the spam. Conrad

Aryeh Gregor wrote: Search isn't cached, so it may be easier to just log it at the backend. I expect many people using things like "please tell me how many people live in China", as revealed by such titles being created. My conclusion is that some people (10%?) don't know how to search in a encyclopedia. I mean, we have an article called [[China]] with a proper Population section... While reading this thread I have deleted a page called "Why do ghosts manifest themselves?" with content "fogcpijkñldjlkcmvlkmc.,vmblcjgmlkjglkjmf,.mfdgfdolfgdjk" [1]. I'm thinking in an extension to feed with regex extracting the actual title they may be loking for. Sampled search logs are unlikely to reveal them though, since what they are repeating are the non-keywords, not the full query. 1-http://es.wikipedia.org/w/index.php?title=Special:Log&page=Por_que_se…

On Thu, Jan 14, 2010 at 12:22 PM, Conrad Irwin <conrad.irwin(a)googlemail.com> wrote: You've missed the point of the normalization here. It's not to be helpful to users: As you observe, it's easy for the recipient of the list to perform their own. The reason to normalize is to push more queries above the reporting threshold. For example, 5 people might search for "john f. kinndey" (a misspelling of "John F. Kennedy"?) but all capitalize it differently. A redirect on this misspelling would be useful regardless of the case. All things equal I'd rather *not* normalize the data... it's just more stuff that may have surprising behaviour. But I think this is something which may need to be balanced against the disclosure threshold. It would also be possible to do the disclosure calculation against normalized data while releasing the raw values... but I must admit a little bit of uneasiness that the normalization might be ignoring some piece of information relevant to privacy. For example, if we were to go that route we might employ some fairly aggressive normalization... removing all whitespace and punctuation. If we went as far as also removing all *numbers* from the check we'd run into things like "Greg Maxwell (555)-555-1212" getting published because enough distinct people searched for "greg maxwell". Obviously the answer to that one is "don't remove numbers" from the check, but I worry about the cases I haven't thought of. On Thu, Jan 14, 2010 at 12:51 PM, Aryeh Gregor <Simetrical+wikilist(a)gmail.com> wrote: Yes, it's possible that someone may search 5 times, from 5 IPs (which *might* be from one machine due to proxy round-Robbins), an identical string ... "MyFullName seen on friday night with a woman other than his wife" ... but what to do? Any information which is disclosed has some risk of disclosing something that someone would rather not be. This risk can be made arbitrarily small, but it can't be eliminated. I think the benefit to the readers of having this information available easily outweighs some sufficiently fringe confidentiality concern. At some point your frequently repeated search is a statistic, which no reasonable privacy policy would frown on disclosing. This is important to our operations, disclosing it is in the public interest, and failing to do work in this area puts us at a disadvantage compared to other parties who might be far less scrupulous. (e.g. If WMF's search performs poorly, you might feel compelled to use Search Engine X — which happens to secretly sell your data to the highest bidder.) Is there some sufficiently high number which *no one* paying attention here has a concern about? We could simply start with that.... and possibly lower the threshold over time as the lowest hanging fruit are solved, tracking our disclosure comfort. I think we all have an interest and obligation to take every reasonable means, but no one can ask for more than that. Would anyone feel more comfortable if this ignored queries made via the secure server? Non-HTTPS traffic can be watched by anyone on the path between you and Wikimedia... any illusion of absolute privacy on the insecure traffic is patently false already.