For a little backstory, in discernatron multiple judges provide scores in
from 0 to 3 for results. Typically we only request a single query to be
reviewed by two judges. We would like to measure the level of disagreement
between these two judges, and if it crosses some threshold get two more
scores, so we can then measure disagreement in the group of 4. Somehow
though, we need to define how to measure that level of disagreement and
what the threshold for needing more scores is.
Some specialized concerns:
* It is probably important to include not just that the users gave
different values, but also how far apart they are. The difference between a
3 and a 2 is much smaller than between a 2 and a 0.
* If the users agree that 80% of the results are all 0, but disagree on the
last 20%, even though the average disagreement is low it's probably still
important? Might be worthwhile to take all the agreements about irrelevant
results and remove them before calculating disagreement? Not sure...
I know we have a few math nerds here on the list, so hoping someone has a
few ideas.
The title suggest indices cron were disabled in preparation for this
maintenance. No re-indexing happened this night. Since we plan the
reboot for tomorrow, it does not make sense to re-enable them right
now. We will run on slightly outdated indices until Wednesday.
If the reboot is delayed again tomorrow, I'll manually run the
indexing. If all goes to plan, I'll just re-enable the cron jobs.
On Mon, Oct 31, 2016 at 8:46 AM, Moritz Mühlenhoff
<mmuhlenhoff(a)wikimedia.org> wrote:
>> I'll make a new attempt on Monday (31st) at around 7am UTC. If there's something
>> which is safe to abort and which you know will still be running, please send
>> me an email.
>
> Due to a bigger reindexing job which didn't complete in time, this
> will be moved to tomorrow
> 1st Nov at 7am UTC.
>
> Cheers,
> Moritz
>
> _______________________________________________
> Engineering mailing list
> Engineering(a)lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/engineering
--
Guillaume Lederrey
Operations Engineer, Discovery
Wikimedia Foundation
UTC+2 / CEST
After extensive testing over the last several months using a new search
query scoring method called BM25 (Best Matching) [1], we recently completed
a limited
production
release to the following top languages: English, German, Spanish, Russian,
Portuguese, French, Italian, Polish, Dutch and Arabic. This new release is
replacing the older search method called tf-idf (term frequency-inverse
document frequency) [2].
We have
additional
testing to do [3,4] to figure out if BM25 will work in languages that
don’t use spaces in-between their words
,
i.e.: Japanese, Chinese, etc.
The Discovery team announces much of
our
completed work in weekly status updates [5
, 6
], but some of the work isn’t actually obvious to anyone who uses our
search engine
- t
hat is because it isn’t actually ‘live’ until a complete re-index of the
servers occur. We’ve created a recurring ticket in Phabricator [
7
] to keep track of the work that goes live
in production
after a re-index, such as the one we’ve also just completed. A few
highlights
of the
recent
re-index
are implementing ascii-folding for the French language and
fixing
several
bugs
for French ÿ, and Russian ’Е’ and 'Ё' when
those characters are
entered in a search query.
Cheers from the Discovery Search Team!
[1] https://en.wikipedia.org/wiki/Okapi_BM25
[2] https://en.wikipedia.org/wiki/Tf%E2%80%93idf
[3] https://phabricator.wikimedia.org/T147495
[4] https://phabricator.wikimedia.org/T147501
[5]
https://www.mediawiki.org/wiki/Wikimedia_Discovery#Updates
[
6
] https://www.mediawiki.org/wiki/Discovery/Status_updates
[
7
] https://phabricator.wikimedia.org/T147505
--
deb tankersley
Product Manager, Discovery
irc: debt
Wikimedia Foundation
Fantastic, thanks so much! :)
--
deb tankersley
Product Manager, Discovery
irc: debt
Wikimedia Foundation
On Sun, Oct 23, 2016 at 11:56 AM, Romaine Wiki <romaine.wiki(a)gmail.com>
wrote:
> Hi Deborah,
>
> I have now translated all the (missing) phrases for Dutch.
>
> Romaine
>
> 2016-09-16 23:46 GMT+02:00 Deborah Tankersley <dtankersley(a)wikimedia.org>:
>
>> Hi,
>>
>> The Discovery team has finished up another language in our quest to
>> enable language detection for better search results. We now need your help
>> in translating the phrase "*showing results from" *in Dutch for these
>> languages: English, Chinese, Arabic, Korean, Greek, Hebrew, Japanese,
>> and Russian.
>>
>> It would be great if we can get these new translations into
>> translatewiki using the following message keys (and this message group
>> link): https://translatewiki.net/wiki/Special:Translate?grou
>> p=ext-wikimediainterwikisearchresults:
>>
>> Dutch (Nederlands)
>> <https://translatewiki.net/w/i.php?title=Special:Translate&group=ext-wikimed…>
>> [1]:
>>
>> search-interwiki-results-enwiki
>> search-interwiki-results-ruwiki
>> search-interwiki-results-hewiki
>> search-interwiki-results-jawiki
>> search-interwiki-results-arwiki
>> search-interwiki-results-zhwiki
>> search-interwiki-results-kowiki
>> search-interwiki-results-elwiki
>>
>>
>> Cheers and thanks for your time!
>>
>>
>> [1] https://translatewiki.net/w/i.php?title=Special:Translat
>> e&group=ext-wikimediainterwikisearchresults&language=nl&
>> filter=&action=translate
>>
>
Hello!
I just finished writing a short incident report on the maps issue of
last Friday [1].
Basically, I was stupid and reinitialized Cassandra on the wrong node.
This should have had almost no effect, except that the system_auth
keyspace of Cassandra is configured with a replication factor of 1.
Loosing that node means that we also lost authentication.
Thanks a lot to Brandon for the help in mitigating this!
[1] https://wikitech.wikimedia.org/wiki/Incident_documentation/20161021-Maps
--
Guillaume Lederrey
Operations Engineer, Discovery
Wikimedia Foundation
UTC+2 / CEST
Hi!
> 3) Move the service to labs, not providing any firm guarantee of service
> level ?
I don't see any reason to move it to labs, and I don't think labs has
infrastructure to handle it. It can not run on virtual machines - in
fact, the only reason why it can't be as stable as we wish is because we
don't have hardware to run fully redundant configuration with capacity
to serve all cases. I don't see how using labs would help there - does
labs have more hardware capacity than production that is available for
wdqs use?
> I think there was one exception, which is services that needed a lot of
> resources so they could not run on vms, but don't we have a prototype of
> "labs on real hardware"?
I really wouldn't want to run this service on prototype that wasn't
tried before and introduce both additional point of failure and
unnecessary administration burden. I fail to see any existing issue that
this would solve, but it certainly has potential to introduce some.
> that you are describing- running easily out of resources (DOS). Even
> quarry, which I have publicly complained about in the past, for what you
> say, has a better resource management than wqs (30-minute limit
> execution, concurrency control, etc.).
We have 30-second limit now. People complain about it all the time
because it's too short, but see above about the hardware.
> icinga. But running an unstable service (wdqs) on top of another
> unstable service (wikidata data handling) will never be stable.
> Everytime a bot starts writing to wikidata 600 times per second, s5 dbs
> shake (that is why we are creating s8) and wqs goes down. :-)
> I would suggest using wqs on labs (or anywhere, non-production) with
> regular imports rather than real-time updates. Less headaches. I am
> literally aiming for that for labsdbs, too.
I don't think this is a good scenario. Delayed updates means severely
degraded user experience and won't save much performance as the data
needs to be moved over anyway. We could save something if we had proper
change tracking service that doesn't use recentchages API directly on
wikidata (so we get several uncached rc API calls per each edit) but has
some faster and more efficient intermediary, but AFAIK we don't have
that still. That would be a direction to look if updating is too
resource-consuming.
--
Stas Malyshev
smalyshev(a)wikimedia.org
