March 2016 - Discovery - lists.wikimedia.org

Search box update gets press
by Moiz Syed 13 Apr '16

13 Apr '16

Hey all, Just came across this: http://thenextweb.com/insider/2016/03/11/wikipedia-spruces-search-autocompl… WOOT!!!!

3 4

Re: [discovery] [Ops] Wikidata Query Service (WQDS) regular deployment window
by Greg Grossmeier 06 Apr '16

06 Apr '16

<quote name="Moritz Muehlenhoff" date="2016-03-22" time="22:11:58 +0100"> > On Tue, Mar 22, 2016 at 10:04:21PM +0100, Guillaume Lederrey wrote: > > Let me know if you have any question or if you know of another place > > where I should publicize this deployment window. > > We have a page on wikitech which tracks all deployment/maintenance > windows: https://wikitech.wikimedia.org/wiki/Deployments Yes, please add your window there. As I see you want a 7 hour window, overlapping with other windows shouldn't be a problem. But, if you ever plan to do work that might impact more than just your own systems, please do coordinate with me. Thanks, Greg -- | Greg Grossmeier GPG: B2FA 27B1 F7EB D327 6B8E | | identi.ca: @greg A18D 1138 8E47 FAC8 1C7D |

3 2

Wikipedia.org Portal stats are moving in the right direction!
by Deborah Tankersley 31 Mar '16

31 Mar '16

Hi all, While updating the Q4 goals [1] for the Wikipedia.org Portal Team today, I wanted to see where we currently are with the bounce rate of visitors to the portal and found some really good news on our dashboard [2]. Back in December 2015, we had approximately 60% of our visitors to the portal page arrive and then leave without doing anything (bounce rate). We also had about only 30% of the portal visitors doing a search on the page and other actions were much lower. Now, as of March 22, 2016, our bounce rate is down to 44% and the searches are up to 42%. Which is a very nice improvement (due to many explainable and unexplainable things) and it shows that our numbers are certainly moving in the right direction. Woohoo! Check out this image [3] for the pretty dashboard stats picture. :) Happy Friday! Deb [1] https://www.mediawiki.org/wiki/Wikimedia_Engineering/2015-16_Q4_Goals#Wikip… [2] http://discovery.wmflabs.org/portal/#action_breakdown (go to Smoothing dropdown and select Monthly Median) [3] https://commons.wikimedia.org/wiki/File:Bounce_Rate_for_Wikipedia_Portal_De… -- Deb Tankersley Product Manager, Discovery Wikimedia Foundation

2 1

Re: [discovery] Input requested about this mailing list
by Chris Koerner 30 Mar '16

30 Mar '16

>Not directly related, but should we add the new #wikimedia-interactive channel to the the list of IRC channels? DONE! :) -- Yours, Chris Koerner Community Liaison - Discovery Wikimedia Foundation

1 0

Hardware requests and visibility
by Guillaume Lederrey 30 Mar '16

30 Mar '16

Hello team! I'm trying to get some visibility on our hardware requests, so that I can follow them properly and in turn give you visibility on what's happening. First step is knowing what is requested. Phabricator should help me there. A query for "tag contains Discovery && tag contains hardware request" [1] gives me 2 tasks (Maps backend hardware, Refresh elastic10{01..16}.eqiad.wmnet servers). I know that there are other tasks out there (Relevance forge hardware for example). If you know of any hardware request for the discovery team that is not tagged "Discovery" && "Hardware-requests", can you send it my way? Or suggest another query that could help me get them? Thanks for your help! MrG [1] https://phabricator.wikimedia.org/maniphest/query/Fta7weLW2Pa5/ (no idea if that link is actually shareable...) -- Guillaume Lederrey Operations Engineer, Discovery Wikimedia Foundation

3 3

Input requested about this mailing list
by Kevin Smith 30 Mar '16

30 Mar '16

Hello discovery mailing list members. We'd like to ask for some feedback on how the Discovery team can best use this list. We don't want to muddle the list with information that folks outside the team don't care about, and we want to make sure that what we post is of value. What sort of information do you think we should share here? It seems clear that items like these should go on this list: - Some interesting findings - we ran a test and found x=y! - Something new and we want to suss out interest - What if we moved the search box to a different position on the page‽ - Upcoming projects, or other milestones related to discovery - X users opted-in to the new autocompletion suggester. - Discussions about quarterly goals and roadmaps There are other items that we could share on this list, but would anyone outside the team be interested? For example, process-related items like: - Portal team decided to have 3x standups per week instead of 2. - Team structure changes, or plans to change how we do retrospectives - Notifications of meeting minutes being posted (e.g. Weekly team meeting minutes, retrospective minutes, etc.) For Discovery team members reading this: What criteria do you use (or wish you could use) when thinking, "Should I post this to public list?" Kevin Smith Agile Coach, Wikimedia Foundation

7 8

Completion suggester: zero results rate has decreased!
by Dan Garry 29 Mar '16

29 Mar '16

Coinciding with the launch of the completion suggester, we've seen a very large decrease in the zero results rate for prefix search <http://discovery.wmflabs.org/metrics/#failure_breakdown>! In fact, the zero results rate for prefix search is now lower than that for full text search. Similarly, we've seen a decrease in the overall zero results rate KPI <http://discovery.wmflabs.org/metrics/#kpi_zero_results>. Yay! Dan -- Dan Garry Lead Product Manager, Discovery Wikimedia Foundation

4 5

Wikipedia Portal clickthrough rates & section usage vary by English vs non-English speakers
by Mikhail Popov 25 Mar '16

25 Mar '16

Hi all! On 17 March 2016 the Portal team deployed a patch to Wikipedia Portal to log all user preferred languages. From the analysis of this additional data, we found that approximately 70% of Wikipedia Portal visitors only have English as their preferred language, and approx. 18% set a language other than English. Around 12% of our users are multilingual (according to their browser preferences), but many of those included English. Users whose primary language is English or who included English as a preferred language clicked through and searched at a remarkably higher rate on a daily basis than users whose primary language is another language (60% vs 50% and 50% vs 30%), but the latter group used the language links at a much higher rate than the former (10% vs 20%). In fact, English-speaking visitors' overall clickthrough rate was 12.5%-14.4% higher than non-English-speaking visitors', and were 1.3 times more likely to click through, indicating, perhaps, that increased localization efforts could better engage our non-English-speaking visitors. Based on the data and patterns observed, we strongly support and encourage the language detection and localization efforts the Portal team has begun pursuing. These and other findings can be found in the report on Commons <https://commons.wikimedia.org/wiki/File:Analysis_of_Clickthrough_Rates_and_…>. The graphs are especially cool and informative, even if I do say so myself. Thanks, Mikhail on behalf of Discovery Analytics -- *Mikhail Popov* // Count Logula, Discovery <https://www.mediawiki.org/wiki/Wikimedia_Discovery> https://wikimediafoundation.org/ *Imagine a world in which every single human being can freely share in the **sum of all knowledge. That's our commitment.* Donate <https://donate.wikimedia.org/>.

2 2

Re: [discovery] Discovery IRC Office Hours March 30th 8:30 UTC
by Chris Koerner 24 Mar '16

24 Mar '16

Yes this is a little early for the SF folks. I messed up. The correct time is 18:30 UTC -- Yours, Chris Koerner Community Liaison - Discovery Wikimedia Foundation

2 1

Discovery IRC Office Hours March 30th 8:30 UTC
by Chris Koerner 24 Mar '16

24 Mar '16

Hello Discovery department, To experiment with working with interested people outside of our team I'd like host an IRC office hour next Wednesday, March 30th.* (That's 1:30 p.m. San Francisco time). The topic will be recent efforts the Discovery team has been working on. So come to share with our communities what you've been up to. To mix things up a bit, I've also scheduled a Bluejeans (video chat) for those who wish to participate in that manner. I'll work as an intermediary between the two mediums. https://meta.wikimedia.org/wiki/IRC_office_hours#Upcoming_office_hours https://bluejeans.com/388063933/ -- Yours, Chris Koerner Community Liaison - Discovery Wikimedia Foundation *I didn't come up with this on my own. :) Tomasz and Kevin helped to organize this endeavor.

2 1

Dashboards down (but we have backups!)
by Mikhail Popov 23 Mar '16

23 Mar '16

There's an issue with Vagrant that's preventing our dashboards from running, but does not impact the data processing scripts we have so for now we can use copies of the dashboards running on our beta instance: http://discovery-beta.wmflabs.org/ Bryan Davis is working on resolving this issue ( https://phabricator.wikimedia.org/T130803). Apologies for the inconvenience. -- *Mikhail Popov* // Count Logula, Discovery <https://www.mediawiki.org/wiki/Wikimedia_Discovery> https://wikimediafoundation.org/ *Imagine a world in which every single human being can freely share in the **sum of all knowledge. That's our commitment.* Donate <https://donate.wikimedia.org/>.

2 1

Search Satisfaction/Success/Whatever Metrics and The Relevance Forge
by Erik Bernhardson 22 Mar '16

22 Mar '16

This thread started off list, but I'm hoping all of you watching along can help us along to brainstorm and improve search satisfaction. Note that these aren't all my thoughts, they are a conglomeration of thoughts (many copy/pasted from off-list emails) from Trey, David, Mikhail and I. That's also why this might not all read like one person wrote it. A few weeks ago I attended ElasticON and there was a good presentation about search satisfaction by Paul Nelson. One of the things he thought was incredibly important, that we had already been thinking about but hadn't moved forward enough on, was generating an Engine Score. This week Paul held an online webinar where he gave the same presentation but without such strict time constraints which Trey attended. You can find my summary of this presentation in last weeks email to this list, 'ElasticON notes' Some things of note: - He doesn't like the idea of golden corpora—but his idea is different from Trey's. He imagines a hand-selected set of "important" queries that find "important" documents. I don't like that either (at least not by itself). I always imagine a random selection of queries for a golden corpus. - He lumps BM25 in with TF/IDF and calls them ancient and unmotivated and from the 80s and 90s. David's convinced us that BM25 is a good thing to pursue. Of course, part of Search Technologies' purpose is to drum up business, so they can't say, "hey just use this in Elastic Search" or they'd be out of business. - He explains the mysterious K factor that got all this started in the first place. It controls how much weight changes far down the results list carry. It sounds like he might tune K based on the number of results for every query, but my question about that wasn't answered. In the demo, he's only pulling 25 results, which Erik's click-through data shows is probably enough. - He mentions that 25,000 "clicks" is a good enough sized set for measuring a score (and having random noise come out in the wash). Not clear if he meant 25K clicks, or 25K user sessions, since it was in the Q&A. David and Trey talked about this some, and Trey think's the idea of Paul's metric (Σ power(FACTOR, position) * isRelevant[user, searchResult[Q,position].DocID]) has a lot of appeal. It's based on clicks and user sessions, so we'd have to be able to capture all the relevant information and make it available somewhere to replay in Relevance Forge for assessment. We currently have a reasonable amount of clickthrough data collected from 0.5% of desktop search sessions that we can use for this task. There are some complications though because this is PII data and so has to be treated carefully. Mikhail's goal for our user satisfaction metric is to have a function that maps features including dwell time to user satisfaction ratio. (e.g., 10s = 20% likely to be satisfied, 10m = 94% likely to be satisfied, etc.). The predictive model is going to include a variety of features of varying predictive power, such as dwell time, clickthrough rate, engagement (scrolling), etc. One problem with the user satisfaction metric is that it isn't replayable. We can't re-run the queries in vitro and get data on what users think of the new results. However it does play into Nelson's idea, discussed in the paper and maybe in the video, of gradable relevance. Assigning a user satisfaction score to a given result would allow us to weight various clicks in his metric rather than treating them all as equal (though that works, too, if it's all you have). We need to build a system that we are able to tune in an effective way. As pointed by Trey cirrus does not allow us to tune the core similarity function params. David tend's to think that we need to replace our core similarity function with a new one that is suited for optimizations and BM25 allows it, there are certainly others and we could build our own. But the problem will be: How to tune these parameters in an effective way, with BM25 we will have 7 fields with 2 analyzers : 14 internal lucene fields. BM25 allows to tune 3 params : weight, k1, and b for each field - weight is likely to range between 0 and 1 with maybe 2 digits precision steps - k1 from 1 to 2 - b from 0 to 1 And I'm not talking about the query independent factors like popularity, pagerank & co that we may want to add. It's clear that we will have to tackle hard search performance problems... David tend's to think that we need to apply an optimization algorithm that will search for optimal combination according to an objective. David doesn't think we can run such optimization plan with A/B testing, it's why we need a way to replay a set of queries and compute various search engine scores. We don't know what's the best approach here: - extract the metrics from the search satisfaction schema that do not require user intervention (click and result position). - build our own set of queries with the tool Erik is building (temporary location: http://portal.wmflabs.org/search/index.php) -- Erik thinks we should do both, as they will give us completely different sets of information. The metrics about what our users are doing is a great source of information provides a good signal. The tool Erik is building comes at the problem from a different direction, sourcing search results from wiki/google/bing/ddg and getting humans to rate which results are relevant/not relevant on a scale of 1 to 4. This can be used with other algorithms to generate an independent score. Essentially I think the best Relevance Forge will output a multi-dimensional engine score and not just a single number. -- We should set up records of how this engine score changes over days, months, and longer, so we can see a rate of improvement (or lack thereof. But hopefully improvement :) And in the end will this (BM25 and/or searching with weights per field) work? - not sure, maybe the text features we have today are not relevant and we need to spend more time on extracting relevant text features from the mediawiki content model (https://phabricator.wikimedia.org/T128076) but we should be able to say : this field has no or only bad impact impact. The big picture would be: - Refactor cirrus in a way that everything is suited for optimization - search engine score: the objective (Erik added it as goal) - Optimization algorithm to search/tune the system params. Trey has prior experience working within optimization frameworks. Mikhail also has relevant machine learning experience. - A/B testing with advanced metrics to confirm that the optimization found good combination With a framework like that we could spend more time on big impact text features (wikitext, synonyms, spelling correction ...). But yes it's a huge engineering task with a lot of challenges :/ It's also

4 5

Wikidata Query Service (WQDS) regular deployment window
by Guillaume Lederrey 22 Mar '16

22 Mar '16

Hello! After discussion with Stas, we want to have a regular deployment window for Wikidata Query Service. This should help give better visibility on when new version arrives and help track issues with those new versions. I will take care of the deployments (with Stas' support, of course). The deployment window is: every Monday, from 7pm CET (10am PST - 5pm UTC) starting from Monday April 11th. Let me know if you have any question or if you know of another place where I should publicize this deployment window. Take care, Guillaume -- Guillaume Lederrey Operations Engineer, Discovery Wikimedia Foundation

2 1

Re: [discovery] [Ops] mini-hackathon
by Guillaume Lederrey 22 Mar '16

22 Mar '16

Thanks for the list! It is a long list... I'm sure there are a few interesting tasks for our purpose in there. It will take some time reading them... On Tue, Mar 22, 2016 at 4:21 PM, Filippo Giunchedi <fgiunchedi(a)wikimedia.org> wrote: > Hi Guillaume, > I think it is a nice idea! For #operations specifically I don't think we > have an explicit tag for this kind of tasks. > > Though a list of Low/Lowest #operations tasks sorted by date updated could > help/guide the selection, > https://phabricator.wikimedia.org/maniphest/query/MEKdyfITov4i/#R > > HTH, > filippo > > > On Tue, Mar 22, 2016 at 1:21 PM, Guillaume Lederrey > <glederrey(a)wikimedia.org> wrote: >> >> Hello! >> >> I can't make it to the Wikimedia Hackathon [1]. I like the concept and >> I have a few friends and former coworkers who would be interested in >> sending some contributions to Wikipedia. So we decided to spend the >> day together on Saturday April 9th and see if we can find tasks that >> need some love. I have a small list, but if you have any additional >> idea of something easy to work on, send them my way. The goal is as >> much to expose my friends to what we do as to contribute meaningful >> improvements. >> >> I'm looking for tasks that are mainly Ops oriented (that's the >> background for most of us, even if some have experience in PHP, >> Pyhton, and a few other area). Tasks that are simple enough to be done >> in a day. Tasks that do not require understanding the complete WMF >> architecture to get started. >> >> Let me know... >> >> Guillaume >> >> >> [1] https://www.mediawiki.org/wiki/Wikimedia_Hackathon_2016 >> >> >> -- >> Guillaume Lederrey >> Operations Engineer, Discovery >> Wikimedia Foundation >> >> _______________________________________________ >> Ops mailing list >> Ops(a)lists.wikimedia.org >> https://lists.wikimedia.org/mailman/listinfo/ops > > -- Guillaume Lederrey Operations Engineer, Discovery Wikimedia Foundation

1 0

mini-hackathon
by Guillaume Lederrey 22 Mar '16

22 Mar '16

Hello! I can't make it to the Wikimedia Hackathon [1]. I like the concept and I have a few friends and former coworkers who would be interested in sending some contributions to Wikipedia. So we decided to spend the day together on Saturday April 9th and see if we can find tasks that need some love. I have a small list, but if you have any additional idea of something easy to work on, send them my way. The goal is as much to expose my friends to what we do as to contribute meaningful improvements. I'm looking for tasks that are mainly Ops oriented (that's the background for most of us, even if some have experience in PHP, Pyhton, and a few other area). Tasks that are simple enough to be done in a day. Tasks that do not require understanding the complete WMF architecture to get started. Let me know... Guillaume [1] https://www.mediawiki.org/wiki/Wikimedia_Hackathon_2016 -- Guillaume Lederrey Operations Engineer, Discovery Wikimedia Foundation

1 0

Re: [discovery] WDQS deployments
by Guillaume Lederrey 22 Mar '16

22 Mar '16

Just wondering: why do you want a regular deployment schedule? I'm all for it, and I can see a few reasons to support this decision (if you start delegating deployment, it is easier on a regular schedule would be my main reason). In general, I'd prefer to have this deployment window early enough in my day so that I have time to fix it if I screw up. Ideally, 10-11am UTC Monday, Tuesday of Thursday. It might make sense for you to be available (at least for the first few deployments), so it might be better to have it later (3-4pm UTC?). For the afternoon, any day could work... On Fri, Mar 18, 2016 at 7:05 PM, Stas Malyshev <smalyshev(a)wikimedia.org> wrote: > Hi! > > I thought a bit more about WDQS deployments and I think it's time to > have regular deployment schedule, at least while we're still in > development. So I though about having weekly deployment point where most > of the non-urgent things will be deployed. We could of course deploy > urgent stuff anytime, but I think having a scheduled point would be also > good. > > Since I'd like you to do this in most cases, I wonder what day/time > would work best for you? > > Thanks, > -- > Stas Malyshev > smalyshev(a)wikimedia.org -- Guillaume Lederrey Operations Engineer, Discovery Wikimedia Foundation

1 0

Discovery Weekly Update for the week starting 2016-03-14
by Chris Koerner 21 Mar '16

21 Mar '16

Hello, Here is the Discovery department's weekly status update. * The completion suggester left beta and is now the default search-as-you-type for all wikis (except Wikidata). ** http://blog.wikimedia.org/2016/03/17/completion-suggester-find-what-you-nee… * Last week we enabled Kartographer extension for Wikivoyage sites, allowing users to add maps to wiki pages without any additional wmf labs and JavaScript tricks. ** A demo of Kartographer and VisualEditor integration can be found here: http://vem3.wmflabs.org/wiki/Main_Page This is our second week summarizing our work in this way and our first week sharing it with wikitech-l. Feedback and suggestions are welcome. Read the full update at the following link. https://www.mediawiki.org/wiki/Discovery/Status_updates/2016_03_18 -- Yours, Chris Koerner Community Liaison - Discovery Wikimedia Foundation

1 0

Knowledge discovery: brainstorm & research
by Luigi Assom 19 Mar '16

19 Mar '16

Hello everybody, in late 2015 I was asking for tech info to document myself for a project about knowledge discovery I was implementing. I am now happy to have published it and I will be happy to hear for comments to improve utility. http://nifty.works is a side project I built for brainstorming and help in research of factual knowledge. I used a data model I co-authored during my first entrepreneurial experience in Italy, and that I also experimented in a mobile app. Here, I wanted to bring the things forward and make something publicly available over the web! This project is about experimental interaction to traverse knowledge graphs. I implemented this functionalities: - query the context <http://nifty.works/about/PAyD8rYKqxEM47Vk/machine-learning#topic-article> of a topic - test for suggesting logical paths <http://nifty.works/path/> in between of topics (pathway discovery) - a gamified <http://nifty.works/play/> version of knowledge networks, as test to engage people in learning (it s still a nerdy prototype I know) - edit maps (selectively add or remove nodes, available if signed up) - sharing maps - pairing wikipedia articles with entities in the knowledge graph (hyperlinks in article will trigger corresponding topics) For full-text search, I am now using a very very basic indexing in elasticsearch; you can test a difference with wikipedia indexing by adding test=ab_fts parameter (I don't know which is better). The nice thing is that I could localise the site for wikipedias not in English, and uncover cultural knowledge strongly related to the language they are written. I think it may be interesting for there are languages spoken by millions of people but poorly addressed by recommender systems (at my knowledge at least). Thinking about swedish, thai, chinese, dutch, hispanic... to name a few! I would like to hear from your comments to improve it and help in making a useful service - comments in UX/UI, tech improvements, communication, functionalities as well as opportunities to sustain it (crowd-funding?) are most welcome! Thank you for time and for your help in answering a lot of questions in the forum! I hope to give back a bit here. Cheers, Luigi

1 0

High priority task added to search sprint: cross-namespace redirects
by Dan Garry 18 Mar '16

18 Mar '16

Hello! I've added T130353 <https://phabricator.wikimedia.org/T130353> to the search sprint. It's not quite an "Unbreak now!" priority task, but it's close. It seems that our fix for cross-namespace redirects has created a bit of a regression which David noticed; the redirects are scored too highly. This is a big problem for those queries, but the set of queries it actually affects is very, very, small. We should get this fixed ASAP, so I marked it as high priority and added it to the top of the sprint. Thanks! Dan -- Dan Garry Lead Product Manager, Discovery Wikimedia Foundation

1 1

Still confused about BM25 vs TF*IDF for search relevance?
by Tomasz Finc 18 Mar '16

18 Mar '16

Here is a good write up that breaks it down http://opensourceconnections.com/blog/2015/10/16/bm25-the-next-generation-o… Given the recent threads about exploring BM25, i thought this was a good introduction to the difference between the two. Cheers. --tomasz

4 3

Starting work on WDQS
by Guillaume Lederrey 17 Mar '16

17 Mar '16

Hello! Quick status (mainly for Tomasz). We took some time with Stas to go over what needs to be done on WDQS. Blazegraph seems robust enough and standard enough. I'll start playing around with it... The list of open points that I should address seems fairly thin. No upgrade planned in the short term, no specific maintenance, ... I can try to take care of incidents on the service, but it does not seems that there are many. In short: I should not have any problem starting to work on WDQS, but I'm not sure it will free all that much time from Stas... We'll see... MrG -- Guillaume Lederrey Operations Engineer, Discovery Wikimedia Foundation

2 1

Re: [discovery] [Wikimedia-l] Wikipedia.org portal page update!
by Deborah Tankersley 15 Mar '16

15 Mar '16

Hi Sam, We're glad you're using the Portal page more than usual and we hope that with upcoming future enhancements, you'll continue to enjoy it! For your first question - we are currently focusing on the wikipedia.org portal page at this time. We have no immediate plans to update the individual wiki's with this code but that doesn't mean that anyone in the community can't use our code to update another wiki to use the meta data and images as we did, in the search results. For your second question, we actually have a ticket <https://phabricator.wikimedia.org/T129627> open to look for input exactly like that. When a user types in a few words that are not in the language that has been previously selected, we're hoping to be able to switch to that "new" language on the fly. I agree that it'll be a very neat feature for us to have on the portal! Thanks for your input! Cheers, Deb -- Deb Tankersley Product Manager, Discovery Wikimedia Foundation On Tue, Mar 15, 2016 at 4:27 PM, Sam Klein <sjklein(a)hcs.harvard.edu> wrote: > I'm using the portal much more these days! Two thoughts, a few days in: > > = Is there a way to turn on thumbnailing on the individual language wikis > too? > > = Auto-suggesting language based on charset would be nice. when searching > in multiple languages I always forget once to switch to the target language > and end up getting search results for kanji on he.wp or for hebrew on ja.wp > > Thanks, SJ > > On Tue, Mar 15, 2016 at 4:25 AM, Michael Jahn <michael.jahn(a)wikimedia.de> > wrote: > > > I've played around a bit. Feels very cool! > > Michael > > > > 2016-03-11 3:27 GMT+01:00 Deborah Tankersley <dtankersley(a)wikimedia.org > >: > > > > > Hello! > > > > > > I'm very pleased to announce that we've updated the Wikipedia.org > > > <http://wikipedia.org/> portal page with a brand new search box that > is > > > more prominent and will now display meta data with images (as > available) > > in > > > the search results (see attached image). > > > > > > This was a large effort by the Discovery Portal team to develop a > > > JavaScript-only version of the language picker, so that JavaScript > > enabled > > > browsers will see all the new meta data. Alongside that effort, we also > > > ensured that in JavaScript (JS) disabled browsers (or older Internet > > > Explorer versions), our visitors won't have a bad experience when > > choosing > > > a language to search in. (Note: in older IE versions and JS disabled > > > browsers, the type-ahead and meta data search results information will > > not > > > be displayed.) > > > > > > We also implemented a shorter language code (ie: EN for English, ES for > > > Spanish, etc) to allow for more characters to be typed into the search > > box. > > > When a user toggles the language selector, the full language name will > be > > > displayed in the dropdown for easy finding of the language you prefer > to > > > search in. For the more technical minded - I've attached a screenshot > of > > > one of the ways we test our code, visually. > > > > > > We're interested in hearing your feedback or if you have any questions! > > > > > > On behalf of the very happy Wikipedia.org Portal Team, > > > > > > Deb > > > > > > -- > > > Deb Tankersley > > > Product Manager, Discovery > > > Wikimedia Foundation > > > _______________________________________________ > > > Wikimedia-l mailing list, guidelines at: > > > https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines > > > New messages to: Wikimedia-l(a)lists.wikimedia.org > > > Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, > > > <mailto:wikimedia-l-request@lists.wikimedia.org?subject=unsubscribe> > > > > > > > > > > -- > > Leiter Kommunikation > > Head of Communications > > > > Wikimedia Deutschland e.V. | Tempelhofer Ufer 23-24 | 10963 Berlin > > Tel. (030) 219 158 260 > > > > http://wikimedia.de <http://www.wikimedia.de> > > > > Stellen Sie sich eine Welt vor, in der jeder Mensch freien Zugang zu der > > Gesamtheit des Wissens der Menschheit hat. Helfen Sie uns dabei! > > > > Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e.V. > > Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg > unter > > der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für > > Körperschaften I Berlin, Steuernummer 27/029/42207. > > _______________________________________________ > > Wikimedia-l mailing list, guidelines at: > > https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines > > New messages to: Wikimedia-l(a)lists.wikimedia.org > > Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, > > <mailto:wikimedia-l-request@lists.wikimedia.org?subject=unsubscribe> > > > > > > -- > Samuel Klein @metasj w:user:sj +1 617 529 4266 > _______________________________________________ > Wikimedia-l mailing list, guidelines at: > https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines > New messages to: Wikimedia-l(a)lists.wikimedia.org > Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, > <mailto:wikimedia-l-request@lists.wikimedia.org?subject=unsubscribe> >

1 0

Investing in documentation
by Kevin Smith 15 Mar '16

15 Mar '16

In today's departmental weekly checkin, I asked if it might be worth spending more effort on documentation (e.g. TextCat). That brief discussion generated the following notes (which are already on wiki[1]): - Should we have a semi-formal policy about the importance of documentation? - Concerns about over-documenting (e.g. not agile) - Move this conversation to email - Dan is willing to prioritize phab tasks to write docs on specific topics - Chris can alert us if he is answering the same question multiple times (feature-level) - Multiple audiences (onboarding new staff, end-users on Wikimedia sites, end users on third party sites, other devs) For now, it seems that our immediate next steps could be to create a phab task whenever you notice (or are informed of) a shortcoming with our documentation. That task can be prioritized by whoever manages the workflow of the person/people who would do the documentation. If we end up in situations where documentation work ends up as eternal low priority, of if using phab proves unproductive, we can adapt and adjust. Feel free to continue the discussion here, if there are additional questions, concerns, or suggestions. [1] https://www.mediawiki.org/wiki/Wikimedia_Discovery/Checkin_meeting_minutes/… Kevin Smith Agile Coach, Wikimedia Foundation

1 0

Re: [discovery] Wiki maps news and updates
by Chris Koerner 15 Mar '16

15 Mar '16

Yuri, Great news! I'm glad to see the progress and look forward to helping work with the Wikivoyage and other map-loving communities. Not to be a pain, but shouldn't the IRC channel be #wikimedia-interactive ? :) -- Yours, Chris Koerner Community Liaison - Discovery Wikimedia Foundation

1 0

wikipedia portal browser support
by Jan Drewniak 15 Mar '16

15 Mar '16

Hi All, The portal team has had some discussions about what browsers we support on the wikipedia.org portal over the past week. Currently, our browser support is all over the map (in regards to IE anyway, almost literally any other browser release in the past 10 years works fine). - The typeahead suggestions work on IE6+ (but very slowly) - The baseline event logging works on IE6+ - A/B test event logging works on IE8+ (requires localStorage) - The JS language-picker works on IE9+ (requires CSS capabilities) - The upcoming A/B test (localized top-links) work on IE8+ (hasn't been test on lower versions) I think we should standardize on browser support. Currently our browser-usage from our dashboard <http://discovery.wmflabs.org/portal/#browser_breakdown> shows IE7 at ~ 1%, IE8 at ~1.5% IE9 at ~ 1% and IE6 isn't even on the map. Mediawiki, and by extension wikipedia.org, *has* browser compatibility standards, published here: https://www.mediawiki.org/wiki/Compatibility I think since the wikipedia portal is a gateway to the rest of wikipedia, it makes sense to follow the same guidelines as the rest of wikipedia. This would mean dropping JS support for IE8 and lower. We would still provide 'basic' support to IE8 and below, meaning the non-js version of the portal would still work for these browsers. This change would slightly change our dashboard metrics, but it would certainly speed-up our development time, allowing us to do more with our limited resources. Any objections or concerns with this approach? -Jan

4 3

Sharing my todo list
by Guillaume Lederrey 15 Mar '16

15 Mar '16

Hello! I'm trying to put some order into what I'm doing / want to do. If you have a few minutes to have a look (especially Dan) and let me know if I'm missing something big or if you have some idea on the priorities of all that... T124444 - Look into encrypting Elasticsearch traffic (in progress, *should* be done for codfw switchover) T128786 - improve robustness of es-tool (limitation were seen during elasticsearch 1.7.5 upgrade) T128000 - Refresh elastic10{01..16}.eqiad.wmnet servers (waiting for the hardware) T128616 - Remove hostname from elasticsearch log messages (minor point but shoudl improve visibility on logs) T128433 - Estimate hardware requirements for relevance lab elasticsearch servers (I'm mainly following, Erik is the brain on this one) T129254 - Put elasticsearch upgrade scripts under version control (small task, but should be done) T129256 - Implement proper orchestration mechanics for elasticsearch upgrade (we should at least investigate and see if we want to invest in better orchestration or not) I have a few more things in my head that are not clear enough yet to be written down. Feedback welcomed! MrG

3 5

Wiki maps news and updates
by Yuri Astrakhan 14 Mar '16

14 Mar '16

The quest to bring maps to Wikipedia continues: * Kartographer has launched for WikiVoyage * Julien Girault will help maps with his UI expertise * Talk to us at the FreeNode IRC channel #wikipedia-interactive https://www.mediawiki.org/wiki/Help:Kartographer Last week we enabled Kartographer extension for Wikivoyage sites, allowing users to add maps to wiki pages without any additional wmflabs and JavaScript tricks. Now you can simply add a <mapframe> or <maplink> to a wiki page, or even use the Visual Editor to insert a map. Additionally, you can: * add markers and polygons visually * edit geojson and see how it changes the map on each keystroke * add auto-numbered markers (either numbers or letters), and have multiple counters * have multiple "groups" of markers/polygons and showing them on the same map or on separate maps (e.g. all food and all drink maps and one combined map) * markers can be of any color, 3 sizes, and contain many different icons * markers and polygons can be clicked and will show popups with wiki text and images * very fast full screen popup map Feedback: https://www.mediawiki.org/wiki/Talk:Maps Bugs & TODOs: https://phabricator.wikimedia.org/tag/kartographer/ All maps-related tasks: https://phabricator.wikimedia.org/tag/maps/ == What's next? == There will be plenty of cleanup and polishing work to make Kartographer work seamlessly. We will need to address the missing functionality reported to us by the community, and help migrate existing wmflabs-based maps to the new platform. Lastly, VE editing will need some more work to become indispensable. Yet, our site is still set on the bigger target - maps for all of Wikipedia. For that we are waiting for more hardware, plus we will need to improve our static maps service to be able to handle wiki-load. Hardware task: https://phabricator.wikimedia.org/T125126 Thank you Max Semenik, Ed Sanders, Alex Kosiaris, Brandon Black, Chris Koerner, Chris Steipp, Tomasz Finc, and Wes Moran for making this possible.

1 0

Julien Girault supporting Maps team
by Moiz Syed 14 Mar '16

14 Mar '16

Hi all, Just a quick announcement that Julien will now be supporting Max and Yuri on the Maps team. I'm looking forward to all the great things the Maps team will build now with the additional help of Julien! Jan and Julien will still be reviewing each others code for the next little while. Quick announcement over :) Moiz

2 1

User searches for information about health crises
by Pine W 14 Mar '16

14 Mar '16

Food for thought: http://www.seattletimes.com/business/smartphone-voices-not-always-helpful-i… I wonder if Wikipedia search should have some enhanced capability to help people who perform searches like these. There may be ethical and legal limits to what Wikipedia should provide in the way of "medical advice", but I think if our search tool gets queries like the ones mentioned in that article, Wikipedia Search might suggest to people that they contact their local emergency services rather than try to do self-help via Wikipedia. Pine

1 0

Help with our hiera hierarchy
by Guillaume Lederrey 14 Mar '16

14 Mar '16

Hey! I'm probably missing something obvious, but there is something I do not understand about our hiera hierarchy as part of change 274711 [1]. What I am trying to achieve: 1) activate HTTPS for elastic search in both eqiad and codfw 2) not activate it for logstash (it would not hurt, but it does not seem like we need it at the moment) 3) define base::puppet::dns_alt_names for elasticsearch servers to 'search.svc.eqiad.wmnet' and 'search.svc.codfw.wmnet' respectively How I'm trying to achieve this: 1) Adding the property elasticsearch::https::ensure: 'present' in hieradata/role/common/elasticsearch/server.yaml 2) Adding the property elasticsearch::https::ensure: 'present' in hieradata/role/common/logstash/server.yaml 3) Adding the property base::puppet::dns_alt_names: 'search.svc.[codfw|eqiad].wmnet' in hieradata/role/[codfw|eqiad]/elasticsearch/server.yaml respectively. What makes me think it is not working: The excellent puppet-compiler gives me a view on the change catalog [6], [7] that tells me that * elasticsearch::https is ensured 'present' for both elasticsearch and logstash nodes * dns_alt_names is not defined for any node Anyone has a pointer on what I'm missing? Thanks! MrG [1] https://gerrit.wikimedia.org/r/#/c/274711/ [2] https://gerrit.wikimedia.org/r/#/c/274711/10/hieradata/role/common/elastics… [3] https://gerrit.wikimedia.org/r/#/c/274711/10/hieradata/role/common/logstash… [4] https://gerrit.wikimedia.org/r/#/c/274711/10/hieradata/role/codfw/elasticse… [5] https://gerrit.wikimedia.org/r/#/c/274711/10/hieradata/role/eqiad/elasticse… [6] http://puppet-compiler.wmflabs.org/2048/elastic1001.eqiad.wmnet/change.elas… [7] http://puppet-compiler.wmflabs.org/2048/logstash1001.eqiad.wmnet/change.log… --- Guillaume Lederrey Operations Engineer, Discovery Wikimedia Foundation

1 1

Re: [discovery] [Wikitech-ambassadors] Wikipedia.org portal page update!
by David Abián 12 Mar '16

12 Mar '16

It looks great, well done. However, I leave some comments: * With my configuration, contents slightly overflow the page when using a window of between 481 and 560 px wide. I would increase the border between mobile and desktop view from 480 px to ~550 px (or would modify contents so that we didn't need to modify that limit). http://davidabian.com/screenshots/wikimedia/wikipedia-org-485.png * Having written a topic in the search box, the selected summary overflows the box after clicking. http://davidabian.com/screenshots/wikimedia/wikipedia-org-results-overflowi… * While a license (CC BY-SA 3.0) is specified in the HTML (<link rel="license" href="//creativecommons.org/licenses/by-sa/3.0/">), no license is displayed to the user. And some questions: * Are you planning to make something similar with wikimedia.org? * Do you follow a specific order when linking to the sister projects? I see that wikipedia.org and wikimedia.org show the projects in a different order. Thank you! El 11/03/16 a las 19:18, Deborah Tankersley escribió: > Hello! > > I'm very pleased to announce that we've updated the Wikipedia.org > <http://wikipedia.org/> portal page with a brand new search box that is > more prominent and will now display meta data with images (as available) > in the search results (link > <https://commons.wikimedia.org/wiki/File:New_searchbox_with_metadata.png>). > > This was a large effort by the Discovery Portal team to develop a > JavaScript-only version of the language picker, so that JavaScript > enabled browsers will see all the new meta data. Alongside that effort, > we also ensured that in JavaScript (JS) disabled browsers (or older > Internet Explorer versions), our visitors won't have a bad experience > when choosing a language to search in. (Note 1: in older IE versions and > JS disabled browsers, the type-ahead and meta data search results > information will not be displayed.) > > We also implemented a shorter language code (ie: EN for English, ES for > Spanish, etc) to allow for more characters to be typed into the search > box. When a user toggles the language selector, the full language name > will be displayed in the dropdown for easy finding of the language you > prefer to search in. For the more technical minded - I've also uploaded, > to Commons, a screenshot > <https://commons.wikimedia.org/wiki/File:Sample_browsers_tested.png> of > one of the ways we test our code, visually. > > We're interested in hearing your feedback or if you have any questions! > (Note 2: My apologies for not getting this email out yesterday, but I > had had issues with size limitations of my screenshots.) > > On behalf of the very happy Wikipedia.org Portal Team, > > Deb > -- > Deb Tankersley > Product Manager, Discovery > Wikimedia Foundation > > > _______________________________________________ > Wikitech-ambassadors mailing list > Wikitech-ambassadors(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/wikitech-ambassadors > -- David Abián - davidabian.com Vocal de Comunicación Wikimedia España Vega Sicilia, 2 47008 - Valladolid https://wikimedia.es Wikimedia España es una asociación sin ánimo de lucro española con CIF G-10413698 inscrita en el Registro Nacional de Asociaciones, Grupo 1, Sección 1, Núm. Nacional 597390. «Imagina un mundo en el que cada persona tenga acceso libre a todo el conocimiento»

2 1

Re: [discovery] [Wikimedia-l] Wikipedia.org portal page update!
by Dan Garry 11 Mar '16

11 Mar '16

On 11 March 2016 at 16:09, Samuel Klein <meta.sj(a)gmail.com> wrote: > > Perhaps you could do this w two queries, one to a composite index that is > only updated weekly. > Indeed, there are mechanisms that can make cross-wiki searching more feasible. In fact, one mechanism we are in the very early stages of exploring is merging all the projects in a given language into a single index, so that one could search *all* projects in a language, rather than just a single project in a given language. I have no timeline here; as I said, we're in the very early stages, and of course we have other work on the go at the same time. > > Additionally, it would likely return you a bunch of really irrelevant > results, > > Make this opt-in, add a different background color for results from the > all-language index, & divide their search-relevance by a > language-prominence factor... > I plan to worry more about the user experience implications that I mentioned once we're a bit closer to solving the technical feasibility questions. As you've shown, these are definitely solvable problems, but I don't want to put the cart before the horse, as it were. :-) Thanks, Dan -- Dan Garry Lead Product Manager, Discovery Wikimedia Foundation

1 0

Weekly summary #1
by Chris Koerner 11 Mar '16

11 Mar '16

Hello, This is a quick summary of some of the things happening around Discovery in the past week. Note: This is our first week trying something like this. Feedback and suggestions are welcome. ==Events and News== * Completion Suggester will be coming to a few of the smaller wikis starting on March 10th <https://meta.wikimedia.org/wiki/User:CKoerner_(WMF)/Work/Completion_Suggest…> ==Other Noteworthy Stuff== * Interesting article from former WMF staffer Nik Everett (now at Elastic) on Loading Wikipedia's Search Index for Testing <https://www.elastic.co/blog/loading-wikipedia>. * Wikipedia.org portal page update! <https://lists.wikimedia.org/pipermail/wikimedia-l/2016-March/083269.html> ==Did you know?== * Wikidata Query Service now has over 1 billion data elements (triples) <https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?panelId=7…> . ==Development== * Measuring scoring changes to search engine relevance based on user behavior * Upcoming AB test measure the effect of changing the boost on search phrase matching (brought to you by new measurement above) View the full update on-wiki at: https://www.mediawiki.org/wiki/Discovery/Status_updates/2016_03_11 Help contribute to next week's update at: https://www.mediawiki.org/wiki/Discovery/Status_updates/Next -- Yours, Chris Koerner Community Liaison - Discovery Wikimedia Foundation P.S. A big thanks to Lydia and the other Wikidata folks for leading the way. I unabashedly borrowed many templates from their weekly status updates <https://www.wikidata.org/wiki/Wikidata:Status_updates>.

1 0

Re: [discovery] [Wikimedia-l] Wikipedia.org portal page update!
by Dan Garry 11 Mar '16

11 Mar '16

On 11 March 2016 at 10:52, ViswaPrabha (വിശ്വപ്രഭ) <vp2007(a)gmail.com> wrote: > > Failed my dream :( > > Any string in any language in any wikipedia project. How far is my dream? > I share your dream! :-) Unfortunately, the dream is quite far away from reality. Querying every search index would put a big performance strain on the search servers. Additionally, it would likely return you a bunch of really irrelevant results, so there's a lot of user experience implications that would need to be figured out as well. Discovery is not actively working on this at present. Thanks, Dan -- Dan Garry Lead Product Manager, Discovery Wikimedia Foundation

1 0

Wikipedia.org portal page update!
by Deborah Tankersley 11 Mar '16

11 Mar '16

Hello! I'm very pleased to announce that we've updated the Wikipedia.org <http://wikipedia.org/> portal page with a brand new search box that is more prominent and will now display meta data with images (as available) in the search results (link <https://commons.wikimedia.org/wiki/File:New_searchbox_with_metadata.png>). This was a large effort by the Discovery Portal team to develop a JavaScript-only version of the language picker, so that JavaScript enabled browsers will see all the new meta data. Alongside that effort, we also ensured that in JavaScript (JS) disabled browsers (or older Internet Explorer versions), our visitors won't have a bad experience when choosing a language to search in. (Note 1: in older IE versions and JS disabled browsers, the type-ahead and meta data search results information will not be displayed.) We also implemented a shorter language code (ie: EN for English, ES for Spanish, etc) to allow for more characters to be typed into the search box. When a user toggles the language selector, the full language name will be displayed in the dropdown for easy finding of the language you prefer to search in. For the more technical minded - I've also uploaded, to Commons, a screenshot <https://commons.wikimedia.org/wiki/File:Sample_browsers_tested.png> of one of the ways we test our code, visually. We're interested in hearing your feedback or if you have any questions! (Note 2: My apologies for not getting this email out yesterday, but I had had issues with size limitations of my screenshots.) On behalf of the very happy Wikipedia.org Portal Team, Deb -- Deb Tankersley Product Manager, Discovery Wikimedia Foundation

1 0

New search box on the portal and prefixsearch api call
by David Causse 11 Mar '16

11 Mar '16

Hi, first of all: congrats for this new search box! I noticed that you use action=query with both generator=prefixsearch and list=prefixsearch. IIRC when we looked at this (it was when we looked at the way some API consumers generates their requests) we noticed that it generates 2 prefixsearch queries on the backend. I think this adds unnecessary latency. Unfortunately I don't know why you need to use list and generator at the same time, is this a limitation of the API? If we can't fix the API call could we work on a cache at the backend level to avoid 2 elasticsearch queries? Thanks!

3 2

Wikipedia.org portal page update!
by Deborah Tankersley 10 Mar '16

10 Mar '16

Hello! I'm very pleased to announce that we've updated the Wikipedia.org <http://wikipedia.org/> portal page with a brand new search box that is more prominent and will now display meta data with images (as available) in the search results (see attached image). This was a large effort by the Discovery Portal team to develop a JavaScript-only version of the language picker, so that JavaScript enabled browsers will see all the new meta data. Alongside that effort, we also ensured that in JavaScript (JS) disabled browsers (or older Internet Explorer versions), our visitors won't have a bad experience when choosing a language to search in. (Note: in older IE versions and JS disabled browsers, the type-ahead and meta data search results information will not be displayed.) We also implemented a shorter language code (ie: EN for English, ES for Spanish, etc) to allow for more characters to be typed into the search box. When a user toggles the language selector, the full language name will be displayed in the dropdown for easy finding of the language you prefer to search in. For the more technical minded - I've attached a screenshot of one of the ways we test our code, visually. We're interested in hearing your feedback or if you have any questions! On behalf of the very happy Wikipedia.org Portal Team, Deb -- Deb Tankersley Product Manager, Discovery Wikimedia Foundation

1 0

"Unbreak now!" task in sprint: cross-namespace redirects on Hungarian Wikipedia
by Dan Garry 10 Mar '16

10 Mar '16

Hello! As a result of a user report <https://www.mediawiki.org/wiki/Help:CirrusSearch/CompletionSuggester#Does_n…>, I've filed T129545 <https://phabricator.wikimedia.org/T129545> and set the priority to "Unbreak now!". I've chatted with Erik about this briefly, and he said he'd investigate. I'm writing this to the list because I think it's good practice when super high priority stuff is dropped in the sprint. :-) Thanks, Dan -- Dan Garry Lead Product Manager, Discovery Wikimedia Foundation

1 1

Wikipedia.org portal - New search box deployed to beta, soon in production!
by Julien Girault 09 Mar '16

09 Mar '16

Hi everybody, Following up on last week's email, the team has made significant improvements to the *enhanced search box*: - We developed a JavaScript-only version of the language picker, on top of the previous patch, so that users who have JS enabled (93% of our traffic) get a nicer version of the language picker. - Shorter language code is displayed, allowing more space for text within the search input. - Triggers the native selector on mobile devices. - The search box looks better overall. Users who disable JavaScript, or are using old IE versions, get the native selector already presented in last email. *It's merged into master, and deployed on beta!* *Check it out:* http://www.wikipedia.beta.wmflabs.org/ *Deployment to production * (Phab ticket for reference: Epic T125472 <https://phabricator.wikimedia.org/T125472>) The patch has been submitted for Thursday's Evening SWAT (Thu 00:00–01:00 UTC, Wed 16:00–17:00 PST) <https://wikitech.wikimedia.org/wiki/Deployments#Thursday.2C.C2.A0March.C2.A…> . Last chance to enjoy the "good old search" https://www.wikipedia.org/ ! Chris <https://www.mediawiki.org/wiki/User:CKoerner_(WMF)> and Deborah <https://www.mediawiki.org/wiki/User:DTankersley_(WMF)> will answer any question or concern related to this deployment *Some screenshots:* *Modern browsers:* Chrome, JS: http://i.imgur.com/Jb8JVbd.png Chrome, no-JS: http://i.imgur.com/JcBfz2h.png (basic language picker because no JS) *IE browsers:* IE8, JS: http://i.imgur.com/Nef2rn6.png (basic language picker because old IE) IE9, JS: http://i.imgur.com/o1yrhcA.png (basic language picker because old IE) IE10, JS: http://i.imgur.com/OY6MVwJ.png IE11, JS: http://i.imgur.com/nQUk8fg.png *Note: the CSS for the suggestions needs some attention, currently seeing weird padding.* *Mobile browsers:* iOS, JS: http://i.imgur.com/7mbCKu6.png iOS, no-JS: http://i.imgur.com/F2rQpEE.png (basic language picker because no JS) Note that JS is required for the typeahead/suggestions feature (this is already the case with the current page, no regression). If you don't have JS, you will have to submit the form in order to get results. *Next A/B test: Use language detection to re-arrange the primary links to suit the user better* *Coming soon!* The team has also made significant progress, stay tuned! For the Wikipedia.org portal team, Julien.

9 18

world's largest blind taste test?
by Deborah Tankersley 09 Mar '16

09 Mar '16

I saw this today and thought it very interesting - Kraft changed their mac n cheese recipe, last December, to be more healthful but didn't tell anyone until now. The full story <http://www.eater.com/2016/3/7/11173858/kraft-mac-and-cheese-different> and a few key excerpts: *Kraft announced that the revamped product would hit store shelves in January — but unbeknownst to customers, it quietly began selling the artificial-ingredient-free mac and cheese in December. (Only consumers who paid careful attention to the tiny print of the ingredients list on the back of the box might have seen the change.) Today, Kraft issued a self-laudatory press release revealing its little experiment — which it has deemed "the world's largest blind taste test" — and proclaiming that "fifty million boxes later ... people didn’t notice a difference."* *...* *"We constantly talk to our consumers and get feedback from them, and we knew they wanted to feel better about the ingredients they serve their families," Greg Guidotti, vice president of meals at Kraft Heinz, tells Eater. "We saw an opportunity in the marketplace to improve our ingredient line, but we didn't want make the change before we had the right recipe." While the company only announced its intentions to oust artificial ingredients from mac and cheese in April, Guidotti says they've in fact been working on the new recipe for more than three years.* *The idea to do a surreptitious switcheroo with the new mac and cheese was born out of the skeptical consumer response to Kraft's initial announcement that it was changing the recipe. "When we announced in April [that we were changing the recipe] there was excitement but also concern, so we saw it as an opportunity," Guidotti says. "We knew the recipe still tasted just as good as you expect from Kraft, but whenever you say you've changed something, consumers will say it probably won’t taste as good. Since we knew it tasted exactly the same, we wanted our fans to experience that for themselves without even being prompted."* Cheers, Deb -- Deb Tankersley Product Manager, Discovery Wikimedia Foundation

1 0

Data access guidelines released
by Oliver Keyes 08 Mar '16

08 Mar '16

Hey y'all, As my final hurrah, I've released the data access guidelines used by the Discovery team in research and analysis on to Meta. It can be found at https://meta.wikimedia.org/wiki/Discovery/Data_access_guidelines The intent here is to ensure that we are transparent about what information we have, what we do with it, and what expectations we have on how employees will guarantee the safety and security of the information and the people behind it. To my knowledge the Discovery team is the first team in Engineering to have released this kind of information so prominently. I am pretty happy we could lead the way, and look forward to other groups with research interests hopefully doing the same! Best, -- Oliver Keyes Count Logula Wikimedia Foundation

3 3

Proposed changes to Discovery workboard structure
by Dan Garry 08 Mar '16

08 Mar '16

Hello all! I’m writing to solicit feedback on a plan to slightly rework the Discovery workboards on Phabricator. Back when we were assembling the team and putting the process together, I expressed a strong desire to have a centralised Discovery workboard which would contain every task related to Discovery’s work. In the end, I’d say that hasn’t really worked out, because there’s simply too many tasks in the workboard and too much in flight. It’s been hard to sensibly break things down into different categories of task on a per-project basis. To help alleviate this problem, here’s my proposal: - The Discovery workboard is disabled. - The Discovery tag will continue to exist as a parent-tag for Discovery’s work, but there will be no associated workboard. - Individual backlog workboards can be created for individual projects at the request of that team, for example: - Discovery-Search-Backlog - Discovery-Analysis-Backlog - Sprint boards will remain exactly as they are presently. How will this affect you? Firstly, this won’t affect you at all unless you spend a significant amount of time in the Discovery backlog, which should mostly be! Otherwise, nothing will really change for you. The workflow of engineers working on Maps and Wikidata Query Service, for example, will likely be completely unaffected by this change. As I’m the primary consumer of the Discovery board, and it’s not working very well for me at the minute, I’d like to move forwards with this proposal unless there are any strong objections. Kevin and I will handle all the logistics. Thoughts? Thanks, Dan -- Dan Garry Lead Product Manager, Discovery Wikimedia Foundation

4 8

Re: [discovery] [WikimediaMobile] Results from similar articles A/B test
by Justin Ormont 07 Mar '16

07 Mar '16

[resending due to mailing list byte limits] A/B testing: Slightly off-topic - You may try separating latency effects in an A/B test by running a counter factual test. Run your feature, but don't display any changes. A/B/CF test. The experiment scorecard then would compare your feature & control to the counterfactual test highlighting indirect changes like added latency. --justin

1 0

Help Rename Relevanc* Lab
by Trey Jones 07 Mar '16

07 Mar '16

Greetings Discovery Friends, Foes, and Other Enthusiasts! For various reasons, we're looking to rename the Relevanc(e|y) Lab so that it doesn't have "lab" in its name. If you have any clever or amusing ideas, jot them down over in Phabricator: https://phabricator.wikimedia.org/T128765 If you aren't Phab-enabled, you can read the discussion so far to get calibrated, and reply here and I'll post them back to Phab. Thanks, —Trey

4 5

Fwd: [Wikitech-l] Using Wikipedia/Wikidata in a nonprofit search engine
by Pine W 06 Mar '16

06 Mar '16

Forwarding to the Discovery mailing list, although it sounds like the OP hopes to have any possible discussion on Wikitech-l. I wonder if there would be ways for WMF Discovery to leverage the work that's being done already on Commoncrawl and Commonsearch for use in Wikimedia internal search. Pine ---------- Forwarded message ---------- From: Sylvain Zimmer <sylvain(a)sylvainzimmer.com> Date: Sun, Mar 6, 2016 at 11:46 AM Subject: [Wikitech-l] Using Wikipedia/Wikidata in a nonprofit search engine To: wikitech-l(a)lists.wikimedia.org Hi, Some of you may be familiar with http://commoncrawl.org ; they are doing an excellent job of making large crawls of the web accessible to everyone. I've been working on an open search engine based on these crawls for a while, and I would love to have your feedbacks on the project: https://about.commonsearch.org/ Specifically, I would be curious to know what you would consider to be the best possible integration of Wikipedia & Wikidata in a general search engine? As a first step, we have just started using the "official website" property from Wikidata and we are considering importing the Wikipedia abstracts next (https://github.com/commonsearch/cosr-back/issues/11). I'm looking forward to your feedbacks... or contributions! :-) Thanks in advance, PS: A few wikimedians recommended me to post on wikitech-l to keep the focus on the technical aspects of the project and hopefully avoid linking this project in any way to the KE stuff, which it actually predates by far (https://news.ycombinator.com/item?id=6209088). -- Sylvain Zimmer http://sylvinus.org _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

1 0

Re: [discovery] [Ops] First two weeks as part of the team
by Chad Horohoe 04 Mar '16

04 Mar '16

You may also find these diagrams useful: https://commons.wikimedia.org/wiki/File:Wikipedia_webrequest_flow_2015-10.p… https://wikitech.wikimedia.org/wiki/File:Infrastructure_overview.png -Chad On Feb 24, 2016 6:58 PM, "Mukunda Modell" <mmodell(a)wikimedia.org> wrote: > >> On Feb 17, 2016 1:50 AM, "Guillaume Lederrey" <glederrey(a)wikimedia.org> > >> wrote: > >>> * I still have not found a global architecture schema (something like > >>> a high level component or deplyoment diagram). But I have never seen > >>> any company having those... > > I made a diagram of the scap (mediawiki) deployment architecture a while > back: https://commons.wikimedia.org/wiki/File:Scap-diagram.png .. > > That does not exactly apply to the new scap3 architecture but it's not too > far off. > > .... > > On Thu, Feb 18, 2016 at 10:37 AM, Giuseppe Lavagetto > <glavagetto(a)wikimedia.org> wrote: > > > > About cherry-picks in beta: the problem is not cherry-picking (I think > > it's a reasonable way to test things) but persistent cherry-picking to > > monkey patch problems is. I think if we follow the flow of: > > > > - writing a patch > > - testing it on beta with a cherry-pick > > - get it merged on ops/puppet and production > > There are a lot of patches on beta these days and there have been a lot of > different people cherry-picking without much coordination. This has lead > to breakage quite often. Patches also get lost regularly. I assume this > usually happens because someone has rebased the HEAD and accidentally > dropped a patch. > > It can be really difficult to get a patch merged in ops/puppet within a > week (or even a month). I've seen a lot of patches sit around for weeks and > even now with the Puppet SWAT windows, it's still sometimes unrealistic to > expect patches get merged into production that quickly. (+CC Tyler) > > Without a system to manage things, and with very little coordination > between everyone who is working on beta, I don't expect the situation to > improve too much. > > I intend to propose a solution for beta & puppet patch cherry-picks very > soon, however, I haven't fully formulated my proposal yet. I will write to > the ops list when I have something written in a clear and presentable way. > > _______________________________________________ > Ops mailing list > Ops(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/ops > >

3 2

Wikivoyage mapframe
by Yuri Astrakhan 03 Mar '16

03 Mar '16

Yesterday, I posted a note on all Wikivoyages' traveler pubs, except EN and RU, describing our maps efforts and asking to switch their mapframe template to the Wikimedia maps service. Two languages are having a discussion, some have switched, some have not replied. *== Discussion ==* de Deutsch https://de.wikivoyage.org/wiki/Wikivoyage:Lounge#Maps_for_Wikivoyage nl Nederlands https://nl.wikivoyage.org/wiki/Wikivoyage:Reizigerscaf%C3%A9#Maps_for_Wikiv… *== Waiting ==* fa فارسی he עברית it Italiano pl Polski pt Português sv Svenska uk Українська vi Tiếng Việt *== Done ==* el Ελληνικά en English es Español fr Français ro Română (no embedded maps, fixed myself) ru Русский zh 中文

3 4

DevopsDay Amsterdam - June 29 - July 1
by Guillaume Lederrey 03 Mar '16

03 Mar '16

Hello teams! I'm thinking in going to DevopsDays Amsterdam (June 29th, 30th and July 1st 2016) [1]. It might be a good occasion to meet some of you in real life if you are so inclined. For those who do not know what DevopsDays are, it is a fun conference format, with a fairly small audience (200-400 people) centered around Devops themes (no surprise). The really fun part of this conference is the Open Spaces [2]. It is a good occasion to have great conversations with smart people, exchange idea, spread our knowledge, ... Let me know... MrG [1] http://www.devopsdays.org/events/2016-amsterdam/ [2] http://www.devopsdays.org/pages/open-space-format/

1 0

Re: [discovery] [Ops] Orchestration of deployments
by Guillaume Lederrey 03 Mar '16

03 Mar '16

I have been wanting to get a deeper look at Salt for a long long time... Seems that the time has come! On Tue, Mar 1, 2016 at 10:48 PM, Andrew Otto <otto(a)wikimedia.org> wrote: > Could also put it in puppet in elasticsearch module or role. > > We often use salt for things like this. It works (sometimes?)! > > > > On Tue, Mar 1, 2016 at 2:47 PM, Daniel Zahn <dzahn(a)wikimedia.org> wrote: >> >> >> On Tue, Mar 1, 2016 at 10:02 AM, Guillaume Lederrey >> <glederrey(a)wikimedia.org> wrote: >>> >>> Do we have a central place for those kind of scripts? I'd like to >>> version it in a more obvious place than my personal Github repo. >> >> >> +1 for not using personal Github. If in doubt you can always use this: >> >> operations/software Random software tools for ops tasks (svn2git, >> udpprofile, etc) >> >> and then there are a lot of repos under operations/software/foo >> >> >> -- >> Daniel Zahn <dzahn(a)wikimedia.org> >> Operations Engineer >> >> _______________________________________________ >> Ops mailing list >> Ops(a)lists.wikimedia.org >> https://lists.wikimedia.org/mailman/listinfo/ops >> >

1 0

Tech talk on the topic of Search at Airbnb
by Moiz Syed 02 Mar '16

02 Mar '16

Hey all, Just found out that there is a tech talk happening at Airbnb next Wednesday on the topic of Search. We can all go check it out. We can also say hi to Juliusz while we're there. https://www.airbnb.com/meetups/b8zxexa2q-search-airbnb

2 1

Fwd: [opensource-107] Seattle to host the 2016 OpenStreetMap State of the Map US Conference
by Pine W 02 Mar '16

02 Mar '16

For Wikimedia folks who are interested in possible collaborations with OSM, now seems like a good time to start thinking about possible presentations. Staff from the Wikimedia Foundation, and/or Wikimedia volunteers from around the US outside of the Seattle area, may want to start thinking about travel plans. For Wikimedia volunteers outside of Cascadia Wikimedians territory, you might consider applying for WMF Travel and Participation Support grants [1]. If you're inside of Cascadia Wikimedians territory and would like to attend the conference, we may have funds in our budget that can support your attendance. Contact me off-list for details. Regards, Pine [1] https://meta.wikimedia.org/wiki/Grants:TPS ---------- Forwarded message ---------- From: Clifford Snow <clifford(a)snowandsnow.us> Date: Tue, Mar 1, 2016 at 5:18 PM Subject: [opensource-107] Seattle to host the 2016 OpenStreetMap State of the Map US Conference To: opensource-107-announce(a)meetup.com I am excited to announce that Seattle was chosen to host the OpenStreetMap 2016 State of the Map US Conference. The conference will take place July 23-25 on SeattleU's campus. We chose SeattleU for their low cost, proximity to Seattle and access to public transit. The food trucks near by didn't hurt either. We are looking for help! Let us know if you want to help. Request for presentation proposals should be coming fairly soon. Start thinking about what you want to present or teach. The formal announcement can be found at: https://openstreetmap.us/2016/02/sotmus-2016/ Clifford

2 1

Wikipedia.org portal - update on work in progress
by Julien Girault 02 Mar '16

02 Mar '16

Hi everybody, Here's an update of what the team has been up to: *Stats are updated!* Yesterday we: - updated www.wikipedia.org stats, - updated the sister portals with the latest version from Meta. - www.wikibooks.org - www.wikinews.org - www.wikiquote.org - www.wikiversity.org - www.wiktionary.org In order to improve the process Deborah created a recurring task for it T128546 <https://phabricator.wikimedia.org/T128546>. It's pretty cool to see what's changed :) - www.wikipedia.org (https://github.com/wikimedia/wikimedia-portals/commit/864d5ebed066aa3267a17…) <https://github.com/wikimedia/wikimedia-portals/commit/864d5ebed066aa3267a17…> - sister portals (https://github.com/wikimedia/wikimedia-portals/commit/2e57bfbd83acce979a25b…) <https://github.com/wikimedia/wikimedia-portals/commit/2e57bfbd83acce979a25b…> *Deploying the enhanced search box to production* Phab: https://phabricator.wikimedia.org/T125571 We had a list of improvements to make before pushing to production. It took us longer than expected, specially because the language picker implementation was not 100% production-ready: - IE8(and lower) users were ignored for the A/B test. - We initially took the decision to not support IE8 for the A/B test in order to save development time (in case the test results show no improvements). - Mobile user experience was not optimal because of the custom dropdown. - We took the decision to figure this out only once we got the test results, to save some development time (in case the test results show no improvements). - The language picker relied on JavaScript and no solution was provided for non-JS users. We will learn from our first A/B test. The test showed a significant improvement though, and we are now getting ready for a deployment to production. Here's the latest update: - We changed the custom dropdown, triggered by JavaScript, and replaced it with a native <select> element. - Styling this native <select> element as well as we can (to match what we had in the A/B test), - but it may look a little odd in old browsers (there isn't so much we can do with <select> elements). - Solves mobile user experience issues because the device's native selector will be triggered and will be a lot easier than a custom dropdown. - Solves non-JS traffic (~ 7% <https://upload.wikimedia.org/wikipedia/commons/e/e6/Analysis_of_Wikipedia_P…>) because it's a native <select> element (no JS required) - minor detail: only the custom arrow is not clickable because we need a little JS hack. *Status: *The patch made it to code review today. We will release when it's approved. As of today, here is what you can expect: We have an idea on how to make it even better for old and weird browsers, but we want to move forward with this now :) And the new typeahead is fantastic!!! *Next A/B test: Use language detection to re-arrange the primary links to suit the user better* (Primary links = the 10 wiki links in the screenshot above). Phab: https://phabricator.wikimedia.org/T125472 For the test we read the user's preferred languages (from the browser) and show the corresponding wikis in the most top positions. Let's see if people click on these links more than usual ! Please come to Deborah and us if you have any question :) *Status:* The patch made it to code review today. We will merge it into its feature branch as soon as it's approved. Then we will review the A/B test setup one more time and schedule a launch date. We hope to get the new search box in production before we launch the A/B test. *Improving performance* Performance matters... it's a huge part of the user experience. We only talk about it when it's bad, we often forget it when it's good. Performance improvements can definitely increase user engagement. Take a look at what we've done since November: https://grafana.wikimedia.org/dashboard/db/webpagetest-portals :) For the Wikipedia.org portal team, Julien.

1 0

(no subject)
by Trey Jones 01 Mar '16

01 Mar '16

Greetings language nerds, I've completed the creation of a 21-language balanced (i.e., 200 each) corpus of relatively clean queries for use in evaluating language identification model testing. The 21 languages were chosen based on query volume across wikis in those languages. I've also evaluated our current version of TextCat against this corpus, using the known 21 languages, and all 59 languages I have models for. The 21 languages have pretty good models, because they had lots of query volume to be built on. The full set of 59 is a bit more dodgy, esp. Igbo, which is known to have a lot of English in the training data. Indonesian is the most unexpectedly poor performing of the bunch (most other poor performance is across language or script families and so is expected). The best model size among those test (500 to 10K), was the full 10,000! However performance at the 3,000 ngram model size (what we've been using for A/B tests) was only a few percentage points worse. Full write up with lots more details here: https://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/Balanced_Language_Id… I'll commit models for the rest of these 21 languages after verifying that they won't mess up our A/B tests. Cheers, —Trey Trey Jones Software Engineer, Discovery Wikimedia Foundation

2 2

Wikipedia.org Portal and Wiki article stats updated
by Deborah Tankersley 01 Mar '16

01 Mar '16

Hi there, Just a heads-up that we updated the Wikipedia.org portal page and 5 sister wiki sites statistics this afternoon. The sister wiki sites that were updated from Meta are: https://www.wikibooks.org, https://www.wikinews.org, https://www.wikiquote.org, https://www.wikiversity.org, https://www.wiktionary.org We have a recurring ticket <https://phabricator.wikimedia.org/T128546> to update the stats on a regular basis, optimally every two weeks, as we can. Let us know if there are any questions or concerns! Cheers, Deb -- Deb Tankersley Product Manager, Discovery Wikimedia Foundation

1 0

Updates to Discovery page on mediawiki.org
by Dan Garry 01 Mar '16

01 Mar '16

I have updated the Discovery page <https://www.mediawiki.org/wiki/Wikimedia_Discovery#The_team> on mediawiki.org to convey who is working on what. If you're curious, take a look. Please note the disclaimer: this is only intended to roughly convey who is working on what, and there are no guarantees that this is accurate to any particular level of detail. Thanks! Dan -- Dan Garry Lead Product Manager, Discovery Wikimedia Foundation

1 0

Push completion suggester rollout a week earlier?
by Erik Bernhardson 01 Mar '16

01 Mar '16

I seem to have forgotten when last discussing this, but the week of March 22 when we plan to roll the feature to prod is the same week tech ops is planning to test shifting all traffic to our failover data center in dallas. They have requested bi deployments that week to limit the variance for bugs that will invariably crop up. I think discovery is mostly ready, but we wll need to move comms a bit faster? We could also push back to the following week. I will be out on mid week to Israel but I trust dcausse and gehel can handle the stayed rollout regardless of my availability.

3 3

Orchestration of deployments
by Guillaume Lederrey 01 Mar '16

01 Mar '16

I am finishing the upgrade of elasticsearch to 1.7.5 for codfw (eqiad still to do). For this, I used a small script [1], heavily inspired (copied / stolen / ...) from bd808. The script is ugly, but it does the job. It runs the deployment over a list of hosts, pausing for manual steps / validations along the way. The script runs locally on my workstation, so it is subject to loss of connectivity, local crashes, hard to handover, ... Do we have a central place for those kind of scripts? I'd like to version it in a more obvious place than my personal Github repo. Do we have examples of similar scripts? A specific tool for this? Rundeck [2] comes to mind. Note that I'm not a huge fan of Rundeck as it brings far too much complexity for simple tasks, but the concept of having a central place of re usable operational components is appealing. [1] https://github.com/gehel/elasticsearch-utility-scripts [2] http://rundeck.org/

3 4

Varnish / caching presentation
by Guillaume Lederrey 01 Mar '16

01 Mar '16

We took some time with Brandon last Friday to have a presentation on the Varnish / caching infrastructure. Brandon did not have a lot of time to prepare this, so it ended up being more of a conversation, mainly driven by our questions. Brandon did have some slides, but they were just a very light support to our conversation. Honestly, that's my preferred format. (Thanks you Brandon for being busy on more important stuff and still take time to answer our questions!) If keeping this fairly unstructured format helps to have more regular Ops Sessions, I'm all for it! I also had to friends / former coworkers who were with me for this session. They both work on the caching infrastructure of Nespresso, and they found it really interesting. If we could open some of our Ops Session I think there could be quite a few people interested in watching them. And in my understanding, this would be quite aligned with our mission of disseminating the world's knowledge (and we do have a sizeable body of technical knowledge to disseminate). I can see a few constraints in open in those Ops Sessions to a wider audience. We still need some private time to discuss some of the things that are sensitive. And we need to find a way for this to not had a significant overhead to our busy schedules. Still I think it would be great if we could do that... What do you think? And again, big thanks to Brandon for his time! MrG

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

Discovery March 2016