Discovery May 2016

discovery@lists.wikimedia.org

17 participants
33 discussions

Guillaume out of office Thursday May 26 evening
by Guillaume Lederrey 21 May '16

21 May '16

Hello! Capital Consulting (aka SafeGuard Switzerland) is organizing a few drinks in Geneva next Thursday (May 26). So I'll be in Geneva for the evening. I'll make sure I attend our standup, but will take a couple of hours off in the CEST evening. Exact time to be confirmed... Take care, MrG -- Guillaume Lederrey Operations Engineer, Discovery Wikimedia Foundation

2 1

update: wikipedia.org portal
by Deborah Tankersley 20 May '16

20 May '16

Hello, The Discovery Team recently added descriptive text to the Wikipedia.org page footer in order to give visitors a better idea of what the sister wiki projects are really all about. Check it out at www.wikipedia.org or view a mobile screen capture here <https://www.mediawiki.org/wiki/File:Descriptive-text_sister_projects-wikipe…> . We also wrapped up a quick, one question survey that ran for a week on the portal to determine how visitors arrive at the page. The results showed that many of the visitors arrive by clicking on a bookmarked link or by typing in 'wikipedia' in their browser. We had many encouraging and uplifting comments as well - see the full results here <https://commons.wikimedia.org/wiki/File:Wikipedia_Portal_Survey_-_May_2016.…> . In the coming weeks and months we'll be reaching out to many of those survey takers who graciously provided their name and email addresses to engage in deeper conversations on how they use the portal and Wikipedia in general. As always, more detailed information is available on wiki for the Wikipedia Portal <https://www.mediawiki.org/wiki/Wikipedia.org_Portal> and A/B testing <https://www.mediawiki.org/wiki/Wikipedia.org_Portal_A/B_testing>. On behalf of our happy little Wikipedia.org Portal team, Deb -- Deb Tankersley Product Manager, Discovery Wikimedia Foundation

1 0

Re: [discovery] [Analytics] University project to make entire English Wikipedia history searchable on Hadoop using Solr
by Tilman Bayer 18 May '16

18 May '16

Yes, of course *processing* the entire history (even with text) has been done before - but perhaps not storing or indexing it. BTW is anyone still using "Wikihadoop"? https://blog.wikimedia.org/2011/11/21/do-it-yourself-analytics-with-wikiped… https://github.com/whym/wikihadoop On Wed, May 18, 2016 at 3:09 AM, Dan Andreescu <dandreescu(a)wikimedia.org> wrote: > Hi Tilman, thanks for pointing to this research. We have indeed worked on > this kind of project, for both ORES and the WikiCredit system. There are > many challenges like memory and processing time. Loading the entire history > without text is what we're working on right now for our Wikistats 2.0 > project. Even this has many challenges. > > As far as I can tell right now, any simple attempt to handle all the data > in one way or one place is going to run into some sort of limit. If anybody > finds otherwise, it would be useful to our work. > > *From: *Tilman Bayer > *Sent: *Tuesday, May 17, 2016 02:54 > *To: *A mailing list for the Analytics Team at WMF and everybody who has > an interest in Wikipedia and analytics. > *Reply To: *A mailing list for the Analytics Team at WMF and everybody > who has an interest in Wikipedia and analytics. > *Cc: *A public mailing list about Wikimedia Search and Discovery projects > *Subject: *[Analytics] University project to make entire English > Wikipedia history searchable on Hadoop using Solr > > Detailed technical report on an undergraduate student project at Virginia > Tech (work in progress) to import the entire English Wikipedia history dump > into the university's Hadoop cluster and index it using Apache Solr, to > "allow researchers and developers at Virginia Tech to benchmark > configurations and big data analytics software": > > Steven Stulga, "English Wikipedia on Hadoop Cluster" > https://vtechworks.lib.vt.edu/handle/10919/70932 (CC BY 3.0) > > IIRC this has rarely or never been attempted due to the large size of the > dataset - 10TB uncompressed. And it looks like the author here encountered > an out of memory error that he wasn't able to solve before the end of > term... > > -- > Tilman Bayer > Senior Analyst > Wikimedia Foundation > IRC (Freenode): HaeB > > > -- > Sent from Gmail Mobile > > > _______________________________________________ > Analytics mailing list > Analytics(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/analytics > > -- Tilman Bayer Senior Analyst Wikimedia Foundation IRC (Freenode): HaeB

1 0

University project to make entire English Wikipedia history searchable on Hadoop using Solr
by Tilman Bayer 17 May '16

17 May '16

Detailed technical report on an undergraduate student project at Virginia Tech (work in progress) to import the entire English Wikipedia history dump into the university's Hadoop cluster and index it using Apache Solr, to "allow researchers and developers at Virginia Tech to benchmark configurations and big data analytics software": Steven Stulga, "English Wikipedia on Hadoop Cluster" https://vtechworks.lib.vt.edu/handle/10919/70932 (CC BY 3.0) IIRC this has rarely or never been attempted due to the large size of the dataset - 10TB uncompressed. And it looks like the author here encountered an out of memory error that he wasn't able to solve before the end of term... -- Tilman Bayer Senior Analyst Wikimedia Foundation IRC (Freenode): HaeB -- Sent from Gmail Mobile

1 0

Discovery Weekly Update for the week starting 2016-05-09
by Chris Koerner 13 May '16

13 May '16

Hello, Here is this week's update from the Discovery department. * Closed out the latest A/B test for adding descriptive text [1] on the Portal on May 10 * Started a short banner survey [2] on the wikipedia.org portal page on May 11 * Stalled on updating wikipedia.org portal [3] stats but hope to be resolved soon (with DBA assistance) * Created a testing dev workflow page [4] for wikipedia.org portal site A/B testing * Released Discernatron! [5]. You can help make search better by grading search results. * Presented geo boosted full text search [6], geo queries in WDQS [7], and updates to mapping for wikivoyage [8] in the monthly CREDIT showcase [9]. [1] https://phabricator.wikimedia.org/T131238 [2] https://phabricator.wikimedia.org/T134512 [3] https://phabricator.wikimedia.org/T128546 [4] https://www.mediawiki.org/wiki/Wikipedia.org_Portal_A/B_testing/dev_process [5] https://discernatron.wmflabs.org/ [6] https://www.youtube.com/watch?v=GwTDDWjxoek&feature=youtu.be&t=18m45s [7] https://www.youtube.com/watch?v=GwTDDWjxoek&feature=youtu.be&t=21m16s [8] https://www.youtube.com/watch?v=GwTDDWjxoek&feature=youtu.be&t=27m31s [9] https://www.mediawiki.org/wiki/File:CREDIT_-_May_2016.webm ---- Feedback and suggestions on this weekly update are welcome! The full update, and archive of past updates, can be found on Mediawiki.org: https://www.mediawiki.org/wiki/Discovery/Status_updates -- Yours, Chris Koerner Community Liaison - Discovery Wikimedia Foundation

1 0

Fwd: Request for code review and merge for wikimedia/search-repository-swift
by Chad Horohoe 12 May '16

12 May '16

Anyone in Discovery have the cycles to look at the Swift plugin I wrote? -Chad ---------- Forwarded message ---------- From: "Avik Das" <avdas(a)syr.edu> Date: May 11, 2016 11:58 PM Subject: Request for code review and merge for wikimedia/search-repository-swift To: "chadh(a)wikimedia.org" <chadh(a)wikimedia.org> Cc: Hi, I was working on elasticsearch plugins and was looking for snapshot plugins. Recently, I came across your repository and found it very useful and generic. However I needed plugin for elasticsearch version 2.1.1. I forked from the main branch and created some changes in my local repository. I am writing this email to request for code review and for merging into main branch for elasticsearch 2.1.1 . I tested my plugin on my local machine and swift repository from Soft Layer Object storage. Please let me know if you need any more information or action from my side. Please find the link for my repository below https://github.com/avik1988/search-repository-swift.git <https://github.com/avik1988/search-repository-swift.git> avik1988/search-repository-swift <https://github.com/avik1988/search-repository-swift.git> github.com search-repository-swift - Github mirror of "search/repository-swift" - our actual code is hosted with Gerrit (please see https://www.mediawiki.org/wiki/Developer_access for contributing Thanks, Avik Das

3 3

Fwd: [Wikitech-l] 12-May-2016 CREDIT, going back to Hangouts on Air/YouTube
by Pine W 12 May '16

12 May '16

Forwarding. ---------- Forwarded message ---------- From: Pine W <wiki.pine(a)gmail.com> Date: Thu, May 12, 2016 at 10:21 AM Subject: Re: [Wikitech-l] 12-May-2016 CREDIT, going back to Hangouts on Air/YouTube To: Wikimedia developers <wikitech-l(a)lists.wikimedia.org>, Wikimedia Foundation Multimedia Team <Multimedia(a)lists.wikimedia.org>, mobile-l < mobile-l(a)lists.wikimedia.org> FYI, this week's presentations, according to the Etherpad, are: * *Derk-Jan Hartman:* Video.js progress * *Dmitry Brant*: Wikidata infoboxes in Android app * *Joaquin Hernandez*: Vicky chat bot * *Baha*: mobile printing for offline reading * *Monte*: "smart random" content service endpoint * *Erik*: Geo boosting search queries Cheers, Pine On Thu, May 12, 2016 at 9:17 AM, Adam Baso <abaso(a)wikimedia.org> wrote: > Reminder... > > On Thu, Apr 14, 2016 at 12:13 AM, Adam Baso <abaso(a)wikimedia.org> wrote: > > > Hi all, > > > > The next CREDIT showcase will be Thursday, 12-May-2016 at 1800 UTC (1100 > > SF). > > > > https://www.mediawiki.org/wiki/CREDIT_showcase > > > > For this one we'll use Hangouts on Air for presenters, and the customary > > YouTube stream for viewers. > > > > See you next month! > > -Adam > > > > > > > _______________________________________________ > Wikitech-l mailing list > Wikitech-l(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/wikitech-l >

1 0

Re: [discovery] [Wikidata] WDQS instabilities - Incident report
by Guillaume Lederrey 11 May '16

11 May '16

Hey! No, the issue is unrelated. We are having a few issues with Varnish upgrades. To be honest, I don't entirely understand what the issue is, so I will not even try to explain it. I expect we will have an incident report coming soon. Regards, Guillaume On Wed, May 11, 2016 at 9:30 PM, Jan Macura <macurajan(a)gmail.com> wrote: > Hi Gullaume, all > > Not sure if it's connected somehow, but WDQS GUI is not working properly > this evening. For some time, the https://query.wikidata.org/ was rendering > blank page with no error, now it crashes with "TypeError: wb.ui.visualEditor > is undefined". > Using SPARQL endpoint distantly works fine. > > Best > Jan > > 2016-05-10 14:16 GMT+02:00 Guillaume Lederrey <glederrey(a)wikimedia.org>: >> >> Hello! >> >> I took some notes of the recent Wikidata Query Service instabilities >> and created an incident report [1]. Some people might have additional >> insight that I don't have. If that's the case, let me know and I'll >> update that incident report. >> >> Thanks all for your help and your understanding! >> >> Guillaume >> >> >> [1] >> https://wikitech.wikimedia.org/wiki/Incident_documentation/20160503-Wikidat… >> >> >> -- >> Guillaume Lederrey >> Operations Engineer, Discovery >> Wikimedia Foundation >> >> _______________________________________________ >> Wikidata mailing list >> Wikidata(a)lists.wikimedia.org >> https://lists.wikimedia.org/mailman/listinfo/wikidata > > > > _______________________________________________ > Wikidata mailing list > Wikidata(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/wikidata > -- Guillaume Lederrey Operations Engineer, Discovery Wikimedia Foundation

1 0

Elasticsearch slowdown - incident report
by Guillaume Lederrey 10 May '16

10 May '16

Hello! Yesterday around 21:43 UTC we experienced a slowdown of Elasticsearch. In short, one of our elasticsearch server went crazy with its garbage collector and had to be restarted. You can find more details in the incident report [1] or the corresponding Phabricator task [2]. Thank you all for your understanding! Guillaume [1] https://wikitech.wikimedia.org/wiki/Incident_documentation/20160509-CirrusS… [2] https://phabricator.wikimedia.org/T134829 -- Guillaume Lederrey Operations Engineer, Discovery Wikimedia Foundation

1 0

WDQS instabilities - Incident report
by Guillaume Lederrey 10 May '16

10 May '16

Hello! I took some notes of the recent Wikidata Query Service instabilities and created an incident report [1]. Some people might have additional insight that I don't have. If that's the case, let me know and I'll update that incident report. Thanks all for your help and your understanding! Guillaume [1] https://wikitech.wikimedia.org/wiki/Incident_documentation/20160503-Wikidat… -- Guillaume Lederrey Operations Engineer, Discovery Wikimedia Foundation

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

Discovery May 2016