Wikimedia-search

wikimedia-search@lists.wikimedia.org

1 participants
108 discussions

finding related pages [was Re: Reading tech sessions at hackathon]
by S Page 02 Jun '15

02 Jun '15

Summary: * CirrusSearch has "morelike:*PageName*", who knew? * I sense a developer article brewing, "Finding related content" AIUI, the Wikipedia mobile apps' "Read more" section just performs a full-text search (API [1] ) for the current page title (Android source [2]). Joaquin's nfity demo http://chimeces.com/webkipedia/ 's "Related pages" section calls the GettingStarted extension's gettingstartedgetpages API module [3] with gsgptaskname=morelike . This is implemented by GettingStarted/MoreLikePageSuggester.php... and it seems this just makes a search query for srsearch=morelike:Australia . Who knew Cirrus search had a "morelike:" keyword? It's not in the enwiki search help, but it is in the Cirrus search help [4]. I'm not sure if there's any reason to interpose gettingstartedgetpages instead of querying search directly for morelike:*pagetitle*, it might cache stuff in Redis. The mobile apps might get better "Read more" suggestions using one of these. There's also a srwhat=suggestion, I don't know if that helps getting related pages. I'll be updating https://www.mediawiki.org/wiki/API:Search_and_discovery with this, and it seems article-worthy. Cheers, hope this helps someone. [1] https://en.wikipedia.org/w/api.php?action=help&modules=query%2Bsearch [2] https://github.com/wikimedia/apps-android-wikipedia/blob/de0b8b579f5030f684… [3] https://en.wikipedia.org/w/api.php?action=help&modules=query%2Bgettingstart… [4] https://www.mediawiki.org/wiki/Help:CirrusSearch#Special_prefixes On Fri, May 29, 2015 at 2:18 AM, Joaquin Oltra Hernandez < jhernandez(a)wikimedia.org> wrote: > Sorry forgot to link it: https://github.com/joakin/webkipedia > > Matt Flaschen told me about the gettingstarted 'morelike' mode for other > purposes, but it fitted perfectly my purposes for this reading app. They > developed it on the Growth team about a year ago, but the experiment wasn't > successful so the API has been there dormant and unused for a lot of time > (works pretty well!). > > About the content I'm fetching for the articles, I'm using the extracts > with the exintro option, and embedding the html ( > https://github.com/joakin/webkipedia/blob/master/lib/api/article.js). The > idea would be to have a 'Read more' that would show the full article I > guess. > > The president's chest is pretty good too :D > http://chimeces.com/webkipedia/#/wiki/Barack_Obama > > On Fri, May 29, 2015 at 4:35 AM, S Page <spage(a)wikimedia.org> wrote: > >> (Cc'ing James Douglas, who's also developing API playground code.) >> >> On Thu, May 28, 2015 at 2:52 AM, Joaquin Oltra Hernandez < >> jhernandez(a)wikimedia.org> wrote: >> >>> S, for getting started quickly, I set up a JS web app completely >>> standalone with some basic infrastructure (libraries for calling the api, >>> rendering pipeline of JS views, url routing) so that interested people >>> could just get quickly to render a view within the app and do interesting >>> stuff querying the API. We were also open to just doing a plain html file >>> with some JS and CSS, or a codepen/jsbin style would have worked too. >>> >>> Here's the demo of the lite wikipedia webapp I worked on: >>> http://chimeces.com/webkipedia/ >>> >> >> That's lovely! It's what API developers develop when they develop. >> * Where's the source? >> * I had no idea the gettingstartedgetpages would give you related pages, >> so obscure! >> * I guess RESTBase has no mode that strips the citations and such, or >> gives you just the opening section (prop=extracts & exintro=) >> * Nice closeup of the great man's chest :-), >> http://chimeces.com/webkipedia/#/wiki/Albert_Einstein >> >> -- >> =S Page WMF Tech writer >> > > -- =S Page WMF Tech writer

8 10

Can we drop support for running Cirrus without plugins
by Nikolas Everett 02 Jun '15

02 Jun '15

In production we run CirrusSearch against Elasticsearch with a bunch of plugins installed - wikimedia-extra and the experimental highlighter. But Cirrus is capable of running without them. It'd be simpler for us if we simply mandated that they be installed. I created https://phabricator.wikimedia.org/T101029 to talk about it by my current feeling is that it probably wouldn't be a huge hardship to make those plugins required. Opinions? Nik

2 5

Event logging
by Kevin Smith 01 Jun '15

01 Jun '15

At two recent meetings, several topics around event logging came up. Specifically, how/where our different systems log (or should log) their events. The topics I'm aware of are: 1. Should WDQS use our standard event logging system? I think the answer was yes. Do we have or need a phab ticket to represent that? 2. Should Our new maps work use our standard event logging system? 3. Should Cirrus/Elastic use our standard event logging system? 4. Should Cirrus/Elastic log to Kibana? What are the privacy implications there? It's probably best to reply to only one topic at a time, so this thread gets forked. Ideally, please change the subject line to reflect the specific topic you are replying to. Discuss! Kevin Smith Agile Coach Wikimedia Foundation *Imagine a world in which every single human being can freely share in the sum of all knowledge. That's our commitment. Help us make it a reality.*

6 5

Event logging caution
by Kevin Smith 01 Jun '15

01 Jun '15

As was mentioned in scrum-of-scrums on May 20, the analytics team is running some Visual Editor A/B tests from now through June 11. As a result, analytics has requested that everyone "reach out to us if you're planning any changes in your Event Logging use" during that time. Hopefully this won't mess up what we're trying to do for the next couple weeks. Kevin Smith Agile Coach Wikimedia Foundation *Imagine a world in which every single human being can freely share in the sum of all knowledge. That's our commitment. Help us make it a reality.*

3 2

How do people use the search API?
by Oliver Keyes 21 May '15

21 May '15

Answer: we don't know yet, because there are Infinity ways to search. But for action=opensearch, http://ironholds.org/misc/api_presentation.html might make an interesting read. We're trying to document all the various forms of API search so that we can do more ad-hoc and regular analysis of this stuff. There's an awesome start at https://www.mediawiki.org/wiki/API:Search_and_discovery that looks pretty complete - if anyone is aware of API methods /not/ listed there, please put them in, or reply here and I will :) -- Oliver Keyes Research Analyst Wikimedia Foundation

3 5

Re: [Wikimedia-search] top articles across languages for testing?
by Brian Gerstle 20 May '15

20 May '15

+search On Tue, May 19, 2015 at 3:14 PM, Brian Gerstle <bgerstle(a)wikimedia.org> wrote: > The subject hints at a question that's been nagging me for a while, and > now that I'm going to be hacking on testing in Lyon I wanted to ask: > > Do we have a list of articles we usually run tests against? > > If not, do we have any processes for curating such a list? Would anyone > be interested in a brainstorming session at Lyon to discuss this further? > > Basically, as a developer, I would love to have more confidence that some > code I wrote doesn't break on our most popular articles. Or, if we can get > more sophisticated, that *certain properties of my code hold true for > certain kinds of generated pages*.* > > Please respond with your thoughts and whether you think I should create a > phab task for the hackathon about this. In either case, ping me anytime or > grab me at Lyon to discuss further! > > Regards, > > Brian > > * Yes, I'm talking about using property-based testing generators to create > random, shrinkable MW pages that we can run tests on. Not sure if it's > practical, but could be an interesting experiment. > > -- > EN Wikipedia user page: https://en.wikipedia.org/wiki/User:Brian.gerstle > IRC: bgerstle > -- EN Wikipedia user page: https://en.wikipedia.org/wiki/User:Brian.gerstle IRC: bgerstle

3 4

FYI: Phabricator project renamed
by Kevin Smith 20 May '15

20 May '15

FYI, I just renamed the phabricator Search-Team project to Search-and-Discovery-Department. It still has the #search-team hashtag, so most searches for the old name should still work. This has introduced an inconvenience, thanks to an odd Phabricator behavior. To auto-complete a project name, Phabricator only shows the first 5 matches, AND for some reason it is showing the "Search-and-Discovery-xxx" projects, and not the simpler "Search-and-Discovery". Argh! I have filed a phab ticket for this. I have updated the team page[1] to point to the new phab project. Let me know if there is any other documentation that also needs to be updated. Kevin Smith Agile Coach Wikimedia Foundation *Imagine a world in which every single human being can freely share in the sum of all knowledge. That's our commitment. Help us make it a reality.*

2 1

Removed restriction on non-members posting
by Nikolas Everett 20 May '15

20 May '15

I've removed the restriction that this list had on non-members posting to it. Its a public list and it doesn't make a whole lot of sense to have that restriction. Nik

2 1

Maps, Phab, and project naming
by Kevin Smith 19 May '15

19 May '15

Our plan has been to have all Search&Discovery issues managed in the Search-Team phabricator project, and then for each subteam to have a sprint board for its current work. However, Yuri pointed out that people wanting to create a new task related to maps would expect to add it to a project with "maps" in the name. Folks outside our team often would not realize that maps are part of the Search vertical. We really want to consolidate all of our product backlogs in one place, to help keep Dan sane. So I don't think keeping an external high-level "Maps" project is a viable option. And I don't think adding the word "maps" to the main Search-Team project would make sense either. We might want to rename our top-level project to "Search-and-Discovery-Team" for other reasons, but that wouldn't help this specific case. Even if we had a maps-related sprint board whose name would match a phab search for "map", that wouldn't be what we would want. We don't want random users dropping tasks directly into our sprint board. We really want new tasks to land in the product backlog first. Is this just a case where we have to educate folks that "maps" work goes in "Search-Team"? Or are there other options? Kevin Smith Agile Coach Wikimedia Foundation *Imagine a world in which every single human being can freely share in the sum of all knowledge. That's our commitment. Help us make it a reality.*

4 7

Fwd: Search API documentation
by S Page 16 May '15

16 May '15

(retrying mail aliases, sorry if you've already seen this) ---------- Forwarded message ---------- From: S Page <spage(a)wikimedia.org> Date: Thu, May 14, 2015 at 1:04 PM Subject: Search API documentation To: search-tech <search-tech(a)wikimedia.org> Cc: Quim Gil <qgil(a)wikimedia.org>, Brad Jorsch <bjorsch(a)wikimedia.org> Hey, James Douglas talked to me about search APIs. I started an overview of what's available at mw:API:Search_and_discovery <https://www.mediawiki.org/wiki/API:Search_and_discovery> that links to existing API pages and the generated MediaWiki API help. In particular the API pages have a link to [try in ApiSandbox] so you can load them up and play around with the MediaWiki API search modules. I added a link to this overview on your team's top-level mw:Search and Discovery page <https://www.mediawiki.org/wiki/Search_and_Discovery> page. <https://www.mediawiki.org/wiki/Search_and_Discovery> I'm also writing articles for a planned "Data and developer hub" to encourage third-party developers to use Wikimedia data and APIs. Fortuitously two of the early articles are about how desktop and mobile use search, "Page info in search results" and "Showing nearby wiki information", so API:Search and discovery links to these as well. Though I'm on the Reading team I'll do what I can to document your stuff, particularly what it can do for third-party developers. Cheers, -- =S Page WMF Tech writer

2 1

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

Wikimedia-search