Dynamic search results as category pages

List overview All Threads
Download

newer

older

How to retrieve the page, execute...

Job queue and Redis?

Yuri Astrakhan

24 Jan 2014 24 Jan '14

10:53 a.m.

Hi, I am thinking of implementing a #CATQUERY <query> magic keyword for the category pages. When this keyword is present, the category page would execute a query against the search backend instead of normal category behavior and show result as if those pages were actually marked with this category. For example, this would allow Greek Philosophers category page to be quickly redefined as a cross-section of greeks & philosophers categories: #CATQUERY incategory:Greek incategory:Philosopher Obviously the community will be able to define much more elaborate queries, including the ordering (will be supported by the new search backend)

Show replies by thread

Brian Wolff

24 Jan 24 Jan

4:55 p.m.

On Jan 24, 2014 1:54 AM, "Yuri Astrakhan" <yastrakhan(a)wikimedia.org> wrote:

...

queries,

...

including the ordering (will be supported by the new search backend) _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

I like the idea in principle, but think the syntax could use bikeshedding ;) If we use this as category pages, im a little worried that people could get confused and try to add [[category:Greek philosophers]] to a page, and expect it to work. We would need good error handling in that situation

...

including the ordering (will be supported by the new search backend)

Nik Everett

6:20 p.m.

Sent from my iPhone

...

On Jan 24, 2014, at 7:55 AM, Brian Wolff <bawolff(a)gmail.com> wrote:

On Jan 24, 2014 1:54 AM, "Yuri Astrakhan" <yastrakhan(a)wikimedia.org> wrote: Hi, I am thinking of implementing a #CATQUERY <query> magic keyword for the category pages. When this keyword is present, the category page would execute a query against the search backend instead of normal category behavior and show result as if those pages were actually marked with this category. For example, this would allow Greek Philosophers category page to be quickly redefined as a cross-section of greeks & philosophers categories: #CATQUERY incategory:Greek incategory:Philosopher Obviously the community will be able to define much more elaborate

queries,

Also, at least at first, you couldn't use one of these synthetic categories to filter search results so you couldn't use them to build other synthetic categories.

...

including the ordering (will be supported by the new search backend)

Cool. I didnt realize search would support this. That's a pretty big deal since people expect there categorirs alphabetized.

So I looked into it and it would be easy to implement with Cirrus _but_ I'm not yet sure about the memory implications. We have a ton of headroom on memory now so it might not matter but I need to test it before I can be as confident that it is ok as I was last night at 1am. It may have caveats like synthetic categories must be less than 1000 articles or not all results are included.

...

Another cool project would be to expand intersection/Dyanamic P age List (Wikimedia) to be able to use search as a different backend (however, that extension would need quite a bit of refactoring to get there) -bawolff _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Brad Jorsch (Anomie)

6:40 p.m.

On Fri, Jan 24, 2014 at 9:20 AM, Nik Everett <neverett(a)wikimedia.org> wrote:

...

On Jan 24, 2014, at 7:55 AM, Brian Wolff <bawolff(a)gmail.com> wrote: If we use this as category pages, im a little worried that people could

get

confused and try to add [[category:Greek philosophers]] to a page, and expect it to work. We would need good error handling in that situation

Also, at least at first, you couldn't use one of these synthetic categories to filter search results so you couldn't use them to build other synthetic categories.

Don't forget the API. You'd probably need to bascally override ApiQueryCategoryMembers to check cmtitle and either use parent::run() or a custom implementation. -- Brad Jorsch (Anomie) Software Engineer Wikimedia Foundation On Fri, Jan 24, 2014 at 9:20 AM, Nik Everett <neverett(a)wikimedia.org> wrote:

...

Sent from my iPhone

On Jan 24, 2014, at 7:55 AM, Brian Wolff <bawolff(a)gmail.com> wrote: > On Jan 24, 2014 1:54 AM, "Yuri Astrakhan" <yastrakhan(a)wikimedia.org>

wrote:

queries,

I like the idea in principle, but think the syntax could use

bikeshedding ;)

If we use this as category pages, im a little worried that people could

get

confused and try to add [[category:Greek philosophers]] to a page, and expect it to work. We would need good error handling in that situation

Also, at least at first, you couldn't use one of these synthetic categories to filter search results so you couldn't use them to build other synthetic categories.

including the ordering (will be supported by the new search backend)

Cool. I didnt realize search would support this. That's a pretty big deal since people expect there categorirs alphabetized.

Another cool project would be to expand intersection/Dyanamic P age List (Wikimedia) to be able to use search as a different backend (however,

that

extension would need quite a bit of refactoring to get there) -bawolff _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

_______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

-- Brad Jorsch (Anomie) Software Engineer Wikimedia Foundation

Jamie Thingelstad

25 Jan 25 Jan

10:13 p.m.

You could do something exactly like this very, very easily using Semantic MediaWiki and Concepts. Jamie Thingelstad jamie(a)thingelstad.com mobile: 612-810-3699 find me on AIM Twitter Facebook LinkedIn On Jan 24, 2014, at 9:55 AM, Brian Wolff <bawolff(a)gmail.com> wrote:

...

On Jan 24, 2014 1:54 AM, "Yuri Astrakhan" <yastrakhan(a)wikimedia.org> wrote:

queries,

including the ordering (will be supported by the new search backend)

Cool. I didnt realize search would support this. That's a pretty big deal since people expect there categorirs alphabetized. Another cool project would be to expand intersection/Dyanamic Page List (Wikimedia) to be able to use search as a different backend (however, that extension would need quite a bit of refactoring to get there) -bawolff _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Yuri Astrakhan

26 Jan 26 Jan

3:25 a.m.

Of course - I am actually modeling this feature on Semantic MW. It's just that I doubt WMF will enable SMW on the cluster any time soon :) On Sat, Jan 25, 2014 at 4:13 PM, Jamie Thingelstad <jamie(a)thingelstad.com>wrote;wrote:

...

On Jan 24, 2014 1:54 AM, "Yuri Astrakhan" <yastrakhan(a)wikimedia.org>

wrote:

queries,

I like the idea in principle, but think the syntax could use

bikeshedding ;)

If we use this as category pages, im a little worried that people could

get

confused and try to add [[category:Greek philosophers]] to a page, and expect it to work. We would need good error handling in that situation

including the ordering (will be supported by the new search backend)

that

_______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Gerard Meijssen

9:22 a.m.

Hoi, Did you consider the use of Wikidata in stead of SMW? It should not only indicate all the articles that have the appropriate statements but also the articles written in other languages. Thanks, GerardM On 26 January 2014 03:25, Yuri Astrakhan <yastrakhan(a)wikimedia.org> wrote:

...

wrote:

On Jan 24, 2014 1:54 AM, "Yuri Astrakhan" <yastrakhan(a)wikimedia.org>

wrote: >> >> Hi, I am thinking of implementing a >> >> #CATQUERY <query> >> >> magic keyword for the category pages. >> >> When this keyword is present, the category page would execute a query >> against the search backend instead of normal category behavior and

show

result as if those pages were actually marked with this category. For example, this would allow Greek Philosophers category page to be quickly redefined as a cross-section of greeks & philosophers categories: #CATQUERY incategory:Greek incategory:Philosopher Obviously the community will be able to define much more elaborate

queries,

I like the idea in principle, but think the syntax could use

bikeshedding ;)

If we use this as category pages, im a little worried that people could

get > confused and try to add [[category:Greek philosophers]] to a page, and > expect it to work. We would need good error handling in that situation > >> including the ordering (will be supported by the new search backend) > > Cool. I didnt realize search would support this. That's a pretty big

deal

since people expect there categorirs alphabetized. Another cool project would be to expand intersection/Dyanamic Page List (Wikimedia) to be able to use search as a different backend (however,

that

_______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Lydia Pintscher

10:10 a.m.

On Fri, Jan 24, 2014 at 10:53 AM, Yuri Astrakhan <yastrakhan(a)wikimedia.org> wrote:

...

In the future this will be possible using Wikidata and queries. My dream is that categories will to a very large extend go away then and be replaced with better Wikidata-based tools. I'd _really_ like us not to introduce more incentives to use categories like this one. Also it'd be quite a source for confusion to have both. Cheers Lydia -- Lydia Pintscher - http://about.me/lydia.pintscher Product Manager for Wikidata Wikimedia Deutschland e.V. Tempelhofer Ufer 23-24 10963 Berlin www.wikimedia.de Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 Nz. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/681/51985.

Helder .

12:47 p.m.

On Sun, Jan 26, 2014 at 12:25 AM, Yuri Astrakhan <yastrakhan(a)wikimedia.org>wrote;wrote:

...

Of course - I am actually modeling this feature on Semantic MW. It's just that I doubt WMF will enable SMW on the cluster any time soon :)

That request was already WONTFIXed: https://bugzilla.wikimedia.org/show_bug.cgi?id=8390 Helder

Bartosz Dziewoński

3:02 p.m.

I have a vague feeling that a very similar proposal appeared on this very list a few months ago. (No links handy, though.) -- Matma Rex

Brian Wolff

7:21 p.m.

On Jan 26, 2014 10:02 AM, "Bartosz Dziewoński" <matma.rex(a)gmail.com> wrote:

...

I have a vague feeling that a very similar proposal appeared on this very

list a few months ago. (No links handy, though.)

...

-- Matma Rex _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Category intersection proposals seem to appear on this list about once every 2 years as far as i remember. Usually they dont succede because nobody ends up writing any efficient code to implement it. -bawolff

Gryllida

9:31 p.m.

http://migadv.com/ comes to mind (https://meta.wikimedia.org/wiki/Grants:IEG/MediaWiki_data_browser) but sadly it's barely interactive... gry

Chad

30 Jan 30 Jan

7:50 p.m.

On Sun, Jan 26, 2014 at 1:10 AM, Lydia Pintscher < lydia.pintscher(a)wikimedia.de> wrote:

...

On Fri, Jan 24, 2014 at 10:53 AM, Yuri Astrakhan <yastrakhan(a)wikimedia.org> wrote:

queries,

including the ordering (will be supported by the new search backend)

Indeed. I think it would be very shortsighted to introduce any sort of new syntax here for this right now as proposed. There's lots of exciting query opportunities coming with Wikidata as Lydia says (which will very likely end up using Elasticsearch in some manner). On Sun, Jan 26, 2014 at 10:21 AM, Brian Wolff <bawolff(a)gmail.com> wrote:

...

Category intersection proposals seem to appear on this list about once every 2 years as far as i remember. Usually they dont succede because nobody ends up writing any efficient code to implement it.

Yep, we've been here before. Because category intersection isn't easy with the schema we've got. However, it's incredibly trivial to do in Elasticsearch. So I imagine if we invest more effort in helping Wikidata expand its syntax and uses we'll likely end up with what Yuri's wanting--if maybe in a different form. I'll also note we've already solved category intersection in Cirrus ;-) https://en.wikipedia.org/w/index.php?search=incategory%3A%221941+births%22+… -Chad

Gerard Meijssen

31 Jan 31 Jan

10:51 a.m.

Hoi, In Reasonator we now support calendars [1] It does show you things like people who were born / died on that date, events under way on that date .. Play with and notice that longer periods are also supported. Some longer periods may take some time but we have a waiting bar for you..It does support any of the languages that we support. It uses Wikidata information so if your favourite person or event is not mentioned. Just add the details in Wikidata. Thanks, GerardM PS three cheers for Magnus [1] On 30 January 2014 19:50, Chad <innocentkiller(a)gmail.com> wrote:

...

On Sun, Jan 26, 2014 at 1:10 AM, Lydia Pintscher < lydia.pintscher(a)wikimedia.de> wrote:

On Fri, Jan 24, 2014 at 10:53 AM, Yuri Astrakhan <yastrakhan(a)wikimedia.org> wrote:

queries,

including the ordering (will be supported by the new search backend)

Category intersection proposals seem to appear on this list about once every 2 years as far as i remember. Usually they dont succede because nobody ends up writing any efficient code to implement it.

Dan Garry

10:34 p.m.

Hey Yuri, This is a really cool idea. It's almost like set theory for categories! You can define certain base categories (based upon manually entering them into the category using existing category syntax), then define other categories based on set theoretic operations on categories. Certainly, if we could go back in time and build categories from the ground up again, I think this approach would be far superior. This feature is sufficiently complex from a product standpoint that I think the best thing to do here is to wait to see what happens with CirrusSearch's integration with Wikidata. Many cool features like this may end up being obsoleted by any effort for Wikidata integration. If not, we can revisit this then. Thanks, Dan On 24 January 2014 01:53, Yuri Astrakhan <yastrakhan(a)wikimedia.org> wrote:

...

-- Dan Garry Associate Product Manager for Platform Wikimedia Foundation

Chad

10:44 p.m.

On Thu, Jan 30, 2014 at 10:50 AM, Chad <innocentkiller(a)gmail.com> wrote:

...

I'll also note we've already solved category intersection in Cirrus ;-) https://en.wikipedia.org/w/index.php?search=incategory%3A%221941+births%22+…

That should read "in both the old and new search" actually. It just goes to show you how easy this is in lucene world. -Chad

Gerard Meijssen

1 Feb 1 Feb

10:36 a.m.

Hoi, With the "autolist" [1] and Widar you can make claims for the entries in a category. So you can first claim "occupation" "philosopher" and then make other claims about such philosophers ... Mind you claiming the Greek nationality for people who spoke Greek is problematic when these people were living before the creation of Greece. Possible claims are that they spoke Greek ... Thanks, GerardM [1] http://tools.wmflabs.org/wikidata-todo/autolist.html# On 24 January 2014 10:53, Yuri Astrakhan <yastrakhan(a)wikimedia.org> wrote:

...

3736

days inactive

3744

days old

wikitech-l@lists.wikimedia.org

Manage subscription

16 comments

12 participants

tags (0)

participants (12)

Bartosz Dziewoński
Brad Jorsch (Anomie)
Brian Wolff
Chad
Dan Garry
Gerard Meijssen
Gryllida
Helder .
Jamie Thingelstad
Lydia Pintscher
Nik Everett
Yuri Astrakhan