Hello,

I'm having a hard time setting up Cirrus Search on my Mediawiki / Wikibase instance, I'm quite confused so I thought I could ask for some help. :)

First, I followed the instructions to install both CirrusSearch and WikibaseCirrusSearch (WBCS) extensions, but I don't really know what my LocalSettings.php config file should look like. On one hand, CirrusSearch asks for this option`$wgSearchType = 'CirrusSearch';`, on the other hand WBCS asks for `$wgWBCSUseCirrus = true;`. When I put them both, I get a php error : `mediawiki/extensions/CirrusSearch/includes/Profile/SearchProfileService.php: A profile repository type rescore named wikibase_base is already registered.` 1/ So, do I need only one of them or is something else missing in my config file ? I might have missed a step in the installation ? 2/ The weird thing is, if I remove one of the two option, the search is somewhat working and is different than before using CirrusSearch, but autocompletion seems off, as it autocomplete only strings like "Item:QXXX" where XXX is the QID. I would have expected an autocomplete on item labels, not on QIDs like on Wikidata 3/ I'm not sure either if I have to fire some CirrusSearch maintenance scripts when I'm toying around with WBCS options in LocalSettings.php, because it appears that WBCS extension doesn't have any.

I apology in advance because I'm pretty sure this is a confusing post, but there are quite a lot of questions in my head. :)

Kevin

Hi Michael,

It's the official signup page here? https://lists.wikimedia.org/mailman/listinfo/Wikimedia-search

If not, then perhaps there needs to be a bit of cleanup on the Mailing lists page: https://meta.wikimedia.org/wiki/Mailing_lists/Overview where I saw a few mentions of "discovery".

Thad https://www.linkedin.com/in/thadguidry/

On Fri, Oct 4, 2019 at 8:29 AM Michael Holloway mholloway@wikimedia.org wrote:

Hello Kevin,

Yes, WikibaseCirrusSearch can be tricky to install. I'd recommend asking your question again on the Search and Discovery mailing list ( discovery@lists.wikimedia.org), where it's more likely to reach a member of the Wikimedia Search Platform Team, who are most likely to be able to help.

Good luck! Michael

On Mon, Sep 30, 2019 at 11:27 AM Kévin Bois < kevin.bois@biblissima-condorcet.fr> wrote:

...

Hello,

I'm having a hard time setting up Cirrus Search on my Mediawiki / Wikibase instance, I'm quite confused so I thought I could ask for some help. :)

First, I followed the instructions to install both CirrusSearch and WikibaseCirrusSearch (WBCS) extensions, but I don't really know what my LocalSettings.php config file should look like. On one hand, CirrusSearch asks for this option`$wgSearchType = 'CirrusSearch';`, on the other hand WBCS asks for `$wgWBCSUseCirrus = true;`. When I put them both, I get a php error : `mediawiki/extensions/CirrusSearch/includes/Profile/SearchProfileService.php: A profile repository type rescore named wikibase_base is already registered. ` 1/ So, do I need only one of them or is something else missing in my config file ? I might have missed a step in the installation ? 2/ The weird thing is, if I remove one of the two option, the search is somewhat working and is different than before using CirrusSearch, but autocompletion seems off, as it autocomplete only strings like "Item:QXXX" where XXX is the QID. I would have expected an autocomplete on item labels, not on QIDs like on Wikidata 3/ I'm not sure either if I have to fire some CirrusSearch maintenance scripts when I'm toying around with WBCS options in LocalSettings.php, because it appears that WBCS extension doesn't have any.

I apology in advance because I'm pretty sure this is a confusing post, but there are quite a lot of questions in my head. :)

Kevin _______________________________________________ Wikidata-tech mailing list Wikidata-tech@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-tech

-- Michael Holloway Software Engineer, Product Infrastructure _______________________________________________ Wikidata-tech mailing list Wikidata-tech@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-tech

Hello, Thank you for taking the time to answer.

So, following your advices, this is what my LocalSettings.php looks like now : wfLoadExtension( 'Elastica' ); require_once "$IP/extensions/CirrusSearch/CirrusSearch.php"; wfLoadExtension( 'WikibaseCirrusSearch' ); $wgSearchType = 'CirrusSearch'; $wgDebugLogGroups['CirrusSearch'] = "$IP/extensions/CirrusSearch/error.log"; $wgWBCSUseCirrus = true; $wgWBRepoSettings['disableCirrus'] = true;

I added the debug log line because unfortunately, I get an error message when trying to search for something. This is what the error.log file shows : ` 2019-10-07 12:52:57 kevin.local mediawiki: Search backend error during full_text search for 'test' after 21: script_exception: link error 2019-10-07 12:52:57 kevin.local mediawiki: Running sendData on cluster default 0s after insertion 2019-10-07 12:52:57 kevin.local mediawiki: Search backend error during sending 1 documents to the mediawiki_general_first index(s) after 5: unknown: Error in one or more bulk request actions:

update: /mediawiki_general_first/page/33085 caused blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];

2019-10-07 12:52:57 kevin.local mediawiki: ElasticaWrite job reported failure on cluster default. Requeueing job with delay of 64. `

It looks like an index error, so I tried to rerun updateSearchIndexConfig.php maintenance script, but this one gives me the same kind of error too. ` /mediawiki/extensions/Elastica/vendor/ruflin/elastica/lib/Elastica/Bulk.php: Error in one or more bulk request actions:

index: /mw_cirrus_metastore_first/mw_cirrus_metastore/namespace-mediawiki--2 caused blocked by: [FORBIDDEN/12/index read-only / allow delete (api)]; ` I'm a bit confused as I've managed to run this once when I first installed CirrusSearch, so I don't really understand why it wouldn't work now.

Finally, autocompletion seems to be working even though the search itself doesn't, but it only autocompletes Item:QID instead of working on the item label, do you have any idea why ?

Thanks for your help, Kevin

Le 4 oct. 2019 à 15:55, David Causse dcausse@wikimedia.org a écrit :

Please see my responses below

On Fri, Oct 4, 2019 at 3:41 PM Thad Guidry <thadguidry@gmail.com mailto:thadguidry@gmail.com> wrote: Hi Michael,

It's the official signup page here? https://lists.wikimedia.org/mailman/listinfo/Wikimedia-search https://lists.wikimedia.org/mailman/listinfo/Wikimedia-search

No it is here: https://lists.wikimedia.org/mailman/listinfo/discovery https://lists.wikimedia.org/mailman/listinfo/discovery

On Mon, Sep 30, 2019 at 11:27 AM Kévin Bois <kevin.bois@biblissima-condorcet.fr mailto:kevin.bois@biblissima-condorcet.fr> wrote: Hello,

I'm having a hard time setting up Cirrus Search on my Mediawiki / Wikibase instance, I'm quite confused so I thought I could ask for some help. :)

First, I followed the instructions to install both CirrusSearch and WikibaseCirrusSearch (WBCS) extensions, but I don't really know what my LocalSettings.php config file should look like. On one hand, CirrusSearch asks for this option`$wgSearchType = 'CirrusSearch';`, on the other hand WBCS asks for `$wgWBCSUseCirrus = true;`. When I put them both, I get a php error : `mediawiki/extensions/CirrusSearch/includes/Profile/SearchProfileService.php: A profile repository type rescore named wikibase_base is already registered.` 1/ So, do I need only one of them or is something else missing in my config file ? I might have missed a step in the installation ? 2/ The weird thing is, if I remove one of the two option, the search is somewhat working and is different than before using CirrusSearch, but autocompletion seems off, as it autocomplete only strings like "Item:QXXX" where XXX is the QID. I would have expected an autocomplete on item labels, not on QIDs like on Wikidata 3/ I'm not sure either if I have to fire some CirrusSearch maintenance scripts when I'm toying around with WBCS options in LocalSettings.php, because it appears that WBCS extension doesn't have any.

I apology in advance because I'm pretty sure this is a confusing post, but there are quite a lot of questions in my head. :)

1/ It is very probable that the version of the wikibase code that is installed still contain the CirrusSearch integration code. This code has been extracted in WikibaseCirrusSearch but during a transition period we kept the same code in both extensions (Wikibase and WikibaseCirrusSearch), enabling this code in both Wikibase and WikibaseCirrusSearch might lead to such error. If my assumption is correct setting "disableCirrus" to true in the Wikibase settings array might solve the problem as it will disable the CirrusSearch integration code present in Wikibase letting the one present in WikibaseCirrusSearch works (enabled with $wgWBCSUseCirrus = true). If not: having the exact version (or git revision hash) of these 3 extensions will help.

2/ By removing one of the two options you'll certainly end-up enabling/disabling CirrusSearch

3/ Yes in general you should run updateSearchIndexConfig.php and forceSearchIndex.php when enabling the extension, please see the README file in Cirrus, there are no specific maint script for the wikibase cirrus integration.

Hope it helps,

David.

---

Wikidata-tech mailing list Wikidata-tech@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-tech

Hello, This error was indeed a disk space problem, following your advice I did reset the read-only flag.

I came across multiple errors during the different maintenance scripts that I somehow resolved with a bit of googling, by installing those elasticsearch plugins : https://gerrit.wikimedia.org/r/plugins/gitiles/search/extra/ https://gerrit.wikimedia.org/r/plugins/gitiles/search/extra/

Then I had an error when trying to search something, something like "unknown highlighter of type [experimental"] which was resolved by : https://gerrit.wikimedia.org/r/plugins/gitiles/search/highlighter/ https://gerrit.wikimedia.org/r/plugins/gitiles/search/highlighter/

1/ So at this point, the search is working, but I still have a warning upon requesting something : "A warning has occurred while searching: Mixing entity and article namespaces in search is currently not supported. Only entity namespaces will be searched." It appears there is nothing in the log file for this warning.

2/ My autocompletion is still broken, suggesting only on "Item:QID".

3/ I noticed something odd with search results. I was trying to search an entity which has multiple aliases in different language (atleast 3 or 4), by typing one of its french aliases : - When my UI is in English, the entity is found but the highlight and the alias shown is from another language, neither french or english. - When my UI is in French, it shows me the correct french highlight and alias or label. I don't know if I'm clear, in both cases the entity is found correctly, but its highlighted label or alias is not consistent across the different UI languages.

Thanks for your time, Kevin

Le 7 oct. 2019 à 16:14, David Causse dcausse@wikimedia.org a écrit :

On Mon, Oct 7, 2019 at 3:05 PM Kévin Bois <kevin.bois@biblissima-condorcet.fr mailto:kevin.bois@biblissima-condorcet.fr> wrote: Hello, [...] It looks like an index error, so I tried to rerun updateSearchIndexConfig.php maintenance script, but this one gives me the same kind of error too. ` /mediawiki/extensions/Elastica/vendor/ruflin/elastica/lib/Elastica/Bulk.php: Error in one or more bulk request actions:

index: /mw_cirrus_metastore_first/mw_cirrus_metastore/namespace-mediawiki--2 caused blocked by: [FORBIDDEN/12/index read-only / allow delete (api)]; ` I'm a bit confused as I've managed to run this once when I first installed CirrusSearch, so I don't really understand why it wouldn't work now.

This error may appear when elasticsearch is running low on disk space, it can switch affected indices to read-only, please see https://www.elastic.co/guide/en/elasticsearch/reference/6.7/disk-allocator.h... https://www.elastic.co/guide/en/elasticsearch/reference/6.7/disk-allocator.html and the index.blocks.read_only_allow_delete property to reset the read-only flag. In short: double check disk usage on the partition where Elasticsearch has its data directory, reset the read-only flag for all indices marked as read-only and rerun the maint scripts as described in the CirrusSearch README file.

Finally, autocompletion seems to be working even though the search itself doesn't, but it only autocompletes Item:QID instead of working on the item label, do you have any idea why ?

The reason you still see results for QID queries is that WikibaseCirrusSearch still performs a query on the mysql database when it detects QIDs in the search query it's why you may see search results even though the elasticsearch infrastructure is broken.

---

Wikidata-tech mailing list Wikidata-tech@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-tech

Hello,

Le 16 oct. 2019 à 16:48, David Causse dcausse@wikimedia.org a écrit :

On Wed, Oct 9, 2019 at 6:13 PM Kévin Bois <kevin.bois@biblissima-condorcet.fr mailto:kevin.bois@biblissima-condorcet.fr> wrote: Hello, This error was indeed a disk space problem, following your advice I did reset the read-only flag.

I came across multiple errors during the different maintenance scripts that I somehow resolved with a bit of googling, by installing those elasticsearch plugins : https://gerrit.wikimedia.org/r/plugins/gitiles/search/extra/ https://gerrit.wikimedia.org/r/plugins/gitiles/search/extra/

Then I had an error when trying to search something, something like "unknown highlighter of type [experimental"] which was resolved by : https://gerrit.wikimedia.org/r/plugins/gitiles/search/highlighter/ https://gerrit.wikimedia.org/r/plugins/gitiles/search/highlighter/

Yes indeed these plugins are needed by CirrusSearch when Wikibase integration is enabled, we definitely need to improve the documentation.

1/ So at this point, the search is working, but I still have a warning upon requesting something : "A warning has occurred while searching: Mixing entity and article namespaces in search is currently not supported. Only entity namespaces will be searched." It appears there is nothing in the log file for this warning.

You are running an old version of WikibaseCirrusSearch. We no longer display this warning message. The classic page model and wikibase model are completely different and this is a real challenge to provide a seamless and integrated search experience for both types of data. The consensus was to provide an "optimized" query for the wikibase model only if the namespaces requested do include Items types. Example:

• searching on a Item and Property namespace: wikibase optimized
• searching on Help and Item: warning displayed (your version), wikidata production version: fallback to the classic wiki model
• searching on the default namespaces: it's dependent on your wiki configuration, if you don't setup the list of searched namespaces according to this limitation you will see this warning message for default searches

In other words: wgNamespacesToBeSearchedDefault[1] needs to be configured in a way that does not mix Item/Property namespaces with Wikipage namespaces.

Ok, as I understand I need a more up to date version. So far I'm playing around the 1_33 release because that is the latest release for Mediawiki / Wikibase.

2/ My autocompletion is still broken, suggesting only on "Item:QID".

Are you using the wbsearchentities API (which should be enabled when searching top right for ltr languages)? The completion results displayed when using the Special:Search search box might not use this API leading to QIDs being displayed.

You raised the right issue : I've checked in chrome's console, and the XHR sent is : "?action=opensearch&...". Do you know how can I set the search box to use the wbsearchentities API ? I don't remember seeing this option or disabling it anywhere.

3/ I noticed something odd with search results. I was trying to search an entity which has multiple aliases in different language (atleast 3 or 4), by typing one of its french aliases :

• When my UI is in English, the entity is found but the highlight and the alias shown is from another language, neither french or english. - When my UI is in French, it shows me the correct french highlight and alias or label.

I don't know if I'm clear, in both cases the entity is found correctly, but its highlighted label or alias is not consistent across the different UI languages.

No it's not, it is language dependent, every language has a list of fallback and french is not a fallback for english, but english is a fallback for french. When searching in english there are no guarantees that the ranking nor the data displayed will be the same. This was done to address the ambiguities between languages, e.g. when searching for "or" in french you would like to see Gold but if you were english you would perhaps expect something else.

David.

[1]: https://www.mediawiki.org/wiki/Manual:$wgNamespacesToBeSearchedDefault https://www.mediawiki.org/wiki/Manual:$wgNamespacesToBeSearchedDefault _______________________________________________ Wikidata-tech mailing list Wikidata-tech@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-tech

Thanks, Kevin

Hi Léa,

Sorry, my question was not very clear. Let us take the example of creating a new statement on an item. This can be done with wbeditentity but also with other actions such as wbcreateclaim or wbsetclaim.

Is there any plan to reuse the summaries of these latter actions when creating a single claim via wbeditentity, rather than the current uninformative one?

Broadly speaking, this is something that could be done for all wb* actions whose result can also be achieved via wbeditentity. This does not even require designing new summary messages (as they already exist, they are just not used by wbeditentity yet as far as I am aware).

This would allow clients such as Wikidata-Toolkit to always use wbeditentity, no matter what sort of edit they are doing, and still get informative edit summaries.

Currently, Wikidata-Toolkit has a fairly complicated logic [1] to choose which API action to use depending on the sort of edit. It would intuitively make sense to push that logic in Wikibase itself, so that it could benefit others.

Cheers, Antonin

[1] : https://github.com/Wikidata/Wikidata-Toolkit/blob/master/wdtk-wikibaseapi/sr...

On 30/09/2019 17:16, Léa Lacroix wrote:

In the near future, we will have different edit summaries depending on how many actions are made through the API. # If 5 or less terms (item, description or alias) are changed, we will display the expanded version: <Added/Changed/Removed> [<language code>] <label/description/alias>: <new label/description contents> # Does that answer your question or did you mean something else? :) # Cheers, # Léa

On Mon, 30 Sep 2019 at 16:23, Antonin Delpeuch (lists) <lists@antonin.delpeuch.eu mailto:lists@antonin.delpeuch.eu> wrote:

This is great news! Thank you so much for working on this!

It must be hard to figure out how to generate summaries for a wide range
of edit shapes, but it is definitely useful.

I cannot tell from the Phab ticket whether you considered to reuse
existing summaries from atomic actions (such as addition of a single
description) when the edit was made through wbeditentity?

For instance, as an API user it would be great if I could always use
wbeditentity regardless of the change I am making, and Wikibase would
automatically use a more granular edit summary if possible.

Best,
Antonin


On 30/09/2019 16:07, Léa Lacroix wrote:
> Hello all,
>
> We are about to make some changes to edit summaries that appear
when an
> edit is made using wbeditentity API, which includes for example
editing
> terms (labels, descriptions, aliases) from the new mobile termbox. The
> summary messages currently contain “Changed an Item” as a comment. The
> new summaries will include the message "/Changed label, description
> and/or aliases in # languages/", where # is the count of distinct
> languages that terms in them have been affected.
>
> This change will be deployed on Wednesday, October 2nd, and the
new edit
> summaries will start changing from that point. The old edit summaries
> will not be changed. More improvements will follow in the future (for
> example, more details regarding the terms that have been changed
and the
> languages).
>
> You can see more details in this ticket
> <https://phabricator.wikimedia.org/T220696>. If you encounter any
> issues, please let me know. Thanks,
>
> --
> Léa Lacroix
> Project Manager Community Communication for Wikidata
>
> Wikimedia Deutschland e.V.
> Tempelhofer Ufer 23-24
> 10963 Berlin
> www.wikimedia.de <http://www.wikimedia.de> <http://www.wikimedia.de>
>
> Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens
e. V.
>
> Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg
> unter der Nummer 23855 Nz. Als gemeinnützig anerkannt durch das
> Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207.
>
> _______________________________________________
> Wikidata-tech mailing list
> Wikidata-tech@lists.wikimedia.org
<mailto:Wikidata-tech@lists.wikimedia.org>
> https://lists.wikimedia.org/mailman/listinfo/wikidata-tech
>


_______________________________________________
Wikidata-tech mailing list
Wikidata-tech@lists.wikimedia.org
<mailto:Wikidata-tech@lists.wikimedia.org>
https://lists.wikimedia.org/mailman/listinfo/wikidata-tech

-- Léa Lacroix Project Manager Community Communication for Wikidata

Wikimedia Deutschland e.V. Tempelhofer Ufer 23-24 10963 Berlin www.wikimedia.de http://www.wikimedia.de

Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V.

Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 Nz. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207.

---

Wikidata-tech mailing list Wikidata-tech@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-tech

On Thu, Oct 3, 2019 at 10:57 AM Antonin Delpeuch (lists) lists@antonin.delpeuch.eu wrote:

In Phabricator terms, this corresponds to the following tickets: https://phabricator.wikimedia.org/T191885 https://phabricator.wikimedia.org/T67846

Hey Antonin :)

We hadn't looked at edits other than terms as part of the change but it's something I agree we should have. I'll bump it up and see what we can do.

Cheers Lydia