📢 Hey there, tomorrow it's Live Session again!
We will meet again after a two-month break. For those who are new to the group: Live Sessions are monthly virtual meet-ups, organized to share, discuss and learn from each other. We mostly follow a free agenda, have presentations or tackle specific topics together.
Details about how to participate:
*Time*: 4:00 pm UTC (UTC+2 for Berlin) *Day*: Thursday 27th July 2023 *Jitsi*: https://meet.jit.si/WikibaseLiveSession *Notes*: https://etherpad.wikimedia.org/P/WBUG_2023_07_27
I would love to see as many of you and catch up on what you've been up to these weeks. If you have any questions, wishes, ideas, feel free to come by and share them as well!
See you 👋🌊
Best,
Valerie Wollinger (She/Her) Community Communications Manager Wikibase
Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin Phone: +49 (0)30-577 11 62-0https://wikimedia.de
Keep up to date! Current news and exciting stories about Wikimedia, Wikipedia and Free Knowledge in our newsletter (in German): Subscribe now https://www.wikimedia.de/newsletter/.
Imagine a world in which every single human being can freely share in the sum of all knowledge. Help us to achieve our vision!https://spenden.wikimedia.de
Wikimedia Deutschland — Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Charlottenburg, VR 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207. Geschäftsführende Vorstände: Franziska Heine, Dr. Christian Humborg
Hi all,
Running a Docker instance of Wikibase (wikibase:1.36.3-wmde.4), I've noticed that CirrusSearch has stopped indexing new items. Everything seems to be fine: the ElasticSearch container is up and running, searches over legacy content work as expected, but searches over new content do not yield results. Previously there was no problem.
As mentioned, I've not managed to find anything out of place, so if there were any quick hints on where to look (in what container, log, etc.) to debug indexing new items in CirrusSearch/ElasticSearch, I would greatly appreciate it.
Best, Aidan
Hi Aidan,
Sorry to disappoint you, I don't have any advice for you, but I wanted to chime in nonetheless because we've seen similar behavior on a private project.
I posted this a while back: https://www.mediawiki.org/wiki/Topic:Xmowlgla1gs0aue8
And since then some people have reported that it's not only the prefixes that don't work, but several items are simply not indexed at all.
Unfortunately I didn't have any time yet to dig deeper into this, but I will let you know once I have and I will be closely monitoring this thread for further clues.
Regards, David
On 05/09/2023 16:24, Aidan Hogan wrote:
Hi all,
Running a Docker instance of Wikibase (wikibase:1.36.3-wmde.4), I've noticed that CirrusSearch has stopped indexing new items. Everything seems to be fine: the ElasticSearch container is up and running, searches over legacy content work as expected, but searches over new content do not yield results. Previously there was no problem.
As mentioned, I've not managed to find anything out of place, so if there were any quick hints on where to look (in what container, log, etc.) to debug indexing new items in CirrusSearch/ElasticSearch, I would greatly appreciate it.
Best, Aidan
Hi Aidan,
I just wanted to follow up on this mail with the results of my digging deeper into the subject, even if I'm not entirely sure it will fit your case.
To summarize, in our case, we were facing two distinct problems:
1. First, ElasticSearch had an incomplete index which we found by e.g. comparing the number of results from a SPARQL query to the number of results a search would give us, and 2. Second, specific Wikibase specific CirrusSearch keywords (such as haswbstatement:) were not bringing up any results.
The reason for the first issue seems to have been in either a version incompatibility or a misconfiguration of index shards, or possibly both.
In any case, we had upgraded from MW 1.35 to 1.39 (wmde.13) without also adapting the version of elasticsearch in our docker-compose file.
Once we were running the correct version of elasticsearch, we had to drop the current index (including "--startOver") and then rebuild it. We did this using the three following commands:
docker compose exec mediawiki php extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php    docker compose exec mediawiki php extensions/CirrusSearch/maintenance/ForceSearchIndex.php --skipParse    docker compose exec mediawiki php extensions/CirrusSearch/maintenance/ForceSearchIndex.php --skipLinks --indexOnSkip
Since then, the index has been consistent with the items in wdqs.
The second issue was due to missing configuration. I had to dump the entire index, the queries, and walk through the source code of the WikibaseCirrusSearch extension and the foundation's github repositories to figure out which configuration parameters need to be set to make this work.
In a nutshell, to make haswbstatement work, $wgWBRepoSettings['searchIndexProperties'] has to be configured:
$wgWBRepoSettings['searchIndexProperties'] = [ 'P1', 'P19', 'P23', ]; $wgWBRepoSettings['searchIndexTypes'] = [ Â Â Â Â Â 'string', 'external-id', 'url', 'wikibase-item', 'wikibase-property', Â Â Â Â Â 'wikibase-lexeme', 'wikibase-form', 'wikibase-sense' ];
These are set for WikiData, but not even documented (as far as I've seen) on the Help pages of the WikibaseCirrusSearch page.
To make wbstatementquantity: work, further configuration and most notably a property named "quantity" used as a qualifier on a statement is required.
Like I said, I fear this might not entirely fit your case, but I still wanted to share what I found.
Regards, David
On 05/09/2023 16:29, David Raison wrote:
Hi Aidan,
Sorry to disappoint you, I don't have any advice for you, but I wanted to chime in nonetheless because we've seen similar behavior on a private project.
I posted this a while back: https://www.mediawiki.org/wiki/Topic:Xmowlgla1gs0aue8
And since then some people have reported that it's not only the prefixes that don't work, but several items are simply not indexed at all.
Unfortunately I didn't have any time yet to dig deeper into this, but I will let you know once I have and I will be closely monitoring this thread for further clues.
Regards, David
On 05/09/2023 16:24, Aidan Hogan wrote:
Hi all,
Running a Docker instance of Wikibase (wikibase:1.36.3-wmde.4), I've noticed that CirrusSearch has stopped indexing new items. Everything seems to be fine: the ElasticSearch container is up and running, searches over legacy content work as expected, but searches over new content do not yield results. Previously there was no problem.
As mentioned, I've not managed to find anything out of place, so if there were any quick hints on where to look (in what container, log, etc.) to debug indexing new items in CirrusSearch/ElasticSearch, I would greatly appreciate it.
Best, Aidan
--
*TenTwentyFour S.à r.l.* www.tentwentyfour.lu https://www.tentwentyfour.lu *T*: +352 20 211 1024 *F*: +352 20 211 1023 1 place de l'Hôtel de Ville 4138 Esch-sur-Alzette
Wikibase Community User Group mailing list --wikibaseug@lists.wikimedia.org To unsubscribe send an email towikibaseug-leave@lists.wikimedia.org
One suggestion I'd make is to check that ElasticSearch is not out of disk space. The default setup can result in large log files that may need to be cleared, depending on usage. It is possible that restarting the container may help, but it might require manual action as well.
You can use 'docker ps' to check that the updater is running steadily and not repeatedly restarting, while tools such as 'df -h' and 'du -h -d 1' can be used to try to narrow down usage. 'docker logs --tail -n 50 (container)' may help with logs.
MediaWiki/WB 1.36 is quite old and no longer supported by Wikimedia, having both security and (I think, in Wikibase) performance deficiencies compared to the current LTS 1.39 version: https://github.com/wmde/wikibase-release-pipeline/blob/main/example/template...
It might be worth upgrading to the current version and seeing if that improves the situation. Note that 1.40 is not yet close to ready - https://phabricator.wikimedia.org/T340939 - but the 1.39.4 security release is being prepared: https://phabricator.wikimedia.org/T332786
Unfortunately the abivr
-- Laurence 'GreenReaper' Parry - WBUG https://GreenReaper.co.uk
________________________________ From: Aidan Hogan aidhog@gmail.com Sent: Tuesday, September 5, 2023 3:24:01 PM To: wikibaseug@lists.wikimedia.org wikibaseug@lists.wikimedia.org Subject: [Wikibase] Wikibase: CirrusSearch not indexing new items
Hi all,
Running a Docker instance of Wikibase (wikibase:1.36.3-wmde.4), I've noticed that CirrusSearch has stopped indexing new items. Everything seems to be fine: the ElasticSearch container is up and running, searches over legacy content work as expected, but searches over new content do not yield results. Previously there was no problem.
As mentioned, I've not managed to find anything out of place, so if there were any quick hints on where to look (in what container, log, etc.) to debug indexing new items in CirrusSearch/ElasticSearch, I would greatly appreciate it.
Best, Aidan _______________________________________________ Wikibase Community User Group mailing list -- wikibaseug@lists.wikimedia.org To unsubscribe send an email to wikibaseug-leave@lists.wikimedia.org
Note that 1.40 is not yet close to ready - https://phabricator.wikimedia.org/T340939 - but the 1.39.4 security release is being prepared: https://phabricator.wikimedia.org/T332786
Unfortunately the above process is currently manual, but it looks like the goal is to automate it per https://phabricator.wikimedia.org/T345564, bringing more timely updates in the future.
Yours distracted-by-a-fox, -- Laurence 'GreenReaper' Parry - WBUG https://GreenReaper.co.uk
wikibaseug@lists.wikimedia.org