Hi!

I am building a wiki do document and collect information on the Palme case (https://en.wikipedia.org/wiki/Assassination_of_Olof_Palme). The case was closed two years ago and since then a lot of documents have been released. The police investigation is one of the three largest in world history. The complete material is around 60 000 documents consisting of around 1 000 000 pages, of which we have roughly 5%. My wiki, https://wpu.nu is collecting these documents, OCRs them with Google Cloud Vision, and publishes them using the Proofread Page extension. This is done using a python script running on the server accessing the wiki via the API. Some users are also writing "regular pages" and helps me sort through the material and proof read it. This bit works very well for the most part.

The wiki is running on a bare metal server with AMD Ryzen 5 3600 6-Core Processor (12 logical cores) and 64Gb of RAM. MariaDB (10.3.34) is used for the database. I have used Elastic for SMW data and fulltext search but have been switching back and forth in my debugging efforts.

From the investigation we have almost 60 000 sections (namespace Uppslag), 22 000 chapters (namespace Avsnitt) and 6000 documents (namespace Index). The documents and the Index namespace are handled by the Proofread Page extension. I have changed the PRP templates to suit the annotation and ui needs of wpu. For instance, each Index page has a semantic attribute pointing out which section it is attached to. Between all these pages there are semantic links that represents relations between the sections. This can be for instance the relation between a person and a specific gun, or an organisation or place.

Each namespace is rendered with its corresponding template, which in turn includes several other templates. The templates renders the ui but also contains a lot of business logic, that adds to categories, sets semantic data etc.

I will use an example to try to explain it better. This is an example of a section page: https://wpu.nu/wiki/Uppslag:E13-00 which in the header shows information such as date, document number and relations to other pages. Below that is the meta-information from the semantic data in the corresponding Index page, followed by the pages of that document. The metadata of the page is entered using Page Forms and rendered using the Uppslag_visning template. I use the Uppslag template to set a few variables that are used a lot in the Uppslag_visning template. Uppslag_visning also sets the page semantic data and categories. A semantic query is used to find if there is a corresponding Index page, if so a template is used to render its metadata. Another semantic query is used to get the pages of the Index and render them using template calls in the query.

Oh, and the skin is custom. It is based off of the Pivot-skin but changed extensively.

I have run in to a few problems which have led me to question if MediaWiki + SMW is the right tools for the job, question my sanity and the principle of cause and effect :) It is not a specific problem or bug as such.

Naturally, I often make changes to templates used by the 60 000 section pages. This queues a lot of refreshLinks-jobs in the job queue - initially taking a few hours to clear. I run the jobs as a service and experimented with the options to runJobs.php to get a good usage of the resources. I optimized the templates to reduce the resources needed for each job. (e.g. using proxy-templates to instanciate "variables" to reduce the number of identical function calls, save calculated data in semantic properties etc). This helped a little.

I noticed that a large portion of the refreshLinks-jobs failed with issues locking the Localization Cache table. At that moment i had the runJobs.php --maxjobs parameter set quite high, like 100-500 and --procs around 16 or 32. I lowered the --maxjobs to around 5 and the problem seemed solved. CPU utilization went down and also iowait. The job queue still took a very long time to clear. Looking at the mysql process list i found that a lot of time was spent by jobs trying to delete the Localisation Cache.

I switched to 'array' (and tested 'files' as well). This speed up the queue processing a bit but caused various errors. Sometimes the localisation cache data was read as a int(1) and sometimes the data seemed truncated. Looking at the source I found that the cache file writes and reads was not protected by any mutex or lock. That caused one job to read the LC file when another was writing it, causing a truncated file to be read. I implemented locks and exception handling in the LC-code for jobs to recover should they read corrupted data. I also mounted the $IP/cache dir on a ram-disk. The jobs now went through without LC-errors and a bit faster, but...

MediaWiki must be one of the most battle tested software systems written by man. How come they forgot to lock files that are used in a concurrent setting? I must be doing something wrong here?

Should the jobs be run serially? Why is the LC-cleared for each job when the cache should be clean? Maybe there is some kind of development and deployment path that circumvents this problem?

I could have lived with that the wiki lagged an hour or so, but I also experience data inconsistency. For instance sometimes the query for the Index finds an index, but the query for Pages finds nothing, resulting in that the metadata is filled in on the page but no document is shown. Sometimes when i purge a page the document is shown, and if I purge again, it is gone. Data from other sections in the same chapter is shown wrongly and various sequences of refresh and purge may fix it. The entire wiki is plagued with this type of inconsistency making it a very unreliable source of information for its users.

Any help, tips and pointers would be greatly appreciated.

A first step should probably be to get to a consistent state.

Best regards, Simon

PS: I am probably running a refreshLinks or rebuildData when you visit the wiki, so the info found there might vary.

__________ Versions: Ubuntu 20.04 64bit / Linux 5.4.0-107-generic MediaWiki 1.35.4 Semantic MediaWiki 3.2.3 PHP 7.4.3 (fpm-fcgi) MariaDB 10.3.34-MariaDB-0ubuntu0.20.04.1-log ICU 66.1 Lua 5.1.5 Elasticsearch 6.8.23