Dear Friends,

The Internet Archive is looking for people (especially engineers) to join the Turn All References Blue (TARB) team and build the Wikipedia Citations Database (WCD).

A "Hello World" version of the project, developed by James Hare, can be found here https://wikipediacitations.wiki.opencura.com/wiki/Main_Page.

In a nutshell… the goal of the WDC is to represent every citation, from every Wikipedia article, from every Wikipedia site, in a Wikibase. And to support ad-hoc queries on that database.

Of course we will start with smaller goals and work our way up.

Please email me if you, or someone you know, might be interested in helping out.

- Mark Graham Director, the Wayback Machine @ the Internet Archive (917) 697-0110

Dear Sir, I thank you for your answer. You can the report of this WikiCite project at https://meta.wikimedia.org/wiki/Wikicite/grant/Wikipedia_Citations_in_Wikida.... The report includes links to the source codes and to the database. Yours Sincerely, Houcemeddine Turki ________________________________ De : Houcemeddine A. Turki turkiabdelwaheb@hotmail.fr Envoyé : mercredi 10 novembre 2021 11:45 À : Wikibase Community User Group wikibaseug@lists.wikimedia.org Objet : RE: [Wikibase] Looking for people to help build a Wikipedia Citations Database

Dear Sir, I thank you for your answer. There is already a project in this context. Please refer to https://meta.wikimedia.org/wiki/Wikicite/grant/Wikipedia_Citations_in_Wikida... for further information. You can contact Dr. Silvio Peroni to ask him to include his database as a part of the Internet Archive. Yours Sincerely, Houcemeddine Turki ________________________________ De : Mark Graham mark@archive.org Envoyé : mardi 9 novembre 2021 23:47 À : Wikibase Community User Group wikibaseug@lists.wikimedia.org Objet : [Wikibase] Looking for people to help build a Wikipedia Citations Database

Dear Friends,

The Internet Archive is looking for people (especially engineers) to join the Turn All References Blue (TARB) team and build the Wikipedia Citations Database (WCD).

A "Hello World" version of the project, developed by James Hare, can be found herehttps://wikipediacitations.wiki.opencura.com/wiki/Main_Page.

In a nutshell… the goal of the WDC is to represent every citation, from every Wikipedia article, from every Wikipedia site, in a Wikibase. And to support ad-hoc queries on that database.

Of course we will start with smaller goals and work our way up.

Please email me if you, or someone you know, might be interested in helping out.

- Mark Graham Director, the Wayback Machine @ the Internet Archive (917) 697-0110

Hi Laurence

Many thanks for all the info, totally agree that ARM support could be very useful for the future with the new hardware.

I've only built the Wikibase docker image without "extras" but I'll give it a try.

WM has a complex CI/CD with Jenkins, that's the correct way but needs a lot of effort. What I've done is build manually all required docker images using the Dockerfiles (with some changes) from official git repos, including the base images for stretch/buster using the same build shell script from their repo (with some minor changes too). Finally I was able to generate the "composer-php73" docker image for ARM (my problem in the first message), then I built a wikibase docker image following the instructions: https://github.com/wmde/wikibase-release-pipeline/blob/main/docs/topics/pipe...

I'll write clean notes about all the steps that I followed and pass you, it would be great if WMDE includes ARM64 support.

Best, JM

On Sun, Nov 7, 2021 at 10:23 PM Laurence Parry greenreaper@hotmail.com wrote:

Hi JM (and Maarten),

The prospect of ARM64 support is something I've been looking into for a while now. It's not just Pi, but Apple M1; and relatively cheap servers from Ampere: https://www.anandtech.com/show/16315/the-ampere-altra-review

https://www.anandtech.com/show/16979/the-ampere-altra-max-review-pushing-it-... ...and VMs from Oracle, AWS, GCP, etc. So it'd be great to see what you've done so far so we can perhaps build on it, perhaps even with the ultimate goal of getting WMDE to pick it up as an architecture option for its releases. (We should open a ticket towards that goal, as a full release has quite a few dependencies.)

Not sure exactly how much you achieved so far, so some of this may be redundant, but here's some relevant components I collected for a draft reply to you earlier email:

WMF does not see supporting ARM64 packages as a big priority, because it would be a lot of work for an infrastructure they do not currently use: https://phabricator.wikimedia.org/T274140 https://phabricator.wikimedia.org/T283073 Which I guess is quite reasonable.

However there is still a ticket working towards ARM64 support for development images: https://phabricator.wikimedia.org/T272500

So far an ElasticSearch 6 image has been created which may be important, as I gather newer versions of Wikibase increasingly rely on CirrusSearch: https://gitlab.wikimedia.org/kharlan/wmf-elasticsearch-arm64

See https://mediawiki.org/wiki/Help:Extension:WikibaseCirrusSearch and https://mediawiki.org/wiki/Extension:WikibaseCirrusSearch for details of relevant functionality.

If this changes to OpenSearch (ES 7.10.2+ API): https://phabricator.wikimedia.org/T280482 ...then https://www.docker.elastic.co/r/elasticsearch/elasticsearch-oss:7.10.2-arm64 or a similar build from OpenSearch may be helpful instead. This seems likely to happen for a future release of MediaWiki (1.38+?): https://phabricator.wikimedia.org/T263142

I found an ARM64 build of Blazegraph 2.2.2 here: https://hub.docker.com/r/robcast/researchspace-blazegraph This might not be exactly what is required for WDQS, which has its own modifications, but perhaps it would be a step towards it?

If a base system is needed, this might be useful?

https://hub.docker.com/layers/mediawiki/library/mediawiki/stable-fpm-alpine/...

If you want something beefier for native building, Oracle has free ARM64 VMs up to 1x 4-core/24GB if you can navigate their registration and instantiation process: https://www.oracle.com/uk/cloud/free/

-- Laurence 'GreenReaper' Parry

---

*From:* jm zauron.subs@gmail.com *Sent:* Saturday, November 6, 2021 9:51:37 PM *To:* Wikibase Community User Group wikibaseug@lists.wikimedia.org *Subject:* [Wikibase] Wikibase ARM 64

Hi all,

I would like to run a Wikibase instance in my Raspberry Pi 4 for demo/sandbox purposes. I'm trying to generate a Wikibase docker image for ARM 64 bits, using a Raspberry Pi 4 (ARMv8) but I get this error:

• docker run --volume

/tmp/tmp.dBLABg/Wikibase/composer.json:/tmp/composer.json --volume /tmp/tmp.dBLABg/Wikibase/vendor/:/tmp/vendor/ docker-registry.wikimedia.org/releng/composer-php73:latest install --no-dev --ignore-platform-reqs -vv -d /tmp/ WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested standard_init_linux.go:228: exec user process caused: exec format error

Then I need a "docker-registry.wikimedia.org/releng/composer-php73" image for ARM, where could I find the source code to generate this image from scratch?

Best regards, JM

---

Wikibaseug mailing list -- wikibaseug@lists.wikimedia.org To unsubscribe send an email to wikibaseug-leave@lists.wikimedia.org

I've published the steps to build and run a Wikibase instance in a ARM 64 bits:

Thanks for keeping us up to date and sharing your progress so far! As you say, CI/CD is not necessary for many standalone deployments, especially if intended for personal or hobby use.

If it is taken forward by WMDE, I'd imagine server or at least VM infrastructure being used for it, but this path might be enough for developers to try.

I'm working to add Elasticsearch, WDQS and QuickStatements too, I'm quite near I think.

Yeah, the real power of Wikibase comes with the more advanced querying capabilities - although I've seen people do useful things without WDQS.

I suspect you may need to tweak the memory settings to get WDQS running as well, although it depends on the capacity of your hardware - the 8GB model might fit it easier than 2GB or 4GB.

For example, current WDQS's runBlazegraph.sh specifies a 16GB heap; the updater is set to use a 3GB in its runUpdate.sh (the streaming updater is 1GB, but probably relies on many more pieces).

Of course, they may be intending to be able to load a copy of Wikidata and use it in production, which may not be what you have in mind. Swapping to SSD also isn't the end of the world.

Keep us updated! I may see about getting myself going on that free Oracle VM this wekend... -- Laurence 'GreenReaper' Parry

Completed! I have been able to run a Wikibase bundle with WDQS, Quickstatements and Elasticsearch, thanks to https://github.com/wmde/wikibase-release-pipeline it was quite straightforward, only elasticsearch has been a little tricky.

Instructions here: https://github.com/jmformenti/docker-images/tree/master/raspberrypi/wikibase

Of course it is a beta version, tests will be necessary.

I've pushed the docker images to Docker Hub too, easy way to run Wikibase on ARM64: https://github.com/jmformenti/docker-images/tree/master/raspberrypi/wikibase...

Hope that could help anybody.

On Sat, Nov 13, 2021 at 12:07 PM jm zauron.subs@gmail.com wrote:

Thanks for your useful feedback, my goal is to build a shared Wikibase instance with a low load, just a common place to use as a sandbox between our team in a low-cost hardware.

Of course it would be great if these steps can help to build an official Wikibase support for ARM, and no doubt I would help in anything if it is necessary.

Regarding the hardware I guess that is required minimum 4GB and recommended 8GB model (I'm using the last one).

I was able to run WDQS, Elasticsearch and QuickStatements although I have to solve some issues, only starting without do nothing it takes 3GB RAM.

I'll be updating my progress :)

On Sat, Nov 13, 2021 at 8:57 AM Laurence Parry greenreaper@hotmail.com wrote:

...

...

I've published the steps to build and run a Wikibase instance in a ARM 64 bits:

Thanks for keeping us up to date and sharing your progress so far! As you say, CI/CD is not necessary for many standalone deployments, especially if intended for personal or hobby use.

If it is taken forward by WMDE, I'd imagine server or at least VM infrastructure being used for it, but this path might be enough for developers to try.

...

I'm working to add Elasticsearch, WDQS and QuickStatements too, I'm quite near I think.

Yeah, the real power of Wikibase comes with the more advanced querying capabilities - although I've seen people do useful things without WDQS.

I suspect you may need to tweak the memory settings to get WDQS running as well, although it depends on the capacity of your hardware - the 8GB model might fit it easier than 2GB or 4GB.

For example, current WDQS's runBlazegraph.sh specifies a 16GB heap; the updater is set to use a 3GB in its runUpdate.sh (the streaming updater is 1GB, but probably relies on many more pieces).

Of course, they may be intending to be able to load a copy of Wikidata and use it in production, which may not be what you have in mind. Swapping to SSD also isn't the end of the world.

Keep us updated! I may see about getting myself going on that free Oracle VM this wekend... -- Laurence 'GreenReaper' Parry _______________________________________________ Wikibaseug mailing list -- wikibaseug@lists.wikimedia.org To unsubscribe send an email to wikibaseug-leave@lists.wikimedia.org

Very nice! How much data seems managable on a Pi 4?

On Sat, 20 Nov 2021, 06:41 jm, zauron.subs@gmail.com wrote:

I detected a java bug for ARM64 in WDQS docker image, I've fixed it by re-publishing WDQS docker image "jmformenti/wdqs:0.3.40-wmde.2".

If someone tried to run Wikibase on ARM64 will need to remove wdqs docker image from local and do a new pull.

Instructions have been updated too, adding an extra step.

On Sun, Nov 14, 2021 at 6:40 PM jm zauron.subs@gmail.com wrote:

...

Completed! I have been able to run a Wikibase bundle with WDQS, Quickstatements and Elasticsearch, thanks to https://github.com/wmde/wikibase-release-pipeline it was quite straightforward, only elasticsearch has been a little tricky.

Instructions here: https://github.com/jmformenti/docker-images/tree/master/raspberrypi/wikibase

Of course it is a beta version, tests will be necessary.

I've pushed the docker images to Docker Hub too, easy way to run Wikibase on ARM64: https://github.com/jmformenti/docker-images/tree/master/raspberrypi/wikibase...

Hope that could help anybody.

On Sat, Nov 13, 2021 at 12:07 PM jm zauron.subs@gmail.com wrote:

...

Thanks for your useful feedback, my goal is to build a shared Wikibase instance with a low load, just a common place to use as a sandbox between our team in a low-cost hardware.

Of course it would be great if these steps can help to build an official Wikibase support for ARM, and no doubt I would help in anything if it is necessary.

Regarding the hardware I guess that is required minimum 4GB and recommended 8GB model (I'm using the last one).

I was able to run WDQS, Elasticsearch and QuickStatements although I have to solve some issues, only starting without do nothing it takes 3GB RAM.

I'll be updating my progress :)

On Sat, Nov 13, 2021 at 8:57 AM Laurence Parry greenreaper@hotmail.com wrote:

...

...

I've published the steps to build and run a Wikibase instance in a ARM 64 bits:

Thanks for keeping us up to date and sharing your progress so far! As you say, CI/CD is not necessary for many standalone deployments, especially if intended for personal or hobby use.

If it is taken forward by WMDE, I'd imagine server or at least VM infrastructure being used for it, but this path might be enough for developers to try.

...

I'm working to add Elasticsearch, WDQS and QuickStatements too, I'm quite near I think.

Yeah, the real power of Wikibase comes with the more advanced querying capabilities - although I've seen people do useful things without WDQS.

I suspect you may need to tweak the memory settings to get WDQS running as well, although it depends on the capacity of your hardware - the 8GB model might fit it easier than 2GB or 4GB.

For example, current WDQS's runBlazegraph.sh specifies a 16GB heap; the updater is set to use a 3GB in its runUpdate.sh (the streaming updater is 1GB, but probably relies on many more pieces).

Of course, they may be intending to be able to load a copy of Wikidata and use it in production, which may not be what you have in mind. Swapping to SSD also isn't the end of the world.

Keep us updated! I may see about getting myself going on that free Oracle VM this wekend... -- Laurence 'GreenReaper' Parry _______________________________________________ Wikibaseug mailing list -- wikibaseug@lists.wikimedia.org To unsubscribe send an email to wikibaseug-leave@lists.wikimedia.org

---

Wikibaseug mailing list -- wikibaseug@lists.wikimedia.org To unsubscribe send an email to wikibaseug-leave@lists.wikimedia.org

Updated Wikibase ARM64 to the latest version 1.36.3-wmde.4:

https://github.com/jmformenti/docker-images/tree/master/raspberrypi/wikibase...

Best, JM

On Sat, Nov 20, 2021 at 1:12 PM jm zauron.subs@gmail.com wrote:

I've just started to use with about 800 simple items without any problems, the feeling is good but I don't know the PI behavior with more data

if exists some kind of prepared tests to check the performance would be nice to run

On Sat, Nov 20, 2021 at 9:27 AM Dan Brickley via Wikibaseug < wikibaseug@lists.wikimedia.org> wrote:

...

Very nice! How much data seems managable on a Pi 4?

On Sat, 20 Nov 2021, 06:41 jm, zauron.subs@gmail.com wrote:

...

I detected a java bug for ARM64 in WDQS docker image, I've fixed it by re-publishing WDQS docker image "jmformenti/wdqs:0.3.40-wmde.2".

If someone tried to run Wikibase on ARM64 will need to remove wdqs docker image from local and do a new pull.

Instructions have been updated too, adding an extra step.

On Sun, Nov 14, 2021 at 6:40 PM jm zauron.subs@gmail.com wrote:

...

Completed! I have been able to run a Wikibase bundle with WDQS, Quickstatements and Elasticsearch, thanks to https://github.com/wmde/wikibase-release-pipeline it was quite straightforward, only elasticsearch has been a little tricky.

Instructions here: https://github.com/jmformenti/docker-images/tree/master/raspberrypi/wikibase

Of course it is a beta version, tests will be necessary.

I've pushed the docker images to Docker Hub too, easy way to run Wikibase on ARM64: https://github.com/jmformenti/docker-images/tree/master/raspberrypi/wikibase...

Hope that could help anybody.

On Sat, Nov 13, 2021 at 12:07 PM jm zauron.subs@gmail.com wrote:

...

Thanks for your useful feedback, my goal is to build a shared Wikibase instance with a low load, just a common place to use as a sandbox between our team in a low-cost hardware.

Of course it would be great if these steps can help to build an official Wikibase support for ARM, and no doubt I would help in anything if it is necessary.

Regarding the hardware I guess that is required minimum 4GB and recommended 8GB model (I'm using the last one).

I was able to run WDQS, Elasticsearch and QuickStatements although I have to solve some issues, only starting without do nothing it takes 3GB RAM.

I'll be updating my progress :)

On Sat, Nov 13, 2021 at 8:57 AM Laurence Parry < greenreaper@hotmail.com> wrote:

...

> I've published the steps to build and run a > Wikibase instance in a ARM 64 bits:

Thanks for keeping us up to date and sharing your progress so far! As you say, CI/CD is not necessary for many standalone deployments, especially if intended for personal or hobby use.

If it is taken forward by WMDE, I'd imagine server or at least VM infrastructure being used for it, but this path might be enough for developers to try.

> I'm working to add Elasticsearch, WDQS and > QuickStatements too, I'm quite near I think.

Yeah, the real power of Wikibase comes with the more advanced querying capabilities - although I've seen people do useful things without WDQS.

I suspect you may need to tweak the memory settings to get WDQS running as well, although it depends on the capacity of your hardware - the 8GB model might fit it easier than 2GB or 4GB.

For example, current WDQS's runBlazegraph.sh specifies a 16GB heap; the updater is set to use a 3GB in its runUpdate.sh (the streaming updater is 1GB, but probably relies on many more pieces).

Of course, they may be intending to be able to load a copy of Wikidata and use it in production, which may not be what you have in mind. Swapping to SSD also isn't the end of the world.

Keep us updated! I may see about getting myself going on that free Oracle VM this wekend... -- Laurence 'GreenReaper' Parry _______________________________________________ Wikibaseug mailing list -- wikibaseug@lists.wikimedia.org To unsubscribe send an email to wikibaseug-leave@lists.wikimedia.org

---

Wikibaseug mailing list -- wikibaseug@lists.wikimedia.org To unsubscribe send an email to wikibaseug-leave@lists.wikimedia.org

---

Wikibaseug mailing list -- wikibaseug@lists.wikimedia.org To unsubscribe send an email to wikibaseug-leave@lists.wikimedia.org