Very curious about the options to do something similar! Op za 6 nov. 2021 om 22:52 schreef jm <zauron.subs(a)gmail.com>om>: _______________________________________________ Wikibaseug mailing list -- wikibaseug(a)lists.wikimedia.org To unsubscribe send an email to wikibaseug-leave(a)lists.wikimedia.org

Hi JM (and Maarten), The prospect of ARM64 support is something I've been looking into for a while now. It's not just Pi, but Apple M1; and relatively cheap servers from Ampere: https://www.anandtech.com/show/16315/the-ampere-altra-review https://www.anandtech.com/show/16979/the-ampere-altra-max-review-pushing-it… ...and VMs from Oracle, AWS, GCP, etc. So it'd be great to see what you've done so far so we can perhaps build on it, perhaps even with the ultimate goal of getting WMDE to pick it up as an architecture option for its releases. (We should open a ticket towards that goal, as a full release has quite a few dependencies.) Not sure exactly how much you achieved so far, so some of this may be redundant, but here's some relevant components I collected for a draft reply to you earlier email: WMF does not see supporting ARM64 packages as a big priority, because it would be a lot of work for an infrastructure they do not currently use: https://phabricator.wikimedia.org/T274140 https://phabricator.wikimedia.org/T283073 Which I guess is quite reasonable. However there is still a ticket working towards ARM64 support for development images: https://phabricator.wikimedia.org/T272500 So far an ElasticSearch 6 image has been created which may be important, as I gather newer versions of Wikibase increasingly rely on CirrusSearch: https://gitlab.wikimedia.org/kharlan/wmf-elasticsearch-arm64 See https://mediawiki.org/wiki/Help:Extension:WikibaseCirrusSearch and https://mediawiki.org/wiki/Extension:WikibaseCirrusSearch for details of relevant functionality. If this changes to OpenSearch (ES 7.10.2+ API): https://phabricator.wikimedia.org/T280482 ...then https://www.docker.elastic.co/r/elasticsearch/elasticsearch-oss:7.10.2-arm64 or a similar build from OpenSearch may be helpful instead. This seems likely to happen for a future release of MediaWiki (1.38+?): https://phabricator.wikimedia.org/T263142 I found an ARM64 build of Blazegraph 2.2.2 here: https://hub.docker.com/r/robcast/researchspace-blazegraph This might not be exactly what is required for WDQS, which has its own modifications, but perhaps it would be a step towards it? If a base system is needed, this might be useful? https://hub.docker.com/layers/mediawiki/library/mediawiki/stable-fpm-alpine… If you want something beefier for native building, Oracle has free ARM64 VMs up to 1x 4-core/24GB if you can navigate their registration and instantiation process: https://www.oracle.com/uk/cloud/free/ -- Laurence 'GreenReaper' Parry ------------------------------ *From:* jm <zauron.subs(a)gmail.com> *Sent:* Saturday, November 6, 2021 9:51:37 PM *To:* Wikibase Community User Group <wikibaseug(a)lists.wikimedia.org> *Subject:* [Wikibase] Wikibase ARM 64 Hi all, I would like to run a Wikibase instance in my Raspberry Pi 4 for demo/sandbox purposes. I'm trying to generate a Wikibase docker image for ARM 64 bits, using a Raspberry Pi 4 (ARMv8) but I get this error: + docker run --volume /tmp/tmp.dBLABg/Wikibase/composer.json:/tmp/composer.json --volume /tmp/tmp.dBLABg/Wikibase/vendor/:/tmp/vendor/ docker-registry.wikimedia.org/releng/composer-php73:latest install --no-dev --ignore-platform-reqs -vv -d /tmp/ WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested standard_init_linux.go:228: exec user process caused: exec format error Then I need a "docker-registry.wikimedia.org/releng/composer-php73" image for ARM, where could I find the source code to generate this image from scratch? Best regards, JM _______________________________________________ Wikibaseug mailing list -- wikibaseug(a)lists.wikimedia.org To unsubscribe send an email to wikibaseug-leave(a)lists.wikimedia.org

Hi Laurence Many thanks for all the info, totally agree that ARM support could be very useful for the future with the new hardware. I've only built the Wikibase docker image without "extras" but I'll give it a try. WM has a complex CI/CD with Jenkins, that's the correct way but needs a lot of effort. What I've done is build manually all required docker images using the Dockerfiles (with some changes) from official git repos, including the base images for stretch/buster using the same build shell script from their repo (with some minor changes too). Finally I was able to generate the "composer-php73" docker image for ARM (my problem in the first message), then I built a wikibase docker image following the instructions: https://github.com/wmde/wikibase-release-pipeline/blob/main/docs/topics/pip… I'll write clean notes about all the steps that I followed and pass you, it would be great if WMDE includes ARM64 support. Best, JM On Sun, Nov 7, 2021 at 10:23 PM Laurence Parry <greenreaper(a)hotmail.com> wrote:

Thanks for keeping us up to date and sharing your progress so far! As you say, CI/CD is not necessary for many standalone deployments, especially if intended for personal or hobby use. If it is taken forward by WMDE, I'd imagine server or at least VM infrastructure being used for it, but this path might be enough for developers to try. Yeah, the real power of Wikibase comes with the more advanced querying capabilities - although I've seen people do useful things without WDQS. I suspect you may need to tweak the memory settings to get WDQS running as well, although it depends on the capacity of your hardware - the 8GB model might fit it easier than 2GB or 4GB. For example, current WDQS's runBlazegraph.sh specifies a 16GB heap; the updater is set to use a 3GB in its runUpdate.sh (the streaming updater is 1GB, but probably relies on many more pieces). Of course, they may be intending to be able to load a copy of Wikidata and use it in production, which may not be what you have in mind. Swapping to SSD also isn't the end of the world. Keep us updated! I may see about getting myself going on that free Oracle VM this wekend... -- Laurence 'GreenReaper' Parry _______________________________________________ Wikibaseug mailing list -- wikibaseug(a)lists.wikimedia.org To unsubscribe send an email to wikibaseug-leave(a)lists.wikimedia.org

Completed! I have been able to run a Wikibase bundle with WDQS, Quickstatements and Elasticsearch, thanks to https://github.com/wmde/wikibase-release-pipeline it was quite straightforward, only elasticsearch has been a little tricky. Instructions here: https://github.com/jmformenti/docker-images/tree/master/raspberrypi/wikibase Of course it is a beta version, tests will be necessary. I've pushed the docker images to Docker Hub too, easy way to run Wikibase on ARM64: https://github.com/jmformenti/docker-images/tree/master/raspberrypi/wikibas… Hope that could help anybody. On Sat, Nov 13, 2021 at 12:07 PM jm <zauron.subs(a)gmail.com> wrote: > Thanks for your useful feedback, my goal is to build a shared Wikibase > instance with a low load, just a common place to use as a sandbox between > our team in a low-cost hardware. > > Of course it would be great if these steps can help to build an official > Wikibase support for ARM, and no doubt I would help in anything if it is > necessary. > > Regarding the hardware I guess that is required minimum 4GB and > recommended 8GB model (I'm using the last one). > > I was able to run WDQS, Elasticsearch and QuickStatements although I have > to solve some issues, only starting without do nothing it takes 3GB RAM. > > I'll be updating my progress :) > > > > > > > > > > > On Sat, Nov 13, 2021 at 8:57 AM Laurence Parry <greenreaper(a)hotmail.com> > wrote: > >> > I've published the steps to build and run a >> > Wikibase instance in a ARM 64 bits: >> >> Thanks for keeping us up to date and sharing your progress so far! As >> you say, CI/CD is not necessary for many standalone deployments, especially >> if intended for personal or hobby use. >> >> If it is taken forward by WMDE, I'd imagine server or at least VM >> infrastructure being used for it, but this path might be enough for >> developers to try. >> >> > I'm working to add Elasticsearch, WDQS and >> > QuickStatements too, I'm quite near I think. >> >> Yeah, the real power of Wikibase comes with the more advanced querying >> capabilities - although I've seen people do useful things without WDQS. >> >> I suspect you may need to tweak the memory settings to get WDQS running >> as well, although it depends on the capacity of your hardware - the 8GB >> model might fit it easier than 2GB or 4GB. >> >> For example, current WDQS's runBlazegraph.sh specifies a 16GB heap; the >> updater is set to use a 3GB in its runUpdate.sh (the streaming updater is >> 1GB, but probably relies on many more pieces). >> >> Of course, they may be intending to be able to load a copy of Wikidata >> and use it in production, which may not be what you have in mind. Swapping >> to SSD also isn't the end of the world. >> >> Keep us updated! I may see about getting myself going on that free >> Oracle VM this wekend... >> -- >> Laurence 'GreenReaper' Parry >> _______________________________________________ >> Wikibaseug mailing list -- wikibaseug(a)lists.wikimedia.org >> To unsubscribe send an email to wikibaseug-leave(a)lists.wikimedia.org >> >

Very nice! How much data seems managable on a Pi 4? On Sat, 20 Nov 2021, 06:41 jm, <zauron.subs(a)gmail.com> wrote: _______________________________________________ Wikibaseug mailing list -- wikibaseug(a)lists.wikimedia.org To unsubscribe send an email to wikibaseug-leave(a)lists.wikimedia.org