Wikidata-tech November 2019

wikidata-tech@lists.wikimedia.org

14 participants
9 discussions

how to load a dump of wikidata into a local wikibase
by Miquel Àngel Farré 17 Jan '20

17 Jan '20

Hello, We are having issues launching a local copy of wikidata, when we use the 'importDump.php' tool, below the issues that we are facing. If somebody has an idea of how we could solve this, please let me know. We are also considering professional services to get fixes for this being released in case somebody is professionally consulting around wikibase. Thanks, Miquel Here the issues: if I try to load the full dump, the error I get is: root@4fc8cc9b76b3:/var/www/html/maintenance# php importDump.php --conf ../LocalSettings.php ../images/wikidatawiki-20191101-pages-articles-multistream.xml.bz2 Warning: XMLReader::read(): uploadsource://d0cd78c216b067ffdd60946c258db6a7:45: parser error : Extra content at the end of the document in /var/www/html/includes/import/WikiImporter.php on line 646 Warning: XMLReader::read(): </siteinfo> in /var/www/html/includes/import/WikiImporter.php on line 646 Warning: XMLReader::read(): ^ in /var/www/html/includes/import/WikiImporter.php on line 646 Done! You might want to run rebuildrecentchanges.php to regenerate RecentChanges, If I try to load a partial dump, the warnings that I get (which I think those mean nothing is loading) are: root@4fc8cc9b76b3:/var/www/html/maintenance# php importDump.php --conf ../LocalSettings.php ../images/wikidatawiki-20191020-pages-meta-current1.xml-p1p235321.bz2 Revision 1033865598 using content model wikibase-item cannot be stored on "Q15" on this wiki, since that model is not supported on that page. Revision 1034542603 using content model wikibase-item cannot be stored on "Q17" on this wiki, since that model is not supported on that page. Revision 1032554298 using content model wikibase-item cannot be stored on "Q18" on this wiki, since that model is not supported on that page. Revision 1032534215 using content model wikibase-item cannot be stored on "Q20" on this wiki, since that model is not supported on that page. Revision 1026713626 using content model wikibase-item cannot be stored on "Q21" on this wiki, since that model is not supported on that page. Revision 1023703278 using content model wikibase-item cannot be stored on "Q22" on this wiki, since that model is not supported on that page. Revision 1032815802 using content model wikibase-item cannot be stored on "Q25" on this wiki, since that model is not supported on that page. Revision 1032910600 using content model wikibase-item cannot be stored on "Q26" on this wiki, since that model is not supported on that page.

4 3

Fwd: [Wikitech-l] BREAKING CHANGE: schema update, xml dumps
by Lucas Werkmeister 28 Nov '19

28 Nov '19

Forwarding as this will also be relevant for people who consume Wikidata XML dumps (but not entity dumps), and especially for people who are interested in working with Structured Data on Commons from dumps. ---------- Forwarded message --------- Von: Ariel Glenn WMF <ariel(a)wikimedia.org> Date: Mi., 27. Nov. 2019 um 14:39 Uhr Subject: [Wikitech-l] BREAKING CHANGE: schema update, xml dumps To: Wikipedia Xmldatadumps-l <Xmldatadumps-l(a)lists.wikimedia.org>, Wikimedia developers <wikitech-l(a)lists.wikimedia.org> We plan to move to the new schema for xml dumps for the February 1, 2020 run. Update your scripts and apps accordingly! The new schema contains an entry for each 'slot' of content. This means that, for example, the commonswiki dump will contain MediaInfo information as well as the usual wikitext. See https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/core/+/master/docs… for the schema and https://www.mediawiki.org/wiki/Requests_for_comment/Schema_update_for_multi… for further explanation and example outputs. Phabricator task for the update: https://phabricator.wikimedia.org/T238972 PLEASE FORWARD to other lists as you deem appropriate. Thanks! Ariel Glenn _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l -- Lucas Werkmeister (he/er) Full Stack Developer Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin Phone: +49 (0)30 219 158 26-0 https://wikimedia.de Imagine a world in which every single human being can freely share in the sum of all knowledge. Help us to achieve our vision! https://spenden.wikimedia.de Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207.

1 1

Wikibase: new security fix available for the Query Service GUI
by Léa Lacroix 27 Nov '19

27 Nov '19

Hello all, This message is important to everyone running an instance of Wikibase including the Query Service GUI. We just released a new version of the Wikidata Query Service GUI. This release is primarily to fix several security issues described in T238822 <https://phabricator.wikimedia.org/T238822> and T238824 <https://phabricator.wikimedia.org/T238824> (these tasks will be made public soon). These are different from the previous fix we deployed on November 7th. The fix has been successfully deployed for the Wikidata Query Service. In order to keep your instance safe, please make sure to update your Query Service GUI! Git repositories, releases and currently active version docker images also include the latest fixed code (see links below). If you have a local test setup using the docker-compose example then see: https://gist.github.com/addshore/36f8d6fe2331d28ca8f70df5abda20fd Gerrit repositories: - https://gerrit.wikimedia.org/r/#/c/wikidata/query/gui/+/553311/ - https://gerrit.wikimedia.org/r/#/c/wikidata/query/gui-deploy/+/553313/ Docker images: - latest: digest: sha256:6570acb916b429f10ccb3bf3479b66aa6697b3fb3982166a09aba87eeaba7c90 - legacy: digest: sha256:4503257bbe1744ce389f07f6dcbaf53db7569cc3e570e30dd5a85c8d0073a39d If you have any questions or issues updating your code, please let us know (you can write me an email, or ask in the Wikibase Telegram group <https://t.me/joinchat/HGjGexZ9NE7BwpXzMsoDLA>) Thanks for your understanding, Cheers, -- Léa Lacroix Project Manager Community Communication for Wikidata Wikimedia Deutschland e.V. Tempelhofer Ufer 23-24 10963 Berlin www.wikimedia.de Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 Nz. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207.

1 0

Changed JS API?
by Max Kristen 18 Nov '19

18 Nov '19

Hi Tech-Mailinglist, I'm using this gadget here: https://www.wikidata.org/wiki/Wikidata:Tools/Enhance_user_interface#CiteTool in its current incarnation here: https://www.wikidata.org/wiki/User:MichaelSchoenitzer/CiteTool.js It doesn't seem to work since the last few weeks, maybe because it calls on PropertyValueSnak, which seems to have moved since the last JS API version. https://doc.wikimedia.org/Wikibase/master/js/#!/api/PropertyValueSnak Is the API not ready to be stable yet, and how can this be fixed? Greetings Max

4 5

How to respect throttling and retry-after headers on the Wikidata Query Service.
by Andra Waagmeester 17 Nov '19

17 Nov '19

Hi, I hope this is the right mailing list to discuss this issue. Some time ago I ran into a series of temporary bans, I thought I managed to tackle this basically by doing a full stop once it gets any response header code other than 200. However, this seems not to have fixed it, since I received the following message: "requests.exceptions.HTTPError: 403 Client Error: You have been banned until 2019-10-18T10:21:36.495Z, please respect throttling and retry-after headers. for url: https://query.wikidata.org/sparql" I am looking into this from scratch and see if I can implement a better solution and certainly one that really respects the retry-after time instead of going full stop. Whatever I try now, I keep getting 200 headers and I don't want to start an excessive bot run to get into a ban state to see the exact header that the bot needs to respect. Is there an example of such a header which I can use to make my own test script? Or is there example python could that successfully deals with a retry-after header? Regards, Andra

3 4

Wikibase: security fix available for the Query Service GUI
by Léa Lacroix 07 Nov '19

07 Nov '19

Hello all, -and sorry for cross-posting- This message is important to everyone running an instance of Wikibase including the Query Service GUI. We just released a new version of the Wikidata Query Service GUI. This release is primarily to fix a security issue described in T233213 <https://phabricator.wikimedia.org/T233213> (hidden task, will be made public soon). The fix has been successfully deployed for the Wikidata Query Service. In order to keep your instance safe, please make sure to update your Query Service GUI! Git repositories, releases and currently active version docker images also include the latest fixed code (see links below). If you have a local test setup using the docker-compose example then see: https://gist.github.com/addshore/36f8d6fe2331d28ca8f70df5abda20fd Gerrit repositories: - wikidata/query/gui after commit d9f964b88c01748e278ca8c4b8929a8ef0ef0267 - wikidata/query/gui-deploy after commit 7445472ab0ec61890b42e4d524416fbc6a18aa8a - wikidata/query/deploy after 094d9cda98f3fb706cf9c25aefa3eb33f9f6999a Docker images: - wdqs:0.3.6 (all versions of this tag) - wdqs:latest from digest sha256:04237b42d0b904a2c49ecb7059c82ace8265ba0b7f690ee2d4b3004ad39517ee - wdqs-frontend:latest from digest sha256:1308a7d6622b1e141783336fb52cd6993973321077f58359fbf907b77e105ca3 - wdqs-frontend:legacy from digest sha256:f830abd53fe5e79299011211a2aab7ad947181e56785c06eed6e9bd6b430d4ce Downloadable releases: - https://archiva.wikimedia.org/repository/snapshots/org/wikidata/query/rdf/s… If you have any questions or issues updating your code, please let us know (you can write me an email, or ask in the Wikibase Telegram group <https://t.me/joinchat/HGjGexZ9NE7BwpXzMsoDLA>) Thanks for your understanding, Cheers, -- Léa Lacroix Project Manager Community Communication for Wikidata Wikimedia Deutschland e.V. Tempelhofer Ufer 23-24 10963 Berlin www.wikimedia.de Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 Nz. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207.

1 0

Quick Search using ApostropheFilter ?
by Thad Guidry 07 Nov '19

07 Nov '19

Hi Team, I often see missed results in the Quick Search input box when phrases sometimes are indexed with an apostrophe, but its missed because we sometimes just type simple titles or simple names like so: "no mans sky" (no results) but when I click "containing... no mans sky" then the results page displays <https://www.wikidata.org/w/index.php?search=no%20mans%20sky&title=Special%3…> and shows the video game correctly. It seems this might be because of an indexing choice for the Quick Search input box? (I know that Lucene-Solr can handle this for English input preference https://github.com/apache/lucene-solr/blob/master/lucene/analysis/common/sr… ) Thad https://www.linkedin.com/in/thadguidry/

2 2

[TAIM] Wednesday: Technical Advice IRC Meeting
by Leszek Manicki 06 Nov '19

06 Nov '19

Sorry for cross-posting! Reminder: Technical Advice IRC meeting this week **Wednesday 4-5 pm UTC** on #wikimedia-tech. *Note the time change due to Berlin having switched to winter time!* Questions can be asked in English. The Technical Advice IRC Meeting (TAIM) is a weekly support event for volunteer developers. This week we have a special theme, related to the upcoming Wikimedia Technical Conference focusing on Developer Productivity. In particular we're interested in hearing about your experiences with developer productivity in the context of your work with on-wiki tools (gadgets, templates, modules, etc). Your input is going to be useful for the "Developer Productivity & onwiki tooling" ( https://phabricator.wikimedia.org/T234661) session held at the Conference next week. Hope to see you there! -- Leszek Manicki Engineering Manager Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin Phone: +49 (0)30 219 158 26-0 http://wikimedia.de Imagine a world in which every single human being can freely share in the sum of all knowledge. Help us to achieve our vision! https://spenden.wikimedia.de Wikimedia Deutschland – Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207.

1 0

Next office hour on November 5th on Telegram
by Léa Lacroix 05 Nov '19

05 Nov '19

Hello all, As you know, Lydia and i regularly organize Wikidata office hours, an online meeting where we're presenting an update from the development team, what we're working on at the moment, people can ask questions. The next one will take place on Tuesday, *November 5th* at 18:00 Berlin time (UTC+1) <https://www.timeanddate.com/worldclock/fixedtime.html?msg=Wikidata+Telegram…> on the *the Wikidata Telegram channel <https://t.me/joinchat/AZriqUj5Uag92TB4U9eBdQ>*. <https://t.me/joinchat/AZriqUj5Uag92TB4U9eBdQ> The previous meetings happened on IRC, and the content is later transferred onwiki. You can see the archives here <https://www.wikidata.org/wiki/Wikidata:Events/archive#Past_office_hours>. Since we noticed that over the past year, more and more activity takes place on the Telegram channel, and less on IRC, we decided to make an experiment and run the office hour on Telegram for the first time. As usual, the notes of the meeting will be published on wiki, so everyone can access them. Depending on how it works for us and for you, we may decide to continue the experiment, to go back to IRC, or even to find another tool for such an interactive meeting (I also plan to look at the tools that Wikimedia Space <https://space.wmflabs.org/> is offering). I hope you won't mind us trying new things :) If you have any question or remark, feel free to reach out to me. Cheers, -- Léa Lacroix Project Manager Community Communication for Wikidata Wikimedia Deutschland e.V. Tempelhofer Ufer 23-24 10963 Berlin www.wikimedia.de Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 Nz. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207.

1 2

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

Wikidata-tech November 2019