Hello,
We are having issues launching a local copy of wikidata, when we use the
'importDump.php' tool, below the issues that we are facing.
If somebody has an idea of how we could solve this, please let me know. We
are also considering professional services to get fixes for this being
released in case somebody is professionally consulting around wikibase.
Thanks,
Miquel
Here the issues:
if I try to load the full dump, the error I get is:
root@4fc8cc9b76b3:/var/www/html/maintenance# php importDump.php --conf
../LocalSettings.php
../images/wikidatawiki-20191101-pages-articles-multistream.xml.bz2
Warning: XMLReader::read():
uploadsource://d0cd78c216b067ffdd60946c258db6a7:45: parser error : Extra
content at the end of the document in
/var/www/html/includes/import/WikiImporter.php on line 646
Warning: XMLReader::read(): </siteinfo> in
/var/www/html/includes/import/WikiImporter.php on line 646
Warning: XMLReader::read(): ^ in
/var/www/html/includes/import/WikiImporter.php on line 646
Done!
You might want to run rebuildrecentchanges.php to regenerate RecentChanges,
If I try to load a partial dump, the warnings that I get (which I think
those mean nothing is loading) are:
root@4fc8cc9b76b3:/var/www/html/maintenance# php importDump.php --conf
../LocalSettings.php
../images/wikidatawiki-20191020-pages-meta-current1.xml-p1p235321.bz2
Revision 1033865598 using content model wikibase-item cannot be stored on
"Q15" on this wiki, since that model is not supported on that page.
Revision 1034542603 using content model wikibase-item cannot be stored on
"Q17" on this wiki, since that model is not supported on that page.
Revision 1032554298 using content model wikibase-item cannot be stored on
"Q18" on this wiki, since that model is not supported on that page.
Revision 1032534215 using content model wikibase-item cannot be stored on
"Q20" on this wiki, since that model is not supported on that page.
Revision 1026713626 using content model wikibase-item cannot be stored on
"Q21" on this wiki, since that model is not supported on that page.
Revision 1023703278 using content model wikibase-item cannot be stored on
"Q22" on this wiki, since that model is not supported on that page.
Revision 1032815802 using content model wikibase-item cannot be stored on
"Q25" on this wiki, since that model is not supported on that page.
Revision 1032910600 using content model wikibase-item cannot be stored on
"Q26" on this wiki, since that model is not supported on that page.
Forwarding as this will also be relevant for people who consume Wikidata
XML dumps (but not entity dumps), and especially for people who are
interested in working with Structured Data on Commons from dumps.
---------- Forwarded message ---------
Von: Ariel Glenn WMF <ariel(a)wikimedia.org>
Date: Mi., 27. Nov. 2019 um 14:39 Uhr
Subject: [Wikitech-l] BREAKING CHANGE: schema update, xml dumps
To: Wikipedia Xmldatadumps-l <Xmldatadumps-l(a)lists.wikimedia.org>,
Wikimedia developers <wikitech-l(a)lists.wikimedia.org>
We plan to move to the new schema for xml dumps for the February 1, 2020
run. Update your scripts and apps accordingly!
The new schema contains an entry for each 'slot' of content. This means
that, for example, the commonswiki dump will contain MediaInfo information
as well as the usual wikitext. See
https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/core/+/master/docs…
for the schema and
https://www.mediawiki.org/wiki/Requests_for_comment/Schema_update_for_multi…
for further explanation and example outputs.
Phabricator task for the update: https://phabricator.wikimedia.org/T238972
PLEASE FORWARD to other lists as you deem appropriate. Thanks!
Ariel Glenn
_______________________________________________
Wikitech-l mailing list
Wikitech-l(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
--
Lucas Werkmeister (he/er)
Full Stack Developer
Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin
Phone: +49 (0)30 219 158 26-0
https://wikimedia.de
Imagine a world in which every single human being can freely share in the
sum of all knowledge. Help us to achieve our vision!
https://spenden.wikimedia.de
Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V.
Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter
der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für
Körperschaften I Berlin, Steuernummer 27/029/42207.
Hello all,
This message is important to everyone running an instance of Wikibase
including the Query Service GUI.
We just released a new version of the Wikidata Query Service GUI. This
release is primarily to fix several security issues described in T238822
<https://phabricator.wikimedia.org/T238822> and T238824
<https://phabricator.wikimedia.org/T238824> (these tasks will be made
public soon). These are different from the previous fix we deployed on
November 7th. The fix has been successfully deployed for the Wikidata Query
Service.
In order to keep your instance safe, please make sure to update your Query
Service GUI!
Git repositories, releases and currently active version docker images also
include the latest fixed code (see links below). If you have a local test
setup using the docker-compose example then see:
https://gist.github.com/addshore/36f8d6fe2331d28ca8f70df5abda20fd
Gerrit repositories:
-
https://gerrit.wikimedia.org/r/#/c/wikidata/query/gui/+/553311/
-
https://gerrit.wikimedia.org/r/#/c/wikidata/query/gui-deploy/+/553313/
Docker images:
-
latest: digest:
sha256:6570acb916b429f10ccb3bf3479b66aa6697b3fb3982166a09aba87eeaba7c90
-
legacy: digest:
sha256:4503257bbe1744ce389f07f6dcbaf53db7569cc3e570e30dd5a85c8d0073a39d
If you have any questions or issues updating your code, please let us know
(you can write me an email, or ask in the Wikibase Telegram group
<https://t.me/joinchat/HGjGexZ9NE7BwpXzMsoDLA>)
Thanks for your understanding,
Cheers,
--
Léa Lacroix
Project Manager Community Communication for Wikidata
Wikimedia Deutschland e.V.
Tempelhofer Ufer 23-24
10963 Berlin
www.wikimedia.de
Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V.
Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter
der Nummer 23855 Nz. Als gemeinnützig anerkannt durch das Finanzamt für
Körperschaften I Berlin, Steuernummer 27/029/42207.
Hi,
I hope this is the right mailing list to discuss this issue.
Some time ago I ran into a series of temporary bans, I thought I managed to
tackle this basically by doing a full stop once it gets any response header
code other than 200.
However, this seems not to have fixed it, since I received the following
message:
"requests.exceptions.HTTPError: 403 Client Error: You have been banned
until 2019-10-18T10:21:36.495Z, please respect throttling and retry-after
headers. for url: https://query.wikidata.org/sparql"
I am looking into this from scratch and see if I can implement a better
solution and certainly one that really respects the retry-after time
instead of going full stop.
Whatever I try now, I keep getting 200 headers and I don't want to start an
excessive bot run to get into a ban state to see the exact header that the
bot needs to respect.
Is there an example of such a header which I can use to make my own test
script?
Or is there example python could that successfully deals with a retry-after
header?
Regards,
Andra
Hello all,
-and sorry for cross-posting-
This message is important to everyone running an instance of Wikibase
including the Query Service GUI.
We just released a new version of the Wikidata Query Service GUI. This
release is primarily to fix a security issue described in T233213
<https://phabricator.wikimedia.org/T233213> (hidden task, will be made
public soon). The fix has been successfully deployed for the Wikidata Query
Service.
In order to keep your instance safe, please make sure to update your Query
Service GUI!
Git repositories, releases and currently active version docker images also
include the latest fixed code (see links below). If you have a local test
setup using the docker-compose example then see:
https://gist.github.com/addshore/36f8d6fe2331d28ca8f70df5abda20fd
Gerrit repositories:
-
wikidata/query/gui after commit d9f964b88c01748e278ca8c4b8929a8ef0ef0267
-
wikidata/query/gui-deploy after commit
7445472ab0ec61890b42e4d524416fbc6a18aa8a
-
wikidata/query/deploy after 094d9cda98f3fb706cf9c25aefa3eb33f9f6999a
Docker images:
-
wdqs:0.3.6 (all versions of this tag)
-
wdqs:latest from digest
sha256:04237b42d0b904a2c49ecb7059c82ace8265ba0b7f690ee2d4b3004ad39517ee
-
wdqs-frontend:latest from digest
sha256:1308a7d6622b1e141783336fb52cd6993973321077f58359fbf907b77e105ca3
-
wdqs-frontend:legacy from digest
sha256:f830abd53fe5e79299011211a2aab7ad947181e56785c06eed6e9bd6b430d4ce
Downloadable releases:
-
https://archiva.wikimedia.org/repository/snapshots/org/wikidata/query/rdf/s…
If you have any questions or issues updating your code, please let us know
(you can write me an email, or ask in the Wikibase Telegram group
<https://t.me/joinchat/HGjGexZ9NE7BwpXzMsoDLA>)
Thanks for your understanding,
Cheers,
--
Léa Lacroix
Project Manager Community Communication for Wikidata
Wikimedia Deutschland e.V.
Tempelhofer Ufer 23-24
10963 Berlin
www.wikimedia.de
Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V.
Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter
der Nummer 23855 Nz. Als gemeinnützig anerkannt durch das Finanzamt für
Körperschaften I Berlin, Steuernummer 27/029/42207.
Sorry for cross-posting!
Reminder: Technical Advice IRC meeting this week **Wednesday 4-5 pm UTC**
on #wikimedia-tech.
*Note the time change due to Berlin having switched to winter time!*
Questions can be asked in English.
The Technical Advice IRC Meeting (TAIM) is a weekly support event for
volunteer developers. This week we have a special theme, related to the
upcoming Wikimedia Technical Conference focusing on Developer Productivity.
In particular we're interested in hearing about your experiences with
developer productivity in the context of your work with on-wiki tools
(gadgets, templates, modules, etc). Your input is going to be useful for
the "Developer Productivity & onwiki tooling" (
https://phabricator.wikimedia.org/T234661) session held at the Conference
next week.
Hope to see you there!
--
Leszek Manicki
Engineering Manager
Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin
Phone: +49 (0)30 219 158 26-0
http://wikimedia.de
Imagine a world in which every single human being can freely share in the
sum of all knowledge. Help us to achieve our vision!
https://spenden.wikimedia.de
Wikimedia Deutschland – Gesellschaft zur Förderung Freien Wissens e. V.
Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter
der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für
Körperschaften I Berlin, Steuernummer 27/029/42207.
Hello all,
As you know, Lydia and i regularly organize Wikidata office hours, an
online meeting where we're presenting an update from the development team,
what we're working on at the moment, people can ask questions.
The next one will take place on Tuesday, *November 5th* at 18:00 Berlin
time (UTC+1)
<https://www.timeanddate.com/worldclock/fixedtime.html?msg=Wikidata+Telegram…>
on the *the Wikidata Telegram channel
<https://t.me/joinchat/AZriqUj5Uag92TB4U9eBdQ>*.
<https://t.me/joinchat/AZriqUj5Uag92TB4U9eBdQ>
The previous meetings happened on IRC, and the content is later transferred
onwiki. You can see the archives here
<https://www.wikidata.org/wiki/Wikidata:Events/archive#Past_office_hours>.
Since we noticed that over the past year, more and more activity takes
place on the Telegram channel, and less on IRC, we decided to make an
experiment and run the office hour on Telegram for the first time. As
usual, the notes of the meeting will be published on wiki, so everyone can
access them.
Depending on how it works for us and for you, we may decide to continue the
experiment, to go back to IRC, or even to find another tool for such an
interactive meeting (I also plan to look at the tools that Wikimedia Space
<https://space.wmflabs.org/> is offering). I hope you won't mind us trying
new things :)
If you have any question or remark, feel free to reach out to me.
Cheers,
--
Léa Lacroix
Project Manager Community Communication for Wikidata
Wikimedia Deutschland e.V.
Tempelhofer Ufer 23-24
10963 Berlin
www.wikimedia.de
Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V.
Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter
der Nummer 23855 Nz. Als gemeinnützig anerkannt durch das Finanzamt für
Körperschaften I Berlin, Steuernummer 27/029/42207.