Wikidata-tech

wikidata-tech@lists.wikimedia.org

1 participants
626 discussions

[Significant change] New search profile parameter in Wikidata’s wbsearchentities API module
by Mohammed Sadat Abdulai 26 Jul '22

26 Jul '22

*(apologies for cross-posting)* *This announcement is significant for Wikidata API users.* As a follow-up to the recent improvements we made to the Special:NewLexeme <https://www.wikidata.org/wiki/Special:NewLexeme> page (see the previous announcement <https://lists.wikimedia.org/hyperkitty/list/wikidata@lists.wikimedia.org/th…>) we introduced a new search profile that will prioritize Items representing languages in the Lexeme language input. This will allow editors to more easily find the correct Item for the language of a Lexeme they are creating. Users of the wbsearchentities Action API module can use a new parameter, "profile", to select the search profile to use. The default value, "default", is equivalent to the previous behavior of the API, and should be suitable for most purposes. The "language" profile can be used to search for Items representing languages, as it will prioritize these Items as results. The wbsearch submodule of the query Action API module also gains this parameter (called "wbsprofile", or "gwbsprofile" when wbsearch is being used as a generator), with the same allowed values and meaning. The new parameter is optional, so most API users should not need to update their code; users who use the API to search for language Items can add "profile=language" to their request parameters. New profiles may be added in future without additional significant change announcements, though we will likely mention them in the weekly summary and other channels as appropriate. As with any Action API module, the valid values of the "profile" parameter may be discovered via the "help" or "paraminfo" modules, and the values will have documentation messages explaining what the profile can be used for. The new parameter is already available on Test Wikidata, though the two profiles will produce the same search results there. It will become available on Wikidata on Thursday, 28 July 2022. If you have feedback or questions please let us know on this phabricator ticket <https://phabricator.wikimedia.org/T307869>. Cheers -- Mohammed Sadat *Community Communications Manager for Wikidata/Wikibase* Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin Phone: +49 (0) 30 577 116 2466 https://wikimedia.de Keep up to date! Current news and exciting stories about Wikimedia, Wikipedia and Free Knowledge in our newsletter (in German): Subscribe now <https://www.wikimedia.de/newsletter/>. Imagine a world in which every single human being can freely share in the sum of all knowledge. Help us to achieve our vision! https://spenden.wikimedia.de Wikimedia Deutschland – Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207.

1 0

Upcoming Wikidata Bug Triage Hour on May 5th
by Léa Lacroix 05 May '22

05 May '22

Hello all, The next *Wikidata Bug Triage Hour session* will take place on Thursday, *May 5th, at 16:00 UTC* (18:00 CEST) on Jitsi. This meeting is open to everyone involved in Wikidata and interested in following the progress of the software development, or having specific issues or requests related to software. During this session, we look at tickets of our tracking system Phabricator, we improve their description, tags and keywords together, in order to make them more understandable and easier to discover and to pick by a development team or a volunteer in the future. The upcoming session doesn't have a specific topic, it's an open discussion, so you are welcome to join and *bring a Phabricator task that you particularly care about or that deserves some love*. The meeting is taking place on Jitsi, it is not recorded in video but we are taking collaborative notes. You can find all the links and information here: https://www.wikidata.org/wiki/Wikidata:Events#Wikidata_bug_triage_hour And finally: there is no prerequisite or need to have participated in previous sessions to join - you can also join only to listen and *discover more about how we track and describe bugs and feature requests for Wikidata :)* Looking forward to meet you there! -- Léa Lacroix Community Engagement Coordinator Wikimedia Deutschland e.V. Tempelhofer Ufer 23-24 10963 Berlin www.wikimedia.de Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 Nz. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207.

1 1

New "mul" term language code available on Test Wikidata
by Mohammed Sadat Abdulai 25 Apr '22

25 Apr '22

(This change is relevant for all Wikidata users working with labels, descriptions, and aliases.) Based on a long-standing community request we have enabled a new language code for labels, descriptions, and aliases on Test Wikidata: “mul”, a special language code meaning “multiple languages”. It is intended to replace the current duplication of certain labels and aliases in many languages: instead of the given name Douglas (Q463035) <https://www.wikidata.org/wiki/Q463035> having the label “Douglas” in hundreds of Latin-script languages, it should be enough to add it once as the “mul” label and have all other languages falling back to that (before, as usual, falling back to “en” as a last resort). This should reduce the amount of redundant data in Wikidata, and relieve some pressure from the query service. A big thank you goes to all people involved in the discussions! The purpose of the Test Wikidata version of this feature is to determine whether the current functionality is already sufficient, or whether the feature needs more work before it can be enabled on Wikidata proper. Current implementation: - You can interact with the new language code using the API. - The new language code appears in the table of labels at the top of an item page for users whose Babel information includes “mul” (or who use ? uselang=mul in the URL). If there is already a “mul” label, it will be available for everyone under the usual “all entered languages” option. - The page heading will fall back to the “mul” label if necessary. Fallbacks to “mul” have the usual “indicator” that shows a fallback took place (CSS class wb-language-fallback-mul). For more details, see T285156 <https://phabricator.wikimedia.org/T285156> (or T297393 <https://phabricator.wikimedia.org/T297393> for the Test Wikidata implementation). You are welcome to leave any feedback about the technical implementation of the new language code on these tickets. The new language code will very likely need adjustment of Wikidata policies and guidelines. In case you would like to contribute to drafting preliminary new guidelines, a good starting point is: Help talk:Label#Drafting of guidelines for new language code mul <https://www.wikidata.org/wiki/Help_talk:Label#Drafting_of_guidelines_for_ne…> . Cheers, -- Mohammed Sadat *Community Communications Manager for Wikidata/Wikibase* Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin Phone: +49 (0)30 219 158 26-0 https://wikimedia.de Keep up to date! Current news and exciting stories about Wikimedia, Wikipedia and Free Knowledge in our newsletter (in German): Subscribe now <https://www.wikimedia.de/newsletter/>. Imagine a world in which every single human being can freely share in the sum of all knowledge. Help us to achieve our vision! https://spenden.wikimedia.de Wikimedia Deutschland – Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207.

2 2

Need help with a suppression (oversight) request
by Rich Wales 22 Mar '22

22 Mar '22

I need to contact the person(s) responsible for "oversight" (suppression) on Wikidata. Someone asked the English Wikipedia oversight team to censor their IP address in the revision history of a Wikidata page. However, we (the English Wikipedia oversighters) do not have this sort of access rights on the Wikidata site. Whom can I talk to regarding this type of action on Wikidata? I am intentionally not including any details here, though I will of course be happy to share details with members of the Wikidata oversight team. Thanks. *Richwales* (no relation to Jimbo) -- a member of the English Wikipedia oversight team

2 1

[SIGNIFICANT CHANGE] wbsearchentities returns new display section
by Mohammed Sadat Abdulai 22 Mar '22

22 Mar '22

Hello, This is an announcement for a significant, but not breaking, change to the response format of the wbsearchentities API. So far, each search result has included "label" or "description" members- which included the label/description of the matched entity- so that it could be shown to users. However, these were plain strings with no indication of the language of the label or description: if language fallback happened (e.g. if the search was in Arabic, but the item only had an English description), then there would be no indication of this in the response. Users of the API couldn’t know what the language of the returned label/description was without a separate API call to retrieve the entity’s data. This is for example necessary for screen readers to read out the label/description in the correct language. (We will follow up with other changes to make use of it in the Wikibase UI, so that screen readers there are getting a better indication of which language to read out.) We have resolved this with the addition of a new "display" member to the search result structure, which contains a "label" and "description" that are not plain strings, but rather objects with "language" and "value" members (like labels and descriptions in the normal JSON serialization <https://doc.wikimedia.org/Wikibase/master/php/md_docs_topics_json.html#json…>). The old "label" and "description" fields are still there, but deprecated: we recommend that you don’t use them in new code. Please note that, just like the "label" and "description" were always optional (i.e. could be missing for entities that had no label or description available in the request language or a fallback language), so the "label" and "description" in the new response are also optional (under the same conditions). This change was already deployed to Beta Wikidata, Test Wikidata, and Wikidata. If you have any questions or feedback, please feel free to let us know in this ticket <https://phabricator.wikimedia.org/T104344>. Cheers, -- Mohammed Sadat *Community Communications Manager for Wikidata/Wikibase* Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin Phone: +49 (0)30 219 158 26-0 https://wikimedia.de Keep up to date! Current news and exciting stories about Wikimedia, Wikipedia and Free Knowledge in our newsletter (in German): Subscribe now <https://www.wikimedia.de/newsletter/>. Imagine a world in which every single human being can freely share in the sum of all knowledge. Help us to achieve our vision! https://spenden.wikimedia.de Wikimedia Deutschland – Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207.

1 0

ANN: A Go package providing utilities for processing Wikipedia and Wikidata dumps
by Mitar 09 Jan '22

09 Jan '22

Hi! I just published the first version of a Go package which provides utilities for processing Wikidata entities JSON dumps and Wikimedia Enterprise HTML dumps. It processes them in parallel on multiple cores, so processing is rather fast. I hope it will be useful to others, too. https://gitlab.com/tozd/go/mediawiki Any feedback is welcome. Mitar -- http://mitar.tnode.com/ https://twitter.com/mitar_m

1 0

The Wikibase 2022 Release for MediaWiki 1.36 is now available!
by Mohammed Sadat Abdulai 06 Jan '22

06 Jan '22

We hereby announce that the Wikibase 2022 Release for MediaWiki 1.36 has been published https://releases.wikimedia.org/wikibase/1.36 and on Docker Hub <https://hub.docker.com/u/wikibase>. Here are links to important documentation related to the release: - The full release notes <https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/Wikibas…> - Installation instructions <https://www.mediawiki.org/wiki/Wikibase/Suite> (manual installation) - Installation instructions <https://www.mediawiki.org/wiki/Wikibase/Docker> (Docker) - Updating your Wikibase <https://github.com/wmde/wikibase-release-pipeline/blob/main/docs/topics/upg…> (Docker) This new release is compatible with MediaWiki 1.36 and contains new Wikibase features: - WikibaseEdtf <https://github.com/ProfessionalWiki/WikibaseEdtf#wikibase-edtf> extension that allows to store date and time data following Extended Date/Time Format (EDTF) Specification <https://www.loc.gov/standards/datetime/> using a dedicated datatype. - SyntaxHighlight <https://www.mediawiki.org/wiki/Extension:SyntaxHighlight> extension that provides rich formatting of source code is added to wikibase-bundle container image - WDQS backend is updated to the most recent version 0.3.97 This release also requires ElasticSearch to be in use in order to have case-insensitive search of items and properties in Wikibase. If you have any questions feel free to ask on this mailing list or leave a comment at Talk:Wikibase/FAQ <https://www.mediawiki.org/wiki/Talk:Wikibase/FAQ>. -- Mohammed Sadat *Community Communications Manager for Wikidata/Wikibase* Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin Phone: +49 (0)30 219 158 26-0 https://wikimedia.de Keep up to date! Current news and exciting stories about Wikimedia, Wikipedia and Free Knowledge in our newsletter (in German): Subscribe now <https://www.wikimedia.de/newsletter/>. Imagine a world in which every single human being can freely share in the sum of all knowledge. Help us to achieve our vision! https://spenden.wikimedia.de Wikimedia Deutschland – Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207.

1 0

Wikibase: 1.35.5 security release updates
by Mohammed Sadat Abdulai 17 Dec '21

17 Dec '21

Hello, We would like to announce the following security and maintenance updates to the Wikibase 1.35 container image, which include fixes to severe security issues in MediaWiki and instructions for disabling features in ElasticSearch to mitigate the recently discovered log4shell vulnerability <https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-44228>. Here are links to important documentation related to the release, which include instructions for updating MediaWiki to 1.35.5 and a security fix for Wikibase: - MediaWiki release notes <https://github.com/wikimedia/mediawiki/blob/REL1_35/RELEASE-NOTES-1.35> - Wikibase release notes <https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/Wikibas…> - Upgrade instructions <https://github.com/wmde/wikibase-release-pipeline/blob/main/docs/topics/upg…> If updating your Wikibase installation is not an option, please refer to these instructions on disabling the vulnerable code in MediaWiki in the recent security release announcement. <https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/…> If you have any questions please feel free to ask on this mailing list or leave a comment at Talk:Wikibase/FAQ <https://www.mediawiki.org/wiki/Talk:Wikibase/FAQ>. Cheers, -- Mohammed Sadat *Community Communications Manager for Wikidata/Wikibase* Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin Phone: +49 (0)30 219 158 26-0 https://wikimedia.de Keep up to date! Current news and exciting stories about Wikimedia, Wikipedia and Free Knowledge in our newsletter (in German): Subscribe now <https://www.wikimedia.de/newsletter/>. Imagine a world in which every single human being can freely share in the sum of all knowledge. Help us to achieve our vision! https://spenden.wikimedia.de Wikimedia Deutschland – Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207.

1 0

Follow-up on the new WDQS Updater
by Zbyszko Papierski 27 Oct '21

27 Oct '21

Hello! I’d like to provide a bit more background and summarize a bit our work on the new WDQS updater, from the technical perspective. It has been common knowledge that the old updater had its issues. Main ones among them: - Low throughput, that often caused huge spikes of lag that were very hard to get down from (this is a nice example - [1]). - Reliance on Blazegraph to reconcile the data - Blazegraph’s reads affects writes and vice versa, which quite often caused a cascading failure for both update latency and query performance. - Ineffective handling of eventual consistency - this was one of the reasons for missing data in WDQS. What is worse is that we had very low visibility of what goes missing. We’ll be publishing a series of blog posts that will provide a more in-depth description of the architecture and the challenges during development - stay tuned! In the meantime, I want to explain a few things that about the new updater: - Higher best case lag is the result of the decisions of trading low latency for high consistency - considering the data we lost with the old updater, we think this approach is better in our situation. We would rather have a complete data set than a faster incomplete one. To make sure that we’re keeping the lag manageable, we introduced an SLO [2] and will introduce alerting on the lag being under 10 minutes. - Data is reconciled within the pipeline, which has a dramatically lower effect on Blazegraph. This should help with the updates, which was the goal, but also positively affects query engine stability. - As we previously mentioned in the general announcement, the difference in throughput is substantial (10 edits/sec vs 88 edits/sec) - which means a much faster catch up and more room to grow for Wikidata. The new updater can be scaled even more if necessary. The new Streaming Updater didn’t resolve all the issues magically and there are still two main ones, that we need to address: - Data loss - while the reconciliation mechanism works better than with the old updater, we literally lost updates without any way of knowing about it, other than user feedback - [3] [4]. This is a really bad way of finding out about issues. The new Streaming Updater can still miss data, especially due to late events or eventual consistency, as mentioned before. One thing that changed, however, is that the new updater has better inconsistency/late event reporting, which allows us to build a subsystem around it to reconcile the data. More information here - [5]. - Blazegraph instability - no matter how fast and stable the new updater might be, Blazegraph is still the last node in the process. That means that the whole update process will be affected by Blazegraph’s instability and will in turn produce a lag. One of the most common reasons for that instability is a so-called “GC death spiral”. A server in that state won’t answer any queries (which is a problem in itself), but after restarting, the lag will be high for some time. We are investigating a solution that can help us with this - [6]. I hope that answers at least some of the concerns already raised. Rest assured that we are working on way more things to improve the experience than the updater, all of which is, as always, available to see on our backlog board ([7]) and workboard ([8]). Any and all feedback welcome! Regards, Zbyszko [1] https://grafana.wikimedia.org/d/000000489/wikidata-query-service?viewPanel=… [2] https://grafana-rw.wikimedia.org/d/yCBd7Tdnk/wdqs-lag-slo [3] https://phabricator.wikimedia.org/T272120 [4] https://phabricator.wikimedia.org/T291609 [5] https://phabricator.wikimedia.org/T279541 [6] https://phabricator.wikimedia.org/T293862 [7] https://phabricator.wikimedia.org/tag/wikidata-query-service/ [8] https://phabricator.wikimedia.org/project/view/1227/ -- Zbyszko Papierski (He/Him) Senior Software Engineer Wikimedia Foundation <https://wikimediafoundation.org/>

1 0

[BREAKING CHANGE] Blank node deprecation in WDQS & Wikibase RDF model
by Mike Pham 22 Oct '21

22 Oct '21

Hi all! This is a reminder / retroactive breaking change announcement for the Wikidata Query Service and Wikibase RDF model. The change was already announced <https://lists.wikimedia.org/hyperkitty/list/wikidata@lists.wikimedia.org/me…>, and we invite you to review that announcement if you haven’t seen it before; but that email was quite a while ago, and we also didn’t send it to all of the usual channels, so we wanted to send this email to ensure everyone is aware of the change. The most important part: queries that tested for blank nodes using the standard isBlank() SPARQL function must now use wikibase:isSomeValue() instead. Best regards, — *Mike Pham* (he/him) Sr Product Manager, Search Wikimedia Foundation <https://wikimediafoundation.org/>

2 1

← Newer
1
2
3
4
5
6
7
...
63
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

Wikidata-tech