Hi all,
The next Research Showcase will be live-streamed next Wednesday, September
18, at 9:30 AM PST / 16:30 UTC. Find your local time here
<https://zonestamp.toolforge.org/1726677000>. The theme for this showcase is
*Curation of Wikimedia AI Datasets*.
You are welcome to watch via the YouTube stream:
https://youtube.com/live/USzLGJ5LLC8?feature=share. As usual, you can join
the conversation in the YouTube chat as soon as the showcase goes live.
This month's presentations:
Supporting Community-Driven Data Curation for AI Evaluation on Wikipedia
through WikibenchBy *Tzu-Sheng Kuo, Carnegie Mellon University*AI tools are
increasingly deployed in community contexts. However, datasets used to
evaluate AI are typically created by developers and annotators outside a
given community, which can yield misleading conclusions about AI
performance. How might we empower communities to drive the intentional
design and curation of evaluation datasets for AI that impacts them? We
investigate this question on Wikipedia, an online community with multiple
AI-based content moderation tools deployed. We introduce Wikibench, a
system that enables communities to collaboratively curate AI evaluation
datasets, while navigating ambiguities and differences in perspective
through discussion. A field study on Wikipedia shows that datasets curated
using Wikibench can effectively capture community consensus, disagreement,
and uncertainty. Furthermore, study participants used Wikibench to shape
the overall data curation process, including refining label definitions,
determining data inclusion criteria, and authoring data statements. Based
on our findings, we propose future directions for systems that support
community-driven data curation.WikiContradict: A Benchmark for Evaluating
LLMs on Real-World Knowledge Conflicts from WikipediaBy *Yufang Hou, IBM
Research Europe - Ireland*Retrieval-augmented generation (RAG) has emerged
as a promising solution to mitigate the limitations of large language
models (LLMs), such as hallucinations and outdated information. However, it
remains unclear how LLMs handle knowledge conflicts arising from different
augmented retrieved passages, especially when these passages originate from
the same source and have equal trustworthiness. In this work, we conduct a
comprehensive evaluation of LLM-generated answers to questions that have
varying answers based on contradictory passages from Wikipedia, a dataset
widely regarded as a high-quality pre-training resource for most LLMs.
Specifically, we introduce WikiContradict, a benchmark consisting of 253
high-quality, human-annotated instances designed to assess LLM performance
when augmented with retrieved passages containing real-world knowledge
conflicts. We benchmark a diverse range of both closed and open-source LLMs
under different QA scenarios, including RAG with a single passage, and RAG
with 2 contradictory passages. Through rigorous human evaluations on a
subset of WikiContradict instances involving 5 LLMs and over 3,500
judgements, we shed light on the behaviour and limitations of these models.
For instance, when provided with two passages containing contradictory
facts, all models struggle to generate answers that accurately reflect the
conflicting nature of the context, especially for implicit conflicts
requiring reasoning. Since human evaluation is costly, we also introduce an
automated model that estimates LLM performance using a strong open-source
language model, achieving an F-score of 0.8. Using this automated metric,
we evaluate more than 1,500 answers from seven LLMs across all
WikiContradict instances. To facilitate future work, we release
WikiContradict on: this https URL <https://ibm.biz/wikicontradict>.
Best,Kinneret
--
Kinneret Gordon
Lead Research Community Officer
Wikimedia Foundation <https://wikimediafoundation.org/>
Dear Wikimedians,
Over the past four years, we have witnessed a remarkable transformation in
the WikiForHumanRights campaign
<https://meta.wikimedia.org/wiki/WikiForHumanRights>, driven by the
dedicated efforts of regional coordinators, affiliates, local organizers,
editors, and the entire Wikimedia movement.
This collective impact has reshaped and elevated the campaign in profound
ways. As we look to the future, we are inviting the movement to engage in
regional conversations that explore opportunities to shape their leadership
and participation in the campaign.
Are you a regional coordinator, affiliate, local organizer, editor, or
Wikimedia volunteer in Africa who contributes to the WikiForHumanRights
campaign, either directly or indirectly?
We invite you to register and join the Africa regional conversation.
*Date: Wed 25th Sept 2024*
*Time: 15:00 UTC *
*Via: Zoom*
*Register: *
https://meta.wikimedia.org/wiki/Event:Campaigns/Future_of_WikiForHumanRight…
Please help spread the word!
Best regards,
On behalf of WikiForHumanRights Team
*Find link to graphic: *
https://commons.wikimedia.org/wiki/File:Future_of_WikiForHumanRights_Africa…
*——French——*
Cher-e-s Wikimédien-e-s,
Au cours des 4 dernières années, nous avons assisté à une transformation
remarquable de la campagne WikiForHumanRights,
<https://meta.wikimedia.org/wiki/WikiForHumanRights> grâce aux efforts
dévoués des coordinateurs régionaux, des affiliés, des organisateurs
locaux, des éditeurs et de l'ensemble du mouvement Wikimedia.
Cet impact collectif a permis de remodeler et d'élever la campagne en
profondeur. Alors que nous nous tournons vers l'avenir, nous invitons le
mouvement à s'engager dans des conversations régionales qui explorent les
possibilités de façonner leur leadership et leur participation à la
campagne.
Êtes-vous un affilié, un organisateur local, un éditeur ou un volontaire
Wikimedia en Afrique qui contribue à la campagne W4HR, directement ou
indirectement ?
Nous vous invitons à vous inscrire et à rejoindre la conversation régionale
africaine.
*Date : Mercredi 25 septembre 2024*
*Heure : 15:00 UTC *
*Via: Zoom*
*Lien d’inscription : https://w.wiki/BDRV <https://w.wiki/BDRV>*
Merci de faire passer le message !
Dear all,
I am pleased to share with you the annual Activity Report of the WikiClassics User Group for the year 2023-2024. You can access it via this link https://meta.wikimedia.org/wiki/WikiClassics_User_Group/Reports/2023
The report has also been shared on our social media today.
Best wishes,Alessandro
Hi,
A reminder that the voting period for this year's Board of Trustees
election will close in just over 30 hours. If you have not voted yet,
make sure to have your say
<https://meta.wikimedia.org/wiki/Special:SecurePoll/vote/400>.
Thank you to everyone who has already participated in the 2024 Wikimedia
Foundation Board of Trustees election
<https://meta.wikimedia.org/wiki/Wikimedia_Foundation_elections/2024>!
Voting will close on September 17th at 23:59 UTC.
The official data, including the preliminary results
<https://meta.wikimedia.org/wiki/Special:MyLanguage/Wikimedia_Foundation_ele…>,
is scheduled to be announced in October, after the scrutineers have
completed their work. The official announcement of the new trustees will
happen later, once the selected candidates have been approved (all
selected candidates are subject to a background and media check and are
not eligible to be seated until they pass the checks) and appointed by
the Board.
Data produced during the election will be published
<https://meta.wikimedia.org/wiki/Special:MyLanguage/Wikimedia_Foundation_ele…>
when the results are announced.
Regards,
Katie
On behalf of the Elections Committee and Board Selection Working Group
--
Katie Chan
Any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent the view of any organisation the author is associated with or employed by.
Experience is a good school but the fees are high.
- Heinrich Heine
Dear all,
This is a friendly reminder that the nominations for the WikiWomen Task
Force are open until 15th September 2024. If you are passionate about
advancing gender equity in the Wikimedia movement or know someone who would
be a great fit, we encourage you to submit your nomination before the
deadline.
The WikiWomen Task Force will play a critical role in shaping strategies
and initiatives that support women and gender-diverse individuals within
the Wikimedia community. This is a unique opportunity to contribute
directly to the global efforts for equity and inclusion.
Important Links:
-
Learn more about the WikiWomen Task Force: WikiWomen Task Force Overview
<https://meta.wikimedia.org/wiki/Wiki_Women*_Task_Force>
-
Submit your nomination: WikiWomen Task Force Nominations
<https://meta.wikimedia.org/wiki/Wiki_Women*_Task_Force/Nominations>
-
Join the conversation and stay updated: WikiWomen Collective Telegram
Group <https://t.me/+QqjDgjNb7ONkNmE9>
Please note that the staff and volunteers irrespective of their association
with any Wikimedia entity can apply, but the work on the Task force is
truly volunteer!
We encourage you to take part in this important initiative and help ensure
that the task force is diverse, well-represented, and impactful. If you
have any questions or need additional information, please feel free to
reach out.
Best regards,
Nitesh
WikiWomen Support Team
Hi Folks,
If interested learning or get engaged in the discussion about Wikimedia
Hubs (plans, formation, challenges and benefits), I suggest you come to our
Hubs telegram channel.
We have resource persons from CEE, Wikimedia Europe, ESEAP, LAC and many
others! Please come and we share ideas!
The invite link for Wikimedia Hubs group is https://t.me/+cwcxAJg6yrBjOGRk
You can also know more about it on meta-wiki at
https://meta.wikimedia.org/wiki/Hubs/Ongoing
Kind regards,
Butch Bustria
Dear all,
Have you ever left a workshop or event feeling frustrated by the lack of
tangible results, wondering what impact those discussions would actually
have? We have, too. It's not easy to enable collaboration, build
convergence between diverse viewpoints, and create a space for equitable
participation. We believe that many organizations and individuals face
similar challenges when it comes to bringing people together to advance
common goals.
That’s why Wikimedia Deutschland worked with the Movement Ecology
Collective to create a playbook designed to transform workshops into
outcome-driven deliberations <https://www.wikimedia.de/playbook/>, ensuring
your discussions lead to concrete actions and results. The playbook is
also available
on Commons
<https://commons.wikimedia.org/wiki/File:Good_enough_is_perfect_-_Cover.png>
.
What’s the purpose of the playbook?
This playbook draws from the lessons that Wikimedia Deutschland learned
designing international strategic events with Wikimedians for many years.
It offers a how-to guide to designing and facilitating a process for a
group to discuss, assess and decide upon their collective position on a
strategic or governance proposition. This playbook can be adapted to
individual situations within and outside the Wikiverse, to deliver the
intended outcomes and advance your goals. It is particularly suited for
single organizations and large conglomerates of organizations from
different backgrounds developing a common strategy, writing a foundational
document or developing organization-wide values.
How to use the playbook?
The process consists of 15 steps from gathering feedback on the initial
proposition to developing proposals on how to adapt and improve it and to
making a collective decision about these proposals. It is divided into four
sections which can be read in order or separately, depending on what you’re
most interested in – key learnings, setup, process, and tools. Each section
is illustrated with examples from the Wikimedia Summit 2024.
When adopting the process, adapting it is key: to deliver the intended
outcomes, it needs to be tailored to the group who’s using it, its goals
and its needs. This playbook helps asking the right questions and finding
practical answers.
Please get in touch with our team to share any feedback on the playbook,
your experiences with using and adapting the process, or your ideas for
collaborations: govmov(a)wikimedia.de
Have fun experimenting with the playbook!
Nicole and Eva
--
Nicole Ebber
Director Governance & Movement Relations
Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin
Phone: +49 30 577 11 62-0
https://wikimedia.de
Stay up to date! Current news and exciting stories about Wikimedia,
Wikipedia and Free Knowledge in our newsletter (in German):
https://www.wikimedia.de/newsletter/
Wikimedia Deutschland — Gesellschaft zur Förderung Freien Wissens e. V.
Eingetragen im Vereinsregister des Amtsgerichts Charlottenburg, VR 23855 B.
Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin,
Steuernummer 27/029/42207. Geschäftsführende Vorstände: Franziska Heine,
Dr. Christian Humborg.
