Hi,
On the second day of the Wikimedia Developer Summit (January 10) there will be a Q&A session with Victoria Coleman (Wikimedia Foundation CTO) and Wes Moran (VP of Product). It is a plenary session and it will be video-streamed.
The questions for this session are being crowdsourced at http://www.allourideas.org/wikidev17-product-technology-questions. Anyone can propose questions and vote, anonymously, as many times as you want. At the moment, we have 25 questions and 451 votes.
An important technical detail: questions posted later have also good chances to make it to the top of the list as long as new voters select them. The ranking is made out of comparisons between questions, not accumulation of votes. For instance, the current top question is in fact one of the last that has been submitted so far.
Why posting or voting a good question? One obvious reason is to encourage the Foundation's Technology and Product top managers to bring a good answer in a public session with minutes taken and video recording. :) Beyond that, if the ranking of questions makes sense and is backed by participation numbers, it has a serious chance to influence plans and discussions beyond the Summit.
The current ranking does make sense, but maybe you could help covering more areas, other perspectives?
1. How do we deal with the lack of maintainers for all Wikimedia deployed code? 2. Do we have a plan to bring our developer documentation to the level of a top Internet website, a major free software project? 3. For WMF dev teams, what is the right balance between pushing own work versus seeking and supporting volunteer contributors? 4. During the next year or so, what balance do you think we should strike between new projects and technical debt? 5. When are we going to work on a modern talk pages system for good? 6. Whose responsibility is to assure that all MediaWiki core components and the extensions deployed in Wikimedia have active maintainers? 7. How important is to have a well maintained and well promoted catalog of tools, apps, gadgets, bots, templates, extensions...? 8. Will MediaWiki ever become easier to install and manage? (e.g. plugin manager à la Wordpress). How much do we care about enterprise users? 9. What should be the role of the Architecture Committee in WMF planning (priorities, goals, resources...) and are we there yet? 10. In addition to Community Tech, should the other WMF Product teams prioritize their work taking into account the Community Wishlist results?
The full list: http://www.allourideas.org/wikidev17-product-technology-questions/results
On Tue, Dec 20, 2016 at 12:45 AM, Quim Gil qgil@wikimedia.org wrote:
The questions for this session are being crowdsourced at http://www.allourideas.org/wikidev17-product-technology-questions. Anyone can propose questions and vote, anonymously, as many times as you want. At the moment, we have 25 questions and 451 votes.
An important technical detail: questions posted later have also good chances to make it to the top of the list as long as new voters select them. The ranking is made out of comparisons between questions, not accumulation of votes. For instance, the current top question is in fact one of the last that has been submitted so far.
Right now the top question has a score of 70 based on 88 votes; the second question has a score of 67 based on 1 vote. (This is not some super-rare accident, either: number 8 and 9 on the popularity list both have 4 votes.) I argued that All Our Ideas is too experimental to be relied on back when it was considered as the voting tool for an early iteration of what ended up being the Community Tech Wishlist, and I still think that's the case.
The way their voting system works is that they assume each idea has some appeal (an arbitrary real number) for each voter, the appeals for a given idea are normally distributed, and when a voter is shown a question pair, their probability of voting a given way is a certain function of the difference in appeals. They then use various statistical methods to come up with random values for the appeals which match the observed votes, and using those values they can calculate the probability for each question that a randomly selected voter would prefer that question to a randomly selected alternative; those probabilities are used to score the questions.
That means that the scores can be heavily underspecified (ie. mostly result from the random numbers generated by their algorithm and not actual votes) for some questions; this is especially true for recently submitted questions, which have a very small number of votes, so they will basically get a random position in the ranking. As far as I can see, the journal article [1] where they present their method doesn't discuss this problem at all. This is not terribly useful as a real-world ranking model IMO, so I hope that 1) there will be some human oversight when evaluating the results, and 2) that we don't intend to use this system for any voting that actually matters (getting weirdly prioritized results for a Q&A session is, of course, not a huge deal).
[1] http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0123483
Hi,
http://www.allourideas.org/wikidev17-product-technology-questions is closed for new questions but still open for votes until the end of Thursday. A couple of new questions were added in the past days. Please contribute a couple of minutes submitting some more votes!
The results so far: http://www.allourideas.org/wikidev17-product-technology-questions/results
On Sat, Dec 31, 2016 at 3:26 AM, Gergo Tisza gtisza@wikimedia.org wrote:
On Tue, Dec 20, 2016 at 12:45 AM, Quim Gil qgil@wikimedia.org wrote:
The questions for this session are being crowdsourced at http://www.allourideas.org/wikidev17-product-technology-questions.
Anyone
can propose questions and vote, anonymously, as many times as you want.
At
the moment, we have 25 questions and 451 votes.
An important technical detail: questions posted later have also good chances to make it to the top of the list as long as new voters select them. The ranking is made out of comparisons between questions, not accumulation of votes. For instance, the current top question is in fact one of the last that has been submitted so far.
Right now the top question has a score of 70 based on 88 votes; the second question has a score of 67 based on 1 vote. (This is not some super-rare accident, either: number 8 and 9 on the popularity list both have 4 votes.)
Right now the top 10 have questions that have received as low as 8-15 votes and as high as 80-101. These numbers will be more balanced if/when more people vote this week.
I will not attempt to make a big fuss over participation theories, but IMHO Wikimedia processes are quite biased towards What Is Said By Who Talks First. This is a humble and harmless experiment in a different direction. While seeing a question with eight votes among the top 10 defies the traditional democracy paradigm, it also means that an idea that came later had any chance over those who were submitted early on.
At the end what counts is the final result of the experiment. Regardless of the numbers, I think the current list makes sense, and I in fact it has been making sense all along since its second day or so.
That means that the scores can be heavily underspecified (ie. mostly result from the random numbers generated by their algorithm and not actual votes)
Well, I am not sure. If a question with eight votes is among the top ten, it probably means that it has been systematically preferred over other questions scoring similarly high.
Currently the very last question has only two votes, which means that the same algorithm that can put new questions in the top segment can also bury them down.
The solution to these potential biases is simple: more opinions submitted by more people, which is the basis of any healthy group participation.
Gergo, I am not saying you are wrong (you have clearly done more research than myself). I am just saying that I don't think choosing this tool for this purpose was a wrong idea either. :)
After 25 days, 40 questions, and 2480 votes, the results are... https://www.mediawiki.org/wiki/Wikimedia_Developer_Summit/2017/Program/Wikim...
Thank you to all participants. This experiment has been very interesting so far. The Q&A session will happen next Tuesday at 9:30am Pacific, and there will be a live-broadcast.
https://www.mediawiki.org/wiki/Wikimedia_Developer_Summit/2017/Program#Tuesd...
On Tue, Jan 3, 2017 at 10:05 AM, Quim Gil qgil@wikimedia.org wrote:
Hi,
http://www.allourideas.org/wikidev17-product-technology-questions is closed for new questions but still open for votes until the end of Thursday. A couple of new questions were added in the past days. Please contribute a couple of minutes submitting some more votes!
The results so far: http://www.allourideas.org/wikidev17-product- technology-questions/results
On Sat, Dec 31, 2016 at 3:26 AM, Gergo Tisza gtisza@wikimedia.org wrote:
On Tue, Dec 20, 2016 at 12:45 AM, Quim Gil qgil@wikimedia.org wrote:
The questions for this session are being crowdsourced at http://www.allourideas.org/wikidev17-product-technology-questions.
Anyone
can propose questions and vote, anonymously, as many times as you want.
At
the moment, we have 25 questions and 451 votes.
An important technical detail: questions posted later have also good chances to make it to the top of the list as long as new voters select them. The ranking is made out of comparisons between questions, not accumulation of votes. For instance, the current top question is in fact one of the last that has been submitted so far.
Right now the top question has a score of 70 based on 88 votes; the second question has a score of 67 based on 1 vote. (This is not some super-rare accident, either: number 8 and 9 on the popularity list both have 4 votes.)
Right now the top 10 have questions that have received as low as 8-15 votes and as high as 80-101. These numbers will be more balanced if/when more people vote this week.
I will not attempt to make a big fuss over participation theories, but IMHO Wikimedia processes are quite biased towards What Is Said By Who Talks First. This is a humble and harmless experiment in a different direction. While seeing a question with eight votes among the top 10 defies the traditional democracy paradigm, it also means that an idea that came later had any chance over those who were submitted early on.
At the end what counts is the final result of the experiment. Regardless of the numbers, I think the current list makes sense, and I in fact it has been making sense all along since its second day or so.
That means that the scores can be heavily underspecified (ie. mostly result from the random numbers generated by their algorithm and not actual votes)
Well, I am not sure. If a question with eight votes is among the top ten, it probably means that it has been systematically preferred over other questions scoring similarly high.
Currently the very last question has only two votes, which means that the same algorithm that can put new questions in the top segment can also bury them down.
The solution to these potential biases is simple: more opinions submitted by more people, which is the basis of any healthy group participation.
Gergo, I am not saying you are wrong (you have clearly done more research than myself). I am just saying that I don't think choosing this tool for this purpose was a wrong idea either. :)
-- Quim Gil Engineering Community Manager @ Wikimedia Foundation http://www.mediawiki.org/wiki/User:Qgil
wikitech-l@lists.wikimedia.org