Hello everyone,
I’m happy to announce that the John S. and James L. Knight Foundation has awarded an exploratory grant of $250,000 to the Wikimedia Foundation’s Discovery department [1], in order to conduct research and prototyping to improve how people discover and engage with information on Wikipedia and Wikimedia projects.
The Discovery team has begun six months of research and prototyping, with the goal of building better experiences to help people discover knowledge. You can learn more about the team’s work here http://blog.wikimedia.org/2015/12/23/search-and-discovery-on-wikipedia/.
Our deliverables include:
- User testing and research on current user flows to understand the search and discovery experience - Creation and maintenance of a dashboard of core metrics to use in product development - Research on search relevancy and the possibility of integrating open data sources - Open discussion with the Wikimedia community of volunteer editors - Creation of sample prototypes to showcase discovery possibilities
The need to improve our search experience has long been recognized across the Wikimedia projects. We need better ways to help everyone discover the most relevant, reliable information on Wikipedia and its sister projects. For example, while people can search within one project (like Wikipedia or Wikimedia Commons), they can’t easily search across the different projects. Some people still receive zero results if they search and do not include the right words in a search. There are open data sources that have the potential to improve how people find information, and that should be explored.
We look forward to discussing these projects with communities and anyone with an interest. You can collaborate with the Discovery department in the following ways:
- Subscribe https://lists.wikimedia.org/mailman/listinfo/discovery to the Discovery team’s public mailing list - Read about these projects and others at the MediaWiki Discovery https://www.mediawiki.org/wiki/Wikimedia_Discovery page - Reach out to the Discovery team on their IRC channel: #wikimedia-discovery on Freenode https://freenode.net/.
A press release and blog post will follow shortly, and more information in the form of an FAQ has been posted here [2].
Wes Moran, VP of Product
User: WMoran_(WMF)
[1]: https://www.mediawiki.org/wiki/Wikimedia_Discovery [2]: https://www.mediawiki.org/wiki/Wikimedia_Discovery/KnightFAQ
That's an excellent news. :)
On 6 January 2016 at 19:16, Wes Moran wmoran@wikimedia.org wrote:
Hello everyone,
I’m happy to announce that the John S. and James L. Knight Foundation has awarded an exploratory grant of $250,000 to the Wikimedia Foundation’s Discovery department [1], in order to conduct research and prototyping to improve how people discover and engage with information on Wikipedia and Wikimedia projects.
The Discovery team has begun six months of research and prototyping, with the goal of building better experiences to help people discover knowledge. You can learn more about the team’s work here http://blog.wikimedia.org/2015/12/23/search-and-discovery-on-wikipedia/.
Our deliverables include:
- User testing and research on current user flows to understand the
search and discovery experience
- Creation and maintenance of a dashboard of core metrics to use in
product development
- Research on search relevancy and the possibility of integrating open
data sources
- Open discussion with the Wikimedia community of volunteer editors
- Creation of sample prototypes to showcase discovery possibilities
The need to improve our search experience has long been recognized across the Wikimedia projects. We need better ways to help everyone discover the most relevant, reliable information on Wikipedia and its sister projects. For example, while people can search within one project (like Wikipedia or Wikimedia Commons), they can’t easily search across the different projects. Some people still receive zero results if they search and do not include the right words in a search. There are open data sources that have the potential to improve how people find information, and that should be explored.
We look forward to discussing these projects with communities and anyone with an interest. You can collaborate with the Discovery department in the following ways:
the Discovery team’s public mailing list
- Read about these projects and others at the MediaWiki Discovery
https://www.mediawiki.org/wiki/Wikimedia_Discovery page
- Reach out to the Discovery team on their IRC channel:
#wikimedia-discovery on Freenode https://freenode.net/.
A press release and blog post will follow shortly, and more information in the form of an FAQ has been posted here [2].
Wes Moran, VP of Product
User: WMoran_(WMF)
Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines New messages to: Wikimedia-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, mailto:wikimedia-l-request@lists.wikimedia.org?subject=unsubscribe
Wes Moran wrote:
The Discovery team has begun six months of research and prototyping, with the goal of building better experiences to help people discover knowledge. You can learn more about the team’s work here http://blog.wikimedia.org/2015/12/23/search-and-discovery-on-wikipedia/.
This blog post doesn't mention the Knight Foundation. So we already find ourselves wondering whether the "Search and Discovery" (SAD) team's goals are its own or that of people at the Knight Foundation.
Our deliverables include:
- User testing and research on current user flows to understand the
search and discovery experience
- Creation and maintenance of a dashboard of core metrics to use in
product development
- Research on search relevancy and the possibility of integrating open
data sources
- Open discussion with the Wikimedia community of volunteer editors
- Creation of sample prototypes to showcase discovery possibilities
The need to improve our search experience has long been recognized across the Wikimedia projects. We need better ways to help everyone discover the most relevant, reliable information on Wikipedia and its sister projects. For example, while people can search within one project (like Wikipedia or Wikimedia Commons), they can’t easily search across the different projects.
Yes. My frustration here is that nothing you're proposing will actually address the real problems facing search. Do you think a "dashboard of core metrics" is going to bring about inter-wiki search? If anything, over the next six to eight months, the SAD team will be even more useless than usual as it focuses on fulfilling obligations of a restricted grant.
It would be great to have a search engine that searches all wiki pages on a local wiki. We haven't even reached that point yet. It would be great to have a search engine that is aware of non-text content and can filter media results. We're nowhere near that point yet. But in a few months, maybe your team will have a better understanding of "user flows" and that will help... something. I certainly won't be holding my breath.
MZMcBride
There is a piece by Wes on the website of the Knight Foundation[1] and an additional FAQ on Discovery's "Knowledge Engine" development on MediaWiki.[2]
The Knight Foundation funds many things, and is a past donor to the Wikimedia Foundation--but I'm aware that the Knight Foundation is also Google's partner in "Newsgeist".[3][4]
[1] http://www.knightfoundation.org/blogs/knightblog/2016/1/6/exploring-how-peop... [2] https://www.mediawiki.org/wiki/Wikimedia_Discovery/FAQ [3] http://alldigitocracy.org/what-is-newsgeist/ [4] https://twitter.com/newsgeist
On Wed, Jan 6, 2016 at 3:17 PM, MZMcBride z@mzmcbride.com wrote:
Wes Moran wrote:
The Discovery team has begun six months of research and prototyping, with the goal of building better experiences to help people discover knowledge. You can learn more about the team’s work here <http://blog.wikimedia.org/2015/12/23/search-and-discovery-on-wikipedia/ .
This blog post doesn't mention the Knight Foundation. So we already find ourselves wondering whether the "Search and Discovery" (SAD) team's goals are its own or that of people at the Knight Foundation.
Our deliverables include:
- User testing and research on current user flows to understand the
search and discovery experience
- Creation and maintenance of a dashboard of core metrics to use in
product development
- Research on search relevancy and the possibility of integrating open
data sources
- Open discussion with the Wikimedia community of volunteer editors
- Creation of sample prototypes to showcase discovery possibilities
The need to improve our search experience has long been recognized across the Wikimedia projects. We need better ways to help everyone discover the most relevant, reliable information on Wikipedia and its sister projects. For example, while people can search within one project (like Wikipedia or Wikimedia Commons), they can’t easily search across the different projects.
Yes. My frustration here is that nothing you're proposing will actually address the real problems facing search. Do you think a "dashboard of core metrics" is going to bring about inter-wiki search? If anything, over the next six to eight months, the SAD team will be even more useless than usual as it focuses on fulfilling obligations of a restricted grant.
It would be great to have a search engine that searches all wiki pages on a local wiki. We haven't even reached that point yet. It would be great to have a search engine that is aware of non-text content and can filter media results. We're nowhere near that point yet. But in a few months, maybe your team will have a better understanding of "user flows" and that will help... something. I certainly won't be holding my breath.
MZMcBride
Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines New messages to: Wikimedia-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, mailto:wikimedia-l-request@lists.wikimedia.org?subject=unsubscribe
Would it be possible to share the actual grant agreement with the community, so that people can understand what the Knight Foundation want you to accomplish, and what is going to be built with this money?
Seeing the specifications would help.
I wrote grants and grant reports on behalf of the Wikimedia Foundation from 2009-2011. I was impressed at the time with the emphasis on transparency and other factors around grants, especially restricted grants. I'm beginning to doubt those values have survived, since their main champions at the WMF have all departed.
I just wrote about this in greater depth: http://wikistrategies.net/grant-transparency/
-Pete [[User:Peteforsyth]]
On Wed, Jan 6, 2016 at 2:07 PM, Andreas Kolbe jayen466@gmail.com wrote:
Would it be possible to share the actual grant agreement with the community, so that people can understand what the Knight Foundation want you to accomplish, and what is going to be built with this money?
Seeing the specifications would help. _______________________________________________ Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines New messages to: Wikimedia-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, mailto:wikimedia-l-request@lists.wikimedia.org?subject=unsubscribe
The Knight Foundation website previously announced a $250,000 grant for "Knowledge Engine by Wikipedia", due to run from 1 September 2015 until 31 August 2016, "to advance new models for finding information by supporting stage one development of the Knowledge Engine by Wikipedia, a system for discovering reliable and trustworthy public information on the Internet."
There was nothing on the WMF blog in September 2015, but it seems from this the work on the Knowledge Engine has been ongoing for months.
Could someone please explain why this grant is only being announced now, as though it's something that just happened?
[1] http://www.knightfoundation.org/grants/201551260/ https://archive.is/gZ50b
The Knight Foundation's September 2015 announcement of the $250,000 grant[1] speaks of "supporting stage one development of the Knowledge Engine by Wikipedia". Could we have an explanation of what the other "stages" of this search engine project will be about?
Could we see the grant application?
I am also struck by the fact that the grant is really a very paltry one, compared to the resources the Foundation is investing in this. The MediaWiki page on Discovery[2] lists sixteen people working on this. $250,000 would hardly begin to cover their salaries.
In fact, Risker said as long ago as May last year,[3]
---o0o---
Search and Discovery, a new team, seems to be extraordinarily well-staffed with a disproportionate number of engineers at the same time as other areas seem to be wanting for them. I don't see "fix search" in the Call to Action https://meta.wikimedia.org/wiki/Communications/State_of_the_Wikimedia_Foundation#2015_Call_to_Action document; even if it fell into the heading "Improve technology and execution", this seems like an abnormally large concentration of the top WMF Engineering minds to be focusing on a topic that didn't even rate its own mention in the CtA. More explanation of why Search and Discovery has suddenly become such a major focus is required to assess whether this is appropriate resourcing.
---o0o---
Why isn't there more transparency about this search engine project?
Was this matter ever the subject of disagreements between James and the board members who voted to expel him?
[1] http://www.knightfoundation.org/grants/201551260/ https://archive.is/gZ50b [2] https://www.mediawiki.org/wiki/Wikimedia_Discovery [3] https://meta.wikimedia.org/wiki/Talk:Wikimedia_Foundation_Annual_Plan/2015-1...
On 7 January 2016 at 22:45, Andreas Kolbe jayen466@gmail.com wrote:
...
I am also struck by the fact that the grant is really a very paltry one, compared to the resources the Foundation is investing in this. The MediaWiki page on Discovery[2] lists sixteen people working on this. $250,000 would hardly begin to cover their salaries.
In fact, Risker said as long ago as May last year,[3] ...
I'm not sure what the standard is for grant applications is in the US, but I know locally that is it is extremely rare that allow the funding to be used to pay for salaries and the likes, Although the grant applications I used to have common knowledge were designed to have a physical end goal as per the agreement (example: Replace kitchen cabinets in a Scout den) compared to what will be software changes.
On 1/8/2016 2:01 PM, K. Peachey wrote:
On 7 January 2016 at 22:45, Andreas Kolbe jayen466@gmail.com wrote:
...
I am also struck by the fact that the grant is really a very paltry one, compared to the resources the Foundation is investing in this. The MediaWiki page on Discovery[2] lists sixteen people working on this. $250,000 would hardly begin to cover their salaries.
In fact, Risker said as long ago as May last year,[3] ...
I'm not sure what the standard is for grant applications is in the US, but I know locally that is it is extremely rare that allow the funding to be used to pay for salaries and the likes, Although the grant applications I used to have common knowledge were designed to have a physical end goal as per the agreement (example: Replace kitchen cabinets in a Scout den) compared to what will be software changes.
While it depends on the purpose of the grant, for the deliverables identified in the original post it seems clear that the most natural costs to pay would be salaries in software engineering, broadly speaking. As to the comment about how the grant amount aligns with the size and salary cost of this particular team - in the grantmaking world, it is entirely normal to make awards that pay for only fractions of people's salaries. Let's say you pay for 5% of X's salary and 10% of Y's salary, and as part of the agreement those people are then expected to spend the corresponding percentage of their time dedicated to working on the grant project. I'm sure that the Discovery team has more things to work on than just this one project, but the reason the Foundation would accept this grant is presumably that it overlaps enough with what the organization wants to do anyway.
--Michael Snow
On Sat, Jan 9, 2016 at 3:48 AM, Michael Snow wikipedia@frontier.com wrote:
While it depends on the purpose of the grant, for the deliverables identified in the original post it seems clear that the most natural costs to pay would be salaries in software engineering, broadly speaking. As to the comment about how the grant amount aligns with the size and salary cost of this particular team - in the grantmaking world, it is entirely normal to make awards that pay for only fractions of people's salaries. Let's say you pay for 5% of X's salary and 10% of Y's salary, and as part of the agreement those people are then expected to spend the corresponding percentage of their time dedicated to working on the grant project. I'm sure that the Discovery team has more things to work on than just this one project, but the reason the Foundation would accept this grant is presumably that it overlaps enough with what the organization wants to do anyway.
This is confusing. If you look at the Discovery FAQ[1], it says,
------ *"Knowledge Engine" (KE) was an early term used to describe a number of initiatives that related to search and discovery of content. It was/is not a product and instead was meant to easily reference what the Discovery team was focusing on. We've since stopped using the term as it caused confusion.* ------
So the Knowledge Engine is what the Discovery team is all about. The two terms are described as practically synonymous in the FAQ: the Knowledge Engine term (now deprecated) was a shorthand way of referring to the Discovery team's work. From that, it doesn't sound like the Discovery team has anything else to work on than that.
The Knight Foundation (KF) grant first announced by the Knight Foundation in September last year[1], and announced by the WMF only a few days ago, used the same language:
------ *To advance new models for finding information by supporting stage one development of the Knowledge Engine by Wikipedia, a system for discovering reliable and trustworthy public information on the Internet.* ------
To me at least this means that the KF grant was indeed intended to fund the Discovery team's work, something which the $250,000 named can clearly only do in part, given the amount of personnel involved. The rest of the funding thus must claim a share of the Foundation's own resources.
There is an interesting post by "Eagle" on Wikipediocracy,[3] which I suspect may go some way towards explaining the wider technological background of the Knowledge Engine effort and the strategic decisions underlying it:
------ *Today, a fork is possible because the Mediawiki software is open source and all of the database has been licensed by the contributors. We do not know if the Knowledge Engine software will be open source or how the Knowledge Engine database will be licensed. *
*This is a very interesting trend. A large base of volunteers have gathered a lot of the world's knowledge. The original model, created by Messrs. Wales and Sanger, is that the collection should be delivered in the form of an encyclopedia. *
*Then, IBM's Watson, Apple's Siri, and Microsoft Cortana came forward and created proprietary natural language systems to use Wikipedia (plus other data sources) to provide access to general knowledge in [a] way far more attractive than a text encyclopedia. *
*The question becomes would a fork away from the WMF be possible once Wikipedia shifts away from a Mediawiki based "encyclopedia?" Even if the entire Wikipedia community shifted to working on the fo[r]k, would the user interface of the Knowledge Engine keep the user traffic (and the Google juice) with the WMF after the shift? *
*Conversely, if the world of information seekers is shifting away from a text encyclopedia model to access information, does anyone seriously believe that the WMF technical staff (even if enhanced by the Knight Foundation grant) can compete with the best that IBM, Apple and Microsoft will continue to develop? *
*This is a very serious problem that Mr. Wales, Lila, Doc James and seven others can not solve by themselves locked into a secret sound-proof chamber.* ------
While I am pretty sure that any Knowledge Engine software developed by the Wikimedia Foundation will be open source (if I am wrong on this, please put me right!), and am not proposing to initiate a discussion about forking here, some of what Eagle says about the wider technological background feels like it might be very relevant to the motivations underlying the Knowledge Engine (or "Discovery") project.
People looking up Wikipedia on their smartphone in the pub will indeed not read a long encyclopedia article. They just want a snippet of information. But does that mean that, given developments like the Knowledge Graph, Siri, Watson etc., the writing is on the wall for Wikipedia's -- presently at least -- immensely popular and much-loved encyclopedia format?
I don't understand what happened between the first announcement of the grant by the KF in September, and the renewed announcement of it now. What the KF says in its January 6, 2016, announcement[4] has morphed somewhat from the earlier announcement of 12 months' support for a Knowledge Engine project designed to enable the public to discover "reliable and trustworthy public information on the internet."
What the KF is now talking about is funding
------ *exploratory research and prototyping to improve how people find and engage with knowledge on Wikimedia projects. Knight’s support will fund six months of investigation around search and browsing on the projects, with the ultimate goal of building better experiences to help people discover knowledge* ------
Is this "exploratory research" -- now shortened from twelve to six months -- what the Wikimedia Foundation pitched for?
Or was the pitch for the more ambitious plans described in the "Discovery Year 0-1-2" presentation[5] and the Discovery FAQ?[1]
What further stages are envisaged?
Jimmy Wales said on his user talk page yesterday[6] that, in his opinion, and pending confirmation that there are no contractual reasons standing in the way of this, the grant letter should be published on Meta, and that "it would be best to clear the air around that completely as soon as possible."
For once, I agree with him. To clear the air completely, the grant application documentation should be made public as well.
Please draw the public and the community into your confidence on this, and work with the community rather than in isolation from it.
Andreas
[1] https://www.mediawiki.org/wiki/Wikimedia_Discovery/FAQ [2] http://www.knightfoundation.org/grants/201551260/ [3] http://wikipediocracy.com/forum/viewtopic.php?p=169310#p169310 [4] http://www.knightfoundation.org/blogs/knightblog/2016/1/6/exploring-how-peop... [5] https://commons.wikimedia.org/wiki/File:Discovery_Year_0-1-2.pdf [6] https://en.wikipedia.org/w/index.php?title=User_talk:Jimbo_Wales&diff=69...
wikimedia-l@lists.wikimedia.org