In cases of excessive resource usage we have several options. Contact the source, throttle them, or flat out disable access depending on what each case calls for.
I have seen the dev team to this liberally in the past when needed. If any one person or group is exploiting us by using unproportionate amounts of resources thats one thing, if we are just trying to make money by selling access to what we already have thats another. Limiting abusive sources shouldnt be an issue, but as soon as we start selling access we loose sight of our mission.
On Sat, Jan 16, 2016 at 9:11 PM, Denny Vrandecic dvrandecic@wikimedia.org wrote:
I find it rather surprising, but I very much find myself in agreement with most what Andreas Kolbe said on this thread.
To give a bit more thoughts: I am not terribly worried about current crawlers. But currently, and more in the future, I expect us to provide more complex and this expensive APIs: a SPARQL endpoint, parsing APIs, etc. These will be simply expensive to operate. Not for infrequent users - say, to the benefit of us 70,000 editors - but for use cases that involve tens or millions of requests per day. These have the potential of burning a lot of funds to basically support the operations of commercial companies whose mission might or might not be aligned with our.
Is monetizing such use cases really entirely unthinkable? Even under restrictions like the ones suggested by Andreas, or other such restrictions we should discuss? On Jan 16, 2016 3:49 PM, "Risker" risker.wp@gmail.com wrote:
Hmm. The majority of those crawlers are from search engines - the very search engines that keep us in the top 10 of their results (and often in the top 3), thus leading to the usage and donations that we need to survive. If they have to pay, then they might prefer to change their algorithm, or reduce the frequency of scraping (thus also failing to
catch
updates to articles including removal of vandalism in the lead
paragraphs,
which is historically one of the key reasons for frequently crawling the same articles). Those crawlers are what attracts people to our sites, to read, to make donations, to possibly edit. Of course there are lesser crawlers, but they're not really big players.
I'm at a loss to understand why the Wikimedia Foundation should take on
the
costs and indemnities associated with hiring staff to create a for-pay
API
that would have to meet the expectations of a customer (or more than one customer) that hasn't even agreed to pay for access. If they want a specialized API (and we've been given no evidence that they do), let THEM hire the staff, pay them, write the code in an appropriately open-source way, and donate it to the WMF with the understanding that it could be modified as required, and that it will be accessible to everyone.
It is good that the WMF has studied the usage patterns. Could a link be given to the report, please? It's public, correct? This is exactly the point of transparency. If only the WMF has the information, then it
gives
an excuse for the community's comments to be ignored "because they don't know the facts". So let's lay out all the facts on the table, please.
Risker/Anne
On 16 January 2016 at 15:06, Vituzzu vituzzu.wiki@gmail.com wrote:
Thank you for sharing this but, above all, to focus on digging real
data.
IMHO we shouldn't forget our mission, so licenses must be as free as possible. Turning into something "more closed" would definitely deplete
one
of the most valuable source (the open source world) of volunteering we
have.
Crawlers' owner should definitely share our increasing expenses but any kind of agreement with them should include ways to improve our
userbase.
I'm wondering about an agreement with Google (or any other player) to
add
an "edit" button to knowledge graph. Sort of a "knowledge vs. users" agreement.
So, we definitely need a long term strategy which the Foundation will pursue in *negotiating* with anyone who wants a big scale access to
*our
resources* (while access to our knowledge will have no limits, as
usual).
Vito
Il 16/01/2016 19:21, Lila Tretikov ha scritto:
To share some context of the discussion the board had around this -- I don't think the minutes give enough detail. APIs -- as we are freely
and
rapidly creating them today are important, but are not quite at the
core
of the issue we are facing.
Today Wikimedia is the largest internet channel for open free
knowledge
in
the world. But the trends are against us. We have to face them
together.
We have to have the answers on how. The strategic discussion next week
will
help guide us.
Over the last year we looked at the trends in Wikimedia traffic,
internet
as a whole and user behaviors. It took a lot of research. When we
started
the process we have not had solid internal data about unique visitors
or
human vs. crawler usage on the site. For a top 10 website this is a
big
issue; it hurts our ability to make smart decisions. We've learned a
lot.
We found data that supports Leigh's point -- our permissive license supports our core value, we are (I know I am) here for free knowledge.
Yet
it allows others to use the content in ways that truncates, simplifies
and
reduces it. More importantly this type of reuse separates our readers
from
our site, disconnecting readers from our contributors (no edit
buttons)
and ultimately reduces traffic. Is this a problem? I'd like to hear if
people
on this list see it as such. And how we sustain contributions over
time.
Meanwhile estimated half of our hosting is used to support crawlers
that
scan our content. This has an associated cost in infrastructure,
power,
servers, employees to support some well-funded organizations. The
content
is used for a variety of commercial purposes, sometimes having nothing
to
do with putting our contributor's work in front of more readers.
Still,
we
can say this is tangentially supportive of our mission.
As these two trends increase without our intervention, our traffic
decline
will accelerate, our ability to grow editors, content and cover costs
will
decline as well.
The first question on the upcoming consultation next week will be
squarely
on this. Please help us. API conversation is a consequence of this challenge. If we were to build more for reuse: APIs are a good way to
do
so. If we are to somehow incentivize users of SIri to come back to Wikipedia, what would we need to do? Should we improve our site so
more
people come to us directly as the first stop? How do we bring people
into
our world vs. the world of commercial knowledge out there? How do we
fund
this if the people moved to access our content through other
interfaces
(a
trend that has been accelerating)?
Those are the core questions we need to face. We will have to have
some
uncomfortable, honest discussions as we test our hypothesis this year.
The
conversation next week is a good start to prioritize those. Please
join
it.
Lila
On Sat, Jan 16, 2016 at 6:11 AM, Leigh Thelmadatter <
osamadre@hotmail.com
wrote:
If we are concerned about Google taking unfair advantage of Wikipedia,
one
simple solution is to allow content donations with a non-commercial restriction. Right now, the concept of "free" include commercial use.
An
added bonus to this is that we would get a lot more institutional donations of content if we allowed an non-commercial option. My problem with allowing for paying for "premium access" is that we
are
allowing Google to have a priviledged position. There is no way
around
that. What is the impetus behind this proposal? Its not like we are lacking money. And limiting growth of the Foundation is not a bad thing...
at
least not to the community.
To: wikimedia-l@lists.wikimedia.org
From: ricordisamoa@openmailbox.org Date: Sat, 16 Jan 2016 14:13:06 +0100 Subject: Re: [Wikimedia-l] Monetizing Wikimedia APIs
"Imagine a world in which every single human being can freemiumly
share
in the sum of all knowledge." XD
Il 16/01/2016 10:23, Pete Forsyth ha scritto:
> I'm interested to hear some perspectives on the following line of > thinking:
Lisa presented some alternative strategies for revenue needs for the > Foundation, including the possibility of charging for premium
access
> to the
services and APIs, expanding major donor and foundation fundraising, > providing specific services for a fee, or limiting the Wikimedia > Foundation's growth. The Board emphasized the importance of keeping > free
access to the existing APIs and services, keeping operational growth
in
> line with the organization's effectiveness, providing room for > innovation
in the Foundation's activities, and other potential fundraising > strategies.
The Board asked Lila to analyze and develop some of these potential > strategies for further discussion at a Board meeting in 2016. > Source: https://wikimediafoundation.org/wiki/Minutes/2015-11-07 > -Pete[[User:Peteforsyth]] > _______________________________________________ > Wikimedia-l mailing list, guidelines at: > https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
New messages to: Wikimedia-l@lists.wikimedia.org > Unsubscribe:
https://lists.wikimedia.org/mailman/listinfo/wikimedia-l,
> <mailto:wikimedia-l-request@lists.wikimedia.org
?subject=unsubscribe>
Wikimedia-l mailing list, guidelines at:
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
New messages to: Wikimedia-l@lists.wikimedia.org Unsubscribe:
https://lists.wikimedia.org/mailman/listinfo/wikimedia-l
,
mailto:wikimedia-l-request@lists.wikimedia.org?subject=unsubscribe
Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines New messages to: Wikimedia-l@lists.wikimedia.org Unsubscribe:
https://lists.wikimedia.org/mailman/listinfo/wikimedia-l,
mailto:wikimedia-l-request@lists.wikimedia.org?subject=unsubscribe
Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines New messages to: Wikimedia-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, mailto:wikimedia-l-request@lists.wikimedia.org?subject=unsubscribe
Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines New messages to: Wikimedia-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, mailto:wikimedia-l-request@lists.wikimedia.org?subject=unsubscribe
Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines New messages to: Wikimedia-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, mailto:wikimedia-l-request@lists.wikimedia.org?subject=unsubscribe