[Foundation-l] Fundraising Letter Feb 2012

Tue Feb 14 10:10:37 UTC 2012

Indeed Yaroslav

I agree these are concerns and it is likely no solution will ever be 
perfect.

I think the best way to limit (not avoid) such concerns is to make the 
process as transparent as possible. This one is quite a challenge since 
one of the key point in the process is that the name of reviewers are 
kept confidential to allow them to be plain honest in their review.
We may not be able to do that since it is not so much in our culture.

The abuse of the system that I have personally witnessed is one related 
to COI. I have been wondering how strategic committee was populated. My 
memory is that it was represented by three types of people
* representants of major labs (mostly from Paris)
* representants from majors industrial actors (eg, France Telecom)
* individual experts

The desire to have several representants of major industrial actors is 
an explicite choice. The gouvernment wanted to make sure that money 
given to fund research would have industrial outcomes. Even though some 
very fundamental projects end up being funded, putting many industrial 
in the committee is a good way to ensure that most funded projects would 
be "practical" with some real-life outcomes and economical benefits in 
the end.

As for the fact of having experts from major paris-based labs, I think 
it was not necessarily a explicite choice. I think it rather come from 
facility. France is largely focused on its capital (the rest of the 
country is easily considered by the government to be peasants). 
Financially speaking, it is more effective to invite someone working 10 
mn walks from the meeting place than 5 hours train. And last, people 
living in Paris are more likely to lobby ANR to be part of the 
committee. Of course, I am projecting there and it may be more complex, 
but this is the impression I got.

(as Wikimedians, we could well imagine that major chapters will in 
effect have members of the chapters on the FDC just because it is easier 
and more effective than having members of a new-born far-away chapter. 
This will not necessarily be an explicite choice but will simply 
naturally happen.
However, as Wikimedia, we could well imagine that if we want to put a 
specific focus in terms of funding on ... say... Africa and women... it 
would probably make sense to put on the committee people interested in 
Africa and in women. That would be an explicite choice)

The one type of "abuse" I have witnessed was this one.

Imagine the 20 experts of the strategy committee in the room, ranking 
projects list A and list B.

Project P is considered. Unfortunately, one of the experts is not only 
an expert but also involved in Project P or is possibly working in one 
of the company involved in Project P. He announces a COI and leave the 
room. In apparence, all is fine.

In reality, there is another expert in the room, involved in Project J 
with the same COI situation. Before the meeting, the two experts agreed 
that they will mutually support the other expert project. So in the end, 
their COI was poorly handled.

I am not quite sure we can really avoid this situation.

The other point that is slightly annoying is that terms of experts are 
limited to two years.
Consequently, individual experts can only be there 2 years and be done 
with it.
However, big industrial actors (eg, France Telecom) will always have 
somebody to propose so the organization itself will be on that 
committee, regardless of the 2 years terms.
Of course, one might argue that FT needs to be on that committee anyway 
since it is an actor you may not do without, but in this case, the 
choice of limiting terms for individuals is rather dishonnest and not 
helping. Practically speaking, if an individual is a fabulous expert on 
robotics, it is unfortunate to refuse his help after two years based on 
a question of terms limitation if he is still willing and helpful.

(I do not need to make any parallele with some wikimedia chapters right ?)

These were the other points which came to my mind as I read your answers.

On 2/10/12 12:35 PM, Yaroslav M. Blanter wrote:
> Florence, I think you gave a great description of the process, and I agree
> that we should aim at the degree of transparency achieved in it. Actually,
> if I would be in charge of setting such a review panel, this is close to
> how I would do it. However, I also have similar personal experience. I am
> an academic researcher, and I also participated in such panels on many
> occasions - as a panel member, as a referee, and, obviously, as an
> applicant. I do not claim I fully understand all the details, which also
> vary case-to-case, but there are a couple of things I learned from my
> participation, and it is probably good to list them here.
>
> 1) If I am a panel member and I want to kill a proposal (for instance, for
> personal reasons), it is fairly easy. I just need to find an issue which
> experts did not comment on and represent it like a very serious issue. If
> there is another panel member who wants to kill the same proposal, the
> proposal is dead. You do not even need to conspire.
>
> 2) If I want to promote a proposal, it is very difficult, since everybody
> assumes I am somehow personally involved. The most efficient way to promote
> a borderline proposal is to kill the competitors, see above.
>
> 3) If I see that there are panel members with personal agenda, trying to
> kill good proposals, it is usually very difficult to withstand them. The
> majority of the panel do not care, and if a panel member is killing a
> proposal, and another panel member is defending it, the proposal is most
> likely dead.
>
> 4) You mentioned that there are "other issues" like opening a new lab
> which come on top of the proposal quality and can change ranking. My
> experience is that these issues often tend to dominate, and they are the
> easiest to use by panel members who want to change the ranking with respect
> to the expert evaluation.
>
> Even such an  open procedure can be subject to manipulation, and one has
> to be extra careful here. I hope this helps.
>
> Cheers
> Yaroslav
>
> On Thu, 09 Feb 2012 23:52:20 +0100, Florence Devouard<anthere9 at yahoo.com>
> wrote:
>> I wanted to share an experience with regards to a future FDC.
>>
>> During two years, I was a member of the "comité de pilotage" (which I
>> will here translate in "steering committee") of the ANR (National
>> Research Agency in France).
>>
>> The ANR distributes every year about 1000 M€ to support research in
>> France.
>>
>> The ANR programmatic activity is divided in 6 clearly defined themes + 1
>
>> unspecific area. Some themes are further divided for more granularity.
>> For example, I was in the steering committee of CONTINT, which is one of
>
>> the four programs of the main theme "information and communication
>> technologies". My program was about "production and sharing of content
>> and knowledge (creation, edition, search, interface, use, trust,
>> reality, social network, futur of the internet), associated services and
>
>> robotics"
>>
>> Every year, the steering committee of each group define the strategic
>> goals of the year and list keywords to better refine the description of
>> what could be covered or not covered.
>>
>> Then a public call for projects is made. People have 2 months to present
>
>> their project. From memory, the projects received by CONTINT were
>> possibly 200.
>>
>> The projects are peer-reviewed by community members (just as research
>> articles are reviewed by peers) and annotation/recommandation for
>> support or not are provided by the peers. There is no administrative
>> filter at this point.
>>
>> Then a committee constituted of peers review all the projects and their
>> annotation/comments and rank them in three groups. C rejected. B why
>> not. A proposed. Still not administrative filtering at this point.
>>
>> The steering committee, about 20 people made of community members
>> (volunteers) and ANR staff review the A and B. Steering committee is
>> kindly ask to try to keep A projects in A list and B projects in B list.
>
>> However, various considerations will make it so that some projects are
>> pushed up and others pushed down. It may range from "this lab is great,
>> they need funding to continue a long-going research" to "damned, we did
>> not fund any robotic project this year even though it is within our
>> priorities; what would be the best one to push up ?" or "if we push down
>
>> this rather costly project, we could fund these three smaller ones". We
>> may also make recommandation to a project team to rework its budget if
>> we think it was a little bit too costly compared to the impact expected.
>>
>> At the end of the session, we have a brand new list of A followed by B.
>> All projects are ranked. At this point, the budget is only an
>> approximation, so we usually know that all A will be covered but 0 to a
>> few Bs may be.
>>
>> The budget is known slightly later and the exact list of projects funded
>
>> published.
>>
>> How do we make sure what we fund is the best choice ?
>> Not by administrative decision.
>> But by two rounds of independent peer-review who can estimate the
>> quality of the project proposed and the chance for the organisations to
>> do it well.
>> And by a further round through which we know that all projects are
>> interesting and feasible, but will be selected according to strategic
>> goals defined a year earlier.
>>
>> There are also "special calls" if there is a budget to support a highly
>> specific issue. Projects leaders have to decide if their project is
>> related to a "regular theme", or a "white" or a "special call".
>>
>> The idea behind this is also that they have to make the effort to
>> articulate their needs clearly and show what would be the outcome.
>>
>> The staff do not really make decisions. The staff is here to make sure
>> all the process work smoothly, to receive the propositions and make sure
>
>> they fit the basic requirements, to recruit peer for the reviews (upon
>> suggestion made... by steering committee or other peers), to organise
>> the meetings, to publish the results, and so on. Of course, some of them
>
>> do impact the process because of their strong inner knowledge of all the
>
>> actors involved. The staff is overall 30 people.
>>
>> How do we evaluate afterwards that we made the good choice and funded
>> the right ones ?
>> First because as in any funding research, there are some deliverables;
>> Second because once a year, there is a sort of conference where all
>> funded organizations participate and show their results. If an
>> organization does not respect the game or repeatedly fail to produce
>> results, they inevitably fall in the C range at some point in the
>> peer-review process.
>>
>> I present a simplify process, but that generally is it. I am not saying
>> either that it is a perfect system, it is not. But according to what I
>> hear, the system is working fairly well and is not manipulated as much
>> as other funding system may be ;)
>>
>> Last, members of the steering committee may only do a 2 years mandate.
>> No more. There is a due COI agreement to sign and respect as well.
>> Thanks to the various steps in the process and thanks to the good
>> (heavy) work provided by the staff, the workload of volunteers is
>> totally acceptable.
>>
>> Note that this is governement money but the government does not review
>> each proposition. The governement set up a process in which there is
>> enough trust (through the peer-review system and through the themes and
>> keyword methodology) to delegate the decision-making. The program is a 3
>
>> years-program defined by the board of the organization. The majority of
>> the board are high level members of the governement (Education, Budget,
>> Research, Industry etc.). This board does define the program and the
>> dispatching of the budget between the various themes. But the board does
>
>> not make the decision-making with regards to which programs are accepted
>
>> or not.
>>
>> Florence
>>
>>
>
>
> _______________________________________________
> foundation-l mailing list
> foundation-l at lists.wikimedia.org
> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l