[Wikimedia-l] Effectiveness of meetings (Was: Affcom ...)

Tue May 14 12:54:49 UTC 2013

Theoratically every penny spent should be measured wheter it was spent
well, or was wasted, including salaries. Ideally you could find out all the
direct and indirect investment in a given program. Since the latter is
extremely hard (or impossible) to measure, it is always an estimated extra
cost.

The emphasis here is not on the programs a meetup may trigger, but the
meetup itself; its effectiveness. (it is a program itself)

Regarding evalulation of programs through a meetup's perspective (if I
understood your words correctly), there is no point going any deeper than
the name of it, the type of it, and a note that it was"done", "in progress"
, "failed" or "postponed" as of the date of publishing. Details should be
carefully adjusted for all questions you wish to answer.

90% of evaluating an event is actually done through surveys collected from
participants on site (!) and after a given period (somewhere between three
to six month),  by evaluating the answers and feedbacks you collect
_proactively_ (a.k.a. "by asking") The rest 10% is an evaluation summary of
the catering, the venue and the executing staff. Six month later a summary
was published and an evaluation meetup was held before we started
organizing the next similar event (what's content was heavily influenced by
that that report).

Cheers,
Balázs

2013/5/14 Lodewijk <lodewijk at effeietsanders.org>

> I totally understand (or rather, I think I do). However, just like with any
> project, also such calculations would be subject to cost/benefit ratios.
> What information do you expect to win, and how much overhead does it
> create? It sounds great to be able to put a number on things, but up to a
> certain size, things will be mostly intangible anyway (enthusiasm,
> inspiration etc.) so you will have little valuable output of such
> measurements, while the costs are relatively high (they need to learn speak
> your measure language, they need to understand concepts, document
> thoroughly, measure etc.) and for small events that seems unreasonable to
> me to expect it from them - such as with the article writing contests in
> 'new' countries.
>
> However, this is of course different for large international events with
> many participants. Anyway, I am glad to hear you're not trying to compare
> different categories of events with each other.
>
> Lodewijk
>
>
> 2013/5/14 Balázs Viczián <balazs.viczian at wikimedia.hu>
>
> > I see a few misunderstanding here: I do not wish to compare them to each
> > other, but to the previous one(s). At least I hope this is what I've
> > written because this is what I was ment. So compare GLAM 2013 to GLAM
> 2012,
> > and GLAM 2011, ..., compare Brussels meetup with previous similar
> thematic
> > meetups and so forth. Create a breakdown about the costs and the
> benefits.
> > Aimed at GLAM? then the main thing to measue how did it boost GLAM
> > cooperations. Is there any initiative that had been picked up or created
> > upon the inspiration they got there? If yes, how much? What types? What
> are
> > their outcomes? Any other side goals (like international cooperations -
> > like a new WLM participant who decided to join after discussions there)?
> > Any other unexpected benefits? (like an out of the blue content donation
> > from a participating G,L,A or M representative) etc. Easy to identify a
> > dozen questions, plus if you start thinking it over, you'll find a dozen
> > more that are as well pretty important, just not that much visible or in
> > front. Count them and divide the total number with the total costs, and
> > you'll get how much was invested in a single initiative there.
> >
> > If an idea was brought home by a chapter from that event (or the local
> > ideas have been influenced by it), note even if locally it would cost
> > almost nothing, on an absolute level, there is this cost/benefit ratio,
> > both the chapter's and the event's, what you have to further divide it
> > locally amongst the number of projects. If an example helps you, lets say
> > you brought home 2 article writing contest ideas, both of them conducted
> > and did cost nothing, except a few merchandise (total of 200USD) your
> > travel and stay there should be added to the total costs, since without
> it,
> > these would likely never been conducted at you. Hence the absolute
> numbers
> > would include your flight and stay in London, divided equally in between
> > the projects that meeting inspired. Got me?
> >
> > Balázs
> >
> >
> > 2013/5/14 Lodewijk <lodewijk at effeietsanders.org>
> >
> > > I think this is very true, and we could perhaps improve our procedures
> > and
> > > documentation in many ways. However, I do think it is important to
> > realize
> > > that you're comparing apples with oranges here. The meeting in Brussels
> > for
> > > example was on the topic of influencing legislative processes - a topic
> > by
> > > definition long term and hard to measure. The goals there would be
> better
> > > (copyright) legislation than without these initiatives - and rather
> > > strategic. The bootcamp in DC was intended as a boost for volunteers,
> > where
> > > they would get a lot of information and experience in a short time:
> > > capacity building. This is also long term, but much better to measure.
> > And
> > > finally there's the hackathon, which is much more short-term focused
> (at
> > > least partially), and will have very concrete results (but probably
> zero
> > > measured in new articles created).
> > >
> > > What I think is most important, is that for every meeting we have, we
> > > identify desired outcomes. These outcomes can be very tangible, or
> > > intangible - but they should always be defined, and the agenda should
> be
> > > built around it.
> > > Based on these desired outcomes, you can estimate up front if the
> meeting
> > > is likely to be 'worth it'. If not, you can reduce the costs, increase
> > the
> > > outcomes (different setup) or cancel all together.
> > > Finally, after the meeting a report (private or public - there should
> > > always be a report or documentation) should be produced and if possible
> > > published. In that report you should always return to these desired
> > > outcomes, and see if they were met - and how/why not.
> > >
> > > But even then on the strategy side of things meetings are always hard
> to
> > > estimate.
> > >
> > > Of course you're totally right that there are expectations towards
> > > participants of meetings. Not only because of the money invested, but
> > also
> > > because of the attention of the other participants. I definitely have
> > been
> > > in meetings where people literally fell asleep or played games during
> the
> > > meeting - and that is simply insulting to the other people in the room.
> > So
> > > yeah, if that is your mindset, perhaps it is better not to go at all.
> But
> > > then I am assuming good faith, and think that everyone will be going to
> > > meetings with the best of intentions, and not simply to play tourist.
> > >
> > > Lodewijk
> > >
> > > 2013/5/14 Balázs Viczián <balazs.viczian at wikimedia.hu>
> > >
> > > > 40k for a single meetup can be justified if the results worth that
> much
> > > > money. (I've already argued about this regarding Milan chapters
> meetup
> > > > btw). Providing a "more than basic" travel and accomodation can be a
> > way
> > > of
> > > > appreciation as well for their work, what they do as volunteers, that
> > > > should be calculated into the costs.
> > > >
> > > > Ain't these intrenational events' (not just this meetup's, but all
> > > events')
> > > > "success ratio" being measured by some way already?
> > > >
> > > > The cost/benefit ratio [1] is a pretty basic (and extreme important)
> > > thing
> > > > we like to calculate with here in Hungary about all of our
> activities.
> > > For
> > > > example the number of articles created divided with the total costs
> of
> > > the
> > > > article writing contest they've been created within gives a number we
> > can
> > > > work a lot with to improve cost-effectiveness. Seemingly very few
> > > chapters
> > > > doing anything similar (or not in a visible way)
> > > >
> > > > AffCom already measures itself in some ways in their reports, but
> > > regarding
> > > > other meetups, I've barely seen at least a basic followup or
> aftercare
> > > and
> > > > especially not a detailed overall (measurement) report what is
> usually
> > > /at
> > > > least in those commercial events I was involved with/ being published
> > > after
> > > > about six month of the last day of the event, and gives a detailed
> > > summary
> > > > of its pros and cons, dos and don'ts, successes and fails, overall
> > impact
> > > > (upon proactively collected feedbacks), etc.
> > > >
> > > > There were plenty of international events already this year (Brussels
> > > > meeting about EU policies, Milan, London glam, Amsterdam hackathon
> that
> > > are
> > > > coming in my mind right now from 2013, and this is just the first 4
> > month
> > > > (only 1/3rd of the year) and not the full list!) most of them with no
> > or
> > > > very low visible results yet (ok they need some time to evolve, but
> > > > regarding events in 2012, I barely read anything to remember nor any
> > > > followups or summares, reports). Compared to the "GLAM camp" in the
> US
> > > > recently, it seems definitely true. The latter looks like a very good
> > > > example of a beneficial meetup, having a lot of potential and it
> seems
> > > > there is a chance that it will be followed up by WMDC and other
> > > > participating parties well after the event. Why to have a long term
> > > > followup? To give a definite answer wheter those potentials actually
> > > > resulted in anything at all (was there any "real benefit") or it was
> > > just a
> > > > very good mooded, fun and positive, but totally fruitless event
> (wasted
> > > > money from the movement's POV)
> > > >
> > > > I see compared to 2012 costs going up without any visible rise in
> > > > effectiveness or more worse, a decline in it. This is solely based
> upon
> > > > what I can (or can not) read about them on meta and other places,
> like
> > > the
> > > > comments here. Wikimania 2014 was the first event ever where this
> thing
> > > was
> > > > taken seriously, but rather on the cost cutting side, than on the
> > > > effectiveness improving side
> > > >
> > > > It would be great to see detailed measurements of these events, like
> > how
> > > > many new projects or international cooperations (or whatever it aims)
> > > were
> > > > boosted/inspired by the given thematic meetup up until the next
> similar
> > > > meetup. If that number is X, while the costs were Y, and X/Y does not
> > > look
> > > > good,  than you can start thinking how can you improve X without
> > > expanding
> > > > Y (or even lowering it) to get a much friendly ratio, thus creating
> an
> > > even
> > > > more fruitful (better quality) event next time. The best would be a
> > > > detailed breakdown, like main goals, side goals, unexpected or
> "extra"
> > > > things that that meeting had inspired/boosted/hosted/etc.
> > > >
> > > > Note, there ain't no such thing as free lunch [2]
> > > >
> > > > Cheers,
> > > > Balázs
> > > >
> > > > PS: WMHU has 68k budget for 2013.
> > > >
> > > > [1] http://en.wikipedia.org/wiki/Benefit%E2%80%93cost_ratio
> > > > [2]
> > > >
> > http://en.wikipedia.org/wiki/There_ain%27t_no_such_thing_as_a_free_lunch
> > > >
> > > >
> > > >
> > > > 2013/5/14 Lodewijk <lodewijk at effeietsanders.org>
> > > >
> > > > > I don't think it is trivial either, and having a discussion is
> fine.
> > > > > However, the bigger discussion is perhaps more relevant - because
> > > AffCom
> > > > is
> > > > > simply following WMF policy here, which is in place for many
> > employees,
> > > > > board members (and others? not sure).
> > > > >
> > > > > So I think there are two relevant discussions to be had: First of
> > all,
> > > > > whether there should be an AffCom meeting at all. Fair question of
> > > > course.
> > > > > (analogous you could wonder if certain chapters should really send
> a
> > > > > representative to Wikimania from their budget, whether certain
> > > employees
> > > > > really should be there and whether all chapters should be present
> at
> > > the
> > > > > Chapters meeting - all fair discussions to be had in their time,
> > too).
> > > > The
> > > > > second relevant question is, in my opinion, whether the WMF travel
> > > policy
> > > > > is good, proportional etc. This policy has mostly received
> criticism
> > > from
> > > > > two sides - some think it is too elaborate, and WMF should get
> 'less
> > > > > luxury' (again fair discussion, but we should then focus on the
> whole
> > > > > question) - another criticism is that it should be equalized for
> all
> > > > > Wikimedia-sponsored trips, including individual engagement grants,
> > > trips
> > > > to
> > > > > the chapters meeting etc. When this question was brought up last
> > time,
> > > I
> > > > > believe it was Sue who mentioned that this would simply result in
> > much
> > > > less
> > > > > travel and participation. Again, a fair question.
> > > > >
> > > > > Personally, I feel that WMF Committee (and board) members should
> not
> > be
> > > > > treated to a lower standard than staff members, simply because they
> > are
> > > > not
> > > > > being paid for their work. But maybe I'm the only one in thát
> opinion
> > > > > though...
> > > > >
> > > > > Lodewijk
> > > > >
> > > > >
> > > > > 2013/5/14 Andrea Zanni <zanni.andrea84 at gmail.com>
> > > > >
> > > > > > On Tue, May 14, 2013 at 12:30 AM, David Gerard <
> dgerard at gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > https://en.wikipedia.org/wiki/Parkinson%27s_law_of_triviality
> > > > > >
> > > > > >
> > > > > > I don't really think this is a triviality.
> > > > > > Cents and single dollars of chapters are weighted and analyzed,
> > > > > > FDC and GACs nitpicks and their goal is to assure that donor
> moneys
> > > is
> > > > > not
> > > > > > wasted.
> > > > > > We all know that there are issues on that and a lot of chapters
> had
> > > to
> > > > > work
> > > > > > hard to be able to draw their budget, and receive their money.
> > > > > >
> > > > > > 40'000 $ for Wikimania travels are a lot, especially compared to
> > the
> > > > > travel
> > > > > > budget of chapters. Period.
> > > > > > I would assume that WMF and the Board budgets should be
> reasonably
> > > > > > proportional to chapters ones.
> > > > > > Maybe I'm the only one, though.
> > > > > > _______________________________________________
> > > > > > Wikimedia-l mailing list
> > > > > > Wikimedia-l at lists.wikimedia.org
> > > > > > Unsubscribe:
> > > https://lists.wikimedia.org/mailman/listinfo/wikimedia-l
> > > > > >
> > > > > _______________________________________________
> > > > > Wikimedia-l mailing list
> > > > > Wikimedia-l at lists.wikimedia.org
> > > > > Unsubscribe:
> > https://lists.wikimedia.org/mailman/listinfo/wikimedia-l
> > > > >
> > > > _______________________________________________
> > > > Wikimedia-l mailing list
> > > > Wikimedia-l at lists.wikimedia.org
> > > > Unsubscribe:
> https://lists.wikimedia.org/mailman/listinfo/wikimedia-l
> > > >
> > > _______________________________________________
> > > Wikimedia-l mailing list
> > > Wikimedia-l at lists.wikimedia.org
> > > Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l
> > >
> > _______________________________________________
> > Wikimedia-l mailing list
> > Wikimedia-l at lists.wikimedia.org
> > Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l
> >
> _______________________________________________
> Wikimedia-l mailing list
> Wikimedia-l at lists.wikimedia.org
> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l
>