It seems that people are well enough informed by the field studies that our team runs to want us to continue to run them.  In fact, demand has sky-rocketed both within and outside of the Wikimedia Foundation.  You hold a minority opinion that testing software in the field is unnecessary.  Yet, field tests are considered a best-practice and have become a critical part of our strategy for minimizing the disruption (and maximizing the benefits) of software changes.  I don't think that many people would appreciate your proposed strategy of releasing the software and waiting for people to complain.  Given how difficult it is to develop good user-facing software, it's likely that every major deployment would be disruptive if we adopted that strategy.  I can speak for a few disruptions that my research helped prevent and some opportunities that it helped us explore. 

Allow me to share a specific example.  In this study[1], we found that telling anonymous editors to register dropped their productivity by 25%.  Yet we didn't identify substantial issues in user testing.  If we had not run this field experiment, we might have deployed the change thinking that we were improving Wikipedia when we were really driving good editors away.  During the experiment, we received no substantial negative feedback 

For a large collection of field experiments that were used to iterate on Wikimedia software, see:

Really, what I want to say is this: If you want to improve privacy protections, I am your ally.  We're merely disagreeing about whether it is good to assume that DNT means something it wasn't intended to mean or not.  However, when you say that my work has no value, it's hard to talk to you productively because, honestly, I don't think your opinion is well-informed. 



On Thu, Jan 15, 2015 at 7:22 AM, Christian Aistleitner <> wrote:

On Wed, Jan 14, 2015 at 12:07:57PM -0600, Aaron Halfaker wrote:
> For example, not collecting usage data about certain sections of our
> population (e.g. IE10 users where DNT is set by default) means that we
> don't know if our software works for them.

If WMF's main form of QA was through automated usage data collection,
you'd have a point.

But actually, I think WMF is doing better than that.

>From my point of view, a central pillar in QA is “software getting tested”.
That's happening widely across WMF.
Both manually and automated.
It's great already and getting better every day.

And for me the main QA ingredient is listening to feedback from the
users. Besides studies and dog-fooding, WMF's bugtracker is a
testament to that and contains reports that “$X is not working on
browser $Y” or “$X needs to also do $Z”.
And that's really great!

To me, user behaviour data collection is a way to support and assist
the above two. But it is not a requirement when trying to determine
“if our software works for them”.

Users are sending us emails about issues, come to IRC to discuss
issues, file a ticket, or they just tell someone.
All without having their usage data collected.

I am convinced “IE10 users that do not want to unset DNT” are no
exception to that.

Have fun,

P.S.: I for one received bug reports from IE10 users. (But I do not
know whether or they used DNT.)

---- quelltextlich e.U. ---- \\ ---- Christian Aistleitner ----
                           Companies' registry: 360296y in Linz
Christian Aistleitner
Kefermarkterstrasze 6a/3     Email:
4293 Gutau, Austria          Phone:          +43 7946 / 20 5 81
                             Fax:            +43 7946 / 20 5 81

Analytics mailing list