On Thu, Aug 21, 2014 at 5:21 PM, Tomasz Finc <tfinc@wikimedia.org> wrote:

On Thu, Aug 21, 2014 at 4:29 PM, Kristen Lans <klans@wikimedia.org> wrote:
> *Explore paid testing services [Tomasz]

The background on this is that we've only been able to amass a small
community of iOS testers who while being extremely helpful have not
reached the consistent testing that we need. We'll still continue to
crowd source as much as we can but in the mean time we'd like to
explore some paid services and I'm curious to hear from the smart
people on this list about any they would recommend.

Testing by volunteers is hard, and agile development makes it harder for volunteer testers to follow projects moving at speed. I have a friend working on this at Mozilla, and they have similar issues even with significantly more investment of resources than WMF has.

Of the crowd-sourced testing services, I like uTest the most http://www.utest.com/. They are one of the oldest such services and they seem ethical. While I have not employed them for any projects, I have dealt with them directly from time to time in the past.

As with any such project, defining a proper focus for the test effort is key to getting good results.

-Chris