[Labs-l] Fall 2016 Tool Labs user survey results

[Bryan Davis]

Between 2016-10-20 and 2016-11-04, the Wikimedia Foundation ran a direct
response user survey of registered Tool Labs users. 175 email recipients
participated in the survey which represents 18% of those who were contacted.

Based on responses to demographic questions, the average* respondent:

   -

   Has used Tool Labs for 1-3 years
   -

   Developed & maintains 1-2 tools
   -

   Spends an hour or less a week using Tool Labs
   -

   Programs using PHP and/or Python
   -

   Does the majority of their work locally
   -

   Uses source control
   -

   Was not a developer or maintainer on Toolserver


[1]: "Average" here means a range of responses covering 50% or more of
responses to the question. This summarization is coarse, but useful as a
broad generalization. Detailed demographic response data is available on
wiki <https://meta.wikimedia.org/wiki/Research:Annual_Tool_Labs_Survey/2016>
.

Qualitative questions:

   -

   87% agree that services have high reliability (up time). Up from 64%
   last year.
   -

   71% agree that it is easy to write code and have it running on Tool
   Labs. Up from 69% last year.
   -

   71% agree that they feel they are supported by the Tool Labs team when
   they contact them via labs-l mailing list, #wikimedia-labs IRC channel, or
   phabricator. Up from 67% last year.
   -

   46% agree that they receive useful information via labs-announce /
   labs-l mailing lists. Down from 53% last year.
   -

   49% disagree that documentation is easy-to-find. Down from 52% last year.
   -

   89% find the support they receive when using Tool Labs as good or better
   than the support they received when using Toolserver. Up from 71% last
   year.
   -

   50% agree that Tool Labs documentation is comprehensive. Up from 46%
   last year.
   -

   48% agree that Tool Labs documentation is clear. Down from 50% last year.


Service usage:

   -

   39% use LabsDB often. Down from 45% last year.
   -

   49% use webservices often. Down from 60% last year.
   -

   42% use cronjobs often. Down from 54% last year.
   -

   66% never use redis. Up from 75% last year.
   -

   48% never use continuous jobs. Down from 41% last year.


The survey included several free form response sections. Survey
participants were told that we would only publicly share their responses or
survey results in aggregate or anonymized form. The free form responses
include comments broadly falling into these categories:

   -

   Documentation (54 comments)
   -

   Workflow (22 comments)
   -

   Stability & performance (11 comments)
   -

   Software versions (8 comments)
   -

   Monitoring (8 comments)
   -

   Kubernetes / PaaS (6 comments)
   -

   Storage (4 comments)
   -

   Support (4 comments)
   -

   Discovery (3 comments)
   -

   Other (15 comments)


Documentation: Over a third of the comments concerned the state of Tool
Labs documentation. Given the survey scores on documentation related
questions this is not surprising. Many suggestions for improvement focus on
step by step guides, task based tutorials, and new user on-boarding guides.
Better coverage of database related issues also received multiple mentions.

Workflow: Workflow related comments included several calls for easier
methods of uploading and downloading files in shared tool accounts. Others
hoped for some form of automated deployment related to GitHub repositories
and web-based editing capabilities. Several people commented on
difficulties learning and using Linux commands with a Windows background.

Stability & performance: Three comments asked for overall stability to be a
focus for the coming year's work. Increased performance of Tool Labs
provided services such as database servers and the hosted tools themselves
also received several mentions.

Software versions: Java 8, PHP 7, and general software and library upgrades
were requested.

Monitoring: Services for monitoring and alerting on webservice uptime, grid
jobs, and log files were included in the comments.

Kubernetes / PaaS: Comments asked for expanded Kubernetes services, Docker
support, and Platform as a Service rollout.

Storage: Storage comments mentioned expanded capacity for database and
large file storage. NFS speed improvements were also mentioned.

Support: Comments on the topic of support called for expanded staffing by
the WMF and additional coverage in the #wikimedia-labs irc channel.

Discovery: Better systems to track and search existing tools are desired.


Thanks to all who participated. Comments are welcome on the meta talk page
<https://meta.wikimedia.org/wiki/Research_talk:Annual_Tool_Labs_Survey/2016>
.

Bryan
-- 
Bryan Davis              Wikimedia Foundation    <bd808 at wikimedia.org>
[[m:User:BDavis_(WMF)]]  Sr Software Engineer            Boise, ID USA
irc: bd808                                        v:415.839.6885 x6855
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/labs-l/attachments/20161127/fb19b27d/attachment-0001.html>