[0]: https://meta.wikimedia.org/wiki/Research:Annual_Tool_Labs_Survey
On Wed, May 4, 2016 at 7:57 PM, Bryan Davis bd808@wikimedia.org wrote:
Apologies for that abrupt initial message, that was a great example of hitting the wrong key in a mail client. :)
Between 2015-09-25 and 2015-10-08, the Wikimedia Foundation ran a direct response user survey of registered Tool Labs users. 106 users responded to the survey.
Based on responses to demographic questions, the average[1] respondent: * Has used Tool Labs for 1-3 years * Developed & maintains 1-3 tools * Spends an hour or less a week using Tool Labs * Programs using PHP and/or Python * Does the majority of their work locally * Uses source control * Was not a developer or maintainer on Toolserver
[1]: "Average" here means a range of responses covering 50% or more of responses to the question. This summarization is coarse, but useful as a broad generalization. Detailed demographic response data will be made available on wiki.
Qualitative questions: 64% agree that services have high reliability (up time). 69% agree that it is easy to write code and have it running on Tool Labs. 67% agree that they feel they are supported by the Tool Labs team when they contact them via labs-l mailing list, #wikimedia-labs IRC channel, or phabricator. 53% agree that they receive useful information via labs-announce / labs-l mailing lists. 52% disagree that documentation is easy-to-find. 71% find the support they receive when using Tool Labs as good or better than the support they received when using Toolserver. 50% disagree that Tool Labs documentation is comprehensive. 50% agree that Tool Labs documentation is clear.
Service usage: 45% use LabsDB often. 60% use webservices often. 54% use cronjobs often. 75% never use redis. 41% never use continuous jobs .
The survey included several free form response sections. Survey participants were told that we would only publicly share their responses or survey results in aggregate or anonymized form. The freeform responses include comments broadly falling into these categories:
Documentation (33 comments) Stability and performance (18 comments) Version control and Deployment (14 comments) Logs, Metrics, and Monitoring (12 comments) Package management (10 comments) SGE (8 comments) Database (7 comments) Account/Tool creation (5 comments) SSH (5 comments) Other (11 comments)
Additonal details are available on meta [2].
[2]: https://meta.wikimedia.org/wiki/Research:Annual_Tool_Labs_Survey
Bryan
Hi Bryan,
Thanks for the report. With this information in hand, what follow up is planned?
Pine On May 4, 2016 19:01, "Bryan Davis" bd808@wikimedia.org wrote:
On Wed, May 4, 2016 at 7:57 PM, Bryan Davis bd808@wikimedia.org wrote:
Apologies for that abrupt initial message, that was a great example of hitting the wrong key in a mail client. :)
Between 2015-09-25 and 2015-10-08, the Wikimedia Foundation ran a direct response user survey of registered Tool Labs users. 106 users responded to the survey.
Based on responses to demographic questions, the average[1] respondent:
- Has used Tool Labs for 1-3 years
- Developed & maintains 1-3 tools
- Spends an hour or less a week using Tool Labs
- Programs using PHP and/or Python
- Does the majority of their work locally
- Uses source control
- Was not a developer or maintainer on Toolserver
[1]: "Average" here means a range of responses covering 50% or more of responses to the question. This summarization is coarse, but useful as a broad generalization. Detailed demographic response data will be made available on wiki.
Qualitative questions: 64% agree that services have high reliability (up time). 69% agree that it is easy to write code and have it running on Tool Labs. 67% agree that they feel they are supported by the Tool Labs team when they contact them via labs-l mailing list, #wikimedia-labs IRC channel, or phabricator. 53% agree that they receive useful information via labs-announce / labs-l mailing lists. 52% disagree that documentation is easy-to-find. 71% find the support they receive when using Tool Labs as good or better than the support they received when using Toolserver. 50% disagree that Tool Labs documentation is comprehensive. 50% agree that Tool Labs documentation is clear.
Service usage: 45% use LabsDB often. 60% use webservices often. 54% use cronjobs often. 75% never use redis. 41% never use continuous jobs .
The survey included several free form response sections. Survey participants were told that we would only publicly share their responses or survey results in aggregate or anonymized form. The freeform responses include comments broadly falling into these categories:
Documentation (33 comments) Stability and performance (18 comments) Version control and Deployment (14 comments) Logs, Metrics, and Monitoring (12 comments) Package management (10 comments) SGE (8 comments) Database (7 comments) Account/Tool creation (5 comments) SSH (5 comments) Other (11 comments)
Additonal details are available on meta [2].
Bryan
Bryan Davis Wikimedia Foundation bd808@wikimedia.org [[m:User:BDavis_(WMF)]] Sr Software Engineer Boise, ID USA irc: bd808 v:415.839.6885 x6855
Labs-l mailing list Labs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/labs-l
On Wed, May 4, 2016 at 8:35 PM, Pine W wiki.pine@gmail.com wrote:
Hi Bryan,
Thanks for the report. With this information in hand, what follow up is planned?
The primary purpose of the study was to set a baseline of qualitative ("how are we doing") responses. These will be used to gauge increase/decrease in satisfaction in comparison to future surveys. Tool Labs staff get regular feedback from people who are either extremely happy or extremely unhappy with services, but a broad survey like this is helpful to determine the general satisfaction of those who are not motivated to provide feedback by a single incident. No specific timetable has been set yet, but I expect to see another survey run between October and December of this year.
Data from this survey, especially the free form comments, has already been shaping the quarterly planning for the Labs techops team and my more recent Tool Labs support initiative. I would say that a focus on making Tool Labs easier to use, especially by individuals who are not experts in the Unix command-line, is a general theme that is being pursued as we also seek to replace aging services like Sun Grid Engine and increase overall stability of the platform.
One of the things that stood out the most for me personally in the feedback was a general need for tutorials and other task focused technical documentation. This is an area that I would like to encourage the existing Tool Labs developers to help with. There are some ideas of things to write in Phabricator [0][1]. Generally we are lacking on "big picture" documentation. We have a lot of documentation on solving particular problems but even those documents can be difficult to find and understand as they are mostly written with an assumption of familiarity with the services and terminology used in Tool Labs.
[0]: https://phabricator.wikimedia.org/T101659 [1]: https://phabricator.wikimedia.org/T123425
Bryan
Sounds good. Thank you.
Pine
Pine On May 5, 2016 08:19, "Bryan Davis" bd808@wikimedia.org wrote:
On Wed, May 4, 2016 at 8:35 PM, Pine W wiki.pine@gmail.com wrote:
Hi Bryan,
Thanks for the report. With this information in hand, what follow up is planned?
The primary purpose of the study was to set a baseline of qualitative ("how are we doing") responses. These will be used to gauge increase/decrease in satisfaction in comparison to future surveys. Tool Labs staff get regular feedback from people who are either extremely happy or extremely unhappy with services, but a broad survey like this is helpful to determine the general satisfaction of those who are not motivated to provide feedback by a single incident. No specific timetable has been set yet, but I expect to see another survey run between October and December of this year.
Data from this survey, especially the free form comments, has already been shaping the quarterly planning for the Labs techops team and my more recent Tool Labs support initiative. I would say that a focus on making Tool Labs easier to use, especially by individuals who are not experts in the Unix command-line, is a general theme that is being pursued as we also seek to replace aging services like Sun Grid Engine and increase overall stability of the platform.
One of the things that stood out the most for me personally in the feedback was a general need for tutorials and other task focused technical documentation. This is an area that I would like to encourage the existing Tool Labs developers to help with. There are some ideas of things to write in Phabricator [0][1]. Generally we are lacking on "big picture" documentation. We have a lot of documentation on solving particular problems but even those documents can be difficult to find and understand as they are mostly written with an assumption of familiarity with the services and terminology used in Tool Labs.
Bryan
Bryan Davis Wikimedia Foundation bd808@wikimedia.org [[m:User:BDavis_(WMF)]] Sr Software Engineer Boise, ID USA irc: bd808 v:415.839.6885 x6855
Labs-l mailing list Labs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/labs-l
wikitech-l@lists.wikimedia.org