September 2020 - Toolhub-dev - lists.wikimedia.org

Draft goals for October-December 2020
by Bryan Davis 30 Sep '20

30 Sep '20

Hey folks! Srishti and I have put together a list of the Toolhub bits we think we can commit to working on in the next quarter. We would like y'all to have a look and give us any feedback you may have: too aggressive somewhere, missing a key feature, things like that. Our goal is to implement the features described at <https://meta.wikimedia.org/wiki/Toolhub/Roadmap#Milestone_2:_Initial_API_an…> along with supporting features which are needed to build a stable foundation for the project. This set of features will result in an application which includes the core features of Hay's Directory and a platform that can be extended in future iterations to achieve the "first customer release" functionality of milestones 3, 4, 5, and 6. By the close of Q2 we should have (roughly in priority order): * Local development tooling and instructions for use * Dockerfile(s) generated using Blubber * Project hosted in gerrit * CI tests run via PipelineLib * Django backend account creation via OAuth with meta.wikimedia.org as authorization server * Django backend authentication via OAuth with meta.wikimedia.org as authorization server * User interface for creating/authenticating to a Toolhub account * Django backend API described by an OpenAPI schema * API endpoints for CRUD operations on toolinfo.json source URLs to be used by the crawler * User interface for CRUD operations on toolinfo.json source URLs to be used by the crawler * Integration with translatewiki.net for static messages (dynamic content translations are out of scope at this point) * Working user acceptance testing/staging deployment somewhere in Toolforge/Cloud VPS/??? * API endpoints for viewing actions taken on Toolhub (think Special:Log & Special:RecentChanges) * User interface for viewing actions taken on Toolhub (think Special:Log & Special:RecentChanges) * Crawler process that loads toolinfo.json source URLs, validates their content, and populates the Toolhub database * User interface to view crawler status information * User interface to display toolinfo records created by the crawler * Django backend OAuth authorization server allowing registration of new client applications * API endpoints for OAuth client registration * User interface to register a new OAuth client * User interface to authorize an OAuth client to contribute to Toolhub as the authorizing user * User interface to view OAuth clients authorized to act as the currently authenticated user * User interface to revoke authorization from OAuth clients * API authentication via OAuth with Toolhub as authorization server Bryan -- Bryan Davis Technical Engagement Wikimedia Foundation Principal Software Engineer Boise, ID USA [[m:User:BDavis_(WMF)]] irc: bd808

1 0

Weekly updates now also on metawiki
by Bryan Davis 25 Sep '20

25 Sep '20

I started https://meta.wikimedia.org/wiki/Toolhub/Progress_reports and created pages for the two reports that have been sent to this list so far. I also have stubbed out https://meta.wikimedia.org/wiki/Toolhub/Progress_reports/2020-10-02 for the coming week. Everyone is welcome to add notes on the progress report pages at any time. My current plan is to pretty up the page, remove the draft marker template, and send a summary to this list each week around 16:30 UTC on Fridays. Let's see how long I can stick with this! :) If anyone is bored and interested in messing around with the progress reports page for fun, it would be neat to have something similar to https://wikitech.wikimedia.org/wiki/Incident_documentation's use of <inputbox> and preloads to make starting the stub for a new week trivial. Ideas about navigation aids between the report pages would be welcome as well. I feel like there used to be a nice setup for this kind of project status reporting on either metawiki or mw.o circa 2014, but a few minutes of searching did not find anything super obvious. Bryan -- Bryan Davis Technical Engagement Wikimedia Foundation Principal Software Engineer Boise, ID USA [[m:User:BDavis_(WMF)]] irc: bd808

1 0

2020-09-25 weekly Toolhub update
by Bryan Davis 25 Sep '20

25 Sep '20

Report on activities in the Toolhub project for the week ending 2020-09-25. == More research tasks resolved == * T261020: Explore RTL language support issues ** Documented at https://meta.wikimedia.org/wiki/Toolhub/Decision_record#RTL_language_support * T259838: Explore translation options for Toolhub dynamic content ** Initial investigation documented at https://meta.wikimedia.org/wiki/Toolhub/Decision_record#Translations_for_dy… ** Created tech spike task T263303 for exploration of Action API integration with TWN == Toolinfo v1.2.0-draft02 json schema published for review == A new v1.2.0-draft02 json schema is now on metawiki for review [0]. A "final" v1.2.0 schema will probably not be released until Toolhub reaches a milestone of a working toolinfo.json submission and storage system to reduce the documentation churn that theory meeting practice is likely to expose in the current draft. Changes from v1.2.0-draft01: * MaxLength constraints added for all string types * Extracted #/definitions/url * Extracted #/definitions/url_multilingual_or_array * toolinfo_version replaced by $schema * toolinfo_language replaced by $language This schema is also now being managed with the @wikimedia/jsonschema-tools library [1] developed by the Foundation's Analytics team. This tool allows us to use a YAML file to describe the schema [2] and produce versioned artifacts that are point-in-time copies of that master document. This should make updating and revising the schema easier in the future. == Work in progress on Blubber and PipelineLib integration == Giuseppe suggested that we start working towards integration with the Wikimedia deployment pipeline tooling sooner rather than later after last week's announcement of a working development environment based on Docker and Docker Compose. Bryan spent time several days in this past week becoming more familiar with Blubber [3] and PipelineLib [4]. These tools are intended to make it easier to produce high quality Docker containers and test the code they contain. Using standard Foundation tooling will reduce friction when we get to the point of deploying the application to production. The use of Poetry [5] by Toolhub for Python packaging and dependency management appears to be a novel practice for Foundation projects. Bryan had some discussions with Dan Duvall on irc about various ways that Poetry could be used with Blubber. This in turn led to some exploratory coding to see how the various approaches would work in practice. The first experiment was using Blubber's "builder" configuration to install Poetry using their isolated installation bootstrapping script. This worked, but it was later found that Blubber's order of operations does not support copying an artifact from an earlier stage of a multi-stage build in time to use that artifact as an input to a later build stage--reported as T263597. The second experiment was creating a custom base image with Poetry pre-installed. This was functionally lifting the builder configuration experiment's bootstrapping step up into the base image. The problem found with this approach was managing directory permissions from the Blubber integration. The user account that needs to own the $POETRY_VIRTUALENVS_PATH directory is added by Blubber. Blubber has the ability to generate chown instructions for a directory (by setting the lives.in setting), but this ability is only extended for a single directory per variant. Trying to get around this using multi-stage chaining runs back into the artifact copying ordering issues from T263597. The third experiment was extending Blubber itself to understand how to interact with Poetry. Bryan wrote and tested locally a patch for Blubber extending the syntax of Blubber's 'python' configuration to allow specifying a version of Poetry to install and use [6]. With this patch and the proper configuration, Blubber will install Poetry using pip and then use Poetry rather than pip to install additional project dependencies. This allowed placing the correct instructions in each stage of Blubber's opinionated Dockerfile generation to avoid the issues encountered in the first two experiments. Assuming that Bryan can work with the Blubber project maintainer to refine and merge the patch, this seems to unlock moving on to integrating Blubber into the local development workflow and then starting work on PipelineLib integration (which uses Blubber as a prerequisite). == Wrap up == Next week includes the end of the Foundation's FY20/21 Q1 and the start of Q2. The key result for Toolhub in the Q1 plan is currently showing 93% complete. The remaining work in the KR is closing out two mostly complete (Bryan thinks...) research tasks: * T261017: Determine basic hosting parameters for Toolhub ** Waiting on feedback from Giuseppe and Reedy * T261023: Explore content moderation issues ** A review from Risker would be appreciated [0]: https://meta.wikimedia.org/wiki/Toolhub/Data_model#Version_1.2.0 [1]: https://www.npmjs.com/package/@wikimedia/jsonschema-tools [2]: https://github.com/bd808/toolhub/blob/main/jsonschema/toolinfo/current.yaml [3]: https://wikitech.wikimedia.org/wiki/Blubber [4]: https://wikitech.wikimedia.org/wiki/PipelineLib [5]: https://python-poetry.org/ [6]: https://gerrit.wikimedia.org/r/c/blubber/+/629877 Bryan -- Bryan Davis Technical Engagement Wikimedia Foundation Principal Software Engineer Boise, ID USA [[m:User:BDavis_(WMF)]] irc: bd808

1 0

2020-09-18 "weekly" Toolhub update
by Bryan Davis 21 Sep '20

21 Sep '20

Welcome to the first weekly project update for Toolhub! In an attempt to keep our loosely connected group informed about happenings across Toolhub we will try to establish a regular practice of collecting updates and sending them out once a week. This week things reported will be off the top of Bryan's head, but work will happen soon™ to create some place (probably on metawiki) to record notes during the week so that they can be summarized and distributed periodically. Since this is the first "weekly" summary, and the project has been doing things for about 2 months, this first update will cover a bit more than just one week of events. :) == Advisory council "complete" == T258185 was closed by Bryan on 2020-09-14 after Risker agreed to join the advisory council. We now have representation on the advisory committee of folks with experience in SRE, Security, Community Relations, technical volunteering, and on-wiki contributions. More folks may be invited to the committee if we find large gaps during discussions of features and needs of the system, but this completes the list of skills that was previously identified as critical to initial work. == Vue.js picked as UI framework == T261029 was closed by Bryan on 2020-09-14 after Srishti selected Vue.js. Srishti initially narrowed the choices for a Javascript UI framework down to React and Vue.js. Moriel and others made a good case of adopting Vue.js in alignment with the relatively recent decision by the now-disbanded Front-end Architecture Working Group (FAWG) to recommend Vue.js as the future replacement for OOJS and other component frameworks in MediaWiki. Ultimately Srishti and Bryan decided that using Vue.js would be technically possible and politically advantageous. == Decision record created on metawiki == Bryan has started <https://meta.wikimedia.org/wiki/Toolhub/Decision_record> as a location to collect evidence of decisions made duing the course of designing and implementing Toolhub. The intent is to create a document that will help anyone joining the advisory council or implementation team catch up on the important bits of past discussions. Currently it documents the UI framework selection and a prior fiat decision to use an "API first" design process when building application features. == Dynamic translation research == Toolhub will not be attempting to establish its own translator community. Instead we are hoping to integrate with the existing translatewiki.net (TWN) community. Currently how that integration will actually work is undefined, but there are 2 ideas along with some pros and cons outlined in T259838. We expect to do some tech spike work in the coming quarter to get a better idea of the feasibility of a web API integration approach. If that does not seem viable after some testing we are likely to go with the idea of exporting strings to a TWN supported catalog format, shipping that catalog to TWN via git repos (their normal workflow), and ultimately getting translations back for integration with the application. Bryan documented an afternoon of research at T259838#6460927 existing Django apps intended to help with the technical issues of translating user provided content stored in the application's database. High level summaries of the functionality of each app are available on the Phabricator task. == Toolinfo v1.2.0-draft01 json schema published for review == Bryan did a review of the data model documented by James Hare during the 2018 work on Toolhub to refresh his understanding of the design. During this review, the lack of toolinfo.json support for declaring the URL of end-user facing documents seemed like something that should be revisited. This had been brought up in 2018 by Kunal on the data model talk page [0]. James explained that he had set this field as an "annotation" value to be added thorough Toolhub rather than a toolinfo.json publisher controlled value in a hope of supporting links to both 'official' and community created documentation. Bryan agrees with the need to allow documenting community created documentation links, but felt that we can figure out how to deal with competing links in the user interfaces. A new v1.2.0-draft01 json schema is now on metawiki for review [1]. A "final" v1.2.0 schema will probably not be released until Toolhub reaches a milestone of a working toolinfo.json submission and storage system to reduce the documentation churn that theory meeting practice is likely to expose in the current draft. == Prototype development environment == Bryan has been poking at a local development environment [2] for the project over the last few weeks. This week he reached a point where he shared the system with Srishti to see if it could be made to work anywhere other than Bryan's laptops. That test turned out to be successful! Srishti reported back that she was able to get the system booted up on her laptop and returning the expected empty Django project landing page. The current system is very experimental and the tooling it uses may change significantly before we hit any deployment milestones. The current system is using Docker Compose to manage multiple Docker containers (initial one for the Django app and one for a MariaDB database), Poetry for Python dependency management and versioning, and a custom Makefile to make using it all a bit easier for the developer. Today only git, Docker, Docker Compose, and GNU make are needed on the developer's laptop. All of the Python and other tooling requirements are being packed into the development mode Docker containers. This may or may not be sustainable as we progress out of experimental mode and start to integrate with the Wikimedia development ecosystem (Gerrit, Jenkins, etc), but it has been a useful experiment for Bryan who has not worked locally with Docker extensively. == Wrap up == That's all the news that Bryan can remember today. :) [0]: https://meta.wikimedia.org/wiki/Talk:Toolhub/Data_model#1.1.0_feedback [1]: https://meta.wikimedia.org/wiki/Toolhub/Data_model#Version_1.2.0 [2]: https://github.com/bd808/toolhub Bryan -- Bryan Davis Technical Engagement Wikimedia Foundation Principal Software Engineer Boise, ID USA [[m:User:BDavis_(WMF)]] irc: bd808

3 2

Closing out FY20/21 Q1 related tasks
by Bryan Davis 19 Sep '20

19 Sep '20

The Foundation's FY20/21 Q1 closes in a week and half on 2020-09-30. There are still the following open tasks related to Toolhub OKRs (objectives and key results) for Q1: * T261017: Determine basic hosting parameters for Toolhub ** Waiting on feedback from Giuseppe and Reedy * T261023: Explore content moderation issues ** A review from Risker would be appreciated * T261020: Explore RTL language support issues ** Needs to be documented on [[meta:Toolhub/Decision record]] * T259838: Explore translation options for Toolhub dynamic content ** Needs related "tech spike" task for exploration of Action API integration with TWN ** Needs related "tech spike" task for exploration of git repo integration with TWN ** Needs to be documented on [[meta:Toolhub/Decision record]] I will take on closing out the two tasks that are just needing some documentation. Giuseppe, Reedy, and Risker: I would appreciate it if y'all could pick an hour in your week next week to check in on the tasks that are wanting your review and give your feedback. These are both relatively high level directional questions, so don't worry about giving exhaustive recommendations. We are just looking for consensus that our ideas about hosting and moderation are headed in the right direction. Bryan -- Bryan Davis Technical Engagement Wikimedia Foundation Principal Software Engineer Boise, ID USA [[m:User:BDavis_(WMF)]] irc: bd808

1 0

Welcome Risker to the advisory council
by Bryan Davis 14 Sep '20

14 Sep '20

I am happy to announce that Anne Clin ([[User:Risker]]) has agreed to join the Toolhub advisory council. Risker is a long time contributor to English Wikipedia. She has also served on ArbCom and currently holds CU and Oversighter rights on enwiki. I have talked on and off with Risker about the importance of tools for the on-wiki communities for years. I am excited that she will be making time to help us make decisions for Toolhub. I hope that her insights will lead us to better interfaces and workflows generally, and especially where we are making things that relate to interfaces and workflows that Wikimedia editors and admins are used to using on the project wikis. Bryan -- Bryan Davis Technical Engagement Wikimedia Foundation Principal Software Engineer Boise, ID USA [[m:User:BDavis_(WMF)]] irc: bd808

2 1

Welcome Eran Roz to the advisory council
by Bryan Davis 12 Sep '20

12 Sep '20

I am happy to announce that Eran Roz ([[User:ערן]]) has agreed to join the Toolhub advisory council. When I asked Moriel for recommendations of technical volunteers with really strong RTL web knowledge she suggested I contact Eran. As you can see on their mw.o user page, Eran has a history of contributing in many technical areas including building tools and bots and helping find and fix RTL bugs. I am excited to have Eran join us and look forward to their feedback on many aspects of the design and implementation of Toolhub. Bryan -- Bryan Davis Technical Engagement Wikimedia Foundation Principal Software Engineer Boise, ID USA [[m:User:BDavis_(WMF)]] irc: bd808

1 0

Welcome Taavi to the list
by Bryan Davis 10 Sep '20

10 Sep '20

Hey folks, I approved a list subscription by Taavi Väänänen today. Taavi is [[User:Majavah]] on wiki. Taavi has been getting more involved in the Wikimedia technical community since participating in Google Code-In with us in the 2019 round. A few months ago he also contributed a major feature patch for Striker that added Phabricator project creation to the app. It took me forever to review it and then even longer to deploy it, but the patch was really solid work. It is also the only volunteer contributed feature since the Striker project began in 2016! :) Taavi, you are the first person on this list that I did not drag here to help Sristi and I with questions as we start implementing Toolhub. You are very welcome to participate in the discussions we have here or just lurk. Everyone, please do remember that this list is a part of the Wikimedia technical spaces and thus covered by the <https://www.mediawiki.org/wiki/Code_of_Conduct>. The archives of the list are also publicly available, so keep that in mind as well when posting here. As long as we stick to the topic of the list, developing Toohub, I think everyone will be fine. :) Bryan -- Bryan Davis Technical Engagement Wikimedia Foundation Principal Software Engineer Boise, ID USA [[m:User:BDavis_(WMF)]] irc: bd808

1 0

2024

2023

2022

2021

2020

Toolhub-dev September 2020