Hey folks!
Srishti and I have put together a list of the Toolhub bits we think we
can commit to working on in the next quarter. We would like y'all to
have a look and give us any feedback you may have: too aggressive
somewhere, missing a key feature, things like that.
Our goal is to implement the features described at
<https://meta.wikimedia.org/wiki/Toolhub/Roadmap#Milestone_2:_Initial_API_an…>
along with supporting features which are needed to build a stable
foundation for the project. This set of features will result in an
application which includes the core features of Hay's Directory and a
platform that can be extended in future iterations to achieve the
"first customer release" functionality of milestones 3, 4, 5, and 6.
By the close of Q2 we should have (roughly in priority order):
* Local development tooling and instructions for use
* Dockerfile(s) generated using Blubber
* Project hosted in gerrit
* CI tests run via PipelineLib
* Django backend account creation via OAuth with meta.wikimedia.org as
authorization server
* Django backend authentication via OAuth with meta.wikimedia.org as
authorization server
* User interface for creating/authenticating to a Toolhub account
* Django backend API described by an OpenAPI schema
* API endpoints for CRUD operations on toolinfo.json source URLs to be
used by the crawler
* User interface for CRUD operations on toolinfo.json source URLs to
be used by the crawler
* Integration with translatewiki.net for static messages (dynamic
content translations are out of scope at this point)
* Working user acceptance testing/staging deployment somewhere in
Toolforge/Cloud VPS/???
* API endpoints for viewing actions taken on Toolhub (think
Special:Log & Special:RecentChanges)
* User interface for viewing actions taken on Toolhub (think
Special:Log & Special:RecentChanges)
* Crawler process that loads toolinfo.json source URLs, validates
their content, and populates the Toolhub database
* User interface to view crawler status information
* User interface to display toolinfo records created by the crawler
* Django backend OAuth authorization server allowing registration of
new client applications
* API endpoints for OAuth client registration
* User interface to register a new OAuth client
* User interface to authorize an OAuth client to contribute to Toolhub
as the authorizing user
* User interface to view OAuth clients authorized to act as the
currently authenticated user
* User interface to revoke authorization from OAuth clients
* API authentication via OAuth with Toolhub as authorization server
Bryan
--
Bryan Davis Technical Engagement Wikimedia Foundation
Principal Software Engineer Boise, ID USA
[[m:User:BDavis_(WMF)]] irc: bd808
I started https://meta.wikimedia.org/wiki/Toolhub/Progress_reports and
created pages for the two reports that have been sent to this list so
far. I also have stubbed out
https://meta.wikimedia.org/wiki/Toolhub/Progress_reports/2020-10-02
for the coming week.
Everyone is welcome to add notes on the progress report pages at any
time. My current plan is to pretty up the page, remove the draft
marker template, and send a summary to this list each week around
16:30 UTC on Fridays. Let's see how long I can stick with this! :)
If anyone is bored and interested in messing around with the progress
reports page for fun, it would be neat to have something similar to
https://wikitech.wikimedia.org/wiki/Incident_documentation's use of
<inputbox> and preloads to make starting the stub for a new week
trivial. Ideas about navigation aids between the report pages would be
welcome as well. I feel like there used to be a nice setup for this
kind of project status reporting on either metawiki or mw.o circa
2014, but a few minutes of searching did not find anything super
obvious.
Bryan
--
Bryan Davis Technical Engagement Wikimedia Foundation
Principal Software Engineer Boise, ID USA
[[m:User:BDavis_(WMF)]] irc: bd808
Report on activities in the Toolhub project for the week ending 2020-09-25.
== More research tasks resolved ==
* T261020: Explore RTL language support issues
** Documented at
https://meta.wikimedia.org/wiki/Toolhub/Decision_record#RTL_language_support
* T259838: Explore translation options for Toolhub dynamic content
** Initial investigation documented at
https://meta.wikimedia.org/wiki/Toolhub/Decision_record#Translations_for_dy…
** Created tech spike task T263303 for exploration of Action API
integration with TWN
== Toolinfo v1.2.0-draft02 json schema published for review ==
A new v1.2.0-draft02 json schema is now on metawiki for review [0]. A
"final" v1.2.0 schema will probably not be released until Toolhub
reaches a milestone of a working toolinfo.json submission and storage
system to reduce the documentation churn that theory meeting practice
is likely to expose in the current draft.
Changes from v1.2.0-draft01:
* MaxLength constraints added for all string types
* Extracted #/definitions/url
* Extracted #/definitions/url_multilingual_or_array
* toolinfo_version replaced by $schema
* toolinfo_language replaced by $language
This schema is also now being managed with the
@wikimedia/jsonschema-tools library [1] developed by the Foundation's
Analytics team. This tool allows us to use a YAML file to describe the
schema [2] and produce versioned artifacts that are point-in-time
copies of that master document. This should make updating and revising
the schema easier in the future.
== Work in progress on Blubber and PipelineLib integration ==
Giuseppe suggested that we start working towards integration with the
Wikimedia deployment pipeline tooling sooner rather than later after
last week's announcement of a working development environment based on
Docker and Docker Compose.
Bryan spent time several days in this past week becoming more familiar
with Blubber [3] and PipelineLib [4]. These tools are intended to make
it easier to produce high quality Docker containers and test the code
they contain. Using standard Foundation tooling will reduce friction
when we get to the point of deploying the application to production.
The use of Poetry [5] by Toolhub for Python packaging and dependency
management appears to be a novel practice for Foundation projects.
Bryan had some discussions with Dan Duvall on irc about various ways
that Poetry could be used with Blubber. This in turn led to some
exploratory coding to see how the various approaches would work in
practice.
The first experiment was using Blubber's "builder" configuration to
install Poetry using their isolated installation bootstrapping script.
This worked, but it was later found that Blubber's order of operations
does not support copying an artifact from an earlier stage of a
multi-stage build in time to use that artifact as an input to a later
build stage--reported as T263597.
The second experiment was creating a custom base image with Poetry
pre-installed. This was functionally lifting the builder configuration
experiment's bootstrapping step up into the base image. The problem
found with this approach was managing directory permissions from the
Blubber integration. The user account that needs to own the
$POETRY_VIRTUALENVS_PATH directory is added by Blubber. Blubber has
the ability to generate chown instructions for a directory (by setting
the lives.in setting), but this ability is only extended for a single
directory per variant. Trying to get around this using multi-stage
chaining runs back into the artifact copying ordering issues from
T263597.
The third experiment was extending Blubber itself to understand how to
interact with Poetry. Bryan wrote and tested locally a patch for
Blubber extending the syntax of Blubber's 'python' configuration to
allow specifying a version of Poetry to install and use [6]. With this
patch and the proper configuration, Blubber will install Poetry using
pip and then use Poetry rather than pip to install additional project
dependencies. This allowed placing the correct instructions in each
stage of Blubber's opinionated Dockerfile generation to avoid the
issues encountered in the first two experiments.
Assuming that Bryan can work with the Blubber project maintainer to
refine and merge the patch, this seems to unlock moving on to
integrating Blubber into the local development workflow and then
starting work on PipelineLib integration (which uses Blubber as a
prerequisite).
== Wrap up ==
Next week includes the end of the Foundation's FY20/21 Q1 and the
start of Q2. The key result for Toolhub in the Q1 plan is currently
showing 93% complete. The remaining work in the KR is closing out two
mostly complete (Bryan thinks...) research tasks:
* T261017: Determine basic hosting parameters for Toolhub
** Waiting on feedback from Giuseppe and Reedy
* T261023: Explore content moderation issues
** A review from Risker would be appreciated
[0]: https://meta.wikimedia.org/wiki/Toolhub/Data_model#Version_1.2.0
[1]: https://www.npmjs.com/package/@wikimedia/jsonschema-tools
[2]: https://github.com/bd808/toolhub/blob/main/jsonschema/toolinfo/current.yaml
[3]: https://wikitech.wikimedia.org/wiki/Blubber
[4]: https://wikitech.wikimedia.org/wiki/PipelineLib
[5]: https://python-poetry.org/
[6]: https://gerrit.wikimedia.org/r/c/blubber/+/629877
Bryan
--
Bryan Davis Technical Engagement Wikimedia Foundation
Principal Software Engineer Boise, ID USA
[[m:User:BDavis_(WMF)]] irc: bd808
Welcome to the first weekly project update for Toolhub!
In an attempt to keep our loosely connected group informed about
happenings across Toolhub we will try to establish a regular practice
of collecting updates and sending them out once a week. This week
things reported will be off the top of Bryan's head, but work will
happen soon™ to create some place (probably on metawiki) to record
notes during the week so that they can be summarized and distributed
periodically.
Since this is the first "weekly" summary, and the project has been
doing things for about 2 months, this first update will cover a bit
more than just one week of events. :)
== Advisory council "complete" ==
T258185 was closed by Bryan on 2020-09-14 after Risker agreed to join
the advisory council. We now have representation on the advisory
committee of folks with experience in SRE, Security, Community
Relations, technical volunteering, and on-wiki contributions. More
folks may be invited to the committee if we find large gaps during
discussions of features and needs of the system, but this completes
the list of skills that was previously identified as critical to
initial work.
== Vue.js picked as UI framework ==
T261029 was closed by Bryan on 2020-09-14 after Srishti selected
Vue.js. Srishti initially narrowed the choices for a Javascript UI
framework down to React and Vue.js. Moriel and others made a good case
of adopting Vue.js in alignment with the relatively recent decision by
the now-disbanded Front-end Architecture Working Group (FAWG) to
recommend Vue.js as the future replacement for OOJS and other
component frameworks in MediaWiki. Ultimately Srishti and Bryan
decided that using Vue.js would be technically possible and
politically advantageous.
== Decision record created on metawiki ==
Bryan has started
<https://meta.wikimedia.org/wiki/Toolhub/Decision_record> as a
location to collect evidence of decisions made duing the course of
designing and implementing Toolhub. The intent is to create a document
that will help anyone joining the advisory council or implementation
team catch up on the important bits of past discussions. Currently it
documents the UI framework selection and a prior fiat decision to use
an "API first" design process when building application features.
== Dynamic translation research ==
Toolhub will not be attempting to establish its own translator
community. Instead we are hoping to integrate with the existing
translatewiki.net (TWN) community. Currently how that integration will
actually work is undefined, but there are 2 ideas along with some pros
and cons outlined in T259838. We expect to do some tech spike work in
the coming quarter to get a better idea of the feasibility of a web
API integration approach. If that does not seem viable after some
testing we are likely to go with the idea of exporting strings to a
TWN supported catalog format, shipping that catalog to TWN via git
repos (their normal workflow), and ultimately getting translations
back for integration with the application.
Bryan documented an afternoon of research at T259838#6460927 existing
Django apps intended to help with the technical issues of translating
user provided content stored in the application's database. High level
summaries of the functionality of each app are available on the
Phabricator task.
== Toolinfo v1.2.0-draft01 json schema published for review ==
Bryan did a review of the data model documented by James Hare during
the 2018 work on Toolhub to refresh his understanding of the design.
During this review, the lack of toolinfo.json support for declaring
the URL of end-user facing documents seemed like something that should
be revisited. This had been brought up in 2018 by Kunal on the data
model talk page [0]. James explained that he had set this field as an
"annotation" value to be added thorough Toolhub rather than a
toolinfo.json publisher controlled value in a hope of supporting links
to both 'official' and community created documentation. Bryan agrees
with the need to allow documenting community created documentation
links, but felt that we can figure out how to deal with competing
links in the user interfaces.
A new v1.2.0-draft01 json schema is now on metawiki for review [1]. A
"final" v1.2.0 schema will probably not be released until Toolhub
reaches a milestone of a working toolinfo.json submission and storage
system to reduce the documentation churn that theory meeting practice
is likely to expose in the current draft.
== Prototype development environment ==
Bryan has been poking at a local development environment [2] for the
project over the last few weeks. This week he reached a point where he
shared the system with Srishti to see if it could be made to work
anywhere other than Bryan's laptops. That test turned out to be
successful! Srishti reported back that she was able to get the system
booted up on her laptop and returning the expected empty Django
project landing page.
The current system is very experimental and the tooling it uses may
change significantly before we hit any deployment milestones. The
current system is using Docker Compose to manage multiple Docker
containers (initial one for the Django app and one for a MariaDB
database), Poetry for Python dependency management and versioning, and
a custom Makefile to make using it all a bit easier for the developer.
Today only git, Docker, Docker Compose, and GNU make are needed on the
developer's laptop. All of the Python and other tooling requirements
are being packed into the development mode Docker containers. This may
or may not be sustainable as we progress out of experimental mode and
start to integrate with the Wikimedia development ecosystem (Gerrit,
Jenkins, etc), but it has been a useful experiment for Bryan who has
not worked locally with Docker extensively.
== Wrap up ==
That's all the news that Bryan can remember today. :)
[0]: https://meta.wikimedia.org/wiki/Talk:Toolhub/Data_model#1.1.0_feedback
[1]: https://meta.wikimedia.org/wiki/Toolhub/Data_model#Version_1.2.0
[2]: https://github.com/bd808/toolhub
Bryan
--
Bryan Davis Technical Engagement Wikimedia Foundation
Principal Software Engineer Boise, ID USA
[[m:User:BDavis_(WMF)]] irc: bd808
The Foundation's FY20/21 Q1 closes in a week and half on 2020-09-30.
There are still the following open tasks related to Toolhub OKRs
(objectives and key results) for Q1:
* T261017: Determine basic hosting parameters for Toolhub
** Waiting on feedback from Giuseppe and Reedy
* T261023: Explore content moderation issues
** A review from Risker would be appreciated
* T261020: Explore RTL language support issues
** Needs to be documented on [[meta:Toolhub/Decision record]]
* T259838: Explore translation options for Toolhub dynamic content
** Needs related "tech spike" task for exploration of Action API
integration with TWN
** Needs related "tech spike" task for exploration of git repo
integration with TWN
** Needs to be documented on [[meta:Toolhub/Decision record]]
I will take on closing out the two tasks that are just needing some
documentation.
Giuseppe, Reedy, and Risker: I would appreciate it if y'all could pick
an hour in your week next week to check in on the tasks that are
wanting your review and give your feedback. These are both relatively
high level directional questions, so don't worry about giving
exhaustive recommendations. We are just looking for consensus that our
ideas about hosting and moderation are headed in the right direction.
Bryan
--
Bryan Davis Technical Engagement Wikimedia Foundation
Principal Software Engineer Boise, ID USA
[[m:User:BDavis_(WMF)]] irc: bd808
I am happy to announce that Anne Clin ([[User:Risker]]) has agreed to
join the Toolhub advisory council.
Risker is a long time contributor to English Wikipedia. She has also
served on ArbCom and currently holds CU and Oversighter rights on
enwiki. I have talked on and off with Risker about the importance of
tools for the on-wiki communities for years. I am excited that she
will be making time to help us make decisions for Toolhub. I hope that
her insights will lead us to better interfaces and workflows
generally, and especially where we are making things that relate to
interfaces and workflows that Wikimedia editors and admins are used to
using on the project wikis.
Bryan
--
Bryan Davis Technical Engagement Wikimedia Foundation
Principal Software Engineer Boise, ID USA
[[m:User:BDavis_(WMF)]] irc: bd808
I am happy to announce that Eran Roz ([[User:ערן]]) has agreed to join
the Toolhub advisory council.
When I asked Moriel for recommendations of technical volunteers with
really strong RTL web knowledge she suggested I contact Eran. As you
can see on their mw.o user page, Eran has a history of contributing in
many technical areas including building tools and bots and helping
find and fix RTL bugs. I am excited to have Eran join us and look
forward to their feedback on many aspects of the design and
implementation of Toolhub.
Bryan
--
Bryan Davis Technical Engagement Wikimedia Foundation
Principal Software Engineer Boise, ID USA
[[m:User:BDavis_(WMF)]] irc: bd808
Hey folks,
I approved a list subscription by Taavi Väänänen today. Taavi is
[[User:Majavah]] on wiki. Taavi has been getting more involved in the
Wikimedia technical community since participating in Google Code-In
with us in the 2019 round. A few months ago he also contributed a
major feature patch for Striker that added Phabricator project
creation to the app. It took me forever to review it and then even
longer to deploy it, but the patch was really solid work. It is also
the only volunteer contributed feature since the Striker project began
in 2016! :)
Taavi, you are the first person on this list that I did not drag here
to help Sristi and I with questions as we start implementing Toolhub.
You are very welcome to participate in the discussions we have here or
just lurk.
Everyone, please do remember that this list is a part of the Wikimedia
technical spaces and thus covered by the
<https://www.mediawiki.org/wiki/Code_of_Conduct>. The archives of the
list are also publicly available, so keep that in mind as well when
posting here. As long as we stick to the topic of the list, developing
Toohub, I think everyone will be fine. :)
Bryan
--
Bryan Davis Technical Engagement Wikimedia Foundation
Principal Software Engineer Boise, ID USA
[[m:User:BDavis_(WMF)]] irc: bd808