Report on activities in the Toolhub project for the week ending 2020-11-20.
See the on-wiki progress report [0] for additional details and clickable links.
== TL;DR ==
* No report for week of 2020-11-13
* Toolinfo read API
* Toolinfo.json crawler and status API
* Locale switching in Vue frontend
* Improved error responses from API
* Miscellaneous code cleanup changes & bug fixes
Even with the vacation time in the last 2 weeks, we are making good
progress. The implementation of the crawler and toolinfo storage and
their related APIs unblocks a lot of UI work. Both the Django and Vue
systems are ready to start on the process of integrating with TWN for
message translation, which could start happening as early as next week
(although the first week of December is more likely). We are also very
close to the point of having enough functionality to make demoing with
others a useful process.
We are also starting to evaluate our actual progress vs our announced
plans for the quarter. Audit logs, TWN integration, and OAuth for API
consumers all remain to be done. With only 6 calendar weeks, and
functionally 4 working weeks left in the quarter we are cautiously
optimistic that all of these remaining components will be started. It
is unlikely that they will all be completed however. We should be able
to make better estimates in the next 2 weeks about what will and will
not be possible in the remaining timebox.
[0]: https://meta.wikimedia.org/wiki/Toolhub/Progress_reports/2020-11-20
Bryan
--
Bryan Davis Technical Engagement Wikimedia Foundation
Principal Software Engineer Boise, ID USA
[[m:User:BDavis_(WMF)]] irc: bd808
Report on activities in the Toolhub project for the week ending 2020-11-06.
See the on-wiki progress report [0] for additional details and clickable links.
== TL;DR ==
* Eslint configured for project
* Django locale file generation setup
* We have a logo!
* UI for adding/removing toolinfo.json URLs implemented
* Work started on API for retrieving tool data collected from
toolinfo.json files
This was another week of great progress for the project. Bryan and
Srishti continue to discover more bootstrapping issues, but they are
being resolved quickly without major impediment to over all progress.
Bryan is adjusting his planned work to try and get ahead of making API
endpoints for Srishti to integrate into the frontend. Bryan's tool
description API work will continue as Srishti takes some time to work
on localization for the existing Vue frontend.
Progress will slow during the week of 2020-11-13. The Foundation has
made 3 days of that week holidays, and Bryan is going to take
advantage of that by adding in 2 PTO days for a week long break. No
progress report is expected to be sent to the mailing list next week,
but additional progress may be recorded on metawiki.
[0]: https://meta.wikimedia.org/wiki/Toolhub/Progress_reports/2020-11-06
Bryan
--
Bryan Davis Technical Engagement Wikimedia Foundation
Principal Software Engineer Boise, ID USA
[[m:User:BDavis_(WMF)]] irc: bd808
Report on activities in the Toolhub project for the week ending 2020-10-30.
See the on-wiki progress report [0] for additional details and clickable links.
== TL;DR ==
* Initial backend API created
* OpenAPI v3 schema being automatically generated for backend API
* OpenAPI schema viewer with "try" functionality added to Vue frontend
* CRUD API for managing toolinfo.json URLs for the crawler created
* Work in progress on frontend for toolinfo.json URL management
Progress, and learning about the new to us tools being used in the
project, continues at a steady pace. As a greenfield project that is
using many components new to the developers and largely new to the
Foundation there has been some "two steps forward, one step back"
activities as making progress on a given milestone uncovers weaknesses
in or omissions of tertiary features. This sort of thing is to be
expected and thus far has not become disruptive to the overall
progress and timeline. We are in the last week of the first month of
the quarter, but due to the number of holiday and vacation breaks in
November and December we are closer to half way through the available
working hours for the quarter. We will continue to evaluate progress
versus the roadmap and try to keep everyone informed of potential
changes in what we believe will be {{done}} by the end of the quarter.
[0]: https://meta.wikimedia.org/wiki/Toolhub/Progress_reports/2020-10-30
Bryan
--
Bryan Davis Technical Engagement Wikimedia Foundation
Principal Software Engineer Boise, ID USA
[[m:User:BDavis_(WMF)]] irc: bd808
Report on activities in the Toolhub project for the week ending 2020-10-23.
See the on-wiki progress report [0] for additional details and clickable links.
== TL;DR ==
* The project's canonical git repo has been moved to wikimedia/toolhub
in Gerrit.
* Jenkins is running tests for the gerrit repo using the PipelineLib
project's in-repo configuration. No images are being published to the
Foundation's Docker repo yet.
* Toolhub now has a Vue.js powered user interface!
* Configuration has been added to the Vue frontend and Django backend
creating a workflow for the OAuth 2.0 authentication process and
informing Vue of the current user's authentication state.
This was a good week for progress on the project. As hoped for last
week, the migration to Gerrit and initial Vue setup have been
completed. Finding a workflow for the OAuth login and communicating
user data to the Vue frontend was a nice bonus completion. Bryan plans
on starting work on the Django backend's API endpoints for adding and
removing toolinfo.json external URLs to configure the crawler in the
coming week. Srishti is planning on UI design for the external URLs
management.
For those of you with access to the Foundation's Slack and/or Matrix
chat systems, there is now a #toolhub-team channel that Bryan and
Srishti will be using to discuss Toolhub dev questions rather than
private messages. Join us if you are interested! (/me apologizes to
the non-Foundation staff who do not currently have access to this
chat.)
[0]: https://meta.wikimedia.org/wiki/Toolhub/Progress_reports/2020-10-23
Bryan
--
Bryan Davis Technical Engagement Wikimedia Foundation
Principal Software Engineer Boise, ID USA
[[m:User:BDavis_(WMF)]] irc: bd808
Report on activities in the Toolhub project for the week ending 2020-10-16.
See the on-wiki progress report [0] for additional details and clickable links.
== TL;DR ==
* T264807: The Django backend now has working integration with
MediaWiki's OAuth 2.0 server for user account creation and
authentication.
* T264813: A .pipeline/config.yaml configuration has been written.
Testing is blocked on moving the git repo from GitHub to Gerrit which
is expected in the coming week.
* Srishti has been reviewing the UI wireframes from 2018 and thinking
about how implementation will happen. She also has made a POC git repo
to show Bryan how to setup the parallel frontend and backend
components.
Bootstrapping work is nearly done with migration to Gerrit and initial
Vue.js setup remaining. We expect to complete these two steps in the
coming week. Bryan will then move on to the initial API bootstrapping
and endpoints for managing toolinfo.json URLs while Srishti starts
designing the basics of the frontend and connecting it to the backend
for authentication.
[0]: https://meta.wikimedia.org/wiki/Toolhub/Progress_reports/2020-10-16
Bryan
--
Bryan Davis Technical Engagement Wikimedia Foundation
Principal Software Engineer Boise, ID USA
[[m:User:BDavis_(WMF)]] irc: bd808
As has become the early tradition here, I would like to welcome
Joaquin Oltra Hernandez, [[User:JHernandez (WMF)]], to the list.
Joaquin is a Developer Advocate at the Foundation and is interested in
following along with our experiment in using this list to keep folks
connected to the Toolhub project work and related discussions. I also
have hopes of getting him involved in the technical community facing
communications that will be needed as we get closer to deploying the
tool. :)
Everyone, please do remember that this list is a part of the Wikimedia
technical spaces and thus covered by the
<https://www.mediawiki.org/wiki/Code_of_Conduct>. The archives of the
list are also publicly available, so keep that in mind as well when
posting here.
Bryan
--
Bryan Davis Technical Engagement Wikimedia Foundation
Principal Software Engineer Boise, ID USA
[[m:User:BDavis_(WMF)]] irc: bd808
Report on activities in the Toolhub project for the week ending 2020-10-09.
See the on-wiki progress report [0] for additional details and clickable links.
Now that we have a place on metawiki for these reports to be written
and archived, I'm going to try making the mailing list post be more of
a TL;DR summary than a comprehensive report. Feedback on that
arbitrary decision is appreciated. If folks would rather have all the
info by email I am happy to accommodate them.
== TL;DR ==
* Goals for the quarter (October-December 2020) have been set and
recorded in Betterworks and Phabricator.
* Research and proof of concept work on Vue.js related front end
concepts has progressed with a result of Vuetify
(<https://vuetifyjs.com/>) being selected as the user interface
library for the project.
* Work has started on OAuth 2.0 integration with MediaWiki including
an upstream patch to make MediaWiki's OAuth 2.0 identity responses
include data that OAuth 1.0a has historically returned.
* The local development environment is converging with the expected
production CI and deployment systems.
[0]: https://meta.wikimedia.org/wiki/Toolhub/Progress_reports/2020-10-09
Bryan
--
Bryan Davis Technical Engagement Wikimedia Foundation
Principal Software Engineer Boise, ID USA
[[m:User:BDavis_(WMF)]] irc: bd808
Report on activities in the Toolhub project for the week ending 2020-10-02.
See the on-wiki progress report [0] for prettier formatting and clickable links.
== Progress reports now published on metawiki ==
The weekly progress reports for Toolhub development from the weeks of
2020-09-18 and 2020-09-25 have been retroactively posted to the new
Toolhub/Progress reports area on metawiki. Going forward these reports
will start on metawiki as pages marked with {{Draft}} and then be
tidied up and summarized to the toolhub-dev mailing list at the end of
the work week.
== Content moderation exploration ==
Where there is free form text content, there will be vandalism. This
is an Internet truism that Wikimedians are well aware of. MediaWiki
includes many components to help support patrolling content
submissions. Toolhub will need systems for this as well, but what
systems? How can we make the process of patrolling Toolhub feel
friendly to folks who are used to doing content patrolling with
MediaWiki?
Bryan, Keegan, Johan, Harej, Xover, and Risker all provided input on
the discussion of needs in T261023. The combined recommendations have
been summarized at Toolhub/Decision record#Content moderation support.
== Draft goals for October-December 2020 ==
Srishti and Bryan have drafted a proposed set of functionality to be
implemented in Q2 of the Foundation's fiscal year 2020/2021
(October-December 2020). This list has been shared with the advisory
council and will likely be finalized as objectives and key results
(OKRs) in the week of October 5th.
Our goal is to implement the features described at
Toolhub/Roadmap#Milestone 2: Initial API and UI; toolinfo crawler
along with supporting features which are needed to build a stable
foundation for the project. This set of features will result in an
application which includes the core features of Hay's Directory and a
platform that can be extended in future iterations to achieve the
"first customer release" functionality of milestones 3, 4, 5, and 6.
== Wrap up ==
The week saw the end of the Foundation's FY20/21 Q1 and the start of
Q2. The key result for Toolhub in the Q1 plan ("Define the technical
plan for Toolhub") ended the quarter with an assessment of 97%
complete. The remaining work in the KR is closing out the "Determine
basic hosting parameters for Toolhub" research task. This task is
currently waiting on feedback from members of the advisory board.
Planning for the next phase of the project (coding!) is well underway.
Official goals are being finalized and should be documented by the
time we send out the progress report for the week ending 2020-10-09.
Bryan is happy in general with the progress that was made in Q1. His
main takeaway for improvement in the future is to try and get issues
needing feedback from other teams and the advisory council surfaced
earlier in the quarter to avoid colliding with the inevitable time
crunch that folks fall under towards the end of the quarter when they
are trying to close out their own goals.
[0]: https://meta.wikimedia.org/wiki/Toolhub/Progress_reports/2020-10-02
Bryan
--
Bryan Davis Technical Engagement Wikimedia Foundation
Principal Software Engineer Boise, ID USA
[[m:User:BDavis_(WMF)]] irc: bd808
Hey folks!
Srishti and I have put together a list of the Toolhub bits we think we
can commit to working on in the next quarter. We would like y'all to
have a look and give us any feedback you may have: too aggressive
somewhere, missing a key feature, things like that.
Our goal is to implement the features described at
<https://meta.wikimedia.org/wiki/Toolhub/Roadmap#Milestone_2:_Initial_API_an…>
along with supporting features which are needed to build a stable
foundation for the project. This set of features will result in an
application which includes the core features of Hay's Directory and a
platform that can be extended in future iterations to achieve the
"first customer release" functionality of milestones 3, 4, 5, and 6.
By the close of Q2 we should have (roughly in priority order):
* Local development tooling and instructions for use
* Dockerfile(s) generated using Blubber
* Project hosted in gerrit
* CI tests run via PipelineLib
* Django backend account creation via OAuth with meta.wikimedia.org as
authorization server
* Django backend authentication via OAuth with meta.wikimedia.org as
authorization server
* User interface for creating/authenticating to a Toolhub account
* Django backend API described by an OpenAPI schema
* API endpoints for CRUD operations on toolinfo.json source URLs to be
used by the crawler
* User interface for CRUD operations on toolinfo.json source URLs to
be used by the crawler
* Integration with translatewiki.net for static messages (dynamic
content translations are out of scope at this point)
* Working user acceptance testing/staging deployment somewhere in
Toolforge/Cloud VPS/???
* API endpoints for viewing actions taken on Toolhub (think
Special:Log & Special:RecentChanges)
* User interface for viewing actions taken on Toolhub (think
Special:Log & Special:RecentChanges)
* Crawler process that loads toolinfo.json source URLs, validates
their content, and populates the Toolhub database
* User interface to view crawler status information
* User interface to display toolinfo records created by the crawler
* Django backend OAuth authorization server allowing registration of
new client applications
* API endpoints for OAuth client registration
* User interface to register a new OAuth client
* User interface to authorize an OAuth client to contribute to Toolhub
as the authorizing user
* User interface to view OAuth clients authorized to act as the
currently authenticated user
* User interface to revoke authorization from OAuth clients
* API authentication via OAuth with Toolhub as authorization server
Bryan
--
Bryan Davis Technical Engagement Wikimedia Foundation
Principal Software Engineer Boise, ID USA
[[m:User:BDavis_(WMF)]] irc: bd808
I started https://meta.wikimedia.org/wiki/Toolhub/Progress_reports and
created pages for the two reports that have been sent to this list so
far. I also have stubbed out
https://meta.wikimedia.org/wiki/Toolhub/Progress_reports/2020-10-02
for the coming week.
Everyone is welcome to add notes on the progress report pages at any
time. My current plan is to pretty up the page, remove the draft
marker template, and send a summary to this list each week around
16:30 UTC on Fridays. Let's see how long I can stick with this! :)
If anyone is bored and interested in messing around with the progress
reports page for fun, it would be neat to have something similar to
https://wikitech.wikimedia.org/wiki/Incident_documentation's use of
<inputbox> and preloads to make starting the stub for a new week
trivial. Ideas about navigation aids between the report pages would be
welcome as well. I feel like there used to be a nice setup for this
kind of project status reporting on either metawiki or mw.o circa
2014, but a few minutes of searching did not find anything super
obvious.
Bryan
--
Bryan Davis Technical Engagement Wikimedia Foundation
Principal Software Engineer Boise, ID USA
[[m:User:BDavis_(WMF)]] irc: bd808