Report on activities in the Toolhub project for the week ending 2021-04-02.
See the on-wiki progress report [0] for additional details and clickable links.
== TL;DR ==
* Crawler error tracking improved
* Data consistency improved with custom model data field classes
* Toolinfo origin exposed via API and used in UI access control
* April-June 2021 planning continues
This week marks the completion of the second full quarter of
implementation work on Toolhub. During the quarter, Bryan and Srishti
have:
* Given two presentations to Wikimedia Foundation staff about the
Toolhub project
* Completed the user interface to register, authorize, view, and
revoke Toolhub OAuth grants
* Implemented faceted search
* Implemented creation and editing of new toolinfo records via the API and UI
* Implemented the backend for viewing history and diffs of toolinfo
records over time
* Designed and implemented a uniform notification system for the UI
* Improved localization with changes to the translation layer
* Made many improvements to our frontend tooling to call the backend API
* Added "soft delete" support for toolinfo records
* Made various improvements in the functionality and reporting for the
URL crawler
The frontend for history, diffs, and rollback/revert of toolinfo
revisions is under review and expected to be merged in the coming
week. We also expect to solidify our April-June planning and update
our target release date following additional discussion next week.
Thanks to everyone who is following along on our progress and giving
us support with encouraging words and helpful ideas. We continue to be
very excited about the impact that this project will have on the
Wikimedia community and are looking forward to its release. :)
[0]: https://meta.wikimedia.org/wiki/Toolhub/Progress_reports/2021-04-02
Bryan
--
Bryan Davis Technical Engagement Wikimedia Foundation
Principal Software Engineer Boise, ID USA
[[m:User:BDavis_(WMF)]] irc: bd808
Report on activities in the Toolhub project for the week ending 2021-03-26.
See the on-wiki progress report [0] for additional details and clickable links.
== TL;DR ==
* Editing of full toolinfo records merged and deployed to demo server
* Work started on display of edit history and diffs
* Crawler updated to delete stale records and avoid edit wars between URLs
* Some data transforms for human consumption moved from backend to frontend
* Initial research done on Elastic Common Schema debug log formatting
* Thinking about April-June goals for the project
The January-March quarter of planned work will end mid-week next week.
Our remaining named goal for the quarter is nearly complete. It is
possible that work will be completed before the arbitrary 2021-03-31
end date, but it may also extend into the early days of the next
quarter. Either way is fine, we are pushing ourselves to move forward
but are also more concerned with doing things well than exact
timelines.
Bryan and Srishti will be attempting to finalize plans for work in the
April-June quarter in the coming week. In our high level planning for
the July 2020 - June 2021 fiscal year we projected deploying a "1.0"
version of Toolhub by the end of June 2021. One of the questions that
will be examined next week is if the more important part of that goal
is the release date or the feature set. We have three more planned
sets of features on the roadmap: lists of tools, annotations
(community maintained notes/details for tools), and moderation and
patrolling. We must implement moderation and patrolling support before
the 1.0 release. The other two features are also strongly desired, but
could easily be reprioritized as 1.x follow up features to be deployed
after the initial launch.
[0]: https://meta.wikimedia.org/wiki/Toolhub/Progress_reports/2021-03-26
Bryan
--
Bryan Davis Technical Engagement Wikimedia Foundation
Principal Software Engineer Boise, ID USA
[[m:User:BDavis_(WMF)]] irc: bd808
Report on activities in the Toolhub project for the week ending 2021-03-19.
See the on-wiki progress report [0] for additional details and clickable links.
== TL;DR ==
* Work continues to enable editing of a full toolinfo record
* Several smaller issues have been fixed or are in review
Work continues on our remaining named goal of direct tool
registration. It is still difficult to say with complete certainty
whether this work will be entirely complete on March 31, but we feel
reasonably confident that if it is not done by then it will be
completed very early in April.
We will soon begin formalizing our goals for the final quarter of the
Wikimedia Foundation's 2020-2021 fiscal year. Our ideal goal has been
to reach a stable "1.0" level of functionality, receive a security
review, and deploy into production by the end of June 2021. Over the
next couple of weeks we need to estimate and prioritize the remaining
roadmap features. This will help us decide if our ideal goal is still
within reach or if we will need to either defer a feature or push the
deployment date into July 2021.
[0]: https://meta.wikimedia.org/wiki/Toolhub/Progress_reports/2021-03-12
Bryan
--
Bryan Davis Technical Engagement Wikimedia Foundation
Principal Software Engineer Boise, ID USA
[[m:User:BDavis_(WMF)]] irc: bd808
Report on activities in the Toolhub project for the week ending 2021-03-12.
See the on-wiki progress report [0] for additional details and clickable links.
== TL;DR ==
* Faceted search live on demo server!
* Direct tool registration making progress
* New demo server setup created to support search
We have search! With this functionality in the core product we have
reached and exceeded feature parity with Hay's Directory. Bryan has
been doing some comparison searches using the two tools and found that
the results are comparable. In some instances Hay's Directory will
return more results than Toolhub, but thus far each of these extra
results has proven to be information for a tool that is no longer
active on Toolforge. This also marks completion of the second of three
named goals for our January-March quarter.
Work is likely to continue on direct tool registration through the end
of March, and possibly into the early part of April. We will be
spending some time in the coming weeks to prioritize the planned work
remaining to reach our "1.0" feature targets.
[0]: https://meta.wikimedia.org/wiki/Toolhub/Progress_reports/2021-03-12
Bryan
--
Bryan Davis Technical Engagement Wikimedia Foundation
Principal Software Engineer Boise, ID USA
[[m:User:BDavis_(WMF)]] irc: bd808
Hey folks,
Peter Pelberg joined our mailing list today. Peter is a Product
Manager at the Foundation working with the Editing team. I have not
talked to Peter directly about his reasons for joining us, but I'll
make an educated guess that he saw Sristi and I present about Toolhub
at yesterday's Foundation staff meeting and wanted to learn more.
Everyone, please do remember that this list is a part of the Wikimedia
technical spaces and thus covered by the
<https://www.mediawiki.org/wiki/Code_of_Conduct>. The archives of the
list are also publicly available, so keep that in mind as well when
posting here. As long as we stick to the topic of the list, developing
Toohub, I think everyone will be fine. :)
Bryan
--
Bryan Davis Technical Engagement Wikimedia Foundation
Principal Software Engineer Boise, ID USA
[[m:User:BDavis_(WMF)]] irc: bd808
Report on activities in the Toolhub project for the week ending 2021-03-05.
See the on-wiki progress report [0] for additional details and clickable links.
== TL;DR ==
* UI to create new toolinfo record merged
* Code review feedback to address in search UI
* Patches from Eran and Reedy merged
* Crawler made more tolerant of extra data in toolinfo.json
* Keywords case folded to lower case
* Thinking about end user documentation and production deployment
I'm worried that I sound like a boring cheerleader, but I feel that
this was another great week for Toolhub progress. Srishti did some
really solid work on updating her tool creation patch, and I think she
found patterns that we will be able to repeat to make building editing
forms easier. We finally got our chance to present to the collective
Foundation staff, and the initial feedback seems very positive. We
have a lot of things to keep us busy building UI screens through the
end of the month/quarter, but it is looking possible to actually get
them all done. :)
One of my personal goals for next week is getting a new build of
Toolhub pushed out to the demo server
(<https://toolhub-demo.wmcloud.org/>) so that y'all can see the
progress that we have made in weeks since the current build deployed
there was created. I'm so excited to see search working locally, and I
really want the rest of you to be able to play with it too.
[0]: https://meta.wikimedia.org/wiki/Toolhub/Progress_reports/2021-03-05
Bryan
--
Bryan Davis Technical Engagement Wikimedia Foundation
Principal Software Engineer Boise, ID USA
[[m:User:BDavis_(WMF)]] irc: bd808
Report on activities in the Toolhub project for the week ending 2021-02-26.
See the on-wiki progress report [0] for additional details and clickable links.
== TL;DR ==
* Unexpectedly quick progress on faceted search!
* Work continues on direct tool registration
The faceted search feature has a work in progress UI in gerrit
already! A combination of the backend being easier than expected and
Bryan's burning desire to see an actual working search with facet
display and drill-down made this quick burst possible. We should
probably also be begrudgingly thankful for lockdown boredom as a
contributing factor. :)
The unexpected speed of getting a working UI for search is encouraging
for our quarterly goals. We still have quite a bit of work left to
implement user interfaces for the toolinfo registration, editing, and
history APIs, but with the vast majority of backend work to support
these done Bryan can start helping more with UI work in the last month
of the quarter.
[0]: https://meta.wikimedia.org/wiki/Toolhub/Progress_reports/2021-02-19
Bryan
--
Bryan Davis Technical Engagement Wikimedia Foundation
Principal Software Engineer Boise, ID USA
[[m:User:BDavis_(WMF)]] irc: bd808
Report on activities in the Toolhub project for the week ending 2021-02-19.
See the on-wiki progress report [0] for additional details and clickable links.
== TL;DR ==
* Revert and undo actions added to history/diff API
* Work started on faceted search
* Various smaller bug/feature fixes
The big news this week is work starting in earnest on the faceted
search functionality. As documented in more detail on meta, this work
is using Elasticsearch as the backend search service. This has some
risk due to the recent relicensing of new versions of Elasticsearch
under the gratis but not libre Server Side Public License. Bryan is
part of a working group inside the Technology department that is
already looking into the future of Elasticsearch for Wikimedia
projects. At this time, it seems best to continue with the prior plan
to use Elasticsearch for Toolhub--even if that means a rewrite prior
to production launch--rather than hold all work until we find out what
the working group recommends.
If y'all have strong opinions about this decision that you would like
to share, please do send them either to this list or directly to
Bryan.
[0]: https://meta.wikimedia.org/wiki/Toolhub/Progress_reports/2021-02-19
Bryan
--
Bryan Davis Technical Engagement Wikimedia Foundation
Principal Software Engineer Boise, ID USA
[[m:User:BDavis_(WMF)]] irc: bd808
Report on activities in the Toolhub project for the week ending 2021-02-12.
See the on-wiki progress report [0] for additional details and clickable links.
== TL;DR ==
* Security Readiness Review for Toolhub planned for Q4
* Detour to fix tech debt because of pyca/cryptography packaging changes
* API for history and diff of toolinfo records merged
* Direct tool registration UI work started
* Various small bug/feature fixes
See <https://meta.wikimedia.org/wiki/Toolhub/Progress_reports/2021-02-12#Fun_tim…>
for what Bryan ended up spending time on this week. Our drama was
pretty minor compared to the frustrations that other projects felt
[1][2]. It's hard to blame the upstream for moving forward, but it is
really tricky to keep track of the planning boards of the long chain
of indirect dependencies in a project like Toolhub.
Srishti will be taking some well deserved time off next week, so there
will probably not be a large amount of progress on the toolinfo
creation screens in next week's report.
[0]: https://meta.wikimedia.org/wiki/Toolhub/Progress_reports/2021-02-12
[1]: https://github.com/pyca/cryptography/issues/5771
[2]: https://github.com/PyO3/pyo3/issues/1420
Bryan
--
Bryan Davis Technical Engagement Wikimedia Foundation
Principal Software Engineer Boise, ID USA
[[m:User:BDavis_(WMF)]] irc: bd808
Report on activities in the Toolhub project for the week ending 2021-02-05.
See the on-wiki progress report [0] for additional details and clickable links.
== TL;DR ==
* Toolhub API OAuth fully implemented
* Tracking for per-edit changes to tool information implemented
* SPDX license identifier API and validation added
* API error response format documented in OpenAPI spec
* Direct tool registration UI wireframes created and reviewed
* API for history and diff of toolinfo records in progress
* Toolhub advisory council members asked to stay involved through June 2021
We were able to complete the first of our three major goals for the
January-March quarter of planned work this week with the resolution of
Toolhub API OAuth.
Bryan estimates that the backend APIs for our second major goal of
Direct tool registration are 60% complete, and only blocked by
competing time commitments. Srishti is just starting work on the UI
components to go with this backend.
It is still too early to know if we will complete all of our goal work
in the quarter, but things are looking good at the moment. :)
The presentation to Foundation staff that was planned for 2021-02-04
has been postponed until 2021-03-04. It turns out that there was some
more pressing news [1] that preempted the February staff meeting's
planned schedule. The good news here for Toolhub is that we should
have even more features to show off in March!
[0]: https://meta.wikimedia.org/wiki/Toolhub/Progress_reports/2021-02-05
[1]: https://lists.wikimedia.org/pipermail/wikimedia-l/2021-February/096166.html
Bryan
--
Bryan Davis Technical Engagement Wikimedia Foundation
Principal Software Engineer Boise, ID USA
[[m:User:BDavis_(WMF)]] irc: bd808