TLDR: fresh-node now uses Node 18. To learn more or install/upgrade, refer to
https://gerrit.wikimedia.org/g/fresh.
Hi all,
Fresh 23.08.1 is upon is. It's a fairly big release!
*What's new?*
After a delay of more than two years, Node.js 18 and npm 9 (replacing Node.js 16 and npm
7) are available to WMF CI as of yesterday, and can now be used locally through Fresh!
We've also added early support for Node.js 20, you can opt-in via the fresh-node20
command.
The default fresh-node command was updated to Node.js 18. You can continue to use older
versions for another 6-12 months via the fresh-node16 and fresh-node14 commands. The
fresh-node12 command has been removed (unsupported since last year
<https://github.com/nodejs/Release#end-of-life-releases>).
We've also made a few minor tweaks to improve support for Podman
<https://podman.io/> to encourage competition and use of freely-licensed software
(Docker for Mac/Windows require the proprietary Docker Desktop). We've also improved
experimental support for Apple ARM-based devices. Node, npm, and Firefox work out of the
box using Docker's default emulation (including the faster Rosetta-based emulation).
For example, using `npx grunt karma:firefox`. Chrome has yet to support emulation within
Docker.
Full changelog:
https://gerrit.wikimedia.org/g/fresh/+/23.08.1/CHANGELOG.md
Commits:
https://gerrit.wikimedia.org/g/fresh/+log/23.08.1/
*What was the hold up?*
The updating and creation of new Docker images for WMF CI is rather simple — involving
little more than a directory copy or a one-line change (example 1
<https://gerrit.wikimedia.org/r/c/integration/config/+/891648/1/dockerfiles/node16/Dockerfile.template>,
example 2, <https://gerrit.wikimedia.org/r/c/integration/config/+/894125/3> example
3 <https://gerrit.wikimedia.org/r/c/integration/config/+/946960>). We run a very
efficient operation here, with people like James Forrester, Antoine Musso, and myself
routinely migrating the entire Wikimedia movement's CI workloads in mere minutes! We
use Zuul to automatically churn through hundreds of Jenkins jobs, used on a daily basis by
thousands of Gerrit's Git repositories. It's quite the feat of automation
<https://www.mediawiki.org/wiki/Continuous_integration/Docker> really.
The holdup was in upgrading the browser tests of several product features to Webdriver.io
v7. (Browser tests are sometimes associated with "selenium" for hysterical
raisins, they involve no Selenium tech today.)
Webdriver 6 and earlier use the deprecated Node.js Fibers
<https://www.npmjs.com/package/fibers> functionality to emulate async code as
synchronous code. This allowed our developers to write test cases using a simpler syntax
in which async-await statements could be omitted. The underlying functionality for this
was discontinued in Node.js 16. Thus until a team migrated their tests to Webdriver 7, the
associated CI pipeline would have to remain on Node.js 14.
One of the strengths of our WMF CI setup is that projects can bi-directionally integrate.
This is one of many capabilities that Zuul and a number of other large CI systems have
provided for over a decade, that e.g. GitHub/GitLab (still in their infancy around CI) are
only just starting to explore. Zuul allows MediaWiki core to ensure its changes can't
break extensions, and likewise allows each extension to ensure it can't break other
extensions that it is meant to work together with, e.g. the MediaWiki bundle, or WMF
production. We currently run this on a single Docker container, which supports local
testing without a new and different dev environment for each of the 1000+ extensions. This
is efficient from a big picture perspective, especially as a non-profit (though big
corporations seem to make the same choice). We've also designed our infastructure to
generally support two or three major versions simultanously, which allows individual teams
to schedule maintenance at their own convenience.
But, all this relies on the assumption that teams are aligned, that products in production
have an owner, and that owners discover and (eventually) prioritise routine maintenance
within 3-6 months.
Those assumptions allow for wide margins, they we don't meet those anymore, with
dozens of major capabilities and products having lapsed
<https://www.mediawiki.org/wiki/Developers/Maintainers> in ownership over the years
(or with existing ownership insufficiently excercised). In addition, the CI infrastructure
lacks a directly responsible individual. Thus team's upgrade tasks
<https://phabricator.wikimedia.org/T256626> can linger for years, with nobody
accountable for executing a chosen set of high-level priorities or knowing the needs of
individual MediaWiki development teams. Everybody is waiting for somebody else while we
continue to progress within our local maxima.
Our support window covered Node 12, 14 and 16. The volunteers maintaining the CI jobs
(James and myself in this case) choose not to start supporting Node 18 until we can start
dropping older stuff. Despite individual teams expressing
<https://phabricator.wikimedia.org/T314051> their blockage
<https://phabricator.wikimedia.org/T337647> and a desire to start working with newer
versions, there exists no incentive for anyone to act on this.
What resolved it in this case is that last week, I too became someone wanted to start
testing something on Node 18. And I'm not willing to maintain a custom dev env or CI
pipeline just for myself. So I spent several days identifying the four remaining projects
(analysis <https://phabricator.wikimedia.org/T256626#9063307>), upgrading each
project, communicating with teams at WMDE/WMF, turning off the remaining unmaintained
tests, dusting off James' patches for upgrading Quibble CI jobs from Node 14 to Node
16, adding Node 18, and finally releasing Fresh with Node 18 support.
*Welcome Release Engineering!*
We're transitioning ownership of Fresh from the Performance Team to the Release
Engineering team. This release was done in collaboration with Antoine Musso!
*What is Fresh?*
**
Fresh is a fast way to launch isolated environments from your terminal. These can be used
to work more securely and responsibly
<https://timotijhof.net/posts/2019/protect-yourself-from-npm/> with Node.js-based
developer tools, especially those installed from npm such as ESLint, QUnit, Grunt,
Webdriver, and more. Example guide:
https://www.mediawiki.org/wiki/Manual:JavaScript_unit_testing.
To file tasks or browse existing ones, check the Phabricator board at
https://phabricator.wikimedia.org/tag/fresh/.
--
Timo Tijhof,
Principal Engineer,
Wikimedia Foundation.