[Wikitech-l] Re: Score, Kubernetes and switching to Shellbox

Hi everyone, tl;dr: External shell outs are now run via Shellbox. Any deployed code needs to use Shellbox/BoxedCommand, and documentation is available to help migrate. To safely re-enable Score (LilyPond) on Wikimedia wikis, we developed Shellbox, a way to run shell commands in a remote, isolated container. This is (hopefully) a stronger level of isolation than we previously had with firejail, since it's relying on Linux containers and Kubernetes to do the isolation. At the same time, this helps us in moving towards running MediaWiki on Kubernetes, as we don't want to include all these external commands inside the MediaWiki container. For the most part, any new shelling out to external commands needs to be done via Shellbox. A lot of the design and rationale behind Shellbox is captured in the RfC: <https://phabricator.wikimedia.org/T260330>. In Wikimedia production, so far Score, Timeline, SyntaxHighlight and Wikidata constraint regex checking are all using Shellbox. Details about that and links to dashboards are available at <https://wikitech.wikimedia.org/wiki/Shellbox>. The main things that are left are media-handling code that extracts metadata: DjVu, PdfHandler and PagedTiffHandler, which is tracked at <https://phabricator.wikimedia.org/T289228>, and videoscaling (TimedMediaHandler). Some work has to be done in MediaWiki to make code compatible with Shellbox, specifically switching to "BoxedCommand", which now has its own documentation page: <https://www.mediawiki.org/wiki/Manual:BoxedCommand>. BoxedCommand works transparently whether you have a separate Shellbox service set up or not. This is the preferred way to write new shellouts going forward, though Shell::command() isn't officially deprecated yet. So far all shellouts that are used in Wikimedia production have already been converted except for TimedMediaHandler. Looking forward, I think this also gives us a lot of flexibility in using more external commands in the future. First, we're less tied to whatever OS version MediaWiki is running on, as long as it can be built/shipped in a container, we can use it. And secondly, it's probably OK if external commands aren't super well behaved (e.g. use too much memory) since they're no longer sharing the same resources as an appserver (this shouldn't be interpreted as a free pass for super inefficient stuff of course). I tried to keep this summary short, and am intending to write a longer blog post that explains some more history in detail. But if you have any questions or something isn't clear, please ask! -- Kunal