I've done a major update to a number of diagrams on Wikitech.
Usually, I don't mention an update here, but I'm highlighting it now as it's
been a while since we mentioned them on-list and the community and foundation have grown a
lot so some of these may be new to you.
Given how much has changed in recent changes, I also included a changelog and a link to
where in the docs you'd normally discover this diagram on-wiki:
*== 1. File:Wikipedia_webrequest_2022.png
<https://wikitech.wikimedia.org/wiki/File:Wikipedia_webrequest_2022.png> (Updated)
==*
This is a highly simplified diagram, covering the general shape of our stack through the
example of a typical Wikipedia webrequest.
**
Previous:
https://upload.wikimedia.org/wikipedia/commons/b/b3/Wikipedia_webrequest_fl…
New:
https://upload.wikimedia.org/wikipedia/commons/4/4d/Wikipedia_webrequest_20…
Documentation: wikitech:MediaWiki_at_WMF
<https://wikitech.wikimedia.org/wiki/MediaWiki_at_WMF> and wikitech:Caching_overview
<https://wikitech.wikimedia.org/wiki/Caching_overview>.
Notable changes:
* Change edge TLS termination ("HTTPS") from ats-tls to HAProxy. I wrote a
"Caching overview § History
<https://wikitech.wikimedia.org/wiki/Caching_overview>" section.
* Change appserver TLS from Nginx- to Envoy.
* Add new MainStash DB.
* Include storage ExternalStore DB, ParserCache DB, and Swift media.
* Include services Shellbox, Mathoid, and Kask.
*== 2. File:WMF_infrastructure_2022.png
<https://wikitech.wikimedia.org/wiki/File:WMF_infrastructure_2022.png> *(Updated)
==**
This is a continous attempt at an overview of tier-1/user-facing infrastructure. It will
likely never be complete from all POV, but.. it is more accurate and complete than it has
been. Thanks to all that contributed by entertaining my many questions over the years.
****
Previous (2016 by Elukey):
https://upload.wikimedia.org/wikipedia/labs/4/4d/Infrastructure_overview.png
New:
https://upload.wikimedia.org/wikipedia/commons/4/48/WMF_infrastructure_2022…
Documentation: wikitech:Wikimedia_infrastructure
<https://wikitech.wikimedia.org/wiki/Wikimedia_infrastructure> and wikitech:Purged
<https://wikitech.wikimedia.org/wiki/Purged>
Notable changes:
* Add new Drmrs data center in Marseille, France.
* Add new services: purged.go, EventStreams, Thumbor, mcrouter, Envoy, etcd.
* Add new distinction for Multi-DC between primary and secondary data center.
* Change sessionstore from Redis to Kask/Cassandra.
* Change jobqueue from Redis to EventGate/Kafka.
* Include distinct MediaWiki server roles and clusters.
* Include high-level MediaWiki platform components.
* Include example flow for "JobQueue job" and "CDN purge".
*== 3. File:MediaWiki_infrastructure_2022.png
<https://wikitech.wikimedia.org/wiki/File:MediaWiki_infrastructure_2022.png> (New)
==*
Similar to WMF Infra diagram, but more abstract around DC and services, and more detailed
within the platform. Including more core services, and recognising extensions as their own
layer.
**
New:
https://upload.wikimedia.org/wikipedia/commons/e/ee/MediaWiki_infrastructur…
Documentation: wikitech:MediaWiki_at_WMF
<https://wikitech.wikimedia.org/wiki/MediaWiki_at_WMF>
*== 4. File:Wikipedia_Memcached_flow_2022.png
<https://wikitech.wikimedia.org/wiki/File:Wikipedia_Memcached_flow_2022.png>
*(Updated)**
****
Previous:
https://upload.wikimedia.org/wikipedia/commons/d/db/Wikipedia_Memcached_flo…
New:
https://upload.wikimedia.org/wikipedia/commons/4/45/Wikipedia_Memcached_flo…
Documentation: wikitech:Memcached_for_MediaWiki
<https://wikitech.wikimedia.org/wiki/Memcached_for_MediaWiki>
Notable changes:
* Include the three tiers of ParserCache.
* Add WANCache legend to explain different keytypes you may encounter on the network.
* Add full name of the mcrouter-with-onhost-tier service for greppability.
* Add new WRStats service (T310662 <https://phabricator.wikimedia.org/T310662>).
This was part of Multi-DC work
<https://www.mediawiki.org/wiki/Wikimedia_Performance_Team/Multi-DC_MediaWiki> to
reduce primary DB writes and (not bi-di replicated) Redis use in AbuseFilter. This service
also replaces the old "User ping limiter" in core and is now able to serve both
use cases.
* Remove "on-host: soon" labels. Adopting on-host memc for WANCache was
considered not worth the added runtime complexity (T264604
<https://phabricator.wikimedia.org/T264604>). Note that SRE's work on adding 10G
network links for memcached hosts, and the addition of mcrouter-managed gutter pools take
care of the general usecase that we were exploring on-host for. We kept it for ParserCache
however (T244340 <https://phabricator.wikimedia.org/T244340>.
*== Edit link ==*
As before, each diagram file page has an "Edit" link in the description that
takes you directly to the open-source
Diagrams.net web app (loading file read-only from
Google Drive). You can fork by using "Save as" in the web app. See also:
wikitech:Performance/Runbook/diagrams.net_conventions
<https://wikitech.wikimedia.org/wiki/Performance/Runbook/diagrams.net_conventions>
-- Timo