Huge thanks for this. As someone who comes into contact with for example the WMF infrastructure but irregularly, having these diagrams somewhat up to date just to be able to follow along in conversations is very helpful.

//Johan Jönsson --

On Mon, Oct 24, 2022 at 3:51 PM Krinkle krinkle@fastmail.com wrote:

I've done a major update to a number of diagrams on Wikitech.

Usually, I don't mention an update here, but I'm highlighting it now as it's been a while since we mentioned them on-list and the community and foundation have grown a lot so some of these may be new to you.

Given how much has changed in recent changes, I also included a changelog and a link to where in the docs you'd normally discover this diagram on-wiki:

*== 1. File:Wikipedia_webrequest_2022.png https://wikitech.wikimedia.org/wiki/File:Wikipedia_webrequest_2022.png (Updated) ==*

This is a highly simplified diagram, covering the general shape of our stack through the example of a typical Wikipedia webrequest.

Previous: https://upload.wikimedia.org/wikipedia/commons/b/b3/Wikipedia_webrequest_flo... New: https://upload.wikimedia.org/wikipedia/commons/4/4d/Wikipedia_webrequest_202... Documentation: wikitech:MediaWiki_at_WMF https://wikitech.wikimedia.org/wiki/MediaWiki_at_WMF and wikitech:Caching_overview https://wikitech.wikimedia.org/wiki/Caching_overview. Notable changes:

• Change edge TLS termination ("HTTPS") from ats-tls to HAProxy. I wrote a

"Caching overview § History https://wikitech.wikimedia.org/wiki/Caching_overview" section.

• Change appserver TLS from Nginx- to Envoy.
• Add new MainStash DB.
• Include storage ExternalStore DB, ParserCache DB, and Swift media.
• Include services Shellbox, Mathoid, and Kask.

*== 2. File:WMF_infrastructure_2022.png https://wikitech.wikimedia.org/wiki/File:WMF_infrastructure_2022.png (Updated) ==*

This is a continous attempt at an overview of tier-1/user-facing infrastructure. It will likely never be complete from all POV, but.. it is more accurate and complete than it has been. Thanks to all that contributed by entertaining my many questions over the years.

Previous (2016 by Elukey): https://upload.wikimedia.org/wikipedia/labs/4/4d/Infrastructure_overview.png New: https://upload.wikimedia.org/wikipedia/commons/4/48/WMF_infrastructure_2022.... Documentation: wikitech:Wikimedia_infrastructure https://wikitech.wikimedia.org/wiki/Wikimedia_infrastructure and wikitech:Purged https://wikitech.wikimedia.org/wiki/Purged Notable changes:

• Add new Drmrs data center in Marseille, France.
• Add new services: purged.go, EventStreams, Thumbor, mcrouter, Envoy,

etcd.

• Add new distinction for Multi-DC between primary and secondary data

center.

• Change sessionstore from Redis to Kask/Cassandra.
• Change jobqueue from Redis to EventGate/Kafka.
• Include distinct MediaWiki server roles and clusters.
• Include high-level MediaWiki platform components.
• Include example flow for "JobQueue job" and "CDN purge".

*== 3. File:MediaWiki_infrastructure_2022.png https://wikitech.wikimedia.org/wiki/File:MediaWiki_infrastructure_2022.png (New) ==*

Similar to WMF Infra diagram, but more abstract around DC and services, and more detailed within the platform. Including more core services, and recognising extensions as their own layer.

New: https://upload.wikimedia.org/wikipedia/commons/e/ee/MediaWiki_infrastructure... Documentation: wikitech:MediaWiki_at_WMF https://wikitech.wikimedia.org/wiki/MediaWiki_at_WMF

*== 4. File:Wikipedia_Memcached_flow_2022.png https://wikitech.wikimedia.org/wiki/File:Wikipedia_Memcached_flow_2022.png (Updated)*

Previous: https://upload.wikimedia.org/wikipedia/commons/d/db/Wikipedia_Memcached_flow... New: https://upload.wikimedia.org/wikipedia/commons/4/45/Wikipedia_Memcached_flow... Documentation: wikitech:Memcached_for_MediaWiki https://wikitech.wikimedia.org/wiki/Memcached_for_MediaWiki Notable changes:

• Include the three tiers of ParserCache.
• Add WANCache legend to explain different keytypes you may encounter on

the network.

• Add full name of the mcrouter-with-onhost-tier service for greppability.
• Add new WRStats service (T310662

https://phabricator.wikimedia.org/T310662). This was part of Multi-DC work https://www.mediawiki.org/wiki/Wikimedia_Performance_Team/Multi-DC_MediaWiki to reduce primary DB writes and (not bi-di replicated) Redis use in AbuseFilter. This service also replaces the old "User ping limiter" in core and is now able to serve both use cases.

• Remove "on-host: soon" labels. Adopting on-host memc for WANCache was

considered not worth the added runtime complexity (T264604 https://phabricator.wikimedia.org/T264604). Note that SRE's work on adding 10G network links for memcached hosts, and the addition of mcrouter-managed gutter pools take care of the general usecase that we were exploring on-host for. We kept it for ParserCache however (T244340 https://phabricator.wikimedia.org/T244340.

*== Edit link ==*

As before, each diagram file page has an "Edit" link in the description that takes you directly to the open-source Diagrams.net web app (loading file read-only from Google Drive). You can fork by using "Save as" in the web app. See also: wikitech:Performance/Runbook/diagrams.net_conventions https://wikitech.wikimedia.org/wiki/Performance/Runbook/diagrams.net_conventions

-- Timo

---

Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

Thanks so much, Timo! Having these snapshots of our systems and infrastructure, and also capturing how it evolves over time, is invaluable.

-selena

On Mon, Oct 24, 2022 at 6:51 AM Krinkle krinkle@fastmail.com wrote:

I've done a major update to a number of diagrams on Wikitech.

Usually, I don't mention an update here, but I'm highlighting it now as it's been a while since we mentioned them on-list and the community and foundation have grown a lot so some of these may be new to you.

Given how much has changed in recent changes, I also included a changelog and a link to where in the docs you'd normally discover this diagram on-wiki:

*== 1. File:Wikipedia_webrequest_2022.png https://wikitech.wikimedia.org/wiki/File:Wikipedia_webrequest_2022.png (Updated) ==*

This is a highly simplified diagram, covering the general shape of our stack through the example of a typical Wikipedia webrequest.

Previous: https://upload.wikimedia.org/wikipedia/commons/b/b3/Wikipedia_webrequest_flo... New: https://upload.wikimedia.org/wikipedia/commons/4/4d/Wikipedia_webrequest_202... Documentation: wikitech:MediaWiki_at_WMF https://wikitech.wikimedia.org/wiki/MediaWiki_at_WMF and wikitech:Caching_overview https://wikitech.wikimedia.org/wiki/Caching_overview. Notable changes:

• Change edge TLS termination ("HTTPS") from ats-tls to HAProxy. I wrote a

"Caching overview § History https://wikitech.wikimedia.org/wiki/Caching_overview" section.

• Change appserver TLS from Nginx- to Envoy.
• Add new MainStash DB.
• Include storage ExternalStore DB, ParserCache DB, and Swift media.
• Include services Shellbox, Mathoid, and Kask.

*== 2. File:WMF_infrastructure_2022.png https://wikitech.wikimedia.org/wiki/File:WMF_infrastructure_2022.png (Updated) ==*

This is a continous attempt at an overview of tier-1/user-facing infrastructure. It will likely never be complete from all POV, but.. it is more accurate and complete than it has been. Thanks to all that contributed by entertaining my many questions over the years.

Previous (2016 by Elukey): https://upload.wikimedia.org/wikipedia/labs/4/4d/Infrastructure_overview.png New: https://upload.wikimedia.org/wikipedia/commons/4/48/WMF_infrastructure_2022.... Documentation: wikitech:Wikimedia_infrastructure https://wikitech.wikimedia.org/wiki/Wikimedia_infrastructure and wikitech:Purged https://wikitech.wikimedia.org/wiki/Purged Notable changes:

• Add new Drmrs data center in Marseille, France.
• Add new services: purged.go, EventStreams, Thumbor, mcrouter, Envoy,

etcd.

• Add new distinction for Multi-DC between primary and secondary data

center.

• Change sessionstore from Redis to Kask/Cassandra.
• Change jobqueue from Redis to EventGate/Kafka.
• Include distinct MediaWiki server roles and clusters.
• Include high-level MediaWiki platform components.
• Include example flow for "JobQueue job" and "CDN purge".

*== 3. File:MediaWiki_infrastructure_2022.png https://wikitech.wikimedia.org/wiki/File:MediaWiki_infrastructure_2022.png (New) ==*

Similar to WMF Infra diagram, but more abstract around DC and services, and more detailed within the platform. Including more core services, and recognising extensions as their own layer.

New: https://upload.wikimedia.org/wikipedia/commons/e/ee/MediaWiki_infrastructure... Documentation: wikitech:MediaWiki_at_WMF https://wikitech.wikimedia.org/wiki/MediaWiki_at_WMF

*== 4. File:Wikipedia_Memcached_flow_2022.png https://wikitech.wikimedia.org/wiki/File:Wikipedia_Memcached_flow_2022.png (Updated)*

Previous: https://upload.wikimedia.org/wikipedia/commons/d/db/Wikipedia_Memcached_flow... New: https://upload.wikimedia.org/wikipedia/commons/4/45/Wikipedia_Memcached_flow... Documentation: wikitech:Memcached_for_MediaWiki https://wikitech.wikimedia.org/wiki/Memcached_for_MediaWiki Notable changes:

• Include the three tiers of ParserCache.
• Add WANCache legend to explain different keytypes you may encounter on

the network.

• Add full name of the mcrouter-with-onhost-tier service for greppability.
• Add new WRStats service (T310662

https://phabricator.wikimedia.org/T310662). This was part of Multi-DC work https://www.mediawiki.org/wiki/Wikimedia_Performance_Team/Multi-DC_MediaWiki to reduce primary DB writes and (not bi-di replicated) Redis use in AbuseFilter. This service also replaces the old "User ping limiter" in core and is now able to serve both use cases.

• Remove "on-host: soon" labels. Adopting on-host memc for WANCache was

considered not worth the added runtime complexity (T264604 https://phabricator.wikimedia.org/T264604). Note that SRE's work on adding 10G network links for memcached hosts, and the addition of mcrouter-managed gutter pools take care of the general usecase that we were exploring on-host for. We kept it for ParserCache however (T244340 https://phabricator.wikimedia.org/T244340.

*== Edit link ==*

As before, each diagram file page has an "Edit" link in the description that takes you directly to the open-source Diagrams.net web app (loading file read-only from Google Drive). You can fork by using "Save as" in the web app. See also: wikitech:Performance/Runbook/diagrams.net_conventions https://wikitech.wikimedia.org/wiki/Performance/Runbook/diagrams.net_conventions

-- Timo

---

Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/