You may also find these diagrams useful:
https://commons.wikimedia.org/wiki/File:Wikipedia_webrequest_flow_2015-10.pn...
https://wikitech.wikimedia.org/wiki/File:Infrastructure_overview.png
-Chad On Feb 24, 2016 6:58 PM, "Mukunda Modell" mmodell@wikimedia.org wrote:
On Feb 17, 2016 1:50 AM, "Guillaume Lederrey" glederrey@wikimedia.org wrote:
- I still have not found a global architecture schema (something like
a high level component or deplyoment diagram). But I have never seen any company having those...
I made a diagram of the scap (mediawiki) deployment architecture a while back: https://commons.wikimedia.org/wiki/File:Scap-diagram.png ..
That does not exactly apply to the new scap3 architecture but it's not too far off.
....
On Thu, Feb 18, 2016 at 10:37 AM, Giuseppe Lavagetto glavagetto@wikimedia.org wrote:
About cherry-picks in beta: the problem is not cherry-picking (I think it's a reasonable way to test things) but persistent cherry-picking to monkey patch problems is. I think if we follow the flow of:
- writing a patch
- testing it on beta with a cherry-pick
- get it merged on ops/puppet and production
There are a lot of patches on beta these days and there have been a lot of different people cherry-picking without much coordination. This has lead to breakage quite often. Patches also get lost regularly. I assume this usually happens because someone has rebased the HEAD and accidentally dropped a patch.
It can be really difficult to get a patch merged in ops/puppet within a week (or even a month). I've seen a lot of patches sit around for weeks and even now with the Puppet SWAT windows, it's still sometimes unrealistic to expect patches get merged into production that quickly. (+CC Tyler)
Without a system to manage things, and with very little coordination between everyone who is working on beta, I don't expect the situation to improve too much.
I intend to propose a solution for beta & puppet patch cherry-picks very soon, however, I haven't fully formulated my proposal yet. I will write to the ops list when I have something written in a clear and presentable way.
Ops mailing list Ops@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/ops
I created 2 diagrams about RCStream and IRC change logs. Those are meant to be a medium for my analysis of those flow [1], they are not meant to be reference documentation. They are much uglier than the other diagrams I have seen, but who knows, maybe they will interest someone...
* https://wikitech.wikimedia.org/wiki/File:Irc-rcstream-sequence.svg * https://wikitech.wikimedia.org/wiki/File:Irc-rcstream-deployment.svg
[1] https://phabricator.wikimedia.org/T126472
On Thu, Feb 25, 2016 at 8:16 AM, Chad Horohoe chorohoe@wikimedia.org wrote:
You may also find these diagrams useful:
https://commons.wikimedia.org/wiki/File:Wikipedia_webrequest_flow_2015-10.pn...
https://wikitech.wikimedia.org/wiki/File:Infrastructure_overview.png
-Chad
On Feb 24, 2016 6:58 PM, "Mukunda Modell" mmodell@wikimedia.org wrote:
On Feb 17, 2016 1:50 AM, "Guillaume Lederrey" glederrey@wikimedia.org wrote:
- I still have not found a global architecture schema (something like
a high level component or deplyoment diagram). But I have never seen any company having those...
I made a diagram of the scap (mediawiki) deployment architecture a while back: https://commons.wikimedia.org/wiki/File:Scap-diagram.png ..
That does not exactly apply to the new scap3 architecture but it's not too far off.
....
On Thu, Feb 18, 2016 at 10:37 AM, Giuseppe Lavagetto glavagetto@wikimedia.org wrote:
About cherry-picks in beta: the problem is not cherry-picking (I think it's a reasonable way to test things) but persistent cherry-picking to monkey patch problems is. I think if we follow the flow of:
- writing a patch
- testing it on beta with a cherry-pick
- get it merged on ops/puppet and production
There are a lot of patches on beta these days and there have been a lot of different people cherry-picking without much coordination. This has lead to breakage quite often. Patches also get lost regularly. I assume this usually happens because someone has rebased the HEAD and accidentally dropped a patch.
It can be really difficult to get a patch merged in ops/puppet within a week (or even a month). I've seen a lot of patches sit around for weeks and even now with the Puppet SWAT windows, it's still sometimes unrealistic to expect patches get merged into production that quickly. (+CC Tyler)
Without a system to manage things, and with very little coordination between everyone who is working on beta, I don't expect the situation to improve too much.
I intend to propose a solution for beta & puppet patch cherry-picks very soon, however, I haven't fully formulated my proposal yet. I will write to the ops list when I have something written in a clear and presentable way.
Ops mailing list Ops@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/ops
On Wed, Feb 24, 2016 at 11:16 PM, Chad Horohoe chorohoe@wikimedia.org wrote:
You may also find these diagrams useful:
https://commons.wikimedia.org/wiki/File:Wikipedia_webrequest_flow_2015-10.pn...
I don't know the source of this image, but there are a couple corrections that could be made. EventLogging comes from either the web or the MediaWiki instances. I believe there was a recent change so MediaWiki instances produce EventLogging data by hitting an http endpoint much like the web based EventLogging does. The EventLogging data then flows into Kafka. From there it goes somewhere (not sure what happens here) and ends up in the analytics DB servers, along with HDFS.
I may be missing a few steps, and this may be partially incorrect. But it's my current understanding.
Additionally CirrusSearch logs flow from MediaWiki into kafaka. Very soon API logs will also start flowing from MediaWiki into Kafka. Yet another use case is that EventBus will soon (or already does?) ship events about edit's from MediaWiki into Kafka. Initially those EventBus events will be read out of kafka by RestBase.
https://wikitech.wikimedia.org/wiki/File:Infrastructure_overview.png
-Chad On Feb 24, 2016 6:58 PM, "Mukunda Modell" mmodell@wikimedia.org wrote:
On Feb 17, 2016 1:50 AM, "Guillaume Lederrey" <glederrey@wikimedia.org
wrote:
- I still have not found a global architecture schema (something like
a high level component or deplyoment diagram). But I have never seen any company having those...
I made a diagram of the scap (mediawiki) deployment architecture a while back: https://commons.wikimedia.org/wiki/File:Scap-diagram.png ..
That does not exactly apply to the new scap3 architecture but it's not too far off.
....
On Thu, Feb 18, 2016 at 10:37 AM, Giuseppe Lavagetto glavagetto@wikimedia.org wrote:
About cherry-picks in beta: the problem is not cherry-picking (I think it's a reasonable way to test things) but persistent cherry-picking to monkey patch problems is. I think if we follow the flow of:
- writing a patch
- testing it on beta with a cherry-pick
- get it merged on ops/puppet and production
There are a lot of patches on beta these days and there have been a lot of different people cherry-picking without much coordination. This has lead to breakage quite often. Patches also get lost regularly. I assume this usually happens because someone has rebased the HEAD and accidentally dropped a patch.
It can be really difficult to get a patch merged in ops/puppet within a week (or even a month). I've seen a lot of patches sit around for weeks and even now with the Puppet SWAT windows, it's still sometimes unrealistic to expect patches get merged into production that quickly. (+CC Tyler)
Without a system to manage things, and with very little coordination between everyone who is working on beta, I don't expect the situation to improve too much.
I intend to propose a solution for beta & puppet patch cherry-picks very soon, however, I haven't fully formulated my proposal yet. I will write to the ops list when I have something written in a clear and presentable way.
Ops mailing list Ops@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/ops
discovery mailing list discovery@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/discovery