Setting up multiple Parsoid servers behind load balancer

List overview All Threads
Download

newer

older

Discussion on adding a...

Commons Android app update - v2.4.2

James Montalvo

7 Jun 2017 7 Jun '17

10:26 p.m.

I'm trying to setup two Parsoid servers to play nicely with two MediaWiki application servers and am having some issues. I have no problem getting things working with Parsoid on a single app server, or multiple Parsoid servers being used by a single app server, but ran into issues when I increased to multiple app servers. To try to get this working I starting making the app and Parsoid servers communicate through my load balancer. So an overview of my config is: Load balancer = 192.168.56.63 App1 = 192.168.56.80 App2 = 192.168.56.60 Parsoid1 = 192.168.56.80 Parsoid2 = 192.168.56.60 Note, App1 and Parsoid1 are the same server, and App2 and Parsoid2 are the same server. I can only spin up so many VMs on my laptop. The load balancer (HAProxy) is configured as follows: * 80 forwards to 443 * 443 forwards to App1 and App2 port 8080 * 8081 forwards to App1 and App2 port 8080 (this will be a private network connection later) * 8001 forwards to Parsoid1 and Parsoid2 port 8000 (also will be private) On App1/Parsoid1 I can run `curl 192.168.56.63:8001` and get the appropriate response from Parsoid. I can run `curl 192.168.56.63:8081` and get the appropriate response from MediaWiki. The same is true for both on App2/Parsoid2. So the servers can get the info they need from the services. Currently I'm getting a the error "Error loading data from server: 500: docserver-http: HTTP 500. Would you like to retry?" when attempting to use Visual Editor. I've tried various different settings and have not always gotten that specific error, but am getting it with the settings I currently have in localsettings.js and LocalSettings.php (shown below in this email). Removing the proxy config lines from these settings gave slightly better results. I did not get the 500 error, but instead it sometimes after a very long time it would work. It also may have been throwing errors in the parsoid log (with debug on). I have those logs saved if they help. I'm hoping someone can just point out some misconfiguration, though. Here are snippets of my config files: On App1/Parsoid1, relevant localsettings.js: parsoidConfig.setMwApi({ uri: 'http://192.168.56.80:8081/demo/api.php', proxy: { uri: 'http://192.168.56.80:8081/' }, domain: 'demo', prefix: 'demo' } ); parsoidConfig.serverInterface = '192.168.56.80'; On App2/Parsoid2, relevant localsettings.js: parsoidConfig.setMwApi({ uri: 'http://192.168.56.80:8081/demo/api.php', proxy: { uri: 'http://192.168.56.80:8081/' }, domain: 'demo', prefix: 'demo' } ); parsoidConfig.serverInterface = '192.168.56.60'; On App1/Parsoid1, relevant LocalSettings.php: $wgVirtualRestConfig['modules']['parsoid'] = array( 'url' => '192.168.56.80:8001', 'HTTPProxy' => 'http://192.168.56.80:8001', 'domain' => $wikiId, 'prefix' => $wikiId ); On App2/Parsoid2, relevant LocalSettings.php: $wgVirtualRestConfig['modules']['parsoid'] = array( 'url' => '192.168.56.80:8001', 'HTTPProxy' => 'http://192.168.56.80:8001', 'domain' => $wikiId, 'prefix' => $wikiId ); Thanks! --James

Show replies by date

C. Scott Ananian

8 Jun 8 Jun

1:11 a.m.

I think in general the first thing you should do for performance is set up restbase in front of parsoid? Caching the parsoid results will be faster than running multiple parsoids in parallel. That would also match the wmf configuration more closely, which would probably help us help you. I wrote up instructions for configuring restbase on the VE and Parsoid wiki pages. As it turns out I updated these today to use VRS configuration. Let me know if you run into trouble, perhaps some further minor updates are necessary. --scott On Jun 7, 2017 6:26 PM, "James Montalvo" <jamesmontalvo3(a)gmail.com> wrote:

...

James Montalvo

3:30 a.m.

Setting up RESTBase is very involved. I'd really prefer not to add that complexity at this time. Also I'm not sure at my scale RESTBase would provide much performance benefit (though I don't know much about it so that's just a hunch). The parsoid and VE configs have fields for proxy (as shown in my snippets), so it seems like running them this way is intended. Am I wrong? Thanks, James On Jun 7, 2017 8:12 PM, "C. Scott Ananian" <cananian(a)wikimedia.org> wrote:

...

an overview of my config is: Load balancer = 192.168.56.63 App1 = 192.168.56.80 App2 = 192.168.56.60 Parsoid1 = 192.168.56.80 Parsoid2 = 192.168.56.60 Note, App1 and Parsoid1 are the same server, and App2 and Parsoid2 are

the

same server. I can only spin up so many VMs on my laptop. The load balancer (HAProxy) is configured as follows: * 80 forwards to 443 * 443 forwards to App1 and App2 port 8080 * 8081 forwards to App1 and App2 port 8080 (this will be a private

network

connection later) * 8001 forwards to Parsoid1 and Parsoid2 port 8000 (also will be private) On App1/Parsoid1 I can run `curl 192.168.56.63:8001` and get the appropriate response from Parsoid. I can run `curl 192.168.56.63:8081`

and

get the appropriate response from MediaWiki. The same is true for both on App2/Parsoid2. So the servers can get the info they need from the

services.

Currently I'm getting a the error "Error loading data from server: 500: docserver-http: HTTP 500. Would you like to retry?" when attempting to

use

Visual Editor. I've tried various different settings and have not always gotten that specific error, but am getting it with the settings I

currently

have in localsettings.js and LocalSettings.php (shown below in this

email).

Removing the proxy config lines from these settings gave slightly better results. I did not get the 500 error, but instead it sometimes after a

very

long time it would work. It also may have been throwing errors in the parsoid log (with debug on). I have those logs saved if they help. I'm hoping someone can just point out some misconfiguration, though. Here are snippets of my config files: On App1/Parsoid1, relevant localsettings.js: parsoidConfig.setMwApi({ uri: 'http://192.168.56.80:8081/demo/api.php', proxy: { uri: 'http://192.168.56.80:8081/' }, domain: 'demo', prefix: 'demo' } ); parsoidConfig.serverInterface = '192.168.56.80'; On App2/Parsoid2, relevant localsettings.js: parsoidConfig.setMwApi({ uri: 'http://192.168.56.80:8081/demo/api.php', proxy: { uri: 'http://192.168.56.80:8081/' }, domain: 'demo', prefix: 'demo' } ); parsoidConfig.serverInterface = '192.168.56.60'; On App1/Parsoid1, relevant LocalSettings.php: $wgVirtualRestConfig['modules']['parsoid'] = array( 'url' => '192.168.56.80:8001', 'HTTPProxy' => 'http://192.168.56.80:8001', 'domain' => $wikiId, 'prefix' => $wikiId ); On App2/Parsoid2, relevant LocalSettings.php: $wgVirtualRestConfig['modules']['parsoid'] = array( 'url' => '192.168.56.80:8001', 'HTTPProxy' => 'http://192.168.56.80:8001', 'domain' => $wikiId, 'prefix' => $wikiId ); Thanks! --James _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

_______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

James Montalvo

4:33 a.m.

I had a mistake in the config written in my original email, but it was correct on my servers. 192.168.56.80 LocalSettings.php snippet: ``` $wgVirtualRestConfig['modules']['parsoid'] = array( 'url' => 'http://192.168.56.63:8001', 'HTTPProxy' => 'http://192.168.56.63:8001', 'domain' => $wikiId, 'prefix' => $wikiId ); ``` 192.168.56.80 localsettings.js snippet: ``` parsoidConfig.setMwApi({ uri: 'http://192.168.56.63:8081/demo/api.php', proxy: { uri: 'http://192.168.56.63:8081/' }, domain: 'demo', prefix: 'demo' }); parsoidConfig.serverInterface = '192.168.56.80'; ``` 192.168.56.60 LocalSettings.php snippet: ``` $wgVirtualRestConfig['modules']['parsoid'] = array( 'url' => 'http://192.168.56.63:8001', 'HTTPProxy' => 'http://192.168.56.63:8001', 'domain' => $wikiId, 'prefix' => $wikiId ); ``` 192.168.56.60 localsettings.js snippet: ``` parsoidConfig.setMwApi({ uri: 'http://192.168.56.63:8081/demo/api.php', proxy: { uri: 'http://192.168.56.63:8081/' }, domain: 'demo', prefix: 'demo' }); parsoidConfig.serverInterface = '192.168.56.60'; ``` Sorry for the bad info, and thanks to Subramanya for pointing it out. --James On Wed, Jun 7, 2017 at 10:30 PM, James Montalvo <jamesmontalvo3(a)gmail.com> wrote:

...

I'm trying to setup two Parsoid servers to play nicely with two

MediaWiki

application servers and am having some issues. I have no problem getting things working with Parsoid on a single app server, or multiple Parsoid servers being used by a single app server, but ran into issues when I increased to multiple app servers. To try to get this working I starting making the app and Parsoid servers communicate through my load

balancer. So

the

network

connection later) * 8001 forwards to Parsoid1 and Parsoid2 port 8000 (also will be

private)

On App1/Parsoid1 I can run `curl 192.168.56.63:8001` and get the appropriate response from Parsoid. I can run `curl 192.168.56.63:8081`

and

get the appropriate response from MediaWiki. The same is true for both

App2/Parsoid2. So the servers can get the info they need from the

services.

Currently I'm getting a the error "Error loading data from server: 500: docserver-http: HTTP 500. Would you like to retry?" when attempting to

use

Visual Editor. I've tried various different settings and have not always gotten that specific error, but am getting it with the settings I

currently

have in localsettings.js and LocalSettings.php (shown below in this

email).

Removing the proxy config lines from these settings gave slightly better results. I did not get the 500 error, but instead it sometimes after a

very

_______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

James Montalvo

4:40 a.m.

So I'm getting the 500 error message from VE, and this is my parsoid log file on the parsoid server that attempted to handle the request. Not sure if this helps. [info][master][7904] initializing 4 workers [info][worker][7924] loading ... [info][worker][7912] loading ... [info][worker][7918] loading ... [info][worker][7930] loading ... [info][worker][7912] ready on 192.168.56.80:8000 [info][worker][7918] ready on 192.168.56.80:8000 [info][worker][7924] ready on 192.168.56.80:8000 [info][worker][7930] ready on 192.168.56.80:8000 { "0": "Starting HTTP request: ", "1": { "method": "GET", "followRedirect": true, "uri": "http://192.168.56.63:8081/demo/api.php", "qs": { "format": "json", "action": "query", "meta": "siteinfo", "siprop": "namespaces|namespacealiases|magicwords|functionhooks|extensiontags|general|interwikimap|languages|protocols|specialpagealiases", "rawcontinue": 1 }, "timeout": 40000, "agent": { "domain": null, "_events": {}, "_eventsCount": 1, "defaultPort": 80, "protocol": "http:", "options": { "maxSockets": 15, "connectTimeout": 5000, "path": null }, "requests": {}, "sockets": {}, "freeSockets": {}, "keepAliveMsecs": 1000, "keepAlive": false, "maxSockets": 15, "maxFreeSockets": 256 }, "headers": { "X-Request-ID": null, "User-Agent": "Parsoid/0.5.1+git", "Connection": "close" }, "strictSSL": true, "proxy": "http://192.168.56.63:8081/" } } (on a functioning installation normally there are several more entries after the JSON above) --James On Wed, Jun 7, 2017 at 11:33 PM, James Montalvo <jamesmontalvo3(a)gmail.com> wrote:

...

I'm trying to setup two Parsoid servers to play nicely with two

MediaWiki

application servers and am having some issues. I have no problem

getting

things working with Parsoid on a single app server, or multiple Parsoid servers being used by a single app server, but ran into issues when I increased to multiple app servers. To try to get this working I

starting

making the app and Parsoid servers communicate through my load

balancer. So

the

network

connection later) * 8001 forwards to Parsoid1 and Parsoid2 port 8000 (also will be

private)

On App1/Parsoid1 I can run `curl 192.168.56.63:8001` and get the appropriate response from Parsoid. I can run `curl 192.168.56.63:8081`

and

get the appropriate response from MediaWiki. The same is true for both

App2/Parsoid2. So the servers can get the info they need from the

services.

Currently I'm getting a the error "Error loading data from server: 500: docserver-http: HTTP 500. Would you like to retry?" when attempting to

use

Visual Editor. I've tried various different settings and have not

always

gotten that specific error, but am getting it with the settings I

currently

have in localsettings.js and LocalSettings.php (shown below in this

email).

Removing the proxy config lines from these settings gave slightly

better

results. I did not get the 500 error, but instead it sometimes after a

very

_______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

C. Scott Ananian

12:43 p.m.

RESTBase actually adds a lot of immediate performance, since it lets VE load the editable representation directly from cache, instead of requiring the editor to wait for Parsoid to parse the page before it can be edited. I documented the RESTBase install; it shouldn't actually be any more difficult than Parsoid. They both use the same service runner framework now. At any rate: in your configurations you have URL and HTTPProxy set to the exact same string. This is almost certainly not right. I believe if you just omit the proxy lines entirely from the configuration you'll find things work as you expect. --scott On Wed, Jun 7, 2017 at 11:30 PM, James Montalvo <jamesmontalvo3(a)gmail.com> wrote:

...

I think in general the first thing you should do for performance is set

restbase in front of parsoid? Caching the parsoid results will be faster than running multiple parsoids in parallel. That would also match the

wmf

configuration more closely, which would probably help us help you. I

wrote

up instructions for configuring restbase on the VE and Parsoid wiki

pages.

As it turns out I updated these today to use VRS configuration. Let me

know

if you run into trouble, perhaps some further minor updates are

necessary.

--scott On Jun 7, 2017 6:26 PM, "James Montalvo" <jamesmontalvo3(a)gmail.com>

wrote:

> I'm trying to setup two Parsoid servers to play nicely with two

MediaWiki

> application servers and am having some issues. I have no problem

getting

> things working with Parsoid on a single app server, or multiple Parsoid > servers being used by a single app server, but ran into issues when I > increased to multiple app servers. To try to get this working I

starting

> making the app and Parsoid servers communicate through my load

balancer.

the

network > connection later) > * 8001 forwards to Parsoid1 and Parsoid2 port 8000 (also will be

private)

On App1/Parsoid1 I can run `curl 192.168.56.63:8001` and get the appropriate response from Parsoid. I can run `curl 192.168.56.63:8081`

and > get the appropriate response from MediaWiki. The same is true for both

App2/Parsoid2. So the servers can get the info they need from the

services.

Currently I'm getting a the error "Error loading data from server: 500: docserver-http: HTTP 500. Would you like to retry?" when attempting to

use > Visual Editor. I've tried various different settings and have not

always

gotten that specific error, but am getting it with the settings I

currently

have in localsettings.js and LocalSettings.php (shown below in this

email). > Removing the proxy config lines from these settings gave slightly

better

results. I did not get the 500 error, but instead it sometimes after a

very

_______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

-- (http://cscott.net)

Alexandros Kosiaris

9 Jun 9 Jun

7:56 a.m.

Hi James, I don't know if you have noticed the following in C. Scott's response

...

At any rate: in your configurations you have URL and HTTPProxy set to the exact same string. This is almost certainly not right. I believe if you just omit the proxy lines entirely from the configuration you'll find things work as you expect. --scott

but I could not help but notice the error too. AFAIK setting these variables instruct both software to use http://192.168.56.63:8001/ as a forward proxy which is NOT what you have there. HAproxy is a reverse proxy software, not a forward proxy (although you can abuse it to achieve that functionality). In the setup you describe there is no need for forward proxies so neither parsoid nor mediawiki need a proxy configuration. I also don't think you need RESTBase as long as you are willing to wait for parsoid to finish parsing and returning the result. It should be fine for small articles, but as these grow larger, you will start having various performance related problems (for example you might have to adjust haproxy timeouts). But from what I gather, you are not there yet.

Gabriel Wicke

2:57 p.m.

On Fri, Jun 9, 2017 at 12:56 AM, Alexandros Kosiaris < akosiaris(a)wikimedia.org> wrote:

...

I also don't think you need RESTBase as long as you are willing to wait for parsoid to finish parsing and returning the result.

Apart from performance, there is also functionality that is missing without RESTBase: - Diffs are going to contain a lot of extra changes (commonly called "dirty diffs"), as no original HTML or data-parsoid is available to Parsoid's selective serialization algorithm. This might make it difficult to review changes. - Switching between wikitext and visual editing won't work. - Visual editing in general will very likely stop working once we reduce the size of HTML by separating out metadata (see https://phabricator.wikimedia.org/T78676). We keep pushing this back due to a lack of resources, but it is still planned, and might happen within the next six months. In short, using Parsoid directly for visual editing is an unsupported configuration, and is likely to stop working altogether in the foreseeable future. On Thu, Jun 8, 2017 at 7:10 AM, James Montalvo <jamesmontalvo3(a)gmail.com> wrote:

...

I've read through the documentation I think you're talking about. It's kind of hard to determine where to start since the docs are spread out between multiple VE, Parsoid and RESTBase pages. Installing RESTBase is, as you say, straightforward (git clone, npm install, basically). Configuring is not clear to me, and without clear docs it's the kind of thing that takes hours of trial and error.

The RESTBase install instructions <https://github.com/wikimedia/restbase#installation> point to a fairly well-commented example documentation file: https://github.com/wikimedia/restbase/blob/master/config.example.yaml For a basic install, all you should need is adjust the lines marked with XXX in there. The default backend will use SQLite. Cassandra offers better scalability and distribution for large scale, but this is not likely something you need. A single SQLite-backed RESTBase instance and a single Parsoid instance should be all you need. We are aware of the complexity of setting up a fully featured MediaWiki system, and are working on a Kubernetes-based solution right now (see https://github.com/wikimedia/mediawiki-containers/blob/k8s/README.k8s.md for current work in progress). The early prototype already sets up MediaWiki, VisualEditor, RESTBase, Parsoid, Math, as well as other services like EventBus. The current work is primarily aimed at development and testing, but we expect it to also offer a quick way to spin up a complete & fully-featured containerized MediaWiki system for small installs. Hope this helps, Gabriel

James HK

3:25 p.m.

Hi,

...

In short, using Parsoid directly for visual editing is an unsupported configuration, and is likely to stop working altogether in the foreseeable future.

...

for current work in progress). The early prototype already sets up MediaWiki, VisualEditor, RESTBase, Parsoid, Math, as well as other services like EventBus. The current work is primarily aimed at development and testing, but we expect it to also offer a quick way to spin up a complete & fully-featured containerized MediaWiki system for small installs.

This sounds like a lot of sublayers that can potentially disrupt a simple editing process and I wonder from the many non-WMF MediaWiki installations and administrators, who will be able and capable to debug those once an issue arise. MediaWiki-core itself isn't a walk in a park and now and then you find a bug that is addressed post six month, left alone for a year, or worse never see a response. Now, I can imagine that those added layers don't necessarily make this process easier and the current response times on phabricator make me skeptic about the outlined requirements and hereby the added complexity. Cheers On 6/9/17, Gabriel Wicke <gwicke(a)wikimedia.org> wrote: > On Fri, Jun 9, 2017 at 12:56 AM, Alexandros Kosiaris < > akosiaris(a)wikimedia.org> wrote: >> >> I also don't think you need RESTBase as long as you are willing to >> wait for parsoid to finish parsing and returning the result. > > > Apart from performance, there is also functionality that is missing without > RESTBase: > > - Diffs are going to contain a lot of extra changes (commonly called > "dirty diffs"), as no original HTML or data-parsoid is available to > Parsoid's selective serialization algorithm. This might make it difficult > to review changes. > - Switching between wikitext and visual editing won't work. > - Visual editing in general will very likely stop working once we reduce > the size of HTML by separating out metadata (see > https://phabricator.wikimedia.org/T78676). We keep pushing this back due > to a lack of resources, but it is still planned, and might happen within > the next six months. >

...

In short, using Parsoid directly for visual editing is an unsupported configuration, and is likely to stop working altogether in the foreseeable future.

> > > On Thu, Jun 8, 2017 at 7:10 AM, James Montalvo <jamesmontalvo3(a)gmail.com> > wrote: > >> I've read through the documentation I think you're talking about. It's >> kind >> of hard to determine where to start since the docs are spread out between >> multiple VE, Parsoid and RESTBase pages. Installing RESTBase is, as you >> say, straightforward (git clone, npm install, basically). Configuring is >> not clear to me, and without clear docs it's the kind of thing that takes >> hours of trial and error. > > > The RESTBase install instructions > <https://github.com/wikimedia/restbase#installation> point to a fairly > well-commented example documentation file: > https://github.com/wikimedia/restbase/blob/master/config.example.yaml > > For a basic install, all you should need is adjust the lines marked with > XXX in there. The default backend will use SQLite. Cassandra offers better > scalability and distribution for large scale, but this is not likely > something you need. A single SQLite-backed RESTBase instance and a single > Parsoid instance should be all you need. > > We are aware of the complexity of setting up a fully featured MediaWiki > system, and are working on a Kubernetes-based solution right now (see > https://github.com/wikimedia/mediawiki-containers/blob/k8s/README.k8s.md

...

> > Hope this helps, > > Gabriel > _______________________________________________ > Wikitech-l mailing list > Wikitech-l(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Gabriel Wicke

4:17 p.m.

On Fri, Jun 9, 2017 at 8:25 AM, James HK <jamesin.hongkong.1(a)gmail.com> wrote:

...

This is a familiar pattern in the history of computers. Early computers were programmed in assembly, until complexity was added with compilers. Early wikis were simple Perl CGI scripts backed by files, until Wikipedia's scale (traffic and organizational), security and feature requirements made it necessary to add caching layers, isolated services, and distributed storage systems. Each of these steps added layers of abstraction and complexity, and concerns about understanding all those layers was (rightfully) brought up at each step along the way. And yet the move towards higher levels of abstraction has been highly successful. Complex systems like web browsers or even entire distributed system clusters can now be deployed with a single click, on largely commoditized platforms. We are not yet at the point where we can offer you this degree of automation for MediaWiki, but we are working on it. -- Gabriel Wicke Principal Engineer, Wikimedia Foundation

Subramanya Sastry

5:57 p.m.

New subject: Setting up multiple Parsoid servers behind load balancer

On 06/09/2017 09:57 AM, Gabriel Wicke wrote:

...

On Fri, Jun 9, 2017 at 12:56 AM, Alexandros Kosiaris < akosiaris(a)wikimedia.org> wrote:

I also don't think you need RESTBase as long as you are willing to wait for parsoid to finish parsing and returning the result.

What Gabriel said there about dirty diffs. So, this depends on whether wikis are concerned about their wikitext getting normalized to "Parsoid-determined canonical" formats (wrt choice of whitespaces, quotes, for ex.). For example, this is a extremely important for wikimedia wikis, but may be less so for some smaller wikis, if they take a one-time normalization dirty diff and adopt identical norms in source editing.

...

- Switching between wikitext and visual editing won't work.

This is because of the dirty-diff requirement. As far as I understand, even if wikis are okay with dirty diffs, VE's source <-> html switching functionality requires restbase right now.

...

- Visual editing in general will very likely stop working once we reduce the size of HTML by separating out metadata (see https://phabricator.wikimedia.org/T78676). We keep pushing this back due to a lack of resources, but it is still planned, and might happen within the next six months.

...

In short, using Parsoid directly for visual editing is an unsupported configuration, and is likely to stop working altogether in the foreseeable future.

Just to be clear, we haven't yet made any formal decision to go down this route, but Gabriel articulates the reasons why it might make sense to do this. There are some aspects to consider here: (a) whether we want to support this combination behind a config flag at all given that some functionality may not be available (unless Parsoid clients figure out ways to support some functionality without RESTBase) (b) the complexity (maintenance, testing, documentation, support) of supporting multiple combinations. We don't have fully resolved answers to this yet. I don't know what VE's take on this is -- so there is also that to consider. But, when we have firm resolutions on all of this, we will make suitable announcements on lists, suggest upgrade options, and update wikis. But, also, what Gabriel said earlier about RESTBase. If you are already installing Parsoid, adding RESTBase (since it is also node.js) with the default sqlite backend might not be a whole lot more complexity. So, if VE-editing wikis that use Parsoid start adopting this, that would also inform our decisions above. Subbu.

James Montalvo

8:28 p.m.

Thanks to everyone for all the responses. I'm learning a lot. In the short term we need to figure out how to make this work without RESTBase, but I've been convinced by this email chain that in the long term we'll need to incorporate RESTBase into our setup. At this point I think I've determined that the problem we're having is not actually a Parsoid problem, but somehow related to MediaWiki Core (PHP) response times. Something about my multi-server setup is causing 25% of MW core response times to be 25x longer than normal. I didn't notice this in my dev setup, prior to testing Parsoid, probably because I just assumed my laptop was old and underpowered. In other words, normal page loads were slower but I just figured that having multiple VMs up on my laptop functioning as full app servers was the reason. Parsoid evidently has a default timeout short enough that when Parsoid makes MW core API requests I was getting failures, causing me to misinterpret it as a Parsoid issue. To ensure it was not my underpowered laptop I moved my testing to a machine with 12 CPUs and 64 GB RAM. Our configuration script that allows us to define our setup as follows: load balancers = list, of, IP, addresses, ... app servers = list, of, IP, addresses, ... memcached servers = list, of, IP, addresses, ... db master = a.single.ip.address db replicas = list, of, IP, addresses, ... parsoid servers = list, of, IP, addresses, ... elasticsearch servers = list, of, IP, addresses, ... I have not run it with that many servers yet, but it's theoretically possible. A single server does not need to fill a single role, so in testing thus far my configs look more like: load balancers = server.3.ip.addr app servers = server.1.ip.addr, server.2.ip.addr memcached servers = server.1.ip.addr, server.2.ip.addr db master = server.1.ip.addr db replicas = server.2.ip.addr parsoid servers = server.1.ip.addr, server.2.ip.addr elasticsearch servers = server.1.ip.addr, server.2.ip.addr In short: three servers, one exclusively a load balancer, two with everything installed albeit one acting as DB master and the other as DB replica. We're running this setup in production with all servers configured as "localhost", e.g. everything installed on one server. I'm pretty sure I've narrowed down the 25x-longer-response-times to being a multiple app-server problem because I can take the dev config above (server.1.ip.addr, server.2.ip.addr, server.3.ip.addr) and comment out various servers and re-run deploy. This allows me to quickly switch from a single app server to two, two DBs to one, etc. I see the issue with multiple app servers. I don't see it with a single app server, regardless of whether the other services have 1 or 2 servers. My LocalSettings.php files are are at [1] and [2] for dual app servers. These reference Extensions.php which _shouldn't_ have any impact but can be found at [3]. The files are written by Ansible and I'm kind of bad at getting the indenting correct...so, sorry about that if it looks funny. All of this is created by our project called meza [4]. We weren't really planning on announcing meza yet, but basically its purpose is to simplify MediaWiki install with all the bells and whistles for "enterprise" (whatever that means :) ) use cases. We've been running it on a single server for about a year, but need to migrate to a high availability setup to support 24/7 mission critical operations. Any ideas what may cause two load-balanced app servers to respond slowly 25% of the time? Thanks! --James [1] https://gist.github.com/jamesmontalvo3/5adf207623454c9eff98e93152b43108#fil… [2] https://gist.github.com/jamesmontalvo3/5adf207623454c9eff98e93152b43108#fil… [3] https://gist.github.com/jamesmontalvo3/5adf207623454c9eff98e93152b43108#fil… [4] https://github.com/enterprisemediawiki/meza On Fri, Jun 9, 2017 at 12:57 PM, Subramanya Sastry <ssastry(a)wikimedia.org> wrote:

...

On 06/09/2017 09:57 AM, Gabriel Wicke wrote: On Fri, Jun 9, 2017 at 12:56 AM, Alexandros Kosiaris <

akosiaris(a)wikimedia.org> wrote:

I also don't think you need RESTBase as long as you are willing to wait for parsoid to finish parsing and returning the result.

This is because of the dirty-diff requirement. As far as I understand, even if wikis are okay with dirty diffs, VE's source <-> html switching functionality requires restbase right now. - Visual editing in general will very likely stop working once we > reduce > the size of HTML by separating out metadata (see > https://phabricator.wikimedia.org/T78676). We keep pushing this back > due > to a lack of resources, but it is still planned, and might happen > within > the next six months.

There are some unresolved questions about how willing (Parsoid) clients are to work with this stripped-html format. That and the matter of us being resource-strapped means we keep kicking this down the road. But, when this happens, this will break VE-editing unless VE and Parsoid hide the data-mw stripping behind a config flag. In short, using Parsoid directly for visual editing is an unsupported > configuration, and is likely to stop working altogether in the foreseeable > future.

C. Scott Ananian

10:46 p.m.

It may be a lock issue. IIRC mediawiki can invoke parsoid which can then reinvoke the mediawiki api. I remember there being some corner case with locking which caused this recursive invocation to deadlock in some (not anything like production) situations. If you can get a trace from the "slow" mediawiki perhaps you will find it waiting for a lock to time out. --scott On Jun 9, 2017 4:29 PM, "James Montalvo" <jamesmontalvo3(a)gmail.com> wrote:

...

Thanks to everyone for all the responses. I'm learning a lot. In the short term we need to figure out how to make this work without RESTBase, but I've been convinced by this email chain that in the long term we'll need to incorporate RESTBase into our setup. At this point I think I've determined that the problem we're having is not actually a Parsoid problem, but somehow related to MediaWiki Core (PHP) response times. Something about my multi-server setup is causing 25% of MW core response times to be 25x longer than normal. I didn't notice this in my dev setup, prior to testing Parsoid, probably because I just assumed my laptop was old and underpowered. In other words, normal page loads were slower but I just figured that having multiple VMs up on my laptop functioning as full app servers was the reason. Parsoid evidently has a default timeout short enough that when Parsoid makes MW core API requests I was getting failures, causing me to misinterpret it as a Parsoid issue. To ensure it was not my underpowered laptop I moved my testing to a machine with 12 CPUs and 64 GB RAM. Our configuration script that allows us to define our setup as follows: load balancers = list, of, IP, addresses, ... app servers = list, of, IP, addresses, ... memcached servers = list, of, IP, addresses, ... db master = a.single.ip.address db replicas = list, of, IP, addresses, ... parsoid servers = list, of, IP, addresses, ... elasticsearch servers = list, of, IP, addresses, ... I have not run it with that many servers yet, but it's theoretically possible. A single server does not need to fill a single role, so in testing thus far my configs look more like: load balancers = server.3.ip.addr app servers = server.1.ip.addr, server.2.ip.addr memcached servers = server.1.ip.addr, server.2.ip.addr db master = server.1.ip.addr db replicas = server.2.ip.addr parsoid servers = server.1.ip.addr, server.2.ip.addr elasticsearch servers = server.1.ip.addr, server.2.ip.addr In short: three servers, one exclusively a load balancer, two with everything installed albeit one acting as DB master and the other as DB replica. We're running this setup in production with all servers configured as "localhost", e.g. everything installed on one server. I'm pretty sure I've narrowed down the 25x-longer-response-times to being a multiple app-server problem because I can take the dev config above (server.1.ip.addr, server.2.ip.addr, server.3.ip.addr) and comment out various servers and re-run deploy. This allows me to quickly switch from a single app server to two, two DBs to one, etc. I see the issue with multiple app servers. I don't see it with a single app server, regardless of whether the other services have 1 or 2 servers. My LocalSettings.php files are are at [1] and [2] for dual app servers. These reference Extensions.php which _shouldn't_ have any impact but can be found at [3]. The files are written by Ansible and I'm kind of bad at getting the indenting correct...so, sorry about that if it looks funny. All of this is created by our project called meza [4]. We weren't really planning on announcing meza yet, but basically its purpose is to simplify MediaWiki install with all the bells and whistles for "enterprise" (whatever that means :) ) use cases. We've been running it on a single server for about a year, but need to migrate to a high availability setup to support 24/7 mission critical operations. Any ideas what may cause two load-balanced app servers to respond slowly 25% of the time? Thanks! --James [1] https://gist.github.com/jamesmontalvo3/5adf207623454c9eff98e93152b431 08#file-localsettings-app1-php [2] https://gist.github.com/jamesmontalvo3/5adf207623454c9eff98e93152b431 08#file-localsettings-app2-php [3] https://gist.github.com/jamesmontalvo3/5adf207623454c9eff98e93152b431 08#file-extensions-php [4] https://github.com/enterprisemediawiki/meza On Fri, Jun 9, 2017 at 12:57 PM, Subramanya Sastry <ssastry(a)wikimedia.org> wrote:

On 06/09/2017 09:57 AM, Gabriel Wicke wrote: On Fri, Jun 9, 2017 at 12:56 AM, Alexandros Kosiaris <

akosiaris(a)wikimedia.org> wrote:

I also don't think you need RESTBase as long as you are willing to wait for parsoid to finish parsing and returning the result.

quotes,

for ex.). For example, this is a extremely important for wikimedia wikis, but may be less so for some smaller wikis, if they take a one-time normalization dirty diff and adopt identical norms in source editing. - Switching between wikitext and visual editing won't work.

back

due to a lack of resources, but it is still planned, and might happen within the next six months.

There are some unresolved questions about how willing (Parsoid) clients are to work with this stripped-html format. That and the matter of us

being

resource-strapped means we keep kicking this down the road. But, when

this

happens, this will break VE-editing unless VE and Parsoid hide the

data-mw

stripping behind a config flag. In short, using Parsoid directly for visual editing is an unsupported > configuration, and is likely to stop working altogether in the

foreseeable

future.

_______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

2537

days inactive

2539

days old

wikitech-l@lists.wikimedia.org

Manage subscription

12 comments

6 participants

tags (0)

participants (6)

Alexandros Kosiaris
C. Scott Ananian
Gabriel Wicke
James HK
James Montalvo
Subramanya Sastry