What's the best way to improve performance, with regard to edit rate

List overview All Threads
Download

newer

older

Re: [MediaWiki-l] image rebuilding...

help importing image into a new...

Star Struck

28 Nov 2018 28 Nov '18

7:40 a.m.

I have a wiki with a high volume of edits that need to be made (via API), about 2-3 per second. The wiki can't keep up with them; there ends up being a backlog of edits that need to be made as, e.g., attempts are being made by 10 different bots to make 10 edits simultaneously (all to different pages) and it takes a long time for them to save so that the bots can move on to their next edits. What's the best way to boost performance in the ways needed to reduce the time needed to save edits, or to be able to save more edits per second by concurrently running bots? Thanks, -- Starstruck

Show replies by date

Hershel Robinson

28 Nov 28 Nov

8:18 a.m.

New subject: What's the best way to improve performance, with regard to edit rate

...

What's the best way to boost performance ...

Depends on a myriad of factors, such as the OS, web server and database and hardware etc. The simplest answer is to increase your hardware resources, meaning if the site has one CPU, give it two. For anything more specific, we would need more details about the server software and hardware. Hershel On Wed, Nov 28, 2018 at 2:41 PM Star Struck <starstruck7200(a)gmail.com> wrote:

...

-- http://civihosting.com/ Simply the best in shared hosting

Star Struck

9:17 a.m.

New subject: What's the best way to improve performance, with regard to edit rate

The server is just the localhost that I use for testing; it's Apache and MySQL running on Ubuntu 18.04 on an HP Elite 3.0ghz with 4GB of RAM <https://www.amazon.com/HP-Elite-Professional-Certified-Refurbished/dp/B0094JF1HA>. I haven't really figured out what kind of hardware or software I want to use for production, because I haven't done a lot of server administration (I've typically just used a VPS when I needed webhosting, but perhaps my needs now have expanded beyond that, because this is going to be a huge wiki, the same scale as Wikipedia; although my main concern at the moment is with making page saves, rather than page loads, more efficient, since I don't necessarily anticipate having a lot of visitors from the Internet reading the wiki, or else I'd be focusing more on stuff like caching; I mostly just want to set up a workable proof-of-concept for the moment.) I've heard that Wikimedia splits their enwiki database up among more than one server; is that how they're able to handle several page saves per second on the master? On Wed, Nov 28, 2018 at 8:19 AM Hershel Robinson <hershelsr(a)gmail.com> wrote:

...

What's the best way to boost performance ...

Hershel Robinson

9:33 a.m.

New subject: What's the best way to improve performance, with regard to edit rate

...

I've heard that Wikimedia splits their enwiki database up among more than one server; is that how they're able to handle several page saves per second on the master?

That is correct. See here https://meta.wikimedia.org/wiki/Wikimedia_servers for more details. -- http://civihosting.com/ Simply the best in shared hosting

Ariel Glenn WMF

6 Feb 6 Feb

3:54 a.m.

New subject: What's the best way to improve performance, with regard to edit rate

It's not split up (sharded) across servers, a least as far as page and revision tables go. There is one active master at any given time hat handles all writes; the current host has 160GB of memory and 10 physical cores (20 with hyperthreading). The actual revision *content* for all projects is indeed split up across several servers, in an 'external storage' cluster. The current server configuration is available at https://noc.wikimedia.org/conf/highlight.php?file=db-eqiad.php You can get basic specs as well as load information on these servers by looking up each one in grafana; here's db1067 (the current enwiki master) as an example: https://grafana.wikimedia.org/d/000000607/cluster-overview?orgId=1&var-… Ariel On Wed, Nov 28, 2018 at 4:34 PM Hershel Robinson <hershelsr(a)gmail.com> wrote:

...

I've heard that Wikimedia splits their enwiki database up among more than one server; is that how they're able to handle several page saves per second on the master?

That is correct. See here https://meta.wikimedia.org/wiki/Wikimedia_servers for more details. -- http://civihosting.com/ Simply the best in shared hosting _______________________________________________ MediaWiki-l mailing list To unsubscribe, go to: https://lists.wikimedia.org/mailman/listinfo/mediawiki-l

Brian Wolff

5:46 a.m.

New subject: What's the best way to improve performance, with regard to edit rate

What is your caching setup (e.g. $wgMainCacheType and friends)? Caching probably has more of an effect on read time than save time, but it will also have an effect on save time, probably a significant one. If its just one server, apcu (i.e. CACHE_ACCEL) is probably the easiest to setup. There's a number of factors that can effect page save time. In many cases it can depend on what the content of your page edits (e.g. If your edits have lots of embedded images, 404 handling can result in significant improvements). The first step I would suggest would be to do profiling - https://www.mediawiki.org/wiki/Manual:Profiling This will tell you what part is being slow, and we can give more specific advice based on that -- Brian On Wed, Nov 28, 2018 at 2:17 PM Star Struck <starstruck7200(a)gmail.com> wrote:

...

I haven't really figured out what kind of hardware or software I want to use for production, because I haven't done a lot of server administration (I've typically just used a VPS when I needed webhosting, but perhaps my needs now have expanded beyond that, because this is going to be a huge wiki, the same scale as Wikipedia; although my main concern at the moment is with making page saves, rather than page loads, more efficient, since I don't necessarily anticipate having a lot of visitors from the Internet reading the wiki, or else I'd be focusing more on stuff like caching; I mostly just want to set up a workable proof-of-concept for the moment.) I've heard that Wikimedia splits their enwiki database up among more than one server; is that how they're able to handle several page saves per second on the master? On Wed, Nov 28, 2018 at 8:19 AM Hershel Robinson <hershelsr(a)gmail.com> wrote:

What's the best way to boost performance ...

_______________________________________________ MediaWiki-l mailing list To unsubscribe, go to: https://lists.wikimedia.org/mailman/listinfo/mediawiki-l

1903

days inactive

1973

days old

mediawiki-l@lists.wikimedia.org

Manage subscription

5 comments

4 participants

tags (0)

participants (4)

Ariel Glenn WMF
Brian Wolff
Hershel Robinson
Star Struck