Hey all,
I've been scheming for a while on how to reduce the number of calls up to the server for CentralNotice. At the same time I want to greatly reduce the number of objects I have in cache.
To do this I propose to change the architecture to having an intermediate proxy server with a static head JS section in mediawiki page head. The proxy would map down all the variables to only what is required at the time.
Right now I am imagining 4 servers, 2 in eqiad and 2 in ams, that would host a node.js proxy and/or possibly a local varnish instance.
The more detailed architecture is sketched out here: https://www.mediawiki.org/w/index.php?title=Extension:CentralNotice/Caching_...
I would appreciate any comments.
~Matt Walker Wikimedia Foundation Fundraising Technology Team
On 09/18/2013 06:06 PM, Matthew Walker wrote:
Hey all,
I've been scheming for a while on how to reduce the number of calls up to the server for CentralNotice. At the same time I want to greatly reduce the number of objects I have in cache.
To do this I propose to change the architecture to having an intermediate proxy server with a static head JS section in mediawiki page head. The proxy would map down all the variables to only what is required at the time.
+1 for limiting the application logic in regular text Varnishes, both from a performance and risk management perspective. Having your own banner proxies should make it easier to tweak its behavior to your needs without the risk of taking down the entire site.
Node is a good choice for this kind of task. If the total size of all unique banners is relatively small you might even be able to cache the banners in-memory instead of doing backend cache requests.
Gabriel
"Node is a good choice for this kind of task. If the total size of all unique banners is relatively small you might even be able to cache the banners in-memory instead of doing backend cache requests."
Though, not very explicitly proposed; but I was thinking that the best plan would be to have both node and varnish on the proxy box. I'd rather not write a caching layer in node when varnish does a fine job at it; but I also think it's somewhat silly to have symmetric traffic on the proxy when I can avoid it by having varnish on box. And the amount of data is small enough that we can easily fit it into < 16GB RAM. (Probably less than 8; but I don't know how it's all going to work itself out in production.)
~Matt Walker Wikimedia Foundation Fundraising Technology Team
On Thu, Sep 19, 2013 at 8:48 AM, Gabriel Wicke gwicke@wikimedia.org wrote:
On 09/18/2013 06:06 PM, Matthew Walker wrote:
Hey all,
I've been scheming for a while on how to reduce the number of calls up to the server for CentralNotice. At the same time I want to greatly reduce
the
number of objects I have in cache.
To do this I propose to change the architecture to having an intermediate proxy server with a static head JS section in mediawiki page head. The proxy would map down all the variables to only what is required at the
time.
+1 for limiting the application logic in regular text Varnishes, both from a performance and risk management perspective. Having your own banner proxies should make it easier to tweak its behavior to your needs without the risk of taking down the entire site.
Node is a good choice for this kind of task. If the total size of all unique banners is relatively small you might even be able to cache the banners in-memory instead of doing backend cache requests.
Gabriel
wikitech-l@lists.wikimedia.org