Hi!
In another thread Oliver asked about the progress of One Machine To Rule Them All :-)
In fact it looks like it will now be two machines to rule them all, or rather, two machines to cooperatively rule them all in roughly equal capacity. I know, it doesn't have the same ring to it...
I posted the following to RT 6383, but who knows who reads RT, so here it is again:
-- quote --
An update on this. Some Analytics folk have probably already heard bits
and pieces via mailing lists, but my fellow Opsen on RT duty rightly
begin to wonder about this ticket.
We have procured dbstore100[12], both with which will be replicating
shards S[1-7] into a single MariaDB 10 instance each, using the new
multi-source replication. The boxes are still being setup (because
recombining the shards requires full dump/reload, plus getting all seven
in sync, plus compressing tables -- slow going). The x1 shard and event
logging will replicate to dbstore too, but that's pending RT 7081.
Analytics will have direct, but read-only, access to dbstore1002.
db1047, the current s1-analytics-slave, has the required disk space so
it will likely become a slave to dbstore1002, or else make use of the
MariaDB 10 CONNECT engine to access the data (like federation, but
better than FEDERATED engine was, thanks to ECP:
engine-condition-pushdown). As ever, Analytics will have read/write
access to db1047 with scratch space.
The situation will result in:
- cross joins/unions on any wiki on either db1047 or dbstore1002
- ability to spread load across both boxes with a single SQL query
- less likely to block others due to locking
- less likely to cause replag
I'm happy to go into more technical detail if anyone is interested.
When will it be ready, you ask? :-) Not until after the Ops meet in Athens, which realistically means: in May.
-- endquote --
The bit about spreading load across two machines with one query will require people to be a bit careful in designing the SQL. Alternatively you guys might simply choose to dictate which box should run expensive queries, to avoid tripping each other up.
Incidentally, MariaDB 10 has the Cassandra storage engine which might be of some use to you guys in time. But so far I've only been trialing CONNECT+ECP.
BR
Sean