Hi,
at the office hour yesterday
(cf. http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-office/20130430.txt):
| [...]
| <Coren> multichill: The long story short; replicating
| databases is happening soon (Within the month)
| Replicating multiple copies of commons and wikidata
| isn't going to happen that way; it needs to be built
| into application logic or using federated tables.
| Almost /all/ of the problems the TS have had with
| replication were caused by that redundancy and
| trying to keep it synced.
| <multichill> Coren: So you're basically saying Toollabs is
| useless for me
| <Platonides> {{ref needed}}
| <Coren> We're all more than happy to help you (and any other
| maintainer) with adapting your tools to work in that
| setup.
| <scfc_de> Coren: In that case, Tools will not be able to
| replace Toolserver.
| <giftpfla1ze> what are federated tables btw?
| <Ryan_Lane> AFAIK toolserver will also have this limitation
| at some point
| <scfc_de> Ryan_Lane: What do you mean?
| [...]
There were never answers to this, so I bring it up here
again:
1. How were "almost /all/ of the problems the TS have had
with replication" "caused by that redundancy and trying
to keep it synced"?
2. What limitation will the Toolserver have at some point?
Tim
Hi.
I've been a big proponent of the idea that we should work toward the
creation of multiple Toolservers. Redundancy in this area would be a good
thing. Setting up infrastructure that allows any person with an AWS
account or extra server capacity to set up and run their own full
Toolserver (that is, set up real-time database replication + shared
hosting) would be great.
That said, I think it would be foolish to ignore the fact that the German
Toolserver is slowly dying and that Wikimedia Labs is its natural
successor. There was an IRC office hours earlier today and my takeaway
from it was that Labs is on the right path, but that it isn't marketing
itself well.
Lately, the Toolserver has been horribly unstable. Web tools stop working,
queries are killed, databases are corrupt, replication lag hurts, and
everything feels overloaded and tired. Labs should better advertise itself
as a robust, stable hosting platform. Stability is important to developers
with limited time, who don’t want to spend their time debugging hosting
issues. The Toolserver had a tendency to make breaking changes (changing
the operating system, changing the type of Web server, killing and then
resuscitating support for Vixie cron, requiring the use of SGE, etc.).
Labs should implement and emphasize a better approach to these hosting
issues, to garner adopters and supporters.
Database replication came up yet again in the office hours. Many
developers (myself included) seem to be holding off on Labs until database
replication is up and running. The sooner this can happen, the better. But
the remaining sticking point seems to be cross-database joins, which
people in the office hours suggested using federated tables or application
logic to replace. It would help if the Labs folks could better explain
_why_ cross-database joins won't be supported (I think most developers
would agree with the reasoning) and offer better guidance and
documentation for how to work around this hurdle. (For example, what is a
federated table?)
Yes, there will always be some contingent that's upset at the seemingly
underhanded way in which Labs was brought into the world (simply announced
one day, with the implicit acceptance that the German Toolserver would be
put down in time or left to rot), but I think there are far more people
who don't support or are ambivalent toward Labs when these people could
and should be proponents for it. Labs just needs a bit better public
relations, in my opinion.
MZMcBride