We currently have no plans for having the user databases on the same servers as the replicated databases. Direct joins will not be possible, so tools will need to be modified.
-50
It's such a useful feature, that it would be worth making a local mysql slaves for having them. I know, the all-powerful labs environment is unable to run a mysql instance, but we could use MySQL cluster, trading memory (available) to get joins (denied).
I'm not the one setting up the databases. If you want information about why this won't be available, talk to Asher (binasher in #wikimedia-operations on Freenode). Maybe he can be convinced otherwise.
Of course, in the production cluster we don't do joins this way. We handle the joins in the app logic, which is a more appropriate way of doing this.
SGE is a strong queue system. We have people and tools already trained to use it. It would be my first option. That said, if the presented alternative has the same user interface, it shouldn't be a problem. For instance, I don't have an opinion about which of the SGE forks would be preferable.
In general in Labs we don't have a large need for a queuing system right now. If Toolserver folks need it very badly, it's possible to add, someone just needs to put the effort into it. It likely wouldn't be amazingly hard to puppetize this to run in a single project. Making things multi-project is difficult and takes effort. Anyone can do the single-project version in a project, multi-project will likely take engineering effort.
- no support for servlets
I'm not sure what you mean by servlet?
J2EE, I guess.
Well, if it's available in the ubuntu repos, or if it's open source, then it's available in Labs.
I'd love DaB to help us improve Labs.
Everything about Labs is fully open. Anyone can help build it, even the production portions.
Would it be worth our efforts? I sometimes wonder why we should work on that (yes, I'm pessimistic right now). For instance the squid in front of *.beta.wmflabs.org. It was configured by Petan and me. We had absolutely no support from the WMF. The squid wasn't purging correctly. It worked on production, so there was a config error somewhere. We begged to see the squid config for months. But as it was in the private repository, no, it can't be shown, just in case it has something secret (very unlikely for squid config). Yes, we will clean them up and publish, eventually. Months passed (not to mention how publishing the config had been requested years ago). It could have been quickly reviewed before handing out, and we weren't going to abuse it if there really something weird was there. Replicating the WMF setup was done without viewing that same setup. I finally fixed it. I was quite proud of having solved it.
And you should be. Your changes kept that project moving along for months until I broke it.
Where is that file right now? It vanished. The file was lost in one of the multiple corruptions of labs instances. It was replaced with a copy of the cluster config (which was finally published in the meantime). So it feels like wasted effort now. I'd have liked to save a local copy at least.
To be fair, there's only been a single occurrence of instance corruption, which was due to a bug in KVM.
Also, yes, the squid configuration was finally published because ones of the devs spent the time to do so. I was working on stabilizing things most of that time.
Does this mean your efforts were wasted? Of course not. Your efforts helped keep the project running, which is important. Just because your file was replaced with the production copy doesn't mean the work put into it was for nothing.
It's not enough to leave tools there and say "It is fully open. Anyone can help build it"
We're also putting effort into making the migration happen, but we're focusing our efforts in different places. We can't do everything, which is why I'm trying to encourage others to help out. If we work on separate pieces of the work it'll go much quicker.
- Ryan