Greetings, programs!
This week's update will be brief, if optimistic:
On the news front, there have been performance and reliability issues with Gluster than are being worked on by Ryan Lane. He is experimenting with an upcoming network driver and tweaking with automount timing to improve the situation. Along with the database replication, the shared storage is one of the key component of the infrastructure that are going to receive the most attention in the short term.
There is now a new project (named, predictably enough, "tools") that is the intended destination of the new architecture for the Tool Labs. We have a functionnal webservice environment, as well as a workable compute cluster to support work and long running processes. Already, a few brave souls have stepped forward to test that new environment; others are welcome to peek in or join the project with the usual "beta" caveats.
I'm not ready to open the gates entirely yet, since none of the management is currently automatized (that's a big part of my TODO for the week); but tools which have no dependencies on access to databases should already be able to be experimentally moved to the new project.
This week, I plan to concentrate on the first draft of an interface through which tool maintainers can manage their assets on the project, and do a first documentation pass regarding how to write and/or move a tool on the new project.
-- Marc
Marc A. Pelletier wrote:
There is now a new project (named, predictably enough, "tools") that is the intended destination of the new architecture for the Tool Labs. We have a functional webservice environment, as well as a workable compute cluster to support work and long running processes. Already, a few brave souls have stepped forward to test that new environment; others are welcome to peek in or join the project with the usual "beta" caveats.
I meant to ask this last week and it may be documented somewhere already (if so, just reply with a link), but I'm having a little trouble wrapping my head around Toolserver --> Tool Labs right now.
The Toolserver is a pool of shared resources, with a few roots, and a structure that's largely single user. That is, each Toolserver user gets his or her own directory inside /home and then works in that area.
Wikimedia Labs, as I understand it, is generally a pool of shared resources, but the idea is to create virtual machines that each individually have roots and only a few users per virtual machine (i.e., per project).
My confusion is how this "tools" project within Wikimedia Labs will be structured.
MZMcBride
On 04/03/13 19:30, MZMcBride wrote:
The Toolserver is a pool of shared resources, with a few roots, and a structure that's largely single user. That is, each Toolserver user gets his or her own directory inside /home and then works in that area.
Wikimedia Labs, as I understand it, is generally a pool of shared resources, but the idea is to create virtual machines that each individually have roots and only a few users per virtual machine (i.e., per project).
My confusion is how this "tools" project within Wikimedia Labs will be structured.
MZMcBride
A few machines where you install the different tools. So you wouldn't need to manage a complete VM, or install apache for your web app. You would place the tool in the appropiate dir and it _should_ just work. Quite similar to how toolserver worked, but more tool-centric (as if you created a MMP per tool, but easier).
Disclaimer: This my own conception, I may have understood everything wrong, etc.
On 03/04/2013 01:58 PM, Platonides wrote:
A few machines where you install the different tools. So you wouldn't need to manage a complete VM, or install apache for your web app. You would place the tool in the appropiate dir and it_should_ just work. Quite similar to how toolserver worked, but more tool-centric (as if you created a MMP per tool, but easier).
That's a very good nutshell.
The basic outline is thus: there is a compute grid and a webserver cloud (and, in the future, databases and other common services) that are managed at the project levels by the roots. Those are made available to all the tools.
Each "tool" is, essentially, storage, a webspace, a set of permissions, a user, and a group. Members of the group are the tool's maintainers and have complete control over it. They can queue jobs or continuous tasks, and put what they need in their webspace to function. They'll also have permission over future resources (like databases, etc).
-- Marc
toolserver-l@lists.wikimedia.org