[Labs-l] RFC: Webtools setup

[Tim Landscheidt]

(anonymous) wrote:

> [...]

> Imho adding another layer of directory clutter (/web/) is a bad idea (I
> already wrote that on Nova Resource Talk:Webtools). As well as is
> /htdocs/cgi-bin/. It would be better to place all files into
> /data/project/TOOL (use subdirectories if you want to). That would also
> be the toolserver way (~USER/public_html respectively
> ~PROJECT/public_html), as a side note.

I don't have any opinion on /data/project/TOOL vs.
/data/project/web/TOOL :-).  But IIRC there are some differ-
ences between a "normal" "public_html" and an explicit
"cgi-bin", so a subdirectory "cgi-bin" is probably neces-
sary.

> Nova Resource:Webtools also proposes the public entry-point address to
> be http://web.wmflabs.org/TOOL. As I also said on the talk page, I would
> prefer tool(s).wmflabs.org. ;) Unfortunately, that went unnoticed. :(

> [...]

Sounds good to me, *but* we need to check if we can have a
catch-all DNS entry for *.wmflabs.org and still other ser-
vers in this domain.

(anonymous) wrote:

>>>> Dependencies of tools on other software are specified ex-
>>>> plicitely so that tools can be moved to other servers or
>>>> servers can be split by other software needed (i. e., a
>>>> server that only handles PHP, Ruby on Rails, etc.).  Depen-
>>>> dencies can be different for development (command line) and
>>>> deployment (web).

>>> This could be more complex to organise, since cli and web may still need
>>> to be grouped in the same tool.

>> Hmmm.  What do you mean?

> You talk about different prerequisites for parts of a tool, but both
> would belong to the same tool.

Yes, but specifying those differences is done by using two
Puppet classes instead of one, so the complexity isn't re-
ally there.

>>> This looks good, although I wonder if it's practical to ask users to add
>>> puppet configuration for each tool. Maybe it could be templated to the
>>> point where you only provide the tool name as a parameter to get all
>>> those points.

>> I had imagined a workflow where a user asks an admin: "I
>> want to create PHP tool TOOL!", and the admin then uses a
>> template (or maybe even freehands).  This has the added
>> bonus that the admin doesn't have to check whether the user-
>> supplied puppet module is correct, but can rely on the cor-
>> rectness of his own template.

>> It should be pretty easy to write a script that takes a tool
>> name and submits a change of the puppet configuration to
>> Gerrit, but I would wait with that until we're handling,
>> say, dozens of new tools per day :-).

> I would prefer not to require an admin in the process of creating a
> tool. It should be possible to do through eg. a privileged helper.
> Although if it's a standarised change, maybe it could be merged
> automatically by jenkins.

We do require admin interaction and manual "review" to add
feeds to Planet Wikimedia, and still get turn-around times
in the range of minutes, so I don't think that we should
spend much thought in optimizing the creation times of new
tools :-).

> Probably, most projects wouldn't need to be in puppet.

Yes, of course, but Webtools is certainly one of the others
as it will be used by dozens of developers and receive prob-
ably a million hits per day.  It should be as robust as pos-
sible from the get-go.  We should definitely learn from the
Toolserver (or the earlier WMF cluster) where ex post docu-
mentation of setup, including all hot fixes, to survive re-
boots and updates is painful and pushes the admin workload
*much* higher than necessary.

> [...]

>> I think LDAP accounts would open up lots of new security
>> questions.  The users only need to exist on the instances of
>> the Webtools project, and so local accounts should be
>> enough.

> Those are not local accounts, since they should be shared through the
> several instances of the webtools project.

I meant local as the opposite of LDAP.

> I thought about setting up nis for that, but it would be better if those
> uids were also handled by ldap.

What's the advantage of NIS?

> Not being on that project group, those 'users' couldn't join to other
> projects. Maybe a problem would appear if we created an project with the
> same name as used later by an ubuntu package. We could prefix with
> "tool-" the tool names to avoid that, although that'd be uglier IMHO.

Collisions between LDAP and local (or instance-shared :-))
users could be a problem, yes.  We need to find out if a lo-
cal user created by Puppet shadows an LDAP one of the same
name.

The idea of the prefix is neat and it may be easy to black-
list just all usernames with this prefix from LDAP creation
to avoid any collisions; I initially thought that we then
would have to name the directories accordingly as well, but
as we will probably heavily rewrite URLs in any case, this
shouldn't be a problem that is visible to the public.

One more thing that came to my mind is sudo.  We need to
allow developers to sudo as their tools.  "/etc/sudoers.d"
should be ideal for that.

Tim