[Labs-l] RFC: Webtools setup

[Tim Landscheidt]

(anonymous) wrote:

> [...]

>>> Nova Resource:Webtools also proposes the public entry-point address to
>>> be http://web.wmflabs.org/TOOL. As I also said on the talk page, I would
>>> prefer tool(s).wmflabs.org. ;) Unfortunately, that went unnoticed. :(

>>> [...]

>> Sounds good to me, *but* we need to check if we can have a
>> catch-all DNS entry for *.wmflabs.org and still other ser-
>> vers in this domain.

> Wait-a-minute, you meant adding a dns entry *for each tool*??

No, I mean if we have a wildcard DNS entry for
*.wmflabs.org, we need to ensure that
non-webtools-project-1.wmflabs.org and
non-webtools-project-2.wmflabs.org are still reachable.

>>> I would prefer not to require an admin in the process of creating a
>>> tool. It should be possible to do through eg. a privileged helper.
>>> Although if it's a standarised change, maybe it could be merged
>>> automatically by jenkins.

>> We do require admin interaction and manual "review" to add
>> feeds to Planet Wikimedia, and still get turn-around times
>> in the range of minutes, so I don't think that we should
>> spend much thought in optimizing the creation times of new
>> tools :-).

> Well, perhaps if it was just a few minutes..., but  zero-time is still
> infinitely faster. You don't want to put off that developer that awoke
> at 5 am with a neat idea for a tool just because nobody was there to
> approve its creation. :)

You're still in Toolserver land where everything hinges on
DaB.'s sleep cycle (and health, and time, and mood,
and ... :-)).  At 5 AM anywhere on the world, there will be
probably an admin, but certainly an experienced user on IRC
who can either guide the developer how to submit the change-
set or do it for him, and a few minutes later it can be live
as WMF ops will probably be staffed 24/7.

IMVHO, though, if developers want to work in a collaborative
environment, they have to collaborate.  If the world must
stop just because they think so, how will they handle issues
where they definitely need to wait for other humans?

>>> Probably, most projects wouldn't need to be in puppet.

>> Yes, of course, but Webtools is certainly one of the others
>> as it will be used by dozens of developers and receive prob-
>> ably a million hits per day.  It should be as robust as pos-
>> sible from the get-go.  We should definitely learn from the
>> Toolserver (or the earlier WMF cluster) where ex post docu-
>> mentation of setup, including all hot fixes, to survive re-
>> boots and updates is painful and pushes the admin workload
>> *much* higher than necessary.

> I was refering to tools projects. Of course that the setup for creating
> a webtools apache or mysql should be puppetized. What I don't see the
> need is for having a puppet entry for an editcounter. It just needs to
> be in the proper folder of an instance for php tools (and remember that
> project storage is automatically shared).

> [...]

We discussed this in the past at
http://thread.gmane.org/gmane.org.wikimedia.toolserver.announce/485
(though to no avail then :-)): Servers are added and/or get
updated.  So you need to know the requirements of tools, and
if packages are no longer shipped in future versions of
Ubuntu, you need to find out which tool needed them so that
you can decide if the tool can be changed or you have to
package (and maintain) the software yourself.  If you have
already packaged software locally, on an update where you
can't rebuild the package at the first try, you want to see
which tool needs it, so you can (again) decide which path of
action is most appropriate.

What *I* don't understand is the resistance to having a Pup-
pet entry for an editcounter: It will do no harm, it isn't
rocket science, but it adds information and eases adminis-
tration.  When I shop at IKEA, I get asked for my postal
code.  When I visit the Miniatur Wunderland, they ask me for
my country or, speaking German, for my state.  They could
speed up the checkout process if they didn't, and probably
save thousands of euros on wages per year, but they prefer
to have some actual data to having to act on the hunch:
"Most of our customers are locals/Germans/Europeans/etc."
And especially PHP is one of those languages where you need
a lot of discipline in coding because there are practically
no safeguards against shooting yourself in the foot.  I
can't imagine a PHP developer who has written more than a
line of code, but fails to fill out a Puppet template for
PHP tools :-).

>>> I thought about setting up nis for that, but it would be better if those
>>> uids were also handled by ldap.

>> What's the advantage of NIS?

> It provides a “shared passwd”. See
> http://en.wikipedia.org/wiki/Network_Information_Service
> There's not a special advantage of using it instead of LDAP other than
> "it manages global accounts" and "It's not LDAP", you already have
> libraries for that and it'd be trivial to chain ldap and nis.

> [...]

No, I meant what is the advantage of setting up a NIS server
to "sharing passwd" using Puppet's users?

"Marc A. Pelletier" <marc at uberbox.org> wrote:

> [...]

> Automating is a fixed cost; manual processing is an
> unbounded cost against a finite resource.  :-)

That is (sort of) true, but if you focus on, say, a human's
lifetime, the fixed cost can be much higher.  IIRC Matthias
Wandel of woodgears.ca once commented on a suggestion for a
quick release for one of his jigs that if it took him
15 minutes to build on, he would have to use it hundreds of
time just to break even.

Also, "automating" can be very different from "automating".
Take Planet Wikimedia as a more non-solid example: Creating
a web form to add a URL to a planet is easy.  But if you re-
ally want to cut out the human part of the process, you have
test the URL that it is actually a link to an Atom/RSS feed.
And that it is neither a link to the planet's feed itself.
Also, you want to exclude spam sites.  So you add a review
queue.  Now you need permissions for that queue.  And you
need to protect this queue against DOS attacks.  Oh, and if
anything on the business end of the planet changes, you have
to adapt the ("fixed-cost") automation as well :-).

Tim