[Labs-admin] Replacing instance pages on Wikitech with a tool using the keystone/nova api

[Bryan Davis]

On Mon, Mar 27, 2017 at 8:18 AM, Andrew Bogott <abogott at wikimedia.org> wrote:
> On 3/25/17 8:12 PM, Bryan Davis wrote:
>>
>> I wanted to learn a bit about the keystone rest api, so I started
>> building a tool that would display things from it. One thing led to
>> another and I ended up building something that lists all projects in
>> our cluster and lets you see details on instances. Take a look at
>> <https://tools.wmflabs.org/keystone-browser/>.
>
> I think replacing the wikitech pages with this tool is totally fine, as long
> as
>
> * The new tool needs to cache like crazy, otherwise it's a ready-made
> keystone DOS.  (Maybe it does this already).

The tool currently doesn't cache at all, but has been very performant.
Adding caching via redis would be trivial if actually needed.

The big thing I did to help with performance was to get user details
from LDAP directly rather than from the keystone api. Keystone will
return uid data in bulk for each role in a project, but not detailed
user information. Detailed user information is only available via
per-user REST endpoint. LDAP however is happy to give me bulk results
for a query with a search filter like
'(|(uid=bd808)(uid=rush)(uid=...)...)' and the python ldap3 client
seems to properly handle paging over this result set to avoid the
per-request LDAP limit. Take a look at the bastion project to see this
in action (<https://tools.wmflabs.org/keystone-browser/project/bastion>).
Pulling that list of users up with the `openstack` cli client takes 5+
minutes vs about 7 seconds or so when getting the uid->cn mappings
from LDAP.

> * The tool is easy to discover (possibly via a sidebar link in wikitech, for
> the short term).

I was thinking that we would update [[Template:Nova Project]] to link
to the page in the tool for the given project.

> * We still have an established system for users to write documentation for
> their projects (maybe by preserving the existing namespace for that)
> ** The new tool should embed a link to ^

Yeah. There is value in giving each project a well known place to
write docs. I don't want to kill that at all.

> Thanks for making this!  The callback-based page updates are something I've
> spent a lot of time on and, hence, I have some attachment and reluctance to
> kill them -- but, honestly, the OpenStack upstream have been unknowingly
> hostile to that callback system and I have to rewrite things with basically
> every release, so it will save me a bunch of time to not maintain them.

The bits of the hooks system I've seen looked fragile, but honestly
the motivation is killing SMW usage more than anything else. I'd be
fine with changing the hooks to write static pages that are
transcluded into the project page too for that goal. I have seen a few
folks complain about the bot spam in wikitech's recent changes for the
instances that nodepool creates and destroys like crazy, but Chad made
that patch to ignore them that will roll out eventually.


Bryan
-- 
Bryan Davis              Wikimedia Foundation    <bd808 at wikimedia.org>
[[m:User:BDavis_(WMF)]]  Sr Software Engineer            Boise, ID USA
irc: bd808                                        v:415.839.6885 x6855