Hi!
As we have been working on gathering and defining user stories for the Toolforge Build Service and Toolforge itself, I have been thinking about the next steps for both of them, and surroundings and I wanted to share them and have some discussion to try to give a bit more direction to our work in those areas.
== Tl;Dr;
Let's think without constraints on what we want toolforge to become.
My opinion: * Move towards full Platform as a Service ** this means users only interface with our platform ** this might mean offering k8s as a service on top of CloudVPS if needed * Simple thin client * Simple thin UI (for people that can't/don't want to use the client) * API that supports both the above
== Long description
I think that this is somewhat a popular idea, but I would like to but I would like toolforge to be as easy to use as digitalocean and heroku, that is, a PaaS platform.
This means: * No need for ssh * Very simple cli (from the user's computer) * Simple web UI (same capabilities as the cli, for anyone that can't install the cli)
This also means: * No k8s as a service (discussed later) * Detaching the users from the underlying implementation
I know that this might require lots of changes, and those are not easy, but let's focus on the features we want, not the design underneath yet.
What I would like is to have some set of "components" that I can use and combine to create my tool:
Storage: * Store structured data somewhere (db) * Store unstructured data somewhere (storage/file-like?/s3?)
Compute: * Something that runs periodically (cron-like) * Something that runs once (one-off) * Something that runs continuously (daemon)
Network: * Create a public entry point for a web service * Connect between my components
So inspired by the digitalocean[1] and heroku[2] clis, the toolforge cli could just do:
* toolforge run * toolforge run-once * toolforge run-every * toolforge db * toolforge storage * toolforge expose-port (--public|--local)
Some side-commands could be: * toolforge tool -> to manage tools themselves, (create/add-maintainer/remove-maintainer/...) * toolforge get-all -> to list all my components * toolforge logs -> get the logs for a component * toolforge shell -> start a shell inside a component container (similar to heroku bash), for debugging * toolforge edit-config -> to allow to do the above trough some kind of structured spec
This is not an exhaustive list, but this should cover most of the usecases.
You might be asking now, what about people that needs some extra features from k8s?
For those, we can offer k8s as a service (through CloudVPS + terraform for example), so they have full control of their k8s instances.
Note that I have tried to refrain myself from adding any implementation details yet, as I think that we should do the exercise of thinking what we want without limiting ourselves on how we think it could be done. The limitations will come later :)
== Some random stats for current k8s toolforge usage
Total number of namespaces: 3163 Of which, namespaces that are empty: 1496 That means that only 1667 have something, for those, number of k8s webservices: 1276 Number of grid webservices: 307 Number of tools with cronjobs: 71 Number of tools with >1 cronjob 47 Number of tools with >10 cronjob 6 Number of tools with manually defined resources: 51 Of which I checked a few, and could be sorted out with "continuous jobs", as in daemons, though I have not reviewed all of them in detail.
[1] https://docs.digitalocean.com/reference/doctl/reference/apps [2] https://devcenter.heroku.com/categories/command-line
On Mon, Dec 12, 2022 at 8:18 AM David Caro dcaro@wikimedia.org wrote:
As we have been working on gathering and defining user stories for the Toolforge Build Service and Toolforge itself, I have been thinking about the next steps for both of them, and surroundings and I wanted to share them and have some discussion to try to give a bit more direction to our work in those areas.
== Tl;Dr;
Let's think without constraints on what we want toolforge to become.
My opinion:
- Move towards full Platform as a Service
** this means users only interface with our platform ** this might mean offering k8s as a service on top of CloudVPS if needed
- Simple thin client
- Simple thin UI (for people that can't/don't want to use the client)
- API that supports both the above
== Long description
I think that this is somewhat a popular idea, but I would like to but I would like toolforge to be as easy to use as digitalocean and heroku, that is, a PaaS platform.
This means:
- No need for ssh
- Very simple cli (from the user's computer)
- Simple web UI (same capabilities as the cli, for anyone that can't install the cli)
This also means:
- No k8s as a service (discussed later)
- Detaching the users from the underlying implementation
I know that this might require lots of changes, and those are not easy, but let's focus on the features we want, not the design underneath yet.
What I would like is to have some set of "components" that I can use and combine to create my tool:
Storage:
- Store structured data somewhere (db)
- Store unstructured data somewhere (storage/file-like?/s3?)
Compute:
- Something that runs periodically (cron-like)
- Something that runs once (one-off)
- Something that runs continuously (daemon)
Network:
- Create a public entry point for a web service
- Connect between my components
So inspired by the digitalocean[1] and heroku[2] clis, the toolforge cli could just do:
- toolforge run
- toolforge run-once
- toolforge run-every
- toolforge db
- toolforge storage
- toolforge expose-port (--public|--local)
Some side-commands could be:
- toolforge tool -> to manage tools themselves, (create/add-maintainer/remove-maintainer/...)
- toolforge get-all -> to list all my components
- toolforge logs -> get the logs for a component
- toolforge shell -> start a shell inside a component container (similar to heroku bash), for debugging
- toolforge edit-config -> to allow to do the above trough some kind of structured spec
This is not an exhaustive list, but this should cover most of the usecases.
You might be asking now, what about people that needs some extra features from k8s?
For those, we can offer k8s as a service (through CloudVPS + terraform for example), so they have full control of their k8s instances.
Note that I have tried to refrain myself from adding any implementation details yet, as I think that we should do the exercise of thinking what we want without limiting ourselves on how we think it could be done. The limitations will come later :)
== Some random stats for current k8s toolforge usage
Total number of namespaces: 3163 Of which, namespaces that are empty: 1496 That means that only 1667 have something, for those, number of k8s webservices: 1276 Number of grid webservices: 307 Number of tools with cronjobs: 71 Number of tools with >1 cronjob 47 Number of tools with >10 cronjob 6 Number of tools with manually defined resources: 51 Of which I checked a few, and could be sorted out with "continuous jobs", as in daemons, though I have not reviewed all of them in detail.
[1] https://docs.digitalocean.com/reference/doctl/reference/apps [2] https://devcenter.heroku.com/categories/command-line
This high level idea sounds good to me except for the part about attempting to hide the Kubernetes runtime from the users. I would like to hear more about the complexities David believes this would insulate either the WMCS team or the Toolforge users from.
It seems unlikely to me that WMCS would be able to produce abstractions robust enough that they would not leak Kubernetes implementation details to the tenants. In my experience, once users can see that there is a more powerful system just out of reach they will start figuring out how to tunnel down and reach it. Making direct use of Kubernetes possible, but generally unnecessary for the majority of users seems like a better goal than attempting to hide it entirely.
Bryan
On 12/12 16:39, Bryan Davis wrote:
On Mon, Dec 12, 2022 at 8:18 AM David Caro dcaro@wikimedia.org wrote:
As we have been working on gathering and defining user stories for the Toolforge Build Service and Toolforge itself, I have been thinking about the next steps for both of them, and surroundings and I wanted to share them and have some discussion to try to give a bit more direction to our work in those areas.
== Tl;Dr;
Let's think without constraints on what we want toolforge to become.
My opinion:
- Move towards full Platform as a Service
** this means users only interface with our platform ** this might mean offering k8s as a service on top of CloudVPS if needed
- Simple thin client
- Simple thin UI (for people that can't/don't want to use the client)
- API that supports both the above
== Long description
I think that this is somewhat a popular idea, but I would like to but I would like toolforge to be as easy to use as digitalocean and heroku, that is, a PaaS platform.
This means:
- No need for ssh
- Very simple cli (from the user's computer)
- Simple web UI (same capabilities as the cli, for anyone that can't install the cli)
This also means:
- No k8s as a service (discussed later)
- Detaching the users from the underlying implementation
I know that this might require lots of changes, and those are not easy, but let's focus on the features we want, not the design underneath yet.
What I would like is to have some set of "components" that I can use and combine to create my tool:
Storage:
- Store structured data somewhere (db)
- Store unstructured data somewhere (storage/file-like?/s3?)
Compute:
- Something that runs periodically (cron-like)
- Something that runs once (one-off)
- Something that runs continuously (daemon)
Network:
- Create a public entry point for a web service
- Connect between my components
So inspired by the digitalocean[1] and heroku[2] clis, the toolforge cli could just do:
- toolforge run
- toolforge run-once
- toolforge run-every
- toolforge db
- toolforge storage
- toolforge expose-port (--public|--local)
Some side-commands could be:
- toolforge tool -> to manage tools themselves, (create/add-maintainer/remove-maintainer/...)
- toolforge get-all -> to list all my components
- toolforge logs -> get the logs for a component
- toolforge shell -> start a shell inside a component container (similar to heroku bash), for debugging
- toolforge edit-config -> to allow to do the above trough some kind of structured spec
This is not an exhaustive list, but this should cover most of the usecases.
You might be asking now, what about people that needs some extra features from k8s?
For those, we can offer k8s as a service (through CloudVPS + terraform for example), so they have full control of their k8s instances.
Note that I have tried to refrain myself from adding any implementation details yet, as I think that we should do the exercise of thinking what we want without limiting ourselves on how we think it could be done. The limitations will come later :)
== Some random stats for current k8s toolforge usage
Total number of namespaces: 3163 Of which, namespaces that are empty: 1496 That means that only 1667 have something, for those, number of k8s webservices: 1276 Number of grid webservices: 307 Number of tools with cronjobs: 71 Number of tools with >1 cronjob 47 Number of tools with >10 cronjob 6 Number of tools with manually defined resources: 51 Of which I checked a few, and could be sorted out with "continuous jobs", as in daemons, though I have not reviewed all of them in detail.
[1] https://docs.digitalocean.com/reference/doctl/reference/apps [2] https://devcenter.heroku.com/categories/command-line
This high level idea sounds good to me except for the part about attempting to hide the Kubernetes runtime from the users. I would like to hear more about the complexities David believes this would insulate either the WMCS team or the Toolforge users from.
This goes both ways, for the users: * They become dependent only on the abstraction we provide on top of whichever infrastructure, not caring about where it is running, or how it is running * This means that any upgrade, redesign, transformation or otherwise of the underlying implementation has more ways of being transparent for them, and thus, uneventful and not requiring any action on their part.
For us (maintainers, toolforge roots): * We can make implementation changes without changing the abstraction itself, this is a huge benefit, that allows: ** avoiding a Grid 2.0, users are not bound to implementations anymore ** being able to swap the way we do storage (NFS) without user's caring about it ** being able to upgrade the cluster without breaking any user's setup ** split the k8s cluster in two transparently for users (opening the door to blue-green deployments, etc.) ** not needing bastions as development environments ** not needing unix users for security eventually (if we don't want to)
It seems unlikely to me that WMCS would be able to produce abstractions robust enough that they would not leak Kubernetes implementation details to the tenants.
Nobody is perfect, so we might leak some details, but as long as we have the abstraction layer in between, we can handle/deprecate/transform those without getting coupled with the implementation itself.
In my experience, once users can see that there is a more powerful system just out of reach they will start figuring out how to tunnel down and reach it. Making direct use of Kubernetes possible, but generally unnecessary for the majority of users seems like a better goal than attempting to hide it entirely.
That is why, for those users, we will have to provide an alternative to use that more powerful system that is easier to use than hacking around to get to the toolforge underlying bits. That is the k8s as a service through cloudvps+terraform (or whatever).
But even before that, if a user has to hack around to get something done, it's pretty clear that that is not supported by the system and will break without the need of a notice and will not block any operations on the infra.
The goal being separating the offerings of a platform as a service, and kubernetes runtime.
Bryan
Bryan Davis Technical Engagement Wikimedia Foundation Principal Software Engineer Boise, ID USA [[m:User:BDavis_(WMF)]] irc: bd808 _______________________________________________ Cloud-admin mailing list -- cloud-admin@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-admin.lists.wikimedia.org/
cloud-admin@lists.wikimedia.org