Hi all,
Matt & I have published our draft goals for the nascent Services team [1] at
https://www.mediawiki.org/wiki/Services/Roadmap
Your feedback would be much appreciated.
Thanks,
Gabriel
On Thu, Jun 05, 2014 at 10:21:29AM -0700, Gabriel Wicke wrote:
Matt & I have published our draft goals for the nascent Services team [1] at
https://www.mediawiki.org/wiki/Services/Roadmap
Your feedback would be much appreciated.
- These goals are vast and in very exciting in general. Clearly the team has a vision and a lot of thought has been put into compiling this big list of tasks. Kudos to both of you for writing this up and sharing it with us, this is incredibly useful!
- At the same time, some of these goals are /too/ specific and do not leave much up to interpretation and deliberation.
For some of these, we haven't agreed if they're towards the direction that we're going to take (minor example: "leverage packages as much as possible for deployment, DRY") and the goals process is not really the proper forum to make those decisions.
Making them a bit more abstract ("work on improving our production deployment strategy") would help keep our options open and avoid misunderstandings when the time comes.
- On a more specific note, I'm still personally in doubt on whether the MWCore team has signed off on a) the storage service plans b) static Parsoid HTML5 for all page views plan, especially in the detail articulated in your goals. As this is a big part of your plan for next year, I think it's worth clarifying.
- This is *a lot* of work. Are you sure you can pull this off?
As more of these core services get deployed, you're going to incur some of the costs supporting those deployments. This is essentially the "ops problem" of scheduling work; both us and MWCore know well how this can throw off our plans and schedules and we tend to account for it. Have you considered this?
Moreover, my impression of the team's charter is that it will also have a supportive role to other teams that want to write or use services. I see you've accounted for some of that (e.g. mobile) but I think you should probably expect more of it as SOA catches up within the foundation.
Finally, I think you should expect some (healthy, I hope!) amount of debate and consideration for certain things you intend to do, something that can also be quite time-consuming, obviously :)
(what are the week numbers in parenthesis supposed to be, btw? Is this FTE? If so, e.g. Q2 accounts for 18 weeks for Gabriel, probably a bit too much for a single quarter ;))
- Several of the things listed there will require hardware, possibly a non-trivial amount of it. Could you make a very rough list of your expectations so that we can plan for it, or is it too early to tell?
Regards, Faidon
Hi Faidon,
thank you for your input!
On 06/07/2014 08:30 PM, Faidon Liambotis wrote:
On Thu, Jun 05, 2014 at 10:21:29AM -0700, Gabriel Wicke wrote:
Matt & I have published our draft goals for the nascent Services team [1] at
- At the same time, some of these goals are /too/ specific and do not leave much up to interpretation and deliberation.
I hope that they are specific enough to invite critical input and identify points of contention. These plans are also not immutable, and I'm happy to make them less specific in areas where there's more need for discussion.
- This is *a lot* of work. Are you sure you can pull this off?
I'm cautiously optimistic. Q1 and to some degree Q2 will be tight, but it's important for other teams to get the basic functionality done early. We have prototypes for parts of this, so have an idea of what we are getting ourselves into. There are also some optional items in Q3 / Q4 that could be canceled or deferred if earlier tasks take longer than expected.
Moreover, my impression of the team's charter is that it will also have a supportive role to other teams that want to write or use services. I see you've accounted for some of that (e.g. mobile) but I think you should probably expect more of it as SOA catches up within the foundation.
The supportive role is indeed intended. Initially much of this support is going to be in the form of building general infrastructure (storage service, REST API, caching), but over time we'll spend more time on helping other teams. Some of this will be supporting specific use cases, but we'll also help other teams develop their own specialized services and API end points. This experience will help us refine the general infrastructure, and might also surface repeating problems that could be solved generically. Overall I think that we should have enough capacity in Q3 & Q4 to work on this.
Finally, I think you should expect some (healthy, I hope!) amount of debate and consideration for certain things you intend to do, something that can also be quite time-consuming, obviously :)
I'd be very disappointed if this was the end of all technical debate. ;)
But more seriously, I think a good amount of time is going to be needed for working with the community. This is the reason why we didn't set a deadline for using Parsoid HTML5 for page views. Our goal is to get the technical preliminaries in place, and then test, refine and discuss things based on the real thing. Much of this is probably going to be handled by Parsoid and the Community Engagement team, but we'll still need to be prepared to make technical adjustments as needed.
(what are the week numbers in parenthesis supposed to be, btw? Is this FTE? If so, e.g. Q2 accounts for 18 weeks for Gabriel, probably a bit too much for a single quarter ;))
Indeed ;) These are (very rough) estimates of how long it might take one of us to tackle individual tasks. So yes, FTEs with an element of planning poker.
- Several of the things listed there will require hardware, possibly a non-trivial amount of it. Could you make a very rough list of your expectations so that we can plan for it, or is it too early to tell?
I don't expect major hardware needs for the storage service or REST API before Q2. Until then we should be able to make do with little more than the misc servers we've been using for testing so far. The smaller services (mathoid, citations etc) will need two shared boxes for redundancy. Lets do some more detailed planning on this soon.
Gabriel
Hello,
I came kind of late to this discussion so you guys might have already talked about this, please ignore if that is the case. I see that in the goals page there is a bunch of work regarding current projects but there is little mentioning of the 'background work' needed to deploy services. It seems that the work of building the infrastructure should be called out explicitly.
For example:
1. Transport: have we chosen what is it going to be our transport protocol? as in json-rpc, protocol buffers...
2. Authentication: are services for internal consumption only (and thus inside our firewall) or do they provide an authentication scheme?
3. Logging: How would you follow requests across services, or at least, follow the jump from client to server?
4.Versioning
5.Testing (version aware)
6.Client Configs, how does the client know where services are?
7.Code generation from service api descriptor
On Thu, Jun 5, 2014 at 7:21 PM, Gabriel Wicke gwicke@wikimedia.org wrote:
Hi all,
Matt & I have published our draft goals for the nascent Services team [1] at
https://www.mediawiki.org/wiki/Services/Roadmap
Your feedback would be much appreciated.
Thanks,
Gabriel
Ops mailing list Ops@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/ops
Hi Nuria!
On 06/12/2014 09:53 AM, Nuria Ruiz wrote:
Hello,
I came kind of late to this discussion so you guys might have already talked about this, please ignore if that is the case.
Much of this has been discussed before, but I'm happy to give you pointers.
I see that in the goals page there is a bunch of work regarding current projects but there is little mentioning of the 'background work' needed to deploy services. It seems that the work of building the infrastructure should be called out explicitly.
I think some of that is already called out. We see it very much as one of our responsibilities to improve practices across services. We also already have some experience with production services like Parsoid, so we are not starting from scratch.
For example:
- Transport: have we chosen what is it going to be our transport protocol?
as in json-rpc, protocol buffers...
We are aiming for more REST and less RPC. We'll start with JSON and HTML resources, but could consider other representations later.
- Authentication: are services for internal consumption only (and thus
inside our firewall) or do they provide an authentication scheme?
As stated in the goals, we'll start the first phase with simple internal authentication, but invest time in a proper solution later on in the fiscal, in collaboration with platform. See https://www.mediawiki.org/wiki/Requests_for_comment/SOA_Authentication.
- Logging: How would you follow requests across services, or at least,
follow the jump from client to server?
Matt just set up a GELF sink feeding into logstash. Tracking requests across the infrastructure should be possible by including a unique request id in the structured log information.
4.Versioning 5.Testing (version aware)
https://www.mediawiki.org/wiki/Requests_for_comment/Storage_service#API_vers...
https://www.mediawiki.org/wiki/Services/Roadmap#Drive_automated_service_test...
Re versioning: I'm leaning towards testing version transformations separately from the main functionality, as it reduces the number of cases we need to cover.
6.Client Configs, how does the client know where services are?
Are you more interested in discovery of internal services or external API end points?
7.Code generation from service api descriptor
Are you interested in clients or mocks?
Gabriel
I will do my best to try to take a look at the docs you sent and add comments where it pertains.
We are aiming for more REST and less RPC. We'll start with JSON and HTML
resources, but could consider other representations later.
I see, in case you did not see it the heroku guys jus released a very useful set of guidelines for rest apis. The most interesting bit is pagination, I think: https://github.com/interagent/http-api-design
Are you more interested in discovery of internal services or external API end
points? Not discovery but rather configuration, so I can test serviceX locally in vagrant and also deploy the client in prod and client has access to the production configuration. In our case probably we resolve this with puppet as it seems to be the mother of all things at wikimedia.
7.Code generation from service api descriptor
Client generation that can be used for both, see for example: https://github.com/interagent/heroics
On Thu, Jun 12, 2014 at 10:28 PM, Gabriel Wicke gwicke@wikimedia.org wrote:
Hi Nuria!
On 06/12/2014 09:53 AM, Nuria Ruiz wrote:
Hello,
I came kind of late to this discussion so you guys might have already talked about this, please ignore if that is the case.
Much of this has been discussed before, but I'm happy to give you pointers.
I see that in the goals page there is a bunch of work regarding current projects but there is little mentioning of the 'background work' needed
to
deploy services. It seems that the work of building the infrastructure should be called out explicitly.
I think some of that is already called out. We see it very much as one of our responsibilities to improve practices across services. We also already have some experience with production services like Parsoid, so we are not starting from scratch.
For example:
- Transport: have we chosen what is it going to be our transport
protocol?
as in json-rpc, protocol buffers...
We are aiming for more REST and less RPC. We'll start with JSON and HTML resources, but could consider other representations later.
- Authentication: are services for internal consumption only (and thus
inside our firewall) or do they provide an authentication scheme?
As stated in the goals, we'll start the first phase with simple internal authentication, but invest time in a proper solution later on in the fiscal, in collaboration with platform. See https://www.mediawiki.org/wiki/Requests_for_comment/SOA_Authentication.
- Logging: How would you follow requests across services, or at least,
follow the jump from client to server?
Matt just set up a GELF sink feeding into logstash. Tracking requests across the infrastructure should be possible by including a unique request id in the structured log information.
4.Versioning 5.Testing (version aware)
https://www.mediawiki.org/wiki/Requests_for_comment/Storage_service#API_vers...
https://www.mediawiki.org/wiki/Services/Roadmap#Drive_automated_service_test...
Re versioning: I'm leaning towards testing version transformations separately from the main functionality, as it reduces the number of cases we need to cover.
6.Client Configs, how does the client know where services are?
Are you more interested in discovery of internal services or external API end points?
7.Code generation from service api descriptor
Are you interested in clients or mocks?
Gabriel
mediawiki-core@lists.wikimedia.org