*tl;dr: Mobile Apps will, in partnership with the Services, investigate building a content service for the Mobile Apps.*
The Mobile Apps Team currently has quite a few pain points with the way we fetch article content currently:
- We have to make a lot of API requests to load an article: article HTML, lead image, read more recommendations, and more - We send the user HTML that we then discard, needlessly increasing data usage - We do transforms to the HTML in JavaScript on the client side, which causes code duplication across the apps and degrades user-perceived performance - Trivial changes to the API (e.g. renaming a parameter) can break the app which is problematic since apps can't be hotfixed easily
To address these challenges, we are considering performing some or all of these tasks in a service developed by the Mobile Apps Team with help from Services. This service will hit the APIs we currently hit on the client, aggregate the content we need on the server side, perform transforms we're currently doing on the client on the server instead, and serve the full response to the user via RESTBase. In addition to providing a public API end point, RESTBase would help with common tasks like monitoring, caching and authorisation.
So the Mobile Apps Team is going to spend a bit of time investigating whether using RESTBase with Node.js is an option for building a content service for the Wikipedia app to replace our current method of retrieving article content. Our initial scope for this is feature parity with our current content retrieval method.
Our action items are as follows:
- Wait for RESTBase to be deployed. - Timescale: Weeks - Owner: All of us :-) - Figure out what information the service should serve for the first iteration (i.e. for feature parity) and what APIs it needs to hit to do that - Timescale: Wed 4th Feb - Owner: Dan Garry - Start implementing the service and see whether it meets our needs - Timescale: Planning a spike for next apps sprint (16th Feb - 27th Feb) to perform initial investigation - Owner: Currently undecided engineer from Mobile Apps, with Services engineers serving as consultants
As always, feel free to ask if there are any questions.
Dan
Thanks for getting this ball rolling, Dan! Couldn't agree more with the points you raised—having in fact raised a few of them myself. Put me down as one of the mobile/full-stack engineers who wants to work on this service :-).
On Tue, Feb 3, 2015 at 8:46 PM, Dan Garry dgarry@wikimedia.org wrote:
*tl;dr: Mobile Apps will, in partnership with the Services, investigate building a content service for the Mobile Apps.*
The Mobile Apps Team currently has quite a few pain points with the way we fetch article content currently:
- We have to make a lot of API requests to load an article: article
HTML, lead image, read more recommendations, and more
- We send the user HTML that we then discard, needlessly increasing data
usage
- We do transforms to the HTML in JavaScript on the client side, which
causes code duplication across the apps and degrades user-perceived performance
- Trivial changes to the API (e.g. renaming a parameter) can break the
app which is problematic since apps can't be hotfixed easily
To address these challenges, we are considering performing some or all of these tasks in a service developed by the Mobile Apps Team with help from Services. This service will hit the APIs we currently hit on the client, aggregate the content we need on the server side, perform transforms we're currently doing on the client on the server instead, and serve the full response to the user via RESTBase. In addition to providing a public API end point, RESTBase would help with common tasks like monitoring, caching and authorisation.
So the Mobile Apps Team is going to spend a bit of time investigating whether using RESTBase with Node.js is an option for building a content service for the Wikipedia app to replace our current method of retrieving article content. Our initial scope for this is feature parity with our current content retrieval method.
Our action items are as follows:
- Wait for RESTBase to be deployed.
- Timescale: Weeks
- Owner: All of us :-)
- Figure out what information the service should serve for the first
iteration (i.e. for feature parity) and what APIs it needs to hit to do that
- Timescale: Wed 4th Feb
- Owner: Dan Garry
- Start implementing the service and see whether it meets our needs
- Timescale: Planning a spike for next apps sprint (16th Feb - 27th Feb) to perform initial investigation
engineers serving as consultants
- Owner: Currently undecided engineer from Mobile Apps, with Services
As always, feel free to ask if there are any questions.
Dan _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On 04/02/15 12:46, Dan Garry wrote:
To address these challenges, we are considering performing some or all of these tasks in a service developed by the Mobile Apps Team with help from Services. This service will hit the APIs we currently hit on the client, aggregate the content we need on the server side, perform transforms we're currently doing on the client on the server instead, and serve the full response to the user via RESTBase. In addition to providing a public API end point, RESTBase would help with common tasks like monitoring, caching and authorisation.
I don't really understand why you want it to be integrated with RESTBase. As far as I can tell (it is hard to pin these things down), RESTBase is a revision storage backend and possibly a public API for that backend. I thought the idea of SOA was to separate concerns. Wouldn't monitoring, caching and authorization would be best done as a node.js library which RESTBase and other services use?
-- Tim Starling
On Tue, Feb 3, 2015 at 8:42 PM, Tim Starling tstarling@wikimedia.org wrote:
I don't really understand why you want it to be integrated with RESTBase. As far as I can tell (it is hard to pin these things down), RESTBase is a revision storage backend and possibly a public API for that backend.
Actually, RESTBase's logic applies to the Mobile Apps case quite naturally. When a page is fetched and transformed, it can be stored so that consequent requests can simply retrieve the transformed document form storage.
I thought the idea of SOA was to separate concerns. Wouldn't monitoring, caching and authorization would be best done as a node.js library which RESTBase and other services use?
Good point. Ideally, what we would need to do is provide the right tools to developers to create services, which can then be placed "strategically" around DCs (in cooperation with Ops, ofc). For v1, however, we plan to provide only logical separation (to a certain extent) via modules which can be dynamically loaded/unloaded from RESTBase. In return, RESTBase will provide them with routing, monitoring, caching and authorisation out of the box. The good point here is that this 'modularisation' eases the transition to a more-decomposed orchestration SOA model. Going in that direction, however, requires some prerequisites to be fulfilled, such as [1].
Marko
[1] https://phabricator.wikimedia.org/T84923
Marko Obrovac Senior Services Engineer Wikimedia Foundation
tl;dr; Before the Mobile Apps Team embarks on its own path, I really think we should work together as an organization to figure out a better strategy for implementing a set of consistent services that address the needs of all platforms.
So… I've been involved in similar conversations at other organizations. They generally sound like this:
"These APIs were designed before mobile. Mobile needs mobile friendly services to be great. This services currently don't exist and/or aren't being implemented fast enough for us to build the features we want. Lets implement our own services!" (Ok that was a bit contrived but you get the point)
The problem I have with this solution is two fold:
1. The owners of this services project should really be the services team. 2. Building a separate service for mobile does not align the needs of the mobile team with the rest of the MediaWiki API consumers.
To be clear - I totally agree that we should be working with services engineers to collaboratively develop APIs. In fact, Mobile Apps, along with other front end teams, should be proactively giving the services team input and support so they can build the services that we need. This results in making the API not only better for Mobile Apps, but the other platforms as well.
Working around our services issues by providing the mobile apps with different set of APIs than the rest of the platforms comes with its own set of problems. In the long run it creates more work (maintaining multiple APIs) and leads ambiguity ("Oh, that only works on the mobile API"). It also creates a new point of failure that would only affect mobile apps. This means at times consumers of the "Real API" may have needs that conflict with mobile apps and compete for the time and attention of the services engineers.
Being on iOS and Android already puts the mobile apps on an island because of the language and framework differences. Using a different API won't help that, but development of an API that serves the needs of all the platforms aligns our needs with the rest of the organization while providing all front end teams a consistent way to access data.
This doesn't even address the fact that the mobile team is staffed with experts in tuning native apps - which is quite at different skill set than writing, deploying, and maintaining server side java script. Write good scalable server code is difficult, and this will be extremely important here, since, for the foreseeable future, mobile traffic is expected to see the most growth.
It should go without saying, but I'm obviously not against Mobile App Engineers submitting patches to the API if they want to, maybe even hack using days (though rumors have it that iOS is a bit busy this quarter). But this is quite different than owning the service.
After reading this, you would probably assume all my experiences with this strategy were bad. Ironically, I was recently on a mobile team that developed its own services successfully. I even helped to come up with the Acronym: MISL (Mobile Intermediary Services Layer). :)
However, on that mobile team we had 2 experienced AND dedicated javascript/python engineers that were specifically tasked to write services/administration panels and who interfaced directly with the main services team.
(Now, If we are going to start talking about embedding web services engineers on the Mobile Apps team, I'm definitely in!)
On Wed, Feb 4, 2015 at 12:59 AM, Marko Obrovac mobrovac@wikimedia.org wrote:
On Tue, Feb 3, 2015 at 8:42 PM, Tim Starling tstarling@wikimedia.org wrote:
I don't really understand why you want it to be integrated with RESTBase. As far as I can tell (it is hard to pin these things down), RESTBase is a revision storage backend and possibly a public API for that backend.
Actually, RESTBase's logic applies to the Mobile Apps case quite naturally. When a page is fetched and transformed, it can be stored so that consequent requests can simply retrieve the transformed document form storage.
I thought the idea of SOA was to separate concerns. Wouldn't monitoring, caching and authorization would be best done as a node.js library which RESTBase and other services use?
Good point. Ideally, what we would need to do is provide the right tools to developers to create services, which can then be placed "strategically" around DCs (in cooperation with Ops, ofc). For v1, however, we plan to provide only logical separation (to a certain extent) via modules which can be dynamically loaded/unloaded from RESTBase. In return, RESTBase will provide them with routing, monitoring, caching and authorisation out of the box. The good point here is that this 'modularisation' eases the transition to a more-decomposed orchestration SOA model. Going in that direction, however, requires some prerequisites to be fulfilled, such as [1].
Marko
[1] https://phabricator.wikimedia.org/T84923
Marko Obrovac Senior Services Engineer Wikimedia Foundation _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Good point. Ideally, what we would need to do is provide the right tools to developers to create services, which can then be placed "strategically" around DCs (in cooperation with Ops, ofc).
Yes. As an organization we should provide good tools that allow developers to create services. I do fail to understand the "strategically" around DCs part though.
For v1, however, we plan to provide only logical separation (to a certain extent) via modules which can be dynamically loaded/unloaded from RESTBase.
modules ? Care to explain a bit more ? AFAIK RESTBase is a revision storage service and to be honest I am fighting to understand what modules you are referring to and the architecture behind those modules.
In return, RESTBase will provide them with routing, monitoring, caching and authorisation out of the box. The good point here is that this 'modularisation' eases the transition to a more-decomposed orchestration SOA model. Going in that direction, however, requires some prerequisites to be fulfilled, such as [1].
While revision caching can very well be done by RESTBase (AFAIK, that is one reason it is being created for), authorization (It's not revision authorization, but generic authentication/authorization I am referring to) and monitoring should not be provided by RESTBase to any service. Especially monitoring. Services (whatever their nature) should provide discoverable (REST if you like, as I suspect you do) endpoints that allow monitoring via third party tools and not depend on an another service for that. My take is that there should be a swagger manifest that describes a basic monitoring framework and services should each independently implement it (including RESTBase)
I am also a bit unclear on the routing aspect. Care to point out an up to date architectural diagram ? I have been told in person that the one https://www.npmjs.com/package/restbase is not up to date so I can't comment on that.
On 04/02/15 16:59, Marko Obrovac wrote:
For v1, however, we plan to provide only logical separation (to a certain extent) via modules which can be dynamically loaded/unloaded from RESTBase. In return, RESTBase will provide them with routing, monitoring, caching and authorisation out of the box.
We already have routing, monitoring and caching in Varnish. That seems like a good place to implement further service routing to me.
It's simple to configure, has excellent performance and scalability, and monitoring and a distributed logging system are already implemented.
It doesn't have authorisation, but I thought that was going to be in a separate service from RESTBase anyway.
-- Tim Starling
Tim, I like Varnish's vcl flexibility, but not the debugging aspects. Still, +1, but could you elaborate on how you see:
* Services communicate with each other - via varnish as well or directly? * Do you see routing varnish layer as non-caching, only to forward request to the second tear service-specific varnish instance that will actually perform caching? * Will there be any problems separating services into separate, non mutually trusted clusters? * service management - varnish would only indicate overall health of the service. What about service-specific real time monitored values (if needed), logs, controlling service instances. Any thoughts on generalizing that infrastructure?
Lastly, semi related - my service will also need a node.js lib to access MW api. How should we manage common libs like that?
Thanks! On Feb 6, 2015 4:15 AM, "Tim Starling" tstarling@wikimedia.org wrote:
On 04/02/15 16:59, Marko Obrovac wrote:
For v1, however, we plan to provide only logical separation (to a certain extent) via modules which
can
be dynamically loaded/unloaded from RESTBase. In return, RESTBase will provide them with routing, monitoring, caching and authorisation out of
the
box.
We already have routing, monitoring and caching in Varnish. That seems like a good place to implement further service routing to me.
< https://git.wikimedia.org/blob/operations%2Fpuppet/4a7f5ce62d9cdd1ace20ca6c4...
It's simple to configure, has excellent performance and scalability, and monitoring and a distributed logging system are already implemented.
It doesn't have authorisation, but I thought that was going to be in a separate service from RESTBase anyway.
-- Tim Starling
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On Thu, Feb 5, 2015 at 9:00 PM, Yuri Astrakhan yastrakhan@wikimedia.org wrote:
Lastly, semi related - my service will also need a node.js lib to access MW api. How should we manage common libs like that?
Libraries published on npm using semantic versioning and imported to the individual services as needed.
Bryan
On Wed, Feb 4, 2015 at 6:59 AM, Marko Obrovac mobrovac@wikimedia.org wrote:
On Tue, Feb 3, 2015 at 8:42 PM, Tim Starling tstarling@wikimedia.org wrote:
I don't really understand why you want it to be integrated with RESTBase. As far as I can tell (it is hard to pin these things down), RESTBase is a revision storage backend and possibly a public API for that backend.
Actually, RESTBase's logic applies to the Mobile Apps case quite naturally. When a page is fetched and transformed, it can be stored so that consequent requests can simply retrieve the transformed document form storage.
Ok, so in this vision RESTBase is a caching layer for revisions
I thought the idea of SOA was to separate concerns. Wouldn't monitoring, caching and authorization would be best done as a node.js library which RESTBase and other services use?
Good point. Ideally, what we would need to do is provide the right tools to developers to create services, which can then be placed "strategically" around DCs (in cooperation with Ops, ofc). For v1, however, we plan to provide only logical separation (to a certain extent) via modules which can be dynamically loaded/unloaded from RESTBase. In return, RESTBase will provide them with routing, monitoring, caching and authorisation out of the box. The good point here is that this 'modularisation' eases the transition to a more-decomposed orchestration SOA model. Going in that direction, however, requires some prerequisites to be fulfilled, such as [1].
So, now RESTBase has become a router and an auth provider as well? (Gabriel already clarified me that "monitoring" means that RESTbase will expose its own metrics for that specific service, so this is not monitoring of the service at all, rather accounting).
I need some clarification at this point - what is RESTBase really going to do? I'm asking because when I read "RESTBase will provide them with routing, [...] and authorisation" I immediately think of a request router and a general on-wiki auth provider. And we already have both, and re-doing them in RESTBase would be plainly wrong.
Maybe you intend very specific things when you say "routing" and not request routing, which is what everybody here will think of. And when you say "auth" you mean that RESTBase implements an auth schema for its clients, so that no client can access data from another one. If this is the case, I have some further questions: is this going to be RBAC? Which permissions models are you implementing? Are we sure it is what we will need? And foremost: will this be exposable to external consumers? will it be able to hook up to our traditional wiki auth scheme?
Can you please expand a bit on those concepts? Or a lot of confusion, uncertainty and doubt will spread amongst your fellow engineers, resulting in an hostile view of what may be a perfectly well designed software.
Cheers,
Giuseppe
On Wed, Feb 4, 2015 at 6:59 AM, Marko Obrovac mobrovac@wikimedia.org wrote:
On Tue, Feb 3, 2015 at 8:42 PM, Tim Starling tstarling@wikimedia.org wrote:
I don't really understand why you want it to be integrated with RESTBase. As far as I can tell (it is hard to pin these things down), RESTBase is a revision storage backend and possibly a public API for that backend.
Actually, RESTBase's logic applies to the Mobile Apps case quite naturally. When a page is fetched and transformed, it can be stored so that consequent requests can simply retrieve the transformed document form storage.
Ok, so in this vision RESTBase is a caching layer for revisions
I thought the idea of SOA was to separate concerns. Wouldn't monitoring, caching and authorization would be best done as a node.js library which RESTBase and other services use?
Good point. Ideally, what we would need to do is provide the right tools to developers to create services, which can then be placed "strategically" around DCs (in cooperation with Ops, ofc). For v1, however, we plan to provide only logical separation (to a certain extent) via modules which can be dynamically loaded/unloaded from RESTBase. In return, RESTBase will provide them with routing, monitoring, caching and authorisation out of the box. The good point here is that this 'modularisation' eases the transition to a more-decomposed orchestration SOA model. Going in that direction, however, requires some prerequisites to be fulfilled, such as [1].
So, now RESTBase has become a router and an auth provider as well? (Gabriel already clarified me that "monitoring" means that RESTbase will expose its own metrics for that specific service, so this is not monitoring of the service at all, rather accounting).
I need some clarification at this point - what is RESTBase really going to do? I'm asking because when I read "RESTBase will provide them with routing, [...] and authorisation" I immediately think of a request router and a general on-wiki auth provider. And we already have both, and re-doing them in RESTBase would be plainly wrong.
Maybe you intend very specific things when you say "routing" and not request routing, which is what everybody here will think of. And when you say "auth" you mean that RESTBase implements an auth schema for its clients, so that no client can access data from another one. If this is the case, I have some further questions: is this going to be RBAC? Which permissions models are you implementing? Are we sure it is what we will need? And foremost: will this be exposable to external consumers? will it be able to hook up to our traditional wiki auth scheme?
Can you please expand a bit on those concepts? Or a lot of confusion, uncertainty and doubt will spread amongst your fellow engineers, resulting in an hostile view of what may be a perfectly well designed software.
Cheers,
Giuseppe
On Wed, Feb 4, 2015 at 5:42 AM, Tim Starling tstarling@wikimedia.org wrote:
On 04/02/15 12:46, Dan Garry wrote:
To address these challenges, we are considering performing some or all of these tasks in a service developed by the Mobile Apps Team with help from Services. This service will hit the APIs we currently hit on the client, aggregate the content we need on the server side, perform transforms we're currently doing on the client on the server instead, and serve the full response to the user via RESTBase. In addition to providing a public API end point, RESTBase would help with common tasks like monitoring, caching and authorisation.
I don't really understand why you want it to be integrated with RESTBase. As far as I can tell (it is hard to pin these things down), RESTBase is a revision storage backend and possibly a public API for that backend. I thought the idea of SOA was to separate concerns. Wouldn't monitoring, caching and authorization would be best done as a node.js library which RESTBase and other services use?
I agree with Tim. Using RESTBase as an integration layer for everything is SOA done wrong. If we need to have an authorization system, which is different from our APIs, we need to build it separately, not to add levels of indirection.
Doing 4 things from one single service is basically rebuilding the mediawiki monolith, only in a different language :)
What you need, IMO, is a thin proxy layer in front of all the separate APIs you have to call, including restbase for caching/revision storage. It may be built into the app or, if it is consumed by multiple apps, built as a thin proxy service itself.
(I also don't get what "monitoring" means here, but someone could probably explain it to me)
Cheers,
Giuseppe
On 02/03/2015 05:46 PM, Dan Garry wrote:
*tl;dr: Mobile Apps will, in partnership with the Services, investigate building a content service for the Mobile Apps.*
The Mobile Apps Team currently has quite a few pain points with the way we fetch article content currently:
- We have to make a lot of API requests to load an article: article
HTML, lead image, read more recommendations, and more
Is there a bug for this and the other issues? I'm subscribed to [1], but I don't see anything like the issues you've mentioned on it.
What are "read more recommendations"?
- We send the user HTML that we then discard, needlessly increasing data
usage
- We do transforms to the HTML in JavaScript on the client side, which
causes code duplication across the apps and degrades user-perceived performance
- Trivial changes to the API (e.g. renaming a parameter) can break the
app which is problematic since apps can't be hotfixed easily
Is there an actual instance where an API change has broken an app or is this merely a hypothetical concern? Speaking as an API developer, we occasionally have to make breaking changes, and if we broke something too fast, it would be nice to have feedback if that was the case so we can improve in the future.
[1] https://phabricator.wikimedia.org/T75616
-- Legoktm
Hey Kunal,
Responses in-line.
On 3 February 2015 at 21:09, Legoktm legoktm.wikipedia@gmail.com wrote:
Is there a bug for this and the other issues? I'm subscribed to [1], but I don't see anything like the issues you've mentioned on it.
There's this, which documents some of them, but it's more descriptive of a specific proposed solution: https://phabricator.wikimedia.org/T87824
I welcome any other ideas that anyone has for solving our problems. We're not tied to a specific solution.
What are "read more recommendations"?
When you scroll to the end of an article, we give three suggestions as to what you could read next. The suggestions are generated quite naively by running a full text search using the current page's title as a query. Our metrics have shown a lot of positive engagement with the feature so far.
Here's a screenshot on [[Bern]]: http://i.imgur.com/qnHcsdT.png
Is there an actual instance where an API change has broken an app or is this merely a hypothetical concern? Speaking as an API developer, we occasionally have to make breaking changes, and if we broke something too fast, it would be nice to have feedback if that was the case so we can improve in the future.
I'm referring to a possible upcoming breaking change in MobileFrontend, which is the API we depend on for our content. A certain API parameter may be removed. Since the apps were made under the assumption that that value would always be in the API response, the app doesn't handle this issue well. It leaves users unable to see file pages.
This issue is particularly prominent on apps since if the user never updates the app then they'll never get the fix.
Dan
On Tue, Feb 3, 2015 at 5:46 PM, Dan Garry dgarry@wikimedia.org wrote:
To address these challenges, we are considering performing some or all of these tasks in a service developed by the Mobile Apps Team with help from Services. This service will hit the APIs we currently hit on the client, aggregate the content we need on the server side, perform transforms we're currently doing on the client on the server instead, and serve the full response to the user via RESTBase. In addition to providing a public API end point, RESTBase would help with common tasks like monitoring, caching and authorisation.
Using https://phabricator.wikimedia.org/T87824 as a reference point for what you're talking about -
I think you will generally find agreement that moving client-side transformations that only live in the app to server-side code that enables access by multiple consumers and caching is a good idea. If there are reasons not do to this, now'd be a good time to speak up.
If not, then I think one thing to keep in mind is how to organize the transformation code in a manner that it doesn't just become a server-side hodgepodge still only useful to one consumer, to avoid some of the pitfalls Brian mentions. Say you want to reformat infoboxes on the mobile web, but not do all the other stuff the mobile app does. Can you just get that specific transformation? Are some transformations dependent on others? Or say we want to make a change only for the output that gets fed into the PDF generator, but not for any other outputs. Can we do that?
Or a more pressing concern for the app team itself, what about alpha, beta, stable version of the apps -- how would those get more/less experimental versions of the output? Or languages -- are there cases where we apply a transformation only in one language, but not another?
Do we need a way to register schemas so we can easily get a certain set of inter-dependent transformations, like "mobile app stable", "desktop web", etc.? Or are these all just API/service parameters?
Just some early questions as we're thinking this through.
Erik
On Tue, Feb 3, 2015 at 11:33 PM, Erik Moeller erik@wikimedia.org wrote:
I think you will generally find agreement that moving client-side transformations that only live in the app to server-side code that enables access by multiple consumers and caching is a good idea. If there are reasons not do to this, now'd be a good time to speak up.
If not, then I think one thing to keep in mind is how to organize the transformation code in a manner that it doesn't just become a server-side hodgepodge still only useful to one consumer, to avoid some of the pitfalls Brian mentions. Say you want to reformat infoboxes on the mobile web, but not do all the other stuff the mobile app does. Can you just get that specific transformation? Are some transformations dependent on others? Or say we want to make a change only for the output that gets fed into the PDF generator, but not for any other outputs. Can we do that?
Right now the plan is to start from plain Parsoid HTML. The mobile app service would be called for each new revision to prime the cache / storage. Chaining transformations might be possible, but right now it's not clear that it would be worth the complexity. Currently AFAIK only OCG and mobile apps have strong transformation needs, and there seems to be little overlap in the way they transform the content. Mobile web still wraps sections into divs, but we are looking into eliminating that by possibly integrating the section markup into the regular Parsoid output.
Regarding general-purpose APIs vs. mobile: I think mobile is in some ways a special case as their content transformation needs are closely coupled with the way the apps are presenting the content. Additionally, at least until SPDY is deployed there is a strong performance incentive to bundle information in a single response tailored to the app's needs. One strategy employed by Netflix is to introduce a second API layer http://techblog.netflix.com/2012/07/embracing-differences-inside-netflix.html on top of the general content API to handle device-specific needs. I think this is a sound strategy, as it contains the volatility in a separate layer while ensuring that everything is ultimately consuming the general-purpose API. If the need for app-specific massaging disappears over time, we can simply shut down the custom service / API end point without affecting the general API.
Gabriel
One strategy employed by Netflix is to introduce a second API layer < http://techblog.netflix.com/2012/07/embracing-differences-inside-netflix.htm...
on top of the general content API to handle device-specific needs. I think this is a sound strategy, as it contains the volatility in a separate layer while ensuring that everything is ultimately consuming the general-purpose API.
This design appears often enough that it can likely be called a "design pattern". The Selenium/Webdriver project did exactly the same thing[1]. The API for Selenium v2 has about 1/3 as many functions as Selenium v1. People that use Selenium v2 build their own high-level APIs based on the basic core set of functions available.
Defining the scope of the "general content API" can be challenging.
On Wed, Feb 4, 2015 at 8:41 AM, Gabriel Wicke gwicke@wikimedia.org wrote:
Regarding general-purpose APIs vs. mobile: I think mobile is in some ways a special case as their content transformation needs are closely coupled with the way the apps are presenting the content. Additionally, at least until SPDY is deployed there is a strong performance incentive to bundle information in a single response tailored to the app's needs.
A notion of schemas that declare a specific set of transformations to be applied/not applied might help avoid overcomplicating things early on while addressing different transformation needs even within the growing number of mobile use cases (Android app alpha/beta/stable, iOS app alpha/beta/stable, mobile web alpha/beta/stable, third party apps), and potentially making code re-usable for desktop needs down the road. Since the number of schemas would be limited, and specifying the correct schema would result in a single response, performance could be optimized for each use case.
Erik
On Wed, Feb 4, 2015 at 11:41 AM, Gabriel Wicke gwicke@wikimedia.org wrote:
Regarding general-purpose APIs vs. mobile: I think mobile is in some ways a special case as their content transformation needs are closely coupled with the way the apps are presenting the content. Additionally, at least until SPDY is deployed there is a strong performance incentive to bundle information in a single response tailored to the app's needs. One strategy employed by Netflix is to introduce a second API layer < http://techblog.netflix.com/2012/07/embracing-differences-inside-netflix.htm...
on top of the general content API to handle device-specific needs. I think this is a sound strategy, as it contains the volatility in a separate layer while ensuring that everything is ultimately consuming the general-purpose API. If the need for app-specific massaging disappears over time, we can simply shut down the custom service / API end point without affecting the general API.
I can definitely understand that motivation for providing mobile specific service layer - so if the services team wants to implement the API in this way and support that architecture, I am totally on board.
My remaining hesitation here is that from the reading of this proposal, the mobile team is the owner of implementing this service, not the services team (Maybe I am misreading?).
This leads me to ask questions like: Why is the mobile apps team investigating which is the best server side technology? That seems outside of our domain knowledge. Who will be responsible for maintaining this code? Who will be testing it to make sure that is performant?
I'm new, so maybe these answers are obvious to others, but to me they seem fuzzy when responsibilities are divided between two teams.
I would propose that this be a project that the Services Team owns. And that the Mobile Apps Team defines specs on what they need the new service to provide.
I think the way we'd want to go is roughly to have a *partnership between* the Services and Mobile teams produce and maintain the service.
(Note that the state of the art is that Mobile Apps are using Mobile Web's MobileFrontend extension as an intermediate API to aggregate & format page data -- which basically means Max has done the server-side API work for Mobile Apps so far.)
I'd expect to see Max and/or someone else from the Mobile team collaborating with the Services team to create what y'all need: 1) something that does what Mobile Apps needs it to... 2) and can be maintained like Services needs it to.
In general I'm in favor of more ad-hoc project-specific teams rather than completely siloing every service to the Services group, or every mobile UI to the Mobile group.
-- brion
On Wed, Feb 4, 2015 at 2:29 PM, Corey Floyd cfloyd@wikimedia.org wrote:
On Wed, Feb 4, 2015 at 11:41 AM, Gabriel Wicke gwicke@wikimedia.org wrote:
Regarding general-purpose APIs vs. mobile: I think mobile is in some
ways a
special case as their content transformation needs are closely coupled
with
the way the apps are presenting the content. Additionally, at least until SPDY is deployed there is a strong performance incentive to bundle information in a single response tailored to the app's needs. One
strategy
employed by Netflix is to introduce a second API layer <
http://techblog.netflix.com/2012/07/embracing-differences-inside-netflix.htm...
on top of the general content API to handle device-specific needs. I think this is a sound strategy, as it contains the volatility in a separate
layer
while ensuring that everything is ultimately consuming the
general-purpose
API. If the need for app-specific massaging disappears over time, we can simply shut down the custom service / API end point without affecting the general API.
I can definitely understand that motivation for providing mobile specific service layer - so if the services team wants to implement the API in this way and support that architecture, I am totally on board.
My remaining hesitation here is that from the reading of this proposal, the mobile team is the owner of implementing this service, not the services team (Maybe I am misreading?).
This leads me to ask questions like: Why is the mobile apps team investigating which is the best server side technology? That seems outside of our domain knowledge. Who will be responsible for maintaining this code? Who will be testing it to make sure that is performant?
I'm new, so maybe these answers are obvious to others, but to me they seem fuzzy when responsibilities are divided between two teams.
I would propose that this be a project that the Services Team owns. And that the Mobile Apps Team defines specs on what they need the new service to provide. _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
In general I'm in favor of more ad-hoc project-specific teams rather than
completely siloing every service to the Services group, or every mobile UI to the Mobile group.
I strongly agree. Based on experience on both sides of this spectrum, I recommend (when feasible) favoring feature teams over functional teams.
On Wed, Feb 4, 2015 at 3:00 PM, Brion Vibber bvibber@wikimedia.org wrote:
I think the way we'd want to go is roughly to have a *partnership between* the Services and Mobile teams produce and maintain the service.
(Note that the state of the art is that Mobile Apps are using Mobile Web's MobileFrontend extension as an intermediate API to aggregate & format page data -- which basically means Max has done the server-side API work for Mobile Apps so far.)
I'd expect to see Max and/or someone else from the Mobile team collaborating with the Services team to create what y'all need:
- something that does what Mobile Apps needs it to...
- and can be maintained like Services needs it to.
In general I'm in favor of more ad-hoc project-specific teams rather than completely siloing every service to the Services group, or every mobile UI to the Mobile group.
-- brion
On Wed, Feb 4, 2015 at 2:29 PM, Corey Floyd cfloyd@wikimedia.org wrote:
On Wed, Feb 4, 2015 at 11:41 AM, Gabriel Wicke gwicke@wikimedia.org wrote:
Regarding general-purpose APIs vs. mobile: I think mobile is in some
ways a
special case as their content transformation needs are closely coupled
with
the way the apps are presenting the content. Additionally, at least
until
SPDY is deployed there is a strong performance incentive to bundle information in a single response tailored to the app's needs. One
strategy
employed by Netflix is to introduce a second API layer <
http://techblog.netflix.com/2012/07/embracing-differences-inside-netflix.htm...
on top of the general content API to handle device-specific needs. I think this is a sound strategy, as it contains the volatility in a separate
layer
while ensuring that everything is ultimately consuming the
general-purpose
API. If the need for app-specific massaging disappears over time, we
can
simply shut down the custom service / API end point without affecting
the
general API.
I can definitely understand that motivation for providing mobile specific service layer - so if the services team wants to implement the API in
this
way and support that architecture, I am totally on board.
My remaining hesitation here is that from the reading of this proposal,
the
mobile team is the owner of implementing this service, not the services team (Maybe I am misreading?).
This leads me to ask questions like: Why is the mobile apps team investigating which is the best server side technology? That seems outside of our domain knowledge. Who will be responsible for maintaining this code? Who will be testing it to make sure that is performant?
I'm new, so maybe these answers are obvious to others, but to me they
seem
fuzzy when responsibilities are divided between two teams.
I would propose that this be a project that the Services Team owns. And that the Mobile Apps Team defines specs on what they need the new service to provide. _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
In general I'm in favor of more ad-hoc project-specific teams rather than
completely siloing every service to the Services group, or every mobile UI to the Mobile group.= Agreed, as long as everyone deploying services communicates through services team so there is no duplication of solutions. We should have 1 routing layer, standard monitoring and standard caching as it is been pointed out before. Specifics of services might be different but it seems the core mission of service team to provide common infrastructure, right?. That means that the 1st service we develop will be more work for services team than subsequent services. *Ideally* the mobile team should develop the service without having to solve (for example) any caching problems that are not mobile specific.
On Wed, Feb 4, 2015 at 3:04 PM, James Douglas jdouglas@wikimedia.org wrote:
In general I'm in favor of more ad-hoc project-specific teams rather than
completely siloing every service to the Services group, or every mobile UI to the Mobile group.
I strongly agree. Based on experience on both sides of this spectrum, I recommend (when feasible) favoring feature teams over functional teams.
On Wed, Feb 4, 2015 at 3:00 PM, Brion Vibber bvibber@wikimedia.org wrote:
I think the way we'd want to go is roughly to have a *partnership
between*
the Services and Mobile teams produce and maintain the service.
(Note that the state of the art is that Mobile Apps are using Mobile
Web's
MobileFrontend extension as an intermediate API to aggregate & format
page
data -- which basically means Max has done the server-side API work for Mobile Apps so far.)
I'd expect to see Max and/or someone else from the Mobile team collaborating with the Services team to create what y'all need:
- something that does what Mobile Apps needs it to...
- and can be maintained like Services needs it to.
In general I'm in favor of more ad-hoc project-specific teams rather than completely siloing every service to the Services group, or every mobile
UI
to the Mobile group.
-- brion
On Wed, Feb 4, 2015 at 2:29 PM, Corey Floyd cfloyd@wikimedia.org
wrote:
On Wed, Feb 4, 2015 at 11:41 AM, Gabriel Wicke gwicke@wikimedia.org wrote:
Regarding general-purpose APIs vs. mobile: I think mobile is in some
ways a
special case as their content transformation needs are closely
coupled
with
the way the apps are presenting the content. Additionally, at least
until
SPDY is deployed there is a strong performance incentive to bundle information in a single response tailored to the app's needs. One
strategy
employed by Netflix is to introduce a second API layer <
http://techblog.netflix.com/2012/07/embracing-differences-inside-netflix.htm...
on top of the general content API to handle device-specific needs. I
think
this is a sound strategy, as it contains the volatility in a separate
layer
while ensuring that everything is ultimately consuming the
general-purpose
API. If the need for app-specific massaging disappears over time, we
can
simply shut down the custom service / API end point without affecting
the
general API.
I can definitely understand that motivation for providing mobile
specific
service layer - so if the services team wants to implement the API in
this
way and support that architecture, I am totally on board.
My remaining hesitation here is that from the reading of this proposal,
the
mobile team is the owner of implementing this service, not the services team (Maybe I am misreading?).
This leads me to ask questions like: Why is the mobile apps team investigating which is the best server side technology? That seems outside of our domain knowledge. Who will be responsible for maintaining this code? Who will be testing it to make sure that is performant?
I'm new, so maybe these answers are obvious to others, but to me they
seem
fuzzy when responsibilities are divided between two teams.
I would propose that this be a project that the Services Team owns. And that the Mobile Apps Team defines specs on what they need the new
service
to provide. _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On Wed, Feb 4, 2015 at 4:00 PM, Brion Vibber bvibber@wikimedia.org wrote:
I think the way we'd want to go is roughly to have a *partnership between* the Services and Mobile teams produce and maintain the service.
(Note that the state of the art is that Mobile Apps are using Mobile Web's MobileFrontend extension as an intermediate API to aggregate & format page data -- which basically means Max has done the server-side API work for Mobile Apps so far.)
I'd expect to see Max and/or someone else from the Mobile team collaborating with the Services team to create what y'all need:
- something that does what Mobile Apps needs it to...
- and can be maintained like Services needs it to.
In general I'm in favor of more ad-hoc project-specific teams rather than completely siloing every service to the Services group, or every mobile UI to the Mobile group.
+1. This is the only thing that will scale in my opinion. "Full stack" teams involving design, front end, back end, ops, release, project management, and testing resources should be formed to work on vertical slices of functionality ("features" or "products") that are prioritized by the entire organization. Thinking that some team can be called on to fulfill all of the cross-feature/product needs is madness. Services is 3 people, MediaWiki-Core is 9 people (minus standing obligations like security and performance reviews). Teams of this size cannot be expected to service all the "backend" needs of the myriad product/feature verticals that are under the WMF umbrella. If we don't have enough people to staff projects this way we are trying to do too many things at once. (Which I'm pretty sure is actually the case.)
Bryan
On 4 February 2015 at 15:00, Brion Vibber bvibber@wikimedia.org wrote:
In general I'm in favor of more ad-hoc project-specific teams rather than completely siloing every service to the Services group, or every mobile UI to the Mobile group.
Agreed. This also ensures that the service exactly meets the functional requirements, no more and no less.
Dan
wikitech-l@lists.wikimedia.org