In case you didn't know, all API calls support 2 caching parameters, "maxage" and "smaxage". The first one controls client caching (basically sends headers back that will make the client cache the request), the second one controls server caching (varnish) across users. If you use the latter for things like mobileview, it's best if you take into account the fact that page content can change. In which case you'll probably want to add a bogus revisionid parameter (even if that's not a valid parameter for that particular API call), which will act as a cachebuster. Of course that means that you'll need at least one dynamic API call, or one with short caching values, to catch the fact that the revision id changed for that page. That is, if you care about not showing slightly stale wiki content... it really depends on the use case.

On Wed, Jun 3, 2015 at 10:43 PM, Brian Gerstle <bgerstle@wikimedia.org> wrote:
I'm definitely interested. I'll add some engineering tasks to the iOS backlog t optimize siteinfo queries as a starting point.  If we're going to make some additional patches to mobileview, adding an extracts snippet and adding cache headers seem like great improvements.


On Wed, Jun 3, 2015 at 2:39 PM, Adam Baso <abaso@wikimedia.org> wrote:
There are others interested in caching, too, and as third parties look to integrate api.php calls, it is important there as well. You're in good company!

Anyone in Reading web engineering or infrastructure know how easy/hard it would be to have the caching related headers for action=mobileview mimic MFE HTML as well as whether that would be advisable? If we can shave, say, 100ms on respones and lower utilization at the Apaches, seems like a good investment while the mobile app content service is stood up and polished. 

Brian, from what I heard you say yesterday, the sitematrix and siteinfo stuff shouldn't be too risky. Interested in tackling the PHP? Need a pairing partner?

-Adam


On Wednesday, June 3, 2015, Brian Gerstle <bgerstle@wikimedia.org> wrote:
Also, I'm the guy complaining about cache control :-).

The other related conversation is the future of mobileview itself, and if/when/how the apps will interact w/ MW-core vs. RESTBase micro services.  Just something to keep in mind that might affect how much (if any) effort we put into working on mobileview as opposed to investing in RESTBase-powered services, which (IIUC) already have some caching mechanisms to hook into.  

That said, some MW API's which are not page related could benefit from cache-control, like sitematrix & siteinfo.

On Wed, Jun 3, 2015 at 10:09 AM, Brian Gerstle <bgerstle@wikimedia.org> wrote:
Side note, do we need a lot of these headers?  

X-Powered-By: HHVM/3.6.1
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Varnish: 2583358918, 1834266080
Via: 1.1 varnish, 1.1 varnish
X-Cache: cp1047 miss (0), cp1060 frontend miss (0)
X-Analytics: https=1
Set-Cookie: WMF-Last-Access=03-Jun-2015;Path=/;HttpOnly;Expires=Sun, 05 Jul 2015 12:00:00 GMT

Could save a few bytes down the wire if we nixed the unused ones.


On Wed, Jun 3, 2015 at 9:28 AM, Adam Baso <abaso@wikimedia.org> wrote:
Hi team -

There was a question yesterday if we had caching headers in the APIs consumed by the apps.

I did a quick cURL on a couple URLs for Android and iOS and it seems like maybe requests are served cold from the Varnish cache (as with many API responses), even if there may be some memcached/Redis hot storage at the web server for the underlying content.

Does anyone know if there's a safe approach to implement caching headers (e.g., Cache-Control, Expires, ETag, etc.) to optimize for hot cache hits for the API endpoints used by the apps and experimental webapps? In particular, would we benefit from doing so on action=mobileview? Is there a way to canonicalize the request parameters to increase the odds of a cache hit regardless of whether iOS or Android made the first hit (either client-side or perhaps at the edge with scare VCL)? What about for other API endpoints?

-Adam

$ curl -s -D - "http://en.m.wikipedia.org/w/api.php?action=mobileview&format=json&page=Fish&prop=text%7Csections%7Cthumb%7Cimage%7Cid%7Crevision%7Cdescription%7Clastmodified%7Cnormalizedtitle%7Cdisplaytitle%7Cprotection%7Ceditable&onlyrequestedsections=1&sections=0&sectionprop=toclevel%7Cline%7Canchor&noheadings=true&thumbsize=640" -o /dev/null | grep -v GeoIP

HTTP/1.1 200 OK

Server: Apache

X-Powered-By: HHVM/3.6.1

X-Content-Type-Options: nosniff

X-Frame-Options: SAMEORIGIN

Vary: Accept-Encoding,X-Forwarded-Proto,Cookie

Content-Type: application/json; charset=utf-8

X-Varnish: 2886917462, 1761146385

Via: 1.1 varnish, 1.1 varnish

Transfer-Encoding: chunked

Date: Wed, 03 Jun 2015 13:20:08 GMT

Age: 0

Connection: keep-alive

X-Cache: cp1046 miss (0), cp1046 frontend miss (0)

Set-Cookie: WMF-Last-Access=03-Jun-2015;Path=/;HttpOnly;Expires=Sun, 05 Jul 2015 12:00:00 GMT

Cache-Control: private, s-maxage=0, max-age=0, must-revalidate


$  curl -s -D - "https://en.m.wikipedia.org/w/api.php?action=mobileview&format=json&noheadings=true&page=SM%20U-66&prop=sections%7Ctext%7Clastmodified%7Clastmodifiedby%7Clanguagecount%7Cid%7Cprotection%7Ceditable%7Cdisplaytitle%7Cthumb%7Cdescription%7Cimage&sectionprop=toclevel%7Cline%7Canchor%7Clevel%7Cnumber%7Cfromtitle%7Cindex&sections=all&thumbwidth=640" -o /dev/null | grep -v GeoIP

HTTP/1.1 200 OK

Server: nginx/1.6.2

Date: Wed, 03 Jun 2015 13:22:23 GMT

Content-Type: application/json; charset=utf-8

Transfer-Encoding: chunked

Connection: keep-alive

X-Powered-By: HHVM/3.6.1

X-Content-Type-Options: nosniff

X-Frame-Options: SAMEORIGIN

Vary: Accept-Encoding,X-Forwarded-Proto,Cookie

X-Varnish: 2583358918, 1834266080

Via: 1.1 varnish, 1.1 varnish

Age: 0

X-Cache: cp1047 miss (0), cp1060 frontend miss (0)

X-Analytics: https=1

Set-Cookie: WMF-Last-Access=03-Jun-2015;Path=/;HttpOnly;Expires=Sun, 05 Jul 2015 12:00:00 GMT

Cache-Control: private, s-maxage=0, max-age=0, must-revalidate







_______________________________________________
Mobile-l mailing list
Mobile-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mobile-l




--
EN Wikipedia user page: https://en.wikipedia.org/wiki/User:Brian.gerstle
IRC: bgerstle



--
EN Wikipedia user page: https://en.wikipedia.org/wiki/User:Brian.gerstle
IRC: bgerstle



--
EN Wikipedia user page: https://en.wikipedia.org/wiki/User:Brian.gerstle
IRC: bgerstle

_______________________________________________
Mobile-l mailing list
Mobile-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mobile-l