Face detection service? - Mobile-l

Greg Grossmeier

3 Feb 3 Feb

9:20 p.m.

This was going to be my question during the dev summit: why *don't* we do this, and improve our detection over time (and retroactively) instead of relying on user's devices? Which is a long way of saying: +1 :) -- Sent from my phone, please excuse brevity. On Feb 3, 2015 12:10 PM, "Max Semenik" <maxsem.wiki(a)gmail.com> wrote:

...

Problem: while apps have face detection available from iOS and Android for use in lead image positioning/cropping, mobile web doesn't have it, and even for apps detection is quite slow, resulting in battery drain and UX-problematic slowdown, especially on low-end Android devices. With Dmitry's help, I discovered Android's face detection library sources ([1], separated out to a standalone library at [2]). This means that we can build a face detection service and supply its results to all users, be that apps, web or third parties. Thoughts? ---- [1] https://android.googlesource.com/platform/external/neven/+/master [2] https://github.com/lqs/neven -- Best regards, Max Semenik ([[User:MaxSem]]) _______________________________________________ Mobile-l mailing list Mobile-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l

Reply

Brian Gerstle

9:53 p.m.

In general, I just want an API that provides a way to get & set cropping of images in different scenarios. As clever as it sounds to wrap some algo in a service, it won't cover all cropping cases (just faces at first) and will probably need a way for users to fine tune it anyway. TL;DR; +1 but I think an image cropping & preference service is more valuable. On Tue, Feb 3, 2015 at 3:38 PM, Brion Vibber <bvibber(a)wikimedia.org> wrote:

...

Happy thoughts. :D Note that face detection alone doesn't work well for all images, so we should also have a way for images to override the focal point for cropping purposes. This could be implemented as parser functions that save page properties (thus giving them standard editing/versioning/audit trail) and give them a nice user-friendly UI for seeing/overriding the crop points. -- brion On Tue, Feb 3, 2015 at 12:20 PM, Greg Grossmeier <greg(a)wikimedia.org> wrote:

This was going to be my question during the dev summit: why *don't* we do this, and improve our detection over time (and retroactively) instead of relying on user's devices? Which is a long way of saying: +1 :) -- Sent from my phone, please excuse brevity. On Feb 3, 2015 12:10 PM, "Max Semenik" <maxsem.wiki(a)gmail.com> wrote:

Problem: while apps have face detection available from iOS and Android for use in lead image positioning/cropping, mobile web doesn't have it, and even for apps detection is quite slow, resulting in battery drain and UX-problematic slowdown, especially on low-end Android devices. With Dmitry's help, I discovered Android's face detection library sources ([1], separated out to a standalone library at [2]). This means that we can build a face detection service and supply its results to all users, be that apps, web or third parties. Thoughts? ---- [1] https://android.googlesource.com/platform/external/neven/+/master [2] https://github.com/lqs/neven -- Best regards, Max Semenik ([[User:MaxSem]]) _______________________________________________ Mobile-l mailing list Mobile-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l

_______________________________________________ Mobile-l mailing list Mobile-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l

-- EN Wikipedia user page: https://en.wikipedia.org/wiki/User:Brian.gerstle IRC: bgerstle

Reply

Corey Floyd

10:38 p.m.

I think using server side face detection to provide a good initial focal area is good idea. However, I think the API itself should be agnostic to how that bounding box was determined, be it user manipulation, or automatic face detection. (Although we probably want to store the method of how it was determined) Because of the different device sizes and browsers, it may be better to just deliver the images uncropped and just let each app/browser zoom in on the suggested focal area in the best way it can. I would want to make sure any server side service dogfoods a publicly exposed API. Note to Brion: I hear that Wikidata people aren't too interested in storing image metadata like this. Maybe we need to add this to Commons or create a separate Wikidata instance just image metadata. On Tue, Feb 3, 2015 at 4:21 PM, Brion Vibber <bvibber(a)wikimedia.org> wrote:

...

I think face detection gets us a great initial default on many important images, so I'm in favor of going forward with a server-side service that can set the initial position, which can then get used on desktop, mobile web, and mobile apps for display. In general better more flexible cropping tools for image thumbnailing would be really interesting as well, though for now we can get away with client-side cropping based on the focal area as the apps do. Perhaps adding a 'scenario' into the thumbnail identifier: 320px-Foobar.jpg - normal image at 320px width 96px-square-Foobar.jpg - square-cropped image at 96x96 640px-landscape-Foobar.jpg - landscape-cropped "lead image style" at 640xsomething which would pick out square or landscape crops based on the defined focal area (whether provided by the face detection service or manually set). A couple possibilities for setting the data: * stick it in WikiData somehow? * or - use a parser function on the image description page that saves the coords to page props table Using a parser function is relatively straightforward to implement, and a pretty UI could be made to set the coordinates without manually editing the {{#image-focalpoint:12,35}} or whatever. But something in the wikibase/wikidata world might be awesomer. -- brion On Tue, Feb 3, 2015 at 12:53 PM, Brian Gerstle <bgerstle(a)wikimedia.org> wrote:

In general, I just want an API that provides a way to get & set cropping of images in different scenarios. As clever as it sounds to wrap some algo in a service, it won't cover all cropping cases (just faces at first) and will probably need a way for users to fine tune it anyway. TL;DR; +1 but I think an image cropping & preference service is more valuable. On Tue, Feb 3, 2015 at 3:38 PM, Brion Vibber <bvibber(a)wikimedia.org> wrote:

Happy thoughts. :D Note that face detection alone doesn't work well for all images, so we should also have a way for images to override the focal point for cropping purposes. This could be implemented as parser functions that save page properties (thus giving them standard editing/versioning/audit trail) and give them a nice user-friendly UI for seeing/overriding the crop points. -- brion On Tue, Feb 3, 2015 at 12:20 PM, Greg Grossmeier <greg(a)wikimedia.org> wrote:

This was going to be my question during the dev summit: why *don't* we do this, and improve our detection over time (and retroactively) instead of relying on user's devices? Which is a long way of saying: +1 :) -- Sent from my phone, please excuse brevity. On Feb 3, 2015 12:10 PM, "Max Semenik" <maxsem.wiki(a)gmail.com> wrote: > Problem: while apps have face detection available from iOS and Android > for use in lead image positioning/cropping, mobile web doesn't have it, and > even for apps detection is quite slow, resulting in battery drain and > UX-problematic slowdown, especially on low-end Android devices. > > With Dmitry's help, I discovered Android's face detection library > sources ([1], separated out to a standalone library at [2]). This means > that we can build a face detection service and supply its results to all > users, be that apps, web or third parties. > > Thoughts? > > ---- > [1] https://android.googlesource.com/platform/external/neven/+/master > [2] https://github.com/lqs/neven > > -- > Best regards, > Max Semenik ([[User:MaxSem]]) > > _______________________________________________ > Mobile-l mailing list > Mobile-l(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/mobile-l > > _______________________________________________ Mobile-l mailing list Mobile-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l

_______________________________________________ Mobile-l mailing list Mobile-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l

-- EN Wikipedia user page: https://en.wikipedia.org/wiki/User:Brian.gerstle IRC: bgerstle

_______________________________________________ Mobile-l mailing list Mobile-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l

-- Corey Floyd Software Engineer Mobile Apps / iOS Wikimedia Foundation

Reply

Dan Garry

11:22 p.m.

On 3 February 2015 at 14:00, Brian Gerstle <bgerstle(a)wikimedia.org> wrote:

...

In the end, I don't know if the apps team is even looking to make these improvements right now (since we're focusing on other new features instead). However, if we are looking to refine this feature across platforms, maybe we should schedule a hangout which might help us address various concerns faster than email.

These things are definitely on the plate for us soon. But, as you say, not right now. The reason for that is that historically the iOS team has had far more work on its plate than it can feasibly handle. We're still no way near our full velocity yet. I'd prefer that we not divide our focus at this critical time. Let's revisit this in a sprint or two when we're closer to our velocity and we know how likely we are to be able to actually implement that work. Feel free to hold me to this by reminding me in a few weeks that I said it. ;-) Dan -- Dan Garry Associate Product Manager, Mobile Apps Wikimedia Foundation

Reply

Maryana Pinchuk

5 Feb 5 Feb

1:35 a.m.

Moved to Phab for some input from Gabriel & the Services crew: https://phabricator.wikimedia.org/T88633 Feel free to keep hashing things out there :) On Wed, Feb 4, 2015 at 2:44 AM, Sam Smith <samsmith(a)wikimedia.org> wrote:

...

I hear the Foundation uses Node.js. Have some OpenCV Node.js bindings: https://github.com/peterbraden/node-opencv Like Brian, I agree with Corey's comments. First, create the API that allows us to ask the question that we want to ask, then, with the API in place, start with a simple solution and iterate. If we happen to extract a simple stateless facial recognition service from the solution while iterating, then… awesome! –Sam On Tue, Feb 3, 2015 at 10:22 PM, Dan Garry <dgarry(a)wikimedia.org> wrote:

On 3 February 2015 at 14:00, Brian Gerstle <bgerstle(a)wikimedia.org> wrote:

In the end, I don't know if the apps team is even looking to make these improvements right now (since we're focusing on other new features instead). However, if we are looking to refine this feature across platforms, maybe we should schedule a hangout which might help us address various concerns faster than email.

These things are definitely on the plate for us soon. But, as you say, not right now. The reason for that is that historically the iOS team has had far more work on its plate than it can feasibly handle. We're still no way near our full velocity yet. I'd prefer that we not divide our focus at this critical time. Let's revisit this in a sprint or two when we're closer to our velocity and we know how likely we are to be able to actually implement that work. Feel free to hold me to this by reminding me in a few weeks that I said it. ;-) Dan -- Dan Garry Associate Product Manager, Mobile Apps Wikimedia Foundation _______________________________________________ Mobile-l mailing list Mobile-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l

_______________________________________________ Mobile-l mailing list Mobile-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l

-- Maryana Pinchuk Product Manager, Wikimedia Foundation wikimediafoundation.org

Reply