Problem: while apps have face detection available from iOS and Android for use in lead image positioning/cropping, mobile web doesn't have it, and even for apps detection is quite slow, resulting in battery drain and UX-problematic slowdown, especially on low-end Android devices.
With Dmitry's help, I discovered Android's face detection library sources ([1], separated out to a standalone library at [2]). This means that we can build a face detection service and supply its results to all users, be that apps, web or third parties.
Thoughts?
---- [1] https://android.googlesource.com/platform/external/neven/+/master [2] https://github.com/lqs/neven
This was going to be my question during the dev summit: why *don't* we do this, and improve our detection over time (and retroactively) instead of relying on user's devices?
Which is a long way of saying: +1 :)
-- Sent from my phone, please excuse brevity. On Feb 3, 2015 12:10 PM, "Max Semenik" maxsem.wiki@gmail.com wrote:
Problem: while apps have face detection available from iOS and Android for use in lead image positioning/cropping, mobile web doesn't have it, and even for apps detection is quite slow, resulting in battery drain and UX-problematic slowdown, especially on low-end Android devices.
With Dmitry's help, I discovered Android's face detection library sources ([1], separated out to a standalone library at [2]). This means that we can build a face detection service and supply its results to all users, be that apps, web or third parties.
Thoughts?
[1] https://android.googlesource.com/platform/external/neven/+/master [2] https://github.com/lqs/neven
-- Best regards, Max Semenik ([[User:MaxSem]])
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Happy thoughts. :D
Note that face detection alone doesn't work well for all images, so we should also have a way for images to override the focal point for cropping purposes. This could be implemented as parser functions that save page properties (thus giving them standard editing/versioning/audit trail) and give them a nice user-friendly UI for seeing/overriding the crop points.
-- brion
On Tue, Feb 3, 2015 at 12:20 PM, Greg Grossmeier greg@wikimedia.org wrote:
This was going to be my question during the dev summit: why *don't* we do this, and improve our detection over time (and retroactively) instead of relying on user's devices?
Which is a long way of saying: +1 :)
-- Sent from my phone, please excuse brevity. On Feb 3, 2015 12:10 PM, "Max Semenik" maxsem.wiki@gmail.com wrote:
Problem: while apps have face detection available from iOS and Android for use in lead image positioning/cropping, mobile web doesn't have it, and even for apps detection is quite slow, resulting in battery drain and UX-problematic slowdown, especially on low-end Android devices.
With Dmitry's help, I discovered Android's face detection library sources ([1], separated out to a standalone library at [2]). This means that we can build a face detection service and supply its results to all users, be that apps, web or third parties.
Thoughts?
[1] https://android.googlesource.com/platform/external/neven/+/master [2] https://github.com/lqs/neven
-- Best regards, Max Semenik ([[User:MaxSem]])
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
In general, I just want an API that provides a way to get & set cropping of images in different scenarios. As clever as it sounds to wrap some algo in a service, it won't cover all cropping cases (just faces at first) and will probably need a way for users to fine tune it anyway.
TL;DR; +1 but I think an image cropping & preference service is more valuable.
On Tue, Feb 3, 2015 at 3:38 PM, Brion Vibber bvibber@wikimedia.org wrote:
Happy thoughts. :D
Note that face detection alone doesn't work well for all images, so we should also have a way for images to override the focal point for cropping purposes. This could be implemented as parser functions that save page properties (thus giving them standard editing/versioning/audit trail) and give them a nice user-friendly UI for seeing/overriding the crop points.
-- brion
On Tue, Feb 3, 2015 at 12:20 PM, Greg Grossmeier greg@wikimedia.org wrote:
This was going to be my question during the dev summit: why *don't* we do this, and improve our detection over time (and retroactively) instead of relying on user's devices?
Which is a long way of saying: +1 :)
-- Sent from my phone, please excuse brevity. On Feb 3, 2015 12:10 PM, "Max Semenik" maxsem.wiki@gmail.com wrote:
Problem: while apps have face detection available from iOS and Android for use in lead image positioning/cropping, mobile web doesn't have it, and even for apps detection is quite slow, resulting in battery drain and UX-problematic slowdown, especially on low-end Android devices.
With Dmitry's help, I discovered Android's face detection library sources ([1], separated out to a standalone library at [2]). This means that we can build a face detection service and supply its results to all users, be that apps, web or third parties.
Thoughts?
[1] https://android.googlesource.com/platform/external/neven/+/master [2] https://github.com/lqs/neven
-- Best regards, Max Semenik ([[User:MaxSem]])
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
I think face detection gets us a great initial default on many important images, so I'm in favor of going forward with a server-side service that can set the initial position, which can then get used on desktop, mobile web, and mobile apps for display.
In general better more flexible cropping tools for image thumbnailing would be really interesting as well, though for now we can get away with client-side cropping based on the focal area as the apps do.
Perhaps adding a 'scenario' into the thumbnail identifier:
320px-Foobar.jpg - normal image at 320px width 96px-square-Foobar.jpg - square-cropped image at 96x96 640px-landscape-Foobar.jpg - landscape-cropped "lead image style" at 640xsomething
which would pick out square or landscape crops based on the defined focal area (whether provided by the face detection service or manually set).
A couple possibilities for setting the data: * stick it in WikiData somehow? * or - use a parser function on the image description page that saves the coords to page props table
Using a parser function is relatively straightforward to implement, and a pretty UI could be made to set the coordinates without manually editing the {{#image-focalpoint:12,35}} or whatever. But something in the wikibase/wikidata world might be awesomer.
-- brion
On Tue, Feb 3, 2015 at 12:53 PM, Brian Gerstle bgerstle@wikimedia.org wrote:
In general, I just want an API that provides a way to get & set cropping of images in different scenarios. As clever as it sounds to wrap some algo in a service, it won't cover all cropping cases (just faces at first) and will probably need a way for users to fine tune it anyway.
TL;DR; +1 but I think an image cropping & preference service is more valuable.
On Tue, Feb 3, 2015 at 3:38 PM, Brion Vibber bvibber@wikimedia.org wrote:
Happy thoughts. :D
Note that face detection alone doesn't work well for all images, so we should also have a way for images to override the focal point for cropping purposes. This could be implemented as parser functions that save page properties (thus giving them standard editing/versioning/audit trail) and give them a nice user-friendly UI for seeing/overriding the crop points.
-- brion
On Tue, Feb 3, 2015 at 12:20 PM, Greg Grossmeier greg@wikimedia.org wrote:
This was going to be my question during the dev summit: why *don't* we do this, and improve our detection over time (and retroactively) instead of relying on user's devices?
Which is a long way of saying: +1 :)
-- Sent from my phone, please excuse brevity. On Feb 3, 2015 12:10 PM, "Max Semenik" maxsem.wiki@gmail.com wrote:
Problem: while apps have face detection available from iOS and Android for use in lead image positioning/cropping, mobile web doesn't have it, and even for apps detection is quite slow, resulting in battery drain and UX-problematic slowdown, especially on low-end Android devices.
With Dmitry's help, I discovered Android's face detection library sources ([1], separated out to a standalone library at [2]). This means that we can build a face detection service and supply its results to all users, be that apps, web or third parties.
Thoughts?
[1] https://android.googlesource.com/platform/external/neven/+/master [2] https://github.com/lqs/neven
-- Best regards, Max Semenik ([[User:MaxSem]])
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- EN Wikipedia user page: https://en.wikipedia.org/wiki/User:Brian.gerstle IRC: bgerstle
I think using server side face detection to provide a good initial focal area is good idea. However, I think the API itself should be agnostic to how that bounding box was determined, be it user manipulation, or automatic face detection. (Although we probably want to store the method of how it was determined)
Because of the different device sizes and browsers, it may be better to just deliver the images uncropped and just let each app/browser zoom in on the suggested focal area in the best way it can.
I would want to make sure any server side service dogfoods a publicly exposed API.
Note to Brion: I hear that Wikidata people aren't too interested in storing image metadata like this. Maybe we need to add this to Commons or create a separate Wikidata instance just image metadata.
On Tue, Feb 3, 2015 at 4:21 PM, Brion Vibber bvibber@wikimedia.org wrote:
I think face detection gets us a great initial default on many important images, so I'm in favor of going forward with a server-side service that can set the initial position, which can then get used on desktop, mobile web, and mobile apps for display.
In general better more flexible cropping tools for image thumbnailing would be really interesting as well, though for now we can get away with client-side cropping based on the focal area as the apps do.
Perhaps adding a 'scenario' into the thumbnail identifier:
320px-Foobar.jpg - normal image at 320px width 96px-square-Foobar.jpg - square-cropped image at 96x96 640px-landscape-Foobar.jpg - landscape-cropped "lead image style" at 640xsomething
which would pick out square or landscape crops based on the defined focal area (whether provided by the face detection service or manually set).
A couple possibilities for setting the data:
- stick it in WikiData somehow?
- or - use a parser function on the image description page that saves the
coords to page props table
Using a parser function is relatively straightforward to implement, and a pretty UI could be made to set the coordinates without manually editing the {{#image-focalpoint:12,35}} or whatever. But something in the wikibase/wikidata world might be awesomer.
-- brion
On Tue, Feb 3, 2015 at 12:53 PM, Brian Gerstle bgerstle@wikimedia.org wrote:
In general, I just want an API that provides a way to get & set cropping of images in different scenarios. As clever as it sounds to wrap some algo in a service, it won't cover all cropping cases (just faces at first) and will probably need a way for users to fine tune it anyway.
TL;DR; +1 but I think an image cropping & preference service is more valuable.
On Tue, Feb 3, 2015 at 3:38 PM, Brion Vibber bvibber@wikimedia.org wrote:
Happy thoughts. :D
Note that face detection alone doesn't work well for all images, so we should also have a way for images to override the focal point for cropping purposes. This could be implemented as parser functions that save page properties (thus giving them standard editing/versioning/audit trail) and give them a nice user-friendly UI for seeing/overriding the crop points.
-- brion
On Tue, Feb 3, 2015 at 12:20 PM, Greg Grossmeier greg@wikimedia.org wrote:
This was going to be my question during the dev summit: why *don't* we do this, and improve our detection over time (and retroactively) instead of relying on user's devices?
Which is a long way of saying: +1 :)
-- Sent from my phone, please excuse brevity. On Feb 3, 2015 12:10 PM, "Max Semenik" maxsem.wiki@gmail.com wrote:
Problem: while apps have face detection available from iOS and Android for use in lead image positioning/cropping, mobile web doesn't have it, and even for apps detection is quite slow, resulting in battery drain and UX-problematic slowdown, especially on low-end Android devices.
With Dmitry's help, I discovered Android's face detection library sources ([1], separated out to a standalone library at [2]). This means that we can build a face detection service and supply its results to all users, be that apps, web or third parties.
Thoughts?
[1] https://android.googlesource.com/platform/external/neven/+/master [2] https://github.com/lqs/neven
-- Best regards, Max Semenik ([[User:MaxSem]])
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- EN Wikipedia user page: https://en.wikipedia.org/wiki/User:Brian.gerstle IRC: bgerstle
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
+1 to what Corey said. I think we should focus on the main problem here: how to choose and crop images for a given frame. Facial recognition is part of the solution, but w/o a more generic approach the experience will still be broken.
In the end, I don't know if the apps team is even looking to make these improvements right now (since we're focusing on other new features instead). However, if we are looking to refine this feature across platforms, maybe we should schedule a hangout which might help us address various concerns faster than email.
On Tue, Feb 3, 2015 at 4:38 PM, Corey Floyd cfloyd@wikimedia.org wrote:
I think using server side face detection to provide a good initial focal area is good idea. However, I think the API itself should be agnostic to how that bounding box was determined, be it user manipulation, or automatic face detection. (Although we probably want to store the method of how it was determined)
Because of the different device sizes and browsers, it may be better to just deliver the images uncropped and just let each app/browser zoom in on the suggested focal area in the best way it can.
I would want to make sure any server side service dogfoods a publicly exposed API.
Note to Brion: I hear that Wikidata people aren't too interested in storing image metadata like this. Maybe we need to add this to Commons or create a separate Wikidata instance just image metadata.
On Tue, Feb 3, 2015 at 4:21 PM, Brion Vibber bvibber@wikimedia.org wrote:
I think face detection gets us a great initial default on many important images, so I'm in favor of going forward with a server-side service that can set the initial position, which can then get used on desktop, mobile web, and mobile apps for display.
In general better more flexible cropping tools for image thumbnailing would be really interesting as well, though for now we can get away with client-side cropping based on the focal area as the apps do.
Perhaps adding a 'scenario' into the thumbnail identifier:
320px-Foobar.jpg - normal image at 320px width 96px-square-Foobar.jpg - square-cropped image at 96x96 640px-landscape-Foobar.jpg - landscape-cropped "lead image style" at 640xsomething
which would pick out square or landscape crops based on the defined focal area (whether provided by the face detection service or manually set).
A couple possibilities for setting the data:
- stick it in WikiData somehow?
- or - use a parser function on the image description page that saves the
coords to page props table
Using a parser function is relatively straightforward to implement, and a pretty UI could be made to set the coordinates without manually editing the {{#image-focalpoint:12,35}} or whatever. But something in the wikibase/wikidata world might be awesomer.
-- brion
On Tue, Feb 3, 2015 at 12:53 PM, Brian Gerstle bgerstle@wikimedia.org wrote:
In general, I just want an API that provides a way to get & set cropping of images in different scenarios. As clever as it sounds to wrap some algo in a service, it won't cover all cropping cases (just faces at first) and will probably need a way for users to fine tune it anyway.
TL;DR; +1 but I think an image cropping & preference service is more valuable.
On Tue, Feb 3, 2015 at 3:38 PM, Brion Vibber bvibber@wikimedia.org wrote:
Happy thoughts. :D
Note that face detection alone doesn't work well for all images, so we should also have a way for images to override the focal point for cropping purposes. This could be implemented as parser functions that save page properties (thus giving them standard editing/versioning/audit trail) and give them a nice user-friendly UI for seeing/overriding the crop points.
-- brion
On Tue, Feb 3, 2015 at 12:20 PM, Greg Grossmeier greg@wikimedia.org wrote:
This was going to be my question during the dev summit: why *don't* we do this, and improve our detection over time (and retroactively) instead of relying on user's devices?
Which is a long way of saying: +1 :)
-- Sent from my phone, please excuse brevity. On Feb 3, 2015 12:10 PM, "Max Semenik" maxsem.wiki@gmail.com wrote:
Problem: while apps have face detection available from iOS and Android for use in lead image positioning/cropping, mobile web doesn't have it, and even for apps detection is quite slow, resulting in battery drain and UX-problematic slowdown, especially on low-end Android devices.
With Dmitry's help, I discovered Android's face detection library sources ([1], separated out to a standalone library at [2]). This means that we can build a face detection service and supply its results to all users, be that apps, web or third parties.
Thoughts?
[1] https://android.googlesource.com/platform/external/neven/+/master [2] https://github.com/lqs/neven
-- Best regards, Max Semenik ([[User:MaxSem]])
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- EN Wikipedia user page: https://en.wikipedia.org/wiki/User:Brian.gerstle IRC: bgerstle
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- Corey Floyd Software Engineer Mobile Apps / iOS Wikimedia Foundation
On 3 February 2015 at 14:00, Brian Gerstle bgerstle@wikimedia.org wrote:
In the end, I don't know if the apps team is even looking to make these improvements right now (since we're focusing on other new features instead). However, if we are looking to refine this feature across platforms, maybe we should schedule a hangout which might help us address various concerns faster than email.
These things are definitely on the plate for us soon. But, as you say, not right now. The reason for that is that historically the iOS team has had far more work on its plate than it can feasibly handle. We're still no way near our full velocity yet. I'd prefer that we not divide our focus at this critical time.
Let's revisit this in a sprint or two when we're closer to our velocity and we know how likely we are to be able to actually implement that work. Feel free to hold me to this by reminding me in a few weeks that I said it. ;-)
Dan
I hear the Foundation uses Node.js. Have some OpenCV Node.js bindings: https://github.com/peterbraden/node-opencv
Like Brian, I agree with Corey's comments. First, create the API that allows us to ask the question that we want to ask, then, with the API in place, start with a simple solution and iterate.
If we happen to extract a simple stateless facial recognition service from the solution while iterating, then… awesome!
–Sam
On Tue, Feb 3, 2015 at 10:22 PM, Dan Garry dgarry@wikimedia.org wrote:
On 3 February 2015 at 14:00, Brian Gerstle bgerstle@wikimedia.org wrote:
In the end, I don't know if the apps team is even looking to make these improvements right now (since we're focusing on other new features instead). However, if we are looking to refine this feature across platforms, maybe we should schedule a hangout which might help us address various concerns faster than email.
These things are definitely on the plate for us soon. But, as you say, not right now. The reason for that is that historically the iOS team has had far more work on its plate than it can feasibly handle. We're still no way near our full velocity yet. I'd prefer that we not divide our focus at this critical time.
Let's revisit this in a sprint or two when we're closer to our velocity and we know how likely we are to be able to actually implement that work. Feel free to hold me to this by reminding me in a few weeks that I said it. ;-)
Dan
-- Dan Garry Associate Product Manager, Mobile Apps Wikimedia Foundation
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Moved to Phab for some input from Gabriel & the Services crew: https://phabricator.wikimedia.org/T88633
Feel free to keep hashing things out there :)
On Wed, Feb 4, 2015 at 2:44 AM, Sam Smith samsmith@wikimedia.org wrote:
I hear the Foundation uses Node.js. Have some OpenCV Node.js bindings: https://github.com/peterbraden/node-opencv
Like Brian, I agree with Corey's comments. First, create the API that allows us to ask the question that we want to ask, then, with the API in place, start with a simple solution and iterate.
If we happen to extract a simple stateless facial recognition service from the solution while iterating, then… awesome!
–Sam
On Tue, Feb 3, 2015 at 10:22 PM, Dan Garry dgarry@wikimedia.org wrote:
On 3 February 2015 at 14:00, Brian Gerstle bgerstle@wikimedia.org wrote:
In the end, I don't know if the apps team is even looking to make these improvements right now (since we're focusing on other new features instead). However, if we are looking to refine this feature across platforms, maybe we should schedule a hangout which might help us address various concerns faster than email.
These things are definitely on the plate for us soon. But, as you say, not right now. The reason for that is that historically the iOS team has had far more work on its plate than it can feasibly handle. We're still no way near our full velocity yet. I'd prefer that we not divide our focus at this critical time.
Let's revisit this in a sprint or two when we're closer to our velocity and we know how likely we are to be able to actually implement that work. Feel free to hold me to this by reminding me in a few weeks that I said it. ;-)
Dan
-- Dan Garry Associate Product Manager, Mobile Apps Wikimedia Foundation
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l