thanks Dan, this is awesome – I’ll give it a try this morning with some of the recent mobile cohorts. On Nov 21, 2013, at 7:39 AM, Dan Andreescu <dandreescu(a)wikimedia.org> wrote:

On Thu, Nov 21, 2013 at 9:57 AM, Dario Taraborelli <dtaraborelli(a)wikimedia.org> wrote: I tried uploading a cohort from a recent A/B test (1,780 unique user_id’s). The async validation took about 5 minutes to complete. If I create a temporary table with the data in my CSV and run a join with the user table against a slave, the query to validate that these users exist takes about 400ms if I use user_id (primary key in enwiki.user) and about 3s using user_name (unique in enwiki.user). What’s the reason why it takes so long to validate a cohort in the application? My understanding is that this is due to Labs being slow compared to stat1? -- Steven Walling, Product Manager https://wikimediafoundation.org/ _______________________________________________ Wikimetrics mailing list Wikimetrics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikimetrics

I don't think labs is that much slower though, we're talking orders of magnitude here. So, I think the reason is that currently it's validating one user at a time. Since for each record I have to check against a potential user_id and user_name match, this takes forever. Two ways to make it much faster: * batch every X users and do a where user_id in (...) or user_name in (...) query instead of checking each one * create temporary tables just like Dario did The problem is that cohorts can have users from multiple projects. That makes both approaches harder, but should still be doable. The reason I haven't done this yet is that when we scheduled 818 we broke out the performance issue and agreed we'd work on it later. Sounds important though, I'll look at it now.

fantastic, is there any chance we could get an even better performance if we allowed users to specify the field type in the upload form (if it’s just user_ids, validation will be faster and the app doesn’t need to check every single entry for a valid user_name too). I understand that by design the application makes no assumption about the type of that field (and in fact it accepts a mix of user_id’s and user_names, correct)?

is there any foreseeable use case for mixed cohorts? If not, it sounds like this would be a useful enhancement.

Thanks for this! Just did a cohort of 15K and it worked fine. I unintentionally hit the "back" button, but after clicking "forward" and my cohort was validated, not even 10 seconds later. Also, I can't seem to find the place that listed which users were not valid. Did we lose that ability? - E On Thu, Nov 21, 2013 at 3:16 PM, Dan Andreescu <dandreescu(a)wikimedia.org>wrote;wrote:

Hi Edward. Yes, we temporarily lost the UI that shows what users are invalid. I wasn't sure what exactly people needed here so I didn't hazard a guess. The data is all there though and I can easily show you invalid users and invalid reasons for your cohort. I just maybe need you and someone else to say how you'd like it to work and I can whip up a view for it tomorrow. Dario, any opinion on how invalid users should be displayed? The only weird part now is that you can't upload again. You'd have to delete the whole cohort and start over... — Sent from Mailbox for iPhone On Thu, Nov 21, 2013 at 7:20 PM, Edward Galvez <egalvez(a)wikimedia.org> wrote: Also! Just to introduce myself, I'm one of the interns with the Program Evaluation & Design team - thus this upgrade is very timely. Thank you! On Thu, Nov 21, 2013 at 3:45 PM, Edward Galvez <egalvez(a)wikimedia.org> wrote: Thanks for this! Just did a cohort of 15K and it worked fine. I unintentionally hit the "back" button, but after clicking "forward" and my cohort was validated, not even 10 seconds later. Also, I can't seem to find the place that listed which users were not valid. Did we lose that ability? - E On Thu, Nov 21, 2013 at 3:16 PM, Dan Andreescu <dandreescu(a)wikimedia.org> wrote: On Thu, Nov 21, 2013 at 6:00 PM, Dario Taraborelli <dtaraborelli(a)wikimedia.org> wrote: fantastic, is there any chance we could get an even better performance if we allowed users to specify the field type in the upload form (if it’s just user_ids, validation will be faster and the app doesn’t need to check every single entry for a valid user_name too). I understand that by design the application makes no assumption about the type of that field (and in fact it accepts a mix of user_id’s and user_names, correct)? absolutely, it would run up to 2x faster if the file was all user_ids and the user specified that up front. But currently, yes, you can mix user_ids and user_names _______________________________________________ Wikimetrics mailing list Wikimetrics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikimetrics _______________________________________________ Wikimetrics mailing list Wikimetrics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikimetrics

Thanks for this! Just did a cohort of 15K and it worked fine. I unintentionally hit the "back" button, but after clicking "forward" and my cohort was validated, not even 10 seconds later. Also, I can't seem to find the place that listed which users were not valid. Did we lose that ability?