On Mon, Jun 9, 2014 at 11:20 AM, Aaron Halfaker ahalfaker@wikimedia.org wrote:
Hmm... It's hard to evaluate your strategy without more context. Why are you limiting your query to users with more than 10k lifetime edits? Are your trying to generate a proportion of a subset of users? If so, what's the denominator?
I ran the query with various editcount limits (0, 1K, 10K, 100K, 1M); the goal was to see whether power users are affected differently. (They are, users with 100K+ edits have the highest opt-out rates.) I am basically taking the ratio of ( <users with more than X total edits who have been active since the rollout and disabled MediaViewer> / <users with more than X total edits who have been active since the rollout> ) which in this case should be ( <count(*) for up_value = ' ' group> / ( <count(*) for up_value = ' ' group> + <count(*) for up_value is NULL ' group>) )
Also, opt-out rates tend to be low no matter how obvious and desired they are. If the goal of this analysis is to find out if opt-out rates are high (or low), then I'd recommend comparing them with opt-out rates for another feature.
Any tip for what that feature should be? Are there features for which opt-out rates have been recorded a few days after they have been deployed? (I imagine one problem is that opt-out happens slowly - although in the past two days our opt-out count grew about 20%, but the ratio to active users has been remarkably stable.)