>QA in beta labs is good but not enough. We still need to do QA when a feature goes to production and currently
This is true but at the same time, I do not see anything in the description of your FF events that could not be tested on beta-labs. If we are talking add-block that can be tested even earlier, vagrant will be a fine venue. All the issues related to the client (browser) not emitting events can be tested on the development environment with ease. 



On Mon, Dec 15, 2014 at 4:18 PM, Leila Zia <leila@wikimedia.org> wrote:

On Mon, Dec 15, 2014 at 10:06 AM, Toby Negrin <tnegrin@wikimedia.org> wrote:
I share Christian's concerns -

Dario/Leila - can you comment based on your recent experiences with WikiGrok?

I agree with Christian.

QA in beta labs is good but not enough. We still need to do QA when a feature goes to production and currently, it's very hard to figure out if there's a problem with logging. An example:

While testing WikiGrok in production, we learned that after some point tests from Firefox browser from my machine were not logged. We did not get any errors for this. I found out about this because I was trying to manually make a trace of activities and see if I can stitch them together and make sense of them. We eventually figured out what was going on in that case [1], but it concerns me that there may be other important events that we don't log in the DB and we never know that we're not logging.

 

Thanks

-Toby


> On Dec 15, 2014, at 9:42 AM, Christian Aistleitner <christian@quelltextlich.at> wrote:
>
> Hi,
>
>> On Mon, Dec 15, 2014 at 08:34:39AM -0800, Kevin Leduc wrote:
>> I closed the Phabricator task with a links to this thread and the wikitech
>> doc for testing on beta cluster.
>
> I am fine with keeping the task closed.
>
> But I am somewhat surprised to see beta mentioned in the
> resolution. Note that Dario's request set scope as [1]
>
>  However, there are types of data quality issues that we only
>  discover when collecting data at scale and in the wild (on
>  browsers/platforms that we don’t necessarily test for internally).
>
> . That's a valid scope, but from my point of view, beta does not match
> that scope.
>
> Neither is beta large scale, nor is it hammered on with crazy devices.
>
> Beta is just a halfing the distance between EventLogging's devserver
> (Vagrant!) and production.
>
> Have fun,
> Christian
>
>
>
> [1] https://lists.wikimedia.org/pipermail/analytics/2014-December/002884.html
>
>
>
> --
> ---- quelltextlich e.U. ---- \\ ---- Christian Aistleitner ----
>                           Companies' registry: 360296y in Linz
> Christian Aistleitner
> Kefermarkterstrasze 6a/3     Email:  christian@quelltextlich.at
> 4293 Gutau, Austria          Phone:          +43 7946 / 20 5 81
>                             Fax:            +43 7946 / 20 5 81
>                             Homepage: http://quelltextlich.at/
> ---------------------------------------------------------------
> _______________________________________________
> Analytics mailing list
> Analytics@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/analytics

_______________________________________________
Analytics mailing list
Analytics@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/analytics

_______________________________________________
Analytics mailing list
Analytics@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/analytics