From where would you propose measuring these data
points? Obviously
network latency will have a great impact on some of the metrics
and a
consistent location would help to define the pass/fail of each test. I do
think another benchmark Ops "features" would be a set of
latency-to-datacenter values, but I know that is a much harder taks. Thanks
for putting this together.
On Thu, Mar 21, 2013 at 6:40 PM, Asher Feldman <afeldman(a)wikimedia.org>wrote;wrote:
I'd like to push for a codified set of minimum
performance standards that
new mediawiki features must meet before they can be deployed to larger
wikimedia sites such as English Wikipedia, or be considered complete.
These would look like (numbers pulled out of a hat, not actual
suggestions):
- p999 (long tail) full page request latency of 2000ms
- p99 page request latency of 800ms
- p90 page request latency of 150ms
- p99 banner request latency of 80ms
- p90 banner request latency of 40ms
- p99 db query latency of 250ms
- p90 db query latency of 50ms
- 1000 write requests/sec (if applicable; writes operations must be free
from concurrency issues)
- guidelines about degrading gracefully
- specific limits on total resource consumption across the stack per
request
- etc..
Right now, varying amounts of effort are made to highlight potential
performance bottlenecks in code review, and engineers are encouraged to
profile and optimize their own code. But beyond "is the site still up for
everyone / are users complaining on the village pump / am I ranting in
irc", we've offered no guidelines as to what sort of request latency is
reasonable or acceptable. If a new feature (like aftv5, or flow) turns out
not to meet perf standards after deployment, that would be a high priority
bug and the feature may be disabled depending on the impact, or if not
addressed in a reasonable time frame. Obviously standards like this can't
be applied to certain existing parts of mediawiki, but systems other than
the parser or preprocessor that don't meet new standards should at least be
prioritized for improvement.
Thoughts?
Asher
_______________________________________________
Wikitech-l mailing list
Wikitech-l(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l