[Labs-l] Performance of crosswiki query tools

Marc A. Pelletier marc at uberbox.org
Sun Sep 1 17:30:55 UTC 2013


On 09/01/2013 01:02 PM, Jesse (Pathoschild) wrote:
> This is /very/ slow on Tools Labs; does the Tools Labs infrastructure
> make "use `dbname`" significantly slower? Is there a more efficient way
> to do this?

Not that I know, but the command line clients takes 2-3 seconds when it
connects to a database to fetch table names and schemas -- perhaps your
library is doing something comparable behind the scenes?

I've tested with perl DBI, and there doesn't seem to be a cost to 'use'
proper, that is no significant overhead compared to using fully
qualified table names without taking the time to connect to a database
beforehand (i.e.: select * from foo.table; rather than use foo; select *
from table;)

One thing that might slow things down for you is that, at this time,
labs is in a /different/ datacenter from where the replicas are.  While
bandwidth isn't an issue, that does introduce /latency/ into the
equation (about 26ms per rountrip) so that doing 1000 operations over
one row is going to be *significantly* slower than doing one operation
over 1000 rows -- anything that requires a roundtrip with the database
will take 26ms to complete.

This does have performance consequences; it's likely that doing 100
simple queries in parallel would take nearly the same time as taking
just one, and that fetching a 100 row result at once would be nearly 100
times faster than fetching 100 rows one at a time.

At least one dev that had performance issue with lazily fecthing a
result one row at a time found an very significant boots simply by using
fetch_many instead.

All of that said, labs will be moving to our primary datacenter in
Ashburn in the medium term and that issue will definitely become much
less salient then -- but it is still good practice to limit the number
of roundtrips when doing lots of DB activity.

-- Marc




More information about the Labs-l mailing list