Hi everyone,
I'm getting duplicate rows in the cargo tables. The instances I have found have the same data with the exception of the _ID. It happens on main tables, tables created for fields with multiple values, and tables created for multiple instances of a template.
This is creating cartesian products on calls to #cargo_query.
I'm running MySQL with InnoDB tables and the version of cargo I have is 0.10 last updated on 11/22. If there is a way to turn on a trace for the SQL queries or add debug statements in the code I could provide a log.
Thoughts?
Thanks!!
Hi Ed,
Yes, this sometimes happens with Cargo - it seems to only happen under fairly special circumstances, which makes debugging it difficult. For now, if it happens, I would just recreate the relevant table(s), either via the web interface or with the command line script cargoRecreateData.php.
-Yaron
On Tue, Nov 24, 2015 at 12:12 AM, Ed edward.hoo@gmail.com wrote:
Hi everyone,
I'm getting duplicate rows in the cargo tables. The instances I have found have the same data with the exception of the _ID. It happens on main tables, tables created for fields with multiple values, and tables created for multiple instances of a template.
This is creating cartesian products on calls to #cargo_query.
I'm running MySQL with InnoDB tables and the version of cargo I have is 0.10 last updated on 11/22. If there is a way to turn on a trace for the SQL queries or add debug statements in the code I could provide a log.
Thoughts?
Thanks!! _______________________________________________ MediaWiki-l mailing list To unsubscribe, go to: https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
Hi Yaron,
If you give me some pointers I can attempt to debug it as it seems to happen somewhat often in my set up.
I have also found sql expressions where the parser interprets quoted parenthesis as function calls. I could also try to debug that.
My email is edward.hoo in the email service named after a very large number :)
Sent from my iPhone
On Nov 24, 2015, at 9:41 AM, Yaron Koren yaron@wikiworks.com wrote:
Hi Ed,
Yes, this sometimes happens with Cargo - it seems to only happen under fairly special circumstances, which makes debugging it difficult. For now, if it happens, I would just recreate the relevant table(s), either via the web interface or with the command line script cargoRecreateData.php.
-Yaron
On Tue, Nov 24, 2015 at 12:12 AM, Ed edward.hoo@gmail.com wrote:
Hi everyone,
I'm getting duplicate rows in the cargo tables. The instances I have found have the same data with the exception of the _ID. It happens on main tables, tables created for fields with multiple values, and tables created for multiple instances of a template.
This is creating cartesian products on calls to #cargo_query.
I'm running MySQL with InnoDB tables and the version of cargo I have is 0.10 last updated on 11/22. If there is a way to turn on a trace for the SQL queries or add debug statements in the code I could provide a log.
Thoughts?
Thanks!! _______________________________________________ MediaWiki-l mailing list To unsubscribe, go to: https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
-- WikiWorks · MediaWiki Consulting · http://wikiworks.com _______________________________________________ MediaWiki-l mailing list To unsubscribe, go to: https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
Hi,
The duplicate rows seem to happen as a result of a race condition when the new record has been queued and somewhere in the process we reedit or regenerate and queue a second one.
I'm not sure if it is buffering, using multiple connections, something else.
I was going to mess around with the DBO_NOBUFFER setting but the comments in "includes/db/Database.php" are a bit ominous:
* Unbuffered queries are very troublesome in MySQL: * * - If another query is executed while the first query is being read * out, the first query is killed. This means you can't call normal * MediaWiki functions while you are reading an unbuffered query result * from a normal wfGetDB() connection. * * - Unbuffered queries cause the MySQL server to use large amounts of * memory and to hold broad locks which block other queries.
Thoughts?
On Tue, Nov 24, 2015 at 9:41 AM, Yaron Koren yaron@wikiworks.com wrote:
Hi Ed,
Yes, this sometimes happens with Cargo - it seems to only happen under fairly special circumstances, which makes debugging it difficult. For now, if it happens, I would just recreate the relevant table(s), either via the web interface or with the command line script cargoRecreateData.php.
-Yaron
On Tue, Nov 24, 2015 at 12:12 AM, Ed edward.hoo@gmail.com wrote:
Hi everyone,
I'm getting duplicate rows in the cargo tables. The instances I have
found
have the same data with the exception of the _ID. It happens on main tables, tables created for fields with multiple values, and tables
created
for multiple instances of a template.
This is creating cartesian products on calls to #cargo_query.
I'm running MySQL with InnoDB tables and the version of cargo I have is 0.10 last updated on 11/22. If there is a way to turn on a trace for the SQL queries or add debug statements in the code I could provide a log.
Thoughts?
Thanks!! _______________________________________________ MediaWiki-l mailing list To unsubscribe, go to: https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
-- WikiWorks · MediaWiki Consulting · http://wikiworks.com _______________________________________________ MediaWiki-l mailing list To unsubscribe, go to: https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
Yes, it probably is due to a race condition.
Hopefully, there's a better solution than using unbuffered queries - if that is a solution.
-Yaron
On Sun, Dec 13, 2015 at 12:51 AM, Ed edward.hoo@gmail.com wrote:
Hi,
The duplicate rows seem to happen as a result of a race condition when the new record has been queued and somewhere in the process we reedit or regenerate and queue a second one.
I'm not sure if it is buffering, using multiple connections, something else.
I was going to mess around with the DBO_NOBUFFER setting but the comments in "includes/db/Database.php" are a bit ominous:
* Unbuffered queries are very troublesome in MySQL: * * - If another query is executed while the first query is beingread * out, the first query is killed. This means you can't call normal * MediaWiki functions while you are reading an unbuffered query result * from a normal wfGetDB() connection. * * - Unbuffered queries cause the MySQL server to use large amounts of * memory and to hold broad locks which block other queries.
Thoughts?
Hi,
I went over the code and noticed that DB_SLAVE was being used in read-only queries that are part of update processes. I do not know if this is what generated the issues I'm experiencing (I'm not running replication). However, it could generate race conditions in a replicated environment as the slave would not always have the most current information. Therefore I updated the calls to DB_MASTER for the functions that seemed to be called as part of an update/delete/insert sequence. That left only one or two instances using DB_SLAVE. So I updated all calls to DB_MASTER for good measure -- in case I had missed something.
I will run it like this for a while and see if we get more duplicate entries in the cargo tables.
Thoughts?
On Sun, Dec 13, 2015 at 8:39 AM, Yaron Koren yaron@wikiworks.com wrote:
Yes, it probably is due to a race condition.
Hopefully, there's a better solution than using unbuffered queries - if that is a solution.
-Yaron
On Sun, Dec 13, 2015 at 12:51 AM, Ed edward.hoo@gmail.com wrote:
Hi,
The duplicate rows seem to happen as a result of a race condition when
the
new record has been queued and somewhere in the process we reedit or regenerate and queue a second one.
I'm not sure if it is buffering, using multiple connections, something else.
I was going to mess around with the DBO_NOBUFFER setting but the comments in "includes/db/Database.php" are a bit ominous:
* Unbuffered queries are very troublesome in MySQL: * * - If another query is executed while the first query isbeing
read * out, the first query is killed. This means you can't call normal * MediaWiki functions while you are reading an unbuffered query result * from a normal wfGetDB() connection. * * - Unbuffered queries cause the MySQL server to use large amounts of * memory and to hold broad locks which block other queries.
Thoughts?
MediaWiki-l mailing list To unsubscribe, go to: https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
Hi Ed,
I have no idea if that change will fix anything - but if so, I'll be glad to add it in!
-Yaron
On Sun, Dec 13, 2015 at 4:14 PM, Ed edward.hoo@gmail.com wrote:
Hi,
I went over the code and noticed that DB_SLAVE was being used in read-only queries that are part of update processes. I do not know if this is what generated the issues I'm experiencing (I'm not running replication). However, it could generate race conditions in a replicated environment as the slave would not always have the most current information. Therefore I updated the calls to DB_MASTER for the functions that seemed to be called as part of an update/delete/insert sequence. That left only one or two instances using DB_SLAVE. So I updated all calls to DB_MASTER for good measure -- in case I had missed something.
I will run it like this for a while and see if we get more duplicate entries in the cargo tables.
Thoughts?
On Sun, Dec 13, 2015 at 8:39 AM, Yaron Koren yaron@wikiworks.com wrote:
Yes, it probably is due to a race condition.
Hopefully, there's a better solution than using unbuffered queries - if that is a solution.
-Yaron
On Sun, Dec 13, 2015 at 12:51 AM, Ed edward.hoo@gmail.com wrote:
Hi,
The duplicate rows seem to happen as a result of a race condition when
the
new record has been queued and somewhere in the process we reedit or regenerate and queue a second one.
I'm not sure if it is buffering, using multiple connections, something else.
I was going to mess around with the DBO_NOBUFFER setting but the
comments
in "includes/db/Database.php" are a bit ominous:
* Unbuffered queries are very troublesome in MySQL: * * - If another query is executed while the first query isbeing
read * out, the first query is killed. This means you can't
call
normal * MediaWiki functions while you are reading an unbuffered query result * from a normal wfGetDB() connection. * * - Unbuffered queries cause the MySQL server to use large amounts of * memory and to hold broad locks which block other
queries.
Thoughts?
MediaWiki-l mailing list To unsubscribe, go to: https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
MediaWiki-l mailing list To unsubscribe, go to: https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
mediawiki-l@lists.wikimedia.org