I'm trying to use mwdumper to insert the English Wikipedia enwiki database into MySQL (enwiki-20070908-pages-articles.xml), but the SSH connection seems to timeout/disconnect after about 890K rows (out of about 10 million I believe) have been uploaded. How can I keep SSH from disconnecting?
Is there some mwdumper command line option I can use, or some client or server side SSH setting? (server is Linux with OpenSSH server)
Thanks, Saqib
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Saqib Kadri wrote:
I'm trying to use mwdumper to insert the English Wikipedia enwiki database into MySQL (enwiki-20070908-pages-articles.xml), but the SSH connection seems to timeout/disconnect after about 890K rows (out of about 10 million I believe) have been uploaded. How can I keep SSH from disconnecting?
Is there some mwdumper command line option I can use, or some client or server side SSH setting? (server is Linux with OpenSSH server)
Usually SSH shouldn't drop your connection if data is being actively sent over it, and mwdumper should be happily feeding data over the pipe as fast as the mysql on the other end will take it.
You might try fiddling with some SSH config settings such as ServerAliveInterval and TCPKeepAlive if there is a connection problem.
The problem might be on the MySQL end, though; double-check that you're not getting disconnected from the MySQL socket due to timeouts or oversized packets. Check the MySQL settings net_read_timeout, net_write_timeout, and max_packet_size.
- -- brion vibber (brion @ wikimedia.org)
On 9/19/07, Saqib Kadri email@saqibkadri.net wrote:
Is there some mwdumper command line option I can use, or some client or server side SSH setting? (server is Linux with OpenSSH server)
and the client (that is the program you are using)?
If you look at the manual page ssh_config (5), you will find some parameter in the client configuration that my play a role on this behaviour (I am not an expert so I am not giving direct response here): ServerAliveCountMax ServerAliveInterval TCPKeepAlive
In the manual page sshd_config you can find some parameter about the server side (for instance ClientAliveCountMax ClientAliveInterval TCPKeepAlive), but you need root access on the server to change this parameters.
As suggested in a previous reply screen may solve your problem, but it has a rather steep learning-curve (but you just need a few command to use it for your purpose)
AnyFile
Thanks, Saqib
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org http://lists.wikimedia.org/mailman/listinfo/wikitech-l
What about using nohup to launch the command? This backgrounds the process in the proper way...
Luca
On 9/19/07, Any File anysomefile@gmail.com wrote:
On 9/19/07, Saqib Kadri email@saqibkadri.net wrote:
Is there some mwdumper command line option I can use, or some client or server side SSH setting? (server is Linux with OpenSSH server)
and the client (that is the program you are using)?
If you look at the manual page ssh_config (5), you will find some parameter in the client configuration that my play a role on this behaviour (I am not an expert so I am not giving direct response here): ServerAliveCountMax ServerAliveInterval TCPKeepAlive
In the manual page sshd_config you can find some parameter about the server side (for instance ClientAliveCountMax ClientAliveInterval TCPKeepAlive), but you need root access on the server to change this parameters.
As suggested in a previous reply screen may solve your problem, but it has a rather steep learning-curve (but you just need a few command to use it for your purpose)
AnyFile
Thanks, Saqib
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org http://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org http://lists.wikimedia.org/mailman/listinfo/wikitech-l
wikitech-l@lists.wikimedia.org