Hello,
I'm trying to tunnel to my VPS Trove instance, so that I can test some code against the production database.
I'm using the following:
ssh -N -L 4177:tdlqt33y3nt.svc.trove.eqiad1.wikimedia.cloud:3306 login.toolforge.org
I've also tried:
ssh -N -L 4177:tdlqt33y3nt.svc.trove.eqiad1.wikimedia.cloud:3306 mwcurator.mwoffliner.eqiad1.wikimedia.cloud
The initial connection seems to work, but in my application I'm getting dropped connections. I see the following in the terminal where I set up the tunnel:
channel 2: open failed: connect failed: Connection refused channel 2: open failed: connect failed: Connection refused
I imagine this isn't a fully supported workflow but I'm wondering if there's some way to get it to work?
Thanks, -Travis
I noticed you say it's a VPS as in Cloud VPS project but you attempt to tunnel via login.toolforge.org.
Maybe try tunneling via bastion.wmcloud.org or primary.bastion.wmcloud.org
as the bastion hosts for the cloud VPS projects.
https://wikitech.wikimedia.org/w/index.php?go=Go&search=bastion.wmcloud....
On Fri, Jun 21, 2024 at 9:36 AM Travis Briggs audiodude@gmail.com wrote:
Hello,
I'm trying to tunnel to my VPS Trove instance, so that I can test some code against the production database.
I'm using the following:
ssh -N -L 4177:tdlqt33y3nt.svc.trove.eqiad1.wikimedia.cloud:3306 login.toolforge.org
I've also tried:
ssh -N -L 4177:tdlqt33y3nt.svc.trove.eqiad1.wikimedia.cloud:3306 mwcurator.mwoffliner.eqiad1.wikimedia.cloud
The initial connection seems to work, but in my application I'm getting dropped connections. I see the following in the terminal where I set up the tunnel:
channel 2: open failed: connect failed: Connection refused channel 2: open failed: connect failed: Connection refused
I imagine this isn't a fully supported workflow but I'm wondering if there's some way to get it to work?
Thanks, -Travis _______________________________________________ Cloud mailing list -- cloud@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/
When I try tunnelling through bastion.wmcloud.org or primary.bastion.wmcloud.org I just get:
pymysql.err.OperationalError: (2003, "Can't connect to MySQL server on 'localhost' ([Errno 61] Connection refused)")
I can resolve and connect to the trove instance from toolforge, and as I mentioned in my original email, I tried connecting through mwcurator.mwoffliner.eqiad1.wikimedia.cloud which is our VPS box.
Thanks, -Travis
On Fri, Jun 21, 2024 at 10:21 AM Daniel Zahn dzahn@wikimedia.org wrote:
I noticed you say it's a VPS as in Cloud VPS project but you attempt to tunnel via login.toolforge.org.
Maybe try tunneling via bastion.wmcloud.org or primary.bastion.wmcloud.org
as the bastion hosts for the cloud VPS projects.
https://wikitech.wikimedia.org/w/index.php?go=Go&search=bastion.wmcloud....
On Fri, Jun 21, 2024 at 9:36 AM Travis Briggs audiodude@gmail.com wrote:
Hello,
I'm trying to tunnel to my VPS Trove instance, so that I can test some code against the production database.
I'm using the following:
ssh -N -L 4177:tdlqt33y3nt.svc.trove.eqiad1.wikimedia.cloud:3306 login.toolforge.org
I've also tried:
ssh -N -L 4177:tdlqt33y3nt.svc.trove.eqiad1.wikimedia.cloud:3306 mwcurator.mwoffliner.eqiad1.wikimedia.cloud
The initial connection seems to work, but in my application I'm getting dropped connections. I see the following in the terminal where I set up the tunnel:
channel 2: open failed: connect failed: Connection refused channel 2: open failed: connect failed: Connection refused
I imagine this isn't a fully supported workflow but I'm wondering if there's some way to get it to work?
Thanks, -Travis _______________________________________________ Cloud mailing list -- cloud@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/
-- Daniel Zahn dzahn@wikimedia.org Site Reliability Engineer _______________________________________________ Cloud mailing list -- cloud@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/
I also have just now tried the command line mysql client through the tunnel and things seem to work, so maybe it's some Python setting that's causing the connection to drop?
-Travis
On Fri, Jun 21, 2024 at 12:06 PM Travis Briggs audiodude@gmail.com wrote:
When I try tunnelling through bastion.wmcloud.org or primary.bastion.wmcloud.org I just get:
pymysql.err.OperationalError: (2003, "Can't connect to MySQL server on 'localhost' ([Errno 61] Connection refused)")
I can resolve and connect to the trove instance from toolforge, and as I mentioned in my original email, I tried connecting through mwcurator.mwoffliner.eqiad1.wikimedia.cloud which is our VPS box.
Thanks, -Travis
On Fri, Jun 21, 2024 at 10:21 AM Daniel Zahn dzahn@wikimedia.org wrote:
I noticed you say it's a VPS as in Cloud VPS project but you attempt to tunnel via login.toolforge.org.
Maybe try tunneling via bastion.wmcloud.org or primary.bastion.wmcloud.org
as the bastion hosts for the cloud VPS projects.
https://wikitech.wikimedia.org/w/index.php?go=Go&search=bastion.wmcloud....
On Fri, Jun 21, 2024 at 9:36 AM Travis Briggs audiodude@gmail.com wrote:
Hello,
I'm trying to tunnel to my VPS Trove instance, so that I can test some code against the production database.
I'm using the following:
ssh -N -L 4177:tdlqt33y3nt.svc.trove.eqiad1.wikimedia.cloud:3306 login.toolforge.org
I've also tried:
ssh -N -L 4177:tdlqt33y3nt.svc.trove.eqiad1.wikimedia.cloud:3306 mwcurator.mwoffliner.eqiad1.wikimedia.cloud
The initial connection seems to work, but in my application I'm getting dropped connections. I see the following in the terminal where I set up the tunnel:
channel 2: open failed: connect failed: Connection refused channel 2: open failed: connect failed: Connection refused
I imagine this isn't a fully supported workflow but I'm wondering if there's some way to get it to work?
Thanks, -Travis _______________________________________________ Cloud mailing list -- cloud@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/
-- Daniel Zahn dzahn@wikimedia.org Site Reliability Engineer _______________________________________________ Cloud mailing list -- cloud@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/
If you have a small snippet of code doing the connection that has the issue that can be looked at it would help a lot debugging.
Something that comes to mind might be that it's trying to use a unix socket (as it's connecting to `localhost`), but that might depend on your configuration and the options passed to the connection.
Connectivity-wise, as you checked with the mysql cli there should be no difference :/
On 06/21 12:08, Travis Briggs wrote:
I also have just now tried the command line mysql client through the tunnel and things seem to work, so maybe it's some Python setting that's causing the connection to drop?
-Travis
On Fri, Jun 21, 2024 at 12:06 PM Travis Briggs audiodude@gmail.com wrote:
When I try tunnelling through bastion.wmcloud.org or primary.bastion.wmcloud.org I just get:
pymysql.err.OperationalError: (2003, "Can't connect to MySQL server on 'localhost' ([Errno 61] Connection refused)")
I can resolve and connect to the trove instance from toolforge, and as I mentioned in my original email, I tried connecting through mwcurator.mwoffliner.eqiad1.wikimedia.cloud which is our VPS box.
Thanks, -Travis
On Fri, Jun 21, 2024 at 10:21 AM Daniel Zahn dzahn@wikimedia.org wrote:
I noticed you say it's a VPS as in Cloud VPS project but you attempt to tunnel via login.toolforge.org.
Maybe try tunneling via bastion.wmcloud.org or primary.bastion.wmcloud.org
as the bastion hosts for the cloud VPS projects.
https://wikitech.wikimedia.org/w/index.php?go=Go&search=bastion.wmcloud....
On Fri, Jun 21, 2024 at 9:36 AM Travis Briggs audiodude@gmail.com wrote:
Hello,
I'm trying to tunnel to my VPS Trove instance, so that I can test some code against the production database.
I'm using the following:
ssh -N -L 4177:tdlqt33y3nt.svc.trove.eqiad1.wikimedia.cloud:3306 login.toolforge.org
I've also tried:
ssh -N -L 4177:tdlqt33y3nt.svc.trove.eqiad1.wikimedia.cloud:3306 mwcurator.mwoffliner.eqiad1.wikimedia.cloud
The initial connection seems to work, but in my application I'm getting dropped connections. I see the following in the terminal where I set up the tunnel:
channel 2: open failed: connect failed: Connection refused channel 2: open failed: connect failed: Connection refused
I imagine this isn't a fully supported workflow but I'm wondering if there's some way to get it to work?
Thanks, -Travis _______________________________________________ Cloud mailing list -- cloud@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/
-- Daniel Zahn dzahn@wikimedia.org Site Reliability Engineer _______________________________________________ Cloud mailing list -- cloud@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/
Cloud mailing list -- cloud@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/
This could be the same issue as T367464 [0], due to Trove databases having a default wait_timeout=120, which means that idle connections get terminated after 2 minutes. Interactive connections (the ones created when using the "mysql" CLI) have a different `interactive_timeout` so are not affected.
You can customize the value of wait_timeout for your Trove instance by creating a "configuration group". [1]
HTH, Francesco
[0] https://phabricator.wikimedia.org/T367464 [1] https://wikitech.wikimedia.org/wiki/Help:Trove_database_user_guide#Modify_da...
On Mon, Jun 24, 2024 at 10:47 AM David Caro dcaro@wikimedia.org wrote:
If you have a small snippet of code doing the connection that has the issue that can be looked at it would help a lot debugging.
Something that comes to mind might be that it's trying to use a unix socket (as it's connecting to `localhost`), but that might depend on your configuration and the options passed to the connection.
Connectivity-wise, as you checked with the mysql cli there should be no difference :/
On 06/21 12:08, Travis Briggs wrote:
I also have just now tried the command line mysql client through the tunnel and things seem to work, so maybe it's some Python setting that's causing the connection to drop?
-Travis
On Fri, Jun 21, 2024 at 12:06 PM Travis Briggs audiodude@gmail.com wrote:
When I try tunnelling through bastion.wmcloud.org or primary.bastion.wmcloud.org I just get:
pymysql.err.OperationalError: (2003, "Can't connect to MySQL server on 'localhost' ([Errno 61] Connection refused)")
I can resolve and connect to the trove instance from toolforge, and as I mentioned in my original email, I tried connecting through mwcurator.mwoffliner.eqiad1.wikimedia.cloud which is our VPS box.
Thanks, -Travis
On Fri, Jun 21, 2024 at 10:21 AM Daniel Zahn dzahn@wikimedia.org wrote:
I noticed you say it's a VPS as in Cloud VPS project but you attempt to tunnel via login.toolforge.org.
Maybe try tunneling via bastion.wmcloud.org or primary.bastion.wmcloud.org
as the bastion hosts for the cloud VPS projects.
https://wikitech.wikimedia.org/w/index.php?go=Go&search=bastion.wmcloud....
On Fri, Jun 21, 2024 at 9:36 AM Travis Briggs audiodude@gmail.com wrote:
Hello,
I'm trying to tunnel to my VPS Trove instance, so that I can test some code against the production database.
I'm using the following:
ssh -N -L 4177:tdlqt33y3nt.svc.trove.eqiad1.wikimedia.cloud:3306 login.toolforge.org
I've also tried:
ssh -N -L 4177:tdlqt33y3nt.svc.trove.eqiad1.wikimedia.cloud:3306 mwcurator.mwoffliner.eqiad1.wikimedia.cloud
The initial connection seems to work, but in my application I'm getting dropped connections. I see the following in the terminal where I set up the tunnel:
channel 2: open failed: connect failed: Connection refused channel 2: open failed: connect failed: Connection refused
I imagine this isn't a fully supported workflow but I'm wondering if there's some way to get it to work?
Thanks, -Travis _______________________________________________ Cloud mailing list -- cloud@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/
-- Daniel Zahn dzahn@wikimedia.org Site Reliability Engineer _______________________________________________ Cloud mailing list -- cloud@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/
Cloud mailing list -- cloud@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/
-- David Caro SRE - Cloud Services Wikimedia Foundation https://wikimediafoundation.org/ PGP Signature: 7180 83A2 AC8B 314F B4CE 1171 4071 C7E1 D262 69C3
"Imagine a world in which every single human being can freely share in the sum of all knowledge. That's our commitment." _______________________________________________ Cloud mailing list -- cloud@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/
Hi Francesco,
That actually sounds very plausible! I think I'm creating the tunnel, then not starting the script for a few minutes. Or there could be several minutes of idle time as the script is running.
I've done configuration groups before, so I'll try that and report back.
Thanks! -Travis
On Mon, Jun 24, 2024 at 6:11 AM Francesco Negri fnegri@wikimedia.org wrote:
This could be the same issue as T367464 [0], due to Trove databases having a default wait_timeout=120, which means that idle connections get terminated after 2 minutes. Interactive connections (the ones created when using the "mysql" CLI) have a different `interactive_timeout` so are not affected.
You can customize the value of wait_timeout for your Trove instance by creating a "configuration group". [1]
HTH, Francesco
[0] https://phabricator.wikimedia.org/T367464 [1] https://wikitech.wikimedia.org/wiki/Help:Trove_database_user_guide#Modify_da...
On Mon, Jun 24, 2024 at 10:47 AM David Caro dcaro@wikimedia.org wrote:
If you have a small snippet of code doing the connection that has the
issue
that can be looked at it would help a lot debugging.
Something that comes to mind might be that it's trying to use a unix
socket (as
it's connecting to `localhost`), but that might depend on your
configuration
and the options passed to the connection.
Connectivity-wise, as you checked with the mysql cli there should be no difference :/
On 06/21 12:08, Travis Briggs wrote:
I also have just now tried the command line mysql client through the
tunnel
and things seem to work, so maybe it's some Python setting that's
causing
the connection to drop?
-Travis
On Fri, Jun 21, 2024 at 12:06 PM Travis Briggs audiodude@gmail.com
wrote:
When I try tunnelling through bastion.wmcloud.org or primary.bastion.wmcloud.org I just get:
pymysql.err.OperationalError: (2003, "Can't connect to MySQL server
on
'localhost' ([Errno 61] Connection refused)")
I can resolve and connect to the trove instance from toolforge, and
as I
mentioned in my original email, I tried connecting through mwcurator.mwoffliner.eqiad1.wikimedia.cloud which is our VPS box.
Thanks, -Travis
On Fri, Jun 21, 2024 at 10:21 AM Daniel Zahn dzahn@wikimedia.org
wrote:
I noticed you say it's a VPS as in Cloud VPS project but you
attempt to
tunnel via login.toolforge.org.
Maybe try tunneling via bastion.wmcloud.org or primary.bastion.wmcloud.org
as the bastion hosts for the cloud VPS projects.
https://wikitech.wikimedia.org/w/index.php?go=Go&search=bastion.wmcloud....
On Fri, Jun 21, 2024 at 9:36 AM Travis Briggs audiodude@gmail.com wrote:
Hello,
I'm trying to tunnel to my VPS Trove instance, so that I can test
some
code against the production database.
I'm using the following:
ssh -N -L 4177:tdlqt33y3nt.svc.trove.eqiad1.wikimedia.cloud:3306 login.toolforge.org
I've also tried:
ssh -N -L 4177:tdlqt33y3nt.svc.trove.eqiad1.wikimedia.cloud:3306 mwcurator.mwoffliner.eqiad1.wikimedia.cloud
The initial connection seems to work, but in my application I'm
getting
dropped connections. I see the following in the terminal where I
set up the
tunnel:
channel 2: open failed: connect failed: Connection refused channel 2: open failed: connect failed: Connection refused
I imagine this isn't a fully supported workflow but I'm wondering
if
there's some way to get it to work?
Thanks, -Travis _______________________________________________ Cloud mailing list -- cloud@lists.wikimedia.org List information:
https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/
-- Daniel Zahn dzahn@wikimedia.org Site Reliability Engineer _______________________________________________ Cloud mailing list -- cloud@lists.wikimedia.org List information:
https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/
Cloud mailing list -- cloud@lists.wikimedia.org List information:
https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/
-- David Caro SRE - Cloud Services Wikimedia Foundation https://wikimediafoundation.org/ PGP Signature: 7180 83A2 AC8B 314F B4CE 1171 4071 C7E1 D262 69C3
"Imagine a world in which every single human being can freely share in
the
sum of all knowledge. That's our commitment." _______________________________________________ Cloud mailing list -- cloud@lists.wikimedia.org List information:
https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/
-- Francesco Negri (he/him) -- IRC: dhinus Site Reliability Engineer, Cloud Services team Wikimedia Foundation _______________________________________________ Cloud mailing list -- cloud@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/
There's no fundamental reason why this can't work. I've done similar things:
ssh -t dev.toolforge.org -L 3306:tools.db.svc.wikimedia.cloud:3306"
Sometimes it takes a few tries to find the right ssh incantation, but port tunneling is definitely something that works. One thing you might try is running ssh in verbose mode (ssh -v, or -vv, or -vvv to get increasingly more detailed logging). A lot of it is low-level details which may not mean anything, but it's worth a shot to see if anything jumps out at you.
Another good debugging tool is netcat (i.e. the "nc" command). You can set to listen for connections on your VPS server (on some arbitrary port) and then try to connect to it. That should at least tell you if you're getting as far as executing a TCP handshake through your tunnel.
On Jun 21, 2024, at 12:35 PM, Travis Briggs audiodude@gmail.com wrote:
I imagine this isn't a fully supported workflow but I'm wondering if there's some way to get it to work?
Yes I've definitely tunnelled successfully to toolforge DBs, and the wiki replica databases, before. I've never had a problem with it, but I suspect the issue is with Trove. Like i said, I can establish the connection and even query my Trove production db from the mysql command line. But it times out/fails in my Python program (which never had a problem before).
Thanks, -Travis
On Fri, Jun 21, 2024 at 1:55 PM Roy Smith roy@panix.com wrote:
There's no fundamental reason why this can't work. I've done similar things:
ssh -t dev.toolforge.org -L 3306:tools.db.svc.wikimedia.cloud:3306"
Sometimes it takes a few tries to find the right ssh incantation, but port tunneling is definitely something that works. One thing you might try is running ssh in verbose mode (ssh -v, or -vv, or -vvv to get increasingly more detailed logging). A lot of it is low-level details which may not mean anything, but it's worth a shot to see if anything jumps out at you.
Another good debugging tool is netcat (i.e. the "nc" command). You can set to listen for connections on your VPS server (on some arbitrary port) and then try to connect to it. That should at least tell you if you're getting as far as executing a TCP handshake through your tunnel.
On Jun 21, 2024, at 12:35 PM, Travis Briggs audiodude@gmail.com wrote:
I imagine this isn't a fully supported workflow but I'm wondering if there's some way to get it to work?
Cloud mailing list -- cloud@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/