Hi,
I've been working on some sanity checks for our labtestn (mitaka/neutron) deployment [0].
I found a weird issue when trying to create an instance in the 'tooling' network (the vxlan based network):
<< Network requires port_security_enabled and subnet associated in order to apply security groups.
This error is seen in the /var/log/nova/nova-conductor.log file when I create an instance with this cmdline:
% openstack server create --flavor 2 \ --image 66e544e8-fe4f-41f7-9809-6723e53b5a99 aborrero-test1 \ --nic port-id=9389a984-d58e-4776-8b7a-30ff93073917 \ --property subnet=3ec06de7-3b9e-4de3-86c6-67ba1895b253
However:
% neutron port-show 9389a984-d58e-4776-8b7a-30ff93073917 \ | grep port_security_enabled
| port_security_enabled | True
(this port was created manually by me)
Not sure if the 'server create' command is lacking some additional option. I generated it following what I saw in our bootstrap docs + labtestcontrol2003 history
I also tried with this command:
% openstack server create --flavor 2 \ --image 66e544e8-fe4f-41f7-9809-6723e53b5a99 aborrero-test1 \ --nic net-id=60aa9467-253c-4fdf-9fa0-eba42dafc975
with the same result (i.e, a net instead of a port)
I'm probably misunderstanding some openstack concepts: nics, ports, subnets, nets, etc.
[0] https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Deployment_sanity...
I this is both expected and a bug :D
There is an option to disable port security for a Network. This is potentially the right thing in a world where Tooling here is really Toolforge and we want ferm/iptables to manage rulesets and not be overlapping with security groups. If this is its own VXLAN network that could make sense.
So to test this I issued:
# optional mass turning off of port-security mode for a network neutron net-update --port-security-enabled=false <net uuid>
This documented in modules/openstack/templates/bootstrap/neutron/neutron_seed.sh.erb
port-security (security group application) can be disabled per-port or per-network. Per-port seems to be fairly well worked out, but per network seems to come with confusion between nova and neutron. Nova assumes in some cases that every port will have a security group and so balks. I see https://review.openstack.org/#/c/59578/ in relation but haven't found a similar bug for disabled for an entire network but I'm sure I've seen one I just can't find it. If only we could read redhat's magic docs at https://access.redhat.com/solutions/2143121. Possibly https://bugzilla.redhat.com/show_bug.cgi?id=1291210
I don't recall doing anything special to create the existing instances so that's interesting, and this seems not to happen if we first make a port with security groups disabled and then attach an instance. So in general I think this is just mismatched expectations between nova and neutron when security groups are disabled for an entire network. There a few known upstream reports of similar things.
On Fri, Jun 1, 2018 at 7:43 AM, Arturo Borrero Gonzalez < aborrero@wikimedia.org> wrote:
Hi,
I've been working on some sanity checks for our labtestn (mitaka/neutron) deployment [0].
I found a weird issue when trying to create an instance in the 'tooling' network (the vxlan based network):
<< Network requires port_security_enabled and subnet associated in order to apply security groups.
This error is seen in the /var/log/nova/nova-conductor.log file when I create an instance with this cmdline:
% openstack server create --flavor 2 \ --image 66e544e8-fe4f-41f7-9809-6723e53b5a99 aborrero-test1 \ --nic port-id=9389a984-d58e-4776-8b7a-30ff93073917 \ --property subnet=3ec06de7-3b9e-4de3-86c6-67ba1895b253
However:
% neutron port-show 9389a984-d58e-4776-8b7a-30ff93073917 \ | grep port_security_enabled
| port_security_enabled | True
(this port was created manually by me)
Not sure if the 'server create' command is lacking some additional option. I generated it following what I saw in our bootstrap docs + labtestcontrol2003 history
I also tried with this command:
% openstack server create --flavor 2 \ --image 66e544e8-fe4f-41f7-9809-6723e53b5a99 aborrero-test1 \ --nic net-id=60aa9467-253c-4fdf-9fa0-eba42dafc975
with the same result (i.e, a net instead of a port)
I'm probably misunderstanding some openstack concepts: nics, ports, subnets, nets, etc.
[0] https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/ Admin/Deployment_sanity_checklist
openstack network show 60aa9467-253c-4fdf-9fa0-eba42dafc975 | grep -e name -e port_security | name | tooling | | port_security_enabled | False
On Fri, Jun 1, 2018 at 9:08 AM, Chase Pettet cpettet@wikimedia.org wrote:
I this is both expected and a bug :D
There is an option to disable port security for a Network. This is potentially the right thing in a world where Tooling here is really Toolforge and we want ferm/iptables to manage rulesets and not be overlapping with security groups. If this is its own VXLAN network that could make sense.
So to test this I issued:
# optional mass turning off of port-security mode for a network neutron net-update --port-security-enabled=false <net uuid>
This documented in modules/openstack/templates/bootstrap/neutron/neutron_ seed.sh.erb
port-security (security group application) can be disabled per-port or per-network. Per-port seems to be fairly well worked out, but per network seems to come with confusion between nova and neutron. Nova assumes in some cases that every port will have a security group and so balks. I see https://review.openstack.org/#/c/59578/ in relation but haven't found a similar bug for disabled for an entire network but I'm sure I've seen one I just can't find it. If only we could read redhat's magic docs at https://access.redhat.com/solutions/2143121. Possibly https://bugzilla. redhat.com/show_bug.cgi?id=1291210
I don't recall doing anything special to create the existing instances so that's interesting, and this seems not to happen if we first make a port with security groups disabled and then attach an instance. So in general I think this is just mismatched expectations between nova and neutron when security groups are disabled for an entire network. There a few known upstream reports of similar things.
On Fri, Jun 1, 2018 at 7:43 AM, Arturo Borrero Gonzalez < aborrero@wikimedia.org> wrote:
Hi,
I've been working on some sanity checks for our labtestn (mitaka/neutron) deployment [0].
I found a weird issue when trying to create an instance in the 'tooling' network (the vxlan based network):
<< Network requires port_security_enabled and subnet associated in order to apply security groups.
This error is seen in the /var/log/nova/nova-conductor.log file when I create an instance with this cmdline:
% openstack server create --flavor 2 \ --image 66e544e8-fe4f-41f7-9809-6723e53b5a99 aborrero-test1 \ --nic port-id=9389a984-d58e-4776-8b7a-30ff93073917 \ --property subnet=3ec06de7-3b9e-4de3-86c6-67ba1895b253
However:
% neutron port-show 9389a984-d58e-4776-8b7a-30ff93073917 \ | grep port_security_enabled
| port_security_enabled | True
(this port was created manually by me)
Not sure if the 'server create' command is lacking some additional option. I generated it following what I saw in our bootstrap docs + labtestcontrol2003 history
I also tried with this command:
% openstack server create --flavor 2 \ --image 66e544e8-fe4f-41f7-9809-6723e53b5a99 aborrero-test1 \ --nic net-id=60aa9467-253c-4fdf-9fa0-eba42dafc975
with the same result (i.e, a net instead of a port)
I'm probably misunderstanding some openstack concepts: nics, ports, subnets, nets, etc.
[0] https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/ Deployment_sanity_checklist
-- Chase Pettet chasemp on phabricator https://phabricator.wikimedia.org/p/chasemp/ and IRC
So I have removed the VXLAN project here as it's speculative at the moment and it has caused several time-sucking rabbit holes. Safe to say either Newton or a lot of detailed walk through is needed to validate its ready for prime time.
this looks like:
root@labtestcontrol2003:~# openstack server list
+--------------------------------------+-----------+--------+--------------------------------------------------+
| ID | Name | Status | Networks |
+--------------------------------------+-----------+--------+--------------------------------------------------+
| 07ecaa54-9541-4865-8d69-c4ed89257a06 | stretch-4 | ACTIVE | flatcloudinstancesb=172.16.128.48 |
| c5e839de-5e02-45c8-bf3e-4f319ab5d2e3 | stretch-3 | ACTIVE | tooling=172.16.130.15 |
| ae5fb971-bb5b-4a4b-a2e0-1a7d82b304d7 | stretch-2 | ACTIVE | tooling=172.16.130.14 |
| a762d405-1bef-403b-8812-b940b2ee75df | stretch-1 | ACTIVE | tooling=172.16.130.13, 172.16.129.15 |
| ffc269f9-320b-4371-8415-792ef4dc16d3 | jessie-2 | ACTIVE | tooling=172.16.130.12 |
| 89338d1c-9ecb-414b-90a9-14d97a3a7f8b | jessie-1 | ACTIVE | flatcloudinstancesb=172.16.128.47 |
| 690d6cf6-86ba-4028-a093-f15eadc36aad | ti5 | ACTIVE | flatcloudinstancesb=172.16.128.46, 172.16.129.14 |
+--------------------------------------+-----------+--------+--------------------------------------------------+
root@labtestcontrol2003:~#
root@labtestcontrol2003:~#
root@labtestcontrol2003:~# openstack server delete c5e839de-5e02-45c8-bf3e-4f319ab5d2e3
root@labtestcontrol2003:~# openstack server delete ae5fb971-bb5b-4a4b-a2e0-1a7d82b304d7
root@labtestcontrol2003:~# openstack server delete a762d405-1bef-403b-8812-b940b2ee75df
root@labtestcontrol2003:~# openstack server delete ffc269f9-320b-4371-8415-792ef4dc16d3
neutron router-interface-delete 5712e22e-134a-40d3-a75a-1c9b441717ad 3ec06de7-3b9e-4de3-86c6-67ba1895b253
<router> <subnet_of_internal_gw_port>
neutron port-list | grep 130
neutron port-delete db573423-e707-4c84-896d-5d015e04eef0
| 60aa9467-253c-4fdf-9fa0-eba42dafc975 | tooling | 3ec06de7-3b9e-4de3-86c6-67ba1895b253 172.16.130.0/24 |
root@labtestcontrol2003:~# neutron net-delete 60aa9467-253c-4fdf-9fa0-eba42dafc975
Deleted network: 60aa9467-253c-4fdf-9fa0-eba42dafc975
labtestneutron2001
brq60aa9467-25 8000.02fbea93bbc3 no vxlan-2
reboot of labtestneutron200[12] to clear out the host level dynamic bridge was needed
On Fri, Jun 1, 2018 at 9:10 AM, Chase Pettet cpettet@wikimedia.org wrote:
openstack network show 60aa9467-253c-4fdf-9fa0-eba42dafc975 | grep -e name -e port_security | name | tooling | | port_security_enabled | False
On Fri, Jun 1, 2018 at 9:08 AM, Chase Pettet cpettet@wikimedia.org wrote:
I this is both expected and a bug :D
There is an option to disable port security for a Network. This is potentially the right thing in a world where Tooling here is really Toolforge and we want ferm/iptables to manage rulesets and not be overlapping with security groups. If this is its own VXLAN network that could make sense.
So to test this I issued:
# optional mass turning off of port-security mode for a network neutron net-update --port-security-enabled=false <net uuid>
This documented in modules/openstack/templates/bo otstrap/neutron/neutron_seed.sh.erb
port-security (security group application) can be disabled per-port or per-network. Per-port seems to be fairly well worked out, but per network seems to come with confusion between nova and neutron. Nova assumes in some cases that every port will have a security group and so balks. I see https://review.openstack.org/#/c/59578/ in relation but haven't found a similar bug for disabled for an entire network but I'm sure I've seen one I just can't find it. If only we could read redhat's magic docs at https://access.redhat.com/solutions/2143121. Possibly https://bugzilla.redhat.com/show_bug.cgi?id=1291210
I don't recall doing anything special to create the existing instances so that's interesting, and this seems not to happen if we first make a port with security groups disabled and then attach an instance. So in general I think this is just mismatched expectations between nova and neutron when security groups are disabled for an entire network. There a few known upstream reports of similar things.
On Fri, Jun 1, 2018 at 7:43 AM, Arturo Borrero Gonzalez < aborrero@wikimedia.org> wrote:
Hi,
I've been working on some sanity checks for our labtestn (mitaka/neutron) deployment [0].
I found a weird issue when trying to create an instance in the 'tooling' network (the vxlan based network):
<< Network requires port_security_enabled and subnet associated in order to apply security groups.
This error is seen in the /var/log/nova/nova-conductor.log file when I create an instance with this cmdline:
% openstack server create --flavor 2 \ --image 66e544e8-fe4f-41f7-9809-6723e53b5a99 aborrero-test1 \ --nic port-id=9389a984-d58e-4776-8b7a-30ff93073917 \ --property subnet=3ec06de7-3b9e-4de3-86c6-67ba1895b253
However:
% neutron port-show 9389a984-d58e-4776-8b7a-30ff93073917 \ | grep port_security_enabled
| port_security_enabled | True
(this port was created manually by me)
Not sure if the 'server create' command is lacking some additional option. I generated it following what I saw in our bootstrap docs + labtestcontrol2003 history
I also tried with this command:
% openstack server create --flavor 2 \ --image 66e544e8-fe4f-41f7-9809-6723e53b5a99 aborrero-test1 \ --nic net-id=60aa9467-253c-4fdf-9fa0-eba42dafc975
with the same result (i.e, a net instead of a port)
I'm probably misunderstanding some openstack concepts: nics, ports, subnets, nets, etc.
[0] https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/D eployment_sanity_checklist
-- Chase Pettet chasemp on phabricator https://phabricator.wikimedia.org/p/chasemp/ and IRC
-- Chase Pettet chasemp on phabricator https://phabricator.wikimedia.org/p/chasemp/ and IRC
cloud-admin@lists.wikimedia.org