So I have removed the VXLAN project here as it's speculative at the moment and it has caused several time-sucking rabbit holes.  Safe to say either Newton or a lot of detailed walk through is needed to validate its ready for prime time.

this looks like:

root@labtestcontrol2003:~# openstack server list

+--------------------------------------+-----------+--------+--------------------------------------------------+

| ID                                   | Name      | Status | Networks                                         |

+--------------------------------------+-----------+--------+--------------------------------------------------+

| 07ecaa54-9541-4865-8d69-c4ed89257a06 | stretch-4 | ACTIVE | flatcloudinstancesb=172.16.128.48                |

| c5e839de-5e02-45c8-bf3e-4f319ab5d2e3 | stretch-3 | ACTIVE | tooling=172.16.130.15                            |

| ae5fb971-bb5b-4a4b-a2e0-1a7d82b304d7 | stretch-2 | ACTIVE | tooling=172.16.130.14                            |

| a762d405-1bef-403b-8812-b940b2ee75df | stretch-1 | ACTIVE | tooling=172.16.130.13, 172.16.129.15             |

| ffc269f9-320b-4371-8415-792ef4dc16d3 | jessie-2  | ACTIVE | tooling=172.16.130.12                            |

| 89338d1c-9ecb-414b-90a9-14d97a3a7f8b | jessie-1  | ACTIVE | flatcloudinstancesb=172.16.128.47                |

| 690d6cf6-86ba-4028-a093-f15eadc36aad | ti5       | ACTIVE | flatcloudinstancesb=172.16.128.46, 172.16.129.14 |

+--------------------------------------+-----------+--------+--------------------------------------------------+

root@labtestcontrol2003:~#

root@labtestcontrol2003:~#

root@labtestcontrol2003:~# openstack server delete c5e839de-5e02-45c8-bf3e-4f319ab5d2e3

root@labtestcontrol2003:~# openstack server delete ae5fb971-bb5b-4a4b-a2e0-1a7d82b304d7

root@labtestcontrol2003:~# openstack server delete a762d405-1bef-403b-8812-b940b2ee75df

root@labtestcontrol2003:~# openstack server delete ffc269f9-320b-4371-8415-792ef4dc16d3


neutron router-interface-delete 5712e22e-134a-40d3-a75a-1c9b441717ad 3ec06de7-3b9e-4de3-86c6-67ba1895b253


<router> <subnet_of_internal_gw_port>


neutron port-list | grep 130

neutron port-delete db573423-e707-4c84-896d-5d015e04eef0



| 60aa9467-253c-4fdf-9fa0-eba42dafc975 | tooling                                            | 3ec06de7-3b9e-4de3-86c6-67ba1895b253 172.16.130.0/24  |


root@labtestcontrol2003:~# neutron net-delete 60aa9467-253c-4fdf-9fa0-eba42dafc975

Deleted network: 60aa9467-253c-4fdf-9fa0-eba42dafc975


labtestneutron2001

> brq60aa9467-25 8000.02fbea93bbc3 no vxlan-2


reboot of labtestneutron200[12] to clear out the host level dynamic bridge was needed



On Fri, Jun 1, 2018 at 9:10 AM, Chase Pettet <cpettet@wikimedia.org> wrote:
openstack network show 60aa9467-253c-4fdf-9fa0-eba42dafc975 | grep -e name -e port_security
| name                      | tooling                              |
| port_security_enabled     | False

On Fri, Jun 1, 2018 at 9:08 AM, Chase Pettet <cpettet@wikimedia.org> wrote:
I this is both expected and a bug :D

There is an option to disable port security for a Network.  This is potentially the right thing in a world where Tooling here is really Toolforge and we want ferm/iptables to manage rulesets and not be overlapping with security groups.  If this is its own VXLAN network that could make sense.

So to test this I issued:

# optional mass turning off of port-security mode for a network
neutron net-update --port-security-enabled=false <net uuid>

This documented in modules/openstack/templates/bootstrap/neutron/neutron_seed.sh.erb

port-security (security group application) can be disabled per-port or per-network.  Per-port seems to be fairly well worked out, but per network seems to come with confusion between nova and neutron.  Nova assumes in some cases that every port will have a security group and so balks.  I see https://review.openstack.org/#/c/59578/ in relation but haven't found a similar bug for disabled for an entire network but I'm sure I've seen one I just can't find it.  If only we could read redhat's magic docs at https://access.redhat.com/solutions/2143121.  Possibly https://bugzilla.redhat.com/show_bug.cgi?id=1291210

I don't recall doing anything special to create the existing instances so that's interesting, and this seems not to happen if we first make a port with security groups disabled and then attach an instance.  So in general I think this is just mismatched expectations between nova and neutron when security groups are disabled for an entire network.  There a few known upstream reports of similar things.  

On Fri, Jun 1, 2018 at 7:43 AM, Arturo Borrero Gonzalez <aborrero@wikimedia.org> wrote:
Hi,

I've been working on some sanity checks for our labtestn
(mitaka/neutron) deployment [0].

I found a weird issue when trying to create an instance in the 'tooling'
network (the vxlan based network):

<<
Network requires port_security_enabled and subnet associated in order to
apply security groups.
>>

This error is seen in the /var/log/nova/nova-conductor.log file when I
create an instance with this cmdline:

% openstack server create --flavor 2 \
--image 66e544e8-fe4f-41f7-9809-6723e53b5a99 aborrero-test1 \
--nic port-id=9389a984-d58e-4776-8b7a-30ff93073917 \
--property subnet=3ec06de7-3b9e-4de3-86c6-67ba1895b253

However:

% neutron port-show 9389a984-d58e-4776-8b7a-30ff93073917 \
| grep port_security_enabled

| port_security_enabled | True

(this port was created manually by me)

Not sure if the 'server create' command is lacking some additional
option. I generated it following what I saw in our bootstrap docs +
labtestcontrol2003 history

I also tried with this command:

% openstack server create --flavor 2 \
--image 66e544e8-fe4f-41f7-9809-6723e53b5a99 aborrero-test1 \
--nic net-id=60aa9467-253c-4fdf-9fa0-eba42dafc975

with the same result (i.e, a net instead of a port)

I'm probably misunderstanding some openstack concepts: nics, ports,
subnets, nets, etc.

[0]
https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Deployment_sanity_checklist



--
Chase Pettet
chasemp on phabricator and IRC



--
Chase Pettet
chasemp on phabricator and IRC



--
Chase Pettet
chasemp on phabricator and IRC