| *** hamalq_ has quit IRC | 00:05 | |
| *** artom has quit IRC | 00:11 | |
| openstackgerrit | Merged openstack/nova master: libvirt: Add vTPM config support https://review.opendev.org/739209 | 00:17 |
|---|---|---|
| openstackgerrit | Merged openstack/nova master: zuul: remove legacy-tempest-dsvm-neutron-dvr-multinode-full https://review.opendev.org/740447 | 00:31 |
| *** xek_ has quit IRC | 00:32 | |
| *** gyee has quit IRC | 00:33 | |
| *** tetsuro has joined #openstack-nova | 00:33 | |
| *** rcernin has quit IRC | 00:34 | |
| *** rcernin has joined #openstack-nova | 00:39 | |
| *** zhanglong has joined #openstack-nova | 01:01 | |
| *** Liang__ has joined #openstack-nova | 01:10 | |
| *** huaqiang has quit IRC | 01:14 | |
| sean-k-mooney | mnaser: that general indeicates an issue with neutron or rabbitmq performance | 01:29 |
| mnaser | sean-k-mooney: yeah, so based on my log tracing, the PUT is sent at :33 but n-ovs-agent doesnt actually plug until :37 | 01:29 |
| sean-k-mooney | the PUT? | 01:30 |
| mnaser | sorry, the PUT from n-cpu to neutron to plug the port | 01:30 |
| sean-k-mooney | that is not pluging | 01:30 |
| sean-k-mooney | that is port binding | 01:31 |
| sean-k-mooney | they are two differnt things | 01:31 |
| sean-k-mooney | binding happens before pluging starts | 01:31 |
| mnaser | sean-k-mooney: what triggers the plug? | 01:31 |
| sean-k-mooney | after teh ports are bound nova plugs the interface which in the case of ovs is the act of createing the ovs port | 01:32 |
| *** yaawang has quit IRC | 01:33 | |
| sean-k-mooney | when the port is created the external_id in other_cofnig on the ovs interface is set to the neutron port uuid | 01:33 |
| sean-k-mooney | that triggeres an event on a unix socket that the l2 agent is monitoring | 01:33 |
| sean-k-mooney | and then the l2 agent start wiring up the port | 01:33 |
| mnaser | the PUT is the only operation i'm seeing towards neutron-server which isn't read-only (aka not a GET) | 01:33 |
| *** yaawang has joined #openstack-nova | 01:33 | |
| sean-k-mooney | when the port is wired up and the dhcps agnet has also updated the dhcp config the pluging evnet is sent | 01:34 |
| sean-k-mooney | mnaser: nova does not talk to neutron during plugin | 01:34 |
| sean-k-mooney | it configure the network backend and neutron watches for that | 01:34 |
| sean-k-mooney | when the l2 agent finsihs it send a rpc to the neutron server | 01:34 |
| sean-k-mooney | and that trivers teh event to nova | 01:35 |
| sean-k-mooney | *triggers | 01:35 |
| mnaser | so n-ovs-agent waits for os_vif to do its thing, and then when it sees a new port, it starts the process of wiring it up (aka the whole vlan dance), tells neutron its plugged and then neutron-server does the notify http call? | 01:35 |
| sean-k-mooney | yes | 01:36 |
| mnaser | (fwiw this is stein) | 01:36 |
| sean-k-mooney | yep this dance has not change since neutron was qunantum | 01:36 |
| sean-k-mooney | at least not for spwan | 01:36 |
| mnaser | TIL a lot | 01:36 |
| sean-k-mooney | live migration has changed but spawn is still the same | 01:37 |
| mnaser | ok so once the port is setup by nova and it starts waiting for external event, n-ovs-agent should technically start acting right away | 01:37 |
| sean-k-mooney | yes on older version when the l2 agent was using vsctl instead of the ovs python binding | 01:37 |
| sean-k-mooney | it used to poll for the port updates | 01:37 |
| sean-k-mooney | so there was more latency but it was minimal | 01:38 |
| mnaser | sean-k-mooney: well in that case, the theory is that n-ovs-agent is taking a long time to notice the port appearing | 01:38 |
| sean-k-mooney | mnaser: the condition for sending the network_vif_plugged event have 3 parts | 01:39 |
| sean-k-mooney | 1 the prot is bound and active on a host | 01:39 |
| sean-k-mooney | 2 the agent has finishiend its vlan dance | 01:39 |
| sean-k-mooney | 3 the dhcp agent has complted configing dhcp | 01:39 |
| mnaser | sean-k-mooney: because when i grep n-ovs-agent logs, i see the first 'record' of something happening with that port id is 21:37, few seconds before nova gets notified. that tells me that n-ovs-agent didn't see the port right away? | 01:39 |
| sean-k-mooney | 3 can get very slow at scale | 01:39 |
| mnaser | ah | 01:40 |
| mnaser | so maybe #3 is the issue here | 01:40 |
| mnaser | fwiw, this is doing like | 01:40 |
| mnaser | almost 250 near-concurrent port plugs | 01:40 |
| sean-k-mooney | mnaser: i would the time stamps for 1 os-vif adding the port to ovs, 2 l2 agent seeign the port, 3 dhcp agent seting it up | 01:41 |
| sean-k-mooney | mnaser: ah ha... and that causes issues :) | 01:41 |
| sean-k-mooney | let me see if i have any of the down stream bugs | 01:41 |
| sean-k-mooney | i know our networkg folk have had issue form customer related to the dhcp agent | 01:41 |
| sean-k-mooney | it might not be the same but it sound similar | 01:42 |
| mnaser | oh this is interesting | 01:42 |
| mnaser | https://www.irccloud.com/pastebin/vjt0J52B/ | 01:43 |
| mnaser | sean-k-mooney: almost 3 minutes for os_vif to plug the port? | 01:43 |
| mnaser | but i guess, "Plugging" is the step 1 | 01:43 |
| mnaser | and then "Plugged" happens after getting the external event | 01:43 |
| sean-k-mooney | yes i think so but i cna check quickly where that log is | 01:44 |
| mnaser | the first 'reference' to that port id in the neutron logs is at 2020-07-16 21:37:27.517 (port xx updated) | 01:44 |
| mnaser | (neutron ovs logs that is) | 01:44 |
| mnaser | 2020-07-16 21:37:34.415 3862298 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-bda6aed7-6f44-474a-9e4a-ff6082e7bc9e - - - - -] Configuration for devices up ['66278068-7509-40cb-8773-ed4d36801723'] and devices down [] completed. | 01:45 |
| mnaser | so the 'successfully plugged' comes in 15 seconds after neutron says im all good | 01:45 |
| sean-k-mooney | so pluging started at 40,342,704 | 01:46 |
| sean-k-mooney | that didnt work | 01:47 |
| sean-k-mooney | 2020-07-16 21:34:21.726 | 01:47 |
| sean-k-mooney | so ya the l2 agent too 3 minutes | 01:47 |
| mnaser | os_vif considers it plugged when the external id gets filled in? | 01:48 |
| sean-k-mooney | no | 01:49 |
| sean-k-mooney | https://github.com/openstack/os-vif/blob/8042e41f1bfc4b4d279934a0bed94f77454faeff/os_vif/__init__.py#L78 | 01:49 |
| sean-k-mooney | that message gets retruned when we finish adding the port to ovs | 01:49 |
| mnaser | so it took us 3 minutes to simply add a port into ovs o_O | 01:50 |
| sean-k-mooney | that is what it looks like | 01:50 |
| sean-k-mooney | mnaser: are you using the ovs-vsctl backend or python bindings | 01:50 |
| mnaser | sean-k-mooney: i haven't touched it -- but looking at the code, os_vif defaults to ovs-vsctl but neutron switched to python bindings a while back | 01:51 |
| sean-k-mooney | the default is vsctl | 01:51 |
| sean-k-mooney | https://github.com/openstack/os-vif/blob/stable/stein/vif_plug_ovs/ovs.py#L65-L68 | 01:51 |
| sean-k-mooney | in stien | 01:51 |
| mnaser | im looking at the ovs logs | 01:52 |
| sean-k-mooney | actully its still the defaul... i really though i changed that before neutron did | 01:52 |
| mnaser | i guess that means | 01:53 |
| mnaser | i can grep the logs to find an ovs-vsctl command | 01:53 |
| sean-k-mooney | yes but i know where to check the code :) im wondering if we are using --no-wait. i think we are | 01:54 |
| mnaser | this is interesting though https://www.irccloud.com/pastebin/DERdqYGW/ | 01:55 |
| sean-k-mooney | well its making an external call so eventlit will context switch and start running other code | 01:56 |
| mnaser | sean-k-mooney: with debug=true, im not seeing any executions grepping vsctl or ovs | 01:56 |
| sean-k-mooney | https://github.com/openstack/os-vif/blob/stable/ussuri/vif_plug_ovs/ovsdb/impl_vsctl.py#L333-L335 | 01:57 |
| sean-k-mooney | im not sure we are using --no-wait | 01:57 |
| sean-k-mooney | --no-wait return when the port is added to the db | 01:57 |
| sean-k-mooney | without it ovs-vsctl blocks for the dataplane to acknolage that it has create teh port | 01:58 |
| mnaser | i guess i have to find privsep logs | 01:58 |
| mnaser | cause n-cpu doesnt contain any execution logs | 01:59 |
| sean-k-mooney | they are part of the nova compute logs | 01:59 |
| *** yaawang has quit IRC | 01:59 | |
| mnaser | ps show 25% cpu usage for the privsep proc for ovs | 02:00 |
| sean-k-mooney | we reduce the privsep logs to info by default https://github.com/openstack/nova/blob/master/nova/config.py#L58 | 02:00 |
| mnaser | ah that's probably why it's not visible | 02:01 |
| sean-k-mooney | ya prvisep in debug mode dupms sensitive info live vm console to the logs | 02:01 |
| sean-k-mooney | so we do not enable privsep debug logs wehn you enable nova's debug logging | 02:01 |
| sean-k-mooney | anyway there are a few things that could be happening. | 02:02 |
| *** yaawang has joined #openstack-nova | 02:02 | |
| *** mriedem has left #openstack-nova | 02:02 | |
| sean-k-mooney | 1 nova could be bussy and wehn we context switch away it might take a while before we get back | 02:02 |
| sean-k-mooney | 2 ovs could be taking a long time to add the port | 02:03 |
| sean-k-mooney | if it is you would see this in the ovs db log | 02:03 |
| sean-k-mooney | it will have a log of when the port add is started and complete i belive | 02:03 |
| mnaser | ovsdb-server has no output (well, almost nothing) but vswitchd has a bunch | 02:04 |
| melwitt | I thought it was pluggin' | 02:04 |
| sean-k-mooney | melwitt: it is | 02:04 |
| sean-k-mooney | but there is a 3 minitu gap betwen it starting and finishing | 02:05 |
| melwitt | I know, just replying super late | 02:05 |
| mnaser | i see a bunch of this: `ovs_rcu(urcu5)|WARN|blocked 1000 ms waiting for main to quiesce` | 02:05 |
| sean-k-mooney | mnaser: in the vswitchd log you should seee somthign like this | 02:06 |
| sean-k-mooney | 2020-05-06T19:52:07.865Z|00031|bridge|INFO|bridge br-int: added interface tap36660a66-00 on port 3 | 02:06 |
| mnaser | ah, yes | 02:06 |
| sean-k-mooney | i think you are looking for tap66278068-75 | 02:07 |
| mnaser | 2020-07-16T21:37:25.283Z|11608|bridge|INFO|bridge br-int: added interface qvo66278068-75 on port 32307 | 02:07 |
| sean-k-mooney | ah you are using iptables | 02:08 |
| sean-k-mooney | not the contrack security group driver? | 02:08 |
| mnaser | yes, there's a plan to move towards openvswitch driver | 02:08 |
| mnaser | iptables_hybrid right now | 02:08 |
| sean-k-mooney | yep that is why the interface name is diffrent | 02:08 |
| sean-k-mooney | given it started plugging at 21:34:21.726 | 02:09 |
| sean-k-mooney | that took a while | 02:09 |
| sean-k-mooney | plugging finished at 21:37:49.943 | 02:10 |
| sean-k-mooney | there is still a 24 second gap but that is more resonably | 02:11 |
| mnaser | so either: nova is 'distracted' doing something else and not actually running ovs-vsctl commands? | 02:11 |
| mnaser | or the ovs-vsctl command is actually taking 3 minutes | 02:11 |
| sean-k-mooney | it could be that its in the queue of peneding funciton to execute in privsep for a while too | 02:12 |
| mnaser | that's very likely because | 02:13 |
| mnaser | https://www.irccloud.com/pastebin/hv0DDE8m/ | 02:13 |
| mnaser | 25% cpu usage for privsep sounds off | 02:13 |
| *** markvoelker has joined #openstack-nova | 02:13 | |
| sean-k-mooney | i dont know didnt you say your creating 250 port/vms at the same time | 02:14 |
| sean-k-mooney | but i think stein predates privsep multi threading | 02:14 |
| mnaser | i mean, they're not all 250 ports there, but i've seen a case of 15 vms spawning at teh same time here | 02:14 |
| mnaser | so 15 vms * 5 ports each = 75 ports being attached together at elast | 02:15 |
| sean-k-mooney | so there are a few things you could try | 02:18 |
| sean-k-mooney | if i am remeberign correctly the native ovsdb interface in os-vif does not use privsep | 02:18 |
| *** markvoelker has quit IRC | 02:18 | |
| mnaser | https://docs.openstack.org/releasenotes/oslo.privsep/stein.html -- "Privsep now uses multithreading to allow concurrency in executing privileged commands. The number of concurrent threads defaults to the available CPU cores, but can be adjusted by the new thread_pool_size config option." | 02:19 |
| mnaser | and i have privsep 1.32.1 -- cool, so that's taken care of -- i think moving to native ovsdb is probably much better/cleaner | 02:20 |
| sean-k-mooney | you can set ovsdb_interface="native" in the nova.conf | 02:20 |
| sean-k-mooney | but im just checkign the group | 02:20 |
| sean-k-mooney | i think its somthign like vif_plug_ovs but give me a sec | 02:21 |
| sean-k-mooney | i also need to chagne the default on master | 02:21 |
| *** zhanglong has quit IRC | 02:21 | |
| mnaser | sean-k-mooney: https://github.com/openstack/os-vif/blob/d588708f2149b4503e63a2c2165201c9fe399bdb/vif_plug_ovs/tests/functional/ovsdb/test_ovsdb_lib.py#L54 tests show os_vif_ovs | 02:21 |
| sean-k-mooney | ah yes [os_vif_ovs] | 02:22 |
| *** zhanglong has joined #openstack-nova | 02:22 | |
| mnaser | ok, setting to native | 02:23 |
| *** sapd__x has joined #openstack-nova | 02:23 | |
| mnaser | the plugs that happen on start up were also not very fast, taking ~5s each | 02:23 |
| mnaser | perhaps this will show difference | 02:23 |
| mnaser | 2020-07-17 02:23:44.773 1550960 DEBUG ovsdbapp.backend.ovs_idl.vlog [-] tcp:127.0.0.1:6640: entering ACTIVE _transition /openstack/venvs/nova-19.0.8/lib/python2.7/site-packages/ovs/reconnect.py:485 | 02:24 |
| sean-k-mooney | ah this is is where that gets generateed https://github.com/openstack/os-vif/blob/master/os_vif/plugin.py#L79 | 02:24 |
| sean-k-mooney | mnaser: that looks like its connect with the native backend alright | 02:24 |
| mnaser | yeah but nova-compute is pegged at 100% cpu now on start up | 02:25 |
| mnaser | time to see what its doing | 02:25 |
| sean-k-mooney | well it does a lot | 02:25 |
| sean-k-mooney | including pluging all the port of every vm on the host | 02:25 |
| sean-k-mooney | but its going to run all the reousce tracker stuff before that form init_host | 02:25 |
| mnaser | just sent USR2 | 02:26 |
| sean-k-mooney | it that the guru meditation report | 02:26 |
| mnaser | yeah | 02:27 |
| sean-k-mooney | ya i have no idea how to read those | 02:27 |
| mnaser | oddly enough though, its still taking 5 seconds to plug a port on start up | 02:27 |
| sean-k-mooney | has the privsep load dropped | 02:28 |
| sean-k-mooney | i dont know if ovsdbapp uses privsep internaly but os-vif nolonger need to use prvisep for the ovs db updates at least | 02:29 |
| mnaser | oooou i have an idea | 02:29 |
| mnaser | i think ipv6 being enabled is hurting this host | 02:29 |
| sean-k-mooney | it still needs privsep for other thngs | 02:29 |
| sean-k-mooney | oh hum | 02:29 |
| mnaser | seems like it was stuck on /openstack/venvs/nova-19.0.8/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:649 in _check_my_ip | 02:29 |
| mnaser | on start up | 02:29 |
| mnaser | im going to time that | 02:30 |
| sean-k-mooney | ya maybe | 02:30 |
| mnaser | specifically `ips = compute_utils.get_machine_ips()` | 02:30 |
| mnaser | yep, just ran that and its eating up a lot of cpu time | 02:30 |
| sean-k-mooney | that might be trying to do a revers dns lookup | 02:30 |
| mnaser | # ip addr | grep inet | wc -l => 746 | 02:31 |
| sean-k-mooney | am im going to have to call it a night but ill be around in a coule of hour. | 02:31 |
| sean-k-mooney | oh all the link local addresses | 02:31 |
| sean-k-mooney | on all the tap devices | 02:31 |
| sean-k-mooney | mnaser: artom saw an issue with this before | 02:32 |
| mnaser | len(netifaces.interfaces()) => 973 | 02:32 |
| mnaser | yeah, i think i'm onto something here | 02:32 |
| sean-k-mooney | i think he fixed it but maybe not on master | 02:32 |
| sean-k-mooney | * stien | 02:32 |
| sean-k-mooney | maybe we didnt backport it | 02:32 |
| mnaser | t = time.time(); netifaces.ifaddresses('qvo64d61663-06'); print(time.time() - t) => 133ms | 02:34 |
| mnaser | 973*133ms = 129s | 02:35 |
| sean-k-mooney | https://review.opendev.org/#/c/672161/ | 02:35 |
| mnaser | we should probably just skip qbr/qvo/qvb/tap in there | 02:35 |
| sean-k-mooney | ya proably | 02:36 |
| sean-k-mooney | so artom moved it to init_host to fix https://bugs.launchpad.net/nova/+bug/1837075 | 02:36 |
| openstack | Launchpad bug 1837075 in OpenStack Compute (nova) stein "Evacuation takes too long when destination host has a large number of NICs" [Low,Fix committed] - Assigned to Artom Lifshitz (notartom) | 02:36 |
| mnaser | but instead it takes a long time to init_host :) | 02:36 |
| sean-k-mooney | yes but we used to call it at runtime several times | 02:37 |
| sean-k-mooney | so we look it up once now and cache it | 02:37 |
| sean-k-mooney | we used to call it every time we called get_host_ip_addr | 02:38 |
| sean-k-mooney | which we did for all move operations | 02:38 |
| mnaser | yeah, perhaps that function could use skipping nova-managed devices | 02:38 |
| sean-k-mooney | this is all its used for | 02:39 |
| *** zhanglong has quit IRC | 02:39 | |
| sean-k-mooney | ips = compute_utils.get_machine_ips() | 02:39 |
| sean-k-mooney | if CONF.my_ip not in ips: | 02:39 |
| sean-k-mooney | LOG.warning('my_ip address (%(my_ip)s) was not found on ' | 02:39 |
| sean-k-mooney | 'any of the interfaces: %(ifaces)s', | 02:39 |
| sean-k-mooney | {'my_ip': CONF.my_ip, 'ifaces': ", ".join(ips)}) | 02:39 |
| sean-k-mooney | it validating that the ip we set in the config if its set is on the host | 02:39 |
| sean-k-mooney | there is proably a better way to do that | 02:39 |
| * sean-k-mooney that might be an understatement... | 02:40 | |
| mnaser | yeah i cant find a better thing but i think skipping qbr/qvo/qvb/tap might speed it up.. a lot | 02:40 |
| sean-k-mooney | where you debuging startup latency | 02:40 |
| *** zhanglong has joined #openstack-nova | 02:41 | |
| sean-k-mooney | or were you debuging an ongoing issue with new vms | 02:41 |
| sean-k-mooney | this will be a nice fix but it wont affect the later | 02:41 |
| mnaser | yeah i agree it wont affect the latter, i just switched to native and running the 'benchmark' now against the cloud | 02:41 |
| sean-k-mooney | ok well if it helps or not let me know. ill be back in 8 hours or so after i get some sleep | 02:43 |
| sean-k-mooney | o/ | 02:43 |
| mnaser | yeah, i think i'll leave it to get soak tested overnight with those changes | 02:44 |
| mnaser | but we're onto something ,thanks sean-k-mooney | 02:44 |
| sean-k-mooney | no worries | 02:45 |
| *** mkrai has joined #openstack-nova | 02:54 | |
| *** tetsuro has quit IRC | 03:13 | |
| *** psachin has joined #openstack-nova | 03:37 | |
| *** zhanglong has quit IRC | 03:47 | |
| *** markvoelker has joined #openstack-nova | 03:48 | |
| *** markvoelker has quit IRC | 03:53 | |
| *** vishalmanchanda has joined #openstack-nova | 04:01 | |
| *** huaqiang has joined #openstack-nova | 04:12 | |
| openstackgerrit | Huaqiang Wang proposed openstack/nova master: hardware: create 'mixed' instance for realtime CPUs https://review.opendev.org/728485 | 04:21 |
| openstackgerrit | Huaqiang Wang proposed openstack/nova master: compute: bump nova-compute version and check in API https://review.opendev.org/740933 | 04:21 |
| openstackgerrit | Huaqiang Wang proposed openstack/nova master: metadata: export the vCPU IDs that are pinning on the host CPUs https://review.opendev.org/688936 | 04:21 |
| openstackgerrit | Huaqiang Wang proposed openstack/nova master: release note: add new features introduces by use-pcpu-and-vcpu-in-one-instance https://review.opendev.org/736997 | 04:21 |
| openstackgerrit | Huaqiang Wang proposed openstack/nova master: test: add some notes and new tests for mixed instance https://review.opendev.org/741395 | 04:21 |
| *** bbowen has quit IRC | 04:30 | |
| *** bbowen has joined #openstack-nova | 04:30 | |
| *** rcernin has quit IRC | 04:47 | |
| *** tetsuro has joined #openstack-nova | 04:50 | |
| *** Liang__ has quit IRC | 04:54 | |
| *** KeithMnemonic has quit IRC | 04:55 | |
| *** Liang__ has joined #openstack-nova | 04:55 | |
| *** udesale has joined #openstack-nova | 04:57 | |
| *** rcernin has joined #openstack-nova | 04:57 | |
| *** Liang__ has quit IRC | 05:01 | |
| *** Liang__ has joined #openstack-nova | 05:02 | |
| *** yaawang has quit IRC | 05:04 | |
| *** yaawang has joined #openstack-nova | 05:05 | |
| *** rcernin has quit IRC | 05:12 | |
| *** suryasingh has joined #openstack-nova | 05:18 | |
| *** rcernin has joined #openstack-nova | 05:20 | |
| *** tetsuro has quit IRC | 05:21 | |
| *** ratailor has joined #openstack-nova | 05:29 | |
| *** links has joined #openstack-nova | 05:37 | |
| *** zhanglong has joined #openstack-nova | 05:44 | |
| *** yaawang has quit IRC | 05:45 | |
| *** yaawang has joined #openstack-nova | 05:45 | |
| *** Liang__ has quit IRC | 05:47 | |
| *** Liang__ has joined #openstack-nova | 05:48 | |
| *** markvoelker has joined #openstack-nova | 05:49 | |
| *** JamesBen_ has quit IRC | 05:51 | |
| *** JamesBenson has joined #openstack-nova | 05:53 | |
| *** markvoelker has quit IRC | 05:54 | |
| *** xiaolin has joined #openstack-nova | 05:54 | |
| *** JamesBenson has quit IRC | 05:58 | |
| openstackgerrit | Huaqiang Wang proposed openstack/nova master: hardware: create 'mixed' instance for realtime CPUs https://review.opendev.org/728485 | 06:07 |
| openstackgerrit | Huaqiang Wang proposed openstack/nova master: compute: bump nova-compute version and check in API https://review.opendev.org/740933 | 06:07 |
| openstackgerrit | Huaqiang Wang proposed openstack/nova master: metadata: export the vCPU IDs that are pinning on the host CPUs https://review.opendev.org/688936 | 06:07 |
| openstackgerrit | Huaqiang Wang proposed openstack/nova master: release note: add new features introduces by use-pcpu-and-vcpu-in-one-instance https://review.opendev.org/736997 | 06:07 |
| openstackgerrit | Huaqiang Wang proposed openstack/nova master: test: add some notes and new tests for mixed instance https://review.opendev.org/741395 | 06:07 |
| openstackgerrit | Wenping Song proposed openstack/nova-specs master: Add no user token when get Cyborg client https://review.opendev.org/740184 | 06:12 |
| *** rcernin has quit IRC | 06:19 | |
| *** jsuchome has joined #openstack-nova | 06:25 | |
| *** rcernin has joined #openstack-nova | 06:32 | |
| *** yaawang has quit IRC | 06:32 | |
| *** yaawang has joined #openstack-nova | 06:33 | |
| *** JamesBenson has joined #openstack-nova | 06:41 | |
| *** tetsuro has joined #openstack-nova | 06:43 | |
| *** JamesBenson has quit IRC | 06:45 | |
| *** dklyle has quit IRC | 06:46 | |
| *** rcernin has quit IRC | 06:46 | |
| *** tetsuro has quit IRC | 06:48 | |
| *** tetsuro has joined #openstack-nova | 06:50 | |
| *** rcernin has joined #openstack-nova | 06:53 | |
| *** tetsuro has quit IRC | 06:55 | |
| *** xiaolin has quit IRC | 06:55 | |
| *** maciejjozefczyk has joined #openstack-nova | 06:56 | |
| *** zhanglong has quit IRC | 07:04 | |
| *** yaawang has quit IRC | 07:04 | |
| *** yaawang has joined #openstack-nova | 07:05 | |
| *** zhanglong has joined #openstack-nova | 07:05 | |
| *** rcernin has quit IRC | 07:06 | |
| *** tesseract has joined #openstack-nova | 07:15 | |
| *** udesale has quit IRC | 07:17 | |
| *** mkrai has quit IRC | 07:21 | |
| *** ralonsoh has joined #openstack-nova | 07:23 | |
| *** Liang__ has quit IRC | 07:25 | |
| *** brinzhang0 has joined #openstack-nova | 07:26 | |
| *** brinzhang_ has quit IRC | 07:29 | |
| *** tosky has joined #openstack-nova | 07:33 | |
| *** rcernin has joined #openstack-nova | 07:35 | |
| openstackgerrit | Kevin Zhao proposed openstack/nova master: [WIP] CI: add tempest-integrated-compute-aarch64 job https://review.opendev.org/714439 | 07:38 |
| *** tetsuro has joined #openstack-nova | 07:41 | |
| *** rcernin has quit IRC | 07:42 | |
| *** nightmare_unreal has joined #openstack-nova | 07:43 | |
| openstackgerrit | Luigi Toscano proposed openstack/nova stable/ussuri: zuul: remove legacy-tempest-dsvm-neutron-dvr-multinode-full https://review.opendev.org/741590 | 07:45 |
| *** rcernin has joined #openstack-nova | 07:46 | |
| *** tetsuro has quit IRC | 07:47 | |
| *** jsuchome has quit IRC | 07:48 | |
| *** zhanglong has quit IRC | 07:50 | |
| *** rcernin has quit IRC | 07:50 | |
| *** zhanglong has joined #openstack-nova | 07:51 | |
| *** mkrai has joined #openstack-nova | 07:52 | |
| *** jsuchome has joined #openstack-nova | 07:52 | |
| *** udesale has joined #openstack-nova | 07:57 | |
| *** brinzhang_ has joined #openstack-nova | 07:59 | |
| *** jangutter has joined #openstack-nova | 08:01 | |
| *** tetsuro has joined #openstack-nova | 08:02 | |
| *** brinzhang0 has quit IRC | 08:02 | |
| *** jangutter_ has quit IRC | 08:03 | |
| *** ociuhandu has joined #openstack-nova | 08:08 | |
| *** k_mouza has joined #openstack-nova | 08:09 | |
| openstackgerrit | Kevin Zhao proposed openstack/nova master: [WIP] CI: add tempest-integrated-compute-aarch64 job https://review.opendev.org/714439 | 08:14 |
| *** tetsuro has quit IRC | 08:23 | |
| *** derekh has joined #openstack-nova | 08:30 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova-specs master: Support SRIOV interface attach and detach https://review.opendev.org/741601 | 08:37 |
| *** JamesBenson has joined #openstack-nova | 08:42 | |
| *** JamesBenson has quit IRC | 08:47 | |
| *** brinzhang0 has joined #openstack-nova | 08:47 | |
| *** spatel has joined #openstack-nova | 08:49 | |
| *** brinzhang_ has quit IRC | 08:49 | |
| *** xek_ has joined #openstack-nova | 08:51 | |
| *** spatel has quit IRC | 08:53 | |
| openstackgerrit | Tobias Urdin proposed openstack/nova master: [PoC] libvirt: Broadcast RARP post live migration https://review.opendev.org/741529 | 09:02 |
| openstackgerrit | Tobias Urdin proposed openstack/nova master: [PoC] libvirt: Send announce-self post live migration https://review.opendev.org/741529 | 09:03 |
| *** zhanglong has quit IRC | 09:04 | |
| tobias-urdin | sean-k-mooney: yes I agree, it's a dirty workaround but until it is solved I will need to carry this downstream | 09:04 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Remove broken legacy zuul jobs https://review.opendev.org/741603 | 09:05 |
| *** zhanglong has joined #openstack-nova | 09:05 | |
| gibi | gmann, tosky: I proposed some clean of our jobs ^^ | 09:07 |
| tosky | gibi: as long as it works for you, I'm fine :) | 09:08 |
| gibi | tosky: they never passed | 09:08 |
| gibi | tosky: and they are really old jobs | 09:08 |
| *** brinzhang_ has joined #openstack-nova | 09:09 | |
| gibi | and anyhow if somebody needs them and cares about them then we can resurrect them in zuul v3 form later | 09:09 |
| tosky | I don't remember if I mentioned it yesterday: even though it's not part of the main goal (just stretch), if you plan to backport the fixes to the older branches, that will help cleaning up openstack-zuul-jobs | 09:09 |
| tosky | that's why I proposed a backport of the fix that was merged yesterday | 09:09 |
| gibi | tosky: I'm happy to backport these job deletes to older branches | 09:09 |
| tosky | thanks :) | 09:10 |
| *** brinzhang0 has quit IRC | 09:12 | |
| *** sapd__x has quit IRC | 09:12 | |
| *** martinkennelly has joined #openstack-nova | 09:14 | |
| *** zhanglong has quit IRC | 09:14 | |
| *** zhanglong has joined #openstack-nova | 09:17 | |
| *** sapd1 has joined #openstack-nova | 09:18 | |
| *** martinkennelly has quit IRC | 09:19 | |
| *** dtantsur|afk is now known as dtantsur | 09:19 | |
| gibi | lyarwood, melwitt: if you have time could you hit this pike backport? https://review.opendev.org/#/q/topic:bug/1862633+status:open | 09:23 |
| *** rcernin has joined #openstack-nova | 09:29 | |
| *** zhanglong has quit IRC | 09:30 | |
| *** zhanglong has joined #openstack-nova | 09:31 | |
| *** Liang__ has joined #openstack-nova | 09:36 | |
| *** Liang__ has quit IRC | 09:41 | |
| *** rcernin has quit IRC | 09:48 | |
| lyarwood | gibi: ack I can look today | 09:52 |
| gibi | thanks | 09:52 |
| *** tkajinam has quit IRC | 09:52 | |
| gibi | sean-k-mooney, stephenfin: fyi I pushed a small spec for SRIOV attach https://review.opendev.org/#/c/741601/ | 09:55 |
| *** udesale_ has joined #openstack-nova | 10:01 | |
| *** yaawang has quit IRC | 10:02 | |
| *** zhanglong has quit IRC | 10:02 | |
| *** zhanglong has joined #openstack-nova | 10:03 | |
| *** udesale has quit IRC | 10:04 | |
| *** yaawang has joined #openstack-nova | 10:04 | |
| *** dtantsur is now known as dtantsur|brb | 10:07 | |
| *** yaawang has quit IRC | 10:14 | |
| *** zhanglong has quit IRC | 10:15 | |
| *** yaawang has joined #openstack-nova | 10:21 | |
| *** brinzhang0 has joined #openstack-nova | 10:22 | |
| openstackgerrit | Wenping Song proposed openstack/nova master: Cyborg shelve/unshelve support https://review.opendev.org/729563 | 10:22 |
| *** brinzhang_ has quit IRC | 10:25 | |
| brinzhang0 | gibi: hi, I update the nova runway, added the cyborg-shelve-and-unshelve patches to the queue | 10:26 |
| *** brinzhang0 has quit IRC | 10:30 | |
| *** JamesBenson has joined #openstack-nova | 10:43 | |
| stephenfin | gibi: I lost the race for some o.vo versions. Could I get your +2 on this again? https://review.opendev.org/#/c/739210/ | 10:46 |
| *** JamesBenson has quit IRC | 10:47 | |
| stephenfin | gibi: -1 on spec for nits but otherwise perfect | 10:51 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Re-enable live snapshot for paused instances https://review.opendev.org/740334 | 10:53 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Remove workaround for really old QEMU https://review.opendev.org/740335 | 10:53 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Use better variable names for '_create_guest' https://review.opendev.org/740945 | 10:53 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP: Attempt to restore some sanity to snapshot https://review.opendev.org/740336 | 10:53 |
| *** mkrai has quit IRC | 10:53 | |
| stephenfin | lyarwood: The first two of those are pretty simple but would benefit from your eyes, IMO. The third is a FUP for your comments ^ | 10:53 |
| kashyap | stephenfin: Good catch on re-enabling the 'paused' thing | 10:56 |
| kashyap | I recall the old change | 10:56 |
| kashyap | Yep, it makes sense, given how far we've come w/ newer (and largely improved) versions | 10:57 |
| *** martinkennelly has joined #openstack-nova | 10:57 | |
| *** sapd1 has quit IRC | 10:57 | |
| *** mgariepy has quit IRC | 10:58 | |
| *** martinkennelly has quit IRC | 11:01 | |
| sean-k-mooney | gibi: ill read your sepc shortly. did you see my comments on the code change. | 11:37 |
| sean-k-mooney | your code change wont fix PF detach either so i might just add a small patch on top to do that | 11:37 |
| *** raildo has joined #openstack-nova | 11:41 | |
| *** JamesBenson has joined #openstack-nova | 11:45 | |
| *** xek_ has quit IRC | 11:50 | |
| *** mgariepy has joined #openstack-nova | 12:02 | |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: api: Add type hints for 'nova.compute.api' https://review.opendev.org/741623 | 12:04 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP: api: Add 'nova.compute.API.migrate' https://review.opendev.org/741624 | 12:04 |
| openstackgerrit | Elod Illes proposed openstack/nova stable/queens: Reject boot request for unsupported images https://review.opendev.org/708609 | 12:05 |
| *** dtantsur|brb is now known as dtantsur | 12:10 | |
| *** ratailor has quit IRC | 12:20 | |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Handle VIR_ERR_DEVICE_MISSING when detaching devices https://review.opendev.org/741561 | 12:21 |
| *** haleyb has quit IRC | 12:21 | |
| *** JamesBenson has quit IRC | 12:25 | |
| *** JamesBenson has joined #openstack-nova | 12:26 | |
| *** huaqiang has quit IRC | 12:39 | |
| *** huaqiang has joined #openstack-nova | 12:40 | |
| *** spatel has joined #openstack-nova | 12:50 | |
| *** jangutter_ has joined #openstack-nova | 12:54 | |
| *** spatel has quit IRC | 12:55 | |
| *** jangutter has quit IRC | 12:58 | |
| jkulik | How often and how do you clean up your instances table in the cell0 db? | 13:01 |
| sean-k-mooney | i think you do it with nova-manage | 13:01 |
| sean-k-mooney | how often is up to you | 13:02 |
| sean-k-mooney | i think downstream we default to every 30days | 13:02 |
| jkulik | hm ... I thought so, but I can't seem to find the docs or the code for that | 13:03 |
| sean-k-mooney | i think you do "nova-manage db archive_deleted_rows" then "nova-manage db purge" | 13:05 |
| sean-k-mooney | but you need to us a config file with the cell0 credentials | 13:05 |
| gibi | stephenfin: thanks. | 13:06 |
| jkulik | deleted_at is NULL for the instance I'm looking at. it's over 200 days old | 13:06 |
| gibi | sean-k-mooney: I haven't read the comments on the code yet | 13:06 |
| gibi | sean-k-mooney: I have some review to make before I can circle back to the sriov code patches | 13:07 |
| sean-k-mooney | gibi: there is no rush | 13:07 |
| sean-k-mooney | it can wait till next week | 13:08 |
| gibi | ack | 13:08 |
| jkulik | Seems to be just that one instance, that's not deleted, though. Thank you. | 13:08 |
| mnaser | i think the archive code didn't cover cell0 before and recently added support to go over all cells | 13:09 |
| mnaser | if i'm remembering right | 13:09 |
| sean-k-mooney | mnaser: so by default nova manage used to only look at cell1 | 13:09 |
| sean-k-mooney | you could create a second config can use that to run the commands on cell0 | 13:09 |
| mnaser | yeah, i get this feeling i remember there was an improvement that let you run it against all cells more recently | 13:10 |
| sean-k-mooney | but i dont know if we have now made it do that by default or not | 13:10 |
| * mnaser checks | 13:10 | |
| sean-k-mooney | it came up as a downstream bug i think and melwitt? might have imporved it | 13:10 |
| mnaser | sean-k-mooney: good memory https://review.opendev.org/#/c/507486/ | 13:11 |
| sean-k-mooney | 11 months so proably train? | 13:11 |
| mnaser | yes, the earliest tag this commit is part of is train | 13:11 |
| mnaser | or 20.0.0 | 13:12 |
| mnaser | so yeah, train onwards | 13:12 |
| lyarwood | melwitt: https://review.opendev.org/#/c/741561/ - would you be able to take a look at this today? | 13:12 |
| sean-k-mooney | im not sure if this is backportable or not but jkulik are you pre train | 13:13 |
| jkulik | I'm fine for now, thank you. (we're on queens but going to upgrade soon. it can wait. the instance in question just wasn't marked deleted and thus still came up in "server list" and customers wondered ...) | 13:13 |
| sean-k-mooney | if it was in cell0 then it would have been in error | 13:14 |
| jkulik | It was actually in state BUILD | 13:14 |
| sean-k-mooney | really? that should not happen | 13:14 |
| sean-k-mooney | that likely means you had an rpc issue at some point that lost a callback to set it to error | 13:15 |
| jkulik | but since that happened somewhere last year, I also don't have logs anymore. this being the only instance in that state, I think it can be ignored. | 13:15 |
| sean-k-mooney | ya but if you do see this crop up more often let us know | 13:15 |
| sean-k-mooney | queens is sufficently old that if you hit an issue its proably already fixed but still good to know | 13:16 |
| mnaser | yeah, i've ran into a few of the isssues that jkulik ran into and cleaned them up in later release | 13:16 |
| mnaser | mostly around making things happen in a single transaction to avoid BUILD inside cell0 | 13:16 |
| *** k_mouza has quit IRC | 13:17 | |
| sean-k-mooney | maybe that is something we shoudl add to nova audit | 13:17 |
| mnaser | btw - debug from yesterday's issue exposes this https://www.irccloud.com/pastebin/FdpUKxPC/ | 13:18 |
| sean-k-mooney | set instance in build in cell0 to error after x time | 13:18 |
| mnaser | so it sounds like the actual commit to addport happened 34 seconds after | 13:18 |
| sean-k-mooney | ya so if your using iptables os-vif need to create a linux bridge then a veth pair then add one end of the veth pair to the linux bridge and the other end to ovs | 13:19 |
| mnaser | so maybe that part took a long time so i need to do some more log grepping | 13:20 |
| sean-k-mooney | so that can take a few second although i would not normally expect it to be that long | 13:20 |
| mnaser | system has 1157 total devices | 13:20 |
| sean-k-mooney | going form vsctl to native seams to have cut it form 3 mins to ~30 seconds so thats an improvement at least | 13:21 |
| mnaser | from qvo/qvb/qbr | 13:21 |
| mnaser | yeah -- i still have instances that take 900s to provision so it might still be bad too | 13:21 |
| sean-k-mooney | mnaser: if you change form iptables to conntrack then it elimiandes all the hybrid plug stuff so os-vif just add a port to ovs and is done | 13:21 |
| mnaser | sean-k-mooney: yeah i have another one here that took 3 minutes too | 13:22 |
| mnaser | so i suspect native vs ovsdb wasn't the big help here | 13:22 |
| sean-k-mooney | likely the large number of ports is cause some of the operation to be slower | 13:22 |
| sean-k-mooney | ah ok | 13:22 |
| mnaser | sean-k-mooney: but firewall driver is defined in neutron? and this is happening inside nova? | 13:23 |
| mnaser | does nova become somehow aware from what firewall driver is being used? | 13:23 |
| sean-k-mooney | yes | 13:24 |
| sean-k-mooney | the vif binding_details has a hybrid_plug atribute | 13:24 |
| mnaser | ah gotcha | 13:25 |
| sean-k-mooney | its true for iptables and false for contrac | 13:25 |
| sean-k-mooney | os-vif just does what neutron tells it to do | 13:25 |
| mnaser | and so when donig the PUT for the binding | 13:25 |
| mnaser | we get the info back telling us how to do it | 13:25 |
| sean-k-mooney | yep | 13:25 |
| mnaser | ok i see. i wonder if i can restart straight into openvswitch firewall driver | 13:25 |
| sean-k-mooney | we use that respocne both to generate the libvirt xml and pass it to os-vif to tell it how to add the port | 13:26 |
| sean-k-mooney | not quite | 13:26 |
| mnaser | i've looked at the code and it seems like it does have some code which 'remanages' it fwiw | 13:26 |
| sean-k-mooney | you can but the ports on exiting vms wont be rebound | 13:26 |
| *** artom has joined #openstack-nova | 13:26 | |
| sean-k-mooney | so you need to live migrate teh vms or shelve and unshelve them | 13:26 |
| sean-k-mooney | i guess any move operation but hardreboot wont fix it | 13:27 |
| mnaser | sean-k-mooney: have you seen this code? https://github.com/openstack/neutron/blob/master/neutron/agent/linux/openvswitch_firewall/iptables.py | 13:27 |
| *** nhicher has joined #openstack-nova | 13:27 | |
| mnaser | i guess i don't really care that much if the existing vms stay hybrid plugged | 13:27 |
| sean-k-mooney | so wehn you change it i think that code is nolonger loaded or run | 13:28 |
| mnaser | as long as the new ones don't use hybrid plugging which likely will result in faster os_vif plug time | 13:28 |
| sean-k-mooney | so i thnk you will lose the ablity to manage security group rules on the exsting vms | 13:28 |
| sean-k-mooney | i cant remeber if it was stien we added multiple port bining but when we started using that in nova we got the ability to live migrate between hosts with different firewall driver | 13:29 |
| sean-k-mooney | which will allow you to change in a rolling upgade style | 13:30 |
| *** guilhermesp has joined #openstack-nova | 13:30 | |
| mnaser | sean-k-mooney: i think looking at the code, switching can work, it removes the iptables rules and then just starts applying the rules on the existing qvo iface | 13:31 |
| mnaser | https://github.com/openstack/neutron/blob/master/neutron/agent/linux/openvswitch_firewall/firewall.py#L614-L636 | 13:31 |
| sean-k-mooney | ah ok ya i just notice it did not change the hybridg plug state | 13:32 |
| sean-k-mooney | if it actully removes the iptable rules then it will work fine | 13:32 |
| sean-k-mooney | the existging vm will keep using hybrid_plug untill there ports are rebound | 13:32 |
| mnaser | we've switched them in the past and it wasn't an issue | 13:32 |
| mnaser | yep, but the firwall rules will apply on the qvo instead of tap | 13:32 |
| sean-k-mooney | cool | 13:33 |
| mnaser | and new plugs will plumb tap in and eliminate qbr/qvo/qvb | 13:33 |
| sean-k-mooney | yep | 13:33 |
| mnaser | which will significantly reduce the number of ports on the system | 13:33 |
| sean-k-mooney | it will devide it by 3 | 13:33 |
| stephenfin | lyarwood: replied on https://review.opendev.org/#/c/699291/ | 13:33 |
| mnaser | also i think tap interfaces won't have a random ipv6 addr show up | 13:34 |
| sean-k-mooney | that is also likely | 13:34 |
| sean-k-mooney | well maybe not | 13:34 |
| sean-k-mooney | let me check | 13:34 |
| mnaser | dmesg shows ADDRCONF events for qvb and qvo | 13:34 |
| mnaser | but none for tap | 13:34 |
| sean-k-mooney | they still get link local adresses | 13:35 |
| sean-k-mooney | that siad it is some thing that we could disable in os-vif | 13:35 |
| stephenfin | yo, bauzas. Want some d̶r̶u̶g̶s̶ patches? https://review.opendev.org/#/c/729595/ https://review.opendev.org/#/c/729596/ | 13:35 |
| sean-k-mooney | actully no we cant | 13:35 |
| sean-k-mooney | when os-vif creates the port on ovs its before qemu has created the tap | 13:36 |
| bauzas | stephenfin: https://media.tenor.com/images/a2b7c73a67cf6c1a775466e6ad87d8b7/tenor.gif | 13:36 |
| stephenfin | :P | 13:37 |
| bauzas | (you need to know about Weeds, dude) | 13:37 |
| stephenfin | I know GIFs | 13:37 |
| * bauzas is currently writing a spec, so *any* review ask is good to me :) | 13:37 | |
| bauzas | could I provide a French spec ? :p | 13:38 |
| stephenfin | Sure! Just hope you're happy with artom being the only one reviewing it | 13:38 |
| mnaser | sean-k-mooney: check this out -- https://github.com/openstack/neutron/blob/23e3213a07eb0b0fcdd2a1da36a847dde9beba57/neutron/tests/fullstack/test_firewall.py | 13:41 |
| mnaser | neutron tests switching to openvswitch with agent restart :) | 13:41 |
| stephenfin | lyarwood: Also, in case you didn't know already, I removed the auto-branch naming feature from git-review. If you want topics for e.g. https://review.opendev.org/#/c/741561/ you need to create the branch yourself | 13:42 |
| sean-k-mooney | mnaser: cool | 13:43 |
| sean-k-mooney | mnaser: that makes the upgrade path much smother | 13:44 |
| stephenfin | sean-k-mooney, gibi, (others): Need input of vTPM design. What should we do if we resize and the new flavor has a different vTPM config? | 13:44 |
| sean-k-mooney | that was in the spec | 13:44 |
| sean-k-mooney | we have 2 option reject the resize or what the spec says is we recreat it with the new format lossing all data | 13:45 |
| stephenfin | Right, but what about revert? | 13:45 |
| sean-k-mooney | stephenfin: there is no way to convert form one type to the other | 13:45 |
| stephenfin | Is resize expected to be a destructive operation? | 13:45 |
| stephenfin | I know rebuild is | 13:45 |
| stephenfin | but didn't think resize was | 13:45 |
| sean-k-mooney | it should not be an issue unless we are talking about same host resize | 13:45 |
| stephenfin | Well it's awkward to implement, hence why I'm asking :) | 13:46 |
| stephenfin | I need to stash the ID of the old key stored in the key manager service | 13:46 |
| sean-k-mooney | well we dont want to destoy the old tpm untill resize confimr or reviert | 13:47 |
| sean-k-mooney | yes you would | 13:47 |
| sean-k-mooney | but you can do it the same way we do for flavors | 13:47 |
| sean-k-mooney | stephenfin: basically what we said in the spec was pretend it really hardware | 13:48 |
| stephenfin | Can't we just block it like we do for NUMA | 13:49 |
| sean-k-mooney | for reall hardware if we swapped the mother board which is what a resize is then it would be lose the data | 13:49 |
| sean-k-mooney | stephenfin: yes we could that was option 1 | 13:49 |
| stephenfin | I'm tempted to suggest we do that anyway, since I think this is unlikely to be used much in practice | 13:50 |
| openstackgerrit | Merged openstack/nova stable/queens: libvirt: Don't delete disks on shared storage during evacuate https://review.opendev.org/732717 | 13:50 |
| stephenfin | so long as I explicitly block it like you did for NUMA | 13:50 |
| sean-k-mooney | if you block it for resize i assume the same will be true for rebuild | 13:52 |
| sean-k-mooney | rebuild is not ment to be destuctive | 13:52 |
| stephenfin | that would be my thinking, yes | 13:52 |
| sean-k-mooney | but it would be if an only if you cnaged the type | 13:52 |
| stephenfin | config from flavor + image meta must be identical | 13:52 |
| sean-k-mooney | yep whcih is exactly what we do for numa | 13:52 |
| sean-k-mooney | on rebuild at least | 13:53 |
| sean-k-mooney | but before going down this route | 13:53 |
| sean-k-mooney | do you need to have 2 keys | 13:53 |
| sean-k-mooney | could you not just use the same key for both vtpms | 13:53 |
| stephenfin | that's an interesting point | 13:54 |
| stephenfin | we could indeed, given the owner has changed | 13:54 |
| stephenfin | let me see how that works | 13:54 |
| * stephenfin reverts everything he's done in the past 24 hours \o/ | 13:55 | |
| stephenfin | twas all garbage anyway | 13:55 |
| mnaser | sean-k-mooney: fyi, you might find this interesting -- https://bugzilla.redhat.com/show_bug.cgi?id=1782834 and neutron workaround https://review.opendev.org/#/c/733674/ for ovs 2.12 | 13:55 |
| openstack | bugzilla.redhat.com bug 1782834 in openvswitch "Changing protocols in Bridge table doesn't take effect" [High,New] - Assigned to aconole | 13:55 |
| *** tobias-urdin is now known as tobias-urdin|pto | 13:55 | |
| sean-k-mooney | if that proves difficutl to implement we can use that as justification for blocking and move the two thing you tried into alternitives | 13:55 |
| sean-k-mooney | mnaser: huh that inconveniant | 13:57 |
| sean-k-mooney | mnaser: strangly enough i was asked to look at this patch earlier today | 13:57 |
| sean-k-mooney | https://review.opendev.org/#/c/738551/7/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py@2163 | 13:57 |
| melwitt | lyarwood: sure, will do | 13:57 |
| sean-k-mooney | which is also setting the openflow versions | 13:58 |
| *** xek_ has joined #openstack-nova | 14:01 | |
| mnaser | ovs-vsctl set bridge br-int protocols=OpenFlow10,OpenFlow11,OpenFlow12,OpenFlow13,OpenFlow14,OpenFlow15 doesn't really fix much | 14:01 |
| mnaser | let me restart the ovs agent after that | 14:02 |
| *** mlavalle has joined #openstack-nova | 14:04 | |
| *** k_mouza has joined #openstack-nova | 14:12 | |
| *** haleyb has joined #openstack-nova | 14:20 | |
| gibi | stephenfin: regarding vTPM. We can block the resize and rebuild if that result in a loss of vTPM data as a first step. I'm fine with taht | 14:31 |
| *** k_mouza has quit IRC | 14:31 | |
| *** READ10 has joined #openstack-nova | 14:34 | |
| *** READ10 has quit IRC | 14:36 | |
| *** xek_ has quit IRC | 14:41 | |
| *** k_mouza has joined #openstack-nova | 14:42 | |
| *** dklyle has joined #openstack-nova | 14:42 | |
| gibi | dansmith: hi! I talked to the release team about M2 and it turned out that we don't need a nova release just an os-vif and python-novaclient release and those can be made before M2. So I will propose those lib releases next week and then there is nothing to do at M2 from release perspective | 14:43 |
| dansmith | gibi: I saw, cool, I *definitely* volunteer then :) | 14:43 |
| gibi | cool :) | 14:43 |
| gibi | dansmith: can I ask you to run the weekly meeting on the 30th? | 14:45 |
| dansmith | gibi: yeah | 14:46 |
| gibi | thank you | 14:46 |
| gmann | gibi: zero_downtime upgrade job use zero_downtime/hooks/run_tests.sh which i did not find in git history at all when it was added/remvoed - https://opendev.org/openstack/openstack-zuul-jobs/src/branch/master/playbooks/legacy/grenade-dsvm-neutron-multinode-zero-downtime/run.yaml#L45 | 14:49 |
| gmann | we do not test zero downtime upgrade (i think none of project does) in any other job | 14:50 |
| gmann | so i agree to remove this broken job and if we want/do test zero_downtime in future then we write new job on zuulv3 | 14:51 |
| gibi | gmann: thanks for checking | 14:51 |
| gmann | none of the projects use zero_downtime upgrade TC tag https://governance.openstack.org/tc/reference/tags/assert_supports-zero-downtime-upgrade.html | 14:52 |
| lyarwood | kashyap: https://review.opendev.org/#/c/741561/ - if you have time for a review before leaving for the weekend btw | 14:52 |
| *** READ10 has joined #openstack-nova | 14:53 | |
| gmann | I will remove(once all stable branch remove the job ) the job definition also from opensatck-zuul-jobs as this job is only used by nova | 14:53 |
| kashyap | lyarwood: Hiya, definitely | 14:53 |
| gibi | gmann: cool | 14:54 |
| kashyap | lyarwood: Looks good; also bonus marks for the nice reproducer write-up! | 14:57 |
| sean-k-mooney | gmann: im not sure that any of the service got to the poitn where they could have contol planes on two differetn versions did they | 14:58 |
| sean-k-mooney | that would be the main issue for nova | 14:58 |
| sean-k-mooney | n an n+1 would expect diffferent db version and potatiall rpcs version so we cannot run both in parralel | 14:59 |
| sean-k-mooney | even if we can upgrade the contoelr indepently of the comptue nodes | 14:59 |
| gmann | sean-k-mooney: is not load balancer should disable n-api on node which is going to be upgraded and redirect request to other api node. once first api node is upgraded then bring up the n-api and disable 2nd api node .. | 15:06 |
| gmann | i mean HAProxy should be disable the request on one api node when we upgrade that so that request can be served from other api node and zero downtime on api service | 15:09 |
| *** bnemec is now known as beekneemech | 15:10 | |
| sean-k-mooney | gmann: no that wont work | 15:11 |
| *** maciejjozefczyk has quit IRC | 15:14 | |
| bauzas | gibi: dansmith: fwiw, sorry if I was unclear but i'll be there on week from July 27th to 30th (excluding Friday probably) | 15:18 |
| gibi | bauzas: no worries. It turned out that there is nothing pressing to handle at M2 | 15:18 |
| bauzas | but either way, all good if dansmith can release m-2 and lead the meeting :) | 15:18 |
| gibi | :) | 15:18 |
| *** ociuhandu_ has joined #openstack-nova | 15:20 | |
| sean-k-mooney | gmann: its not just the api its also the conductor and schuler that need to agree on the db schemes and rpc verions | 15:20 |
| *** ociuhandu has quit IRC | 15:24 | |
| *** ociuhandu_ has quit IRC | 15:24 | |
| *** gyee has joined #openstack-nova | 15:26 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova-specs master: Support SRIOV interface attach and detach https://review.opendev.org/741601 | 15:29 |
| gibi | stephenfin, sean-k-mooney: ^^ fixed up the spec | 15:29 |
| melwitt | lyarwood: commented, there are some odd things in the tests (args passed but not used) and tests that are simulating non-real situations, not sure if that was intended | 15:30 |
| lyarwood | melwitt: ack looking | 15:32 |
| lyarwood | sean-k-mooney: https://review.opendev.org/#/c/728057/ btw, looks like this change blows up when we log anything in ERROR | 15:32 |
| stephenfin | gibi: +2 for me. Seems like a no-brainer if you're able to do the work | 15:36 |
| sean-k-mooney | it does? | 15:37 |
| *** udesale_ has quit IRC | 15:37 | |
| stephenfin | sean-k-mooney: I thought you were talking to me there for a sec | 15:37 |
| sean-k-mooney | it is not even specific to the error log | 15:37 |
| sean-k-mooney | oh no lyarwood ^ | 15:38 |
| sean-k-mooney | gibi: looking now | 15:39 |
| *** mlavalle has quit IRC | 15:40 | |
| gibi | stephenfin: I think I can do the work. The WIP patch already works for me in devstack so only automatic testing and documentation is missing. | 15:40 |
| gibi | stephenfin, sean-k-mooney: thanks | 15:40 |
| stephenfin | \o/ | 15:40 |
| lyarwood | sean-k-mooney: yeah I assume py27 doesn't like the filter you provided? | 15:41 |
| sean-k-mooney | gibi: +1 | 15:41 |
| *** mlavalle has joined #openstack-nova | 15:41 | |
| sean-k-mooney | lyarwood: ah yes that is possible | 15:41 |
| gibi | awesome, thanks | 15:41 |
| sean-k-mooney | lyarwood: its a generator expression | 15:42 |
| lyarwood | https://docs.python.org/2/library/logging.html#filter-objects | 15:42 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: mypy: Add type annotations to 'nova.pci' https://review.opendev.org/676209 | 15:42 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: mypy: Add nova.cmd, nova.conf, nova.console https://review.opendev.org/705657 | 15:42 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: mypy: Add type annotations to top-level modules https://review.opendev.org/705658 | 15:42 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: rpc: Rework 'get_notifier', 'wrap_exception' https://review.opendev.org/741663 | 15:42 |
| melwitt | lyarwood: fyi guh, the ceph job looks like it's been angry on stable branches | 15:42 |
| sean-k-mooney | return not any(msg in log_record.msg for msg in messages) | 15:42 |
| lyarwood | it needs to be in a filter method | 15:42 |
| lyarwood | melwitt: yeah I have a few changes open to look at | 15:42 |
| melwitt | ack | 15:42 |
| stephenfin | sean-k-mooney, lyarwood: Looks like it needs to be a specific class in Python 2? https://docs.python.org/2/library/logging.html#filter | 15:43 |
| stephenfin | https://docs.python.org/2/library/logging.html#filter-objects | 15:44 |
| lyarwood | right | 15:44 |
| stephenfin | actually idk, nothing in the docs by way of versionedchanged admonitions | 15:44 |
| stephenfin | so if it's a change, it's not well documented | 15:45 |
| lyarwood | between 3 and 2? | 15:45 |
| lyarwood | ha | 15:45 |
| *** hamalq has joined #openstack-nova | 15:45 | |
| sean-k-mooney | stephenfin: ah ok | 15:45 |
| sean-k-mooney | ill go fix that so | 15:46 |
| stephenfin | it might be worth checking what you've done *definitely* works on Python 3 | 15:46 |
| stephenfin | if it does, I'd be game to submit a PR to the Python docs | 15:46 |
| lyarwood | stephenfin: I'm pretty sure we've logged stuff at ERROR in CI ;) | 15:46 |
| lyarwood | stephenfin: oh but you mean actually works | 15:46 |
| sean-k-mooney | https://docs.python.org/3.8/library/logging.html#logging.Filter | 15:47 |
| sean-k-mooney | "You don’t actually need to subclass Filter: you can pass any instance which has a filter method with the same semantics." | 15:47 |
| *** hamalq_ has joined #openstack-nova | 15:47 | |
| stephenfin | sean-k-mooney++ Good spot | 15:47 |
| stephenfin | python-docs++ | 15:47 |
| sean-k-mooney | i didnt see that was py3 only | 15:48 |
| sean-k-mooney | but its easy to fix | 15:48 |
| sean-k-mooney | that said i have dinner/breakfast cooking so ill do it in an hour or so | 15:48 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Handle VIR_ERR_DEVICE_MISSING when detaching devices https://review.opendev.org/741561 | 15:50 |
| lyarwood | ^ melwitt ; hopefully sorted now thanks again | 15:50 |
| *** hamalq has quit IRC | 15:50 | |
| melwitt | lyarwood: I didn't catch you with my latest comment but it's just a thought anyway | 15:52 |
| lyarwood | melwitt: yeah I'd like to do that in a FUP if possible while simplifying the logic | 15:53 |
| melwitt | k | 15:53 |
| *** derekh has quit IRC | 15:55 | |
| kashyap | melwitt: Thanks for tha close reading of the tests review. I wasn't as careful | 16:00 |
| stephenfin | neither was I :( | 16:00 |
| *** psachin has quit IRC | 16:00 | |
| melwitt | those tests are confusing 😝 | 16:02 |
| kashyap | melwitt: Yes! I bow to your eagle eyes. Sometimes, I give up in despair | 16:02 |
| *** k_mouza has quit IRC | 16:02 | |
| *** JamesBenson has quit IRC | 16:02 | |
| melwitt | not by fault of lyarwood but just the number of combos needed. I'm almost thinking ddt would have helped here | 16:03 |
| lyarwood | melwitt: this needs to go back to stable/queens, did we have ddt back then? | 16:03 |
| stephenfin | yup | 16:04 |
| stephenfin | we've had that since Grizzly or something, afaik | 16:04 |
| lyarwood | ah cool | 16:04 |
| lyarwood | kk well I can respin again if it makes it easier to read and update in the future | 16:05 |
| *** k_mouza has joined #openstack-nova | 16:05 | |
| *** READ10 has quit IRC | 16:06 | |
| *** martinkennelly has joined #openstack-nova | 16:06 | |
| melwitt | lyarwood: well, if you're refactoring and making those error catchings go away, then it doesn't matter [after this] right? but I'm afraid some combinations are still missing so brace yourself. I might be wrong though, somehow looking at these makes me really confused | 16:07 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Remove broken legacy zuul jobs https://review.opendev.org/741603 | 16:11 |
| *** mgariepy has quit IRC | 16:13 | |
| *** dtantsur is now known as dtantsur|afk | 16:13 | |
| *** k_mouza has quit IRC | 16:15 | |
| *** nightmare_unreal has quit IRC | 16:16 | |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Use better variable names, types for '_create_guest' https://review.opendev.org/740945 | 16:23 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP: Attempt to restore some sanity to snapshot https://review.opendev.org/740336 | 16:23 |
| dansmith | gah, stephenfin can you tell me what the magic oslo config reno linky thing is? | 16:30 |
| dansmith | my grepping is failing | 16:30 |
| dansmith | or is that docs only? | 16:30 |
| stephenfin | the sphinx roles? You can't use those in renos | 16:30 |
| stephenfin | just use ``literals`` | 16:31 |
| *** links has quit IRC | 16:31 | |
| dansmith | ack, that must be why I can't find it | 16:31 |
| stephenfin | it's :oslo.config:option:`group.opt` for stuff in doc/source | 16:32 |
| dansmith | yeah okay, I was only grepping the reno tree thinking I'd find examples in there | 16:33 |
| dansmith | but makes sense | 16:33 |
| *** JamesBenson has joined #openstack-nova | 16:47 | |
| *** markvoelker has joined #openstack-nova | 16:58 | |
| *** artom has quit IRC | 17:02 | |
| *** maciejjozefczyk has joined #openstack-nova | 17:04 | |
| *** mgariepy has joined #openstack-nova | 17:09 | |
| *** vishalmanchanda has quit IRC | 17:20 | |
| *** owalsh has quit IRC | 17:21 | |
| *** jsuchome has quit IRC | 17:30 | |
| openstackgerrit | sean mooney proposed openstack/nova stable/train: Silence amqp heartbeat warning https://review.opendev.org/728057 | 17:35 |
| sean-k-mooney | stephenfin: lyarwood that should be the python 2 compatiable way to do the filtering | 17:36 |
| sean-k-mooney | but we will see what the ci says | 17:36 |
| stephenfin | sean-k-mooney: Can you add a NOTE explaining what you've changed? | 17:37 |
| *** tesseract has quit IRC | 17:37 | |
| sean-k-mooney | sure one sec ill do it now | 17:37 |
| stephenfin | and normally I'd suggest a 'Changes:' section, but seeing as there's only one file it's probably not necessary :) | 17:37 |
| *** owalsh has joined #openstack-nova | 17:42 | |
| *** artom has joined #openstack-nova | 17:43 | |
| openstackgerrit | sean mooney proposed openstack/nova stable/train: Silence amqp heartbeat warning https://review.opendev.org/728057 | 17:44 |
| *** maciejjozefczyk has quit IRC | 17:44 | |
| artom | mordred, in my continuing sweep of sdk I stumbled upon https://review.opendev.org/#/c/741688/ - it's kinda touchy because we're breaking backwards compat (except not really?), otoh leaving it as is is overly restrictive too... | 17:45 |
| *** JamesBenson has quit IRC | 17:45 | |
| *** JamesBenson has joined #openstack-nova | 17:49 | |
| *** martinkennelly has quit IRC | 17:52 | |
| mnaser | sean-k-mooney: have a look at what moving to openvswitch firewall driver did => https://docs.google.com/spreadsheets/d/14AzjZ8SaA6e-WOJ_0BDYAKVHtUIEobhyFWOZQPhN9s0/edit#gid=1769100043 | 17:52 |
| mnaser | switch was done at around 15UTC | 17:52 |
| sean-k-mooney | it looks like all the long tails are gone | 17:54 |
| *** JamesBenson has quit IRC | 17:54 | |
| sean-k-mooney | well maybe not from 1600 yes | 17:55 |
| sean-k-mooney | i think there is still an improvement? | 17:56 |
| sean-k-mooney | but from the graph alone its hare to tell | 17:57 |
| mnaser | sean-k-mooney: yeah, it's still early but the results look much better so far | 17:58 |
| sean-k-mooney | mnaser: i think the median and mean times has droped and the make latencies are better ? is that correct | 17:58 |
| mnaser | yes, and also max time dropped (with noe or two odd ones) | 17:58 |
| sean-k-mooney | mnaser: in this config os-vif does not need to use privesep at all i belive if you are also using native | 17:58 |
| mnaser | sean-k-mooney: its using native but it still does because i still see a privsep process up | 17:58 |
| mnaser | but i cant imagine why it's needed | 17:59 |
| sean-k-mooney | that is proably novas | 17:59 |
| mnaser | sean-k-mooney: /openstack/venvs/nova-19.0.8/bin/python2 /openstack/venvs/nova-19.0.8/bin/privsep-helper --config-file /etc/nova/nova.conf --privsep_context vif_plug_ovs.privsep.vif_plug --privsep_sock_path /tmp/tmpD6Vj2G/privsep.sock | 17:59 |
| sean-k-mooney | if there is an os vif on i can check quickly | 17:59 |
| mnaser | maybe we should look into taht | 17:59 |
| sean-k-mooney | well my guess that privsep would not be need is just i dont htink we need to do most of the ops that need it in this config | 18:00 |
| sean-k-mooney | it actuly was never a goal | 18:00 |
| mnaser | sean-k-mooney: oh its used for a few funcs that we need for hybrid plug | 18:00 |
| sean-k-mooney | yes | 18:01 |
| mnaser | sean-k-mooney: i suspect because this machine has a mix of both now, it is why privsep-helper went up | 18:01 |
| sean-k-mooney | ah ok that would make sense ya | 18:01 |
| mnaser | if there is no hybrid ports plugged anymore and i restart nova i suspect it would disappear | 18:01 |
| sean-k-mooney | we need it to create he linux bridges | 18:01 |
| mnaser | ya i found those refs here https://github.com/openstack/os-vif/blob/master/vif_plug_ovs/linux_net.py | 18:01 |
| mnaser | if i find a box with only taps, ill try and restart and theoretically it should not go up | 18:02 |
| sean-k-mooney | yep its also need for ovs-vsctl to acess the ovs-db unix sockt but the python lib connect over tcp and does not need it | 18:02 |
| sean-k-mooney | i can check my home deployment | 18:03 |
| mnaser | yeah in my case im using tcp (which is also the default) | 18:03 |
| sean-k-mooney | yep tcp to localhost | 18:04 |
| sean-k-mooney | well 127.0.0.1 | 18:04 |
| sean-k-mooney | same thing | 18:04 |
| sean-k-mooney | root 48 0.0 0.0 3099668 61044 ? Sl Jun20 0:00 /var/lib/kolla/venv/bin/python /var/lib/kolla/venv/bin/privsep-helper --config-file /etc/nova/nova.conf --privsep_context vif_plug_ovs.privsep.vif_plug --privsep_sock_path /tmp/tmp1n9arrof/privsep.sock | 18:05 |
| sean-k-mooney | so i still have it but i suspec its basically unused | 18:05 |
| *** vdrok has quit IRC | 18:07 | |
| *** vdrok has joined #openstack-nova | 18:08 | |
| sean-k-mooney | ah | 18:09 |
| sean-k-mooney | mnaser: https://github.com/openstack/os-vif/blob/989f5a7d92b2459444e6956facb332f19e7c6537/vif_plug_ovs/ovsdb/ovsdb_lib.py#L50 | 18:10 |
| sean-k-mooney | we set the mtu | 18:10 |
| sean-k-mooney | for the bridge | 18:10 |
| *** JamesBenson has joined #openstack-nova | 18:11 | |
| mnaser | ah | 18:11 |
| sean-k-mooney | as part of create port https://github.com/openstack/os-vif/blob/989f5a7d92b2459444e6956facb332f19e7c6537/vif_plug_ovs/ovsdb/ovsdb_lib.py#L107 | 18:11 |
| sean-k-mooney | so ya that is the only privadge call which is a significat improvment | 18:11 |
| *** NobodyCam has quit IRC | 18:12 | |
| *** NobodyCam has joined #openstack-nova | 18:12 | |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: tests: Add helpers for suspend, resume and reboot of server https://review.opendev.org/741285 | 18:15 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Pass context, instance to '_create_domain' https://review.opendev.org/741286 | 18:15 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: api: Reject non-spawn operations for vTPM https://review.opendev.org/741500 | 18:15 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Add emulated TPM support to Nova https://review.opendev.org/631363 | 18:15 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Add docs for vTPM support https://review.opendev.org/739213 | 18:15 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP: Add support for resize and cold migration of emulated TPM files https://review.opendev.org/639934 | 18:15 |
| stephenfin | dansmith: I won't be doing this tonight, but you might be interested in https://review.opendev.org/#/c/639934/40//COMMIT_MSG@37 | 18:16 |
| dansmith | omg | 18:16 |
| dansmith | scary | 18:16 |
| dansmith | I | 18:16 |
| stephenfin | tl;dr: I need access at the virt driver layer to both the old and new flavor when confirming/reverting a resize, and I don't have that at the moment because we're zeroing out those attributes before said calls to virt driver | 18:17 |
| dansmith | am covered up in other stuff (some of which I broke) but .. yeah maybe I can look next week | 18:17 |
| dansmith | yeah | 18:17 |
| stephenfin | I'm planning to just delay that zeroing out 'til after, but I don't know if there'll be knock on effects or not | 18:17 |
| *** bbowen has quit IRC | 18:17 | |
| dansmith | yeah | 18:17 |
| stephenfin | seeing as your name is beside all the zeroing out, I figure you're the person to ask :) | 18:18 |
| stephenfin | next week is good though | 18:18 |
| sean-k-mooney | stephenfin: do you want me to take a look at that too or should i start from the begining of the seriese | 18:18 |
| stephenfin | sean-k-mooney: If you can follow the flow of that code, sure :) | 18:19 |
| stephenfin | You can do earlier but this is the only bit I personally consider somewhat tricky now. The rest of the series is mostly wiring up stuff | 18:20 |
| sean-k-mooney | well i have not looke at the code in about 2 months so that is why i was hesitent to jump into the end of the series | 18:20 |
| sean-k-mooney | i think i need to start form the start to understand all the dettails | 18:20 |
| stephenfin | Sure, whatever works. It's big. You can ignore the functional test cleanup patches, obv | 18:21 |
| *** TheJulia has quit IRC | 18:22 | |
| *** TheJulia has joined #openstack-nova | 18:23 | |
| *** johnsom has quit IRC | 18:24 | |
| *** johnsom has joined #openstack-nova | 18:24 | |
| dansmith | stephenfin: just skimming it looks like a lot is missing from that patch you linked | 18:25 |
| dansmith | like, it doesn't seem to do much of what the tin describes | 18:25 |
| stephenfin | \o/ | 18:25 |
| dansmith | like, it removes some exceptions and adds some privsep things that don't otherwise get used | 18:25 |
| dansmith | just wondering if you forgot to git add or something | 18:25 |
| stephenfin | maybe I did | 18:26 |
| stephenfin | checking | 18:26 |
| dansmith | ohh, wait, sorry | 18:26 |
| dansmith | the sorted-last v-for-virt stuff was getting hidden under tests | 18:26 |
| dansmith | sorry | 18:26 |
| stephenfin | ah, yes | 18:26 |
| stephenfin | this is all libvirt-specific so nothing happens higher than that | 18:27 |
| sean-k-mooney | stephenfin: oh am ill respin the backport after ci completes or monday | 18:28 |
| sean-k-mooney | but ya i can just move the not to the commit and list the updated file | 18:28 |
| stephenfin | yup, that ought to do the trick | 18:28 |
| mnaser | sean-k-mooney: could we not use libvirt to set mtu and avoid that privsep call? | 18:29 |
| mnaser | or maybe we already use it? https://www.irccloud.com/pastebin/ho53ZFfZ/ | 18:30 |
| mnaser | so that mtu set is.. probably not super useful when we're not doing a hybrid plug | 18:30 |
| sean-k-mooney | we do but its needed when using hybrid plug | 18:31 |
| sean-k-mooney | i dont think we need to do it for non hybrid plug | 18:31 |
| sean-k-mooney | mnaser: that said i really want to stop libvirt doing anything related to networking | 18:32 |
| mnaser | ok so architecturally we want os_vif to do the work | 18:32 |
| sean-k-mooney | mnaser: it casues an unavoidble race | 18:32 |
| sean-k-mooney | yes | 18:32 |
| mnaser | that's a fair point, so mtu stays then i guess | 18:32 |
| sean-k-mooney | https://review.opendev.org/#/c/602432/ | 18:33 |
| sean-k-mooney | your familar with https://bugs.launchpad.net/neutron/+bug/1734320 and https://bugs.launchpad.net/neutron/+bug/1815989 | 18:33 |
| openstack | Launchpad bug 1734320 in OpenStack Compute (nova) "Eavesdropping private traffic" [Undecided,In progress] - Assigned to sean mooney (sean-k-mooney) | 18:33 |
| openstack | Launchpad bug 1815989 in OpenStack Compute (nova) "OVS drops RARP packets by QEMU upon live-migration causes up to 40s ping pause in Rocky" [Medium,In progress] | 18:33 |
| mnaser | ah yes | 18:33 |
| mnaser | so our lack of control when tap goes up | 18:33 |
| sean-k-mooney | what happens today is os-vif set up the port and the l2 agent start to wire it up | 18:34 |
| sean-k-mooney | then libvirt deletes an recreates it | 18:34 |
| sean-k-mooney | that is only a problem when using contrack firewall | 18:35 |
| mnaser | oh yikes, i thought it was os-vif that was wiring up what libvirt creates (but then that's just a race in another direction too) | 18:35 |
| sean-k-mooney | noop | 18:35 |
| sean-k-mooney | libvirt basicaly does if exist delete and recreate | 18:35 |
| sean-k-mooney | undoing all our careful external event stuff | 18:36 |
| sean-k-mooney | for hybrid plug/iptables-firewall, ovs-dpdk and hardware offload | 18:36 |
| sean-k-mooney | os-vif and neutron work togeter | 18:37 |
| sean-k-mooney | but when you use libvirt in the non hybridg plug case it just breaks everything | 18:37 |
| sean-k-mooney | i really wish i or someone lese had time to finish off https://review.opendev.org/#/c/640258 | 18:38 |
| sean-k-mooney | but i dont have time to work on that at the moement | 18:39 |
| mnaser | i share that sentiment :) | 18:40 |
| sean-k-mooney | the main proable is while i got it work 15 months ago i dont follow neutron dev that closely anymore so i would basicaly have to start from scrat to figure out why its not working now | 18:42 |
| *** masayukig has quit IRC | 18:58 | |
| *** masayukig has joined #openstack-nova | 18:59 | |
| *** owalsh has quit IRC | 19:20 | |
| *** owalsh has joined #openstack-nova | 19:29 | |
| *** damien_r has quit IRC | 19:32 | |
| *** mloza has quit IRC | 19:35 | |
| *** mloza has joined #openstack-nova | 19:36 | |
| *** mriedem has joined #openstack-nova | 19:39 | |
| *** mriedem is now known as cremefraiche | 19:40 | |
| *** ralonsoh has quit IRC | 19:53 | |
| *** ralonsoh has joined #openstack-nova | 19:53 | |
| *** johnthetubaguy has quit IRC | 19:56 | |
| *** johnthetubaguy has joined #openstack-nova | 19:58 | |
| *** hemna has quit IRC | 19:59 | |
| *** hemna has joined #openstack-nova | 19:59 | |
| *** bbowen has joined #openstack-nova | 20:00 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add test coverage of baremetal nodes policies https://review.opendev.org/741707 | 20:01 |
| *** johnthetubaguy has quit IRC | 20:05 | |
| *** johnthetubaguy has joined #openstack-nova | 20:08 | |
| *** johnthetubaguy has quit IRC | 20:14 | |
| *** johnthetubaguy has joined #openstack-nova | 20:17 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Introduce scope_types in baremetal node https://review.opendev.org/741709 | 20:23 |
| *** mloza has quit IRC | 20:27 | |
| *** johnthetubaguy has quit IRC | 20:32 | |
| *** sapd1_x has quit IRC | 20:32 | |
| *** sapd1_x has joined #openstack-nova | 20:33 | |
| *** johnthetubaguy has joined #openstack-nova | 20:35 | |
| openstackgerrit | Alex Deiter proposed openstack/nova master: Detach is broken for multi-attached fs-based volumes https://review.opendev.org/741712 | 20:41 |
| *** markvoelker has quit IRC | 20:42 | |
| *** johnthetubaguy has quit IRC | 20:44 | |
| *** johnthetubaguy has joined #openstack-nova | 20:48 | |
| openstackgerrit | Merged openstack/nova stable/ussuri: zuul: remove legacy-tempest-dsvm-neutron-dvr-multinode-full https://review.opendev.org/741590 | 20:59 |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add new default roles in baremetal ndoes policies https://review.opendev.org/741716 | 21:06 |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Pass the actual target in baremetal nodes policy https://review.opendev.org/741717 | 21:09 |
| *** raildo has quit IRC | 21:11 | |
| *** rpittau has quit IRC | 21:12 | |
| *** rpittau has joined #openstack-nova | 21:13 | |
| openstackgerrit | Luigi Toscano proposed openstack/nova stable/train: zuul: remove legacy-tempest-dsvm-neutron-dvr-multinode-full https://review.opendev.org/741720 | 21:18 |
| *** hamalq has joined #openstack-nova | 21:28 | |
| *** hamalq_ has quit IRC | 21:31 | |
| *** gyee has quit IRC | 21:40 | |
| *** gyee has joined #openstack-nova | 21:45 | |
| *** melwitt is now known as jgwentworth | 21:49 | |
| *** lbragstad has quit IRC | 21:56 | |
| openstackgerrit | Merged openstack/nova stable/pike: Reproduce bug 1862633 https://review.opendev.org/740634 | 21:56 |
| openstack | bug 1862633 in OpenStack Compute (nova) "unshelve leak allocation if update port fails" [Medium,Fix released] https://launchpad.net/bugs/1862633 - Assigned to Balazs Gibizer (balazs-gibizer) | 21:56 |
| *** lbragstad has joined #openstack-nova | 21:56 | |
| *** ralonsoh has quit IRC | 22:11 | |
| *** k_mouza has joined #openstack-nova | 22:17 | |
| *** k_mouza has quit IRC | 22:22 | |
| *** markvoelker has joined #openstack-nova | 22:22 | |
| *** markvoelker has quit IRC | 22:27 | |
| *** cremefraiche has left #openstack-nova | 22:33 | |
| *** ociuhandu has joined #openstack-nova | 22:39 | |
| openstackgerrit | Alex Deiter proposed openstack/nova master: Detach is broken for multi-attached fs-based volumes https://review.opendev.org/741712 | 22:43 |
| *** ociuhandu has quit IRC | 22:44 | |
| *** markvoelker has joined #openstack-nova | 22:47 | |
| *** dklyle has quit IRC | 22:50 | |
| *** markvoelker has quit IRC | 22:52 | |
| *** tosky_ has joined #openstack-nova | 22:52 | |
| *** tosky has quit IRC | 22:53 | |
| *** mlavalle has quit IRC | 22:59 | |
| *** gyee has quit IRC | 23:01 | |
| *** tosky_ is now known as tosky | 23:01 | |
| *** dklyle has joined #openstack-nova | 23:05 | |
| *** tosky has quit IRC | 23:38 | |
| *** hamalq has quit IRC | 23:38 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
