| *** bhagyashris is now known as bhagyashris|rover | 04:43 | |
| *** abhishekk is now known as akekane|home | 04:50 | |
| *** akekane|home is now known as abhishekk | 04:50 | |
| bauzas | good morning Nova | 07:45 |
|---|---|---|
| opendevreview | Dmitriy Rabotyagov proposed openstack/nova master: Ensure MAC addresses characters are in the same case https://review.opendev.org/c/openstack/nova/+/811947 | 07:52 |
| plibeau4 | hello, I need your help about this proposal change: https://review.opendev.org/c/openstack/nova/+/803713. | 08:24 |
| gibi | good morning | 08:50 |
| gibi | this week feels to be as crazy as the last was for me. you can ping me but I cannot promise things :/ | 08:51 |
| bauzas | plibeau4: ack, will add your change for a review | 08:55 |
| bauzas | thanks for reaching us | 08:56 |
| plibeau4 | thx guys take your time it's just to have your feedback It's already in production on myside :) | 09:24 |
| plibeau4 | and if you have time -> https://review.opendev.org/c/openstack/nova/+/803600 | 09:25 |
| sean-k-mooney | to me https://review.opendev.org/c/openstack/nova/+/803713 kind of feels like intoducing a race condetion | 09:26 |
| sean-k-mooney | i guess its not a race as it does not quiesce the disk until after the dev rebase is complete deterministically | 09:29 |
| sean-k-mooney | plibeau4: im not conviced moving the quiece is safe to have in production | 09:43 |
| sean-k-mooney | plibeau4: i think it can lead to data currportion if we are not very careful | 09:44 |
| gibi | sean-k-mooney: re: waiting for vif-plugged during hard reboot. What is the piece of information in the port that can be used to decide if the networking backend sends plug event at plug time? | 09:50 |
| sean-k-mooney | gibi: https://specs.openstack.org/openstack/neutron-specs/specs/train/port-binding-extended-information.html | 09:53 |
| sean-k-mooney | binding_details: { | 09:53 |
| sean-k-mooney | ... | 09:53 |
| sean-k-mooney | "connectivity": "l2", | 09:53 |
| gibi | thanks | 09:53 |
| sean-k-mooney | "bound_drivers": {"0": "openvswtich"} | 09:53 |
| sean-k-mooney | } | 09:53 |
| sean-k-mooney | gibi: we could use the bound_drivers to infer this | 09:54 |
| sean-k-mooney | the spec orginaly had an events section that told use when it sent events | 09:54 |
| sean-k-mooney | that was unfrotunetly removed because we could not agree on how to model that | 09:54 |
| sean-k-mooney | so we would have to create a lookup table in nova and map the driver name to it | 09:55 |
| sean-k-mooney | opendaylight was bind time, opendaylight_v2 is plug time | 09:55 |
| gibi | ack. I might need to create a solution on Pike level downstream hence my continued digging in it | 09:56 |
| sean-k-mooney | the conneectivy field is waht we need to use to know if we can safely allow unadress port too | 09:56 |
| gibi | but if the infor is in train then I cannot use that in Pike | 09:56 |
| sean-k-mooney | if its l2 it does not need an ip | 09:56 |
| sean-k-mooney | gibi: i think a workaround config option to force_rebind_on_reboot is the way to go and backport that | 09:57 |
| gibi | yeah I thinking about a config too | 09:57 |
| sean-k-mooney | there are reasons to have that on master by the way | 09:58 |
| sean-k-mooney | combidnind with unpluging on reboot | 09:58 |
| sean-k-mooney | basically if you make reboot: power off-> unplug-> unbind->rebind->plug->spwan | 09:58 |
| sean-k-mooney | then you can do things like change the firewall driver or maybe even ml2/backend | 09:59 |
| gibi | why do you want to add bind to the picture? | 09:59 |
| gibi | ahh I see | 09:59 |
| sean-k-mooney | currently you have to live migrate | 09:59 |
| gibi | to allow change thing that needs a binding | 09:59 |
| sean-k-mooney | but we have some customer saying they want to do it inplace and are ok with stoping the vm | 09:59 |
| sean-k-mooney | yes | 09:59 |
| sean-k-mooney | basicaly some large customes have said to use we cant live migrate every vm we have but we can stop them cause we run everything in duplex mode and can go simplex for a period of time without issue | 10:00 |
| gibi | OK so there is two parallel direction to pursue. 1) add a workaround flag for something like wait_for_plug_event_at_reboot and 2) force_rebind_on_reboot to allow in place update of binding related data | 10:01 |
| sean-k-mooney | so they are fine with stop(tearing down all networking) and start(setting up all networking) to "migrate" form one firewall driver to another (iptables->ovs) or between mech drvers(ml2/ovs->,l2/ovn) | 10:02 |
| sean-k-mooney | gibi: am yes i guess so | 10:02 |
| gibi | the problem (most probably) on my side is that they do VM reboots in big batches due to upgrade and nova sends a lot of unplug - plug event to ODL which fails to finish setting up OVS before the VM starts requesting IP via DHCP | 10:02 |
| gibi | so if they can set a WA flag in nova that nova should wait for the network-vif-plugged before letting the VM run at reboot would help to control the situation | 10:04 |
| sean-k-mooney | yep | 10:04 |
| gibi | and they need it on Pike :/ | 10:04 |
| gibi | I will go with a backportable WA flag | 10:05 |
| sean-k-mooney | ya so on master i think we need to look at how and when we use events and for pike workaround flag is definetly the simplest thing as only the custoemr that is affect can opt in by settting it minimising the risk to others that dont need the change | 10:07 |
| sean-k-mooney | gibi: i had planned to bring up ovn live migration at the ptg(which is affected by the vif-plugged-events) but i might just make tha ta general topic since i think the ovn issue will require ovn changes to fully resolve | 10:09 |
| jpic | hi all, i don't understand what is supposed to keep the websocket between the spice html5 client and the spice server? browsers will kill inactive websockets after 60 secs of inactivity, so i suppose the server is supposed to send ping packets regularily | 10:19 |
| gibi | sean-k-mooney: ack | 10:40 |
| gibi | sean-k-mooney: one more thing, I don't see "bound_drivers" ever got implemented in neutron. Only neturon-lib has the field definition https://review.opendev.org/c/openstack/neutron-lib/+/645288/6/neutron_lib/api/definitions/portbindings.py#55 but neutron does not use that | 10:47 |
| gibi | I only see the connectivity field being implemented | 10:47 |
| gibi | in https://review.opendev.org/c/openstack/neutron/+/645645/ | 10:47 |
| sean-k-mooney | i see... am i think it was ment to be implemented at the ml2 plugin level above the drivers but you might be right | 11:05 |
| sean-k-mooney | https://review.opendev.org/c/openstack/neutron-lib/+/645288/6/neutron_lib/api/definitions/portbindings.py added bound_drivers to the extension | 11:06 |
| sean-k-mooney | but im not sure if they have used it | 11:06 |
| sean-k-mooney | looks like its only in neutron-lib https://codesearch.opendev.org/?q=VIF_DETAILS_BOUND_DRIVERS&i=nope&literal=nope&files=&excludeFiles=&repos= | 11:08 |
| gibi | yeah that matches with my grep results | 11:08 |
| sean-k-mooney | so ya looks like we would need to actully implement that on master before we can use it | 11:09 |
| gibi | yepp, it looks like | 11:09 |
| sean-k-mooney | i feel less bad for not using it yet | 11:09 |
| gibi | :) | 11:10 |
| sean-k-mooney | has anyone else had devstack time out on pip resolver issues | 12:15 |
| sean-k-mooney | Collecting oslo.policy>=3.8.1 | 12:22 |
| sean-k-mooney | Using cached oslo.policy-3.8.2-py3-none-any.whl (86 kB) | 12:22 |
| sean-k-mooney | INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking | 12:22 |
| gibi | sean-k-mooney: I haven't seen that pip issue yet | 12:40 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Add a WA flag waiting for vif-plugged event during reboot https://review.opendev.org/c/openstack/nova/+/813419 | 12:47 |
| gibi | sean-k-mooney: ^^ this is my first try on the hard reboot - vif plug issue | 12:47 |
| sean-k-mooney | i think its cause by cinder's dep on oslo.vmware which is pulling in suds-jurko | 12:48 |
| sean-k-mooney | which is an unmaintained soap lib fork | 12:49 |
| sean-k-mooney | gibi: that is just the events right are you working on a second patch for rebinding | 12:50 |
| sean-k-mooney | i guess you might not need that in your case | 12:51 |
| gibi | sean-k-mooney: it is jut the event waiting | 12:51 |
| sean-k-mooney | was pikes version of netwokring-odl new enough to send plug time events | 12:51 |
| gibi | sean-k-mooney: yes, the plug time event patch was merged in pike | 12:51 |
| sean-k-mooney | ah ok cool | 12:51 |
| gibi | https://review.opendev.org/c/openstack/networking-odl/+/465463 | 12:52 |
| gibi | this is the patch | 12:52 |
| sean-k-mooney | do you have a DNM patch that sets that to true or set it in the job so we can see it work | 12:52 |
| gibi | sean-k-mooney: good point I will add that | 12:52 |
| gibi | sean-k-mooney: ml2/ovs works for me with this locally | 12:52 |
| gibi | sean-k-mooney: so we can enable this WA in our normal ovs jobs | 12:53 |
| sean-k-mooney | cool, ya currently its disabled so we wont see if this breaks anything although it will break ovn actully | 12:53 |
| sean-k-mooney | ya we could enable it in nova-next | 12:53 |
| gibi | I think I will enable this flaf in nova-next | 12:53 |
| gibi | exactly :D | 12:53 |
| sean-k-mooney | yep also we might want to renable a multinode jobs with ml2/ovs at some point | 12:53 |
| sean-k-mooney | im not sure what coverage we have/have lost with the ovn default swap | 12:54 |
| sean-k-mooney | i think nova-next might be the only job that is still using ml2/ovs (other then linuxbrige job) in our gate today | 12:55 |
| sean-k-mooney | i dont think we have a multi node job with a backend that send plugtime events currently | 12:55 |
| gibi | sean-k-mooney: nova-next is multinode afaik | 12:56 |
| gibi | name: nova-next | 12:57 |
| gibi | parent: tempest-multinode-full-py3 | 12:57 |
| gibi | yepp it is multinode | 12:57 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Add a WA flag waiting for vif-plugged event during reboot https://review.opendev.org/c/openstack/nova/+/813419 | 12:58 |
| sean-k-mooney | gibi: oh ok cool | 13:05 |
| gibi | bauzas: fyi, I now remember that there will by TC + PTL discussion slot on the PTG. https://etherpad.opendev.org/p/tc-yoga-ptg L15 (Monday 15 UTC) | 13:20 |
| gibi | bauzas: https://etherpad.opendev.org/p/tc-ptl-interaction | 13:20 |
| gibi | I will join too | 13:20 |
| gibi | just to listen in | 13:20 |
| gibi | or fill in info from the past if needed | 13:20 |
| bauzas | gibi: sorry was off | 13:44 |
| bauzas | gibi: ack, I was wondering when the slot was there | 13:44 |
| opendevreview | Balazs Gibizer proposed openstack/nova stable/pike: Add a WA flag waiting for vif-plugged event during reboot https://review.opendev.org/c/openstack/nova/+/813437 | 14:02 |
| opendevreview | norman shen proposed openstack/nova master: Add a timeout for waiting volume snapshot ready https://review.opendev.org/c/openstack/nova/+/813443 | 14:26 |
| dansmith | gibi: wow, pike huh? :) | 15:13 |
| gibi | dansmith: last week I was pulled into a discussion upgrading Mitaka deployments to Victoria. Pike is easy compared to that | 15:13 |
| dansmith | hah | 15:14 |
| gibi | telcos are slooow | 15:14 |
| dansmith | :) | 15:14 |
| melwitt | bauzas: these backports are finally ready for review after the stable/train gate fixes https://review.opendev.org/c/openstack/nova/+/806629 and the patch under it, if you could take a look | 15:24 |
| bauzas | melwitt: ack, opening a tab and looking at them once we discuss about other issues ;) | 15:25 |
| melwitt | thanks :) | 15:25 |
| opendevreview | Elod Illes proposed openstack/nova stable/stein: [stable-only] Pin virtualenv and setuptools https://review.opendev.org/c/openstack/nova/+/813451 | 15:35 |
| opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: [yoga] Add PCI VPD Capability Handling https://review.opendev.org/c/openstack/nova/+/808199 | 17:31 |
| opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: [yoga] Support remote-managed SmartNIC DPU ports https://review.opendev.org/c/openstack/nova/+/812111 | 17:31 |
| mdbooth | Not really an OpenStack issue, but as I'm hitting it running devstack on Ubuntu Focal I'm hoping somebody here has seen it before 🤞 c-vol fails to come up because lvcreate fails to create a thin volume due to: "modprobe: FATAL: Module dm-thin-pool not found in directory /lib/modules/5.4.0-1047-kvm". Anybody seen that? | 19:39 |
| mdbooth | My google-fu is unfortunately not up to the task. | 19:41 |
| clarkb | mdbooth: you might have to install something like linux-image-extra-virtual but for the kvm kernel | 19:57 |
| mdbooth | clarkb: Thanks. I was just coming to the realisation that this image might have a stripped down kernel for kvm in it. I think I'm going to toss it and try a different cloud image. | 19:58 |
| clarkb | linux-image-virtual doesn't have that module either fwiw, but they do ship linux-image-extra-virtual which I expect includes it | 19:59 |
| mdbooth | Thanks! | 20:00 |
| clarkb | I don't see a virtual package for extra kvm image kernel modules but there are package for specific kernel versions like linux-modules-5.4.0-1009-kvm | 20:00 |
| clarkb | I did check the kernel config for the linux-image-virtual kernel I've got and it lists DM_THIN_PROVISIONING=m so there should be a module for it in a package somewhere | 20:01 |
| mdbooth | Looking good: # find /usr/lib/modules/5.4.0-88-generic/ -name "*thin*" | 20:21 |
| mdbooth | /usr/lib/modules/5.4.0-88-generic/kernel/drivers/md/dm-thin-pool.ko | 20:21 |
| mdbooth | clarkb: Thanks again :) | 20:21 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!