Thursday, 2020-10-01

*** k_mouza has joined #openstack-nova00:00
*** k_mouza has quit IRC00:05
*** _erlon_ has quit IRC00:06
*** k_mouza has joined #openstack-nova00:16
*** k_mouza has quit IRC00:21
*** k_mouza has joined #openstack-nova00:31
*** k_mouza has quit IRC00:36
*** k_mouza has joined #openstack-nova00:56
*** k_mouza has quit IRC01:00
*** dklyle has quit IRC01:03
*** hamalq_ has quit IRC01:21
*** gyee has quit IRC01:26
*** rcernin has quit IRC01:28
*** sapd1 has joined #openstack-nova01:34
*** rcernin has joined #openstack-nova01:43
*** rcernin has quit IRC01:43
*** rcernin has joined #openstack-nova01:44
*** kaisers has quit IRC01:45
*** sapd1 has quit IRC02:02
*** brinzhang_ has joined #openstack-nova02:31
*** brinzhang0 has quit IRC02:34
*** songwenping_ has joined #openstack-nova02:35
*** songwenping__ has quit IRC02:38
*** freenzyfriday has joined #openstack-nova03:10
*** freenzyfriday has quit IRC03:15
*** sapd1 has joined #openstack-nova03:22
*** psachin has joined #openstack-nova03:35
*** songwenping__ has joined #openstack-nova03:55
*** mkrai has joined #openstack-nova03:57
*** songwenping_ has quit IRC03:58
*** links has joined #openstack-nova04:24
*** evrardjp has quit IRC04:33
*** evrardjp has joined #openstack-nova04:33
*** ratailor has joined #openstack-nova05:04
*** ratailor_ has joined #openstack-nova05:10
*** ratailor has quit IRC05:12
*** brinzhang0 has joined #openstack-nova05:14
*** brinzhang_ has quit IRC05:17
*** harsha24 has joined #openstack-nova05:42
*** bhagyashri is now known as bhagyashris06:02
*** brinzhang0 has quit IRC06:15
*** ralonsoh has joined #openstack-nova06:25
*** sean-k-mooney1 has joined #openstack-nova06:26
*** sean-k-mooney has quit IRC06:29
*** slaweq has joined #openstack-nova06:31
*** jhesketh has quit IRC06:35
*** maciejjozefczyk_ has joined #openstack-nova06:38
*** maciejjozefczyk has joined #openstack-nova06:44
*** maciejjozefczyk_ has quit IRC06:44
*** xek has joined #openstack-nova06:45
*** sapd1 has quit IRC06:49
*** haleyb has quit IRC06:54
*** jmlowe has quit IRC06:54
*** ganso has quit IRC06:54
*** ralonsoh has quit IRC06:57
*** hamalq has joined #openstack-nova06:57
*** ralonsoh has joined #openstack-nova06:57
*** tesseract has joined #openstack-nova07:01
*** harsha24 has quit IRC07:08
*** jhesketh has joined #openstack-nova07:10
*** zigo has joined #openstack-nova07:12
*** ratailor_ has quit IRC07:15
*** sapd1 has joined #openstack-nova07:17
*** mkrai has quit IRC07:17
*** rcernin has quit IRC07:19
*** mkrai has joined #openstack-nova07:20
*** xek has quit IRC07:32
*** tosky has joined #openstack-nova07:39
*** mkrai has quit IRC07:46
*** mkrai has joined #openstack-nova07:58
*** mkrai has quit IRC08:03
*** mkrai_ has joined #openstack-nova08:03
hemanth_nHi folks, is this time the unofficial office hours slot , if so want to discuss on the following review https://review.opendev.org/#/c/749175/08:05
*** haleyb has joined #openstack-nova08:07
*** jmlowe has joined #openstack-nova08:07
*** ganso has joined #openstack-nova08:07
lyarwoodhemanth_n: I don't think we have any office hours at the moment but looking at the change gibi / sean-k-mooney1 / stephenfin might be able to help once they are online08:11
lyarwoodIn other news, stable/victoria is blocked until we switch the nova-live-migration job out :/08:11
lyarwood++ /opt/stack/new/devstack-gate/devstack-vm-gate.sh:setup_localrc:L316:   /usr/bin/python3 roles/test-matrix/library/test_matrix.py -n -b stable/victoria -f roles/test-matrix/files/features.yaml -r primary08:11
lyarwood2020-09-30 21:04:43,670 ERROR: branch not allowed by features matrix: victoria08:11
hemanth_nthanks lyardwood08:12
hemanth_nlyarwood* thank you08:13
lyarwoodnp08:13
gibihemanth_n: that patch is on my review list for long08:13
gibihemanth_n: I see that sean-k-mooney1 is +1 on it already so I will try to read it today08:14
gibihemanth_n: is there any special aspect I should be aware of?08:15
hemanth_ngibi: cool thanks .. nope except its a specific scenario which is mentioned in the commit message and bug description08:15
gibihemanth_n: I saw that you managed to add a functional test for it. thanks that helps a lot08:17
openstackgerritLee Yarwood proposed openstack/nova stable/victoria: [stable-only] nova-live-migration: Disable libvirtd during negative tests  https://review.opendev.org/75468808:17
*** hamalq has quit IRC08:18
*** ratailor has joined #openstack-nova08:20
fricklerlyarwood: the right solution would be to move that job away from using ds-gate, I'm not sure if someone has started doing that yet. even if we keep ds-gate working for victoria, that should be considered unstable and very much deprecated08:25
lyarwoodfrickler: yeah I'm working on that in master, just need sort out a multinode ceph job and we should be good to backport to stable/victoria08:25
lyarwoodfrickler: and agreed, this is just for a few days hopefully08:26
lyarwoodgibi: are you aware of https://zuul.opendev.org/t/openstack/builds?job_name=nova-next - looks like gate/post_test_hook.sh is failing when it calls `openstack port unset --binding-profile allocation port-normal-qos`08:27
lyarwoodon master at least08:27
gibilyarwood: thanks for the ping08:28
gibiI haven't seen it yet08:28
lyarwoodgibi: np, can't see anything obvious in nova master that could be causing it, I assume something in Neutron has changed?08:29
gibilyarwood: I will ping lajoskatona on Neutron side, he did some recent changes around qos08:32
lyarwoodawesome thanks08:33
openstackgerritLee Yarwood proposed openstack/nova stable/victoria: [stable-only] nova-live-migration: Disable libvirtd during negative tests  https://review.opendev.org/75468808:35
*** derekh has joined #openstack-nova08:42
*** mkrai_ has quit IRC08:45
*** mkrai__ has joined #openstack-nova08:45
*** ratailor has quit IRC08:46
*** tkajinam has quit IRC08:47
*** salmankhan has joined #openstack-nova08:48
*** ratailor has joined #openstack-nova08:52
*** haleyb has quit IRC08:52
*** jmlowe has quit IRC08:52
*** ganso has quit IRC08:52
*** salmankhan has quit IRC08:59
*** hamalq has joined #openstack-nova09:00
*** salmankhan has joined #openstack-nova09:00
*** ratailor has quit IRC09:02
*** ratailor has joined #openstack-nova09:02
*** mkrai__ has quit IRC09:03
*** hamalq has quit IRC09:04
*** jraju__ has joined #openstack-nova09:05
*** links has quit IRC09:06
openstackgerritLee Yarwood proposed openstack/nova master: WIP: nova-next: Start testing the 'q35' machine type  https://review.opendev.org/70870109:10
*** haleyb has joined #openstack-nova09:10
*** jmlowe has joined #openstack-nova09:10
*** ganso has joined #openstack-nova09:10
*** sapd1 has quit IRC09:20
*** sapd1 has joined #openstack-nova09:21
*** mkrai has joined #openstack-nova09:23
*** ociuhandu has joined #openstack-nova09:34
*** priteau has joined #openstack-nova09:48
*** k_mouza has joined #openstack-nova09:51
*** k_mouza has quit IRC09:55
*** k_mouza has joined #openstack-nova09:59
*** ratailor has quit IRC10:04
*** k_mouza has quit IRC10:04
*** hamalq has joined #openstack-nova10:05
*** ratailor has joined #openstack-nova10:07
*** k_mouza has joined #openstack-nova10:08
*** ratailor has quit IRC10:09
*** hamalq has quit IRC10:09
*** ratailor has joined #openstack-nova10:14
*** ratailor has quit IRC10:15
*** ratailor has joined #openstack-nova10:15
stephenfingibi: Think you could spin through https://review.opendev.org/#/q/topic:remove-xenapi+status:open before the week is out?10:17
*** sapd1 has quit IRC10:20
*** rcernin has joined #openstack-nova10:20
*** smcginnis has quit IRC10:22
*** smcginnis has joined #openstack-nova10:22
gibistephenfin: I'm not comfortable making big changes to master during release time, can we take this after the release?10:24
stephenfinoh, of course10:25
*** rcernin has quit IRC10:26
*** rcernin has joined #openstack-nova10:26
openstackgerritBalazs Gibizer proposed openstack/nova-specs master: Support interface attach with qos ports  https://review.opendev.org/75547710:30
*** xek has joined #openstack-nova10:43
*** lyarwood has quit IRC10:50
*** lyarwood has joined #openstack-nova10:50
*** sapd1 has joined #openstack-nova10:59
openstackgerritStephen Finucane proposed openstack/nova-specs master: Add modernize-os-hypervisors-api spec  https://review.opendev.org/75510911:13
gibilyarwood: it seem neutron-lib needs a new release to contain this: https://review.opendev.org/#/c/753230/11:18
lyarwoodgibi: ack thanks for looking into that!11:18
lyarwoodgibi: ./me wonders if neutron have nova-next in their check/gate?11:19
gibithey don't I guess11:19
lyarwoodyeah might be something to suggest given the QoS coverage it has11:19
gibigood point11:22
gibito be fair the normal qos coverage is in tempest. the nova-next only has the coverage for the qos part of the nova-manage placement heal_allocation CLI11:22
lyarwoodah so we could add something to tempest to also cover this?11:23
gibiI think it was added to the nova-next job as it has to create an invalid situation first then heal it11:26
gibiin a real word missing port allocation only happen during upgrade from older openstack version11:27
gibiwe cannot really simulate that11:27
gibiso we instead deleteing the allocation manually11:27
gibithen try to heal it11:27
lyarwoodack11:29
lyarwoodstill would be nice if a neutron job could execute the test somewhere in their gate11:30
lyarwoodif we could break the test out into a seperate script somewhere11:30
gibiyeah, I agree with that11:30
*** xek has quit IRC11:30
*** xek has joined #openstack-nova11:34
gibilyarwood: I guess the whole nova gate is broken now due to this11:37
*** mkrai has quit IRC11:42
*** ratailor has quit IRC11:42
gibilyarwood: I'll try to disable this part of the nova-next job as it will take time to get a new neutron-lib release11:43
*** ratailor has joined #openstack-nova11:43
*** ratailor has quit IRC11:44
*** ratailor has joined #openstack-nova11:45
*** maciejjozefczyk has quit IRC11:47
*** ratailor has quit IRC11:49
gibilyarwood: also stable/victoria could have the same issue if https://review.opendev.org/#/c/755180 merges11:55
*** k_mouza has quit IRC12:01
openstackgerritBalazs Gibizer proposed openstack/nova master: Temporarily disable parts of heal port allocation test  https://review.opendev.org/75549812:05
gibilyarwood: ^^12:05
*** hamalq has joined #openstack-nova12:06
*** k_mouza has joined #openstack-nova12:07
*** mkrai has joined #openstack-nova12:07
openstackgerritBalazs Gibizer proposed openstack/nova master: Temporarily disable parts of heal port allocation test  https://review.opendev.org/75549812:07
*** hamalq has quit IRC12:10
*** k_mouza has quit IRC12:12
openstackgerritBalazs Gibizer proposed openstack/nova master: Temporarily disable parts of heal port allocation test  https://review.opendev.org/75549812:15
lyarwoodgibi: ack thanks12:15
gibislaweq and lajoskatona are aware of the issue and working on a solution12:16
*** raildo has joined #openstack-nova12:16
lyarwoodawesome thanks12:17
lyarwoodstephenfin: ^ the above should unblock the gate on master btw12:17
*** k_mouza has joined #openstack-nova12:21
*** haleyb has quit IRC12:22
openstackgerritRodrigo Barbieri proposed openstack/nova stable/ussuri: Allow tap interface with multiqueue  https://review.opendev.org/75224512:23
openstackgerritRodrigo Barbieri proposed openstack/nova stable/ussuri: Allow tap interface with multiqueue  https://review.opendev.org/75224512:23
*** haleyb has joined #openstack-nova12:24
*** k_mouza has quit IRC12:25
openstackgerritElod Illes proposed openstack/nova stable/ussuri: Allow tap interface with multiqueue  https://review.opendev.org/75224512:31
*** k_mouza has joined #openstack-nova12:31
*** psachin has quit IRC12:35
*** nweinber has joined #openstack-nova12:40
*** k_mouza has quit IRC12:44
*** k_mouza has joined #openstack-nova12:53
*** k_mouza has quit IRC12:58
*** k_mouza has joined #openstack-nova13:03
*** k_mouza has quit IRC13:05
*** k_mouza has joined #openstack-nova13:05
openstackgerritHemanth Nakkina proposed openstack/nova master: Update pci stat pools based on PCI device changes  https://review.opendev.org/74917513:09
openstackgerritLee Yarwood proposed openstack/nova master: WIP: nova-next: Start testing the 'q35' machine type  https://review.opendev.org/70870113:14
lyarwoodkashyap: ^ think I found a n-cpu bug with QEMU v5.0.0, looks like we will need to rewrite our detach logic13:14
lyarwoodI'll write up a bug once the vanilla fedora-latest-virt-preview job has also hit it without q3513:14
*** sean-k-mooney1 is now known as sean-k-mooney13:19
sean-k-mooneylyarwood: is this related the the qemu issue13:20
sean-k-mooneyon ubuntu13:20
sean-k-mooneyi.e. would the same rewite help13:20
lyarwoodsean-k-mooney: it's unrelated13:21
lyarwoodsean-k-mooney: https://github.com/qemu/qemu/blob/cbba3dc6ea3fc9aa66e9f9eb41051536e3ad7cd0/qdev-monitor.c#L886-L898 - n-cpu is tripping up over this check introduced in QEMU v5.0.0 as we fire off detach requests without waiting for libvirt to emit events.13:22
lyarwoodhmm unless13:23
sean-k-mooneyif (dev->pending_deleted_event) {13:23
lyarwoodare we hitting the bug this check was introduced to avoid?13:23
sean-k-mooneythat check13:23
sean-k-mooneyperhaps13:24
sean-k-mooneyhttps://github.com/qemu/qemu/commit/cce8944cc9efab47d4bf29cfffb3470371c3541b13:24
lyarwoodso the commit references PCI devices13:24
lyarwoodoh I wondeer13:24
lyarwoodinteresting13:24
sean-k-mooneyOn PCIe devices, this cancels the hot-unplug process.13:25
lyarwoodI don't recall seeing another request from libvirt to QEMU in the focal examples13:25
lyarwoodbut maybe13:25
sean-k-mooneyif we retry it it would right13:25
sean-k-mooneywe just loop13:25
lyarwoodright but I can't recall seeing the requests overlap13:25
lyarwoodbut the loop is pretty tight13:25
lyarwoodwould explain why we only saw it on overloaded CI hosts13:26
kashyaplyarwood: Hey, was out for a small walk; reading your comments in the change13:27
kashyapThat "unable to execute ... device_del' rings a bell13:28
kashyaplyarwood: I think we _are_ hitting the bug the check was introduced to avoid, no?  The QEMU commit skips repeated QMP 'device_del' calls13:34
*** slaweq has quit IRC13:36
lyarwoodkashyap: right and with >=v5.0.0 that raises an error, I was wondering above if prior to that on focal we are actually hitting the original issue and hot unplugs are being cancelled in QEMU13:37
kashyaplyarwood: Ah, hence your comment on "why we only saw it on overloaded CI hosts"13:37
lyarwoodkashyap: either way the fix here seems to be to use libvirt events to wait until a detach either succeeds or fails before we retry13:38
sean-k-mooneykashyap: right when we race with a slow detach due to load13:38
sean-k-mooneylyarwood: ya so we should do that in either case and then we can use the ci to see if it fixes the ci issue with a DNM patch to swap the job to focal form bionic13:38
*** rcernin has quit IRC13:38
sean-k-mooneylyarwood: we do sometime get random detach failure on bionic too13:39
sean-k-mooneywhich could be caused by this13:39
sean-k-mooneyits much less common but i have seen many jobs fail for seminly unrelated volume detach issues13:40
kashyaplyarwood: Yeah, it's a long-time TODO to wait for proper events, instead of polling :-(  There should also be a documented TODO in driver.py/guest.py13:40
*** slaweq has joined #openstack-nova13:42
*** xek has quit IRC13:45
*** Luzi has joined #openstack-nova14:00
*** belmoreira has joined #openstack-nova14:02
*** hamalq has joined #openstack-nova14:07
*** hamalq has quit IRC14:11
openstackgerritLee Yarwood proposed openstack/nova master: DNM libvirt: Remove loopingcall when detaching device  https://review.opendev.org/75552614:23
*** xek has joined #openstack-nova14:33
*** jdillaman has quit IRC14:36
openstackgerritsean mooney proposed openstack/nova master: Fix unplugging VIF when migrate/resize VM  https://review.opendev.org/75164214:49
openstackgerritsean mooney proposed openstack/nova master: libvirt: delegate ovs plug to os-vif  https://review.opendev.org/60243214:49
*** dklyle has joined #openstack-nova14:51
sean-k-mooneythose are just rebases by the way ^ i want to see if the tempest failure is real or not14:56
sean-k-mooneyi dont think they are but a rebase never hurts14:57
*** mlavalle has joined #openstack-nova15:08
*** Luzi has quit IRC15:15
*** belmoreira has quit IRC15:18
*** jangutter has joined #openstack-nova15:28
*** mkrai has quit IRC15:30
*** mkrai_ has joined #openstack-nova15:30
*** tesseract has quit IRC15:33
gibinova weekly meeting starts in 15 minutes in #openstack-meeting-315:45
*** jraju__ has quit IRC15:53
*** hamalq has joined #openstack-nova16:00
*** hamalq has quit IRC16:04
*** hamalq has joined #openstack-nova16:04
*** hamalq has quit IRC16:07
*** hamalq has joined #openstack-nova16:08
*** gyee has joined #openstack-nova16:12
*** salmankhan1 has joined #openstack-nova16:14
*** salmankhan has quit IRC16:15
*** salmankhan1 is now known as salmankhan16:15
*** nweinber has quit IRC16:23
*** nweinber has joined #openstack-nova16:23
*** ociuhandu has quit IRC16:24
openstackgerritBalazs Gibizer proposed openstack/nova master: Temporarily disable parts of heal port allocation test  https://review.opendev.org/75549816:30
*** salmankhan1 has joined #openstack-nova16:31
gibilyarwood, stephenfin : could you hit it again? ^16:31
*** salmankhan has quit IRC16:32
*** salmankhan1 is now known as salmankhan16:32
lyarwoodgibi: will do once it passes16:32
gibiOK :)16:32
gibithanks16:33
gibiI will leave for today in 30 minutes so I guess I won't see it pass16:33
*** ralonsoh has quit IRC16:35
stephenfin#16:35
stephenfinDone. If it doesn't pass, it simply won't merge :)16:35
gibithanks16:38
*** k_mouza has quit IRC16:45
*** k_mouza has joined #openstack-nova17:00
*** derekh has quit IRC17:03
*** k_mouza has quit IRC17:05
*** k_mouza has joined #openstack-nova17:16
*** k_mouza has quit IRC17:21
*** k_mouza has joined #openstack-nova17:24
*** k_mouza has quit IRC17:28
*** k_mouza has joined #openstack-nova17:41
*** k_mouza has quit IRC17:45
*** jmlowe has quit IRC18:00
*** ganso has quit IRC18:00
*** k_mouza has joined #openstack-nova18:01
*** mkrai_ has quit IRC18:03
*** k_mouza has quit IRC18:05
*** sapd1 has quit IRC18:22
*** k_mouza has joined #openstack-nova18:28
*** k_mouza has quit IRC18:33
*** jmlowe has joined #openstack-nova18:47
*** ganso has joined #openstack-nova18:47
*** k_mouza has joined #openstack-nova18:52
*** bbowen has quit IRC18:53
*** iurygregory has quit IRC18:55
*** k_mouza has quit IRC18:57
*** salmankhan has quit IRC18:57
*** priteau has quit IRC19:11
*** ociuhandu has joined #openstack-nova19:21
*** ociuhandu has quit IRC19:26
*** k_mouza has joined #openstack-nova19:26
*** k_mouza has quit IRC19:31
*** iurygregory has joined #openstack-nova19:43
*** k_mouza has joined #openstack-nova19:46
*** k_mouza has quit IRC19:50
*** k_mouza has joined #openstack-nova20:09
*** k_mouza has quit IRC20:14
*** nweinber has quit IRC20:54
openstackgerritmelanie witt proposed openstack/nova stable/pike: Include only required fields in ironic node cache  https://review.opendev.org/75444421:09
openstackgerritmelanie witt proposed openstack/nova stable/pike: Lowercase ironic driver hash ring and ignore case in cache  https://review.opendev.org/72305521:09
openstackgerritmelanie witt proposed openstack/nova stable/pike: Add resource_class to fields in ironic node cache  https://review.opendev.org/75565321:09
*** freenzyfriday has joined #openstack-nova21:09
*** raildo has quit IRC21:13
*** freenzyfriday has quit IRC21:14
*** xek has quit IRC21:15
*** slaweq has quit IRC21:15
*** mgoddard has quit IRC21:30
*** mgoddard has joined #openstack-nova21:36
*** freenzyfriday has joined #openstack-nova21:55
*** freenzyfriday has quit IRC21:59
*** k_mouza has joined #openstack-nova22:14
*** k_mouza has quit IRC22:18
*** rcernin has joined #openstack-nova22:24
*** tosky has quit IRC22:45
*** freenzyfriday has joined #openstack-nova22:46
*** freenzyfriday has quit IRC22:51
*** k_mouza has joined #openstack-nova22:51
*** k_mouza has quit IRC22:55
*** tkajinam has joined #openstack-nova23:00
*** k_mouza has joined #openstack-nova23:16
*** k_mouza has quit IRC23:20
*** k_mouza has joined #openstack-nova23:36
*** k_mouza has quit IRC23:40
*** mlavalle has quit IRC23:41
*** jmlowe has quit IRC23:54
*** ganso has quit IRC23:54
*** redrobot has quit IRC23:59

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!