Monday, 2022-11-28

*** dasm|off is now known as Guest18805:30
bauzasgmann: thanks for having incepted the placement LP resource https://launchpad.net/placement09:46
bauzasgmann: since I don't know exactly what's left, I just set the maintainer back to you, sure09:46
bauzasgmann: ping me once you're done 09:46
darkhorseHi team, In the compute_nodes table, there are 'host' column and 'hypervisor_hostname' column, which looks the same. Will you explain the difference between these fields?10:14
gibibauzas: so https://review.opendev.org/c/openstack/tempest/+/842240 is not green yet, but gmann pushed a test patch for nova-lvm with it https://review.opendev.org/c/openstack/nova/+/865765 it is also red but with one less failed test case (test_attach_detach_volume is green there now)10:24
bauzasack10:25
bauzasdarkhorse: 'host' is about the name of the nova-compute service10:26
bauzaswhich is opted in by the conf option 10:26
bauzasdarkhorse: while "hypervisor_hostname" is provided by the virt driver10:27
darkhorsebauzas: thank you for clarification!10:30
bauzasnp10:30
bauzasgibi: sean-k-mooney: could you please review my spec https://review.opendev.org/c/openstack/nova-specs/+/861591 before 3pm UTC today ? :)10:32
opendevreviewSylvain Bauza proposed openstack/nova master: Skip live_migration_with_trunk tempest test on ovs-hybrid and nova-next  https://review.opendev.org/c/openstack/nova/+/86565810:34
bauzasgibi: sean-k-mooney: gmann: ^ fwiw, added depends-on gmann's tempest patch on ^ to eventually get it accepted10:35
* bauzas feels like he's playing a limbo dance10:36
gibibauzas: lets wait for gmann to get up and discuss what are our options about the gate regarding the detach issue. If gmann sees a possible fix then I would wait for it, but if he doesn't then I cannot offer any better way forward now than disabling the tests given the amount of time I can spend on this.10:42
bauzasgibi: I need to catch my daughter at school but let's revisit this once gmann's up10:43
gibiyepp10:43
gibibauzas: I'm +2 on https://review.opendev.org/c/openstack/nova-specs/+/86159110:43
* sean-k-mooney is tempted to wait till 14:59 utc10:46
opendevreviewKirill proposed openstack/nova-specs master: new spec: support of vnc console for ironic  https://review.opendev.org/c/openstack/nova-specs/+/86377310:49
bauzasgibi: I very briefly looked at the failing tests and yeah, we need to continue working on it if we want to merge it11:11
bauzasI can try to help but I don't have a lot of context about it11:11
bauzasalso, I wonder *why* we now have a larger number of issues with nova-lvm11:12
bauzasanyway, /me goes afk for lunch11:12
kashyapbauzas: I think not many (if any at all) production users use LVM11:13
kashyap(With Nova, I mean)11:14
sean-k-mooneyit used to be popular for thost that need the better write prefromance11:14
sean-k-mooneybut its much less common then it used to be11:15
kashyapYeah11:15
sean-k-mooneyi previously used it for my home cloud in the past. i never really had issues with it. im currently using cinder lvm with bfv instead11:17
gibibauzas: when we change dependency version, ie changing to the next ubuntu image, the timings change and that teases out new races11:19
gibiit think the jammy change triggered different timings in our tests showing more this issue11:19
opendevreviewMerged openstack/nova-specs master: Proposes cpu power managment in libvirt  https://review.opendev.org/c/openstack/nova-specs/+/86159111:27
opendevreviewMerged openstack/nova-specs master: Re-propose per process healthchecks  https://review.opendev.org/c/openstack/nova-specs/+/86415012:38
opendevreviewAmit Uniyal proposed openstack/nova stable/ussuri: Adds a repoducer for post live migration fail  https://review.opendev.org/c/openstack/nova/+/86400612:53
opendevreviewAmit Uniyal proposed openstack/nova stable/ussuri: [compute] always set instance.host in post_livemigration  https://review.opendev.org/c/openstack/nova/+/86400712:53
bauzasfwiw, I got f***d by Zuul b/c of nova-lvm again even if Depends-on on the Tempest patch https://review.opendev.org/c/openstack/nova/+/86565813:06
sean-k-mooneywell the tempest change was still failing13:07
bauzasyup13:07
bauzasbut the issue is still present13:07
sean-k-mooneyright becaues qemu is broken13:07
bauzasother jobs are failing to due the flakey tempest test I agree13:07
sean-k-mooneyso we can either disabel the volume tests or we can make it non voting but we cant really do anythign else in onva13:08
bauzasbut for nova-lvm, we get the same exception than previously without depending on the tempest changhe13:08
sean-k-mooneythe nova-lvm job is intended to test the lvm image backend13:08
bauzasyeah I know13:08
sean-k-mooneyso we dont striclty need to test cinder there13:08
bauzasI don't disagree13:09
bauzasbut the failure is also for the rescue tests13:10
bauzasin tearDownClass fixture ending13:10
gokhanihello folks, some of my instances image backing file is deleted unexpectedly. And unfortunately glance image is also deleted. I only see a disk file under /var/lib/nova/instances. is there any way rescue this instances ? I tried using guestfish but it also tries to reach backing  file. 13:11
gibibauzas, sean-k-mooney: lets sync with gmann, but sadly I have to agree to disable those tests temprorarily13:12
bauzasgibi: tbh, I don't know which exact cables to unplug13:12
bauzasI mean, I know the cables13:13
bauzasbut I wonder if by unplugging them, it would either shutdown the Big Thing or even just Do Nothing13:13
sean-k-mooneyyou woudl have to disabel all volume related tests more or less13:14
gibisean-k-mooney: no we don't have to, I think some of those has the wait for sshable thing implemented13:14
sean-k-mooneythere is one other tempest serise that might be related13:14
sean-k-mooneygibi: it failed with that13:15
sean-k-mooneyi mean the current patch might not work 13:15
sean-k-mooneyand making it work might be enough13:15
gibisean-k-mooney: it failed with a tempest patch that adds some sshable flag to some tests. But we already merged a lot of sshable flag patches13:15
sean-k-mooneybut the current patch still failed13:15
* bauzas is about to write a 'hold your rechecks' email 13:15
sean-k-mooneyoh i know but bauzas added a depend-on ot the last one in that seriese13:16
bauzasyup13:16
sean-k-mooneyso we seam to need somethign else on top of that13:16
gibiyes, taht depends on was for adding sshable for hopefully the last set of volume tests13:16
bauzasyup13:16
gibithat tempest test is not corret yet13:16
gibibut gmann hopefully knows more about it13:16
* bauzas needs a coffee13:17
bauzasI'm just wrapping up all the LP bug reports 13:17
gibiso I think we need to disable those volume tests that are failing currently, not all volume tests13:17
sean-k-mooneyhttps://review.opendev.org/c/openstack/tempest/+/84041013:17
sean-k-mooneythis one might also be interesting ^13:17
gibiyes, we need that too to disable trunk13:17
gibithere we know ralonsoh working on a fix13:17
sean-k-mooneyhttps://review.opendev.org/c/openstack/tempest/+/840410  fixes a issue witht he cleanup function ordering13:18
sean-k-mooneyrelated to volumes13:18
gibisorry i mixed up things13:18
gibihttps://review.opendev.org/c/openstack/tempest/+/840410 is new to me13:19
sean-k-mooneybauzas: could you perhaps also add a depend on for that13:19
gibiprobably we need a single nova patch that disables trunk (as in https://review.opendev.org/c/openstack/nova/+/865658) and disable the failing volume tests13:19
sean-k-mooneyya that would be simpler to merge13:20
bauzassean-k-mooney: sec, writing an email to inform the world we're busted13:20
bauzasI'm here13:28
bauzasso13:28
sean-k-mooneyjoing an internal call shortly but https://review.opendev.org/c/openstack/tempest/+/840410 may help with the volume issues13:29
sean-k-mooneyif you can add that as a second depends on it might help13:29
bauzassean-k-mooney: I can try13:29
bauzasthat's gonna be fun13:29
sean-k-mooneyalternitivly we can tempoerly disable the volume tests in the same patch that is skiping the tunk tests13:29
sean-k-mooneyand then try that in the followup that turns them back on13:30
bauzashttps://lists.openstack.org/pipermail/openstack-discuss/2022-November/031335.html13:30
bauzasralonsoh: you also have a different tracking bug number for the wrong port state13:30
bauzasralonsoh: maybe we should merge those 2 bug reports we have13:30
bauzassean-k-mooney: editing inline in gerrit, so we'll see13:31
opendevreviewSylvain Bauza proposed openstack/nova master: Skip live_migration_with_trunk tempest test on ovs-hybrid and nova-next  https://review.opendev.org/c/openstack/nova/+/86565813:32
ralonsohbauzas, yes, one sec13:39
ralonsohbauzas, https://bugs.launchpad.net/neutron/+bug/199702513:40
ralonsohpatch: https://review.opendev.org/c/openstack/neutron/+/83778013:40
ralonsohI've increased the priority of th ereview13:41
bauzasralonsoh: would you disagree if I'm closing https://bugs.launchpad.net/neutron/+bug/1997025 as duplicate of https://bugs.launchpad.net/neutron/+bug/1940425 which was already triaged for Neutron ?13:41
ralonsohbauzas, that issue was something different13:42
ralonsohin https://bugs.launchpad.net/neutron/+bug/1997025, the port is not activated13:42
ralonsohand this is because of the os-vif release13:42
bauzasHmmm, OK13:43
ralonsoh--> https://bugs.launchpad.net/neutron/+bug/1997025/comments/113:43
*** akekane is now known as abhishekk13:57
*** Guest188 is now known as dasm13:58
opendevreviewSylvain Bauza proposed openstack/nova master: Skip live_migration_with_trunk tempest test on ovs-hybrid and nova-next  https://review.opendev.org/c/openstack/nova/+/86565814:46
bauzasyet another try ^14:47
bauzasthe previous run was failing 14:47
bauzassean-k-mooney: gibi: so, I'm about to skip the nova-lvm tests that fail17:13
bauzasfwiw, nova-grenade-multinode also hits the port not active isue https://zuul.opendev.org/t/openstack/build/16cc990d758348999cc5c1b2b30aa9a217:13
gibibauzas: ack17:15
ralonsohbauzas, https://review.opendev.org/c/openstack/neutron/+/83778017:18
ralonsohwe have merged the patch in Neutron17:19
ralonsohI think you should be OK now17:19
ralonsoh(I hope so)17:19
bauzasah, nice to hear17:19
bauzasI can try to recheck my original patch17:19
bauzasralonsoh: kudos to the neutron team17:19
ralonsohyw17:20
sean-k-mooneyralonsoh ah an ye removed the config option great17:25
sean-k-mooneythen it should just work17:25
sean-k-mooneywithout us needing to modify jobs17:25
ralonsohyes referring to the Nova patch17:25
bauzasralonsoh: sean-k-mooney: I just rechecked https://review.opendev.org/c/openstack/nova/+/838976/817:26
bauzaswe will see17:26
ralonsohbauzas, I'll keep an eye on it17:26
sean-k-mooneybauzas: we still need to skip the lvm tests17:26
sean-k-mooneyright17:26
bauzaseither way, it will fail due to the nova-lvm issue but we'll see whether ovn and nova-next jobs work17:26
bauzassean-k-mooney: yup17:26
bauzasthis is just for checking against something which is not my skiptest patch17:27
sean-k-mooneyya ok well it might pass put its unlikely to do it twice to get passed both gate and check17:27
bauzasyup17:27
bauzasnova-lvm seems to fail 100% of times17:27
sean-k-mooneyis nova-lvm using pc or q35 by the way i assume pc17:29
sean-k-mooneythe ohter thing we could try is putting it back to focal temporilly17:30
sean-k-mooneythats more invasive however then skiping the flaky tests17:30
sean-k-mooneynot set in the nova.conf so it should be pc17:31
sean-k-mooneyhttps://zuul.opendev.org/t/openstack/build/f076244738904c949e63b21571a5fc9a/log/controller/logs/etc/nova/nova-cpu_conf.txt#11817:31
opendevreviewAmit Uniyal proposed openstack/nova stable/train: Adds a repoducer for post live migration fail  https://review.opendev.org/c/openstack/nova/+/86380617:46
opendevreviewAmit Uniyal proposed openstack/nova stable/train: [compute] always set instance.host in post_livemigration  https://review.opendev.org/c/openstack/nova/+/86405517:46
gmanngibi: bauzas sean-k-mooney: hi, for detach issue in nova-lvm job. tempest patch need more time it seems. I was making server sshable by default which need more work in other tests.17:57
gmanngibi: sean-k-mooney bauzas: I am moving my house this week so might need some more time on that. For now, I think we can go with skipping the failing tests temporarily in nova-lvm17:58
bauzasack17:59
opendevreviewAmit Uniyal proposed openstack/nova stable/ussuri: [compute] always set instance.host in post_livemigration  https://review.opendev.org/c/openstack/nova/+/86400717:59
gibigmann: thanks for looking into it17:59
sean-k-mooneyso i think we shoudl be ok to skip them for a few weeks to unblock things18:00
gibiyepp I agree18:01
sean-k-mooneythe cidner code paths really shoudl not depend on the images_backend that is in use at least not lvm18:01
gibisady we have no better option now18:01
gmannbut thing is those failing rescue test, all use ssh-able server. there is something else might be causing the failure18:01
sean-k-mooneygmann: yep there likely is18:01
gmannsomething to do on Jammy as lvm on focal working fine18:01
sean-k-mooneypossibly18:02
sean-k-mooneyit defiently happens more on jammy im not sure it would not happen on focal 18:02
sean-k-mooneyit might be related to a default config delta between the two or the qemu/libvirt verions18:02
gmannmay be18:02
sean-k-mooneybut we dont currently have a lot of capsity to investigate18:02
gmannyeah, do we have patch up to skip the tests? 18:03
sean-k-mooneyi think bauzas has one but im not sure they updated for the volume tests18:04
gmannk18:04
gmannthis one? https://review.opendev.org/c/openstack/nova/+/86565818:04
sean-k-mooneythat was to solve the other issue so that is not needed18:05
gmannthis does not include nova-lvm failing tests 18:05
gmannk18:05
sean-k-mooneywe shoudl pull that form the gate18:05
sean-k-mooneygmann: i do not see one form sylvain18:06
sean-k-mooneyso i guess there is non e right now for the lvm failures18:06
opendevreviewAmit Uniyal proposed openstack/nova stable/train: [compute] always set instance.host in post_livemigration  https://review.opendev.org/c/openstack/nova/+/86405518:11
*** dasm is now known as dasm|off22:01
opendevreviewGhanshyam proposed openstack/nova master: Temporary skip some volume detach test in nova-lvm job  https://review.opendev.org/c/openstack/nova/+/86592223:59

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!