Friday, 2022-02-04

*** amoralej|off is now known as amoralej07:28
*** hemna3 is now known as hemna07:37
gibisean-k-mooney: hi! could you upgrade my vote from +1 to +2 on https://review.opendev.org/q/topic:bug%252F1952941 ? thanks!11:10
opendevreviewDmitrii Shcherbakov proposed openstack/nova master: [yoga] Add support for VNIC_TYPE_SMARTNIC  https://review.opendev.org/c/openstack/nova/+/82483511:11
opendevreviewDmitrii Shcherbakov proposed openstack/nova master: Filter computes without remote-managed ports early  https://review.opendev.org/c/openstack/nova/+/81211111:11
opendevreviewDmitrii Shcherbakov proposed openstack/nova master: Add supports_remote_managed_ports capability  https://review.opendev.org/c/openstack/nova/+/82783911:11
sean-k-mooney[m]gibi: ah the requestspec fix sure11:12
dmitriisgibi: apologies for an extra respin of https://review.opendev.org/c/openstack/nova/+/812111. I moved the addition of the compute capability to a separate change https://review.opendev.org/c/openstack/nova/+/827839 and made an extra check in the whitelist. This way we can check if Libvirt has the right version and reject remote_managed ports if it11:12
dmitriisdoesn't early at startup.11:12
sean-k-mooney[m]ill review the series shortly11:12
gibidmitriis: no worry I will check after lunch11:16
dmitriisgibi: ack, tyvm11:16
opendevreviewsean mooney proposed openstack/nova master: [WIP] add initial healthcheck support  https://review.opendev.org/c/openstack/nova/+/82501511:18
opendevreviewsean mooney proposed openstack/nova master: [WIP] add healthcheck manager to manager base  https://review.opendev.org/c/openstack/nova/+/82784411:18
sean-k-mooneystephenfin: by th way im also good with your unittest patch https://review.opendev.org/c/openstack/nova/+/714676 gibi was previously +2 on it12:10
sean-k-mooneyi just approved his patches which will conflict with it12:10
sean-k-mooneyand rechecked yours12:11
sean-k-mooneyso waht i woudl propose is if we need to rebase your mock removal patch we shoudl try and fast approve it when you resubmit to avoid conflict with other patches12:11
gibidmitriis: left comments in https://review.opendev.org/c/openstack/nova/+/82783912:17
sean-k-mooneyartom_: did you put up a patch to skip the flaky test in nova-next12:19
sean-k-mooneyif not i might quickly do that12:19
gibisean-k-mooney: I don't see such patch from artom12:20
sean-k-mooneywe are only seeing tempest.api.compute.servers.test_device_tagging.TaggedAttachmentsTest.test_tagged_attachment fail so ill limit it to that 12:21
gibisean-k-mooney: yepp12:21
sean-k-mooneyah https://bugs.launchpad.net/nova/+bug/1959899 is the bug cool12:24
gibiyes12:24
sean-k-mooneyi was just looking for it for the topic/commit message12:24
opendevreviewsean mooney proposed openstack/nova master: skip test_tagged_attachment in nova-next  https://review.opendev.org/c/openstack/nova/+/82785112:35
sean-k-mooneyok ya i think ^ is all that is needed 12:36
opendevreviewMerged openstack/nova master: Reproduce bug 1952941  https://review.opendev.org/c/openstack/nova/+/82012112:42
gibisean-k-mooney: thanks12:44
gibisean-k-mooney: I will fast approve it as soon as zuul is green on the patch12:44
artom_gibi, sean-k-mooney, I hadn't, so thanks!12:52
opendevreviewAttila Fazekas proposed openstack/nova master: Document the floating ip behavior in short  https://review.opendev.org/c/openstack/nova/+/82785612:58
dmitriisgibi: looking13:28
tobias-urdini have a weird issue, back in december 2021 a resize that failed, also failed to cleanup /var/lib/instances/<uuid>_resize directory, when resize was retried now (earlier today) it failed because os.rename got File exists when it tried to rename that https://github.com/openstack/nova/blob/stable/victoria/nova/virt/libvirt/driver.py#L10402 -- see13:31
tobias-urdinlogs https://paste.opendev.org/show/812522/13:31
tobias-urdinthe first issue was because of a libvirt issue (out of fd and socket issues)13:31
tobias-urdindon't have logs to understand if for example SELinux blocked the file deletion, does it make sense to insert something that would catch the issue and remove the _resize directory before trying the rename? since the API would ensure there can't be an race condition for resizes13:32
gibitobias-urdin: do I understand correctly that the original delete of the dir faild and this now prevents the retry of the resize? I think we can robustify nova by retry the delete during the next resize if it sees the stale dir there13:34
*** dasm|off is now known as dasm13:36
tobias-urdingibi: yes, resize failed because of leftover <uuid>_resize directory in instances directory13:51
opendevreviewTobias Urdin proposed openstack/nova master: Cleanup _resize directory in instances dir on resize  https://review.opendev.org/c/openstack/nova/+/82786514:02
tobias-urdingibi: i guess it could be as simple as ^14:03
gibitobias-urdin: that could be it, but would be nice to add a test around it 14:12
gibitobias-urdin: and bug report for tracking purposes :)14:13
tobias-urdinyeah i'll fix everything up if it feels acceptable (bug report, test, release note etc) - should be very low risk backport as well14:16
gibitobias-urdin: I agree this feels a safe thing to do14:16
sean-k-mooneyi have not beeing following but you will need to ensuer its not a shared file system14:17
sean-k-mooneyotherwise the cleanup would delete the vm14:17
gibisean-k-mooney: it is the _resize dir I think that is being created there, so deleting the dir before we creating it feels save14:17
gibisafe14:17
opendevreviewMerged openstack/nova master: Migrate RequestSpec.numa_topology to use pcpuset  https://review.opendev.org/c/openstack/nova/+/82015314:18
sean-k-mooneyah ok we have a parallel dir that we then move into the final location later?14:18
sean-k-mooneyos.rename(inst_base, inst_base_resize)14:18
sean-k-mooneyso that is moving inst_base to inst_base_resize14:19
gibiyepp, to save it if we want to roll back14:19
sean-k-mooneyok so ya that should be safe14:19
gibitobias-urdin: thanks for taking care of14:19
gibiit14:20
opendevreviewBalazs Gibizer proposed openstack/nova stable/xena: Reproduce bug 1952941  https://review.opendev.org/c/openstack/nova/+/82786814:30
opendevreviewBalazs Gibizer proposed openstack/nova stable/xena: Migrate RequestSpec.numa_topology to use pcpuset  https://review.opendev.org/c/openstack/nova/+/82786914:30
opendevreviewBalazs Gibizer proposed openstack/nova stable/wallaby: Reproduce bug 1952941  https://review.opendev.org/c/openstack/nova/+/82787014:34
opendevreviewBalazs Gibizer proposed openstack/nova stable/wallaby: Migrate RequestSpec.numa_topology to use pcpuset  https://review.opendev.org/c/openstack/nova/+/82787114:34
opendevreviewBalazs Gibizer proposed openstack/nova stable/victoria: Reproduce bug 1952941  https://review.opendev.org/c/openstack/nova/+/82787214:37
opendevreviewBalazs Gibizer proposed openstack/nova stable/victoria: Migrate RequestSpec.numa_topology to use pcpuset  https://review.opendev.org/c/openstack/nova/+/82787314:37
opendevreviewMerged openstack/nova master: Revert "Revert resize: wait for events according to hybrid plug"  https://review.opendev.org/c/openstack/nova/+/81949415:23
opendevreviewMerged openstack/nova stable/wallaby: [rt] Apply migration context for incoming migrations  https://review.opendev.org/c/openstack/nova/+/82055515:43
opendevreviewDmitrii Shcherbakov proposed openstack/nova master: Add supports_remote_managed_ports capability  https://review.opendev.org/c/openstack/nova/+/82783916:08
opendevreviewDmitrii Shcherbakov proposed openstack/nova master: [yoga] Add support for VNIC_TYPE_SMARTNIC  https://review.opendev.org/c/openstack/nova/+/82483516:08
opendevreviewDmitrii Shcherbakov proposed openstack/nova master: Filter computes without remote-managed ports early  https://review.opendev.org/c/openstack/nova/+/81211116:08
dmitriisgibi: re-uploaded with the cases mentioned covered16:17
dmitriisgibi: hopefully the approach in general is OK16:17
gibidmitriis: thanks. I will check it on monday 16:36
gibihave a nice weekend16:36
dmitriisgibi: same to you, cheers o/16:37
gibio/16:37
* gibi leaves the building16:42
*** amoralej is now known as amoralej|off17:41
opendevreviewFederico Ressi proposed openstack/nova master: Debug Nova APIs call failures  https://review.opendev.org/c/openstack/nova/+/80668317:42
chateaulavhas anyone run into issues with libvirt in rgards to `error: failed to get CPU model names`18:32
sean-k-mooneyi hit it yesterday when i set an incorrct value on an arm system18:37
sean-k-mooneyi put virt for [libvirt]/cpu_model18:38
sean-k-mooneyit should have been max or host18:38
sean-k-mooneyso if you use virt-type=qemu and cpu_model=virt it will raise that error 18:38
sean-k-mooneyi shoudl have had virt_type=qemu, cpu_mode=custom and cpu_model=max with machine_type=aarch64:virt or set on the image18:39
chateaulavsean-k-mooney: it relates to riscv currently. I can run emulation using the qemu binary directly; however, run into issue because virsh doesnt see any available cpu models18:39
chateaulavhttps://www.irccloud.com/pastebin/SoDG0HXa/error-risc-v18:40
sean-k-mooneyif you do qemu-system-riscv64 --cpu ?18:42
sean-k-mooneyit will list the supported ones18:42
sean-k-mooneyhome ok libvirt may not have support for that yet18:42
sean-k-mooneyyou could try rv6418:42
sean-k-mooneyfor the cpu_model18:43
sean-k-mooneyand see if it work if you spcify one18:43
chateaulavyeah no luck, been going back and forth and though i may have missed something.18:43
sean-k-mooneyno it sound like libvirt is missing supprt18:43
chateaulavyeah thats what i was thinking to18:45
sean-k-mooneyit might work in a later libvirt18:45
sean-k-mooneywhat version are you using18:45
sean-k-mooneythe most recent release is 8.0.0 i belive18:45
chateaulavlibvirtd (libvirt) 6.0.018:45
sean-k-mooneyi mean its ment to have supprot18:46
sean-k-mooneyi see stuff as far back as 4.7.018:46
sean-k-mooneyin the release notes18:46
chateaulavit seems like partial support, gonna test along that route18:47
sean-k-mooneymaybe they dont support emulation with riscv18:47
sean-k-mooneyjust native supprot?18:47
chateaulavI feel that as well, a couple last checks and then ill put a nail in it18:49
opendevreviewArtom Lifshitz proposed openstack/nova master: DNM: Run Nova gate on Neutron notifier patch  https://review.opendev.org/c/openstack/nova/+/82791719:56
opendevreviewAde Lee proposed openstack/nova stable/xena: Add check job for FIPS  https://review.opendev.org/c/openstack/nova/+/82789519:59
opendevreviewAde Lee proposed openstack/nova stable/xena: Add check job for FIPS  https://review.opendev.org/c/openstack/nova/+/82789520:00
opendevreviewAde Lee proposed openstack/nova stable/wallaby: Add check job for FIPS  https://review.opendev.org/c/openstack/nova/+/82789620:01
melwittdansmith: hi :D I have been working on removing tracebacks in the logs in nova-multi-cell that happen when a neutron network-changed event is received after the instance has moved to another cell during a cross-cell migration,20:03
* sean-k-mooney note to self you still hate debuging networkign issues.20:04
melwittthis seemed simple at first until the fact that the exception is being raised from the db access on nova-conductor but the tracing is happening in nova-compute,20:04
sean-k-mooneyit helps if you enable forwarding on the host if you want nat to work for your vms.... sudo sysctl net.ipv4.conf.all.forwarding=1 20:05
dansmithmelwitt: because of indirection I assume20:05
melwittI'm seeing tracing still from the rpc messaging layer, so I tried this https://review.opendev.org/c/openstack/nova/+/826942/5/nova/objects/instance_info_cache.py#77 but it doesn't work as expected. I get oslo_messaging.rpc.client.RemoteError: Remote error: ExpectedException on the client side instead of InstanceNotFound20:06
melwittI had thought when oslo.messaging receives ExpectedException on the client side it translates it into the original exception type that it wrapped on the server side, but that is not occurring20:06
dansmithno, I think that just avoids the traceback on the conductor side20:07
melwitthm ok20:08
sean-k-mooneymelwitt: i assume you are not checkign if the instance exists then updating because it could get deleted in between 20:08
sean-k-mooneywhich is why you are trying to catch the excpiton20:08
melwittsean-k-mooney: yeah20:08
melwittwell, I'm catching DBReferenceError and then raising InstanceNotFound to be handled by upper layers20:09
sean-k-mooneydo the upper layers need to do anything with the not found20:09
sean-k-mooneythe instance has been moved to the other cell yes20:09
melwittmost things handle not found, like "the instance disappeared while <whatever>"20:10
melwittif someone deletes while it's in the middle of <whatever>20:11
melwittit's working fine except I'm still getting tracebacks in nova-compute coming from oslo.messaging and I can't get it to stop20:11
sean-k-mooneyack20:11
melwittit sends over the longass chain of exception tracebacks even though I caught DBReferenceError on the conductor side20:13
dansmithmelwitt: I think oslo.messaging wraps exceptions in a special class called Original_Remote, IIRC20:29
dansmithnot sure you can catch that directly though because I think it's dynamic20:29
dansmithif you catch it with except Exception, then try to print e.__class__20:29
melwittdansmith: yeah, it's weird. it's as if it's logging it somewhere I can't get to (like in oslo.messaging). this is an example from PS3 before I added any @expected_exceptions, it logs at INFO what I wanted and then directly after that it logs the supertraceback https://zuul.opendev.org/t/openstack/build/7c3c755868b848ed841d37457004d99b/log/controller/logs/screen-n-cpu.txt#1002720:33
melwittI don't think? it's just appended to the exception message bc it's logged in a different color but since I'm desperate I might try truncating str(e) to see what happens20:33
dansmithyeah I expect it's logged explicitly somewhere, but not sure20:36
dansmithnothing specific jumps out at me though20:36
melwittdansmith: ok, I'll try some more things. thanks for looking :)20:39
opendevreviewJonathan Race proposed openstack/nova master: Adds Pick guest CPU architecture based on host arch in libvirt driver support  https://review.opendev.org/c/openstack/nova/+/82205321:01
opendevreviewAde Lee proposed openstack/nova stable/wallaby: Add check job for FIPS  https://review.opendev.org/c/openstack/nova/+/82789621:28
opendevreviewmelanie witt proposed openstack/nova master: Raise InstanceNotFound on fkey constraint fail saving info cache  https://review.opendev.org/c/openstack/nova/+/82694222:09
opendevreviewAde Lee proposed openstack/nova stable/wallaby: Add check job for FIPS  https://review.opendev.org/c/openstack/nova/+/82789622:12
*** dasm is now known as dasm|off22:13
*** Corwin is now known as Guest182623:39

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!