| *** macz_ has quit IRC | 00:35 | |
| *** martinkennelly has quit IRC | 00:42 | |
| *** martinkennelly has joined #openstack-nova | 00:43 | |
| *** gyee has quit IRC | 00:53 | |
| *** mlavalle has quit IRC | 01:07 | |
| *** Liang__ has joined #openstack-nova | 01:17 | |
| *** martinkennelly has quit IRC | 01:25 | |
| *** iurygregory has quit IRC | 01:26 | |
| *** k_mouza has joined #openstack-nova | 01:45 | |
| *** k_mouza has quit IRC | 01:50 | |
| *** dansmith has quit IRC | 02:34 | |
| *** dansmith has joined #openstack-nova | 02:34 | |
| *** kaisers has quit IRC | 02:44 | |
| *** rcernin has quit IRC | 02:54 | |
| *** rcernin has joined #openstack-nova | 03:13 | |
| *** sapd1 has joined #openstack-nova | 04:54 | |
| *** macz_ has joined #openstack-nova | 05:01 | |
| *** macz_ has quit IRC | 05:05 | |
| openstackgerrit | Merged openstack/nova master: Update pci stat pools based on PCI device changes https://review.opendev.org/749175 | 05:23 |
|---|---|---|
| *** sapd1 has quit IRC | 05:27 | |
| *** vishalmanchanda has joined #openstack-nova | 05:32 | |
| *** evrardjp has quit IRC | 05:33 | |
| *** evrardjp has joined #openstack-nova | 05:33 | |
| *** rcernin has quit IRC | 05:36 | |
| *** rcernin has joined #openstack-nova | 05:43 | |
| openstackgerrit | Hemanth Nakkina proposed openstack/nova stable/victoria: Update pci stat pools based on PCI device changes https://review.opendev.org/761700 | 05:44 |
| *** rcernin has quit IRC | 05:57 | |
| *** rcernin has joined #openstack-nova | 06:03 | |
| *** ratailor has joined #openstack-nova | 06:04 | |
| *** sapd1 has joined #openstack-nova | 06:05 | |
| *** rcernin has quit IRC | 06:21 | |
| openstackgerrit | Hemanth Nakkina proposed openstack/nova stable/ussuri: Update pci stat pools based on PCI device changes https://review.opendev.org/761701 | 06:23 |
| *** sapd1 has quit IRC | 06:46 | |
| *** rpittau|afk is now known as rpittau | 06:47 | |
| *** ralonsoh has joined #openstack-nova | 06:56 | |
| *** rcernin has joined #openstack-nova | 07:03 | |
| *** obre has quit IRC | 07:10 | |
| *** sapd1 has joined #openstack-nova | 07:30 | |
| *** spatel has joined #openstack-nova | 07:35 | |
| *** ociuhandu has joined #openstack-nova | 07:37 | |
| *** dklyle has quit IRC | 07:37 | |
| *** spatel has quit IRC | 07:39 | |
| *** ociuhandu has quit IRC | 07:41 | |
| openstackgerrit | Jorhson Deng proposed openstack/nova master: To deal instance with soft-deleting in _init_instance https://review.opendev.org/761264 | 07:47 |
| *** luksky has joined #openstack-nova | 07:56 | |
| *** iurygregory_ has joined #openstack-nova | 08:09 | |
| *** rcernin has quit IRC | 08:09 | |
| *** andrewbonney has joined #openstack-nova | 08:11 | |
| *** iurygregory_ is now known as iurygregory | 08:14 | |
| *** tesseract has joined #openstack-nova | 08:15 | |
| *** dtantsur|afk is now known as dtantsur | 08:42 | |
| *** rcernin has joined #openstack-nova | 08:52 | |
| *** rcernin has quit IRC | 09:06 | |
| *** martinkennelly has joined #openstack-nova | 09:11 | |
| *** ociuhandu has joined #openstack-nova | 09:17 | |
| *** ociuhandu has quit IRC | 09:29 | |
| *** ociuhandu has joined #openstack-nova | 09:30 | |
| *** ociuhandu has quit IRC | 09:34 | |
| *** ociuhandu has joined #openstack-nova | 09:34 | |
| *** derekh has joined #openstack-nova | 09:40 | |
| gibi | bauzas: you can remove your -2 on https://review.opendev.org/#/c/761639/ the fix merged to master | 09:54 |
| *** slaweq has joined #openstack-nova | 09:56 | |
| bauzas | gibi: sure okay | 09:59 |
| bauzas | done | 10:00 |
| * bauzas looks at the grenade multinode job to see why it's still -1 | 10:01 | |
| gibi | bauzas: https://zuul.opendev.org/t/openstack/build/aefd1852c72345148e6f62b65b6b49fd/log/logs/screen-n-cpu.txt?severity=4#17226 | 10:07 |
| bauzas | gibi: I was on it :) | 10:07 |
| gibi | https://bugs.launchpad.net/nova/+bug/1901739 | 10:07 |
| openstack | Launchpad bug 1901739 in OpenStack Compute (nova) " libvirt.libvirtError: internal error: missing block job data for disk 'vda'" [High,Confirmed] | 10:07 |
| bauzas | but I was looking at e-r to see whether we also had a bug | 10:07 |
| bauzas | ack, thanks | 10:08 |
| bauzas | ok, I declared it | 10:08 |
| bauzas | http://status.openstack.org/elastic-recheck/#1901739 | 10:08 |
| gibi | yep | 10:09 |
| *** ociuhandu has quit IRC | 10:09 | |
| *** ociuhandu has joined #openstack-nova | 10:10 | |
| lyarwood | `Delay in Elastic Search: Indexing behind by 100 hours` | 10:15 |
| lyarwood | I know melwitt was looking into this a while ago with infra but I think we need to flag that again | 10:15 |
| lyarwood | I hit 1901739 yesterday FWIW, might bump the bug a little | 10:15 |
| gibi | lyarwood: there is a bug that produces 100MBs of libvirt log maybe that causes logstash to get swamped https://bugs.launchpad.net/nova/+bug/1902276 | 10:16 |
| openstack | Launchpad bug 1902276 in OpenStack Compute (nova) "libvirtd going into a tight loop causing instances to not transition to ACTIVE" [Undecided,New] | 10:16 |
| lyarwood | kashyap: https://bugs.launchpad.net/nova/+bug/1901739 - I know you've been under water with other things but did you find anything more about https://bugs.launchpad.net/nova/+bug/1901739 | 10:16 |
| openstack | Launchpad bug 1901739 in OpenStack Compute (nova) " libvirt.libvirtError: internal error: missing block job data for disk 'vda'" [High,Confirmed] | 10:16 |
| lyarwood | gibi: yeah true are we hitting that often? I've not seen it outside of the example job in the bug tbh | 10:17 |
| gibi | let me look | 10:17 |
| kashyap | lyarwood: Hi, yeah; I'm under a "fun" deadline to do an LWN write-up of a couple of KVM Forum talks (including one involving Block Layer) | 10:17 |
| kashyap | lyarwood: Looking... | 10:17 |
| lyarwood | kashyap: ack np | 10:18 |
| lyarwood | kashyap: I can try to push it forward today | 10:18 |
| kashyap | lyarwood: Let me ping Peter Krempa once more to see if he can have a look at - https://kashyapc.fedorapeople.org/LP-1901739/ | 10:18 |
| * lyarwood gulps more coffee | 10:18 | |
| lyarwood | kashyap: awesome thanks | 10:18 |
| bauzas | lyarwood: the e-r indexing allows to set the hits automatically | 10:19 |
| kashyap | It's the interaction of libvirt's interface with QEMU's NBD | 10:19 |
| bauzas | lyarwood: but when you recheck with a bug number, it does increase the counter too | 10:19 |
| bauzas | just manually | 10:19 |
| lyarwood | bauzas: right my point is that we are likely hitting this more than the counter shows anyway | 10:19 |
| bauzas | correct, but fwiw like I said, i incremented the counter at least this morning | 10:20 |
| bauzas | but we can look at logstash to see the occurences if you do feel we could have more | 10:20 |
| bauzas | http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message%3A%5C%22Live%20Migration%20failure%3A%20internal%20error%3A%20missing%20block%20job%20data%20for%20disk%20%27vda%27%3A%20libvirt.libvirtError%3A%20internal%20error%3A%20missing%20block%20job%20data%20for%20disk%20%27vda%27%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22%20AND%20voting%3A1&from=864000s | 10:20 |
| lyarwood | right but if the logs aren't indexed they don't show up in logstash right? | 10:21 |
| lyarwood | so again to my point, due to the delay we are likely missing hits of this | 10:21 |
| lyarwood | that's all I was suggesting | 10:22 |
| bauzas | oh shit, you're right, my bad | 10:22 |
| bauzas | I was just thinking e-r indexing was broken, not the whole logstash munging | 10:22 |
| bauzas | (because we had specific e-r issues in the past) | 10:23 |
| kashyap | lyarwood: Peter points out the version 5.4.0. But weren't we supposed to be using libvirt-6.0.0, from Focal? | 10:29 |
| *** k_mouza has joined #openstack-nova | 10:33 | |
| *** Liang__ has quit IRC | 10:33 | |
| lyarwood | kashyap: huh | 10:38 |
| gibi | lyarwood: regarding https://bugs.launchpad.net/nova/+bug/1902276 and the big libvirtd.log. I don't found any meaningful log I can use as a signature for this bug. so i don't know how frequently octavia hit this. Either my logstash knowledge is not enough or logstash does not index libvirtd.log | 10:38 |
| openstack | Launchpad bug 1902276 in OpenStack Compute (nova) "libvirtd going into a tight loop causing instances to not transition to ACTIVE" [Undecided,New] | 10:38 |
| lyarwood | gibi: kk, I'm going to finish looking at 1901739 then I'll switch back to that | 10:39 |
| kashyap | lyarwood: What do you mean? Yeah, that was also sorta my reaction ... 5.4 is only about a year-ish old | 10:40 |
| gibi | now I go and ask infra about how we can help with the indexing delay of logstash | 10:40 |
| *** brinzhang0 has quit IRC | 10:42 | |
| lyarwood | kashyap: gah so our nova-live-migration job is still bionic | 10:42 |
| lyarwood | kashyap: not sure how I missed this | 10:42 |
| kashyap | lyarwood: Either way; Peter says 6.0.0, which uses the -blockdev infra should fix it | 10:42 |
| lyarwood | kashyap: kk well the new zuulv3 live migration job is using that and passing | 10:43 |
| kashyap | lyarwood: No problem; you do a lot of CI job work too; easy to miss | 10:43 |
| kashyap | Ah-ha; good to know! Got a link? | 10:43 |
| *** martinkennelly has quit IRC | 10:43 | |
| lyarwood | kashyap: is the fix generic or can he point to something specific just so we can document why we need to move to focal? | 10:43 |
| lyarwood | kashyap: I'm thinking of marking the current job as non-voting while we move | 10:44 |
| kashyap | lyarwood: Yeah, that makes sense; anything that gets us away from the "legacy" (-drive) | 10:46 |
| kashyap | lyarwood: He doesn't have a commit hash, afraid. | 10:46 |
| *** ociuhandu has quit IRC | 10:47 | |
| lyarwood | kashyap: kk | 10:48 |
| *** k_mouza has quit IRC | 10:52 | |
| *** k_mouza has joined #openstack-nova | 10:59 | |
| *** sapd1 has quit IRC | 11:04 | |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: zuul: Merge nova-evacuate into nova-multinode-live-migration https://review.opendev.org/755309 | 11:05 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: zuul: Replace nova-live-migration with zuulv3 jobs https://review.opendev.org/752557 | 11:05 |
| lyarwood | gibi / kashyap ; ^ so I think we need to bring this forward to resolve https://bugs.launchpad.net/nova/+bug/1901739 | 11:06 |
| openstack | Launchpad bug 1901739 in OpenStack Compute (nova) " libvirt.libvirtError: internal error: missing block job data for disk 'vda'" [High,In progress] - Assigned to Lee Yarwood (lyarwood) | 11:06 |
| lyarwood | ahead of the ceph stuff also landing | 11:06 |
| lyarwood | tl;dr move the live migration job from bionic to focal | 11:06 |
| gibi | lyarwood: looking... | 11:09 |
| openstackgerrit | Hemanth Nakkina proposed openstack/nova stable/train: Update pci stat pools based on PCI device changes https://review.opendev.org/761725 | 11:10 |
| *** dtantsur is now known as dtantsur|bbl | 11:13 | |
| *** ociuhandu has joined #openstack-nova | 11:14 | |
| *** martinkennelly has joined #openstack-nova | 11:15 | |
| openstackgerrit | Hemanth Nakkina proposed openstack/nova stable/train: Update pci stat pools based on PCI device changes https://review.opendev.org/761725 | 11:16 |
| *** ociuhandu has quit IRC | 11:18 | |
| gibi | lyarwood: so with the new focal job we can avoid 1901739 on master, but we still hit it on stable/victoria isn't it? | 11:19 |
| *** ociuhandu has joined #openstack-nova | 11:20 | |
| openstackgerrit | Hemanth Nakkina proposed openstack/nova stable/stein: Update pci stat pools based on PCI device changes https://review.opendev.org/761727 | 11:25 |
| *** ociuhandu has quit IRC | 11:30 | |
| *** ociuhandu has joined #openstack-nova | 11:33 | |
| *** ociuhandu has quit IRC | 11:40 | |
| *** tesseract has quit IRC | 11:40 | |
| *** rcernin has joined #openstack-nova | 11:40 | |
| *** tesseract has joined #openstack-nova | 11:41 | |
| lyarwood | gibi: yes, unless we backport the job there | 11:48 |
| lyarwood | gibi: that might not be a crazy idea | 11:48 |
| gibi | yeah, for victoria it is OK as we had the focal goal for V | 11:48 |
| gibi | does this bug hits older stable branches too? | 11:48 |
| lyarwood | gibi: I think I've only seen it in master thus far | 11:49 |
| lyarwood | gibi: but we haven't been landing much in stable/victoria | 11:49 |
| lyarwood | gibi: vs master that is | 11:49 |
| openstackgerrit | Hemanth Nakkina proposed openstack/nova stable/stein: Update pci stat pools based on PCI device changes https://review.opendev.org/761727 | 11:49 |
| *** ociuhandu has joined #openstack-nova | 11:55 | |
| gibi | lyarwood: ack | 12:01 |
| *** crazik has left #openstack-nova | 12:02 | |
| gibi | I totally support backporting this job changes to V | 12:03 |
| *** k_mouza has quit IRC | 12:03 | |
| lyarwood | gibi: ack thanks | 12:06 |
| *** k_mouza has joined #openstack-nova | 12:08 | |
| *** rcernin has quit IRC | 12:08 | |
| *** ociuhandu has quit IRC | 12:10 | |
| *** ratailor has quit IRC | 12:30 | |
| *** ociuhandu has joined #openstack-nova | 12:30 | |
| *** jamesdenton has quit IRC | 12:48 | |
| sean-k-mooney | elod: can you take a look at this os-vif backport and the proceeding patch when you have time https://review.opendev.org/#/c/759255/1 | 12:58 |
| gibi | gmann: does this ring a bell to you? https://zuul.opendev.org/t/openstack/build/bf932888821345c1955cda68226b1a87/log/job-output.txt#28429 was there any policy default change in glance or policy config change in devstack recently? | 13:04 |
| *** tosky has joined #openstack-nova | 13:07 | |
| elod | sean-k-mooney: sure! (at first glance the 'refactor' topic tricked me, but as far as i see it's a bugfix :)) | 13:19 |
| *** sapd1 has joined #openstack-nova | 13:22 | |
| sean-k-mooney | ya kind of its pulling out some common code but its need for the follow up | 13:30 |
| *** k_mouza has quit IRC | 13:32 | |
| *** ociuhandu has quit IRC | 13:37 | |
| *** k_mouza has joined #openstack-nova | 13:47 | |
| *** ociuhandu has joined #openstack-nova | 13:56 | |
| mnasiadka | good afternoon | 13:57 |
| mnasiadka | does Nova support having ephemeral volumes from different backends? (e.g. one ceph, second from a local filesystem) | 13:57 |
| *** ociuhandu has quit IRC | 13:59 | |
| *** ociuhandu has joined #openstack-nova | 13:59 | |
| *** eharney_ has joined #openstack-nova | 14:07 | |
| *** kaisers has joined #openstack-nova | 14:08 | |
| *** ociuhandu has quit IRC | 14:10 | |
| *** eharney has quit IRC | 14:10 | |
| *** nweinber has joined #openstack-nova | 14:17 | |
| *** dtantsur|bbl is now known as dtantsur | 14:20 | |
| lyarwood | mnasiadka: for the libvirt virt driver no, we only support a single images_type configurable | 14:22 |
| lyarwood | mnasiadka: for multibackend support you should use cinder | 14:23 |
| mnasiadka | lyarwood: that's what I considered as an alternative | 14:23 |
| mnasiadka | lyarwood: thanks | 14:23 |
| lyarwood | mnasiadka: np | 14:23 |
| sean-k-mooney | stephenfin:.... https://github.com/openstack/nova/commit/b8695de6da56db42b83b9d9d4c330148766644be rewrote the pci generation that is why my fix is broken | 14:25 |
| sean-k-mooney | i guess that needs to go back to queens too | 14:26 |
| *** jamesdenton has joined #openstack-nova | 14:32 | |
| bauzas | gibi: others, some bugfix that could be backported https://review.opendev.org/#/c/758470/ | 14:44 |
| bauzas | easy peasy IMHO | 14:44 |
| * bauzas still works on the RPC API 6.0 bump | 14:44 | |
| gibi | bauzas: will check soonish | 14:45 |
| bauzas | no rush | 14:46 |
| bauzas | but ta | 14:46 |
| gibi | it is Friday I run out of rush alreay ;) | 14:47 |
| *** artom has quit IRC | 14:48 | |
| openstackgerrit | sean mooney proposed openstack/nova master: Fix unplugging VIF when migrate/resize VM https://review.opendev.org/751642 | 14:49 |
| openstackgerrit | sean mooney proposed openstack/nova master: libvirt: delegate ovs plug to os-vif https://review.opendev.org/602432 | 14:49 |
| *** martinkennelly has quit IRC | 14:49 | |
| *** ociuhandu has joined #openstack-nova | 14:49 | |
| *** martinkennelly has joined #openstack-nova | 14:49 | |
| lyarwood | I'm not sure if anyone has been looking at this but I can't see an obvious issue with nova-ceph-multistore aside from instance spawns taking a long time overall | 14:52 |
| lyarwood | I'll look at another example failure after a downstream call and file a bug to track these timeouts | 14:53 |
| gmann | gibi: let me check. we did change one glance policy in nove ceph multistore job. | 14:54 |
| gmann | gibi: ah this is same copy_image policy. checking why it is failing. | 14:54 |
| *** ociuhandu has quit IRC | 14:55 | |
| gibi | gmann: thanks! | 14:55 |
| bauzas | is it only me or you do you get a shit of exceptions when running functional and unittests locally with nova.storage.rbd_utils about the 'rados' and 'rbd' packages missing ? | 14:56 |
| bauzas | or do* | 14:56 |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova stable/victoria: Modify glance's copy_image permission for nova-ceph-multistore https://review.opendev.org/761755 | 14:56 |
| gibi | bauzas: I see it too | 14:56 |
| gibi | and I remember seeing a patch fixing it | 14:56 |
| bauzas | I tested with both py38 and py39 | 14:56 |
| bauzas | but that's mostly annoying | 14:56 |
| gmann | gibi: bauzas lyarwood backporting the nova-ceph-multistore fix https://review.opendev.org/#/c/761755/ | 14:56 |
| gibi | bauzas: https://review.opendev.org/#/c/746873/ | 14:57 |
| gmann | Tempest test merged before we backported this | 14:57 |
| gibi | gmann: thanks! | 14:58 |
| bauzas | gibi: meh, I should have it in my local repo | 14:58 |
| * bauzas sees it was merged 3 weeks ago | 14:58 | |
| gibi | bauzas: then maybe it is not the fix but it is the patch that generates the exception in the first place | 14:58 |
| lyarwood | gmann: ah did that bork the job on stable? | 14:58 |
| gmann | gibi: lyarwood nova-ceph-multistore job is from victoria so backporting until victoria is needed | 14:59 |
| gmann | lyarwood: yeah - https://zuul.opendev.org/t/openstack/build/bf932888821345c1955cda68226b1a87/ | 14:59 |
| gmann | tempest test which started running in nova-ceph-multistore on master as well on stable/victoria started running and needed this change on nova side | 14:59 |
| bauzas | gibi: hah, that's maybe because the exception is not the same | 15:00 |
| bauzas | ah no | 15:00 |
| bauzas | nevermind | 15:00 |
| bauzas | we log the exception | 15:00 |
| lyarwood | gmann: ack I was going to look at that failure so thanks :) | 15:01 |
| bauzas | hence the stracktrace | 15:01 |
| bauzas | but that's horribly annoying | 15:01 |
| bauzas | lyarwood: any ways for us to not getting a shitty number of exception lines anytime we use tox ? | 15:01 |
| gmann | lyarwood: we added tempest test to copy-image for private images also (https://review.opendev.org/#/c/742546/23) which needed this policy to be open for everyone | 15:01 |
| bauzas | because of rados and rbd missing | 15:02 |
| bauzas | lyarwood: context being https://review.opendev.org/#/c/746873/4/nova/storage/rbd_utils.py | 15:02 |
| lyarwood | gmann: ack | 15:02 |
| *** martinkennelly has quit IRC | 15:03 | |
| lyarwood | bauzas: we could add some logic to skip the check based on an ENV var or something in test envs? | 15:03 |
| lyarwood | bauzas: or we add them as test requirements? | 15:04 |
| lyarwood | bauzas: even though they are virt driver specific | 15:04 |
| bauzas | lyarwood: context is http://paste.openstack.org/show/799789/ | 15:05 |
| lyarwood | bauzas: yeah I;ve seen it | 15:05 |
| bauzas | lyarwood: IMHO we should lazy load the modules | 15:05 |
| bauzas | for rdb and rados | 15:05 |
| bauzas | or making them needed for all *nova* | 15:06 |
| bauzas | but, definitely, the former | 15:06 |
| bauzas | if they are optional | 15:06 |
| bauzas | (and only for a specific driver) | 15:06 |
| lyarwood | bauzas: we lazy loaded before silently and it was a PITA to debug issues when they failed to import | 15:06 |
| lyarwood | bauzas: assuming that's what you mean | 15:06 |
| lyarwood | bauzas: could we not just add them as test deps? | 15:07 |
| bauzas | lyarwood: then, lazy load them by creating the libvirt object | 15:07 |
| bauzas | when initting | 15:07 |
| *** suryasingh has joined #openstack-nova | 15:07 | |
| bauzas | gibi: ^ to be honestly, I'm against https://review.opendev.org/#/c/746873/4/nova/storage/rbd_utils.py | 15:07 |
| bauzas | because we create more problems for folks that don't use ceph than we fix the needed ones | 15:08 |
| openstackgerrit | Merged openstack/nova master: zuul: Merge nova-evacuate into nova-multinode-live-migration https://review.opendev.org/755309 | 15:09 |
| dansmith | bauzas: me too, that seems crazy | 15:09 |
| dansmith | surely we could check images_type and decide whether or not to explode there? | 15:10 |
| gibi | bauzas, lyarwood, dansmith: can we move these import into one of the classes in the rbd_utils? | 15:10 |
| gibi | dansmith: it is not exploding the driver, it is logging the import error | 15:11 |
| dansmith | gibi: I think the point here was to make it explode before those get loaded, but that's not really okay, IMHO | 15:11 |
| bauzas | gibi: the problem is that when using import directly, you *need* rdb | 15:11 |
| bauzas | by any module | 15:11 |
| bauzas | hence either lazy-loading it | 15:11 |
| bauzas | or having other ways to support it like we did by telling none | 15:11 |
| dansmith | yeah, go back to the old way and make something else check to see if the rbd import is None during early init, if we're in a dependent code path | 15:12 |
| *** vishalmanchanda has quit IRC | 15:12 | |
| bauzas | by providing an exception you say "sorry, but you need it" | 15:12 |
| dansmith | check config, or put it in a class we only load if we're using ceph | 15:12 |
| bauzas | that too | 15:13 |
| bauzas | there are multiple ways to make sure it's optional | 15:13 |
| gibi | filed a bug https://bugs.launchpad.net/nova/+bug/1903316 | 15:16 |
| openstack | Launchpad bug 1903316 in OpenStack Compute (nova) "excessive logging of rbd and rados import errors" [Undecided,New] | 15:16 |
| *** ociuhandu has joined #openstack-nova | 15:16 | |
| *** rpittau is now known as rpittau|afk | 15:18 | |
| gibi | feel free to push a fix against it ^^ | 15:19 |
| gibi | if nobody then I might get to it at some point next week | 15:20 |
| *** legochen_ has joined #openstack-nova | 15:21 | |
| *** legochen_ is now known as legochen | 15:21 | |
| bauzas | gibi: I can try to fix it | 15:22 |
| bauzas | but ideally, I'd love to understand the dynamics behind us logging this as an exception and what we wanted to fix previously | 15:22 |
| gibi | ack, thanks | 15:22 |
| gibi | I hope lyarwood can provide such context ^^ | 15:23 |
| bauzas | because I don't see a problem with lazy-loading at init time | 15:23 |
| bauzas | we could even make nova-compute refuse to start if config option says "you should have it" | 15:23 |
| dansmith | right | 15:24 |
| lyarwood | bauzas: the context there is that we previously silently failed without logging any context about why the import failed. | 15:24 |
| dansmith | or .. log the warning if configured and missing | 15:24 |
| lyarwood | bauzas: we had a few downstream reports where this failed even with the modules in the container image etc. | 15:24 |
| lyarwood | hmm I was sure there was a reason we couldn't check the config before doing this import | 15:25 |
| * lyarwood looks again | 15:25 | |
| dansmith | lyarwood: well before it was above the CONF, | 15:25 |
| dansmith | lyarwood: but also, where we actually need it, check to see if the import is None, if so, try to import it again and log the exception there just to get the evidence | 15:25 |
| dansmith | i.e. if rbd_utils.rbd is None: import rbd | 15:26 |
| lyarwood | dansmith: yup understood | 15:27 |
| bauzas | dansmith: lyarwood: honestly, I'm on the side we should fix the problem by making rdb and rados mandatory for the use of the right option, and make nova-compute hardstop if not | 15:32 |
| lyarwood | bauzas: it already does | 15:32 |
| bauzas | not in this module then, right? | 15:32 |
| lyarwood | no, we just raise RuntimeError | 15:33 |
| bauzas | when starting the compute service ? | 15:33 |
| lyarwood | from this module that iirc stops the service eventually yes | 15:33 |
| lyarwood | if rbd is used | 15:33 |
| bauzas | the above try/catch is made at import time, not runtime | 15:34 |
| bauzas | so unless we fail ungracefully there, we do check it at run time | 15:34 |
| lyarwood | https://github.com/openstack/nova/blob/c0fe95fcc5aec99a83dd57093dc230ef67b36b39/nova/storage/rbd_utils.py#L139-L140 | 15:35 |
| bauzas | ack, run time | 15:35 |
| bauzas | and when we create the instancez | 15:35 |
| bauzas | which, I suspect, is made at init | 15:35 |
| bauzas | lyarwood: okay, so indeed, we're all good, but why couldn't we just make the imports being made at run time just before this check ? | 15:36 |
| lyarwood | I'm reworking this now | 15:37 |
| lyarwood | the only issue now is that we need to ignore F401 as the imports aren't directly used in __init__ | 15:39 |
| *** artom has joined #openstack-nova | 15:41 | |
| bauzas | lyarwood: why ? | 15:44 |
| bauzas | I'm confused | 15:44 |
| bauzas | hah, so you have a rados object which isn't used | 15:44 |
| lyarwood | no idea, must be the way the pep8 rule is written, rados is used within the class but it's also saying that the import isn't used | 15:45 |
| lyarwood | both imports are listed as not used | 15:45 |
| lyarwood | I've #noqa'd the lines anyway | 15:45 |
| bauzas | lyarwood: use the importlib module | 15:45 |
| sean-k-mooney | stephenfin: test | 15:46 |
| stephenfin | HexChat didn't crash \o/ | 15:46 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: rbd: Only log import failures when the RbdDriver is used https://review.opendev.org/761762 | 15:46 |
| sean-k-mooney | you should still lookinto weechat | 15:46 |
| sean-k-mooney | but glad its working | 15:46 |
| stephenfin | meh, I live the GUI in a separate window | 15:46 |
| dansmith | lyarwood: that's much better, IMHO | 15:47 |
| lyarwood | yup not sure what I was thinking the first time round | 15:47 |
| sean-k-mooney | stephenfin: by the way https://review.opendev.org/#/q/topic:bug/1895220+(status:open+OR+status:merged) are ready for you to review again | 15:47 |
| sean-k-mooney | they are still running in ci however | 15:48 |
| dansmith | lyarwood: now I have to figure out what drug I can take to let me approve that without a test | 15:48 |
| lyarwood | dansmith: hmm I can work something out now that it's in the class | 15:48 |
| dansmith | lyarwood: I'm thinking it'll be hard to tickle the error case with a mock, so I should probably just let it go... | 15:48 |
| bauzas | wait, you were having global objects | 15:48 |
| *** jangutter_ has quit IRC | 15:48 | |
| lyarwood | true it's awkward | 15:48 |
| dansmith | we used to use importutils more for stuff like that so we can mock that instead of the import keyword | 15:49 |
| bauzas | here, you are making them class attributes | 15:49 |
| dansmith | bauzas: no | 15:49 |
| bauzas | dansmith: right, i just said it | 15:49 |
| lyarwood | wait, the imports aren't global? | 15:49 |
| dansmith | they are | 15:49 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova stable/train: Test for disabling greendns https://review.opendev.org/761763 | 15:49 |
| bauzas | lyarwood: importlib does the magic for you | 15:49 |
| bauzas | or importutils, either way | 15:50 |
| dansmith | the imports are global because if they really import, we'll have gotten them up top, | 15:50 |
| dansmith | the import in the class is purely to generate the exception for logging | 15:51 |
| dansmith | using an import helper would be good for the mocking case, but otherwise the noqa seems fine to me, as I'm sure it's just complaining about the unused variable | 15:51 |
| * bauzas double-checking this detail | 15:51 | |
| dansmith | you might be able to avoid it with "import rbd as _rbd" but it doesn't matter | 15:51 |
| sean-k-mooney | i think its an unused import rather then unsued varibale so not suer the as will fix it but noqa i think is still fine | 15:52 |
| sean-k-mooney | preferably with a comment as to why its there | 15:52 |
| * lyarwood adds a comment | 15:54 | |
| sean-k-mooney | for what its worth i prefer not using importlib/utiles simply due to the grep factor e.g. if im grepin for import rbd i wont see it but it is cleaner to use there helpers | 15:54 |
| bauzas | hmmmm | 15:55 |
| *** dklyle has joined #openstack-nova | 15:55 | |
| sean-k-mooney | that not really important but its why i personlly dont tend to use it | 15:55 |
| bauzas | my own test shows me that the import isn't global | 15:55 |
| sean-k-mooney | bauzas: the import in the fuction is not the top levle one is | 15:55 |
| dansmith | bauzas: the import is on L32 | 15:55 |
| dansmith | it's global | 15:55 |
| lyarwood | bauzas: the imports at the top of the file | 15:55 |
| bauzas | http://paste.openstack.org/show/799791/ | 15:55 |
| lyarwood | bauzas: we only retry these later in the class if the originals failed | 15:55 |
| dansmith | this ^ | 15:56 |
| bauzas | lyarwood: right, so my purpose is to say that if you weren't able to import them at import time, you will get them at run time but only within the class namespace | 15:56 |
| sean-k-mooney | and that is to get the reason it failed | 15:56 |
| dansmith | bauzas: no, we won't import them at runtime either | 15:56 |
| bauzas | I maybe missed the point then | 15:57 |
| lyarwood | so we raise | 15:57 |
| lyarwood | L139 | 15:57 |
| bauzas | oh, that's just for logging purpose ? | 15:57 |
| lyarwood | right | 15:57 |
| dansmith | bauzas: the point is not to "try again", the point is to generate the import exception just for logging | 15:57 |
| bauzas | hah, ok | 15:57 |
| bauzas | hence the F401 | 15:57 |
| bauzas | because the local import wasn't used | 15:57 |
| sean-k-mooney | its something danpb suggested wehn we were trying to figure out why imports didnt work | 15:58 |
| bauzas | lyarwood: but fwiw, imports aren't global | 15:58 |
| sean-k-mooney | we think it was due to memory issues in the custoemr env since the had a bunc of OOM issue at the time | 15:58 |
| sean-k-mooney | but we dont have any logs to say | 15:58 |
| sean-k-mooney | bauzas: import happen in your current scope | 15:58 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: rbd: Only log import failures when the RbdDriver is used https://review.opendev.org/761762 | 15:58 |
| bauzas | sean-k-mooney: right, that's what I said | 15:59 |
| sean-k-mooney | so if you are at global scope they are if you are in function scope tehy are just in that scope | 15:59 |
| sean-k-mooney | although that is only half though | 15:59 |
| sean-k-mooney | *true | 15:59 |
| bauzas | either way | 15:59 |
| sean-k-mooney | the modul is actully imported gloablly including inialising all its global state it just made available in your local scope | 15:59 |
| sean-k-mooney | if you import it somewere else that modules globalse state is still there | 16:00 |
| bauzas | ok, the docstring helps | 16:00 |
| bauzas | dansmith: honestly, I won't bother for tests here | 16:01 |
| * lyarwood moves on | 16:01 | |
| dansmith | lyarwood: couple comments on your comments | 16:02 |
| * lyarwood moves back | 16:02 | |
| dansmith | bauzas: well, I understand but a typo in a log message in an exception handler really defeats the point of the helper.. but I'd rather get it fixed than complicate it for testability | 16:02 |
| bauzas | dansmith: fwiw I agree with your comments | 16:03 |
| bauzas | I guess we need both rbd and rados modules to be present in order to work | 16:04 |
| bauzas | so only one exception catch should be enough | 16:04 |
| dansmith | ...like in the actual import.. I'm not sure why it's split in the original change or this one, but maybe lyarwood has a reason | 16:04 |
| lyarwood | yeah I don't recall why, likely just leaving specific breadcrumbs but that's part of the issue with the first patch so I'll merge them back togther now | 16:07 |
| bauzas | I guess because the log is different | 16:07 |
| dansmith | sure, but the exception trace will tell you which one | 16:07 |
| bauzas | yup, I was able to tell it | 16:07 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: rbd: Only log import failures when the RbdDriver is used https://review.opendev.org/761762 | 16:13 |
| dansmith | gibi: ^ | 16:14 |
| * lyarwood really does move on now | 16:14 | |
| gibi | looking.. | 16:14 |
| dansmith | lyarwood: wait, one more thing | 16:14 |
| * lyarwood really does move back now | 16:14 | |
| lyarwood | ^_^ | 16:15 |
| bauzas | dansmith: oh shit, I +Wd | 16:16 |
| dansmith | bauzas: I was just joking | 16:16 |
| bauzas | hah | 16:16 |
| * bauzas whispers | 16:16 | |
| * bauzas was sweating | 16:16 | |
| dansmith | but I figured gibi should look too | 16:16 |
| bauzas | anyway, time to call it a wrap | 16:16 |
| bauzas | gibi: please look at https://review.opendev.org/761762 which is going to the gate for the next hours | 16:17 |
| bauzas | anyway, \o | 16:17 |
| bauzas | man, I just found we powerplayed /o\ | 16:18 |
| bauzas | apologies | 16:18 |
| gibi | no worries. It looks good to me too | 16:19 |
| openstackgerrit | Sylvain Bauza proposed openstack/nova master: rbd: Only log import failures when the RbdDriver is used https://review.opendev.org/761762 | 16:19 |
| gibi | then I will add the Closes-Bug tag the the commit message ^^ | 16:20 |
| *** artom has quit IRC | 16:20 | |
| gibi | or not :D | 16:20 |
| gibi | whatever | 16:20 |
| gibi | bauzas: thanks for the commit message update | 16:21 |
| *** mlavalle has joined #openstack-nova | 16:23 | |
| *** k_mouza has quit IRC | 16:28 | |
| *** k_mouza has joined #openstack-nova | 16:29 | |
| *** macz_ has joined #openstack-nova | 16:30 | |
| *** gyee has joined #openstack-nova | 16:33 | |
| stephenfin | sean-k-mooney: comment on https://review.opendev.org/#/c/751642/8/nova/virt/libvirt/driver.py@1588 | 16:46 |
| *** ociuhandu_ has joined #openstack-nova | 16:52 | |
| sean-k-mooney | i was asked to log it and assert the excption is caught | 16:53 |
| sean-k-mooney | if i remove the logging i cant do either | 16:53 |
| sean-k-mooney | im useing log.debug so i dont think that prints the stack trace the way log.excpetion does | 16:54 |
| *** ociuhandu has quit IRC | 16:55 | |
| sean-k-mooney | stephenfin: im expecting it to log the excption message not the traceback | 16:55 |
| *** ociuhandu_ has quit IRC | 16:56 | |
| stephenfin | sean-k-mooney: aha, right you are. You need to pass a flag for that to happen | 16:57 |
| *** artom has joined #openstack-nova | 17:02 | |
| *** dtantsur is now known as dtantsur|afk | 17:13 | |
| openstackgerrit | Merged openstack/os-vif stable/victoria: Refactor code of linux_net to more cleaner and increase performace https://review.opendev.org/756406 | 17:21 |
| openstackgerrit | Merged openstack/os-vif stable/victoria: Fix - os-vif fails to get the correct UpLink Representor https://review.opendev.org/759255 | 17:21 |
| *** k_mouza has quit IRC | 17:27 | |
| *** k_mouza has joined #openstack-nova | 17:34 | |
| *** sapd1 has quit IRC | 17:36 | |
| *** derekh has quit IRC | 17:38 | |
| stephenfin | gmann, johnthetubaguy: Are either of you going to have time to review https://review.opendev.org/#/q/topic:secure-rbac+(status:open+OR+status:merged)+project:openstack/placement with me. Would definitely benefit from your input | 17:44 |
| *** k_mouza has quit IRC | 17:45 | |
| *** suryasingh has quit IRC | 17:45 | |
| stephenfin | lyarwood: I can respin https://review.opendev.org/#/c/761762/ real quick if you want? | 17:47 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: rbd: Only log import failures when the RbdDriver is used https://review.opendev.org/761762 | 17:51 |
| *** k_mouza has joined #openstack-nova | 18:05 | |
| *** andrewbonney has quit IRC | 18:07 | |
| lyarwood | stephenfin: yeah thanks, slightly confused how this worked before without this? | 18:08 |
| stephenfin | you were only reading the value of the variable | 18:09 |
| stephenfin | the global keyword is necessary when you write to it | 18:09 |
| *** k_mouza has quit IRC | 18:10 | |
| stephenfin | foo = None | 18:10 |
| stephenfin | def bar(): | 18:10 |
| stephenfin | foo = 'test' | 18:10 |
| stephenfin | bar() | 18:10 |
| stephenfin | print(foo) | 18:10 |
| stephenfin | ^ | 18:10 |
| lyarwood | right so the error is misleading, it's not the read in the conditional that's at fault, it's the import trying to overwrite it? | 18:12 |
| stephenfin | sort of | 18:12 |
| stephenfin | by trying to write to it, the variable becomes locally scoped | 18:12 |
| lyarwood | yup got it | 18:13 |
| stephenfin | flake8 can't catch these kind of issues, unfortunately | 18:13 |
| *** ralonsoh has quit IRC | 18:13 | |
| sean-k-mooney | lyarwood: before it was locally importing every time and using the local import if it was not visable | 18:14 |
| sean-k-mooney | you can read gloabls without the keyword | 18:15 |
| sean-k-mooney | you just can write to them it creates a local alias of it | 18:15 |
| lyarwood | sean-k-mooney: yeah got it | 18:16 |
| lyarwood | stephenfin: still failing btw | 18:17 |
| stephenfin | darn, same error? | 18:17 |
| lyarwood | stephenfin: different, this looks more like a test issue | 18:17 |
| lyarwood | stephenfin: https://7d2bb7e91c376d06349f-0e5efcd7961a07f1b6cc368c11ff4914.ssl.cf1.rackcdn.com/761762/5/check/openstack-tox-cover/b207b75/testr_results.html | 18:18 |
| lyarwood | stephenfin: I need to drop now, feel free to leave that and I can address it on Monday | 18:19 |
| stephenfin | Yeah, it's not the easy fix we though so let's leave it | 18:19 |
| stephenfin | lyarwood: Have a good one o/ | 18:19 |
| lyarwood | and you \o | 18:20 |
| sean-k-mooney | damit... i dont have precommit on my laptop and didnt run pep8 | 18:23 |
| mnaser | i've been confused about this for aw hile | 18:28 |
| mnaser | say i have a queens install and enabled_vgpu_types was changed -- do i have to go muck about the placement service? | 18:29 |
| mnaser | i cant find where this is 'defined' and being 'published' to placement | 18:29 |
| mnaser | inventories only shows 3 resorce classes which seem to be disk/vcpu/mem | 18:32 |
| gmann | stephenfin: ack, lance and I are going to have call on Tuesday for testing strategy for these changes, at least we need unit test like we did in nova. | 18:39 |
| gmann | stephenfin: these are in my list for next week. | 18:40 |
| gmann | these are meeting details - http://lists.openstack.org/pipermail/openstack-discuss/2020-November/018512.html | 18:40 |
| *** mlavalle has quit IRC | 18:41 | |
| *** mlavalle has joined #openstack-nova | 18:43 | |
| *** dave-mccowan has quit IRC | 18:49 | |
| *** kaisers has quit IRC | 18:59 | |
| *** k_mouza has joined #openstack-nova | 19:14 | |
| *** k_mouza has quit IRC | 19:15 | |
| mnaser | `virsh nodedev-dumpxml` shows availableInstances = 0 | 19:40 |
| mnaser | so that's .. progress | 19:40 |
| *** luksky has quit IRC | 19:47 | |
| *** luksky has joined #openstack-nova | 19:48 | |
| *** nweinber has quit IRC | 20:01 | |
| *** legochen has quit IRC | 20:06 | |
| mnaser | ok turns out there was devices that were not removed | 20:15 |
| openstackgerrit | Merged openstack/nova master: zuul: Replace nova-live-migration with zuulv3 jobs https://review.opendev.org/752557 | 20:47 |
| *** ociuhandu has joined #openstack-nova | 20:53 | |
| *** ociuhandu has quit IRC | 21:02 | |
| *** martinkennelly has joined #openstack-nova | 21:08 | |
| *** martinkennelly has quit IRC | 21:12 | |
| *** martinkennelly has joined #openstack-nova | 21:12 | |
| *** martinkennelly has quit IRC | 21:16 | |
| *** martinkennelly has joined #openstack-nova | 21:16 | |
| *** melwitt is now known as jgwentworth | 21:26 | |
| *** whoami-rajat__ has quit IRC | 21:26 | |
| *** martinkennelly has quit IRC | 21:26 | |
| *** martinkennelly has joined #openstack-nova | 21:31 | |
| *** martinkennelly has quit IRC | 21:34 | |
| *** raildo has quit IRC | 22:00 | |
| *** tbachman has quit IRC | 22:07 | |
| *** tesseract has quit IRC | 22:12 | |
| *** JamesBen_ has quit IRC | 22:12 | |
| *** luksky has quit IRC | 22:26 | |
| *** k_mouza has joined #openstack-nova | 23:02 | |
| *** hamalq has joined #openstack-nova | 23:05 | |
| *** k_mouza has quit IRC | 23:06 | |
| *** dave-mccowan has joined #openstack-nova | 23:07 | |
| *** tbachman has joined #openstack-nova | 23:16 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!