| opendevreview | melanie witt proposed openstack/nova master: testing: Run functional tests under [testenv:cover] https://review.opendev.org/c/openstack/nova/+/975346 | 01:26 |
|---|---|---|
| opendevreview | melanie witt proposed openstack/nova master: unified limits: Fix openstacksdk usage for endpoint discovery https://review.opendev.org/c/openstack/nova/+/975106 | 01:26 |
| opendevreview | melanie witt proposed openstack/nova master: TPM: prepare to bump service version for live migration https://review.opendev.org/c/openstack/nova/+/962051 | 01:36 |
| opendevreview | melanie witt proposed openstack/nova master: TPM: support live migration of `host` secret security https://review.opendev.org/c/openstack/nova/+/941483 | 01:36 |
| opendevreview | melanie witt proposed openstack/nova master: TPM: support live migration of `deployment` secret security https://review.opendev.org/c/openstack/nova/+/925771 | 01:36 |
| opendevreview | melanie witt proposed openstack/nova master: TPM: test live migration between hosts with different security https://review.opendev.org/c/openstack/nova/+/952629 | 01:36 |
| opendevreview | melanie witt proposed openstack/nova master: TPM: add late check for supported TPM secret security https://review.opendev.org/c/openstack/nova/+/956975 | 01:36 |
| opendevreview | melanie witt proposed openstack/nova master: TPM: opt-in to new TPM secret security via resize https://review.opendev.org/c/openstack/nova/+/962052 | 01:36 |
| opendevreview | melanie witt proposed openstack/nova master: DNM vtpm tempest https://review.opendev.org/c/openstack/nova/+/957477 | 01:36 |
| *** mhen_ is now known as mhen | 02:32 | |
| melwitt | I have noticed nova-next has been failing a lot on the master branch today https://zuul.opendev.org/t/openstack/builds?job_name=nova-next&project=openstack%2Fnova&branch=master&skip=0 | 03:35 |
| melwitt | error like this "Failed to start libvirt guest: libvirt.libvirtError: internal error: process exited while connecting to monitor: 2026-01-30T20:30:26.821021Z qemu-system-x86_64: -device {"driver":"scsi-block","bus":"scsi0.0","channel":0,"scsi-id":0,"lun":0,"share-rw":true,"drive":"libvirt-1-format","id":"ua-409f3368-f41a-4915-aa22-29dff53bf25b","bootindex":1,"logical_block_size":512,"physical_block_size":512}: Property | 03:37 |
| melwitt | 'scsi-block.physical_block_size' not found" | 03:37 |
| *** sambork_ is now known as sambork | 07:31 | |
| gibi | dansmith: sorry I was away | 08:51 |
| opendevreview | Merged openstack/nova stable/2024.2: Reproduce bug/2130881 https://review.opendev.org/c/openstack/nova/+/975166 | 09:05 |
| *** ykarel_ is now known as ykarel | 09:53 | |
| gibi | sean-k-mooney: left two questions in https://review.opendev.org/c/openstack/nova/+/973149 | 10:46 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Use an executor to delay STOPPED events https://review.opendev.org/c/openstack/nova/+/974445 | 10:56 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Remove spawn_after https://review.opendev.org/c/openstack/nova/+/975396 | 10:56 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Cleanup libvirt driver at service stop https://review.opendev.org/c/openstack/nova/+/975128 | 10:56 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Fix full executor warning on noname executor https://review.opendev.org/c/openstack/nova/+/975172 | 10:56 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Run nova-compute in native threading mode https://review.opendev.org/c/openstack/nova/+/965467 | 10:56 |
| gibi | gmaan: ^^ fixed up the wrapper based on your comments | 10:56 |
| sean-k-mooney | gibi: thanks ill take a look shortly | 11:14 |
| sean-k-mooney | gibi: replied inline | 11:31 |
| opendevreview | sean mooney proposed openstack/nova master: enable tap creation in nova-live-migration https://review.opendev.org/c/openstack/nova/+/975500 | 12:01 |
| sean-k-mooney | melwitt: so nova-next is using debian 13 to work around some qemu img issues with encypted volumes so it and the nova-alt-config and ceph jobs actully using a newer qemu then say tempest-integrated-comptue | 12:09 |
| sean-k-mooney | the ‘Property scsi-block.physical_block_size not found’ might be related to that | 12:09 |
| sean-k-mooney | i.e. some chagne in qemu/libvirt or perhaps we are testing a combination there that we dont have enabel elsehwere | 12:10 |
| sean-k-mooney | im not sure what that might be in this case | 12:10 |
| opendevreview | Merged openstack/nova stable/2024.2: Init virt driver before use https://review.opendev.org/c/openstack/nova/+/975167 | 12:19 |
| sean-k-mooney | melwitt: here is the relevent data https://paste.opendev.org/show/b6sWRx6VMAdktImClzCD/ ill ask about it on the virt channel and see if it rigns a bell. it looks like there is a incompatiablty between the libvirt xml we gereate and the qemu command line args | 13:01 |
| sean-k-mooney | the code on teh nova side for this looks like it has not chaged in 2+ years so this looks liek a qemu regression to me | 13:01 |
| gibi | sean-k-mooney: OK, all looks good to me | 13:31 |
| sean-k-mooney | melwitt: so this is a latent nova bug exposed by a regression in libvirt. https://github.com/libvirt/libvirt/commit/718721f0f9d33aa8abafd71bfb9ac1cfef95dc3e assuems physical_block_size is supproted for disk type = lun but it is not | 14:25 |
| sean-k-mooney | https://issues.redhat.com/browse/RHEL-145937 | 14:25 |
| sean-k-mooney | tracks fixing the libvirt validation to reject it. | 14:25 |
| sean-k-mooney | but what that means for nova is we need to update the code to not generate <blockio logical_block_size='512' physical_block_size='512'/> for dev type = lun | 14:26 |
| sean-k-mooney | when we use lun mode we are directly passhtihg though the underlying lun form the stroage server and we cant change thing like the physical_block_size the same way we cant set the serial of the exported lun. | 14:27 |
| sean-k-mooney | so we will need to make this conditoinal just as we did for serial in https://review.opendev.org/c/openstack/nova/+/918089 | 14:27 |
| sean-k-mooney | melwitt: ill write this up in a bug report adn add the gate blocker flag | 14:28 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: DNM: Troubleshoot deadlocks in threading mode https://review.opendev.org/c/openstack/nova/+/975515 | 14:42 |
| sean-k-mooney | melwitt: it looks like the xml we generate has also changed. | 14:50 |
| sean-k-mooney | the last succesfull nova-next run the the xml looked like thsi https://paste.opendev.org/show/bSZXkYi0lSv491NEjhW0/ | 14:51 |
| sean-k-mooney | so this could be a bug in a recent nova or os-brick change | 14:51 |
| sean-k-mooney | os-brick was bumped 3 weesk ago https://github.com/openstack/requirements/commit/d2639f76ce38c8a14ff7bd00d913601963646b1d | 14:54 |
| sean-k-mooney | the lass succesfuly nova-next was 2026-01-30 21:02:10 | 14:54 |
| sean-k-mooney | so that implies somethign changed in nova | 14:54 |
| sean-k-mooney | fun i foudn the code in nova that sets this and that has not chagne since it was intoduced 13 years ago :) the joys of latent bugs | 15:07 |
| opendevreview | Takashi Kajinami proposed openstack/nova master: libvirt: Use firmware auto-selection by libvirt https://review.opendev.org/c/openstack/nova/+/969132 | 15:07 |
| sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/volume/volume.py#L55-L62 | 15:07 |
| sean-k-mooney | ok foudn the root cause | 15:09 |
| sean-k-mooney | https://github.com/openstack/cinder/commit/47107b4c84b060be834d6f430d32c3ebcac7d35a | 15:09 |
| yuval_ | gibi: the iothreads merge https://review.opendev.org/c/openstack/nova/+/939254 - is it happening for any vms I create? | 15:10 |
| sean-k-mooney | yuval_: that on by default yes | 15:11 |
| yuval_ | I see | 15:11 |
| sean-k-mooney | for all new vms | 15:11 |
| sean-k-mooney | its technially on unconditionally for all vms now | 15:11 |
| yuval_ | so, I want to control how many cpu's are allocated for iothreading | 15:11 |
| sean-k-mooney | we are not suprpoting that this cycle | 15:11 |
| yuval_ | that is not possible | 15:11 |
| yuval_ | I see | 15:12 |
| sean-k-mooney | but that was going to be a disucssion topic of next steps | 15:12 |
| yuval_ | I see | 15:12 |
| sean-k-mooney | we will likely add a flavor extra spec of similar next cycle | 15:12 |
| sean-k-mooney | there isnt currently an agreement on how that shoudl look | 15:12 |
| sean-k-mooney | yuval_: do you have a prefered ux/workflow for this? | 15:12 |
| yuval_ | so, currently what I done to make it work, I would edit the xml and choose which cpu allocated for it. but its nightmare. | 15:13 |
| yuval_ | unmanageable | 15:14 |
| sean-k-mooney | and also unsupproted upsteam | 15:14 |
| yuval_ | yes yes | 15:14 |
| sean-k-mooney | the 3 proposal are as follows | 15:14 |
| sean-k-mooney | 1 add a simple hw:io_threads=n extra specs | 15:14 |
| sean-k-mooney | 2 add the ablityto add an addtionl n io thread per cidner voume | 15:15 |
| sean-k-mooney | 3 use libvirt native iothread pooling feature and add hw:io_thread_max extra spec instead | 15:15 |
| sean-k-mooney | 4 somethimg more complicated | 15:15 |
| sean-k-mooney | in all cases we would have nova select the cores for the tread to run on automaticly for you using the cpu_shared_set cores | 15:16 |
| yuval_ | so 1 - we choose which cpu's will be allocated for iothreading. 3 - there is a libvirt feature that does that? | 15:16 |
| sean-k-mooney | no 1 and 3 the value is just an integer so you chosoe the count | 15:16 |
| sean-k-mooney | not the cores | 15:16 |
| sean-k-mooney | https://libvirt.org/formatdomain.html#iothreads-allocation | 15:17 |
| sean-k-mooney | and yes libvit has a feature fo iothread pooling | 15:17 |
| sean-k-mooney | <iothread id="8" thread_pool_min="2" thread_pool_max="32"> | 15:17 |
| sean-k-mooney | so it can be static or dynimc and you can hotplug/unplug iothread at runtime | 15:17 |
| sean-k-mooney | so we could just add 1 per cidner voluem, even on attach/detach, or it could be a static value in teh flavor or a max value in the falvor with a min of 1 | 15:18 |
| sean-k-mooney | for this cycle we just hardcode 1 and we wanted operator feedback before doing anything more complex | 15:19 |
| fwiesel | gibi: I changed the vmwareapi CI to set the OS_NOVA_DISABLE_EVENTLET_PATCHING environment variable when there is a change under the eventlet-removal topic. | 15:23 |
| fwiesel | You can see it here for the latest run: http://openstack-ci-logs.global.cloud.sap/openstack-nova-973468-bj769/index.html#file=devstack@n-cpu-1.service&highlight=11 | 15:23 |
| yuval_ | sean-k-mooney: first of all that amazing | 15:23 |
| yuval_ | but I am a bit confused | 15:25 |
| yuval_ | what I know about is iothreads per disk | 15:25 |
| yuval_ | <disk …> | 15:25 |
| yuval_ | <driver name='qemu' cache=’none’ io=’native’ …> | 15:25 |
| yuval_ | <iothreads> | 15:25 |
| yuval_ | <iothread id='1'></iothread> | 15:25 |
| yuval_ | <iothread id='2'></iothread> | 15:25 |
| yuval_ | <iothread id='3'></iothread> | 15:25 |
| yuval_ | <iothread id='4'></iothread> | 15:25 |
| yuval_ | </iothreads> | 15:25 |
| sean-k-mooney | so we are not currently proposting doign that maping in nova | 15:30 |
| sean-k-mooney | we could but libvirt wil round robbin the disk across the thread iteslf | 15:30 |
| gibi | fwiesel: thanks. I will let you know when I have a WIP patch for the deadlock issue | 15:31 |
| yuval_ | so like just adding: <iothreads>4</iothreads> | 15:32 |
| gibi | fwiesel: env var looks good | 15:33 |
| sean-k-mooney | yuval_: yes | 15:37 |
| yuval_ | do we know for sure the vm will use the iothreads without mapping them to a disk? | 15:38 |
| sean-k-mooney | i belive it will but i have nto tested it | 15:40 |
| sean-k-mooney | base on the docs i expect it shoul | 15:40 |
| dansmith | melwitt: I +2d that prep-service-bump patch above the object but now that I'm thinking about it, I need to remind myself what the plan is for the dance we need to do when we enable that | 15:47 |
| dansmith | we have a call coming up in a bit, so maybe we discuss after, but... | 15:47 |
| dansmith | we're going to add the object field, but not have it used, we're going to add the decorator for the service bump, but no service bump, and then... | 15:47 |
| dansmith | in the next patch we add host support but not deployment and bump the service version | 15:47 |
| dansmith | that will open the flood gates, but only host will work | 15:48 |
| dansmith | I see in the next patch that you remove deployment from the legacy decorator.. is that the way you're preventing it from working? | 15:49 |
| dansmith | the problem with that is that API could be upgraded to the patch that supports both, but it has no way to check that compute has been upgraded to that point, | 15:49 |
| dansmith | which I know is a small window, but.. | 15:49 |
| gmaan | gibi: ack | 15:50 |
| dansmith | I think we can either (a) move the service bump to deployment before we merge host, or we could bump twice.. the latter would sort of regress host support once the api is upgraded, but at least it's safe | 15:51 |
| dansmith | hmm, missed a (b) in there | 15:51 |
| sean-k-mooney | melwitt: so https://bugs.launchpad.net/cinder/+bug/2127196 was the tracker for enableing the geometry reporting in cidner that broke nova-next | 16:04 |
| melwitt | dansmith: ok. yeah, that's how it's preventing deployment mode and that's a good point ... I feel like bumping once at deployment seems better but either way works for me | 16:06 |
| melwitt | sean-k-mooney: ahh thanks! I was curious what caused it | 16:06 |
| dansmith | melwitt: I think that's the cleanest thing, yeah. we could also go back and bump for host if we run out of time before deployment is ready and we want to make the release or something | 16:07 |
| dansmith | I haven't looked at either in detail, and not at all for months so I don't even know, just saying | 16:07 |
| melwitt | dansmith: that was my main concern heh. yeah that makes sense | 16:07 |
| dansmith | melwitt: yeah, because I think host is the most useful of these modes so I definitely don't want to hold it hostage | 16:13 |
| melwitt | ++ | 16:13 |
| raineszm | sean-k-mooney: melwitt: sorry about that. I know you're looking at the issue on the Nova side, but I put up a patch to set the default behavior of disk_geometry to unset https://review.opendev.org/c/openstack/cinder/+/975540 | 16:38 |
| sean-k-mooney | raineszm: i have had ai crete a fix on the nova side while i was on calls ill push it soon and iteragte on it | 16:54 |
| raineszm | sean-k-mooney: ack. ping me if I can help with anything on my end. | 16:55 |
| opendevreview | sean mooney proposed openstack/nova master: Fix blockio generation for LUN volumes https://review.opendev.org/c/openstack/nova/+/975542 | 16:55 |
| sean-k-mooney | raineszm: its failing 100% on the nova-next job so if ^ passes i think we are good | 16:55 |
| sean-k-mooney | ill see if i can pull that into a devstack env and test it too seperately | 16:56 |
| sean-k-mooney | that actully pretty good form a 1 shot promt https://paste.opendev.org/show/bJ8ytZ1Kb6LMmv9CCLe0/ ia sed for 2 commit but it was over egere and did it in 1 but ignoreing that it created a resonable plan and created a potially viable fix | 17:07 |
| melwitt | thanks raineszm and sean-k-mooney. I'm sending a heads up email to openstack-discuss@ | 17:13 |
| sean-k-mooney | oh ya thats a good idea to avoid rechecks | 17:13 |
| sean-k-mooney | melwitt: https://review.opendev.org/c/openstack/nova/+/975542 passed nova-next if you have time to review | 19:34 |
| melwitt | sean-k-mooney: will look, thanks | 19:36 |
| sean-k-mooney | im going to finish for today shortly so if you or others want to update it go ahead otherwise ill take a look at any comments tomorrow | 19:37 |
| opendevreview | Merged openstack/nova stable/2025.1: libvirt: Get info with abs path, rebase with rel path https://review.opendev.org/c/openstack/nova/+/958674 | 20:15 |
| opendevreview | sean mooney proposed openstack/nova master: Fix blockio generation for LUN volumes https://review.opendev.org/c/openstack/nova/+/975542 | 22:07 |
| opendevreview | sean mooney proposed openstack/nova master: Fix blockio generation for LUN volumes https://review.opendev.org/c/openstack/nova/+/975542 | 22:15 |
| sean-k-mooney | melwitt: i think ^ should now verify the regression and fix the issue. im going to head off again but it looks like the libvirt fixutre was not preserving the blockio elements | 22:17 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!