| mikal | sean-k-mooney: I am unsure if I should recheck the final spice-direct patch now that melwitt's tempest fix has landed or not. Discuss. | 07:48 |
|---|---|---|
| opendevreview | Sasha Malberg proposed openstack/nova master: Update SHELVED VM az when attaching disk https://review.opendev.org/c/openstack/nova/+/942561 | 09:50 |
| opendevreview | Vasyl Saienko proposed openstack/nova master: Fix serial console for ironic https://review.opendev.org/c/openstack/nova/+/942575 | 11:38 |
| opendevreview | Amit Uniyal proposed openstack/nova master: Reproducer for cold migration on shared storage https://review.opendev.org/c/openstack/nova/+/940304 | 12:02 |
| opendevreview | Vasyl Saienko proposed openstack/nova master: Fix serial console for ironic https://review.opendev.org/c/openstack/nova/+/942575 | 12:23 |
| opendevreview | Takashi Kajinami proposed openstack/nova master: Use consistent program name for wsgi scripts and entry points https://review.opendev.org/c/openstack/nova/+/942605 | 15:35 |
| opendevreview | Takashi Kajinami proposed openstack/nova master: Use consistent program name for wsgi scripts and entry points https://review.opendev.org/c/openstack/nova/+/942605 | 15:41 |
| sean-k-mooney | Uggla: can you appove this https://review.opendev.org/c/openstack/releases/+/941862 to do the final release of os-vif | 15:46 |
| Uggla | sean-k-mooney, I'll have a look right after the call and will approve it. | 15:48 |
| sean-k-mooney | ack | 15:48 |
| sean-k-mooney | zull is still runing on it anyway | 15:48 |
| Uggla | sean-k-mooney, +1 for me. | 16:13 |
| sean-k-mooney | cool thanks :) | 16:13 |
| danfai | hi, do you think there is a way to make nova aware of a change of the vif_type when live migrating? I'm currently testing live migrations between linuxbridge and OVS/OVN, which seems doable (for our case) but the bridge name is wrong | 16:17 |
| sean-k-mooney | danfai: we do not allow changing the vif_type for port attached to a nova instnace | 16:46 |
| sean-k-mooney | danfai: so you cannot change it form linxu bridge ot ovs/ovn | 16:46 |
| sean-k-mooney | that is unsupproted | 16:47 |
| sean-k-mooney | there are som hacky script to do that on the neutron side | 16:47 |
| sean-k-mooney | but its offically unsuprpoted form a nova perspective | 16:47 |
| pas-ha[m] | Hi all, I'm interested in the status of the 'local disk encryption' work. https://review.opendev.org/q/topic:%22specs/yoga/approved/ephemeral-encryption-libvirt%22 . AFAIU it was kind of abandoned after that CVE flurry last summer... Are there any plans to continue woking on it? can I help with it somehow (rebasing, fixing comments etc)? cc melwitt | 16:59 |
| sean-k-mooney | we coudl add support for it but we might need to chnage the multiple port bidning api respoce to incude vif_type | 17:03 |
| * melwitt pas-ha[m]: hi, yes it has been postponed due to the CVE stuff and the nova team agreed some amount of refactoring of the existing disk image code should be done before we resume the disk encryption series. I have some proposal for the refactor part that I have not formally pushed yet but will be pushing in the near future. once that is reviewed and agreed, local disk encryption can be picked up again. I don't yet know if it will | 17:17 | |
| * melwitt be prioritized however | 17:17 | |
| sean-k-mooney | pas-ha[m]: where we are with that is before adding that functionality there is a set of technial debt that we would like to clean up | 17:23 |
| sean-k-mooney | basically our concern is the CVE showed there is a lot of tech debt in the iamge backend code and we are concerned about making it even more complext by adding local disk encyption | 17:24 |
| pas-ha[m] | yeah, I bet at least one of those patches there is my own.. (the ISO one) | 17:25 |
| sean-k-mooney | as melwitt noted the priority of the feature was downgraded and the piority fo the refactorign raised but so far we have not commited to when eitehr will be done | 17:25 |
| sean-k-mooney | pas-ha[m]: which iso one | 17:25 |
| sean-k-mooney | there is at least one i was hoping to land this cycle | 17:25 |
| pas-ha[m] | is there any discussion /docs/aims on the planned refactor you could point me to? | 17:25 |
| pas-ha[m] | sean-k-mooney: https://review.opendev.org/c/openstack/nova/+/909611 | 17:26 |
| sean-k-mooney | ah yep thats the one im +2 on. so im hoping melwitt or dan will have time to review that before FF or at least before RC1 | 17:26 |
| pas-ha[m] | bottom line - we are really interested in this feature, as the only proper alternative ATM for local ephemeral encryption is LVM and it starts to be a major operational PITA (no migrations/resize etc).. so if there's anything I could help with I'm all ears 🙂 | 17:29 |
| melwitt | pas-ha[m]: to my understanding, the refactor will be basically the first 3 patches here (maybe just the first 2, will depend on review) https://review.opendev.org/c/openstack/nova/+/925635 obviously I need to rebase and clean up them and I'll write up a blueprint about it. if a spec is needed I will also write a spec | 17:29 |
| pas-ha[m] | thanks, will dive in | 17:30 |
| melwitt | sean-k-mooney: ack I will look at the iso patch | 17:30 |
| sean-k-mooney | melwitt: tldr it will make sure we attach isos as a cdrom rather then as a block device | 17:30 |
| sean-k-mooney | its more subtle then that but that the main part | 17:31 |
| melwitt | ok, thanks | 17:32 |
| pas-ha[m] | fwiw, I recently tried to implement a cold migration / resize for LVM image backend... and it even kind of worked but was ugly AF... really not worth actually doing for sure | 17:32 |
| melwitt | ah, interesting | 17:34 |
| sean-k-mooney | pas-ha[m]: it used to wrok a very long time ago | 17:35 |
| sean-k-mooney | it broke at some point but i have no idea when | 17:35 |
| sean-k-mooney | i dont see why we woudl not supprot it for lvm again eventurlly | 17:36 |
| sean-k-mooney | we just need to scp/rsync the data | 17:36 |
| sean-k-mooney | it very similar to raw/qcow local storage migration | 17:36 |
| sean-k-mooney | we have apis to do this | 17:36 |
| sean-k-mooney | via the remotefs dirvers | 17:37 |
| sean-k-mooney | there is a copy image fucntion for this that lvm shoudl be implementing | 17:37 |
| sean-k-mooney | or using | 17:37 |
| sean-k-mooney | the lvm image backend dirver is the highest perfromign stroage option nova currently supports | 17:38 |
| sean-k-mooney | espically for write intensive operations | 17:38 |
| pas-ha[m] | yeah, kind of what I did. the main problem is that I did not thought of a correct way of dd | ssh | dd to skip the intermediate file, so I was dd to local file, scp to remote, on remote make a new lvm and dd local file into that.. | 17:38 |
| sean-k-mooney | you do not need an a intermediate file | 17:39 |
| sean-k-mooney | dd directly to the remote lvm volume shoudl be enough | 17:39 |
| sean-k-mooney | that is an optimistaion howver | 17:40 |
| pas-ha[m] | also, it seems the libvirt itself does support the migration with lvm, even live one, but for that this lvm must be managed by libvirt itself as a storage pool... which is to much of a rewrite for how nova manages local storage I guess.. | 17:41 |
| sean-k-mooney | i think you can live migrate without the storage pool | 17:41 |
| sean-k-mooney | but for cold migration it need it yes | 17:41 |
| sean-k-mooney | and ya we have no plans to use libvirt sotrage pools in the future | 17:42 |
| sean-k-mooney | libivrt is one of those things that is a doubel edge sword. it provides some useful abstration over qemu but at a cost of addtioanl complexity in the tech stack | 17:44 |
| sean-k-mooney | for the most part its been a win | 17:44 |
| sean-k-mooney | but not alwasys | 17:44 |
| melwitt | hi all, I need a second reviewer on this not-yet-released-regression fix (partial revert) related to tpool.Proxy + libvirt listDevices() if anyone could please look https://review.opendev.org/q/topic:%22bug/2098892%22 | 17:57 |
| sean-k-mooney | oh right i have reviewd that already so someone other then melwitt or i | 17:58 |
| sean-k-mooney | bauzas: gibi this impacts gpus and pci passthough so perhaps ye could find time dansmith may also have context since we chatted about it a little when it was found downstream | 17:59 |
| sean-k-mooney | melwitt: thats needed on stable too yes? | 18:00 |
| sean-k-mooney | i dont think we have done a release with it yet | 18:00 |
| sean-k-mooney | but we sould avoid doing one until the latests patches are included | 18:00 |
| melwitt | yes, it is needed on 4 stable branches also :( | 18:00 |
| gibi | sean-k-mooney: I'm out of energy today, made a note to check it tomorrow | 18:04 |
| sshmlbrg | sean-k-mooney: o/ Could u plz review this mr https://review.opendev.org/c/openstack/nova/+/942561 ? | 18:22 |
| dansmith | melwitt: question hre: https://review.opendev.org/c/openstack/nova/+/942334/6/nova/virt/libvirt/host.py | 18:32 |
| dansmith | ah, nevermind, it's in the commit message below the fold | 18:36 |
| melwitt | thanks dansmith! | 18:41 |
| danfai | sean-k-mooney: thanks for the response re vif_type change. sorry only saw the message now. what I wondered was during a migration, the target hypervisor could have a different model. from what I see nova picks up the new vif_type after the migration, but the bridge_name is at that time wrong | 18:48 |
| sean-k-mooney | so it used to be possibale a long time ago to live migrate between linux bridge and ovs | 18:50 |
| sean-k-mooney | danfai: and i had patches up to fix the bridge name | 18:51 |
| sean-k-mooney | but i dont think neuton ever merged them | 18:51 |
| sean-k-mooney | ill see if i still have the fixes in gerrit | 18:51 |
| danfai | thanks, I saw some comments in the code base from you and mriedem :) | 18:52 |
| sean-k-mooney | https://bugs.launchpad.net/neutron/+bug/1788009 | 18:52 |
| sean-k-mooney | we fixed it for ml2/ovs | 18:52 |
| sean-k-mooney | but ml2/ovn didnt populated it proeprly | 18:52 |
| sean-k-mooney | i think that is still a gap in tehr migration tooling but it might have got fixed in the last year or two | 18:53 |
| danfai | oh, ok. I'll have a more detailed look on the neutron side then. We are still in yoga, so... | 18:53 |
| sean-k-mooney | danfai: for what ite work codl migration shoudl still work i think | 18:54 |
| sean-k-mooney | its been a long time since i tried to fix this | 18:54 |
| mikal | Morning | 18:54 |
| sean-k-mooney | danfai: if your using vlan networking with linux breige yoru move will be less painful | 18:54 |
| danfai | cold migration works flawlessly indeed. also two live migrations (we only have provider networks though) | 18:54 |
| sean-k-mooney | danfai: but just be aware ovn does not supprot vxlan or gre tunnes | 18:54 |
| mikal | sean-k-mooney: I feel like I should recheck the spice-direct final patch now that melwitt's tempest fix has landed. Do you see any reason to not do that? | 18:54 |
| sean-k-mooney | mikal:i already did | 18:55 |
| sean-k-mooney | did it fail again | 18:55 |
| sean-k-mooney | mikal: i have started merging your code | 18:55 |
| mikal | sean-k-mooney: doh, I mismashed the refresh button because I am pre coffee. | 18:55 |
| mikal | Nah, it passed. | 18:55 |
| mikal | Ugh, arm passed. | 18:56 |
| mikal | I will be quiet now. | 18:56 |
| sean-k-mooney | mikal: its still in check i think | 18:56 |
| sean-k-mooney | so it failed on the funtional jobs and weill need a recheck | 18:56 |
| sean-k-mooney | but https://zuul.opendev.org/t/openstack/build/1098d7514ba843e28a5477c69892b176 the hybird plug job passed again | 18:56 |
| danfai | yes, regarding vxlans, this was mentioned in the meeting we had with operators. | 18:57 |
| sean-k-mooney | mikal: rechecks wont work until it repors back so lets just let it complete | 18:57 |
| mikal | sean-k-mooney: will do | 18:57 |
| sean-k-mooney | danfai: the main issue iwht the migration scripts that neutron provides is geneve requires a small mtu on the client or extra overhead on the underlying network then vxlan | 18:58 |
| danfai | hmm, I thought there was more, like ID mappings. I'll add the MTU to the notes, thanks | 18:59 |
| sshmlbrg | sean-k-mooney: Hi! Did u see my request on review? | 18:59 |
| sean-k-mooney | danfai: nova does not supprot updatign the mtu on conencted ports for what its worth | 19:00 |
| sean-k-mooney | danfai: the mtu change wil only take effect after a hard reboot | 19:00 |
| sean-k-mooney | danfai: https://github.com/openstack/neutron/commit/baaf240ce3f7802fe1431cc13913b9d93fc7f742 that is the ovn bridge name patch | 19:00 |
| sean-k-mooney | sshmlbrg: yep. we will need to add some nova functioanl tests to replciate the manual procedure you followed in the bug report | 19:01 |
| danfai | Thank you very much | 19:02 |
| sean-k-mooney | sshmlbrg: unit test are good to have but for operations like shelve we try to also have functional tests. and for bugs we prefer to create a functional test that show the broken behavior as an addtion patch and then have the fix as a second commit | 19:02 |
| sean-k-mooney | sshmlbrg: without loading all the context im not sure that the finally block is corect | 19:03 |
| sean-k-mooney | sshmlbrg: you should not be modifying the az of the instnace directly | 19:04 |
| sshmlbrg | sean-k-mooney: ok, thanks. I will add functional tests as a first commit | 19:04 |
| sean-k-mooney | sshmlbrg: we do not really use the az colume in the cell database to store the az of the instance anymore. | 19:04 |
| sean-k-mooney | sshmlbrg: cool i would od it be creating a new file in https://github.com/openstack/nova/tree/master/nova/tests/functional/regressions | 19:05 |
| sean-k-mooney | for the bug ill see if i can quickly find an exmaple for you to copy | 19:05 |
| sean-k-mooney | sshmlbrg: perhaps https://github.com/openstack/nova/blob/master/nova/tests/functional/regressions/test_bug_2025480.py | 19:06 |
| sshmlbrg | sean-k-mooney: then will need to change the methods used and add the az argument to them | 19:06 |
| sean-k-mooney | updatign the object in memory may be ok its the save that is a little suspect | 19:07 |
| sean-k-mooney | we have a temporay mutation context manager | 19:07 |
| sean-k-mooney | i that we may want to condier | 19:07 |
| sean-k-mooney | we can dicuss that on the review | 19:07 |
| sshmlbrg | sean-k-mooney: thank you for example! ok, sounds good | 19:08 |
| sean-k-mooney | if you look at similar files in the same directory there shoudl be a boot form volume example | 19:08 |
| sean-k-mooney | or other shelve example so it should not be too hard to create a repoducer following the steps in the bug | 19:09 |
| sshmlbrg | sean-k-mooney: i got it. I am gonna return with changes tomorrow. Have a nice day or evening :) | 19:11 |
| sean-k-mooney | sshmlbrg: https://github.com/openstack/nova/blob/375d95565d594182c5472f710b3f7bd53f0746d4/nova/tests/functional/regressions/test_bug_1675570.py#L26 | 19:12 |
| sean-k-mooney | that a better example | 19:12 |
| sean-k-mooney | sshmlbrg: o/ | 19:13 |
| sean-k-mooney | sshmlbrg: that actully attaching a server to a shelve offloaded instnace | 19:14 |
| sean-k-mooney | the reason its not failing is cross az attach is allowed by defult | 19:14 |
| sean-k-mooney | so if you were to set that to false in yoru version i woudl eeepct the attach to fail and repoduce the bug :) | 19:14 |
| sean-k-mooney | mikal: https://zuul.openstack.org/status?change=924844&change=926876&change=926877 we will need to do a few rechecks... but we can kick thos off again shortly | 19:16 |
| sshmlbrg | sean-k-mooney: cool idea, I'll look into this | 19:16 |
| sean-k-mooney | sshmlbrg: let us know if you have issues there is a self.flags function aviable on the test case to set fconfig options | 19:17 |
| sean-k-mooney | if you grep for flags you will see examples of how to use it | 19:17 |
| sean-k-mooney | bauzas: RuntimeError: detected leaked greenlets in nova.tests.functional.libvirt.test_vgpu.VGPUMultipleTypesTests.test_create_servers_with_specific_type | 19:19 |
| sean-k-mooney | i dont know how often tthat fails but that ocatinal does | 19:20 |
| sean-k-mooney | mikal: ^ that the error in the funtional test on the third patfch so its not related ot your code | 19:21 |
| sean-k-mooney | Uggla: https://review.opendev.org/c/openstack/releases/+/941862 need your +1 again i updated the version to 4.1.0 based on elodilles comment on the release channel | 19:24 |
| mikal | sean-k-mooney: are you going to do the rechecks or should I? | 19:25 |
| sean-k-mooney | either works ill be finsihing in the next 20 mins or so but ill preodiclly check in | 19:26 |
| mikal | Well, we' | 19:26 |
| mikal | Well, we're still waiting for them to finish right now anyways. | 19:26 |
| sean-k-mooney | ya. if multiple people recheck it wotn kick it out of check | 19:27 |
| sean-k-mooney | i dont it migh kick it out of gate i dont recally so we should be carful in that case | 19:27 |
| sean-k-mooney | mikal: over all the results look pretty green the nova-grenade-multinode failure in gate on the first patch was a kernel panic | 19:29 |
| sean-k-mooney | that not related to yoru code so just bad luck | 19:29 |
| sean-k-mooney | mikal: sometime rechecking at a less busy time help with things like that but its a bug in our cirros image we think | 19:30 |
| mikal | sean-k-mooney: without checking, I feel like cirros hasn't done a release in ages? | 19:31 |
| mikal | Huh, more recent than I expected. September last year. | 19:31 |
| sean-k-mooney | its been a while there is a 6.x branch based on ubuntu 22.04 or maybe 24.04 | 19:31 |
| mikal | The author now works at chainguard IIRC. | 19:32 |
| sean-k-mooney | i dont recall if they have compelte the rebase on the 24.04 kernel yet | 19:32 |
| sean-k-mooney | we have been considering other distos for testign but few if any really meet the min ram requirements we have | 19:32 |
| sean-k-mooney | gento recently started providign cloud-images iwth cloud ini but they are like 930+ MBs | 19:33 |
| sean-k-mooney | vs the like 20-50mb of a cirros image | 19:33 |
| mikal | I have been toying with the idea of a specialised UEFI binary for SPICE testing and skipping the OS entirely. I guess it depends what functionality you need from the instance. | 19:33 |
| mikal | Also, I have way more random tangents than I will ever have time to persue. | 19:34 |
| sean-k-mooney | ya so ironic were not supper ok with the idea of alpine because they worried about he use of musle libc | 19:34 |
| sean-k-mooney | nova does not really care about that which is why i was prposin git | 19:34 |
| sean-k-mooney | mikal: for tempest we need a small usbset of utils | 19:35 |
| sean-k-mooney | toybox/busybox mostly covers it | 19:35 |
| sean-k-mooney | although i belive we use gnu core utils version in cirros today | 19:35 |
| sean-k-mooney | mikal: i think frickler has been somehwat invovled in cirros mantaince lately but im not reallly sure | 19:36 |
| sean-k-mooney | frickler: do you know if they are still working on a 0.7.x release with 24.04 kernel and package rebase? | 19:37 |
| mikal | sean-k-mooney: they certainly did the most recent release. | 19:37 |
| sean-k-mooney | mikal: i know we are hoping to do a release with https://github.com/cirros-dev/cirros/pull/122 after epoxy is released | 19:39 |
| sean-k-mooney | before would also be fine but as of expoxy nova will be able to create vms with igb nic which supprot emulated sriov in the guest | 19:39 |
| sean-k-mooney | mikal: we are hopping to be able to do some sriov tesing in ci once our providers are capabliy fo supproting that | 19:40 |
| sean-k-mooney | it will be a while before thats realsitc however | 19:40 |
| Uggla | sean-k-mooney, elodilles, https://review.opendev.org/c/openstack/releases/+/941862 +1 again. | 20:31 |
| frickler | sean-k-mooney: I'm not aware of anyone actively working on cirros, but I could do a bit if needed. that PR should be in 0.6.3 though? | 20:36 |
| opendevreview | Luis Enrique Morales Mendez proposed openstack/nova master: Add 'LOG' to archive_deleted_rows function https://review.opendev.org/c/openstack/nova/+/942441 | 20:39 |
| opendevreview | Merged openstack/nova master: Reproducer for bug 2098892 https://review.opendev.org/c/openstack/nova/+/942402 | 21:25 |
| sean-k-mooney | frickler: 0.6.3 is using hte latest (at the time) 22.04 hwe kernel | 21:57 |
| sean-k-mooney | frickler: but not 24.04 | 21:57 |
| sean-k-mooney | frickler: unless you ment the igb change which might be in 0.6.3 | 21:57 |
| opendevreview | Merged openstack/nova master: libvirt: Fix regression of listDevices() return type https://review.opendev.org/c/openstack/nova/+/942334 | 23:25 |
| opendevreview | melanie witt proposed openstack/nova stable/2024.2: Reproducer for bug 2098892 https://review.opendev.org/c/openstack/nova/+/942651 | 23:36 |
| opendevreview | melanie witt proposed openstack/nova stable/2024.2: libvirt: Fix regression of listDevices() return type https://review.opendev.org/c/openstack/nova/+/942652 | 23:36 |
| opendevreview | melanie witt proposed openstack/nova stable/2024.1: Reproducer for bug 2098892 https://review.opendev.org/c/openstack/nova/+/942653 | 23:37 |
| opendevreview | melanie witt proposed openstack/nova stable/2024.1: libvirt: Fix regression of listDevices() return type https://review.opendev.org/c/openstack/nova/+/942654 | 23:37 |
| opendevreview | melanie witt proposed openstack/nova stable/2023.2: Reproducer for bug 2098892 https://review.opendev.org/c/openstack/nova/+/942655 | 23:38 |
| opendevreview | melanie witt proposed openstack/nova stable/2023.2: libvirt: Fix regression of listDevices() return type https://review.opendev.org/c/openstack/nova/+/942656 | 23:38 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!