| opendevreview | Jens Harbott proposed openstack/project-config master: Drop project definitions for some x/* repos https://review.opendev.org/c/openstack/project-config/+/925075 | 04:56 |
|---|---|---|
| *** tobias-urdin|pto is now known as tobias-urdin | 06:31 | |
| fungi | just a heads up, i'm going to take the opportunity to catch a film over lunch, so i'll be disappearing between 14:45 utc and about 17:15 utc | 13:11 |
| fungi | pip 24.2 was just released: https://pip.pypa.io/en/stable/news/ | 14:40 |
| fungi | i can see quite a few things in the changelog that could end up subtly breaking prior behavior assumptions, so be on the lookout | 14:42 |
| fungi | heading out now, back in a couple of hours | 14:46 |
| opendevreview | Merged openstack/project-config master: Add openstack/os-test-images project under glance https://review.opendev.org/c/openstack/project-config/+/925043 | 14:50 |
| clarkb | have fun! | 15:03 |
| clarkb | if I can get at least one review on the stack starting at https://review.opendev.org/c/openstack/project-config/+/925029 I'll go ahead and start the process of shutting down that cloud today | 15:04 |
| clarkb | separately I haev received my repaired laptop so need to spend some time testing that the repairs corrected the issue adn reloading a functional os on it | 15:04 |
| corvus | clarkb: +2 | 16:04 |
| frickler | corvus: clarkb: please also check https://review.opendev.org/c/openstack/project-config/+/925075 this has caused some config errors | 16:25 |
| opendevreview | Merged openstack/project-config master: Set linaro cloud's max servers to 0 https://review.opendev.org/c/openstack/project-config/+/925029 | 16:30 |
| opendevreview | Merged openstack/project-config master: Use the osuosl mirror for deb packages in image builds https://review.opendev.org/c/openstack/project-config/+/925048 | 16:30 |
| clarkb | thanks. I'ev approved that change to clean up the config errors frickler | 16:42 |
| opendevreview | Merged openstack/project-config master: Drop project definitions for some x/* repos https://review.opendev.org/c/openstack/project-config/+/925075 | 16:45 |
| clarkb | there are three nodes stuck in deleting in linaro. I think I remember that nodepool hanldes this more gracefully now if we proceed towards complete provider remove. Is that recollection correct corvus ? | 17:45 |
| clarkb | if so I can approve the next change in the stack | 17:46 |
| clarkb | in the meantime /me starts reinstalling this laptop | 17:58 |
| corvus | clarkb: not sure what will happen here, but if it goes wrong ping me and i can help | 18:08 |
| opendevreview | Merged openstack/project-config master: Remove labels and diskimages from the linaro cloud https://review.opendev.org/c/openstack/project-config/+/925049 | 18:50 |
| carloss | o/ fungi https://review.opendev.org/c/openstack/project-config/+/924430 manila-unmaintained-core group was created last week. Could you please add manila-stable-maint group to the manila-unmaintained-core? :) | 18:54 |
| fungi | carloss: sure, i guess they're going to seed that group but then add people outside manila-stable-maint to it as well? (otherwise there was little point in creating a wholly new group in gerrit) | 18:56 |
| carloss | yep | 18:57 |
| fungi | carloss: done | 19:03 |
| carloss | fungi: thank you! | 19:10 |
| fungi | clarkb: the image/label removal change deployed, but `nodepool image-list` is still showing images there (14 deleting, 6 ready). i'll check it again in a bit and see if those counts decrease | 19:56 |
| fungi | looks like it just transitioned to 15 deleting, 5 ready so it's at least wanting to delete them now | 19:56 |
| clarkb | fungi: ack thannks | 19:58 |
| clarkb | I suspect that if we have things stuck in deleting we run the nodepool pruge command or whatever it is after both nodes and images should be deleted but haven't been | 19:59 |
| clarkb | then we can land the last change | 19:59 |
| clarkb | fungi: https://zuul-ci.org/docs/nodepool/latest/operation.html#erase this command. Though now that I've said it I think we may want to remove the provider from the config first (so that we don't accidentally createa anything new) then run that command to get the last bits out | 20:00 |
| fungi | yeah, i haven't seen the total node count fall yet, just more images transitioning from ready to deleting | 20:03 |
| clarkb | ya the nodes are stuck. They are old ones and I think the easiest thing is to just nodepool erase the provider since the cloud is going away | 20:04 |
| johnsom | Hi OpenDev colleagues. I starting to look at being able to test SR-IOV code in tempest. To do this I need an instance that uses the "igb" network driver. This is available in recent distros such as Ubuntu 24.04 and Fedora. Thoughts on setting this up? | 20:04 |
| clarkb | johnsom: we have 24.04 nodes available | 20:20 |
| clarkb | does it also require special hardware or just the driver? | 20:21 |
| clarkb | if it is just the new enough kernel to have the driver then you should be all set | 20:21 |
| johnsom | Just the driver, it simulates SR-IOV | 20:22 |
| johnsom | It's the version of qemu that is running the instance. It has to be new enough to have the igb driver in it | 20:22 |
| clarkb | you should be able to run your job on 24.04. You may run into problems with python3.12 support there as that is slowly making its way into openstack but we have the nodes available | 20:23 |
| johnsom | Ok, so the trick is how to boot an instance in zuul with the igb network driver instead of the virtio (my guess) driver. | 20:25 |
| johnsom | This for example: https://zuul.opendev.org/t/openstack/build/23390a2df40e448c97361a357c101cc3/log/zuul-info/host-info.controller.yaml#347 | 20:26 |
| clarkb | can't you just modprobe it? | 20:31 |
| clarkb | I think I'm not understanding why you're expecting zuul needs to do anything if this is an emulation driver it shouldn't be attached to any hardware right? so modprobe when you need ti then it should be available? | 20:32 |
| fungi | er, sorry, i meant i haven't seen the total image count fall yet | 20:33 |
| clarkb | but for real hardware the driver would be selected by udev rules based on the hardware | 20:33 |
| clarkb | I think | 20:33 |
| fungi | at this point we're just at 20 images in deleting state, none in ready | 20:33 |
| clarkb | and we don't control what virtual devices the underlying clouds attach to our instances | 20:33 |
| johnsom | QEMU needs to be configured to use the qemu igb instead of virtio. It's outside the instance kernel. Once qemu is using the correct driver, then you can go to the kernel and enable the VFs. | 20:35 |
| clarkb | we don't control qemu | 20:35 |
| johnsom | https://www.irccloud.com/pastebin/vGgMKrR4/ | 20:36 |
| johnsom | https://www.irccloud.com/pastebin/9shGFGT5/ | 20:36 |
| clarkb | whether the qemu in question is the one controlling our test instance VM or the one nested in the test node we don't control it | 20:36 |
| clarkb | if you are takling about the qemu at the "base" layer controlling the instance VM I suspect there isn't a whole lot we can do. For the nested qemu you should have control to configurei t however you like within the job | 20:37 |
| johnsom | The qemu that runs the VM that devstack will be installed into. | 20:37 |
| clarkb | ya unfortunately I don't think that is something we can control and one of the clouds involved isn't doing qemu at all | 20:38 |
| johnsom | Yeah, I know one is on Xen | 20:39 |
| clarkb | assuming you are going to be testing things in a nested VM why do we care about the host hardware at all? Cant' you test it with a virtual device? I guess the issue is you want to test the passthrough of pci which means virtual devices aren't very valid? | 20:42 |
| clarkb | but ya I think there are a few issues. The first is qemu isn't universal, second is that clouds tend to udpate their hypervisors more slowly so won't have the options available, and finally they may prefer to lock that stuff down and not expose it anyway | 20:43 |
| johnsom | My intent is to have a few VFs available to devstack such that we can test that nova and neutron are working correctly when "direct" ports are assigned to a VM. Right now we aren't regularly testing any of it. | 20:46 |
| fungi | that seems like it violates the usual attempt to isolate test workloads from underlying implementation details of the providers where they run | 20:50 |
| fungi | it probably needs a third-party ci where the lowest hypervisor layer has a specific kernel and configuration | 20:51 |
| fungi | if one of our generic donors is able to guarantee those parameters, it's possible we could add a custom label limited to that provider (or providers) | 20:52 |
| fungi | similar to the labels for nested virt acceleration support | 20:53 |
| johnsom | It is just a qemu configuration setting when the VM is booted up. | 20:53 |
| fungi | is there an openstacky way for a normal cloud user to request that through the nova api? or does it need a custom flavor? | 20:53 |
| fungi | i.e. is there a cloud provider you have an account in where you are able to boot instances that meet that requirement? | 20:55 |
| johnsom | Yeah, there is an "OpenStack way" with nova, let me find it. | 20:55 |
| Clark[m] | Sorry back to matrix as I'm trying to do laptop stuff but even if there is an openstacky way I doubt any of our clouds run new enough hypervisor based on what you said before. | 20:56 |
| johnsom | You set hw_vif_model=igb as an image property | 20:57 |
| Clark[m] | And even if they did they may not want to expose hardware devices directly for security reasons | 20:57 |
| johnsom | It is not pass through, it's an emulated driver | 20:57 |
| johnsom | It is documented here: https://docs.openstack.org/glance/latest/admin/useful-image-properties.html (sorry, the page doesn't have direct links) | 20:59 |
| fungi | ah, so it's a setting on the image, not the instance itself? | 21:02 |
| Clark[m] | If it isn't pass through why can't you use it at the nested level? (This goes back to my original confusion) | 21:02 |
| johnsom | They are all essentially tap drivers (over simplifying a lot), it's just what features they emulate for the kernel inside the VM that differentiates them. | 21:02 |
| johnsom | Well, that is how nova gets these extra settings passed in. I find it odd, but.... | 21:03 |
| johnsom | What I understand you are proposing is get a 24.04 nodepool instance, then have it spawn a VM inside it with the setting enabled, then install devstack inside that VM and run the tempest tests there. Correct? | 21:05 |
| clarkb | johnsom: no, you have the cloud hypervisor, the nodepool VM, and then any VMs that the devstack cloud boots | 21:09 |
| clarkb | if this is all emulation why can't you emulate between the nodepool VM and teh VMs that the devstack cloud boots | 21:10 |
| johnsom | They setting has to be on the nodepool VM. | 21:11 |
| johnsom | Such that nova and neutron in devstack can access the virtual SR-IOV capabilities of the igb driver. | 21:12 |
| clarkb | right so can't you modprobe that driver and then create an emulated device? | 21:12 |
| johnsom | no, qemu has to expose it. | 21:14 |
| johnsom | Running modprobe from devstack would give you nothing as the nic is virtio by default in the nodepool VM. | 21:15 |
| clarkb | but linux allows you to have virtual network interfaces too completely detached from any hardware | 21:16 |
| clarkb | I guess I was hoping you could just create an interface of that type and then have devstack VMs attach to them since it is all emulated anyway apparently | 21:16 |
| johnsom | Nope, there is no emulated SR-IOV driver in the linux kernel. | 21:17 |
| johnsom | The devstack level kernel would see it as an actual igb nic, but in reality it is qemu emulating one. | 21:18 |
| clarkb | then ya you may need to do the extra level of nesting which I have no idea if that is feasible | 21:24 |
| johnsom | Yeah, RAM might become an issue. | 21:28 |
| clarkb | I'm almost to the point where I'll feel functional on the repaired laptop (it makes a lot of noise now unfortunately, but seems like lenovo doesn't consider that to be an issue) | 21:50 |
| clarkb | fungi: re the image cleanup we can probably give it until tomorrow then plan to land the last cleanup change and ifnally run the erase command. I don't think we're in a super hurry but I also want to get it done | 21:59 |
| clarkb | and now to update the meeting agenda | 21:59 |
| fungi | sgtm | 22:01 |
| clarkb | we also need to land https://review.opendev.org/c/opendev/base-jobs/+/922653 so that we can move forward on cleaning up those images from nodepool | 22:09 |
| clarkb | but other things keep popping up anddistracting me | 22:10 |
| clarkb | I've just updated the meeting agenda. Anything need to be added/removed/edited? | 22:13 |
| fungi | i can't think of anything | 22:27 |
| fungi | i went ahead and approved 922653 too | 22:27 |
| opendevreview | Merged opendev/base-jobs master: Drop CentOS 8 Stream nodesets and sanity checks https://review.opendev.org/c/opendev/base-jobs/+/922653 | 22:30 |
| clarkb | digging in more looks like libvirt 9.3.0 or newer is necessary for igb? | 22:37 |
| clarkb | I think that is also included in centos 9 stream so potentially useable in the openmetal cloud? | 22:37 |
| clarkb | johnsom: do you know if you can set those flags on vm boot as an alternative to being an image setting? Its kinda weird that would be limited to the image. Also any idea if you set it on either an image or vm and the backing cloud doesn't support it if that gets silently ignored or will it explode? | 22:38 |
| clarkb | sean-k-mooney may know but isn't in here | 22:40 |
| clarkb | agenda has bee nsent | 23:43 |
| clarkb | you can tell I'm on the laptop again because my typing is worse here | 23:43 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!