| *** mhen_ is now known as mhen | 03:04 | |
| *** sambork_ is now known as sambork | 07:09 | |
| *** elodilles is now known as elodilles_pto | 07:28 | |
| gibi | Uggla: I will have to drop at the 30 mins mark from today's IRC meeting. I will switch to mostly read only mode after that | 10:21 |
|---|---|---|
| opendevreview | Merged openstack/nova master: Remove remaining reference to Hyper-V driver https://review.opendev.org/c/openstack/nova/+/1014670 | 13:15 |
| opendevreview | Lucas HENRY proposed openstack/placement master: Add [placement_database] synchronous_reader https://review.opendev.org/c/openstack/placement/+/1014751 | 14:01 |
| Uggla | gibi, ok I will set the cclm topic in the first 30mn, so you can share the status. If that's ok for you. | 14:01 |
| opendevreview | Marius Leustean proposed openstack/nova-specs master: Add spec for handling orphan BDMs from late RPC processing https://review.opendev.org/c/openstack/nova-specs/+/975698 | 14:13 |
| gibi | Uggla: I let dansmith share status :) but sure I can chime in with thing if needd | 14:21 |
| opendevreview | Kamil Sambor proposed openstack/nova master: Replace StaticallyDelayingWrapper with futurist DelayingExecutor https://review.opendev.org/c/openstack/nova/+/997410 | 14:41 |
| dansmith | gibi: Uggla not much to say but I will | 14:41 |
| Uggla | dansmith, cool thx | 14:41 |
| *** ChanServ sets mode: +o Uggla | 14:45 | |
| gboutry | Hola Nova, can I get some eyes on: https://review.opendev.org/c/openstack/nova/+/1012727 | 14:46 |
| *** ChanServ changes topic to "This channel is for Nova development | Info & Useful links: https://etherpad.opendev.org/p/nova" | 14:48 | |
| dansmith | Uggla: I saw you just edited the CCLM blueprint.. the last three are not mine and aren't really related, they're part of the test-as-one-microversion proposal AFAIK | 15:03 |
| Uggla | dansmith, you the last 3 entries in the whiteboard ? | 15:04 |
| dansmith | four actually but yeah | 15:04 |
| opendevreview | Dan Smith proposed openstack/nova master: WIP: Disable the default_availability_zone https://review.opendev.org/c/openstack/nova/+/1014772 | 15:33 |
| Uggla | Reminder: Upstream meeting in ~10mn | 15:50 |
| Uggla | dansmith, I will look / clean the whiteboard ASAP. | 15:50 |
| dansmith | Uggla: ack I was just pointing it out in case you thought they were all actually relevant to CCLM | 15:59 |
| Uggla | #startmeeting nova | 16:00 |
| opendevmeet | Meeting started Mon Nov 9 16:00:58 2026 UTC and is due to finish in 60 minutes. The chair is Uggla. Information about MeetBot at http://wiki.debian.org/MeetBot. | 16:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 16:00 |
| opendevmeet | The meeting name has been set to 'nova' | 16:00 |
| Uggla | Hello everyone | 16:01 |
| bauzas | o/ | 16:01 |
| OriginCode | o/ | 16:01 |
| dansmith | o/ | 16:01 |
| tkajinam | o/ | 16:01 |
| gibi | o/ | 16:02 |
| * fungi is around for his topic in open discussion too | 16:03 | |
| Uggla | Ok let's start because we have a lot to cover in open discussion. Btw not sure we will manage to @ all of them. | 16:04 |
| Uggla | #topic Bugs (stuck/critical) | 16:04 |
| Uggla | #info No critical bugs (https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New&field.importance=Critical&orderby=-datecreated) were reported. | 16:04 |
| Uggla | #topic Gate status | 16:04 |
| Uggla | #link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure — Nova gate bugs | 16:04 |
| Uggla | #link https://etherpad.opendev.org/p/nova-ci-failures-minimal | 16:04 |
| Uggla | #link https://zuul.openstack.org/builds?project=openstack%2Fnova&project=openstack%2Fplacement&branch=stable%2F*&branch=master&pipeline=periodic-weekly&skip=0 — Nova and Placement periodic jobs | 16:04 |
| Uggla | #info Investigate gate failures and report bugs tagged gate-failure. | 16:05 |
| Uggla | #info Include a useful explanation when requesting a recheck. | 16:05 |
| Uggla | #link I did that report: https://nova-ci.uggla.fr/ it probably need some refinement, but that's a start. | 16:05 |
| Uggla | This week: | 16:05 |
| Uggla | nova.tests.functional.libvirt.test_live_migration.LiveMigrationQueuedAbortTestLeftoversRemoved.test_queued_live_migration_abort_leftovers_removed seems the most frequent gate job to fail. (only 2 times) | 16:05 |
| Uggla | tempest.scenario.test_volume_boot_pattern.TestVolumeBootPatternV346.test_instance_boot_after_snapshot_deletion[compute,id-77889046-1a75-4f14-9b3a-fbbfdd8e5093,image,slow,volume]. Seems the most flaky "check" test (3 times) | 16:05 |
| sean-k-mooney | is the later the usual kernel panic issue | 16:06 |
| Uggla | sean-k-mooney not sure atm. I have not done analysis. | 16:07 |
| sean-k-mooney | the effort to update the cirros image to aovid some of the failure mode kidn fo stalled out as there was no upstream reivew form the cirros maintainer | 16:07 |
| sean-k-mooney | ack | 16:07 |
| Uggla | sean-k-mooney also I have not reviewed the llm analysis part of the report this time. | 16:08 |
| Uggla | But the number of failure is low. | 16:08 |
| sean-k-mooney | hopefuly that was helped by zigo test stablistation patches | 16:08 |
| sean-k-mooney | i think most if not all of those are now merged | 16:09 |
| Uggla | cool, good to know and thx zigo ! | 16:09 |
| Uggla | something else to report ? | 16:09 |
| Uggla | seems not so moving on | 16:10 |
| Uggla | #topic Release Planning | 16:10 |
| Uggla | #link https://releases.openstack.org/indri/schedule.html | 16:10 |
| Uggla | #info The release schedule includes Nova-specific deadlines. | 16:10 |
| Uggla | #link https://etherpad.opendev.org/p/nova-2027.1-ptg | 16:10 |
| Uggla | nothing specific to report here. | 16:10 |
| Uggla | #topic Review priorities | 16:10 |
| Uggla | #info Starting: https://etherpad.opendev.org/p/nova-2027.1-status --> new file in progress | 16:11 |
| Uggla | ^ new file for Indri (2027.1) | 16:11 |
| Uggla | refinement of it in progress. | 16:11 |
| Uggla | Request to review https://review.opendev.org/c/openstack/nova/+/1004890, https://review.opendev.org/c/openstack/nova/+/1004899 and https://review.opendev.org/c/openstack/nova/+/1005073 , which are ironic backports of the perf fixes. | 16:11 |
| Uggla | from ironic folks ^ | 16:11 |
| Uggla | also I know core are really busy, but it would be great to review (tkajinam) libvirt version bump for Indri and cleanups | 16:12 |
| Uggla | 1008847: Bump MIN_{LIBVIRT,QEMU} for "Indri" | https://review.opendev.org/c/openstack/nova/+/1008847 | 16:12 |
| Uggla | 1008862: Remove version check for old libvirt/qemu | https://review.opendev.org/c/openstack/nova/+/1008862 | 16:12 |
| Uggla | 1008867: libvirt: Remove vTPM version detection for < 8.6.0 | https://review.opendev.org/c/openstack/nova/+/1008867 | 16:12 |
| Uggla | 1009084: libvirt: Stop testing missing sev capability in libvirt < 10.0.0 | https://review.opendev.org/c/openstack/nova/+/1009084 | 16:12 |
| Uggla | #topic Stable Branches | 16:13 |
| Uggla | oops Elod seems on pto, so skipping that topic. | 16:13 |
| Uggla | #topic vmwareapi 3rd-party CI efforts Highlights | 16:13 |
| Uggla | fwiesel something you'd like to report ? | 16:14 |
| fwiesel | Hi, no updates from my side | 16:14 |
| sean-k-mooney | fwiesel: we had one question for you | 16:14 |
| fwiesel | Sure, go ahead | 16:14 |
| sean-k-mooney | we are looking at adding cross cell migration to nova this cycle. is your ci mutli cell and is that something you woudl want to test in it with vmware? | 16:15 |
| sean-k-mooney | note vmware testign is not a must have for the feature | 16:15 |
| sean-k-mooney | we just wanted to highlight this to you for awareness | 16:15 |
| fwiesel | Our ci is single cell. It would be possible to add it | 16:15 |
| sean-k-mooney | ack im not sure it a high priorty so dont go out of your way | 16:16 |
| fwiesel | Okay, thanks for the heads up | 16:16 |
| fwiesel | I don't even think it would be terribly complicated, considering migrations are essentially handled by vsphere | 16:17 |
| Uggla | I guess we can move on ? | 16:18 |
| dansmith | yep, I doubt it will matter as it doesn't change the virt interface, but glad for the testing of course | 16:18 |
| fwiesel | From my side yes | 16:18 |
| Uggla | btw: | 16:18 |
| Uggla | #topic Cross Cell Live Migration (CCLM) | 16:18 |
| * Uggla is giving the mic to dansmith, gibi | 16:19 | |
| dansmith | Not much to say here, spec has two +2s, I've addressed lots of feedback on it | 16:19 |
| dansmith | instance name patches are merged (I think?), consumer types is close I think, just need to iterate on some tests | 16:19 |
| gibi | yepp, instance name is in | 16:20 |
| bauzas | yup, all patches are merged | 16:20 |
| dansmith | AFAIK gibi and bauzas have been testing the first rev of the actual migration PoC and preparing a ton of comments I'm sure | 16:20 |
| bauzas | correct, I'm currently doing some tests on a devstack multinode env | 16:20 |
| gibi | yeah some of the behind the seen work can be seen in https://etherpad.opendev.org/p/nova-cclm | 16:20 |
| gibi | I'm done with the consumer_type testing and that works OK for me | 16:20 |
| gibi | that is it from me | 16:21 |
| bauzas | cool to know, I'll also test the consumer_types patches as well | 16:22 |
| zigo | My pleasure for fixing some of the CI issues. | 16:23 |
| zigo | FYI, these are passing the gate and are still up for review: | 16:23 |
| zigo | https://review.opendev.org/c/openstack/nova/+/1012433 | 16:23 |
| zigo | https://review.opendev.org/c/openstack/nova/+/1012544 | 16:23 |
| zigo | https://review.opendev.org/c/openstack/nova/+/1012435 | 16:23 |
| zigo | https://review.opendev.org/c/openstack/nova/+/1012541 | 16:23 |
| Uggla | Are we good for the CCLM topic ? | 16:23 |
| dansmith | yup | 16:23 |
| Uggla | nothing to add ? | 16:23 |
| Uggla | cool, thanks all for the update! | 16:24 |
| Uggla | I'm not sure lajoskatona is available so skipping this topic for today. To keep some time | 16:25 |
| Uggla | #topic Bug scrubbing | 16:25 |
| Uggla | #info The reported count is at 61 (+2). | 16:25 |
| Uggla | #link https://etherpad.opendev.org/p/nova-bug-triage-roster | 16:25 |
| Uggla | #link https://truc.uggla.fr/ — Bug trend tracking | 16:25 |
| Uggla | #info No call this week 20261111 is FR bank holiday. So I won't do this call. | 16:25 |
| Uggla | #topic Open discussion | 16:26 |
| Uggla | First is on me | 16:26 |
| Uggla | Uggla / amsinha : https://blueprints.launchpad.net/nova/+spec/reject-glance-encrypted-images | 16:27 |
| Uggla | Nova does not support consuming Glance native encrypted images. Reject API requests that would use these images with HTTP 400 and an explanation that encrypted images are unsupported by Nova, before dispatching the operation. | 16:27 |
| Uggla | Basically the idea is to check if we can go ahead with specless blueprint. | 16:27 |
| Uggla | dansmith, bauzas, gibi, sean-k-mooney ok with a SLBP for ^ ? | 16:28 |
| gibi | yeah so this is basically adding new error handling in nova based on glance image metadata info | 16:28 |
| dansmith | we discussed specless at PTG and +1 from me | 16:28 |
| gibi | no new feature support, just early rejecting | 16:28 |
| dansmith | yup | 16:28 |
| gibi | +1 from me tt | 16:28 |
| gibi | too | 16:28 |
| sean-k-mooney | ok with earley rejection | 16:29 |
| Uggla | +1 also for me. | 16:29 |
| * gibi transitions to mostly read only mode due to travelling | 16:29 | |
| sean-k-mooney | if we want to make encypted images work end to end that might need a full spec depending on what it entails but i was not realy following the ptg discussoin on that topic | 16:29 |
| Uggla | so I think we agree on the change and the SLBP | 16:30 |
| Uggla | Second topic: | 16:30 |
| Uggla | (Kaiyang Wu / OriginCode) | 16:30 |
| Uggla | https://blueprints.launchpad.net/nova/+spec/add-loongarch64-support | 16:30 |
| Uggla | Add LoongArch64 architecture support to the libvirt driver | 16:30 |
| Uggla | OriginCode, please go ahead. | 16:30 |
| OriginCode | Thx | 16:30 |
| OriginCode | I'm planning on adding LoongArch64 support to OpenStack, for now I have been working on getting Nova working with LoongArch64 libvirt virtual machines. | 16:31 |
| OriginCode | It mostly involves adding LoongArch64 architecture definitions and handling mechanisms in Nova and os-traits | 16:32 |
| OriginCode | I would like to know if this blueprint requires a spec | 16:32 |
| OriginCode | Afaik there's no change to APIs, we're only introducing a new architecture to the system. | 16:33 |
| bauzas | my main concern is about the CI testing infra it would require | 16:34 |
| tkajinam | OriginCode, is there any simulator-ish platform available to test the feature ? or are you preparing 3rd party CI to run tests in real hardware ? | 16:34 |
| tkajinam | yeah | 16:35 |
| OriginCode | System-level emulation through QEMU is available, we're also happy to supply test hardware, either hosted here at our labs or shipped to your host locations | 16:35 |
| bauzas | this is like supporting a new libvirt driver, we usually ask the proposer whether they can do 3rd party testing on our patches | 16:35 |
| * Uggla has just realized what is LoongArch64 | 16:35 | |
| bauzas | OriginCode: are you familiar with the opendev Zuul infra we have ? | 16:35 |
| OriginCode | Not quite, but I'm happy to research and try to deploy on our machines. | 16:36 |
| bauzas | I think we have a few pointers, lemme dig them | 16:36 |
| OriginCode | And we can do 3rd party patch testing, no problem | 16:36 |
| bauzas | https://docs.opendev.org/opendev/system-config/latest/third_party.html | 16:37 |
| bauzas | there it is | 16:37 |
| OriginCode | Thanks for the information | 16:37 |
| tkajinam | regarding the spec it may depends on the amount of limitations of the architecture, compared to usual x86_64 . | 16:38 |
| bauzas | before accepting any patches, I'd suggest you to start looking at any potential blockers for testing our check pipeline against your servers | 16:38 |
| sean-k-mooney | for what its worth i dont have any general objections to addign an architrue provdie there is testign fo some kind | 16:38 |
| OriginCode | It's similar to limitations already applied to aarch64 | 16:38 |
| sean-k-mooney | im not sure it needs a spec if its just traits | 16:38 |
| sean-k-mooney | and a minor chagne to the enums | 16:38 |
| sean-k-mooney | if some feature just didnt work that might chagne my responce | 16:38 |
| bauzas | paperwork-wise, I do think this can be specless but again, I don't want us to say 'supported' on some drivers we don't test | 16:39 |
| OriginCode | Sure I understand | 16:39 |
| bauzas | OriginCode: we keep track of our different virt drivers here https://docs.openstack.org/nova/latest/user/support-matrix.html | 16:39 |
| bauzas | that will give you the bare minimum for testing | 16:40 |
| OriginCode | Sounds good | 16:40 |
| tkajinam | (the link reminds me of the question whether we should maintain zVM further. just a side note | 16:41 |
| bauzas | could you join a later meeting once you look over all materials and then know more about any potential issues you could have for running 3rd party CI on our pipelines | 16:41 |
| bauzas | ? | 16:41 |
| bauzas | tkajinam: heh | 16:41 |
| OriginCode | Sounds good, I'll join the weekly meeting with more details. | 16:42 |
| bauzas | thank you | 16:42 |
| OriginCode | Regarding test results and CI deployment, potential hardware donations etc. | 16:42 |
| sean-k-mooney | tkajinam: proably not but on the other hand until it cuases use maintiance pain there isnt a lot of motivation to remove zvm | 16:42 |
| sean-k-mooney | we could perhaps formally deprecated it this cycle (2027.1) | 16:42 |
| sean-k-mooney | and see if anyone complains | 16:42 |
| sean-k-mooney | and revist removing it in 2027.2 | 16:43 |
| tkajinam | ok. I'll look into it so that we can discuss it further. for now let's move on. | 16:43 |
| Uggla | OriginCode, are you ok, can we move to next point ? | 16:44 |
| OriginCode | Yes I'm OK | 16:44 |
| OriginCode | Thank you | 16:44 |
| Uggla | OriginCode thx, feel free to join and provide "updates". | 16:45 |
| Uggla | Next point, fungi: Bridging the Gap: Hibiscus Cycle Retrospective | 16:45 |
| Uggla | https://paste.opendev.org/show/bQxjixbTwcKNVoOvch4s/ | 16:45 |
| * Uggla is giving the mic to fungi | 16:45 | |
| fungi | are we skipping tkajinam's topic? | 16:45 |
| tkajinam | fungi, yes. go ahead | 16:45 |
| fungi | thanks | 16:46 |
| Uggla | I agree with tkajinam, in the background | 16:46 |
| fungi | as we've done for the past couple of years now, the openinfra community managers ran maintainer and contributor feedback surveys at the end of the 2026.2/hibiscus cycle, and we're reaching out directly to teams that had multiple responses | 16:46 |
| fungi | the nova team saw 4 contributors fill out surveys this time, though no maintainers; i've got a breakdown of the results along with basic activity metrics linked above (big thanks to ildikov for putting together the data and summarizing answers!) | 16:46 |
| fungi | starting with the metrics, what jumps out at me is that the team got noticeably faster overall at reviewing and merging changes, however the number of changes merged was basically the same as in 2026.1/gazpacho | 16:46 |
| fungi | coupled with a modest increase in the number of new changes opened, this does mean that the ratio of changes closed fell from basically keeping up in the previous cycle (95%) to accumlating some backlog (82%) | 16:46 |
| fungi | the number of active maintainers approving changes in nova deliverables held steady between gazpacho and hibiscus, but there was some increase in non-maintainer reviewers engaged | 16:46 |
| fungi | moving on to survey results, the respondents were mainly new to working on nova but did report an overall positive experience (averaging 4.25-4.75 out of 5); the only thing they scored lower (2.5 out of 5) was useful ci failures and logs | 16:47 |
| fungi | we added some new questions this round related to sustainability: 50% of the survey takers for nova indicated they might be open to the possibility of working toward helping as maintainers in the future (undecided vs merely disinterested) | 16:47 |
| fungi | there's a lot more info in the summary paste i linked, including anonymized characterizations of the freeform feedback; if anyone has questions about specific items there i'm happy to dig in either during the meeting or any time later | 16:47 |
| fungi | i'll close with this quote from the freeform summaries: | 16:47 |
| fungi | "People noted that they appreciate the core reviewer team’s passion and competence that shines through their work, and they also like and appreciate how much experience the team shares overall and find this a great environment to learn and grow. They felt that the team has been very helpful and supportive with answering questions." | 16:47 |
| gibi | cool | 16:49 |
| * dansmith waits for the "but..." | 16:49 | |
| fungi | i didn't have any buts | 16:50 |
| gmaan | :) | 16:50 |
| dansmith | then I say cool also :) | 16:50 |
| Uggla | yes this is nice and cool feedback ! | 16:50 |
| fungi | would love to get more maintainers filling out the survey next cycle, we asked a lot of questions about burnout and such that we don't have answes on from folks in nova | 16:51 |
| dansmith | too busy to fill out the survey = max burnout | 16:51 |
| fungi | but yes, the contributors who filled out surveys were quite positive overall | 16:51 |
| fungi | i assume the comments about test job stability and usefulness of failure logs aren't surprising | 16:52 |
| sean-k-mooney | yes and no | 16:52 |
| fungi | (just based on earlier discussion in this meeting alone) | 16:52 |
| sean-k-mooney | thre has been an uptick in instabligyt realted to finalisign the eventlet removal and some of the graceful sutdown fucntionaltiy | 16:52 |
| sean-k-mooney | i think gibi, zigo and other have burned dwon a lot of that test instablity | 16:53 |
| gmaan | and where we need the most help but get minimal. everyone seems interested to get their code merged or merge code | 16:53 |
| sean-k-mooney | and we are getting back to a more stabel place | 16:53 |
| fungi | keep in mind this was about last cycle, so hopefully we'll see more positive experience with testing for 2027.1/indri | 16:53 |
| fungi | sounds like things are moving in the right direction there anyway | 16:54 |
| sean-k-mooney | i think so too | 16:54 |
| gmaan | is it? i think no | 16:54 |
| sean-k-mooney | well testing | 16:54 |
| sean-k-mooney | not the helping part | 16:54 |
| gmaan | we need help in CI stabilty and get none | 16:54 |
| sean-k-mooney | well zigo for example has strated to help | 16:55 |
| sean-k-mooney | based on the issues they were finding in there local exactuion when packaging | 16:55 |
| sean-k-mooney | and started a thread on how we can impove out testing by using dynmic schduleing | 16:55 |
| sean-k-mooney | that does not mean that most of the fixes didnt fall to the same few | 16:56 |
| fungi | yes, that's good to highlight (the ongoing need, and also that there are folks outside the set of maintainers pitching in more recently). maybe there are specific areas of test instability that would be more productive for folks to dig into? | 16:56 |
| fungi | and direct anyone who's showing interest in helping stabilize tests toward those areas | 16:57 |
| fungi | anyway, i didn't have anything else for today if there are no other questions | 16:58 |
| Uggla | thx fungi | 16:59 |
| tkajinam | maybe we should encourage these people to join the triage meeting (assuming that's the main place where such CI problem may be discussed) | 17:00 |
| tkajinam | just a small idea | 17:00 |
| Uggla | also this meeting to have an overview. | 17:01 |
| Uggla | We are at the top of the hour. So I propose to close for today and postponed the 2 discussions that we have not covered today, next week. | 17:01 |
| YushoYamaguchi | sean-k-mooney: Could you check the slide about guest csum offload on vhost-user environment? https://speakerdeck.com/yushoyamaguchi/guest-csum-offload | 17:02 |
| YushoYamaguchi | For next week | 17:02 |
| Uggla | YushoYamaguchin, apologize punting your topic for next week. | 17:03 |
| opendevreview | Takashi Kajinami proposed openstack/nova master: Deprecate zVM compute driver https://review.opendev.org/c/openstack/nova/+/1014783 | 17:03 |
| YushoYamaguchi | No worries! | 17:03 |
| Uggla | ok time to close. | 17:03 |
| Uggla | Thanks for joining. Have a nice day/evening. | 17:03 |
| Uggla | #endmeeting | 17:04 |
| opendevmeet | Meeting ended Mon Nov 9 17:04:02 2026 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 17:04 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/nova/2026/nova.2026-11-09-16.00.html | 17:04 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/nova/2026/nova.2026-11-09-16.00.txt | 17:04 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/nova/2026/nova.2026-11-09-16.00.log.html | 17:04 |
| tkajinam | thanks | 17:04 |
| sean-k-mooney | YushoYamaguchi: jsut keep in mind we dont offically suprpot migrating to older verison of qemu/libvirt form neweversiosn. it woudl not be incodsitent to say the saem for newer verions of ovs-dpdk to older verions | 17:05 |
| sean-k-mooney | this is a know issue which amy be ok to mitigate via a workarounds config option | 17:06 |
| sean-k-mooney | but in the past we siad it was the operators responisblity to ensure the host is compatible in such cases | 17:06 |
| sean-k-mooney | what we really want to avoid is having to add new config option for every possible network offload, and we woudl also perfer not to expose libvirt raw values | 17:07 |
| tkajinam | we can name the option more generic like disable_vhostuser_host_offload and make it accept a list, to address the former | 17:11 |
| tkajinam | but we still have to maintain per-option code due to the way we currently handle libvirt configuration. | 17:11 |
| sean-k-mooney | but we rejected that in the past | 17:12 |
| sean-k-mooney | for the checksum offloaing specificly | 17:12 |
| sean-k-mooney | and we went to the dpdk, ovs and libvirt comunity to try and fix this | 17:12 |
| sean-k-mooney | we specifly asked libvirt to provide an intenrface to tune the offloads | 17:13 |
| sean-k-mooney | and ovs to adverise the supproted offload adn qemu to expose the negociated offloads | 17:13 |
| sean-k-mooney | libvirt provided a way to tune and qemu and ovs didnt do there path | 17:13 |
| sean-k-mooney | so we previsly said we cant schduler on this as a result sicne we cant know which offloads were negociated in the guest | 17:14 |
| sean-k-mooney | adn we dont knwo what is supproted by ovs | 17:14 |
| sean-k-mooney | to be clear i do agge that havign a config option is better then https://opendev.org/x/networking-ovs-dpdk/src/branch/master/devstack/libs/ovs-dpdk#L81 | 17:15 |
| sean-k-mooney | but ill also note that this issue was identifed over 11 years ago | 17:16 |
| sean-k-mooney | https://opendev.org/x/networking-ovs-dpdk/commit/4e032e110645f97d3a6c807e1869fc6e0a62aef9 | 17:16 |
| opendevreview | Dan Smith proposed openstack/nova master: Add consumer_type for migration placeholder allocations https://review.opendev.org/c/openstack/nova/+/1012083 | 17:16 |
| tkajinam | oh... | 17:16 |
| sean-k-mooney | the ablity fo ovs dpdk and qemu to negociate and advice this was added because of this and similar problems with "smarter" nics | 17:18 |
| tkajinam | I agree that the auto-negotiation is ideal and it works locally but the problem is that we need negotiation "among" multiple compute hosts for migration. | 17:19 |
| sean-k-mooney | right | 17:19 |
| tkajinam | similarly to cpu flag compatibility problems we sometimes have | 17:19 |
| sean-k-mooney | which is garuded today by qemu rejectin gthe migratoin | 17:19 |
| sean-k-mooney | it internally check the vhsot user bitmask | 17:19 |
| sean-k-mooney | and when we do the livbirt call to start the migration | 17:20 |
| tkajinam | yeah the same of the offload flag difference | 17:20 |
| sean-k-mooney | we get the late migraiotn failure | 17:20 |
| tkajinam | yeah | 17:20 |
| sean-k-mooney | im not entirlly agaisnt the idea of having a extraspec, config option or neutron extsion to allow disabling nic offloads | 17:20 |
| sean-k-mooney | but i am noting that in the past nova siad no for workroudn aound ovs-dpdk verion imppatiablities | 17:21 |
| sean-k-mooney | when my team tried to do that like 10 years ago | 17:21 |
| sean-k-mooney | tkajinam: part of it actully was upstreamed for sriov, its why we collect the nic offload capablites and store it in the pci tracker | 17:22 |
| tkajinam | ah | 17:22 |
| sean-k-mooney | but we stalled out on getting ovs(-dpdk) to report the supprot via the ovsdb | 17:22 |
| sean-k-mooney | we did manged to get libvirt to add the falgs, that wrapper script added qemu args | 17:23 |
| sean-k-mooney | becasue orgianly you could not do this via the libvirt xml | 17:23 |
| sean-k-mooney | so at leat that is cleaner now | 17:23 |
| tkajinam | ok. I think that's a good context | 17:24 |
| sean-k-mooney | https://libvirt.org/formatdomain.html#setting-nic-driver-specific-options | 17:24 |
| tkajinam | yeah I was reading that definition | 17:25 |
| sean-k-mooney | as much as i hate host level config opions | 17:27 |
| sean-k-mooney | im alsomost temspte to add [libvirt]host_vif_offloads=csum:off,gso:on | 17:28 |
| sean-k-mooney | and the same for gust | 17:28 |
| sean-k-mooney | and bsiclly say if you use thse you are responsibel for grouping your hosts so that they do not cuase issue when live migrating | 17:28 |
| sean-k-mooney | i.e. disucsgae there use and say its operator error if you try to moe form a host with somehtign forced on to a host with it off | 17:29 |
| sean-k-mooney | but its nova job to provide an abstraction over such low level details and in reality this really shoudl not be expsoed to operators | 17:30 |
| sean-k-mooney | our orgianlly hope was to have a way to report it for all neturkign backend genericly and then have a neutorn exteion where you could ask for ports with spefici offloads | 17:31 |
| sean-k-mooney | but ya no one has asked for that now in many years | 17:31 |
| tkajinam | and one thing we have to consider even if we get that feature is whether we should enable offload which is not explicitly asked | 17:32 |
| sean-k-mooney | yes | 17:32 |
| tkajinam | if you enable new flag automatically then you face the same compatibility problem. | 17:32 |
| sean-k-mooney | not explcitly | 17:32 |
| sean-k-mooney | but yes as it would be a sever performace regression not too | 17:32 |
| tkajinam | yeah. I don't want users to update 1000x ports to enable a new offload | 17:33 |
| sean-k-mooney | tkajinam: to be clear the reason that his has come up is bugs in ovs-dpdk | 17:33 |
| tkajinam | it may not be direct port update but something requiring migration ... | 17:33 |
| tkajinam | ok | 17:33 |
| sean-k-mooney | a fixed bug too | 17:34 |
| sean-k-mooney | so part of the problem here as well is they ran out of bit and had to change the binary represtion in the virtio spec | 17:35 |
| sean-k-mooney | addign the mtu to a vhost user port for example change the respstaion of the vrito feature mask if i recal correctly | 17:36 |
| sean-k-mooney | as they needed to use the extened format | 17:36 |
| sean-k-mooney | qemu was unabel to compare the diffent formats internally and just rejected it even if the vm was not useing any of the extra adversised capabliteis | 17:36 |
| sean-k-mooney | that partly why going form new to old is hard | 17:37 |
| tkajinam | ok | 17:37 |
| sean-k-mooney | and that is also why qemu and libvirt dont officlly supprot that for there own binaries | 17:37 |
| tkajinam | and people do not pay as much attention to new -> old as old -> new | 17:38 |
| sean-k-mooney | it can work but its not offilaly suprpoted to go form a new qemu to an older one via a live migraiton | 17:38 |
| sean-k-mooney | right. for nova we block new to old migration for livbirt and qemu only | 17:38 |
| sean-k-mooney | we never extened that to the host kernel or ovs | 17:38 |
| sean-k-mooney | but instead said if live migtion does not work use cold migration and prefer old->new only in general | 17:39 |
| tkajinam | understood | 17:39 |
| sean-k-mooney | tkajinam: i think a more ideal soltuion from my perspective | 17:39 |
| sean-k-mooney | woudl be to instead contole this form the ovs-dpdk side | 17:39 |
| sean-k-mooney | i.e. if you could limit the aviabel feature via the vswitch config | 17:39 |
| tkajinam | then enable consistent flags in all nodes in ovs level | 17:41 |
| sean-k-mooney | yep and only upgrade ocne all your ovs instnace supprot it | 17:42 |
| sean-k-mooney | sort of like the pmd core mask | 17:43 |
| sean-k-mooney | ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0x6 | 17:43 |
| sean-k-mooney | you can do it for the phsyical ports | 17:44 |
| sean-k-mooney | https://ovs.readthedocs.io/en/latest/howto/dpdk.html#rx-checksum-offload | 17:44 |
| tkajinam | so you first record available/enabled flags before upgrade, hard-code the list in config, once you upgrade all nodes remove the hard-code to enable new flags everywhere | 17:44 |
| sean-k-mooney | perhpas it kind of sucks regardless | 17:45 |
| sean-k-mooney | becase we have now way to knwo what is aviabel or used | 17:45 |
| sean-k-mooney | nova cant do somethign smarter | 17:45 |
| tkajinam | :-( | 17:45 |
| sean-k-mooney | even if we provide the cofnig option in nova | 17:45 |
| sean-k-mooney | you woudl have to basiclly od the same right | 17:45 |
| tkajinam | yeah | 17:46 |
| sean-k-mooney | ovs apprenlty suprpot ovs-vsctl set Open_vSwitch . \ | 17:47 |
| sean-k-mooney | other_config:userspace-tso-enable=true | 17:47 |
| sean-k-mooney | tkajinam: i woudl be very interested if you disabeld th checksumn offloading on the dpdk phsyical ports | 17:48 |
| sean-k-mooney | if it woudl prevent it form beign adverstised on the upgraded ovs for the vhost-user ports | 17:48 |
| sean-k-mooney | its been like 8 years sinc ei worked on this but i think in the past if the phsyical port did not supprot it it was not adversied on the vhost-user side | 17:49 |
| sean-k-mooney | but all of that could have changed | 17:49 |
| tkajinam | ah | 17:49 |
| tkajinam | I've never tried that but we could ask Yusho to check it | 17:50 |
| tkajinam | though I don't know if they accept performance implication of that | 17:51 |
| sean-k-mooney | turning it off apprently speed up vlan and small packet performacne but redutces performace for tunnels and jumbframes | 17:51 |
| sean-k-mooney | which iguess is why they maded it cofnigurable | 17:52 |
| sean-k-mooney | because it depend on your traffic profile | 17:52 |
| tkajinam | yeah | 17:52 |
| sean-k-mooney | """ DPDK vectorization is disabled when checksum offloading is configured on DPDK physical ports which in turn effects the non-tunnel traffic performance. So it is advised to turn off the Rx checksum offload for non-tunnel traffic use cases to achieve the best performance.""" | 17:52 |
| tkajinam | and now I noticed he already disconnected | 17:53 |
| sean-k-mooney | yeah but at least the channel is logged | 17:53 |
| tkajinam | yeah | 17:54 |
| tkajinam | I vaguely remember they use vxlan/geneve heavily but I could be wrong | 17:54 |
| sean-k-mooney | if i recall you can disabel the checksum offloads which are relivity cheap whiel still having the generic udp ecap/decap offlow neeeded to hardware acclelerate vxlan and geneve | 18:01 |
| sean-k-mooney | the reason we had thsoe workaroudn orgianl is in the 2.x release thre were bugs that resulted in incorect checksum caluations | 18:02 |
| sean-k-mooney | which woudl cause the packet to be droped by the guest kernel | 18:02 |
| sean-k-mooney | i.e. because of the check sum erros | 18:02 |
| sean-k-mooney | so i think what they want to achive (portablity without impacting performace signifcantly) is viable | 18:03 |
| sean-k-mooney | if we went with the config options and kept it geenr co host/gust offload flags, with the default being empty menaing we do not add anything and let libvirt/qemu decied i think i coudl reluctantly accpat that | 18:04 |
| sean-k-mooney | i would just hope usign it is the excptin not the rule | 18:04 |
| tkajinam | so the option defines specific overload, and does not control the full options | 18:06 |
| tkajinam | and it may defaults to no override for compatibility | 18:08 |
| sean-k-mooney | exactly | 18:08 |
| sean-k-mooney | that means no change unless you opt into this | 18:08 |
| tkajinam | yeah | 18:08 |
| sean-k-mooney | and then if you want full contol you woudl have to specify all of them | 18:08 |
| sean-k-mooney | otherwise just pass your overreids | 18:08 |
| sean-k-mooney | but there would sill be no schduler supprot | 18:09 |
| sean-k-mooney | so if its not the same everywhere | 18:09 |
| sean-k-mooney | then you need to use host aggrates or soemthign else | 18:09 |
| sean-k-mooney | if libvirt/ovs ever gained the ablity to report this we could impvoe it like we did for sev | 18:09 |
| tkajinam | we could probably document the specific expected use case in help text so that it's not abused for "generic" use. | 18:11 |
| tkajinam | I feel like it now is simple enough to be specless but we can determine that later | 18:12 |
| tkajinam | sean-k-mooney, I'm leaving soon. thanks for the good discussion, as always :-) | 18:12 |
| sean-k-mooney | no worres also apprently they did make one almsot useful change in ovs | 18:14 |
| sean-k-mooney | tatus : features:optional string | 18:14 |
| sean-k-mooney | virtio features bitmap as per virtio specification. | 18:14 |
| sean-k-mooney | so after we create a vhosuser port ther is now a staus:feature column on the port in the ovsdb | 18:15 |
| sean-k-mooney | so we still dont knwo what the vswich supprots | 18:15 |
| sean-k-mooney | but we can now apprently get the feature bitmap after a vm is booted | 18:15 |
| sean-k-mooney | we still do not know however host to use that for comaptiable unles we read the virtio spec an reimplment tha twhich is not soemthign i would want nova ot have to do | 18:16 |
| sean-k-mooney | tkajinam: enjoy the rest of your night o/ | 18:17 |
| sean-k-mooney | dansmith: on the subject of exsitng az weirdnes, are you aware you can define multiple host aggreate each with the saem availablity_zone=AZ1 metadta value and today we consider the AZ to be the union of allof them including supprot in the placment az filter to combing the uuids usign a member_of:in=uuid1,uuid2 request | 18:36 |
| sean-k-mooney | this came up on the watcher spec where i was trying to expalin we cant assume a 1:1 mappiing betwen AZ and host aggreate | 18:37 |
| sean-k-mooney | 1 AZ id defiend by uniing all the hosts that are in a set of host aggreates with the same az key | 18:37 |
| sean-k-mooney | its not a common usecase but its what the code supprots today | 18:38 |
| dansmith | AZ is a nova concept so it can expose whatever it wants as an AZ to the API user with whatever accounting it wants to make sure it's consistent right? by that I mean, placement doesn't know what an AZ is or that a particular aggregate is special in that way | 18:40 |
| sean-k-mooney | correct placement has no idea | 18:40 |
| sean-k-mooney | and i want watcher to not need to care either | 18:40 |
| dansmith | agree on watcher | 18:40 |
| sean-k-mooney | dansmith: the context was doug was condieerign if watcher could try and may scoped host aggrate id back to there az | 18:41 |
| sean-k-mooney | and i was like no if we mean az lets just accapt az as a paramter | 18:41 |
| dansmith | my point above is that because placement doesn't know that the az tag on an aggregate is special, I don't think there's any sane behavior for nova other than to take the union since placement won' | 18:41 |
| dansmith | t/can't enforce uniqueness there | 18:41 |
| sean-k-mooney | oh ya | 18:41 |
| sean-k-mooney | it would not be on placemnet ot do that anyway | 18:42 |
| sean-k-mooney | if we ever wanted to enfocce that we woudl do that in nova | 18:42 |
| sean-k-mooney | im just noting that before placement we didnt enfoce it | 18:42 |
| sean-k-mooney | so when we added the host aggrate mirroring | 18:42 |
| sean-k-mooney | and then the az prefilter | 18:42 |
| sean-k-mooney | we keep this union behvior working | 18:42 |
| sean-k-mooney | to avoid upgrade impacts | 18:42 |
| sean-k-mooney | which si fine | 18:42 |
| sean-k-mooney | it just not docuemented or really tested | 18:43 |
| sean-k-mooney | (tested outside of unit tests) | 18:43 |
| sean-k-mooney | but i have no idea how much its used so i woudl eb reluctant to change it | 18:43 |
| sean-k-mooney | i have not read https://review.opendev.org/c/openstack/nova/+/1014772 yet for the default az haneling but i gues may main question woudl be the upgrade impact | 18:44 |
| sean-k-mooney | for instance that are already pinned to the default az | 18:44 |
| dansmith | the only way (AFAICT) that the default az could be working (properly) is if you have gone against the docs and actually defined it as a real AZ | 18:45 |
| sean-k-mooney | yep | 18:45 |
| sean-k-mooney | and i know that folks have done that in production before | 18:45 |
| dansmith | but as noted, you can change the default config at will and it doesn't change the instance itself (expected) and thus I don't think removing it would be any different than today where the az is wrong anyway as in my example | 18:46 |
| sean-k-mooney | i was very surpessed that doc note existed when i first found it | 18:46 |
| sean-k-mooney | like 10 years ago | 18:46 |
| dansmith | well, I'm glad it does because otherwise it's like _really_ broken. At least we _tell_ you not to do that today | 18:46 |
| sean-k-mooney | ebcause for the first coupel of year of workign with openstack that guidace was not common knowldage | 18:46 |
| sean-k-mooney | the one gap that im aware of si | 18:47 |
| sean-k-mooney | there is no way to explcity target host in no az | 18:47 |
| sean-k-mooney | which is fine | 18:47 |
| dansmith | but as noted in the commit message of that change, we might want some more LOG.warn and/or nova-status stuff to help with the upgrade | 18:47 |
| sean-k-mooney | for what its worht looking at horizon code breifly | 18:49 |
| sean-k-mooney | i belive if we just stop adversing nova as an az | 18:49 |
| sean-k-mooney | then it will do the right thing | 18:49 |
| dansmith | that would make it more correct than today so.. excellent :D | 18:49 |
| sean-k-mooney | if no other az exist it will not add any az request and if 1 real az exists it will use that as the only value | 18:49 |
| sean-k-mooney | well + any | 18:50 |
| dansmith | I don't think using the one AZ exposed implicitly is really the right behavior, but at least it's not obviously-wrong behavior if the only exposed AZ is the default one | 18:50 |
| sean-k-mooney | well actully no you my hwave to tweak it slightly to allwo any in that case | 18:50 |
| sean-k-mooney | but at that poitn we could have a converstaion with the horizon team on what the prefered behvior woudl be | 18:51 |
| sean-k-mooney | anyway ill try and look at your propsoed fix and then feed that back into my spec if you think i should adn other agree | 18:52 |
| dansmith | yeah it should probably be [list of exposed AZs] + ['N/A'] or something, if the first is a nonzero list | 18:52 |
| sean-k-mooney | ya likely jsut remove the specal case fo 1 branch | 18:52 |
| dansmith | ack, as the commit message says, that proposal was mostly just to be able to poke at the api with that behavior to make sure it was sane | 18:53 |
| dansmith | but yep, I suspect it's an easy thing to do as a precursor bug to any more AZ stuff (including mine) if we can get some eyes on it | 18:53 |
| sean-k-mooney | if i removed the sepcial handelign for the defautl az form my spec and just refence your bug would that be ok for you? | 18:54 |
| dansmith | or just remove and not mention it at all as you should be doing whatever create does today (IMHO), whether that changes or not.. but sure | 18:54 |
| sean-k-mooney | i would also like a second pair of eye on the mutlal exclive group or not. i dont really have a stong prefence either way but once i know what folks rpefer ill update the spec and my local poc | 18:55 |
| dansmith | yep, good with me | 18:55 |
| sean-k-mooney | ack ill drop it and say "the handelign of the default az will mirror create" then figure out what that actully is :) | 18:56 |
| dansmith | sounds good | 18:56 |
| sean-k-mooney | my goal tomorwo is to set up a multi node devstack to proplry test my changes locally and owrk on the openstack client and tempest change. then later in the week ill loop back and update teh spec and poc | 18:57 |
| sean-k-mooney | my curent multi node deployment has the pci-groupign serise and im trying not to disturbe that | 18:58 |
| sean-k-mooney | ok im oging to wrap up shortly o/ | 18:59 |
| dansmith | ack | 19:02 |
| bryanfraschetti | Hi folks, I hope you're doing well. Would anybody be able to review this change https://review.opendev.org/c/openstack/nova/+/1006011/8 | 19:52 |
| bryanfraschetti | It has stalled for about a month, but it is a pretty important fix for k8s-related workloads where frequent attach/detach operations can compete causing contention and timeouts | 19:52 |
| dansmith | bryanfraschetti: looks to me like it's waiting on addressing (legit) review feedback, so nothing actionable at the moment | 20:10 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!