| *** tbachman__ is now known as tbachman | 00:16 | |
| sean-k-mooney[m] | ade_lee ack. ill take a look again tomorrow but looks like everything passed. i think it would still be worth trying to enable zswap as a seperate patch but glad the oom issue are gone | 00:48 |
|---|---|---|
| *** tbachman_ is now known as tbachman | 01:09 | |
| *** tbachman_ is now known as tbachman | 01:44 | |
| *** tbachman_ is now known as tbachman | 01:52 | |
| *** tbachman_ is now known as tbachman | 02:01 | |
| *** tobias-urdin5 is now known as tobias-urdin | 02:09 | |
| *** tbachman_ is now known as tbachman | 02:09 | |
| *** tbachman_ is now known as tbachman | 02:43 | |
| *** EugenMayer4 is now known as EugenMayer | 02:45 | |
| *** artom__ is now known as artom | 03:28 | |
| opendevreview | Merged openstack/nova master: Move ReaderWriterLock to the test tree https://review.opendev.org/c/openstack/nova/+/824931 | 06:27 |
| opendevreview | Lior Friedman proposed openstack/nova master: support use_multipath for nvme driver. Update nvme UT. Add release notes. Change-Id: I20f6aa2d422259601c44d024099cd5760a8e21c6 https://review.opendev.org/c/openstack/nova/+/823941 | 07:33 |
| plibeau2 | lyarwood: hello, sorry to push you my pull request again :) https://review.opendev.org/c/openstack/nova/+/820531/5 | 08:23 |
| *** xek_ is now known as xek | 13:26 | |
| *** dasm|off is now known as dasm | 13:31 | |
| elodilles | bauzas: are you updating the nova meeting wiki right now? because if not, then i'll update the stable part there | 14:18 |
| bauzas | elodilles: do the stable things | 14:20 |
| bauzas | elodilles: I'll update later | 14:20 |
| bauzas | gibi: food for thoughts https://bugs.launchpad.net/nova/+bug/1956983 | 14:20 |
| sean-k-mooney1 | bauzas: that is propably realted to the other bug we have in that area | 14:32 |
| sean-k-mooney1 | currently FFU is broken | 14:32 |
| sean-k-mooney1 | we are checkign the min service version not the min service version fo the running computes | 14:32 |
| elodilles | bauzas: thanks, done | 14:32 |
| sean-k-mooney1 | so you cant skip level upgrade the contoler to n+3 and start it as it will fail the check even if you have stopped all compute unless you work aroudn it with force down | 14:33 |
| gibi | bauzas: context L398 https://etherpad.opendev.org/p/nova-wallaby-ptg | 14:38 |
| gibi | we prevent starting services with older than N-1 computes since https://review.opendev.org/c/openstack/nova/+/738482 | 14:38 |
| gibi | which is merged in wallaby | 14:39 |
| sean-k-mooney1 | gibi: yep but that has a bug in it | 14:39 |
| sean-k-mooney1 | i assume you have seen the mail thread | 14:39 |
| gibi | sean-k-mooney1: I saw the ML thread, I did not see the bug report yet | 14:39 |
| sean-k-mooney1 | i dont think they filed one | 14:40 |
| gibi | bauzas: so I think the bug is already fixed since wallaby | 14:40 |
| gibi | :) | 14:40 |
| gibi | sean-k-mooney1: as of the bug in the N-1 check | 14:40 |
| sean-k-mooney1 | but looking at the code you cannot start the nova conductor of you have n-2 comptue in the db not just n-2 compute running | 14:40 |
| gibi | sean-k-mooney1: I still need to dig up more context | 14:40 |
| gibi | sean-k-mooney1: I feel our original intention was to detect even stopped computes | 14:40 |
| gibi | but I'm not sure | 14:40 |
| sean-k-mooney1 | well that makes FFU impossible | 14:41 |
| gibi | if FFU is broken then we have to do someting | 14:41 |
| sean-k-mooney1 | well it depend the only escape hatch in the current code is to force all compute as down | 14:42 |
| gibi | which is probably unintentional too :) | 14:42 |
| sean-k-mooney1 | possible | 14:42 |
| gibi | I'm wondering what would happen in an N could if an N-2 compute that was stopped now suddenly started up | 14:43 |
| sean-k-mooney1 | in the ffucase if you update the contoler to n+3 then the condcutre wont be able to start until after you have updated all compute agent but they wont be abel to start cause the conductor is stopped | 14:43 |
| sean-k-mooney1 | gibi: ideally the comptue should fail to start but the conductor should keep running | 14:43 |
| gibi | sean-k-mooney1: OK, so now we probably fail backward as we not allow to start the new controller, but we should fail forward and only not allow the old compute to start up | 14:44 |
| sean-k-mooney1 | no | 14:45 |
| sean-k-mooney1 | we want the contoler to start we dont want the computes to start | 14:45 |
| sean-k-mooney1 | until they are upgraded | 14:45 |
| gibi | do we allow starting up an N-2 controller in an N cloud? | 14:46 |
| gibi | or even an N-1 controller? | 14:46 |
| sean-k-mooney1 | the contoler process, conductor in particalar need to be running so the compute can update its version in the db | 14:46 |
| sean-k-mooney1 | gibi: today proably | 14:46 |
| sean-k-mooney1 | which woudl be unsupported | 14:46 |
| sean-k-mooney1 | since the contoler shoudl always be upgraded first | 14:47 |
| sean-k-mooney1 | i mean if you clamp the rpc apis in the config you can praobly make it work but the assumtion we have is contoler are always newer or the same version as the computes | 14:47 |
| *** sean-k-mooney1 is now known as sean-k-mooney | 14:51 | |
| sean-k-mooney | gibi: basicaly i think we shoudl either modify the check to only consider up compute servics or we shoudl move the check to the compute agent instead | 14:52 |
| gibi | hehe, I had considered disabled old computes https://review.opendev.org/c/openstack/nova/+/738482/4..10/nova/service.py#430 | 14:52 |
| sean-k-mooney | or add a config option to opt out of the check for ffu | 14:52 |
| sean-k-mooney | for what its worth i did some testing with devstack not so long ago | 14:53 |
| sean-k-mooney | if i disable the version check you can mix wallaby contoler with train computes | 14:53 |
| sean-k-mooney | to enable better upgrades we are considering supproting that as an upgrade path in the future downstream. not for train to wallaby but from wallaby to whatever our next release would be based on | 14:54 |
| gibi | hm, a disable but running old compute can cause trouble in the cluster, a disable + force down compute cannot as we define force_down as "admin-promised-it-is-not-running" | 14:55 |
| gibi | also based on the discussion in the original patch | 14:55 |
| sean-k-mooney | yes force down means this is down nova just has not noticed yet i promise | 14:55 |
| gibi | so at leat keeping the force_down make sense | 14:56 |
| gibi | I'm not sure about the disable but running compute | 14:56 |
| sean-k-mooney | well i was not thinking about disable | 14:56 |
| sean-k-mooney | i was thinking more up=false | 14:56 |
| gibi | ahh | 14:56 |
| sean-k-mooney | so heartbeat is missing and we mark it as down | 14:56 |
| gibi | so the compute definition exists in the db but the compute is timed out | 14:56 |
| sean-k-mooney | ya presujmably because you stop it on the compute hosts as part of the upgrade | 14:57 |
| gibi | it could be a network partition but it also can be a manual stop yes | 14:57 |
| sean-k-mooney | well if it a network partiion it cant talk to the conductor :) but honestly i think the check is better done on the comptue side then the conductor side | 14:58 |
| gibi | yeah but the check on the compute side only work if the compute is restarted | 14:58 |
| gibi | if the compute is running | 14:58 |
| gibi | then no check happens | 14:58 |
| sean-k-mooney | that is fair | 14:58 |
| gibi | that prevent the old compute to send nonsense to new conductor | 14:58 |
| gibi | probably there is no perfect solution | 14:59 |
| gibi | as from the controller perspective a timed out compute can mean multiple things | 14:59 |
| sean-k-mooney | i mean for now we have the workaround of before you FFU stop all compute agaents and force themn down in the api | 14:59 |
| gibi | yeah, and while that is an extra step it does not create an extra inconsistency | 15:00 |
| gibi | you just need to double confirm that those computes are not coming back | 15:00 |
| gibi | without a restart | 15:00 |
| sean-k-mooney | honestly unless we have done a major rpc bump the comptue wont impact the conductor if they start | 15:00 |
| sean-k-mooney | but yes if you use force down you shoudl fulfil its requiremetns | 15:01 |
| gibi | hm an old compute cannot decided if it is too old, as the old code does not know what is the service version of the N+2 release | 15:02 |
| gibi | except if we backport fixes to stable branches after the N | 15:03 |
| gibi | N+2 release | 15:03 |
| sean-k-mooney | well i was more thingink it would try to connect to the conductor and it would say its too old | 15:03 |
| sean-k-mooney | but ya they cant really | 15:03 |
| gibi | yeah the conductor could tell the compute to go away you are too old | 15:04 |
| sean-k-mooney | it will do that if a new compute tries to regeister | 15:04 |
| bauzas | gibi: I could be late for the meeting, could you please start it if so ? | 15:17 |
| bauzas | the agenda is updated | 15:17 |
| * bauzas rushes | 15:17 | |
| bauzas | reminder : nova meeting starts in 42 mins here | 15:18 |
| gibi | bauzas: sure | 15:22 |
| gibi | just a quick reminder nova meeting starts in 13 minutes here in the channel | 15:47 |
| bauzas | I'm back | 15:57 |
| bauzas | right on time | 15:57 |
| * bauzas heads to the coffee machine | 15:57 | |
| bauzas | #startmeeting nova | 16:00 |
| opendevmeet | Meeting started Tue Jan 18 16:00:00 2022 UTC and is due to finish in 60 minutes. The chair is bauzas. Information about MeetBot at http://wiki.debian.org/MeetBot. | 16:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 16:00 |
| opendevmeet | The meeting name has been set to 'nova' | 16:00 |
| gibi | o. | 16:00 |
| gibi | o/ | 16:00 |
| * bauzas bows to the community | 16:00 | |
| chateaulav | \o | 16:00 |
| elodilles | o/ | 16:00 |
| bauzas | #link https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting | 16:01 |
| bauzas | ok, let's start | 16:01 |
| bauzas | #topic Bugs (stuck/critical) | 16:02 |
| bauzas | #info No Critical bug | 16:02 |
| bauzas | #info No Critical bug | 16:02 |
| bauzas | #link https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New 32 new untriaged bugs (-4 since the last meeting) | 16:02 |
| bauzas | #help Nova bug triage help is appreciated https://wiki.openstack.org/wiki/Nova/BugTriage | 16:02 |
| bauzas | #link https://storyboard.openstack.org/#!/project/openstack/placement 23 open stories (-2 since the last meeting) in Storyboard for Placement | 16:02 |
| bauzas | thanks to those who helped to triage | 16:02 |
| bauzas | (I dunno who about placement :p) | 16:02 |
| gibi | bauzas: can be that the any-trait spec merges closed the related ticjket | 16:04 |
| gibi | I will check | 16:04 |
| bauzas | maybe | 16:04 |
| bauzas | anyway | 16:04 |
| bauzas | any bug to want to discuss ? | 16:04 |
| bauzas | looks not | 16:05 |
| bauzas | #topic Gate status | 16:05 |
| bauzas | #link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure Nova gate bugs | 16:05 |
| bauzas | #link https://zuul.openstack.org/builds?project=openstack%2Fplacement&pipeline=periodic-weekly Placement periodic job status | 16:05 |
| bauzas | #info Please look at the gate failures, file a bug, and add an elastic-recheck signature in the opendev/elastic-recheck repo (example: https://review.opendev.org/#/c/759967) | 16:05 |
| bauzas | let's discuss about the centos8 job with elodilles in the stable branches topic | 16:06 |
| gibi | (yepp the two missing placement ticket is the any-trait support, I will reopen them by creating an open task on them) | 16:06 |
| bauzas | apart from this job, nothing to say | 16:06 |
| bauzas | anyone ? | 16:06 |
| bauzas | mmmm, ok, let's move | 16:08 |
| bauzas | #topic Release Planning | 16:08 |
| bauzas | #info FeatureApprovalFreeze was Jan 13st | 16:08 |
| bauzas | #link https://blueprints.launchpad.net/nova/yoga 23 approved blueprints for the Yoga cycle with 16 of them having specs | 16:08 |
| bauzas | #link https://etherpad.opendev.org/p/nova-yoga-blueprint-status Etherpad for blueprints tracking | 16:08 |
| bauzas | so, as you see, I created an etherpad for yoga | 16:08 |
| bauzas | I'll look at each of the blueprints | 16:08 |
| bauzas | and I'll modify the etherpad to explain which ones are started, and which ones need reviews | 16:09 |
| bauzas | but if folks want to modify the etherpad and explain the changes they have for their blueprint, fine by me :) | 16:09 |
| bauzas | every meeting until FeatureFreeze, I'll provide this etherpad during the meeting | 16:10 |
| gibi | cool | 16:10 |
| bauzas | just remember that we now have 6 weeks before it | 16:10 |
| bauzas | anyway, let's discuss about this etherpad by next week as I need to modify it for providing the changes | 16:11 |
| bauzas | also, last thought | 16:11 |
| bauzas | given it's difficult to merge API changes as they're conficting once we merge one | 16:12 |
| bauzas | I'll provide all the changes wanting to add a new microversion | 16:12 |
| sean-k-mooney | ya we can likely come up with a merge order when we get a little closer to FF | 16:12 |
| bauzas | so during the reviews, people could organize themselves to see who and what could be reviewed first etc. | 16:12 |
| bauzas | sean-k-mooney: yeah, here I just want to tell it by now so people see | 16:13 |
| sean-k-mooney | i think there are only 3 specs that would need a microversion maybe 4 | 16:13 |
| bauzas | like, for the moment, I wonder how many blueprints are asking to get a microversion | 16:13 |
| bauzas | sean-k-mooney: that's the question I have for the moment, but I'll look back at the specs | 16:14 |
| bauzas | anyway | 16:14 |
| bauzas | the etherpad is just starting, we'll see how we organize it | 16:14 |
| * artom wonders if some sort of mechanism for grabbing the next available microversion would be worth it | 16:14 | |
| artom | Probably not | 16:14 |
| bauzas | artom: we can't ask slots | 16:15 |
| sean-k-mooney | i think they are https://blueprints.launchpad.net/nova/+spec/remove-tenant-id https://blueprints.launchpad.net/nova/+spec/unified-limits-nova and https://blueprints.launchpad.net/nova/+spec/libvirt-virtiofs-attach-manila-shares | 16:15 |
| bauzas | artom: as we need to merge them one after the other | 16:15 |
| artom | bauzas, I was thinking some sort of gate job that does it | 16:15 |
| bauzas | artom: we don't need a job | 16:15 |
| artom | So the final microversion isn't known until merge, it's all placeholders | 16:15 |
| artom | Anways, I'm inventing complexity for nothing, carry on :) | 16:16 |
| bauzas | artom: you get a merge conflict for your change if another one was merged :p | 16:16 |
| gibi | modifying a code by a gate job... hm | 16:16 |
| bauzas | gibi: agreed, let's keep it simple | 16:16 |
| bauzas | if we only have 4 changes asking microversions, they can organize themselves to be depending on each of them if really needed | 16:17 |
| bauzas | but again, before saying this, I need to look at which ones specifically so we know them | 16:17 |
| * gibi is lucky being the only one wants a new placement microversion... | 16:17 | |
| bauzas | lol | 16:18 |
| bauzas | anyway, moving on ? | 16:18 |
| sean-k-mooney | sure | 16:18 |
| bauzas | #topic Review priorities | 16:19 |
| bauzas | #link https://review.opendev.org/q/status:open+(project:openstack/nova+OR+project:openstack/placement+OR+project:openstack/os-traits+OR+project:openstack/os-resource-classes+OR+project:openstack/os-vif+OR+project:openstack/python-novaclient+OR+project:openstack/osc-placement)+label:Review-Priority%252B1 | 16:19 |
| bauzas | #link https://review.opendev.org/c/openstack/nova/+/816861 bauzas proposing a documentation change for helping contributors to ask for reviews, needs a second +2 | 16:19 |
| bauzas | nothing to tell it for this meeting from me | 16:19 |
| bauzas | oh actually my change was +W and got a CI issue | 16:20 |
| bauzas | because of the centos8 job | 16:20 |
| bauzas | anyway, nothing to tell here | 16:20 |
| bauzas | I guess we should discuss about the job and the stable branches now | 16:20 |
| bauzas | #topic Stable Branches | 16:21 |
| bauzas | elodilles: your turn | 16:21 |
| elodilles | #info stable gates were blocked, due to centos-8-stream job, which is now set as non-voting ( https://review.opendev.org/c/openstack/tempest/+/824740 ) so should be OK | 16:21 |
| elodilles | at least i think so :) | 16:21 |
| elodilles | #info patches waiting for a 2nd +2 (please review some if you have time): https://review.opendev.org/q/project:openstack/nova+branch:%255Estable/.*+is:open+label:Code-Review%253D%252B2 | 16:21 |
| elodilles | these were probably interesting things for now | 16:22 |
| bauzas | agreed | 16:22 |
| sean-k-mooney | the centos-8-stream issues should i belive be fixed now with the release of a new package | 16:22 |
| elodilles | i know that stable is not the top priority between Yoga-2 and Yoga-3, but if anyone have some time... :) | 16:22 |
| sean-k-mooney | so they could go back to voting shortly once we confrim they are stable | 16:23 |
| elodilles | sean-k-mooney: thanks for the info! | 16:23 |
| bauzas | cool | 16:24 |
| clarkb | sean-k-mooney: I don't see commits to systemd or iputils in git.centos.org. Any idea where the fix is? | 16:24 |
| sean-k-mooney | oh i tought it had been fixed and pushed to some of our mirrors. was that just pinning to an old verion | 16:25 |
| sean-k-mooney | if its not fixed ill flag it internally and see if i can find more info | 16:25 |
| bauzas | clarkb: we're in a meeting but should be ended quickly in 5 mins hopefully | 16:26 |
| bauzas | nothing to say more about stable branches ? | 16:26 |
| bauzas | if not, let's move on and wrap this thing :) | 16:26 |
| bauzas | #topic Sub/related team Highlights | 16:26 |
| bauzas | #info No subteam left | 16:26 |
| bauzas | #topic Open discussion | 16:26 |
| bauzas | nothing on the agenda | 16:26 |
| bauzas | so, I guess we're done ? | 16:27 |
| bauzas | anyone whats to shout something ? | 16:27 |
| bauzas | or yelling about something ? | 16:27 |
| bauzas | s/whats/wants (gosh) | 16:27 |
| bauzas | if not, let's pretend we're done | 16:28 |
| bauzas | #endmeeting | 16:28 |
| opendevmeet | Meeting ended Tue Jan 18 16:28:18 2022 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:28 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/nova/2022/nova.2022-01-18-16.00.html | 16:28 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/nova/2022/nova.2022-01-18-16.00.txt | 16:28 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/nova/2022/nova.2022-01-18-16.00.log.html | 16:28 |
| gibi | this was faaast | 16:29 |
| clarkb | I didn't mean to interrupt the meeting, just wanted to point out that I don't think anything has been fixed on centos 8 stream yet. At least not if you look at the package git repos | 16:29 |
| clarkb | maybe they can revert packages without going through git | 16:29 |
| sean-k-mooney | clarkb: i think i miss read the mail thread | 16:30 |
| sean-k-mooney | i tought ye had forced the mirror update in http://lists.openstack.org/pipermail/openstack-discuss/2022-January/026687.html | 16:33 |
| sean-k-mooney | https://bugzilla.redhat.com/show_bug.cgi?id=2037807 is currently on qa | 16:33 |
| bauzas | clarkb: oh my bad, missed the context | 16:33 |
| bauzas | clarkb: that's the problems with the meetings in lieu of the regular IRC chan | 16:34 |
| bauzas | we get interruptions often | 16:34 |
| clarkb | sean-k-mooney: I think fungi had the impression that it was fixed upstream when it wasn't. As far as I can tell there has been no fix made upstream yet. | 16:34 |
| bauzas | so I thought your ping was unintentional | 16:34 |
| clarkb | bauzas: ah | 16:34 |
| clarkb | I could've been more specific that I was referring to the centos 8 ping issues :) | 16:34 |
| sean-k-mooney | clarkb: ya its still on qa so it has not been shiped yet in rhel and presumable not yet in centos stream | 16:34 |
| bauzas | clarkb: but yeah, reading at what you wrote, this makes perfect sense this was on purpose :D my bad. | 16:35 |
| sean-k-mooney | clarkb: the plue reuest seams to have been merged here https://github.com/redhat-plumbers/systemd-rhel8/pull/246 | 16:35 |
| clarkb | sean-k-mooney: we want systemd-239-55 or newer or a revert on the iputils package aiui | 16:35 |
| sean-k-mooney | yes http://pkgs.devel.redhat.com/cgit/rpms/systemd/commit/?id=a4b172eaddf0882435fd2eb4d146012db8e5a2f2 | 16:36 |
| sean-k-mooney | https://git.centos.org/rpms/systemd/commits/c8s | 16:38 |
| sean-k-mooney | its still on 239-54 | 16:38 |
| sean-k-mooney | sorry for the confusion ill see if i can find someoen internally | 16:39 |
| fungi | clarkb: sean-k-mooney: the mirror update i forced was for centos stream 9, because the official mirror we'd been copying from started refusing rsync connections on friday | 16:51 |
| fungi | unrelated to the ping behavior | 16:51 |
| opendevreview | Ade Lee proposed openstack/nova master: Add check job for FIPS https://review.opendev.org/c/openstack/nova/+/790519 | 18:47 |
| opendevreview | Merged openstack/nova master: libvirt: Add announce-self post live-migration workaround https://review.opendev.org/c/openstack/nova/+/741529 | 20:03 |
| opendevreview | sean mooney proposed openstack/nova stable/xena: libvirt: Add announce-self post live-migration workaround https://review.opendev.org/c/openstack/nova/+/825064 | 21:26 |
| opendevreview | sean mooney proposed openstack/nova stable/victoria: libvirt: Add announce-self post live-migration workaround https://review.opendev.org/c/openstack/nova/+/825175 | 21:32 |
| opendevreview | sean mooney proposed openstack/nova stable/ussuri: libvirt: Add announce-self post live-migration workaround https://review.opendev.org/c/openstack/nova/+/825176 | 21:34 |
| opendevreview | sean mooney proposed openstack/nova stable/train: libvirt: Add announce-self post live-migration workaround https://review.opendev.org/c/openstack/nova/+/825177 | 21:37 |
| opendevreview | sean mooney proposed openstack/nova stable/wallaby: libvirt: Add announce-self post live-migration workaround https://review.opendev.org/c/openstack/nova/+/825178 | 21:45 |
| opendevreview | sean mooney proposed openstack/nova stable/victoria: libvirt: Add announce-self post live-migration workaround https://review.opendev.org/c/openstack/nova/+/825175 | 21:48 |
| opendevreview | sean mooney proposed openstack/nova stable/ussuri: libvirt: Add announce-self post live-migration workaround https://review.opendev.org/c/openstack/nova/+/825176 | 21:49 |
| opendevreview | sean mooney proposed openstack/nova stable/train: libvirt: Add announce-self post live-migration workaround https://review.opendev.org/c/openstack/nova/+/825177 | 21:51 |
| opendevreview | Merged openstack/nova master: Remove deprecated opts from VNC conf https://review.opendev.org/c/openstack/nova/+/824478 | 22:46 |
| *** dasm is now known as dasm| | 23:02 | |
| *** dasm| is now known as dasm|off | 23:02 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!