Tuesday, 2022-08-23

auniyalHi 04:37
auniyalplease review these - 04:37
auniyalhttps://review.opendev.org/c/openstack/nova/+/85381104:37
auniyalhttps://review.opendev.org/c/openstack/nova/+/85381204:37
*** tkajinam|off is now known as tkajinam05:48
sean-k-mooney[m]gibi:  i have mostly figured out the issue with my test. its not the migration stub im calling _detach_passthough_vifs in this code path and that was not being mocked by the fixture07:43
gibiahh, OK07:43
sean-k-mooney[m]if i mock that the migration succeed although my post condtions fail07:43
sean-k-mooney[m]so i need to mock it better07:43
sean-k-mooney[m]or perhaps just the detach call07:44
sean-k-mooney[m]im still not sure why this fails in my case and not the sriov case07:44
sean-k-mooney[m]or in my detach test07:44
sean-k-mooney[m]but thats the issue07:45
opendevreviewsean mooney proposed openstack/nova master: Add VDPA support for suspend and livemigrate  https://review.opendev.org/c/openstack/nova/+/85370408:10
sean-k-mooney[m]im just running ^ now locally08:12
sean-k-mooney[m]that should resolve the issue with the tests and i have dropped the service version release mapping08:12
gibisean-k-mooney[m]: I'm +2 and fingers crossed for the test08:13
sean-k-mooney[m]the issue once i mocked it was the _live_migrate helper did not return the server object08:14
sean-k-mooney[m]so i fixed that and then the assert worked08:14
sean-k-mooney[m]hum that apprently broke one functional test im going to leave the unit test finish then ill check that again08:16
sean-k-mooney[m]im not sure why wapping the order of the waits would have that effect08:16
sean-k-mooney[m]i can restore the order of the waits and store teh server responce in a local var08:17
gibior return the server from the _live_migrate helper08:18
sean-k-mooney[m]well i am returning it from _live_migrate08:19
gibiahh yes08:19
sean-k-mooney[m]https://review.opendev.org/c/openstack/nova/+/853704/6/nova/tests/functional/integrated_helpers.py08:19
gibiwould be nice to check the code which state transtion happens last, the migration or the server08:19
gibiand encode that order in the helper08:20
sean-k-mooney[m]the order should not break other test right?08:20
gibiif that make a test case fail then we probably have a bug somwhere08:20
gibiyes, it should not08:20
gibiexcept if the returned server is asserted08:20
gibibut it is not yet in the expected strate08:20
gibistate08:20
gibior vice versa08:20
gibihm08:21
gibiactually what you do is the correct one I think08:21
gibias you return the server after the server state is changed08:21
gibiso I guess the bug is in that other func test case08:21
sean-k-mooney[m]ya that is why i reordered it08:21
sean-k-mooney[m]i think so ya  just getting its name08:22
sean-k-mooney[m]i think its an ovs hardware offload test08:22
sean-k-mooney[m]ya  test_rollback_pre_live_migration08:23
sean-k-mooney[m]ya its a regression but its expected the server state to be migrating and migration state to be failed08:29
sean-k-mooney[m]but the server state is actully now active08:29
sean-k-mooney[m]because i am now waitign until after the migation has failed and we have rolled back08:29
sean-k-mooney[m]so if you are doing fault injection the order actully can matter but this is the only test out of all the unit and fuctional tests that fail08:30
sean-k-mooney[m]so im just going to update it08:30
opendevreviewsean mooney proposed openstack/nova master: Add VDPA support for suspend and livemigrate  https://review.opendev.org/c/openstack/nova/+/85370408:32
sean-k-mooney[m]ok i have been looking at that since 6 am and i have not had coffee this morning so  im going to go correct that08:34
sean-k-mooney[m]brb08:34
gibiyeah, that is a valid fix for the test case08:41
UgglaHi gibi , do we have the nova meeting this evening ?08:44
gibiUggla: it is tuesday so yes 08:44
gibiI will chair it08:44
gibias the last two08:44
Ugglaok cool08:44
sean-k-mooney[m]still embing caffine but assuming im done with vdpa for today. what seriese should i review first08:47
opendevreviewBalazs Gibizer proposed openstack/nova master: Basics for PCI Placement reporting  https://review.opendev.org/c/openstack/nova/+/84618709:06
opendevreviewBalazs Gibizer proposed openstack/nova master: Extend device_spec with resource_class and traits  https://review.opendev.org/c/openstack/nova/+/84621809:06
opendevreviewBalazs Gibizer proposed openstack/nova master: Reject PCI dependent device config  https://review.opendev.org/c/openstack/nova/+/84643509:06
opendevreviewBalazs Gibizer proposed openstack/nova master: Reject mixed VF rc and trait config  https://review.opendev.org/c/openstack/nova/+/84643609:06
opendevreviewBalazs Gibizer proposed openstack/nova master: Ignore PCI devs with physical_network tag  https://review.opendev.org/c/openstack/nova/+/84621909:06
opendevreviewBalazs Gibizer proposed openstack/nova master: Reject devname based device_spec config  https://review.opendev.org/c/openstack/nova/+/84646609:06
opendevreviewBalazs Gibizer proposed openstack/nova master: Support [pci]device_spec reconfiguration  https://review.opendev.org/c/openstack/nova/+/84647009:06
opendevreviewBalazs Gibizer proposed openstack/nova master: Stop if tracking is disable after it was enabled before  https://review.opendev.org/c/openstack/nova/+/84700909:06
opendevreviewBalazs Gibizer proposed openstack/nova master: Move provider_tree RP creation to PciResourceProvider  https://review.opendev.org/c/openstack/nova/+/85054609:06
opendevreviewBalazs Gibizer proposed openstack/nova master: Retry /reshape at provider generation conflict  https://review.opendev.org/c/openstack/nova/+/85135809:06
opendevreviewBalazs Gibizer proposed openstack/nova master: Heal missing simple PCI allocation in the resource tracker  https://review.opendev.org/c/openstack/nova/+/85135909:06
opendevreviewBalazs Gibizer proposed openstack/nova master: Heal PCI allocation during resize  https://review.opendev.org/c/openstack/nova/+/85239609:06
opendevreviewBalazs Gibizer proposed openstack/nova master: Handle PCI dev reconf with allocations  https://review.opendev.org/c/openstack/nova/+/85239709:06
opendevreviewBalazs Gibizer proposed openstack/nova master: Allow enabling PCI tracking in Placement  https://review.opendev.org/c/openstack/nova/+/85046809:06
opendevreviewBalazs Gibizer proposed openstack/nova master: Generate request_id for Flavor based InstancePCIRequest  https://review.opendev.org/c/openstack/nova/+/85383509:06
opendevreviewBalazs Gibizer proposed openstack/nova master: Request filter for PCI in placement  https://review.opendev.org/c/openstack/nova/+/85277109:06
opendevreviewBalazs Gibizer proposed openstack/nova master: Support resource_class and traits in PCI alias  https://review.opendev.org/c/openstack/nova/+/85331609:06
opendevreviewBalazs Gibizer proposed openstack/nova master: Map PCI pools to RP UUIDs  https://review.opendev.org/c/openstack/nova/+/85411809:06
opendevreviewBalazs Gibizer proposed openstack/nova master: Make allocation candidates available for scheduler filters  https://review.opendev.org/c/openstack/nova/+/85411909:06
opendevreviewBalazs Gibizer proposed openstack/nova master: Filter PCI pools based on Placement allocation  https://review.opendev.org/c/openstack/nova/+/85412009:06
opendevreviewBalazs Gibizer proposed openstack/nova master: Store allocated RP in InstancePCIRequest  https://review.opendev.org/c/openstack/nova/+/85412109:06
opendevreviewBalazs Gibizer proposed openstack/nova master: Func test for PCI in placement scheduling  https://review.opendev.org/c/openstack/nova/+/85412209:07
gibisean-k-mooney[m]: the pci stuff can be reviewed up until https://review.opendev.org/c/openstack/nova/+/851358/ and even further if you have the willpower09:09
gibithe top of the series missing unit test coverage and has spotty functional coverage though09:09
sean-k-mooneyack09:09
sean-k-mooneyok so that is everything up to just before the allocation healling09:10
opendevreviewJan Hartkopf proposed openstack/python-novaclient master: add support for microversion 2.93  https://review.opendev.org/c/openstack/python-novaclient/+/81615809:14
Ugglagibi, sean-k-mooney may I request you to check https://review.opendev.org/c/openstack/nova/+/853986 and merge it if ok.09:25
gibiUggla: done, and thanks10:18
sean-k-mooneygibi: by the way i  have started to notice tempest.api.volume.test_volumes_extend.VolumesExtendAttachedTest.test_extend_attached_volume failing semi often in nova-next10:20
Ugglagibi, thank you !10:20
gibisean-k-mooney: do you have an example failure?10:21
sean-k-mooneyhttps://zuul.opendev.org/t/openstack/build/5e64ad2283a24653a424ad91d99a300410:21
gibiack, I will quickly check10:21
sean-k-mooneyit failed on teh vdpa patch10:21
sean-k-mooneybut i have seen it on several others recnetly10:21
sean-k-mooneydo you know where the new elastic search deployment is hosted10:21
sean-k-mooneyim kind of wonderign how often this happens10:22
gibiinteresting10:23
gibihttps://zuul.opendev.org/t/openstack/build/5e64ad2283a24653a424ad91d99a3004/log/compute1/logs/screen-n-cpu.txt#5204010:23
gibilibvirt.libvirtError: invalid argument: disk '/dev/sda' was not found in the domain config10:23
sean-k-mooney ya10:23
sean-k-mooneythats the same failure 10:23
gibisean-k-mooney: https://opensearch.logs.openstack.org/_dashboards/app/login?nextUrl=%2F_dashboards%2Fapp%2Fdiscover%2F%3Fsecurity_tenant%3Dglobal this is the logstash10:24
sean-k-mooneywe dont guarentee the adress of th disk10:24
sean-k-mooney ack i was trying http://logstash.openstack.org/10:24
sean-k-mooneywhich is obviously dead but maybe infra coudl add a redirect to the new location10:24
gibiyah a redirect would be nice10:25
gibihm from the last 10 novan-next failure 9 was due to the above error10:27
sean-k-mooneydo you know the login info?10:27
sean-k-mooneyi think they sent one ot the mail list but i cant find it in my email10:27
sean-k-mooneygibi: ya so we have a log message that says "the libvirt driver cannot honner the request path" or somethign like that when we attach so im supresed we care when detaching10:28
gibiopenstack openstack10:28
sean-k-mooneythanks10:29
gibiit is almost like it started failing on 08.19.10:30
gibiand we merged https://review.opendev.org/c/openstack/nova/+/843700/ in 19th10:30
sean-k-mooneyyep10:31
gibiso I think we enabled an unstable test10:31
gibiI'm proposing a revert10:31
sean-k-mooneyi can revert that i guess10:32
sean-k-mooneyah go for it10:32
sean-k-mooneyso its calling 10:33
sean-k-mooney    def attach_detach_volume(self, server, volume):10:33
sean-k-mooney        attached_volume = self.nova_volume_attach(server, volume)10:33
sean-k-mooney        self.nova_volume_detach(server, attached_volume)10:33
sean-k-mooneyif that is not waiting for sshable then that is l ikely the issue10:33
sean-k-mooneyhttps://review.opendev.org/c/openstack/tempest/+/836065/4/tempest/scenario/test_encrypted_cinder_volumes.py10:34
opendevreviewBalazs Gibizer proposed openstack/nova master: Revert "Test attached volume extend actions in the nova-next job"  https://review.opendev.org/c/openstack/nova/+/85413210:34
sean-k-mooneyi wonder is this another case of that type of error10:34
gibiyeah that can be one issue yes10:35
sean-k-mooneywe might have removed it form teh persitent tomain but not the active domain10:35
sean-k-mooneyand fail with not found on a retry or something10:35
sean-k-mooneyi wonder if the logs shows anything like that10:35
sean-k-mooneyhum no its the resize that is failing10:36
sean-k-mooneybut perhaps the attach has not completed10:36
gibiI'm table this for now. we can merge the revert or somebody can propose a fix to the tempest test but I let that be done by others :)10:38
sean-k-mooneysure :)10:38
gibi(we actually did a naked recheck that showed this error to merge the original patche enabling this test case)10:41
sean-k-mooneythats unfortunet10:42
opendevreviewMerged openstack/nova master: Alphabetizes objects  https://review.opendev.org/c/openstack/nova/+/85398610:43
gibithat shows why we are against naked rechecks :D10:44
sean-k-mooneyi left a comment on the tempest chagne https://review.opendev.org/c/openstack/tempest/+/836065/4#message-a8c9c9794c75555728517230fcab3b7beb8427de10:46
gibithanks10:46
sean-k-mooneyis logsearch a cli for logstash/opensearch ?10:48
sean-k-mooneythat looks like its quite nice to use vs the ui10:49
sean-k-mooneythe ui is not bad bu the cli output is very short and too the point10:49
gibinope it is my own tool grepping logs locally10:50
gibihttps://github.com/gibizer/zuul-log-search10:50
sean-k-mooneyoh ok10:50
sean-k-mooneyso its pulling the logs locally10:50
gibithat is one of my hack and hustle target10:50
gibiyes it pulls the logs locally based on the zuul query you make10:51
gibicaches it locally10:51
gibiand can grep in it10:51
gibialso you can define bug signatures10:51
sean-k-mooneyah nice10:51
gibiand it can try to classify failures by stored signatures10:51
gibiit was my answer to shutting down the old logsearch instance10:52
sean-k-mooneyya10:52
gibi*logstash10:52
sean-k-mooneyonly downside is disk space really10:52
sean-k-mooneybut that is relitivly cheap10:53
sean-k-mooneywhen you do --file compute1/logs/screen-n-cpu.txt --file controller/logs/screen-n-cpu.txt 10:53
sean-k-mooneyis that going to just pull those files10:53
gibiyes10:53
gibiby default it only pulls the job-output.txt10:54
gibito limit disk space usage :)10:54
sean-k-mooneyok so as logn as you knwo what file you want it wont take that mauch space at all10:54
sean-k-mooneyill have to find some time to set it up and try it out10:54
gibimy query for the above failure in nova-next in the last 25 days takes ~1G local disk space10:55
* sean-k-mooney is slightly sad we had 1G of failure logs in 25days10:55
gibithe compute service logs are pretty big10:56
gibi53 compute service logs == 1G10:57
sean-k-mooneyya i guess they are in debug10:57
gibibut I can put the cache dir to a compressed volume and that can safe a lot of space10:57
gibiyes fortunately gate use debug :D10:57
gibis/safe/save/10:57
sean-k-mooneyi rememebr the intel nfv ci used to use a 300G log share that we sotred gziped logs too 10:58
sean-k-mooneythat was enough for about 60 days fo ci logs10:58
sean-k-mooneyonce you compressed everything10:58
gibiyes these logs are highl compressable10:59
sean-k-mooneylike db dumps. the db dump i was working with for the downstream escalation was about 900MB raw or about 60 mb compressed11:00
opendevreviewBalazs Gibizer proposed openstack/nova master: Retry /reshape at provider generation conflict  https://review.opendev.org/c/openstack/nova/+/85135811:44
opendevreviewBalazs Gibizer proposed openstack/nova master: Heal missing simple PCI allocation in the resource tracker  https://review.opendev.org/c/openstack/nova/+/85135911:44
opendevreviewBalazs Gibizer proposed openstack/nova master: Heal PCI allocation during resize  https://review.opendev.org/c/openstack/nova/+/85239611:44
opendevreviewBalazs Gibizer proposed openstack/nova master: Handle PCI dev reconf with allocations  https://review.opendev.org/c/openstack/nova/+/85239711:44
opendevreviewBalazs Gibizer proposed openstack/nova master: Allow enabling PCI tracking in Placement  https://review.opendev.org/c/openstack/nova/+/85046811:44
opendevreviewBalazs Gibizer proposed openstack/nova master: Generate request_id for Flavor based InstancePCIRequest  https://review.opendev.org/c/openstack/nova/+/85383511:44
opendevreviewBalazs Gibizer proposed openstack/nova master: Request filter for PCI in placement  https://review.opendev.org/c/openstack/nova/+/85277111:44
opendevreviewBalazs Gibizer proposed openstack/nova master: Support resource_class and traits in PCI alias  https://review.opendev.org/c/openstack/nova/+/85331611:44
opendevreviewBalazs Gibizer proposed openstack/nova master: Map PCI pools to RP UUIDs  https://review.opendev.org/c/openstack/nova/+/85411811:44
opendevreviewBalazs Gibizer proposed openstack/nova master: Make allocation candidates available for scheduler filters  https://review.opendev.org/c/openstack/nova/+/85411911:44
opendevreviewBalazs Gibizer proposed openstack/nova master: Filter PCI pools based on Placement allocation  https://review.opendev.org/c/openstack/nova/+/85412011:44
opendevreviewBalazs Gibizer proposed openstack/nova master: Store allocated RP in InstancePCIRequest  https://review.opendev.org/c/openstack/nova/+/85412111:44
opendevreviewBalazs Gibizer proposed openstack/nova master: Func test for PCI in placement scheduling  https://review.opendev.org/c/openstack/nova/+/85412211:44
gibisean-k-mooney, stephenfin: solved the todo in https://review.opendev.org/c/openstack/nova/+/851358 so now the pci series is ready for review up until https://review.opendev.org/c/openstack/nova/+/853835/ the rest of the series only ready for high level architectural review, as those patches does not have proper test coverage. The last patch of the series adds some functional tests that proves that 11:46
gibithe scheduling logic is sane though11:46
sean-k-mooneyack11:48
sean-k-mooneyi was looking at some other stuff but ill start on it soon11:48
gibiack11:48
gibistephenfin, sean-k-mooney: I'm really thankful for all these reviews. I know that this is a lot of code and logic11:49
gibi... and now a quick lunch11:50
opendevreviewAmit Uniyal proposed openstack/nova master: Adds check for VM snapshot fail while quiesce  https://review.opendev.org/c/openstack/nova/+/85217113:02
opendevreviewElod Illes proposed openstack/nova stable/stein: DNM: check CI health  https://review.opendev.org/c/openstack/nova/+/85418413:12
*** dasm|off is now known as dasm13:28
sean-k-mooneystephenfin: o/14:40
sean-k-mooneystephenfin: just an fyi i figured out the issue with the live migration test and that is now passing14:41
sean-k-mooneymelwitt: are you around14:41
sean-k-mooneymelwitt: if so can you take a look at this revert https://review.opendev.org/c/openstack/nova/+/85413214:41
sean-k-mooneyits failing pretty consitently14:42
sean-k-mooneyim holding off recheckign the base patch in teh vdpa seriese until that is merged because it will likely get hit by it again14:42
gibielodilles: I've updated the nova meeting agenda for today15:22
elodillesgibi: i've just updated it :D15:22
gibicool, thanks15:22
elodillesgibi: hopefully i did not overwrite any of your changes :S15:23
gibiIt does not seem so15:24
elodilles\o/15:27
gibinova meeting in 2 minutes15:58
gibi#startmeeting nvoa16:00
opendevmeetMeeting started Tue Aug 23 16:00:12 2022 UTC and is due to finish in 60 minutes.  The chair is gibi. Information about MeetBot at http://wiki.debian.org/MeetBot.16:00
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:00
opendevmeetThe meeting name has been set to 'nvoa'16:00
gibi#endmeeting16:00
opendevmeetMeeting ended Tue Aug 23 16:00:17 2022 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:00
opendevmeetMinutes:        https://meetings.opendev.org/meetings/nvoa/2022/nvoa.2022-08-23-16.00.html16:00
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/nvoa/2022/nvoa.2022-08-23-16.00.txt16:00
opendevmeetLog:            https://meetings.opendev.org/meetings/nvoa/2022/nvoa.2022-08-23-16.00.log.html16:00
gibi#startmeeting nova16:00
opendevmeetMeeting started Tue Aug 23 16:00:19 2022 UTC and is due to finish in 60 minutes.  The chair is gibi. Information about MeetBot at http://wiki.debian.org/MeetBot.16:00
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:00
opendevmeetThe meeting name has been set to 'nova'16:00
gibio/16:00
elodilleso/16:01
sean-k-mooneyo/16:01
gibiso lets start16:03
Ugglao/16:03
gibi#topic Bugs (stuck/critical) 16:03
gibi#info One Critical bug16:03
gibihttps://bugs.launchpad.net/nova/+bug/198654516:03
gibithis is the same from last week16:03
gibiabout the novncproxy16:03
gibimelwitt has a fix for the unit test 16:03
gibiand I'm +216:03
gibihttps://review.opendev.org/c/openstack/nova/+/85337916:04
gibiso we need another core on it16:04
gibi#link https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New 8 new untriaged bugs (0 since the last meeting)16:05
gibi#link https://storyboard.openstack.org/#!/project/openstack/placement 27 open stories (+0 since the last meeting) in Storyboard for Placement 16:05
gibi#info Add yourself in the team bug roster if you want to help https://etherpad.opendev.org/p/nova-bug-triage-roster16:06
gibithe next on the roster is artom 16:06
gibiartom: will you take the triage baton?16:06
artomYes16:06
gibicool thanks16:06
gibiand thanks for the triages melwitt 16:06
gibi#info bug baton is being passed to artom16:07
gibiany specific bug we need to discuss here/16:07
gibi?16:07
gibi#topic Gate status16:09
gibiwe have an unstable test in nova-next enabled recently. We are reverting the patch enabled it #link https://review.opendev.org/c/openstack/nova/+/85413216:09
gibithis also needs a second core16:09
gibiuntil that revert lands nova-next will be fairly but not totally red16:10
gibi#link https://zuul.openstack.org/builds?project=openstack%2Fplacement&pipeline=periodic-weekly Placement periodic job status 16:11
gibiit is green16:11
gibi#link https://zuul.openstack.org/builds?job_name=tempest-integrated-compute-centos-9-stream&project=openstack%2Fnova&pipeline=periodic-weekly&skip=0 Centos 9 Stream periodic job status16:11
gibigreen too16:11
gibi#link https://zuul.opendev.org/t/openstack/builds?job_name=nova-emulation&pipeline=periodic-weekly&skip=0 Emulation periodic job runs16:11
gibiand green as well16:11
gibi#info Please look at the gate failures and file a bug report with the gate-failure tag.16:11
gibi#info STOP DOING BLIND RECHECKS aka. 'recheck' https://docs.openstack.org/project-team-guide/testing.html#how-to-handle-test-failures16:11
gibias a note the current nova-next instability could have been caught if we had stopped using blind rechecks :/16:12
gibianything else about the master gate status?16:12
JayFI have a change, https://review.opendev.org/c/openstack/nova/+/853529, which should help with Ironic changes by not running nova-live-migration tests if only Ironic driver is modified. This should help conserve resources and prevent rechecks by avoiding the tests that can't be broken by Ironic. It's my intention to backport this if/when merged. It only needs one additional16:12
JayFcore vote.16:12
gibiJayF: cool added to my list, I will check it after the meeting16:13
JayFthank you gibi, I appreciate it.16:13
gibianything else gate related?16:13
sean-k-mooneyjust yor revert16:13
sean-k-mooneyhttps://review.opendev.org/c/openstack/nova/+/85413216:13
sean-k-mooneyif we can merge that before it wedges the gate it would be good16:13
sean-k-mooneywe have 40 hits today16:13
gibisean-k-mooney: yes, we need melwitt or gmann or stephenfin to look at it and approve the revert16:14
sean-k-mooneyor dansmith if your about16:15
dansmithlooking16:15
gibithanks16:16
gibianything else about the gate?16:16
dansmithgot it16:16
sean-k-mooneyi dont think so16:16
sean-k-mooneyi have not see any other recurrent failures16:17
gibiack, moving on16:17
gibi#topic Release Planning 16:17
gibi#link https://releases.openstack.org/zed/schedule.html16:17
gibi#info Zed-3 is in 1 weeks16:17
gibi#link Zed tracking etherpad: https://etherpad.opendev.org/p/nova-zed-blueprint-status16:17
gibi#link https://etherpad.opendev.org/p/nova-zed-microversions-plan16:17
gibiplease update the https://etherpad.opendev.org/p/nova-zed-blueprint-status etherpad with the status of your work16:18
gibiand if you have someting that blocking your progress then raise it16:19
gibiis there any specific bp we want to talk about today?16:20
gibiI see that https://review.opendev.org/q/topic:bp/volume-backed-server-rebuild+status:open is green but lacking core review16:22
dansmithI initially did a bunch of review on it,16:23
dansmithbut was hoping to get another core to have fresh eyes on it before we merge16:23
sean-k-mooneythe tempet test is red is that related to teh testcases they are adding16:23
dansmithso I'm good to +W that if someone else looks it over, but I had my hands all in it, so I think that'd be better16:23
dansmithsean-k-mooney: is it? it was passing at one point16:23
opendevreviewMerged openstack/nova stable/ussuri: [ironic] Minimize window for a resource provider to be lost  https://review.opendev.org/c/openstack/nova/+/85354016:24
gibiit is behind https://review.opendev.org/q/topic:bp/update-userdata+status:open where I need to go back and re-review. after that I might be able to look at the rebuild sets but no hard promises16:24
sean-k-mooneywell some of the jobs on the tempest repo are red but i dont know if the new ones worked16:24
dansmiththe runs logs are too old, so it needs a recheck16:24
sean-k-mooneyack shall i do it or have you already done it16:25
dansmithI done did it16:25
sean-k-mooneycool16:25
dansmithseems unlikely to merge at this late date if it's behind something else for a microversion,16:25
dansmithbut that kinda sucks because a lot was done on it16:25
dansmithso if someone has time to look that'd be nice, maybe we can FFE it if we need16:26
sean-k-mooneywell it coudl be merge ahead of the other patch16:26
sean-k-mooneywe might want to swap them since there is an openquestion on the update user data pathc16:26
gibiI have the intention to go back the the update user_data patch (that is the next microversion) and also melwitt reviewe that before16:26
dansmithit's a cross-project thing where cinder has done a bunch of work to enable it, so it definitely sucks to cause them to miss again16:26
gibiso we might be able to unblock the it16:27
dansmithalready missed last cycle, IIRC16:27
sean-k-mooneydansmith: there is only one patch in the other fature to merge16:27
sean-k-mooneybut yes it woudl be shame if it missed16:27
dansmithokay well if we can get someone to look it over while we decide or work on the other, that'd be cool16:28
dansmithmaybe melwitt would want to look at it16:28
gibiack16:28
gibiany other bp to raise?16:29
sean-k-mooneyam jsut i think the teanat id one is likely not going to land16:30
gibiit is in merge conflict since we merged the unittest.mock change ~3weeks ago16:30
sean-k-mooneydansmith: i kind of lost track of the SRBAC stuff are there any reverts that still need to land for that16:30
dansmithum16:30
sean-k-mooneyits not really a blue print but basically is there any work we need to have in zed16:31
dansmithI think it merged underneath something from gmann 16:31
sean-k-mooneyok16:31
dansmithbut I'm not seeing it actually16:32
dansmithnope: https://review.opendev.org/c/openstack/nova/+/84802116:32
dansmiththat one and the one above16:32
sean-k-mooneyack so those should be set as a review priorty to land for zed16:32
dansmithmaybe we can get gmann to +2 that and someone else to ack it, yeah16:32
sean-k-mooneyis gmann on pto i have not spoken to them in a few days16:33
dansmithI think something is keeping him busy, but he was around the last few days a little16:33
gibiI added that rbac stuff to the tracking etherpad16:34
dansmithgibi: thanks16:34
sean-k-mooneyi think thats the main set that we needed to talk about16:34
gibiack then moving on16:35
gibi#topic Review priorities 16:35
gibi#link https://review.opendev.org/q/status:open+(project:openstack/nova+OR+project:openstack/placement+OR+project:openstack/os-traits+OR+project:openstack/os-resource-classes+OR+project:openstack/os-vif+OR+project:openstack/python-novaclient+OR+project:openstack/osc-placement)+(label:Review-Priority%252B1+OR+label:Review-Priority%252B2)16:35
gibilooks fairly relevant16:35
gibi#topic Stable Branches16:36
gibielodilles: any news from stable/16:36
gibi?16:36
elodillesyepp16:36
elodilles#info stable/train was unblocked last week (with the backport of the patch that migrated jobs to zuul v3)16:36
elodilles#info now only stable/stein (and older) are blocked: grenade and other devstack based jobs fail with the same timeout issue as stable/train was previously16:36
elodilles#info stable branch status / gate failures tracking etherpad: https://etherpad.opendev.org/p/nova-stable-branch-ci16:36
sean-k-mooney elodilles i think there is an issue with cliff on yoga and older by the way16:37
sean-k-mooneymore for cilometer then us16:37
elodillessean-k-mooney: is it? :-o16:37
sean-k-mooneybut cliff 4. something  droped python 3.6 support16:37
sean-k-mooneyso centos 8 stream jobs are failing16:37
sean-k-mooneyi did see it on stable/victoria on nova however if i am rememebring correctly16:38
elodillesisn't that non-voting?16:38
elodillesi mean in that case it's not blocking the gate16:38
sean-k-mooneynot for nova16:38
sean-k-mooneyi work with peopel that work on ceilometer that are trigng to fix it for there jobs16:39
sean-k-mooneyjust mentioning it as an fyi incase you see stabel review related to it16:39
elodillessean-k-mooney: ack, thanks, i'll look into that16:39
sean-k-mooneyhttps://review.opendev.org/c/openstack/requirements/+/85413016:40
elodillessean-k-mooney: ++16:40
elodillesi've noted that patch16:40
gibianything else about stable?16:41
elodillesnothing else from me16:41
gibithanks elodilles 16:41
elodillesnp16:41
gibi#topic Open discussion 16:42
gibinothing on the agenda. 16:42
gibidoes anybody has a topic still for today?16:42
gibiI take the silence as no16:43
gibiso thanks for joining16:43
gibi#endmeeting16:43
opendevmeetMeeting ended Tue Aug 23 16:43:35 2022 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:43
opendevmeetMinutes:        https://meetings.opendev.org/meetings/nova/2022/nova.2022-08-23-16.00.html16:43
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/nova/2022/nova.2022-08-23-16.00.txt16:43
opendevmeetLog:            https://meetings.opendev.org/meetings/nova/2022/nova.2022-08-23-16.00.log.html16:43
elodillesthanks o/16:43
sean-k-mooneyelodilles: i put the details in the etherpad16:47
sean-k-mooneyhttps://etherpad.opendev.org/p/nova-stable-branch-ci#L2216:47
JayFbtw; thanks to you all for the help with the stable reviews, trying to get the ironic driver as good as it can be in the stable branch. https://review.opendev.org/c/openstack/nova/+/821350 still needs another core vote in victoria16:50
JayFI know you rarely need us, but I'm happy to help if you all run into Ironic shenanigans in the future :D 16:50
gibisean-k-mooney: fyi I created a test case and now I can confirm that different PFs are tracked as separate PCI pools even if they have same kind of VFs. Fortunately both the apply and the consume logic in stats able to consume from two pools for a single pci request. So everything works \o/ even with splitted pools16:51
sean-k-mooneywell there is one open spec that we really need your import on16:51
sean-k-mooneygibi: nice16:52
sean-k-mooneyJayF: if you have not read https://review.opendev.org/c/openstack/nova-specs/+/84201516:52
sean-k-mooneyJayF: we dont currently have a good way forward on that16:52
JayFI've absolutely had backchannel chats with Julia about that (and felt the pain it is trying to solve). I will scratch my brain for it again but I'm not sure I know how to break out of the logjam 16:53
sean-k-mooneywe were hopign to have a face to face session on this at the ptg before it went virtual16:54
sean-k-mooneyit really feels like it needs a white board and some time to talk thing through16:54
JayFI have $800 of Alaska Airlines credit proving my intention to have showed up to that :( 16:54
JayFhopefully we can find time in virtual summit, anyway, although you're right it's often easier to solve things irl16:55
sean-k-mooney+2w on that victoria backport16:56
* sean-k-mooney it may have taken me longer to workout what baranch comes before and after V then to review16:57
JayFwonderful, thanks. I'll have probably three fresh in the morning as I pull everything down another level. 16:57
JayFsean-k-mooney: https://etherpad.opendev.org/p/JayFTODO I put all the branches on this so I'd stop doing that16:57
JayFsean-k-mooney: it's worse for Ironic, we have mid-stream bugfix releases that are numbered, and they might as well be named after colors for all the meaning the names have for me lolsob16:57
sean-k-mooneyi see :)16:58
sean-k-mooneyok im gong to more or less call it a day.17:01
sean-k-mooneyill likely do some code review on my ipad while i cook/order dinner but im going to drop off irc ffor the day17:01
gibisean-k-mooney: have a nice evening o/17:01
opendevreviewElod Illes proposed openstack/nova stable/train: [ironic] Minimize window for a resource provider to be lost  https://review.opendev.org/c/openstack/nova/+/85354617:15
JayFelodilles: thanks, I had that on my todo list but I appreciate it \o/ 17:16
elodillesJayF: np, just re-applied the cherry-pick to have the right format of the commit message, otherwise it seems right, so +2'd it already17:22
JayFSo, what you want in the commit is17:22
elodillesit's not the commit message in general o:)17:23
JayFcherry-picked-from (master SHA)\ncherry-picked-from (master-1 sha)\n etc etc17:23
JayFwhich is achieved by a clean cherry pick from stable N -> stable N-117:23
elodillesbut to do the cherry-picking branch-by-branch,17:23
elodillesso that one can see that the cherry picking was in the right order, from the right (latest) patch sets, etc.17:24
JayFso for my most recent master change, I'd cherry-pick it to stable/yoga, then cherry pick *the yoga change* onto xena, and so on17:25
JayFand if at any point one patch in the cycle is changed or rebased (whoops), the ones after it have to be re-cherry-picked17:25
elodillesJayF: exactly :)17:26
JayFis this a nova-ism? Or something generally that most OpenStack projects do that Ironic has never really enforced?17:26
elodillesthe backports should be done branch-by-branch according to stable policy17:27
JayFack, so I'll update how I do it in Ironic world too17:28
elodillesbut yes, probably only nova has this check as (we) stable cores were picky about this one o:)17:28
JayFYeah, I've been focused on stable reviews in Ironic for a long time, and this is *never* something I've checked for17:29
elodillesto avoid wrong/not latest/missing backports17:29
JayFfrankly, I doubt I'd -1 a change for this now -- but I do always manually check the earlier branches17:29
JayFand I'll start doing it right for me ;D 17:29
JayF**my changes17:29
elodilles:)17:30
elodillesalso note, that the rebase is unnecessary as Zuul is always doing it. the only time when it is needed, when there's a merge conflict17:32
JayFtbh rebase button is my shortcut for "clear verify and re-run tests on this" except now I realize there are negative side effects in this context17:43
opendevreviewBalazs Gibizer proposed openstack/nova master: Request filter for PCI in placement  https://review.opendev.org/c/openstack/nova/+/85277117:59
opendevreviewBalazs Gibizer proposed openstack/nova master: Support resource_class and traits in PCI alias  https://review.opendev.org/c/openstack/nova/+/85331617:59
opendevreviewBalazs Gibizer proposed openstack/nova master: Map PCI pools to RP UUIDs  https://review.opendev.org/c/openstack/nova/+/85411817:59
opendevreviewBalazs Gibizer proposed openstack/nova master: Make allocation candidates available for scheduler filters  https://review.opendev.org/c/openstack/nova/+/85411917:59
opendevreviewBalazs Gibizer proposed openstack/nova master: Filter PCI pools based on Placement allocation  https://review.opendev.org/c/openstack/nova/+/85412017:59
opendevreviewBalazs Gibizer proposed openstack/nova master: Store allocated RP in InstancePCIRequest  https://review.opendev.org/c/openstack/nova/+/85412117:59
opendevreviewBalazs Gibizer proposed openstack/nova master: Func test for PCI in placement scheduling  https://review.opendev.org/c/openstack/nova/+/85412217:59
opendevreviewBalazs Gibizer proposed openstack/nova master: Test PCI scheduling during move operations  https://review.opendev.org/c/openstack/nova/+/85424717:59
opendevreviewMerged openstack/nova master: Revert "Test attached volume extend actions in the nova-next job"  https://review.opendev.org/c/openstack/nova/+/85413218:08
opendevreviewElod Illes proposed openstack/nova stable/queens: WIP: [stable-only] Remove grenade jobs  https://review.opendev.org/c/openstack/nova/+/85425218:23
opendevreviewMerged openstack/nova master: nova-live-migration tests not needed for Ironic  https://review.opendev.org/c/openstack/nova/+/85352918:47
opendevreviewMerged openstack/nova stable/victoria: Ignore plug_vifs on the ironic driver  https://review.opendev.org/c/openstack/nova/+/82135018:47
opendevreviewJay Faulkner proposed openstack/nova stable/ussuri: Ignore plug_vifs on the ironic driver  https://review.opendev.org/c/openstack/nova/+/82135119:14
JayFhttps://review.opendev.org/c/openstack/nova/+/853546 (to stable/train) needs one more core review, then this one will be backported as far as I plan to take it19:15
opendevreviewJay Faulkner proposed openstack/nova stable/yoga: nova-live-migration tests not needed for Ironic  https://review.opendev.org/c/openstack/nova/+/85425719:16
opendevreviewJay Faulkner proposed openstack/nova stable/yoga: nova-live-migration tests not needed for Ironic  https://review.opendev.org/c/openstack/nova/+/85425719:17
opendevreviewJay Faulkner proposed openstack/nova stable/ussuri: Ignore plug_vifs on the ironic driver  https://review.opendev.org/c/openstack/nova/+/82135119:18
JayFelodilles: took a couple of tries, but I think I got them right now ^^ had to do two rounds, first used `-x` then I realized I needed the `-X` :D19:20
JayFty for the guidance19:20
JayFhttps://review.opendev.org/c/openstack/nova/+/821351 and https://review.opendev.org/c/openstack/nova/+/854257 should both be good for reviews now, no-change-backports of stuff already merged, shouldn't be controversial :D 19:22
JayFthank you all again for helping me plow thru this ironic-driver-backport tech debt19:22
melwittsean-k-mooney, gibi: I'm off today and tomorrow, thanks for getting the revert done and sorry for the trouble ☹️ 22:08
*** dasm is now known as dasm|off22:29

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!