16:00:00 #startmeeting nova 16:00:00 Meeting started Tue Jun 14 16:00:00 2022 UTC and is due to finish in 60 minutes. The chair is bauzas. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:00:00 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 16:00:00 The meeting name has been set to 'nova' 16:00:06 howdy back 16:00:19 o/ 16:00:26 and welcome on our first June nova meeting 16:00:51 o/ 16:01:04 o/ 16:01:36 * bauzas just hopes we'll have more people joining 16:01:43 but we can start 16:01:50 o/ 16:02:03 #topic Bugs (stuck/critical) 16:02:08 #info No Critical bug 16:02:13 o/ 16:02:13 #link https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New 14 new untriaged bugs (+0 since the last meeting) 16:02:18 #link https://storyboard.openstack.org/#!/project/openstack/placement 26 open stories (0 since the last meeting) in Storyboard for Placement 16:02:22 #info Add yourself in the team bug roster if you want to help https://etherpad.opendev.org/p/nova-bug-triage-roster 16:02:31 I have to admit publicly I feel ashamed 16:02:47 I forgot about the baton when it was mine 16:02:52 throw me tomatoes 16:03:01 🍅 16:03:04 but alas I triaged one bug :) 16:03:32 apparently, preparing the Summit trip and doing bug triage doesn't mix on my side 16:03:52 I took the baton from bauzas in Berlin in person \o/ 16:03:59 literally 16:04:14 there was a normal amount of bug inflow 16:04:16 https://etherpad.opendev.org/p/nova-bug-triage-20220607 16:04:19 it could have been an olympic torch 16:04:47 does that count as carry on baggage? 16:05:03 it could be your personal item 16:05:08 :) 16:05:08 depends on the size I guess 16:05:19 or it could be seen as a sport gear 16:05:23 anyway 16:05:24 I saw two interesting bugs 16:05:28 thanks gibi 16:05:30 https://bugs.launchpad.net/nova/+bug/1978372 numa_fit_instance_to_host() algorithm is highly ineffective on higher number of NUMA nodes 16:06:06 sean-k-mooney updated me that this is a known ineffciency of our algo 16:06:19 what kind of hardware has this large amount of NUMA nodes ? 16:06:49 * bauzas is always unpleasantly surprised by all the new things that are created around him 16:07:27 I'm not sure 16:07:35 but I accept that it is possible 16:07:38 8 NUMA nodes seems large to me, but I'm not a tech savvy 16:07:38 bauzas: most recnet amd servers 16:07:49 16 numa nodes is not uncommon now 16:08:02 my brain hurts. 16:08:08 you can get 16 numa nodes in a singel socket now 16:08:18 and i have see systems with 64 16:09:09 * bauzas is network-bound by his gigabit switches at home while he can download at 10 16:09:27 our current packing default falls apart after about 4-8 numa nodes 16:09:34 so right now we are slow by default, but if numa spread is enabled instead of the default pack then it is much better a sean-k-mooney discovered 16:09:37 anyway, sounds an opportunity for optimization then 16:10:04 i have a patch in flight to change the default 16:10:13 the whole packing strategy is hidden within the code 16:10:13 ill work on the release note and push it later 16:10:21 we can continue the discussion there if you like 16:10:27 sure 16:10:36 bauzas: yep its also not part of the api contract and never was 16:11:10 * bauzas shrugs 16:11:18 so the other bug I would like to mention 16:11:21 https://bugs.launchpad.net/nova/+bug/1978549 Placement resource_providers table has dangling column "can_host" 16:11:33 anyway, I understand people wondering why our packing stragegy should struggle only after 16 nodes to iterate 16:11:33 I marked as wontfix with a small note in the placement documentation 16:11:48 https://review.opendev.org/c/openstack/placement/+/845730 16:12:24 this was a mistake back in stable/stein 16:12:34 and I don't want to go back and touch DB migrations there 16:13:39 that is all I had for bug triage this week 16:13:45 gibi: can_host is not part of the DB contract ? 16:13:53 I mean the model 16:14:01 it was removed from the DB model since stein 16:14:13 but we never added a DB migration to drop the coulmn from the schema 16:14:33 but when Placement was split out of nova a new initial DB schema was defined but now without can_host 16:14:37 hence the inconsistency 16:14:41 on the schema level 16:14:45 but nothing is uses can_host 16:14:52 oh I understand 16:16:05 but, if this is post-Stein, the table is removed anyway, no ? 16:16:31 as said in 'Finalize the upgrade' 16:17:05 i think the issue here is they are doing a postgres to mariadb migration 16:17:22 so they were using placement manage to create the new db schma 16:17:27 then trying to do a data migration 16:17:37 and there orginal db had the column 16:17:41 btu the target does not 16:18:16 so if they drop the colum on the souce db then do the data migration it would be fine 16:18:17 ok, I didn't want to enter into the details, let's move on, I think it's safe bet what gibi did 16:18:26 ack 16:19:15 melwitt: are you OK with bug triaging this week or do you want me to do it due to my negligence last week as a punition ? 16:19:27 the latter is fine to me 16:19:53 bauzas: sure, maybe better bc I am out on pto next week 16:20:10 melwitt: cool, then I'll steal it from gibi 16:20:16 cool thanks 16:20:23 #info Next bug baton is passed to bauzas 16:20:48 if you don't mind, I'll pass you the baton next week 16:21:01 or we could give it to anyone else 16:21:01 yes that is ok 16:21:27 moving on 16:21:33 #topic Gate status 16:21:35 * gibi feels sudden emptiness in his life 16:21:38 #link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure Nova gate bugs 16:21:44 #link https://zuul.openstack.org/builds?project=openstack%2Fplacement&pipeline=periodic-weekly Placement periodic job status 16:21:48 #link https://zuul.opendev.org/t/openstack/builds?job_name=nova-emulation&pipeline=periodic-weekly&skip=0 Emulation periodic job runs 16:21:48 😂 gibi 16:21:51 #info Please look at the gate failures and file a bug report with the gate-failure tag. 16:21:55 #info STOP DOING BLIND RECHECKS aka. 'recheck' https://docs.openstack.org/project-team-guide/testing.html#how-to-handle-test-failures 16:22:02 voilà 16:22:17 hadn't anything to tell gate-wise 16:22:35 anything anyone ? 16:22:53 gibi: I don't feel exactly empowered with the baton y'know 16:23:42 ok, next topic then 16:23:47 #topic Release Planning 16:23:54 #link https://releases.openstack.org/zed/schedule.html 16:24:01 #info Zed-2 is in 4 weeks, mind your specs 16:24:33 as a reminder, we'll have a SpecApprovalFreeze on Zed-2 16:25:08 fwiw, here is the current list of accepted blueprints, including specless ones : https://blueprints.launchpad.net/nova/zed 16:25:20 (I eventually updated it one hour before...) 16:25:34 Oh snap, that's in 2 weeks 16:25:40 no 16:25:43 July 14 16:25:49 wtf brain 16:25:54 unless I'm counting wrong 16:25:56 Sorry, ignore me, carry on 16:26:19 that being said, as the clock ticks, next week, we'll discuss of a spec review day 16:26:39 just sayin 16:26:56 next topic 16:27:00 #topic OpenInfra Summit 16:27:13 lemme just do a quick wrap-up 16:27:16 #info bauzas, gibi and stephenfin attended the summit 16:27:21 #info Nova meet-and-greet Operators feedback session on Wednesday, June 8, 2:50pm - 3:20pm got positive feedback 16:27:23 (there was beer) 16:27:27 we had a large audience 16:27:36 gibi: was it tasty 16:27:42 fun times 16:27:44 good beer 16:27:46 gibi: not during sessions tho 16:27:52 yeah, bad timing 16:27:57 bauzas: thats good to hear was the nova session well attended 16:28:03 let's claim this was a productive session 16:28:10 sean-k-mooney: packed room 16:28:18 excelent 16:28:22 Full glass, full room, nice 16:28:43 I think this was well deserved, most of the operators thought we disconnected a bit for too long 16:28:53 and the PTG thing doesn't help 16:29:38 one outcome is at least a strong need for a nova recap at every possible gathering 16:30:04 at least they were expecting one at the Summit, but I have to admit I didn't made it 16:30:12 and the OpenInfra Live thing was on April 16:30:26 I guess only a few of them saw it 16:30:34 the project updates went live on youtube 2 weeks ago 16:30:39 I know 16:30:41 but i woudl assume many did not see it 16:30:50 that's the OpenInfra Live thing I mentioned 16:31:11 apparently, people pay more attention to cycle highlights when it's in-person 16:31:17 right but that may have been in april but the videos only got published on youtube in june 16:31:47 anyway, something easily solvable 16:32:01 one other thing, communication 16:32:10 not a surprise, our ML isn't read 16:32:37 and given they lag a lot, they don't think this is a valuable time to chime in 16:32:59 Wait, so ops show up to Summit, but don't read the ML? How do they know when Summit is? ;) 16:33:04 (they lag by the number of releases) 16:33:14 artom: easy answer : Twitter 16:33:27 and infra foundation marketing 16:33:29 yes, we was asked to tweet more 16:33:36 someone very seriously explained to me they'd prefer nova tweets 16:33:43 We as in the developers? o_O 16:33:45 yeah summit info is communicated in many other ML and places not only openstack-discuss 16:33:50 artom: yes, please :) 16:33:51 huh. 16:33:55 could we auto treat the release notes some how 16:34:02 that's crazy, IMHO 16:34:18 sean-k-mooney: they know about our prelude 16:34:27 but again, they laaaag 16:34:30 * sean-k-mooney rememebrs the april fools twitter as a message bus spec 16:34:39 If they lag releases, what's the point of tweeting, presumably about stuff we're working on *now*? 16:35:11 build interest and involvement 16:35:16 sean-k-mooney: I'm half considering to register a Twitter handle like @Yo_the_Openstack_Nova_gang 16:35:26 chateaulav: +1 16:35:48 But what if I'm an anti-social curmudgeon? 16:35:58 artom: heh 16:36:12 anyway, that one wasn't an easy problem to solve 16:36:22 operator involvement with developers is one of the key and open issue in board meeting too. and TC also raised it to them. 16:36:29 fwiw, I proposed them to only register to 'ops' and 'nova' ML tags 16:36:41 some idea is to combine the events ops meetup and developers one but let's see 16:36:41 both in conjunction 16:36:47 i am too, dont have to be a genius. just start with little things. if it goes to the ML and you think is worthwhile then tweet it and reference the ML archive and irc chat 16:36:48 gmann: please 16:37:01 gmann: the best way to adress that woudl proably be to converge the events and bring back the devs summit 16:37:03 gmann: I feel the community more fragmented now we're spît 16:37:05 spliut 16:37:12 sean-k-mooney: +1 16:37:17 yeah, true 16:37:32 chateaulav: as I said, I begged them to correctly use the ML tags 16:37:35 sean-k-mooney: +1 16:37:47 we got separated when we combined the things :) 16:38:09 and I ask people to *not* make use of [nova][ops] for communicating unless here we agree on the need to engage 16:39:09 #action nova team will only exceptionnally make use of [nova][ops] for important communication to ops. If you're an Ops, feel free to register to both tags in the ML 16:39:19 Tbf, the ML lately seems to be openstack-support, anecdotally 16:39:43 not entirly 16:39:50 mostly 16:39:51 artom: alas, we merged openstack@ openstack-dev@ and openstack-ops@ 16:39:58 we do discuss gate issues and some dev issues 16:40:03 So maybe there is room for dev -> ops announcement-type stuff. 16:40:11 openstack@ was the place for troubleshooting 16:40:13 sean-k-mooney, right, I'm missing a "mostly" in there 16:40:39 @Nova_PTL twitter account? :D 16:40:51 anyway, there are things the nova team can solve and there are other things that are way out of our team scope :) 16:40:53 so we spilt the events and merged the lists. if only we did the reverse :) 16:41:04 im not sure there is much we can do right now to adress this topic 16:41:16 sean-k-mooney: correct and I want to move on 16:41:30 this wasn't an ask to find a solution, just a feedback 16:41:53 You can't tell us "plz tweet moar" and not expect the convo to derail :P 16:41:56 #link https://etherpad.opendev.org/p/r.ea2e9bd003ed5aed5e25cd8393cf9362 readonly etherpad of the meet-and-greet session 16:42:20 artom: I personnally stopped twitting unless exceptional opportunities, not the one to blame 16:42:40 now, back to productive things 16:42:44 * artom never tweets, but always twit 16:42:48 you'll see a long list of complains 16:43:14 some of which have been adresed in newer releases 16:43:19 I encourage any of you to go read the etherpad and amend it (with the write URL of course) 16:43:23 bauzas: any feedback on RBAC scope things if that is discussed in nova sessions also other than ops meetup ? 16:43:46 sean-k-mooney: yeah, I've seen you munging a lot of them, thanks 16:44:04 gmann: I asked about it, this was too way advanced for them 16:44:19 but I pointed them the links to the new rules and personans 16:44:22 personas* 16:44:46 also, this was a 30-min session, 16:45:00 bauzas: ok 16:45:01 so, please understand we were basically only able to scratch the surface 16:45:10 gmann: we had another session around service roles 16:45:13 some of the pain points are on our backlog 16:45:33 so its good that operaters have vlaidated that they still care about them 16:45:39 bauzas: I understand, just checking in case any specific feedback we got from nova sessions 16:45:49 for the pain points, I'll diligently try to make sure all of them are adressed 16:45:51 im thinkign of iothread and virtio-multiqueue 16:46:02 gmann: honestly this was frustrating 16:46:20 gibi: ack. do you have any link of that, I will combine those to discuss in RBAC meeting next week 16:46:26 gmann: my sort summary on the serivce roles https://meetings.opendev.org/irclogs/%23openstack-nova/%23openstack-nova.2022-06-13.log.html#t2022-06-13T06:43:52 16:46:28 give me 30 mins more and I could have made operators to sign off for sending herds of contributors to the nova project 16:46:40 gibi: thanks 16:47:08 don't be surprised if I'm pinging some of you 16:47:17 I want the etherpad to be curated 16:47:18 gmann: and this is the session etherpad but it is a bit of a mess https://etherpad.opendev.org/p/deprivilization-of-service-accounts 16:47:37 gibi: this wasn't a mess 16:47:47 this was rather a prank, I guess 16:48:04 then I was pranked :) 16:48:13 exactly my point 16:48:19 step 1 : propose a forum session 16:48:28 step 2: let gibi see it 16:48:38 step 3 : make sure gibi will attend it 16:48:41 in princiapl we shoudl be able to lable all endpoing that are used for inter service comunicatoin as needing the service role 16:48:52 step 4: don't attend your own session and let gibi lead it instead 16:48:59 step 5 : profit. 16:49:15 (just background: when I entered the room for that session I was cornered that there is nobody who can lead the session) 16:49:16 the service role really shoudl not be able to acces any api other then the inter service apis 16:50:01 that woudl allow use to entirly drop our use fo the admin role eventually 16:50:02 sean-k-mooney: yes, that is direction we are going and has to be a careful audit to verify this. 16:50:10 can we stop on the forum discussions ? 16:50:17 yes we can move on 16:50:20 anyone having a last question or remark ? 16:50:35 (just timeboxing, sorry) 16:50:43 no worries 16:50:46 #topic Review priorities 16:50:51 #link https://review.opendev.org/q/status:open+(project:openstack/nova+OR+project:openstack/placement+OR+project:openstack/os-traits+OR+project:openstack/os-resource-classes+OR+project:openstack/os-vif+OR+project:openstack/python-novaclient+OR+project:openstack/osc-placement)+label:Review-Priority%252B1 16:50:55 #link https://review.opendev.org/c/openstack/project-config/+/837595 Gerrit policy for Review-prio contributors flag. Naming bikeshed in there. 16:51:08 #action bauzas to propose a revision of https://review.opendev.org/c/openstack/project-config/+/837595 16:51:15 #link https://docs.openstack.org/nova/latest/contributor/process.html#what-the-review-priority-label-in-gerrit-are-use-for Documentation we already have 16:51:20 that's it on my side 16:51:59 I encourage cores to make use of the flag if they wish 16:52:06 #topic Stable Branches 16:52:12 elodilles: your time 16:52:15 #info stable/train is blocked - melwitt's fix: https://review.opendev.org/c/openstack/nova/+/844530/ 16:52:24 #info stable branch status / gate failures tracking etherpad: https://etherpad.opendev.org/p/nova-stable-branch-ci 16:52:37 release patches proposed (yoga, xena, wallaby): https://review.opendev.org/q/project:openstack/releases+is:open+intopic:nova 16:52:50 yep i was going to proceed with merging https://review.opendev.org/c/openstack/nova/+/844530 but wanted to ask if there were any objections 16:52:57 i have also commended on the release patches 16:53:10 most of the patches i wanted to land are now landed over night 16:53:25 cool, I'll do a bit of reviews then 16:53:52 any other point to raise about stable ? 16:54:01 nothing else i think 16:54:17 cool 16:54:21 sean-k-mooney bauzas : thanks for looking at the release patches 16:54:21 last point then 16:54:32 elodilles: happy too 16:54:34 elodilles: I have to do it, sean-k-mooney told me already :) 16:54:40 :] 16:54:52 #topic Open discussion 16:54:58 there were nothing on the agenda 16:55:10 for the sake of those last 5 mins, any item to raise ? 16:55:18 bauzas: I realized it would be better if I did bugs this week bc if I'm out next week, that's even less time 😆 16:55:37 I won't be at the next meeting but I can put my bug etherpad link on the agenda for yall 16:55:41 melwitt: I'm both flexible and ashamed 16:55:48 melwitt: pick anytime you want 16:55:54 and I'll do the overlap 16:55:57 the only item i was gong to raise was releases. we had a request to do stable release last week but that is proceeding anyway 16:55:57 bauzas: ok, I will do this week. sorry for the confusion 16:56:06 melwitt: np 16:56:32 I guess not a lot of people are reading our weekly meeting and even less of them do bug triage 16:56:47 weekly minutes* 16:57:12 but, not a reason for anarchy with no meetings and agenda ! :D 16:57:27 (and proper highlights) 16:57:33 we should try to have our meeting on twitter ;) 16:57:41 OK, I guess we can call the wrap 16:58:08 gibi: I was surprised noone debated on the tool itself 16:58:23 * sean-k-mooney looks side eyed at gibi 16:58:26 I could instagram nice pictures of me coding 16:58:42 like, me outside coding 16:58:48 me inside in my office room 16:58:49 start a twitch channel 16:58:53 totally we should all just stream our coding on twitch :) 16:59:03 I'm feeling too old 16:59:32 but at least I'm happy to hear the TC be young-minded with Tik-Tok releases 16:59:40 :D 16:59:51 on that last word, 16:59:59 #endmeeting