14:00:35 <efried> #startmeeting nova 14:00:36 <openstack> Meeting started Thu Apr 4 14:00:35 2019 UTC and is due to finish in 60 minutes. The chair is efried. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:00:37 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:00:39 <openstack> The meeting name has been set to 'nova' 14:00:50 <takashin> o/ 14:00:51 <edleafe> \o 14:00:58 <gmann> o/ 14:01:10 <mdbooth> o/ 14:01:11 <artom> o/ 14:01:12 <mriedem> . 14:01:15 <cdent> o/ 14:01:17 <bauzas> \o 14:01:20 <tssurya> o/ 14:01:20 <artom> Actually, ~o~ 14:01:41 <alex_xu> o/ 14:01:42 * bauzas doing a double meeting at the same time, feeling a bit schizophrenic 14:02:07 <efried> artom: are you swimming? 14:02:15 <artom> edleafe, waving :D 14:02:21 <artom> Err, efried 14:02:36 <efried> oo, the robot, rad 14:02:52 <edleafe> Heh, I thought it was a new hairstyle 14:03:05 <efried> whoops, /me forgot to save the agenda, done, refresh if you had it open 14:03:07 <efried> #link agenda https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting 14:03:23 <efried> #link last week's minutes Minutes from last week: http://eavesdrop.openstack.org/meetings/nova/2019/nova.2019-03-28-21.02.html 14:03:30 <efried> no fups from that afaict, anyone? 14:03:39 <edleafe> artom: https://goo.gl/images/NXekAP 14:03:43 <stephenfin> o/ 14:04:07 <artom> edleafe, *shudder* 14:04:18 <efried> #topic Release News 14:04:18 <efried> #link Stein release schedule: https://wiki.openstack.org/wiki/Nova/Stein_Release_Schedule 14:04:18 <efried> #info Stein RC freeze is TODAY, Thursday, April 4th 14:04:18 <efried> #link Stein RC potential changes tracking: https://etherpad.openstack.org/p/nova-stein-rc-potential 14:04:25 <efried> #link RC2 proposed https://review.openstack.org/649656 14:04:25 <efried> #link Fix non-NIC VFs bombing n-cpu (merging) https://review.openstack.org/#/c/649630/ 14:04:25 <efried> #link VGPU docs (merged) https://review.openstack.org/649454 14:04:36 <efried> That last one wouldn't have been enough to prompt an RC2 by itself, but since we're doing one... 14:04:39 <efried> and last but not least... 14:04:44 <efried> #link Keys to the (database) kingdom in versioned notifications https://review.openstack.org/#/c/649775/ 14:04:54 <efried> That *just* got a new PS, please review and merge asap. 14:05:06 <mriedem> in the future, 14:05:16 <mriedem> i'd ask people be mindful about what we shove into versioned notification payloads, 14:05:22 <mriedem> just because we can send something doesn't mean we should 14:05:57 <efried> is there a doc or (probably better) code comment we can throw somewhere obvious to that effect? 14:06:22 <mriedem> docs are here https://docs.openstack.org/nova/latest/reference/notifications.html 14:06:29 <mriedem> i don't know if there is a warning for contributors 14:06:42 <mriedem> code comment would be hard since the payloads are in lots of places 14:06:46 <mriedem> it's really on the core team 14:06:57 <efried> k 14:07:21 <efried> any other release topics? 14:07:22 <mriedem> i can follow up with a docs patch to add a warning 14:07:27 <gmann> guidelines of " donot/careful about exposing this " can be great. 14:07:27 <efried> thanks mriedem 14:07:46 <artom> There's a What should be in the notification payload para 14:07:48 <Kevin_Zheng> o/ 14:07:56 <artom> Could be followed with a What should NOT be in the notification payload 14:07:56 <mriedem> artom: yeah that's where i'm going to put a warning 14:08:03 <efried> ++ 14:08:16 <efried> #topic Bugs (stuck/critical) 14:08:16 <efried> No Critical bugs 14:08:16 <efried> #link 69 new untriaged bugs (down 1 since the last meeting): https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New 14:08:16 <efried> #link 7 untagged untriaged bugs (down 2 since the last meeting): https://bugs.launchpad.net/nova/+bugs?field.tag=-*&field.status%3Alist=NEW 14:08:27 <efried> Gate status 14:08:27 <efried> #link check queue gate status http://status.openstack.org/elastic-recheck/index.html 14:08:41 <efried> looks like mostly timeout-y things. 14:08:51 <efried> 3rd party CI 14:08:51 <efried> #link 3rd party CI status http://ciwatch.mmedvede.net/project?project=nova&time=7+days 14:09:11 <mriedem> grenade py3 jobs will be still wonky until https://review.openstack.org/#/c/649096/ lands 14:09:38 <efried> a few disturbing hits in unit test jobs - look for the ones where one out of the three pyXX jobs fails. Some kind of odd database race. 14:10:19 <mriedem> we likely need a new e-r query for that 14:10:22 <efried> don't know what it's about, but may try to get an e-r def in place, ... 14:10:23 <efried> yeah 14:10:44 <efried> cause I just learned about e-r this week. I know only enough to be dangerous 14:11:02 * mdbooth loves weird database races if we have a couple of examples 14:11:17 <efried> mdbooth: oh, perfect - there's several examples. 14:11:20 <mdbooth> How much of a problem is it? 14:11:50 <efried> Meh, kills one job in a dozen maybe? 14:11:56 <mdbooth> Moderate 14:12:45 <efried> mdbooth: If you look in http://ciwatch.mmedvede.net/project?project=nova&time=7+days along the openstack-tox-pyXX rows, ignore all green or all red, look for the ones with just one X, click on it. The failures are always in the same small handful of tests, weird DB-ish things. 14:13:52 <efried> thanks for the help 14:14:04 <efried> #action efried to look into adding e-r query for ^ 14:14:28 <efried> #topic Reminders 14:14:28 <efried> #link Spec review day proposed next Tuesday, April 9th http://lists.openstack.org/pipermail/openstack-discuss/2019-April/004484.html 14:14:47 <efried> So far a small number of people have responded about ^ 14:15:02 <efried> Now that I have a captive audience 14:15:20 <efried> is next Tuesday good for folks to block off for reviewing/discussing/updating specs? 14:15:26 <artom> +1 14:15:27 <cdent> yes 14:15:57 <bauzas> I'm good with it but I was about to write something 14:16:13 <bauzas> having a second review day in one week later is a bit impactful 14:16:19 <bauzas> not sure it's healthy 14:16:33 <bauzas> so, I'm all sold on this review day, not the later 14:16:40 <bauzas> if that's worth mentioning 14:16:59 <mriedem> unhealthy how? if you can't do a 2nd sprint, that's fine. 14:17:12 <mriedem> i think the idea is flush as much as possible before the ptg 14:17:18 <efried> ^ 14:17:23 <mriedem> because we're all going to be burned out by saturday 14:17:34 <gmann> +1 14:17:47 <gmann> it can cut down the PTG topics also. 14:18:05 <bauzas> well, ok 14:18:13 <bauzas> that's not that I can't 14:19:00 <bauzas> just that I think I'll feel a bit exhausted after two review days in a week, but fair 14:19:09 <bauzas> but yeah, your point on the PTG is valid 14:19:13 <bauzas> so cool with me 14:19:17 <mriedem> just remember to stretch 14:19:37 <bauzas> :) 14:19:46 <artom> And drink loads of water (or others) 14:20:45 <efried> okay, cool, Ima put up an etherpad where people wanting their specs to get attention that day can list them, make it easier for reviewers to discover rather than picking from the 60-odd open specs out there. 14:20:54 <efried> #efried to fup with review day logistics 14:21:06 <efried> #action efried to fup with review day logistics 14:21:06 <efried> rather 14:21:21 <efried> any other reminders? 14:21:35 <artom> Do we have priorities for train? Should those spec get priority for spec review day? 14:21:48 <efried> mm 14:21:55 <efried> we usually set priorities at/after the ptg 14:22:07 <efried> but that doesn't mean we can't do some pre-prioritizing 14:22:17 <mdbooth> Is it worth having a spec for privsep? It's more of a theme. 14:22:43 <efried> We can have a specless bp to continue the rootwrap-to-privsep conversion for train 14:23:12 <efried> and I would like to have a backlog spec proposed at some point describing how we get from the current privsep to the golden age of security perfection that's been discussed recently in the ML. 14:23:30 <mriedem> specs are point in time for a release, privsep is a long-term commitment, so if we need docs, as suggested in that ML thread, then let's add docs 14:23:45 <dansmith> agree 14:23:52 <efried> #link privsep docs https://review.openstack.org/#/c/649997/ 14:24:13 <mriedem> also, related to that, 14:24:30 <efried> having a specless bp could help us keep review focus on it for train so we can get more of that series merged. 14:24:31 <mriedem> do we have any nova-specific docs about configuring nova's rootwrap compute.filters for the privsep-helper? 14:24:36 <mriedem> because that is all black magic for the most part 14:24:36 <efried> It's over a year old. 14:24:51 <mriedem> and i don't think it's documented, deployment tools just had to deal with it back when it was required 14:25:19 <dansmith> mriedem: asI understand it, we only need one rootwrap rule for privsep itself 14:25:22 <mriedem> also related, sighup'ing nova-compute kills nova-compute b/c the privsep-helper child processes are gone 14:25:40 <artom> ... that sounds quite bad... 14:25:45 <efried> we decided that ^ was latent since at least rocky, right? 14:25:48 <mriedem> dansmith: i'm just not sure if there is anything clear in our docs about configuring rootwrap for privsep 14:25:49 <mdbooth> It's also explicitly by design 14:25:54 <dansmith> it's an oslo.service regression 14:26:01 <mriedem> efried: yeah i also recreated it in rocky 14:26:03 <dansmith> mdbooth: what is explicitly by design? 14:26:17 <mdbooth> dansmith: privsep helper doesn't restart if it ends 14:26:29 <dansmith> mdbooth: it's not supposed to end on HUP 14:26:46 <mdbooth> Ah, ok. Is it supposed to reload its context? 14:26:48 <dansmith> mdbooth: and also, it's definitely expected to be able to restart privsep if you're using a root helperto start 14:26:53 <mriedem> this is the only rootwrap mention in our docs https://docs.openstack.org/nova/stein/cli/nova-rootwrap.html - not even in the install guide 14:27:05 <dansmith> mdbooth: the bug is unrelated to privsep, it's in oslo.service killing things it shouldn't 14:27:11 <mdbooth> dansmith: ack 14:27:27 <mriedem> https://docs.openstack.org/nova/stein/search.html?q=privsep 14:27:31 <mriedem> pretty sad 14:27:34 <mriedem> i'll report a docs bug 14:27:46 * mriedem uses trump voice when he says sad 14:28:52 <efried> okay, moving on? 14:29:05 <efried> #topic Stable branch status 14:29:05 <efried> #link stable/stein: https://review.openstack.org/#/q/status:open+(project:openstack/os-vif+OR+project:openstack/python-novaclient+OR+project:openstack/nova)+branch:stable/stein 14:29:05 <efried> #link stable/rocky: https://review.openstack.org/#/q/status:open+(project:openstack/os-vif+OR+project:openstack/python-novaclient+OR+project:openstack/nova)+branch:stable/rocky 14:29:05 <efried> #link stable/queens: https://review.openstack.org/#/q/status:open+(project:openstack/os-vif+OR+project:openstack/python-novaclient+OR+project:openstack/nova)+branch:stable/queens 14:29:05 <efried> #link stable/pike: https://review.openstack.org/#/q/status:open+(project:openstack/os-vif+OR+project:openstack/python-novaclient+OR+project:openstack/nova)+branch:stable/pike 14:29:52 <efried> handful of stein-ish things we're going to wait on until after the release. 14:30:02 <efried> any other stable notes? 14:30:16 <efried> #topic Sub/related team Highlights 14:30:29 <efried> Placement - no meeting this week 14:30:50 <efried> API (gmann) 14:30:50 <efried> #link This week updates: http://lists.openstack.org/pipermail/openstack-discuss/2019-April/004538.html 14:30:50 <efried> #link Asking about API office hour continuation on ML - http://lists.openstack.org/pipermail/openstack-discuss/2019-March/004336.html 14:30:57 <efried> gmann: anything to bring up here? 14:31:09 <gmann> that's all. 14:31:30 <efried> #topic Stuck Reviews 14:31:32 <efried> any? 14:33:18 <efried> #topic Forum Planning 14:33:42 <efried> Agenda has links to all the nova-ish sessions I found. If anyone knows of others, please add. 14:34:13 <artom> Agenda being... https://etherpad.openstack.org/p/DEN-train-nova-brainstorming? 14:34:14 <efried> gibi_off volunteered to help with the onboarding session. We're thinking to do a live-ish demo of fixing a bug by posting a recreate patch followed by a fix patch. 14:34:27 <efried> artom: Sorry, the meeting agenda: https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting 14:35:05 <mriedem> efried: i have some charts that could maybe be useful to link into onboarding session content - not to go over, but for further reading etc 14:35:10 <efried> ...and melwitt and I are going to co-chair the proj update session; she's started a google doc 14:35:14 <mriedem> very targeted deep dive stuff 14:35:15 <efried> mriedem: cool, please do share. 14:36:04 <efried> any other forum topics? 14:36:14 <efried> #topic PTG Planning 14:36:14 <efried> #link PTG: Nova etherpad https://etherpad.openstack.org/p/nova-ptg-train 14:36:22 <efried> put yer stuff in the etherpad 14:36:31 <efried> put yer name next ta yer stuff 14:37:23 <efried> If you have conflicts / time constraints, include those somewhere in the appropriate pad(s) 14:37:30 <efried> Around middle of the week before I'll start trying to shape an agenda taking ^ into account. 14:38:09 <efried> mm, maybe I'll actually start that a little earlier to give people time to complain and readjust. 14:38:23 <efried> anything ptg-ish that needs to be discussed here? 14:38:34 <efried> #topic Open discussion 14:38:45 <mriedem> here is that privsep docs bug for nova https://bugs.launchpad.net/nova/+bug/1823192 if someone wants to take a shot at it 14:38:46 <openstack> Launchpad bug 1823192 in OpenStack Compute (nova) "Lack of documentation for rootwrap and privsep in nova docs" [Undecided,New] 14:39:27 <efried> thanks mriedem 14:39:46 <efried> if there's nothing else... 14:40:00 <efried> Thanks all. 14:40:00 <efried> o/ 14:40:00 <efried> #endmeeting