14:00:35 <efried> #startmeeting nova
14:00:36 <openstack> Meeting started Thu Apr  4 14:00:35 2019 UTC and is due to finish in 60 minutes.  The chair is efried. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:37 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:00:39 <openstack> The meeting name has been set to 'nova'
14:00:50 <takashin> o/
14:00:51 <edleafe> \o
14:00:58 <gmann> o/
14:01:10 <mdbooth> o/
14:01:11 <artom> o/
14:01:12 <mriedem> .
14:01:15 <cdent> o/
14:01:17 <bauzas> \o
14:01:20 <tssurya> o/
14:01:20 <artom> Actually, ~o~
14:01:41 <alex_xu> o/
14:01:42 * bauzas doing a double meeting at the same time, feeling a bit schizophrenic
14:02:07 <efried> artom: are you swimming?
14:02:15 <artom> edleafe, waving :D
14:02:21 <artom> Err, efried
14:02:36 <efried> oo, the robot, rad
14:02:52 <edleafe> Heh, I thought it was a new hairstyle
14:03:05 <efried> whoops, /me forgot to save the agenda, done, refresh if you had it open
14:03:07 <efried> #link agenda https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting
14:03:23 <efried> #link last week's minutes Minutes from last week: http://eavesdrop.openstack.org/meetings/nova/2019/nova.2019-03-28-21.02.html
14:03:30 <efried> no fups from that afaict, anyone?
14:03:39 <edleafe> artom: https://goo.gl/images/NXekAP
14:03:43 <stephenfin> o/
14:04:07 <artom> edleafe, *shudder*
14:04:18 <efried> #topic Release News
14:04:18 <efried> #link Stein release schedule: https://wiki.openstack.org/wiki/Nova/Stein_Release_Schedule
14:04:18 <efried> #info Stein RC freeze is TODAY, Thursday, April 4th
14:04:18 <efried> #link Stein RC potential changes tracking: https://etherpad.openstack.org/p/nova-stein-rc-potential
14:04:25 <efried> #link RC2 proposed https://review.openstack.org/649656
14:04:25 <efried> #link Fix non-NIC VFs bombing n-cpu (merging) https://review.openstack.org/#/c/649630/
14:04:25 <efried> #link VGPU docs (merged) https://review.openstack.org/649454
14:04:36 <efried> That last one wouldn't have been enough to prompt an RC2 by itself, but since we're doing one...
14:04:39 <efried> and last but not least...
14:04:44 <efried> #link Keys to the (database) kingdom in versioned notifications https://review.openstack.org/#/c/649775/
14:04:54 <efried> That *just* got a new PS, please review and merge asap.
14:05:06 <mriedem> in the future,
14:05:16 <mriedem> i'd ask people be mindful about what we shove into versioned notification payloads,
14:05:22 <mriedem> just because we can send something doesn't mean we should
14:05:57 <efried> is there a doc or (probably better) code comment we can throw somewhere obvious to that effect?
14:06:22 <mriedem> docs are here https://docs.openstack.org/nova/latest/reference/notifications.html
14:06:29 <mriedem> i don't know if there is a warning for contributors
14:06:42 <mriedem> code comment would be hard since the payloads are in lots of places
14:06:46 <mriedem> it's really on the core team
14:06:57 <efried> k
14:07:21 <efried> any other release topics?
14:07:22 <mriedem> i can follow up with a docs patch to add a warning
14:07:27 <gmann> guidelines of " donot/careful about exposing this "  can be great.
14:07:27 <efried> thanks mriedem
14:07:46 <artom> There's a What should be in the notification payload para
14:07:48 <Kevin_Zheng> o/
14:07:56 <artom> Could be followed with a What should NOT be in the notification payload
14:07:56 <mriedem> artom: yeah that's where i'm going to put a warning
14:08:03 <efried> ++
14:08:16 <efried> #topic Bugs (stuck/critical)
14:08:16 <efried> No Critical bugs
14:08:16 <efried> #link 69 new untriaged bugs (down 1 since the last meeting): https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New
14:08:16 <efried> #link 7 untagged untriaged bugs (down 2 since the last meeting): https://bugs.launchpad.net/nova/+bugs?field.tag=-*&field.status%3Alist=NEW
14:08:27 <efried> Gate status
14:08:27 <efried> #link check queue gate status http://status.openstack.org/elastic-recheck/index.html
14:08:41 <efried> looks like mostly timeout-y things.
14:08:51 <efried> 3rd party CI
14:08:51 <efried> #link 3rd party CI status http://ciwatch.mmedvede.net/project?project=nova&time=7+days
14:09:11 <mriedem> grenade py3 jobs will be still wonky until https://review.openstack.org/#/c/649096/ lands
14:09:38 <efried> a few disturbing hits in unit test jobs - look for the ones where one out of the three pyXX jobs fails. Some kind of odd database race.
14:10:19 <mriedem> we likely need a new e-r query for that
14:10:22 <efried> don't know what it's about, but may try to get an e-r def in place, ...
14:10:23 <efried> yeah
14:10:44 <efried> cause I just learned about e-r this week. I know only enough to be dangerous
14:11:02 * mdbooth loves weird database races if we have a couple of examples
14:11:17 <efried> mdbooth: oh, perfect - there's several examples.
14:11:20 <mdbooth> How much of a problem is it?
14:11:50 <efried> Meh, kills one job in a dozen maybe?
14:11:56 <mdbooth> Moderate
14:12:45 <efried> mdbooth: If you look in http://ciwatch.mmedvede.net/project?project=nova&time=7+days along the openstack-tox-pyXX rows, ignore all green or all red, look for the ones with just one X, click on it. The failures are always in the same small handful of tests, weird DB-ish things.
14:13:52 <efried> thanks for the help
14:14:04 <efried> #action efried to look into adding e-r query for ^
14:14:28 <efried> #topic Reminders
14:14:28 <efried> #link Spec review day proposed next Tuesday, April 9th http://lists.openstack.org/pipermail/openstack-discuss/2019-April/004484.html
14:14:47 <efried> So far a small number of people have responded about ^
14:15:02 <efried> Now that I have a captive audience
14:15:20 <efried> is next Tuesday good for folks to block off for reviewing/discussing/updating specs?
14:15:26 <artom> +1
14:15:27 <cdent> yes
14:15:57 <bauzas> I'm good with it but I was about to write something
14:16:13 <bauzas> having a second review day in one week later is a bit impactful
14:16:19 <bauzas> not sure it's healthy
14:16:33 <bauzas> so, I'm all sold on this review day, not the later
14:16:40 <bauzas> if that's worth mentioning
14:16:59 <mriedem> unhealthy how? if you can't do a 2nd sprint, that's fine.
14:17:12 <mriedem> i think the idea is flush as much as possible before the ptg
14:17:18 <efried> ^
14:17:23 <mriedem> because we're all going to be burned out by saturday
14:17:34 <gmann> +1
14:17:47 <gmann> it can cut down the PTG topics also.
14:18:05 <bauzas> well, ok
14:18:13 <bauzas> that's not that I can't
14:19:00 <bauzas> just that I think I'll feel a bit exhausted after two review days in a week, but fair
14:19:09 <bauzas> but yeah, your point on the PTG is valid
14:19:13 <bauzas> so cool with me
14:19:17 <mriedem> just remember to stretch
14:19:37 <bauzas> :)
14:19:46 <artom> And drink loads of water (or others)
14:20:45 <efried> okay, cool, Ima put up an etherpad where people wanting their specs to get attention that day can list them, make it easier for reviewers to discover rather than picking from the 60-odd open specs out there.
14:20:54 <efried> #efried to fup with review day logistics
14:21:06 <efried> #action efried to fup with review day logistics
14:21:06 <efried> rather
14:21:21 <efried> any other reminders?
14:21:35 <artom> Do we have priorities for train? Should those spec get priority for spec review day?
14:21:48 <efried> mm
14:21:55 <efried> we usually set priorities at/after the ptg
14:22:07 <efried> but that doesn't mean we can't do some pre-prioritizing
14:22:17 <mdbooth> Is it worth having a spec for privsep? It's more of a theme.
14:22:43 <efried> We can have a specless bp to continue the rootwrap-to-privsep conversion for train
14:23:12 <efried> and I would like to have a backlog spec proposed at some point describing how we get from the current privsep to the golden age of security perfection that's been discussed recently in the ML.
14:23:30 <mriedem> specs are point in time for a release, privsep is a long-term commitment, so if we need docs, as suggested in that ML thread, then let's add docs
14:23:45 <dansmith> agree
14:23:52 <efried> #link privsep docs https://review.openstack.org/#/c/649997/
14:24:13 <mriedem> also, related to that,
14:24:30 <efried> having a specless bp could help us keep review focus on it for train so we can get more of that series merged.
14:24:31 <mriedem> do we have any nova-specific docs about configuring nova's rootwrap compute.filters for the privsep-helper?
14:24:36 <mriedem> because that is all black magic for the most part
14:24:36 <efried> It's over a year old.
14:24:51 <mriedem> and i don't think it's documented, deployment tools just had to deal with it back when it was required
14:25:19 <dansmith> mriedem: asI understand it, we only need one rootwrap rule for privsep itself
14:25:22 <mriedem> also related, sighup'ing nova-compute kills nova-compute b/c the privsep-helper child processes are gone
14:25:40 <artom> ... that sounds quite bad...
14:25:45 <efried> we decided that ^ was latent since at least rocky, right?
14:25:48 <mriedem> dansmith: i'm just not sure if there is anything clear in our docs about configuring rootwrap for privsep
14:25:49 <mdbooth> It's also explicitly by design
14:25:54 <dansmith> it's an oslo.service regression
14:26:01 <mriedem> efried: yeah i also recreated it in rocky
14:26:03 <dansmith> mdbooth: what is explicitly by design?
14:26:17 <mdbooth> dansmith: privsep helper doesn't restart if it ends
14:26:29 <dansmith> mdbooth: it's not supposed to end on HUP
14:26:46 <mdbooth> Ah, ok. Is it supposed to reload its context?
14:26:48 <dansmith> mdbooth: and also, it's definitely expected to be able to restart privsep if you're using a root helperto start
14:26:53 <mriedem> this is the only rootwrap mention in our docs https://docs.openstack.org/nova/stein/cli/nova-rootwrap.html - not even in the install guide
14:27:05 <dansmith> mdbooth: the bug is unrelated to privsep, it's in oslo.service killing things it shouldn't
14:27:11 <mdbooth> dansmith: ack
14:27:27 <mriedem> https://docs.openstack.org/nova/stein/search.html?q=privsep
14:27:31 <mriedem> pretty sad
14:27:34 <mriedem> i'll report a docs bug
14:27:46 * mriedem uses trump voice when he says sad
14:28:52 <efried> okay, moving on?
14:29:05 <efried> #topic Stable branch status
14:29:05 <efried> #link stable/stein: https://review.openstack.org/#/q/status:open+(project:openstack/os-vif+OR+project:openstack/python-novaclient+OR+project:openstack/nova)+branch:stable/stein
14:29:05 <efried> #link stable/rocky: https://review.openstack.org/#/q/status:open+(project:openstack/os-vif+OR+project:openstack/python-novaclient+OR+project:openstack/nova)+branch:stable/rocky
14:29:05 <efried> #link stable/queens: https://review.openstack.org/#/q/status:open+(project:openstack/os-vif+OR+project:openstack/python-novaclient+OR+project:openstack/nova)+branch:stable/queens
14:29:05 <efried> #link stable/pike: https://review.openstack.org/#/q/status:open+(project:openstack/os-vif+OR+project:openstack/python-novaclient+OR+project:openstack/nova)+branch:stable/pike
14:29:52 <efried> handful of stein-ish things we're going to wait on until after the release.
14:30:02 <efried> any other stable notes?
14:30:16 <efried> #topic Sub/related team Highlights
14:30:29 <efried> Placement - no meeting this week
14:30:50 <efried> API (gmann)
14:30:50 <efried> #link This week updates: http://lists.openstack.org/pipermail/openstack-discuss/2019-April/004538.html
14:30:50 <efried> #link Asking about API office hour continuation on ML - http://lists.openstack.org/pipermail/openstack-discuss/2019-March/004336.html
14:30:57 <efried> gmann: anything to bring up here?
14:31:09 <gmann> that's all.
14:31:30 <efried> #topic Stuck Reviews
14:31:32 <efried> any?
14:33:18 <efried> #topic Forum Planning
14:33:42 <efried> Agenda has links to all the nova-ish sessions I found. If anyone knows of others, please add.
14:34:13 <artom> Agenda being... https://etherpad.openstack.org/p/DEN-train-nova-brainstorming?
14:34:14 <efried> gibi_off volunteered to help with the onboarding session. We're thinking to do a live-ish demo of fixing a bug by posting a recreate patch followed by a fix patch.
14:34:27 <efried> artom: Sorry, the meeting agenda: https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting
14:35:05 <mriedem> efried: i have some charts that could maybe be useful to link into onboarding session content - not to go over, but for further reading etc
14:35:10 <efried> ...and melwitt and I are going to co-chair the proj update session; she's started a google doc
14:35:14 <mriedem> very targeted deep dive stuff
14:35:15 <efried> mriedem: cool, please do share.
14:36:04 <efried> any other forum topics?
14:36:14 <efried> #topic PTG Planning
14:36:14 <efried> #link PTG: Nova etherpad https://etherpad.openstack.org/p/nova-ptg-train
14:36:22 <efried> put yer stuff in the etherpad
14:36:31 <efried> put yer name next ta yer stuff
14:37:23 <efried> If you have conflicts / time constraints, include those somewhere in the appropriate pad(s)
14:37:30 <efried> Around middle of the week before I'll start trying to shape an agenda taking ^ into account.
14:38:09 <efried> mm, maybe I'll actually start that a little earlier to give people time to complain and readjust.
14:38:23 <efried> anything ptg-ish that needs to be discussed here?
14:38:34 <efried> #topic Open discussion
14:38:45 <mriedem> here is that privsep docs bug for nova https://bugs.launchpad.net/nova/+bug/1823192 if someone wants to take a shot at it
14:38:46 <openstack> Launchpad bug 1823192 in OpenStack Compute (nova) "Lack of documentation for rootwrap and privsep in nova docs" [Undecided,New]
14:39:27 <efried> thanks mriedem
14:39:46 <efried> if there's nothing else...
14:40:00 <efried> Thanks all.
14:40:00 <efried> o/
14:40:00 <efried> #endmeeting