14:00:07 <melwitt> #startmeeting nova
14:00:08 <openstack> Meeting started Thu Jun 28 14:00:07 2018 UTC and is due to finish in 60 minutes.  The chair is melwitt. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:09 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:00:11 <openstack> The meeting name has been set to 'nova'
14:00:19 <melwitt> howdy everybody
14:00:26 <mnaser> hello :)
14:00:35 <takashin> o/
14:00:36 <gibi> o/
14:00:46 <efried> ō/
14:01:17 <edleafe> \o
14:01:22 <melwitt> #topic Release News
14:01:34 <melwitt> #link Rocky release schedule: https://wiki.openstack.org/wiki/Nova/Rocky_Release_Schedule
14:01:47 <melwitt> #link Rocky review runways: https://etherpad.openstack.org/p/nova-runways-rocky
14:01:55 <melwitt> #link runway #1: Report CPU features as traits - https://blueprints.launchpad.net/nova/+spec/report-cpu-features-as-traits [END DATE: 2018-07-06] https://review.openstack.org/560317
14:01:56 <patchbot> patch 560317 - nova - Add method to get cpu traits
14:02:02 <melwitt> #link runway #2: NUMA aware vSwitches - https://blueprints.launchpad.net/nova/+spec/numa-aware-vswitches (stephenfin) [END DATE: 2018-07-06] https://review.openstack.org/#/q/topic:bp/numa-aware-vswitches
14:02:08 <melwitt> #link runway #3: Complex Anti Affinity Policies: https://blueprints.launchpad.net/nova/+spec/complex-anti-affinity-policies (yikun) [END DATE: 2018-07-06] https://review.openstack.org/#/q/topic:bp/complex-anti-affinity-policies
14:02:16 <mriedem> o/
14:02:33 <melwitt> r-3 is July 26, about a month from now
14:03:04 <melwitt> I don't have anything else for release news or runways. anyone have anything they'd like to add?
14:03:59 <mriedem> seems like all 3 are getting decent review
14:04:43 <melwitt> I noticed that too. kudos to everyone pitching in there
14:05:05 * bauzas waves a bit late
14:05:16 <melwitt> #topic Bugs (stuck/critical)
14:05:27 <melwitt> no bugs in the critical bugs link
14:05:33 <melwitt> #link 49 new untriaged bugs (up 2 since the last meeting): https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New
14:05:39 <melwitt> #link 11 untagged untriaged bugs: https://bugs.launchpad.net/nova/+bugs?field.tag=-*&field.status%3Alist=NEW
14:05:57 * tssurya waves late
14:06:02 <melwitt> untriaged bug count still creeping up. thanks for those who have been helping and we need more help
14:06:09 <melwitt> #link bug triage how-to: https://wiki.openstack.org/wiki/Nova/BugTriage#Tags
14:06:15 <melwitt> #help need help with bug triage
14:06:25 <melwitt> Gate status:
14:06:30 <melwitt> #link check queue gate status http://status.openstack.org/elastic-recheck/index.html
14:07:11 <mriedem> i have a new cells v1 skip patch
14:07:17 <mriedem> https://review.openstack.org/#/c/578125/
14:07:18 <patchbot> patch 578125 - nova - Skip ServerShowV247Test.test_update_rebuild_list_s...
14:07:21 <mriedem> another rebuild test for cells v1
14:07:40 <mriedem> i think that's all i've noticed for new gate issues this week
14:07:57 <melwitt> a-ha, cool, thanks. will review that one
14:08:21 <melwitt> 3rd party CI:
14:08:27 <melwitt> #link 3rd party CI status http://ci-watch.tintri.com/project?project=nova&time=7+days
14:09:09 <melwitt> does anyone have anything else for bugs, gate status or third party CI?
14:09:30 <johnthetubaguy> random note, when do we plan to remove cells v1?
14:09:45 <mriedem> after nova-net
14:09:49 <mriedem> which at this point is at least stein
14:10:05 <melwitt> yep
14:10:08 <johnthetubaguy> cool, thanks
14:10:23 <mriedem> johnthetubaguy: http://lists.openstack.org/pipermail/openstack-dev/2018-June/131247.html
14:10:43 <melwitt> thanks for that link
14:10:56 <melwitt> any other questions or comments on bugs, gate status or third party CI?
14:10:58 <johnthetubaguy> ah, thanks
14:11:30 <melwitt> #topic Reminders
14:11:39 <melwitt> #link Rocky Subteam Patches n Bugs https://etherpad.openstack.org/p/rocky-nova-priorities-tracking
14:12:15 <melwitt> that's all I have for reminders. does anyone have any other reminders to highlight?
14:13:15 <melwitt> #topic Stable branch status
14:13:20 <melwitt> #link stable/queens: https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:stable/queens,n,z
14:13:51 <melwitt> have 6 queens backports proposed needing stable reviews
14:13:59 <melwitt> #link stable/pike: https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:stable/pike,n,z
14:14:18 <melwitt> 7 backports for pike
14:14:30 <melwitt> #link stable/ocata: https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:stable/ocata,n,z
14:14:49 <melwitt> 6 backports for ocata
14:15:37 <melwitt> does anyone have anything else for stable branch status?
14:16:34 <melwitt> #topic Subteam Highlights
14:16:59 * efried scrambles
14:17:02 <melwitt> cells v2, we skipped the weekly meeting yesterday
14:17:15 <melwitt> we chatted about the "handling a down cell" spec
14:17:25 <melwitt> added comments on the review
14:17:33 <melwitt> anything else I'm missing dansmith or mriedem?
14:17:41 <mriedem> i had that marker bug fix patch
14:17:47 <mriedem> and dan has his sub url patch
14:17:53 * mriedem finds links
14:18:03 <melwitt> oh right. those are on my review list
14:18:05 <mriedem> #link handling a down cell spec https://review.openstack.org/#/c/557369/
14:18:06 <patchbot> patch 557369 - nova-specs - Handling a down cell
14:18:18 <mriedem> #link https://review.openstack.org/#/c/576161/
14:18:19 <patchbot> patch 576161 - nova - Fix regression when listing build_requests with ma...
14:18:33 <mriedem> #link https://review.openstack.org/#/c/578163/
14:18:33 <patchbot> patch 578163 - nova - Allow templated cell_mapping URLs
14:18:56 * bauzas needs to disappear for family reasons
14:18:58 <melwitt> cool, thanks
14:20:00 <melwitt> scheduler, cdent or efried or jaypipes?
14:20:08 <efried> We discussed https://bugs.launchpad.net/nova/+bug/1777591
14:20:08 <openstack> Launchpad bug 1777591 in OpenStack Compute (nova) "‘limit’ in allocation_candidates where sometimes make force_hosts invalid" [High,In progress] - Assigned to xulei (605423512-j)
14:20:08 <efried> TL;DR: if you're force_host'ing, the limit on GET /a_c can make you come up dry.
14:20:20 <cdent> i fell off the net midway through
14:20:54 <efried> We agreed on the long-term approach, which is to make GET /a_c accept UUID(s) to limit the results.
14:21:00 <mriedem> looks like they've updated the patch to not pass a limit when forcing a host or node https://review.openstack.org/#/c/576693/
14:21:00 <patchbot> patch 576693 - nova - Disable limits if force_hosts or force_nodes is set
14:21:09 <efried> And the backportable fix, which is to disable the limit qparam when forcing... yeah, what mriedem said.
14:21:28 <mriedem> we also talked about functional ways to test this, but it's not easy
14:21:46 <efried> We also agreed that the long-term approach should be punted to Stein, because we've got too much going on in Rocky to try to add it.
14:22:26 <efried> And cdent and I spent some time starting down the rabbit hole of what it really means to say GET /a_c?rp_uuids=in:X,Y,Z
14:23:12 <efried> Hint: it's not going to be as simple as you think.  Handling nested/shared gets hairy, unless we start exposing the concept of "anchor" (or "target", whatever you want to call it).  Which we have a hard enough time explaining internally in code.
14:23:16 <efried> That's it.
14:23:56 <melwitt> lots of good info there, thanks for the update efried
14:23:57 <melwitt> notifications subteam, gibi?
14:24:11 <gibi> melwitt: there was no meeting this week
14:24:20 <gibi> melwitt: here is the current status mail #link http://lists.openstack.org/pipermail/openstack-dev/2018-June/131827.html
14:24:38 <gibi> melwitt: many closed blueprint so happyness :)
14:24:46 <gibi> melwitt: that is all
14:24:52 <melwitt> woot
14:25:03 <melwitt> thanks gibi
14:25:32 <melwitt> anything else for subteam highlights before we move on?
14:26:27 <melwitt> #topic Stuck Reviews
14:26:38 <melwitt> nothing on the agenda. does anyone in the room have anything for stuck reviews?
14:27:28 <melwitt> #topic Open discussion
14:27:44 <melwitt> we have a couple of items on the agenda, specless blueprints looking for approval
14:27:58 <melwitt> #link specless blueprint approval wanted (jangutter): https://blueprints.launchpad.net/nova/+spec/vrouter-os-vif-conversion
14:28:08 <jangutter> Thanks!
14:28:12 <melwitt> similar to this change that merged earlier this cycle: https://review.openstack.org/534371
14:28:12 <patchbot> patch 534371 - nova - remove IVS plug/unplug as they're moved to separat... (MERGED)
14:28:47 <melwitt> from what I understand, this is just converting existing code to leverage os-vif and fairly simple
14:28:54 <mriedem> famous last words
14:29:04 <melwitt> yeah, true
14:29:13 <jangutter> Yep. I can go into some funzies if you want....
14:29:42 <jangutter> But, the worst seems to be that this is going to move pain out of Nova.
14:29:56 <melwitt> I think my main question about it is, are there any migration issues to consider here?
14:30:13 <melwitt> upgrade/migration
14:30:17 <jangutter> There has to be an upgrade step, yeah:
14:30:20 <mriedem> since we're past spec freeze, i think it would be good to say that anything new coming in as exceptions require at least 2 cores signing up to review them, like we used to do in days of old with FFEs
14:30:35 <jangutter> I'm also 100% on board with that.
14:31:00 <melwitt> that's fair, mriedem
14:31:02 <jangutter> So during the upgrade, Contrail will have to pull in a new os-vif plugin.
14:31:30 <jangutter> However: most Contrail installs lock the version of Contrail/Tungsten Fabric 1:1 with OpenStack.
14:31:36 <mriedem> which is the same os-vif plugin repo that provides the contrail_vrouter vif type support yes?
14:31:45 <jangutter> mriedem: correct.
14:31:53 <mriedem> so chances are it's already there
14:32:01 <mriedem> but you'd need the new version that supports the vrouter vif type
14:32:34 <jangutter> mriedem: correct. And since the next version of Contrail is only targeting Queens, there's going to be at least one cycle of overlap.
14:32:41 <mriedem> since the contrail plugin reviews aren't in our gerrit - can we still depend on them using zuulv3?
14:33:24 <mriedem> in other words, i think the nova change shouldn't land until either the contrail plugin series is merged (if we can depends-on with zuul externally) or those are released (even though they aren't released to pypi....)
14:33:47 <jangutter> mriedem: I think that's a fair ask.
14:33:48 <mriedem> btw, kind of sucks those don't get released to pypi
14:34:17 <jangutter> mriedem: I'll definitely pass along the request to TF/Contrail
14:34:49 <mriedem> so who is signing up to review this?
14:34:54 <melwitt> so there's a possibility that it will be "ok" to merge nova changes after contrail plugin changes merge, _without_ a release?
14:35:11 <melwitt> of the plugin?
14:35:12 <mriedem> melwitt: idk, i thought zuulv3 can depends-on external changes b/c of urls
14:35:20 <mriedem> if so, then that's probably good enough
14:35:31 <mriedem> i've never tried it though, like a nova change depending on a github change
14:35:34 <melwitt> what I mean is, do consumers of the plugin pull it from source?
14:35:44 <mriedem> they get it from their big vendor product install
14:35:57 <melwitt> okay
14:35:58 <jangutter> melwitt: consumers of the plugin grabs it from the massive containers published by TF/Contrail
14:36:09 <mriedem> https://github.com/Juniper/contrail-nova-vif-driver
14:36:16 <mriedem> that does get tagged at least
14:36:17 <jangutter> melwitt: they're pretty far downstream from OpenStack
14:36:25 <melwitt> I see. thanks
14:36:47 <mriedem> normally with dependent libraries we'd say you need at least version x.y of something in the reno
14:36:49 <mriedem> is kind of my point
14:37:02 <melwitt> yeah, that is what I'm used to
14:37:24 <mriedem> so we could hold the nova change until the https://github.com/Juniper/contrail-nova-vif-driver patches are merged and tagged
14:37:47 * melwitt nods
14:37:53 <mriedem> hopefully jangutter is actually testing all of this end to end to make sure it works...
14:37:54 <jangutter> (facepalm) yes, that is a very good point. I'll engage with the TF community to ensure that happens.
14:37:55 <mriedem> since there is no CI
14:38:32 <mriedem> anyway, i'll leave a comment on the nova change and we can move on
14:38:33 <jangutter> mriedem: oh yes, but the coverage is very very tiny :-(
14:38:47 <jangutter> Thanks for indulging!
14:38:54 <mriedem> can i (1) install it and (2) create a server with a port that plugs in properly
14:39:21 <jangutter> mriedem: that's tricky, since Contrail historically has problems with OpenStack master.
14:39:44 <jangutter> mriedem: Tungsten Fabric is trying to get better, but it's going to take a bit of time.
14:39:45 <mriedem> so this nova change is going to be backported internally to queens?
14:40:04 <mriedem> anyway, i don't need to know that
14:40:07 <jangutter> mriedem: this change doesn't need backports to queens.
14:40:11 <mriedem> nor do i expect 3rd party CI
14:40:19 <mriedem> jangutter: i know it doesn't upstream :)
14:40:26 <mriedem> anyway, better move on
14:40:32 <melwitt> okay. so jangutter needs to find 2 cores that can commit to reviewing the series, if so, we'll approve the bp
14:40:46 <jangutter> thanks!
14:40:47 <melwitt> stephenfin might be a good person to ask, he has knowledge in os-vif
14:40:57 <melwitt> okay, next one on the agenda:
14:41:08 <melwitt> #link specless blueprint approval wanted (gmann): https://blueprints.launchpad.net/nova/+spec/api-extensions-merge-rocky
14:41:39 <melwitt> continuation of queens work to merge API extensions in controller and schema code
14:41:57 <melwitt> seems okay to me. other opinions?
14:42:27 <mriedem> it was approved for rocky until about a week ago when i deferred it due to no activity,
14:42:30 <mriedem> since then gmann went nuts
14:42:33 <melwitt> it had gotten untargeted from rocky because there were no changes proposed and now it has several changes proposed
14:42:35 <mriedem> so might as well throw it back into the mix
14:42:54 <melwitt> ++ okay, will approve
14:43:19 <melwitt> okay, next, mnaser has a review he wanted to discuss
14:43:32 <mnaser> hi!
14:43:33 <melwitt> #link Add hostId to metadata service https://review.openstack.org/577933
14:43:34 <patchbot> patch 577933 - nova - Add hostId to metadata service
14:43:49 <mnaser> i was just wondering thought about that with maybe more eyes to discuss it
14:44:03 <mriedem> i would be fine with that *except* for the issues with config drive not changing during live migration and the hostId will be stale
14:44:17 <mnaser> well isnt 'normal' metadata published in config drive?
14:44:24 <mriedem> i don't think we have other fields in the config drive that suffer from that - we have the AZ but you can't live migrate out of the AZ unless you force it
14:44:37 <mnaser> like user pbulished metadata
14:44:49 <mriedem> mnaser: as in boot time metadata and then add more later?
14:44:52 <mnaser> yeah
14:45:18 <mriedem> yes but i'd kind of think those are used differently
14:46:16 * gibi needs to drop of
14:46:21 <johnthetubaguy> does sound a bit like setting you up for a facepalm
14:46:22 <mriedem> if you create a guest and require the ability to be able to change metadata later and have it reflected in the guest, you can't rely on config drive - but i'm not sure that we have a way to say 'definitely no config drive, metadata api or bust'
14:46:35 <melwitt> so hostId would create an inconsistency between config drive metadata vs metadata API metadata?
14:46:50 <mriedem> hostId is hashed instance.host + project_id
14:47:07 <mriedem> if the guest is live migrated, the config drive hostId hash would be from the initial host, not the current host
14:47:14 <johnthetubaguy> and resize, etc
14:47:14 <mriedem> so it can't really be trusted in a public cloud that does a lot of live migratoin
14:47:28 <mriedem> i'm honestly not sure in which cases we rebuild the config drive
14:47:32 <mriedem> besides rebuild and unshelve
14:47:33 <johnthetubaguy> I guess its a non use triggered action that changes a property you might not expect
14:47:36 <melwitt> yeah, understood. and this would be the only piece of metadata that would be like that
14:47:57 <mriedem> melwitt: i haven't done a full audit but a quick glance i tihnk so yes
14:48:00 <mriedem> as noted about AZ
14:48:09 <johnthetubaguy> server metadata is updated by the user, so that feels different somehow
14:48:17 <mriedem> right agree
14:48:31 <mriedem> as a user/guest, i won't/shouldn't know if i've been live migrated
14:48:39 <mriedem> i guess you'd know from the compute REST API on the outside though
14:48:53 <mriedem> if that hostId doesn't match what's in the guest, you know the guest was moved
14:49:14 <melwitt> there's been want to rebuild config drive in more cases in the past, so maybe doing something like that could create consistency? I can't remember if it was ever proposed to rebuild it upon live migration
14:49:18 <mriedem> mnaser: so i know this use case is for infra right now, but would you support this at vexxxhost?
14:49:25 <mnaser> yes
14:49:33 <mriedem> oops i made that too sexy
14:49:37 <mriedem> ha
14:49:45 <mnaser> lol
14:49:48 <mnaser> honestly
14:50:00 <mnaser> i would be okay with us maybe putting a warning saying that this *could* change in configdrive
14:50:14 <mnaser> but in general anything config drive is not 'total source of truth'
14:50:24 <mriedem> sure, that's why i'm not against the idea
14:50:26 <mriedem> just caveats
14:50:33 <mnaser> so that warning would apply in anything config drive.. metadata api service is still giving the best up to date info
14:50:33 <mriedem> and we don't document the metadata api responses at all
14:50:47 <mriedem> right, but some clouds don't even run the metadata api
14:51:07 <mriedem> that's why i was trying to figure out if there is a way to say i only want metadata api for my guest or nothing at all
14:51:29 <mriedem> you can say no config drive when creating a server (and that's the default from the api) but i don't think we have any way from the compute service to know that the metadata api isn't running
14:51:40 <johnthetubaguy> you can --config-drive false
14:51:42 <johnthetubaguy> ah, right
14:51:47 <mriedem> yeah it's not the same
14:51:57 <johnthetubaguy> yeah
14:51:59 <melwitt> it would be interesting to know if there are other fields that can be stale for config drive. maybe this is no different than the current state
14:52:13 <johnthetubaguy> server metadata was a good example
14:52:19 <johnthetubaguy> all the network state would be stale
14:52:27 <johnthetubaguy> don't regenerate on vif plug
14:52:33 <mriedem> sure any ports or volumes you attach after the server is created wouldn't be in config drive
14:52:35 <mnaser> yeah if you unplug or plug new vifs
14:52:37 <johnthetubaguy> ... real answer is we need to document that stuff
14:53:06 <mnaser> so while i agree that my stuff can be stale, other stuff can be too, so this might be a two part problem where i just volunteered myself to have some sort of warning saying
14:53:16 <mnaser> config drive data don't get regenerated and therefore might be stale
14:53:26 <mnaser> s/don't/doesn't/
14:53:33 <mriedem> maybe we just need some docs in https://docs.openstack.org/nova/latest/user/config-drive.html about config drive being stale and ways to refresh it (server actions that is)
14:53:48 <mriedem> if we have that, i'd be more comfortable with adding this
14:53:56 <johnthetubaguy> like, if you need something fresh, do a rebuild?
14:53:58 <melwitt> yeah, I was trying to highlight whether this would be the _first_ stale thing or if there are already stale things and it's a known landscape
14:54:20 <mriedem> this might be the first stale thing not initiated by a user is the wrinkle i think
14:54:23 <efried> I thought config drive wasn't even needed/used after initial boot.  Pretty sure in Power we remove the thing and never reattach it.
14:54:24 <melwitt> so it sounds like, this is probably okay provided that docs patches are landed with it to be clear about config drive
14:54:26 * stephenfin wonders why we still care about config drive if the metadata service seems better in every way
14:54:36 * mnaser can add document update to config drive being stale in a patch under mine
14:54:37 <mriedem> stephenfin: see above
14:54:40 <melwitt> mriedem: I see
14:54:44 <mriedem> stephenfin: some deployments don't run the metadata api
14:55:00 <mriedem> mnaser: ok do that an i'm ok with your change
14:55:01 <melwitt> yeah, Oath don't run the metadata API, for one
14:55:02 <mnaser> but i don't have the time/resources to add an extra patch that implements rebuilding config drive somewhow
14:55:13 <mriedem> heh i'm not asking for that
14:55:18 <mriedem> i think SpamapS wants it though
14:55:22 <mriedem> but he knows how to code i think
14:55:24 <mriedem> :)
14:55:42 <mnaser> ok so i will push up a patch under mine as a doc update to let users know that configdrive could have stale data as it is built when the instance is first created
14:55:49 <mnaser> also im pretty sure that only happens if you use shared storage
14:55:55 <mnaser> (i think?)
14:56:21 <mnaser> maybe just on live migrations would be better to summarize it
14:56:28 <melwitt> okay, sounds like we agree this is okay with some doc updates to go with it
14:56:40 <melwitt> we have 4 minutes left
14:56:55 <mnaser> thanks everyone (for comments)
14:57:04 <melwitt> does anyone in the room have anything for open discussion before we wrap up?
14:58:14 <melwitt> okay, let's call it. thanks everyone
14:58:18 <melwitt> #endmeeting