21:00:36 <mriedem> #startmeeting nova
21:00:36 <openstack> Meeting started Thu Feb 18 21:00:36 2016 UTC and is due to finish in 60 minutes.  The chair is mriedem. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:37 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:00:40 <openstack> The meeting name has been set to 'nova'
21:00:44 <rlrossit> o/
21:00:48 <rdopiera> o/
21:00:49 <edleafe> \o
21:00:49 <melwitt> o/
21:00:51 <tonyb> \o
21:00:51 <takashin> o/
21:01:04 <mriedem> #link meeting agenda https://wiki.openstack.org/wiki/Meetings/Nova
21:01:06 <dansmith> o/
21:01:17 <mriedem> #topic release status
21:01:21 <mriedem> Mar 1-3, Mitaka-3 and feature freeze, Final release for client libraries, Soft String Freeze, etc.
21:01:24 <mriedem> #info Mar 1-3, Mitaka-3 and feature freeze, Final release for client libraries, Soft String Freeze, etc.
21:01:31 * bauzas waves
21:01:50 <mriedem> there is a note to look out for release critical bugs, mitaka-rc-potential
21:01:59 <sdague> o/
21:02:05 <mikal> .
21:02:11 * edleafe waves back
21:02:16 <mriedem> did we ever figure out who was doing bug duty this week?
21:02:39 <mriedem> anyway, if you come across what looks to be a release critical bug, tag it with mitaka-rc-potential
21:03:07 <mriedem> - When to pause long running efforts (centralise config, py34, mox, etc)
21:03:16 <mriedem> i thought we said last week those were paused now
21:03:26 <mriedem> "the doc element to the config work seems important, the moving Opts around should probably halt until Newton?"
21:03:40 <tonyb> mriedem: I thought py34/mox was paused but confifg was allowed to keep going
21:03:52 <auggy> o/
21:03:54 <mriedem> i have no idea how much is left
21:03:58 <mriedem> on the config stuff
21:03:59 <tonyb> mriedem: yeah that thing
21:04:06 <mikal> I recon we should allow extended help strings
21:04:10 <mikal> But stop the large refactors
21:04:17 <mikal> THe help stgring changes are very low risk with high benefit
21:04:22 <tonyb> I think markus_z is out this week so it's hard to know
21:04:31 <mriedem> mikal: yeah at least until string freeze anyway
21:04:37 <mikal> mriedem: agreed
21:04:46 <mriedem> #topic reminders
21:04:47 <edleafe> mriedem: there is a lot left
21:04:51 <mikal> I could go through and -2 the refector reviews if we thought that was reasonable
21:04:57 <mriedem> #link priority review etherpad https://etherpad.openstack.org/p/mitaka-nova-priorities-tracking
21:05:06 <mriedem> mikal: sure, that is probably helpful
21:05:17 <mikal> mriedem: ok, I'll try and get that done today
21:05:22 <mriedem> ty
21:05:25 <mriedem> #topic bugs
21:05:30 <mriedem> gate status http://status.openstack.org/elastic-recheck/index.html
21:05:39 <mriedem> sdague: anything you want to add on this?
21:05:50 <mriedem> i know grenade is hitting messaging timeouts since oslo.messaging 4.1.0
21:05:53 <mriedem> i don't think anyone is working on that
21:06:13 <sdague> mriedem: it was brought up to the oslo folks, and feels like it all petered out
21:06:14 <mriedem> http://status.openstack.org/elastic-recheck/index.html#1545002
21:06:22 <sdague> I have not looped back around on that
21:06:25 <bauzas> the pep8 issue with File Not Found is known, right?
21:06:35 <mriedem> bauzas: dan brought it up in infra this morning
21:06:39 <dansmith> yeah
21:06:39 <mriedem> so i assume it's a known thing
21:06:44 <dansmith> sounds amazingly awesome
21:06:47 <bauzas> okay, nice to know
21:06:54 <mriedem> i hope AFS didn't do it :)
21:07:03 <dansmith> AFS is perfect
21:07:31 <mriedem> #help needed on messaging timeout bug http://bugs.launchpad.net/bugs/1545002
21:07:31 <openstack> Launchpad bug 1545002 in oslo.messaging "Grenade Failure - MessagingTimeout on floating IP remove" [High,Confirmed]
21:07:55 <mriedem> 3rd party CI status - i don't have any news here
21:08:07 <mriedem> critical bugs - anyone have any?
21:08:48 <sdague> mriedem: well this metadata one isn't good
21:08:56 <sdague> but I don't know if it's critical yet
21:09:04 <mriedem> yeah
21:09:20 <mriedem> i'm leaning on that ol' sdague magic to sort that one out :)
21:09:31 <sdague> oof
21:09:43 <mriedem> more volunteers for 1 week of bug skimming duty? https://wiki.openstack.org/wiki/Nova/BugTriage#Weekly_bug_skimming_duty
21:10:01 <mriedem> i know sdague has been hammering the API bug triage, which is appreciated
21:10:10 <mriedem> hopefully more people get in the bug boat after FF
21:10:13 <bauzas> so I'm about to see if some operators could help with bug triaging
21:10:34 <mriedem> ok
21:10:47 <sdague> only 29 bugs in new right now
21:10:48 <bauzas> but that's a long sotry
21:10:52 <bauzas> story
21:10:53 <tonyb> Just putting this out there, a good triaged list of bugs might make the bugsmash more helpful ....
21:11:00 <sdague> that's totally within sight of 0
21:11:17 <auggy> whoohoo that's s a huge a huge improvement!!
21:11:43 <mriedem> stable branch
21:11:45 <dansmith> yuuuge
21:11:49 <mriedem> only thing here is,
21:12:01 <mriedem> we should review stable/liberty stuff for a 12.0.2 release push around m-3
21:12:10 <mriedem> #link open stable/liberty reviews https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:stable/liberty,n,z
21:12:26 <edleafe> tonyb: thanks for volunteering!
21:12:34 <mriedem> or if anyone knows of bugs fixed on master that we'd like to get into stable/liberty, get the backports up soonish
21:12:37 <mikal> edleafe: hey! He's under other buses!
21:12:54 <edleafe> mikal: never enough buses
21:13:08 <mriedem> i've had a todo to write a tool to find liberty-backport-potential nova bugs that are fixed to try and find possible backports,
21:13:11 <mriedem> but time isn't on my side
21:13:22 <mriedem> ^ is open for anyone that wants to write a tool like that
21:13:44 <mriedem> anything else on bugs?
21:14:13 <mriedem> #topic stuck reviews
21:14:18 <mriedem> there was nothing on the agenda
21:14:24 <mriedem> does anyone have anything to discuss here?
21:14:24 <mikal> So, can I raise something here?
21:14:28 <mriedem> uh oh
21:14:29 <mikal> gus: you around?
21:14:35 <gus> mikal: yep.
21:14:36 <mikal> They're not super stuck, so this might be cheating
21:14:41 <tonyb> mikal: that's totaly in open discussion
21:14:44 <mikal> But gus is getting worried about getting privsep done in time
21:14:50 <tonyb> it's even on the agenda
21:14:53 <mriedem> that's in open discussoin
21:14:56 <mikal> Oh, ok. Sure.
21:14:58 <mriedem> belay
21:15:07 <mriedem> #open discussion
21:15:11 <mriedem> :)oops
21:15:12 <tonyb> There are a couple of stable backports that are kinda stuck
21:15:12 <mikal> LOL
21:15:19 <mriedem> #topic open discussion
21:15:28 <mriedem> mine first
21:15:29 <mriedem> (mriedem): get-me-a-network; trouble in paradise
21:15:43 <mriedem> so we have the ML thread on how to microversion the get-me-a-network stuff
21:15:50 <mriedem> #link ML thread on get-me-a-network microversion http://lists.openstack.org/pipermail/openstack-dev/2016-February/086437.html
21:16:05 <mriedem> main question being, do we change default behavior (opt into not having a nic)?
21:16:27 <mriedem> personally i think we should, since the default behavior to allow an instance w/o a network seems weird
21:16:41 <mriedem> and i doubt that is used in practice very often
21:16:56 <mriedem> i know alaski is against that idea,
21:17:02 <mriedem> so i'm looking for input from others that didn't speak up on the ML
21:17:33 <mriedem> dansmith: ^?
21:17:37 <dansmith> well,
21:17:47 <dansmith> I dunno. changing defaults is kinda bad
21:17:53 <dansmith> even if we're suuure the default is better
21:18:10 <dansmith> if someone has a workflow that attaches with no nic and then always does an attach later, we'll break them right?
21:18:13 <sdague> but it's opt in
21:18:26 <mriedem> dansmith: they'd have to opt in with the microversion
21:18:26 <sdague> you had to ask for things after that microversion
21:18:33 <dansmith> yeah, I know,
21:18:55 <dansmith> but in reality, clients aren't going to check for an empty request and make the behavior be the same because they've opted into the new version
21:19:06 <sdague> changing defaults to be better is definitely a thing we should do, we have a signaling mechanism around it
21:19:24 <dansmith> sdague: well, since there was no IMHO on there I guess I can't argue with "definitely" eh?
21:19:25 <cdent> sdague++
21:19:37 <cdent> if we can't make defaults better then we're really in a bad way
21:19:41 <edleafe> microversions don't solve every problem, but at least give you a way to know what's changed
21:19:48 <cdent> because it presumes we got it right the first time, always
21:19:50 <mriedem> it would have been nice if any user/operator in the world would have spoken in that thread saying they have that use case
21:19:50 <cdent> which is never true
21:20:15 <dansmith> mriedem: I can't point you to a specific example person,
21:20:17 <sdague> mriedem: it was a -dev thread right?
21:20:31 <mriedem> sdague: yeah, could go to ops too, but i don't know that ops care, it's really a user use case
21:20:43 <dansmith> but the use case could be starting a spare instance and adding a nic to it when you failover
21:20:55 <dansmith> either way, I bet nobody is going to speak up about it
21:21:21 <dansmith> just to be clear,
21:21:21 <sdague> yeh, we don't really have the right coms channels to find the people that would speak up about it, unfortunately
21:21:45 <dansmith> the microversion would be "you get a nic if you didn't ask" instead of "you get no nics if you didn't ask" right?
21:21:54 <mriedem> rihgt
21:21:54 <sdague> dansmith: yes
21:22:00 <dansmith> because making it "you get an error if you try to boot an instance with no nic" would be a lot more discoverable, IMHO
21:22:27 <dansmith> because the user of the client that opted in would see that
21:22:39 <dansmith> instead of wondering why the hell they're getting different behavior all the sudden
21:23:11 <sdague> so they did have to move their API pin to get different behavior
21:23:19 <dansmith> who did?
21:23:23 <dansmith> the client library, right?
21:23:39 <sdague> or application
21:23:47 <dansmith> if they're consuming directly, sure
21:23:59 <sdague> well, even with our library they have to specify
21:24:16 <dansmith> anyway, I feel like we were just arguing about not changing something like this in a microversion just for the fun of it, because it's surprising,
21:24:20 <dansmith> yet we like this change so we want to do it
21:24:24 <melwitt> if you have to opt in to not get a nic, then wouldn't the behavior be an error if you didn't ask and a net isn't available?
21:24:27 <dansmith> so, my opinion is don't change it
21:25:00 <dansmith> melwitt: you have to opt in to the new behavior, not the old one
21:25:11 <melwitt> oh
21:25:15 <mriedem> well that's what's proposed
21:25:17 <dansmith> melwitt: so you're (supposedly) opting in to having it be magic
21:25:19 <sdague> it's only surprising to people that are used to getting punched in the face
21:25:33 <sdague> and then we stop doing that
21:25:38 <sdague> for the majority of them
21:25:45 <dansmith> sdague: I feel like arguments like that are not very constructive
21:25:52 <dansmith> so anyway, that's my opinion, let's move on
21:26:02 <mriedem> ok, so, just to finalize,
21:26:11 <mriedem> the reason i brought this up is i want to start drafting a spec,
21:26:18 <mriedem> b/c the neutron meetup is local for me next week
21:26:23 <sdague> ok, I feel like the entire "get me a network" thing was driving by users that were really really frustrated that was not the default
21:26:25 <mriedem> i think i'm going to just doc this all as alternatives in the spec,
21:26:29 <mriedem> and we can hash it out in review
21:26:40 <mriedem> sdague: agree
21:26:50 <mriedem> moving on
21:26:54 <mriedem> gus: mikal: go!
21:26:59 <gus> heh.
21:27:26 <gus> So: privsep/os-brick status:  The code exists to do a simple s/rootwrap/privsep/g in os-brick.
21:27:42 <gus> It depends on a bunch (6?) of small changes across other projects (nova, devstack, cinder)
21:28:02 <mriedem> #link main os-brick privsep change https://review.openstack.org/#/c/277224/
21:28:05 <gus> All the isolated tests I've thrown at it work, but the whole thing fails in tempest integration tests atm ;)
21:28:18 <gus> I'm still debugging that latter, it's almost certainly something to do with eventlet.
21:28:20 <gus> (it hangs)
21:28:56 <gus> My concern is that at this point there are still a bunch of small changes ahead of us, mostly to do with juggling global-requirements once there is an appropriate os-brick/privsep release.
21:29:09 <gus> I know we get thingy about changing requirements close to release deadlines.
21:29:19 <tonyb> gus: we have 10 days still
21:29:32 <gus> tonyb: right.
21:30:01 <gus> The last round of minor changes to other projects took 2 weeks to get nova +2s, and haven't seen any +2s from cinder or devstack yet.
21:30:19 <gus> (thanks to those who approved the nova ones over night)
21:30:23 <tonyb> sdague: can you hepl with the devstack side?
21:30:42 <tonyb> sdague: most of the changes are in mitaka-nova-priorities-tracking
21:30:49 <sdague> tonyb: yeh, though I didn't see anything working yet in full stack
21:30:57 <sdague> which was my concern
21:31:06 <gus> so .. we can still make it, but I feel like it's getting tight - particularly wrt juggling the cross-project review latency.
21:31:13 <gus> sdague: yep, agreed.
21:31:47 <gus> the depends-on changes are all safe (afaics) to merge into the other projects regardless of whether we go ahead with the rest of os-brick/privsep or not.
21:31:55 <scottda> os-brick release just went out
21:32:10 <gus> they're things like an additional rootwrap filter, that at worst would be simply unused.
21:33:22 <gus> I don't have any specific request - just a heads-up that when I whack this final eventlet interaction bug, there's going to be half-a-dozen-or-so changes that are going to need some fast review turnaround.
21:34:08 <mriedem> so,
21:34:22 <mriedem> summaries in the ML on big hairy things like this have been common lately
21:34:23 <mriedem> all the rage
21:34:40 <mriedem> b/c i see a bunch of changes on different topic branches, so tracking it all in gerrit is confusing me
21:34:45 <mriedem> some are tracked against a bp and some aren't
21:34:58 <mriedem> so as a reviewer i'm not entirely sure where to start (besides the main os-brick change with the 5 depends-on)
21:34:58 <sdague> right, it's especially hard when the Depends-On is just a wall
21:35:19 <sdague> it would be useful to at least annotate them to which repos they are on to understand
21:35:22 <scottda> gus: I'm not positive, but I think getting privsep changes into  os-brick is too late for Mitaks
21:35:26 <melwitt> is there a section for them in the priorities etherpad?
21:35:41 <melwitt> that gus can just link all of them
21:35:46 <tonyb> melwitt: yeah they're under os-vif
21:35:57 <tonyb> melwitt: somewhere near 144
21:35:59 <sdague> scottda: what?
21:36:06 <mriedem> tonyb: melwitt: yeah but it's not very detailed
21:36:11 <mriedem> it's just some links, i can figure those out via gerrit
21:36:17 <sdague> scottda: because, honestly, this is a release blocker
21:36:18 <scottda> os-brick was just released today.
21:36:25 <gus> Can I mix comments in with the Depends-On change description "meta headers" ?
21:36:31 <sdague> we're in an upgrade crippled state
21:36:33 <scottda> I'm trying to get hemnafk to verify , but he is AFK
21:36:34 * gus isn't sure what the parsing requirements of that are.
21:36:34 <mikal> gus: alternate lines?
21:36:47 <tonyb> mriedem: Okay tell me what else you need in there and I'll add it
21:36:56 <mikal> # The following depends-on is against os-brick
21:37:00 <tonyb> mriedem: I just followed the "template"
21:37:00 <mikal> Depends-On 12345
21:37:15 <sdague> yeh, each Depends-On must be the only thing on it's line
21:37:16 <mriedem> tonyb: well, like what jaypipes and alaski and others have done in the ML recently on where big things are at
21:37:19 <sdague> but you can have other stuff in there
21:37:24 <gus> mikal: yep, that would be useful for me too - I wasn't sure how strict that block needed to be.
21:37:29 <sdague> https://review.openstack.org/#/c/282031/
21:37:53 <gus> I have some mention in the nova priorities etherpad - but of course that doens't affect devstack + cinder (+ os-brick).
21:38:12 <sdague> so, I just want to circle back on the os-brick thing
21:38:16 <mriedem> what we need is like a management type that knows how to put together fancy charts....
21:38:20 <mriedem> where could we find one of those...
21:38:39 <sdague> because this can't be too late for os-brick
21:38:40 <mikal> So, I kind of disagree
21:38:50 <mikal> What we need is the cross project element to work better
21:38:55 * jaypipes hides from mriedem
21:39:09 <mikal> Do we need to try and get everyone into a dedicated meeting early next week?
21:39:09 <mriedem> library freeze is i think the same time as FF
21:39:14 <scottda> hemnafk: keeps reporting that the privsep changes cause Nova to hang...that's the latest report from him I've seen...
21:39:49 <gus> scottda: yep, that's probably a repeat of my latest status update on the main change.
21:39:52 <mriedem> Feb 22-26R-6Final release for non-client libraries
21:39:54 <scottda> I reckon another release of os-brick could still go out if this is fixed. I just noticed today an email that os-brick was released.
21:40:12 <mriedem> yeah 1.0.0 went out with bug fixes
21:40:20 <mriedem> i think we have until next friday for another os-brick release according to the schedule
21:40:42 <sdague> mikal: the answer is yes, who is going to be point person on it?
21:40:45 <mriedem> mikal: so to answer your question, yes i think some kind of project management would be helpful
21:41:00 <mikal> So, the only problem is that next week wastes a few days
21:41:08 <mikal> We could do it tomorrow, but I'm afk all day
21:41:25 <mikal> I don't want to sit idle over the weekend if its avoidable
21:41:55 <mriedem> well i guess it depends on what works for gus
21:41:57 <sdague> mikal: well, how about start with pulling together where we are, and what's blocking moving forward
21:42:10 <mikal> Yep
21:42:15 <mriedem> yeah some high level context would be helpful before a hangout or something
21:42:15 <sdague> even if it's not a meeting, a consolidated view on all of this stuff would be really useful
21:42:17 <mikal> gus and I can work on an email summary to -dev
21:42:24 <mriedem> cool
21:42:25 <gus> sg
21:42:29 <mikal> And make is super clear what we think brick and devstack need to do
21:42:37 <mikal> We'll get that out in the next couple of hours?
21:42:41 <mriedem> awesome
21:42:51 <sdague> yep, and to be clear, I'll approve the devstack changes as soon as I've got a test run showing me it's working
21:43:01 <sdague> I just couldn't find that in the stack
21:43:34 <gus> sdague: there's things like https://review.openstack.org/#/c/277696/ that just sets a new nova.conf entry.
21:44:01 <sdague> gus: right, but I actually want to see something using that to do something :)
21:44:12 <gus> sure, it just means it remains on the critical path then.
21:44:30 <gus> (sdague: but I understand your reluctance)
21:45:18 <sdague> right, but then we know it works vs. land, oh broken, land, oh broken.
21:45:33 <gus> yep.
21:45:55 <mriedem> anything else on this?
21:46:01 <gus> not from me.
21:46:11 <mriedem> anything else from anyone for open discussion?
21:46:16 <scottda> privsep is out for mitaka in os-brick, according to smcginnis
21:46:34 <mriedem> scottda: umm https://review.openstack.org/#/c/277224/
21:46:36 <gus> ^ huh, well then at least I get my weekend back.
21:46:50 <sdague> scottda: um, that's really not possible
21:47:00 <scottda> I'm just the middle man here
21:47:21 <mriedem> you mean out as in not happening
21:47:31 <mriedem> vs out as released :)
21:47:35 <mriedem> important distinction
21:47:39 <smcginnis> mriedem: Not happening.
21:47:42 <mriedem> yeah
21:47:44 <mriedem> i read the cinder channel
21:47:46 <patrickeast> it was discussed in the last cinder meeting briefly, see ~16:06 http://eavesdrop.openstack.org/meetings/cinder/2016/cinder.2016-02-17-16.00.log.html
21:47:49 <smcginnis> mriedem: There are still issues that need to be resolved and tested.
21:47:57 <mriedem> smcginnis: yeah, https://review.openstack.org/#/c/277224/
21:48:04 <mriedem> that's what we were talking about
21:48:19 <edleafe> mriedem: did a quick count on the config options status. 504 total, 331 either merged or in the pipeline, 173 untouched
21:48:37 <sdague> smcginnis: this is a monsterous upgrade problem
21:48:52 <sdague> that we only didn't revert os-brick in liberty because it was going to be fixed in mitaka
21:48:54 <smcginnis> sdague: Agreed. Not happy we weren't able to get it working in time.
21:49:27 <smcginnis> sdague: Well, TBF, there's a lot of goodness in os-brick that far surpasses the pain of copying rootwrap files.
21:49:41 <sdague> smcginnis: it completely breaks our upgrade model
21:49:56 <sdague> I don't think I can bang my shoe on the table hard enough about that
21:50:15 <smcginnis> We are tied, yes. But did we add anything to rootwrap filters in this cycle?
21:50:40 <mriedem> i think there was maybe one thing
21:50:44 <mriedem> i can check after the meeting
21:50:48 <mriedem> i know we held things out
21:50:55 <mikal> So, where to from here?
21:50:59 <mikal> Do we still want that summary email?
21:51:05 <mikal> Or are we dead in the water until newton?
21:51:08 <sdague> ok, so os-brick will never accept another rootwrap change until this is sorted, right?
21:51:34 <gus> "Add os-brick's scsi_id command to rootwrap" - Sep 9
21:51:42 <smcginnis> I don't work in absolutes, but I can strongly discourage it. :)
21:51:51 <sdague> smcginnis: no, this has to be absolutes
21:51:53 <mriedem> mriedem@ubuntu:~/git/os-brick$ git log --oneline --no-merges 0.6.0.. -- etc/os-brick/rootwrap.d/os-brick.filters
21:51:53 <mriedem> d3b9696 os-brick add cinder local_dev lvm code
21:51:53 <mriedem> ebce3c3 DRBD connector class
21:52:09 <mriedem> we kept ebce3c3 out of nova
21:52:14 <mriedem> b/c of the filter add
21:52:25 <mriedem> d3b9696 is not being used in nova i dont think
21:52:31 <mriedem> it was a late add, to prep for newton
21:52:43 <smcginnis> mriedem: Right, not yet.
21:53:01 <mriedem> 0.6.0 is what's in stable/liberty u-c, that's there i get the version from
21:53:53 <mriedem> so i guess let's take this to -nova or -cinder now
21:53:56 <mriedem> and wrap up the nova meeting
21:54:12 <mriedem> i imagine there are some bits of brain and skull on the floor somewhere
21:54:18 <mriedem> anyone have anything else?
21:54:33 <mriedem> ok, ending, thanks everyone
21:54:35 <mriedem> #endmeeting