21:00:36 <mriedem> #startmeeting nova 21:00:36 <openstack> Meeting started Thu Feb 18 21:00:36 2016 UTC and is due to finish in 60 minutes. The chair is mriedem. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:00:37 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:00:40 <openstack> The meeting name has been set to 'nova' 21:00:44 <rlrossit> o/ 21:00:48 <rdopiera> o/ 21:00:49 <edleafe> \o 21:00:49 <melwitt> o/ 21:00:51 <tonyb> \o 21:00:51 <takashin> o/ 21:01:04 <mriedem> #link meeting agenda https://wiki.openstack.org/wiki/Meetings/Nova 21:01:06 <dansmith> o/ 21:01:17 <mriedem> #topic release status 21:01:21 <mriedem> Mar 1-3, Mitaka-3 and feature freeze, Final release for client libraries, Soft String Freeze, etc. 21:01:24 <mriedem> #info Mar 1-3, Mitaka-3 and feature freeze, Final release for client libraries, Soft String Freeze, etc. 21:01:31 * bauzas waves 21:01:50 <mriedem> there is a note to look out for release critical bugs, mitaka-rc-potential 21:01:59 <sdague> o/ 21:02:05 <mikal> . 21:02:11 * edleafe waves back 21:02:16 <mriedem> did we ever figure out who was doing bug duty this week? 21:02:39 <mriedem> anyway, if you come across what looks to be a release critical bug, tag it with mitaka-rc-potential 21:03:07 <mriedem> - When to pause long running efforts (centralise config, py34, mox, etc) 21:03:16 <mriedem> i thought we said last week those were paused now 21:03:26 <mriedem> "the doc element to the config work seems important, the moving Opts around should probably halt until Newton?" 21:03:40 <tonyb> mriedem: I thought py34/mox was paused but confifg was allowed to keep going 21:03:52 <auggy> o/ 21:03:54 <mriedem> i have no idea how much is left 21:03:58 <mriedem> on the config stuff 21:03:59 <tonyb> mriedem: yeah that thing 21:04:06 <mikal> I recon we should allow extended help strings 21:04:10 <mikal> But stop the large refactors 21:04:17 <mikal> THe help stgring changes are very low risk with high benefit 21:04:22 <tonyb> I think markus_z is out this week so it's hard to know 21:04:31 <mriedem> mikal: yeah at least until string freeze anyway 21:04:37 <mikal> mriedem: agreed 21:04:46 <mriedem> #topic reminders 21:04:47 <edleafe> mriedem: there is a lot left 21:04:51 <mikal> I could go through and -2 the refector reviews if we thought that was reasonable 21:04:57 <mriedem> #link priority review etherpad https://etherpad.openstack.org/p/mitaka-nova-priorities-tracking 21:05:06 <mriedem> mikal: sure, that is probably helpful 21:05:17 <mikal> mriedem: ok, I'll try and get that done today 21:05:22 <mriedem> ty 21:05:25 <mriedem> #topic bugs 21:05:30 <mriedem> gate status http://status.openstack.org/elastic-recheck/index.html 21:05:39 <mriedem> sdague: anything you want to add on this? 21:05:50 <mriedem> i know grenade is hitting messaging timeouts since oslo.messaging 4.1.0 21:05:53 <mriedem> i don't think anyone is working on that 21:06:13 <sdague> mriedem: it was brought up to the oslo folks, and feels like it all petered out 21:06:14 <mriedem> http://status.openstack.org/elastic-recheck/index.html#1545002 21:06:22 <sdague> I have not looped back around on that 21:06:25 <bauzas> the pep8 issue with File Not Found is known, right? 21:06:35 <mriedem> bauzas: dan brought it up in infra this morning 21:06:39 <dansmith> yeah 21:06:39 <mriedem> so i assume it's a known thing 21:06:44 <dansmith> sounds amazingly awesome 21:06:47 <bauzas> okay, nice to know 21:06:54 <mriedem> i hope AFS didn't do it :) 21:07:03 <dansmith> AFS is perfect 21:07:31 <mriedem> #help needed on messaging timeout bug http://bugs.launchpad.net/bugs/1545002 21:07:31 <openstack> Launchpad bug 1545002 in oslo.messaging "Grenade Failure - MessagingTimeout on floating IP remove" [High,Confirmed] 21:07:55 <mriedem> 3rd party CI status - i don't have any news here 21:08:07 <mriedem> critical bugs - anyone have any? 21:08:48 <sdague> mriedem: well this metadata one isn't good 21:08:56 <sdague> but I don't know if it's critical yet 21:09:04 <mriedem> yeah 21:09:20 <mriedem> i'm leaning on that ol' sdague magic to sort that one out :) 21:09:31 <sdague> oof 21:09:43 <mriedem> more volunteers for 1 week of bug skimming duty? https://wiki.openstack.org/wiki/Nova/BugTriage#Weekly_bug_skimming_duty 21:10:01 <mriedem> i know sdague has been hammering the API bug triage, which is appreciated 21:10:10 <mriedem> hopefully more people get in the bug boat after FF 21:10:13 <bauzas> so I'm about to see if some operators could help with bug triaging 21:10:34 <mriedem> ok 21:10:47 <sdague> only 29 bugs in new right now 21:10:48 <bauzas> but that's a long sotry 21:10:52 <bauzas> story 21:10:53 <tonyb> Just putting this out there, a good triaged list of bugs might make the bugsmash more helpful .... 21:11:00 <sdague> that's totally within sight of 0 21:11:17 <auggy> whoohoo that's s a huge a huge improvement!! 21:11:43 <mriedem> stable branch 21:11:45 <dansmith> yuuuge 21:11:49 <mriedem> only thing here is, 21:12:01 <mriedem> we should review stable/liberty stuff for a 12.0.2 release push around m-3 21:12:10 <mriedem> #link open stable/liberty reviews https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:stable/liberty,n,z 21:12:26 <edleafe> tonyb: thanks for volunteering! 21:12:34 <mriedem> or if anyone knows of bugs fixed on master that we'd like to get into stable/liberty, get the backports up soonish 21:12:37 <mikal> edleafe: hey! He's under other buses! 21:12:54 <edleafe> mikal: never enough buses 21:13:08 <mriedem> i've had a todo to write a tool to find liberty-backport-potential nova bugs that are fixed to try and find possible backports, 21:13:11 <mriedem> but time isn't on my side 21:13:22 <mriedem> ^ is open for anyone that wants to write a tool like that 21:13:44 <mriedem> anything else on bugs? 21:14:13 <mriedem> #topic stuck reviews 21:14:18 <mriedem> there was nothing on the agenda 21:14:24 <mriedem> does anyone have anything to discuss here? 21:14:24 <mikal> So, can I raise something here? 21:14:28 <mriedem> uh oh 21:14:29 <mikal> gus: you around? 21:14:35 <gus> mikal: yep. 21:14:36 <mikal> They're not super stuck, so this might be cheating 21:14:41 <tonyb> mikal: that's totaly in open discussion 21:14:44 <mikal> But gus is getting worried about getting privsep done in time 21:14:50 <tonyb> it's even on the agenda 21:14:53 <mriedem> that's in open discussoin 21:14:56 <mikal> Oh, ok. Sure. 21:14:58 <mriedem> belay 21:15:07 <mriedem> #open discussion 21:15:11 <mriedem> :)oops 21:15:12 <tonyb> There are a couple of stable backports that are kinda stuck 21:15:12 <mikal> LOL 21:15:19 <mriedem> #topic open discussion 21:15:28 <mriedem> mine first 21:15:29 <mriedem> (mriedem): get-me-a-network; trouble in paradise 21:15:43 <mriedem> so we have the ML thread on how to microversion the get-me-a-network stuff 21:15:50 <mriedem> #link ML thread on get-me-a-network microversion http://lists.openstack.org/pipermail/openstack-dev/2016-February/086437.html 21:16:05 <mriedem> main question being, do we change default behavior (opt into not having a nic)? 21:16:27 <mriedem> personally i think we should, since the default behavior to allow an instance w/o a network seems weird 21:16:41 <mriedem> and i doubt that is used in practice very often 21:16:56 <mriedem> i know alaski is against that idea, 21:17:02 <mriedem> so i'm looking for input from others that didn't speak up on the ML 21:17:33 <mriedem> dansmith: ^? 21:17:37 <dansmith> well, 21:17:47 <dansmith> I dunno. changing defaults is kinda bad 21:17:53 <dansmith> even if we're suuure the default is better 21:18:10 <dansmith> if someone has a workflow that attaches with no nic and then always does an attach later, we'll break them right? 21:18:13 <sdague> but it's opt in 21:18:26 <mriedem> dansmith: they'd have to opt in with the microversion 21:18:26 <sdague> you had to ask for things after that microversion 21:18:33 <dansmith> yeah, I know, 21:18:55 <dansmith> but in reality, clients aren't going to check for an empty request and make the behavior be the same because they've opted into the new version 21:19:06 <sdague> changing defaults to be better is definitely a thing we should do, we have a signaling mechanism around it 21:19:24 <dansmith> sdague: well, since there was no IMHO on there I guess I can't argue with "definitely" eh? 21:19:25 <cdent> sdague++ 21:19:37 <cdent> if we can't make defaults better then we're really in a bad way 21:19:41 <edleafe> microversions don't solve every problem, but at least give you a way to know what's changed 21:19:48 <cdent> because it presumes we got it right the first time, always 21:19:50 <mriedem> it would have been nice if any user/operator in the world would have spoken in that thread saying they have that use case 21:19:50 <cdent> which is never true 21:20:15 <dansmith> mriedem: I can't point you to a specific example person, 21:20:17 <sdague> mriedem: it was a -dev thread right? 21:20:31 <mriedem> sdague: yeah, could go to ops too, but i don't know that ops care, it's really a user use case 21:20:43 <dansmith> but the use case could be starting a spare instance and adding a nic to it when you failover 21:20:55 <dansmith> either way, I bet nobody is going to speak up about it 21:21:21 <dansmith> just to be clear, 21:21:21 <sdague> yeh, we don't really have the right coms channels to find the people that would speak up about it, unfortunately 21:21:45 <dansmith> the microversion would be "you get a nic if you didn't ask" instead of "you get no nics if you didn't ask" right? 21:21:54 <mriedem> rihgt 21:21:54 <sdague> dansmith: yes 21:22:00 <dansmith> because making it "you get an error if you try to boot an instance with no nic" would be a lot more discoverable, IMHO 21:22:27 <dansmith> because the user of the client that opted in would see that 21:22:39 <dansmith> instead of wondering why the hell they're getting different behavior all the sudden 21:23:11 <sdague> so they did have to move their API pin to get different behavior 21:23:19 <dansmith> who did? 21:23:23 <dansmith> the client library, right? 21:23:39 <sdague> or application 21:23:47 <dansmith> if they're consuming directly, sure 21:23:59 <sdague> well, even with our library they have to specify 21:24:16 <dansmith> anyway, I feel like we were just arguing about not changing something like this in a microversion just for the fun of it, because it's surprising, 21:24:20 <dansmith> yet we like this change so we want to do it 21:24:24 <melwitt> if you have to opt in to not get a nic, then wouldn't the behavior be an error if you didn't ask and a net isn't available? 21:24:27 <dansmith> so, my opinion is don't change it 21:25:00 <dansmith> melwitt: you have to opt in to the new behavior, not the old one 21:25:11 <melwitt> oh 21:25:15 <mriedem> well that's what's proposed 21:25:17 <dansmith> melwitt: so you're (supposedly) opting in to having it be magic 21:25:19 <sdague> it's only surprising to people that are used to getting punched in the face 21:25:33 <sdague> and then we stop doing that 21:25:38 <sdague> for the majority of them 21:25:45 <dansmith> sdague: I feel like arguments like that are not very constructive 21:25:52 <dansmith> so anyway, that's my opinion, let's move on 21:26:02 <mriedem> ok, so, just to finalize, 21:26:11 <mriedem> the reason i brought this up is i want to start drafting a spec, 21:26:18 <mriedem> b/c the neutron meetup is local for me next week 21:26:23 <sdague> ok, I feel like the entire "get me a network" thing was driving by users that were really really frustrated that was not the default 21:26:25 <mriedem> i think i'm going to just doc this all as alternatives in the spec, 21:26:29 <mriedem> and we can hash it out in review 21:26:40 <mriedem> sdague: agree 21:26:50 <mriedem> moving on 21:26:54 <mriedem> gus: mikal: go! 21:26:59 <gus> heh. 21:27:26 <gus> So: privsep/os-brick status: The code exists to do a simple s/rootwrap/privsep/g in os-brick. 21:27:42 <gus> It depends on a bunch (6?) of small changes across other projects (nova, devstack, cinder) 21:28:02 <mriedem> #link main os-brick privsep change https://review.openstack.org/#/c/277224/ 21:28:05 <gus> All the isolated tests I've thrown at it work, but the whole thing fails in tempest integration tests atm ;) 21:28:18 <gus> I'm still debugging that latter, it's almost certainly something to do with eventlet. 21:28:20 <gus> (it hangs) 21:28:56 <gus> My concern is that at this point there are still a bunch of small changes ahead of us, mostly to do with juggling global-requirements once there is an appropriate os-brick/privsep release. 21:29:09 <gus> I know we get thingy about changing requirements close to release deadlines. 21:29:19 <tonyb> gus: we have 10 days still 21:29:32 <gus> tonyb: right. 21:30:01 <gus> The last round of minor changes to other projects took 2 weeks to get nova +2s, and haven't seen any +2s from cinder or devstack yet. 21:30:19 <gus> (thanks to those who approved the nova ones over night) 21:30:23 <tonyb> sdague: can you hepl with the devstack side? 21:30:42 <tonyb> sdague: most of the changes are in mitaka-nova-priorities-tracking 21:30:49 <sdague> tonyb: yeh, though I didn't see anything working yet in full stack 21:30:57 <sdague> which was my concern 21:31:06 <gus> so .. we can still make it, but I feel like it's getting tight - particularly wrt juggling the cross-project review latency. 21:31:13 <gus> sdague: yep, agreed. 21:31:47 <gus> the depends-on changes are all safe (afaics) to merge into the other projects regardless of whether we go ahead with the rest of os-brick/privsep or not. 21:31:55 <scottda> os-brick release just went out 21:32:10 <gus> they're things like an additional rootwrap filter, that at worst would be simply unused. 21:33:22 <gus> I don't have any specific request - just a heads-up that when I whack this final eventlet interaction bug, there's going to be half-a-dozen-or-so changes that are going to need some fast review turnaround. 21:34:08 <mriedem> so, 21:34:22 <mriedem> summaries in the ML on big hairy things like this have been common lately 21:34:23 <mriedem> all the rage 21:34:40 <mriedem> b/c i see a bunch of changes on different topic branches, so tracking it all in gerrit is confusing me 21:34:45 <mriedem> some are tracked against a bp and some aren't 21:34:58 <mriedem> so as a reviewer i'm not entirely sure where to start (besides the main os-brick change with the 5 depends-on) 21:34:58 <sdague> right, it's especially hard when the Depends-On is just a wall 21:35:19 <sdague> it would be useful to at least annotate them to which repos they are on to understand 21:35:22 <scottda> gus: I'm not positive, but I think getting privsep changes into os-brick is too late for Mitaks 21:35:26 <melwitt> is there a section for them in the priorities etherpad? 21:35:41 <melwitt> that gus can just link all of them 21:35:46 <tonyb> melwitt: yeah they're under os-vif 21:35:57 <tonyb> melwitt: somewhere near 144 21:35:59 <sdague> scottda: what? 21:36:06 <mriedem> tonyb: melwitt: yeah but it's not very detailed 21:36:11 <mriedem> it's just some links, i can figure those out via gerrit 21:36:17 <sdague> scottda: because, honestly, this is a release blocker 21:36:18 <scottda> os-brick was just released today. 21:36:25 <gus> Can I mix comments in with the Depends-On change description "meta headers" ? 21:36:31 <sdague> we're in an upgrade crippled state 21:36:33 <scottda> I'm trying to get hemnafk to verify , but he is AFK 21:36:34 * gus isn't sure what the parsing requirements of that are. 21:36:34 <mikal> gus: alternate lines? 21:36:47 <tonyb> mriedem: Okay tell me what else you need in there and I'll add it 21:36:56 <mikal> # The following depends-on is against os-brick 21:37:00 <tonyb> mriedem: I just followed the "template" 21:37:00 <mikal> Depends-On 12345 21:37:15 <sdague> yeh, each Depends-On must be the only thing on it's line 21:37:16 <mriedem> tonyb: well, like what jaypipes and alaski and others have done in the ML recently on where big things are at 21:37:19 <sdague> but you can have other stuff in there 21:37:24 <gus> mikal: yep, that would be useful for me too - I wasn't sure how strict that block needed to be. 21:37:29 <sdague> https://review.openstack.org/#/c/282031/ 21:37:53 <gus> I have some mention in the nova priorities etherpad - but of course that doens't affect devstack + cinder (+ os-brick). 21:38:12 <sdague> so, I just want to circle back on the os-brick thing 21:38:16 <mriedem> what we need is like a management type that knows how to put together fancy charts.... 21:38:20 <mriedem> where could we find one of those... 21:38:39 <sdague> because this can't be too late for os-brick 21:38:40 <mikal> So, I kind of disagree 21:38:50 <mikal> What we need is the cross project element to work better 21:38:55 * jaypipes hides from mriedem 21:39:09 <mikal> Do we need to try and get everyone into a dedicated meeting early next week? 21:39:09 <mriedem> library freeze is i think the same time as FF 21:39:14 <scottda> hemnafk: keeps reporting that the privsep changes cause Nova to hang...that's the latest report from him I've seen... 21:39:49 <gus> scottda: yep, that's probably a repeat of my latest status update on the main change. 21:39:52 <mriedem> Feb 22-26R-6Final release for non-client libraries 21:39:54 <scottda> I reckon another release of os-brick could still go out if this is fixed. I just noticed today an email that os-brick was released. 21:40:12 <mriedem> yeah 1.0.0 went out with bug fixes 21:40:20 <mriedem> i think we have until next friday for another os-brick release according to the schedule 21:40:42 <sdague> mikal: the answer is yes, who is going to be point person on it? 21:40:45 <mriedem> mikal: so to answer your question, yes i think some kind of project management would be helpful 21:41:00 <mikal> So, the only problem is that next week wastes a few days 21:41:08 <mikal> We could do it tomorrow, but I'm afk all day 21:41:25 <mikal> I don't want to sit idle over the weekend if its avoidable 21:41:55 <mriedem> well i guess it depends on what works for gus 21:41:57 <sdague> mikal: well, how about start with pulling together where we are, and what's blocking moving forward 21:42:10 <mikal> Yep 21:42:15 <mriedem> yeah some high level context would be helpful before a hangout or something 21:42:15 <sdague> even if it's not a meeting, a consolidated view on all of this stuff would be really useful 21:42:17 <mikal> gus and I can work on an email summary to -dev 21:42:24 <mriedem> cool 21:42:25 <gus> sg 21:42:29 <mikal> And make is super clear what we think brick and devstack need to do 21:42:37 <mikal> We'll get that out in the next couple of hours? 21:42:41 <mriedem> awesome 21:42:51 <sdague> yep, and to be clear, I'll approve the devstack changes as soon as I've got a test run showing me it's working 21:43:01 <sdague> I just couldn't find that in the stack 21:43:34 <gus> sdague: there's things like https://review.openstack.org/#/c/277696/ that just sets a new nova.conf entry. 21:44:01 <sdague> gus: right, but I actually want to see something using that to do something :) 21:44:12 <gus> sure, it just means it remains on the critical path then. 21:44:30 <gus> (sdague: but I understand your reluctance) 21:45:18 <sdague> right, but then we know it works vs. land, oh broken, land, oh broken. 21:45:33 <gus> yep. 21:45:55 <mriedem> anything else on this? 21:46:01 <gus> not from me. 21:46:11 <mriedem> anything else from anyone for open discussion? 21:46:16 <scottda> privsep is out for mitaka in os-brick, according to smcginnis 21:46:34 <mriedem> scottda: umm https://review.openstack.org/#/c/277224/ 21:46:36 <gus> ^ huh, well then at least I get my weekend back. 21:46:50 <sdague> scottda: um, that's really not possible 21:47:00 <scottda> I'm just the middle man here 21:47:21 <mriedem> you mean out as in not happening 21:47:31 <mriedem> vs out as released :) 21:47:35 <mriedem> important distinction 21:47:39 <smcginnis> mriedem: Not happening. 21:47:42 <mriedem> yeah 21:47:44 <mriedem> i read the cinder channel 21:47:46 <patrickeast> it was discussed in the last cinder meeting briefly, see ~16:06 http://eavesdrop.openstack.org/meetings/cinder/2016/cinder.2016-02-17-16.00.log.html 21:47:49 <smcginnis> mriedem: There are still issues that need to be resolved and tested. 21:47:57 <mriedem> smcginnis: yeah, https://review.openstack.org/#/c/277224/ 21:48:04 <mriedem> that's what we were talking about 21:48:19 <edleafe> mriedem: did a quick count on the config options status. 504 total, 331 either merged or in the pipeline, 173 untouched 21:48:37 <sdague> smcginnis: this is a monsterous upgrade problem 21:48:52 <sdague> that we only didn't revert os-brick in liberty because it was going to be fixed in mitaka 21:48:54 <smcginnis> sdague: Agreed. Not happy we weren't able to get it working in time. 21:49:27 <smcginnis> sdague: Well, TBF, there's a lot of goodness in os-brick that far surpasses the pain of copying rootwrap files. 21:49:41 <sdague> smcginnis: it completely breaks our upgrade model 21:49:56 <sdague> I don't think I can bang my shoe on the table hard enough about that 21:50:15 <smcginnis> We are tied, yes. But did we add anything to rootwrap filters in this cycle? 21:50:40 <mriedem> i think there was maybe one thing 21:50:44 <mriedem> i can check after the meeting 21:50:48 <mriedem> i know we held things out 21:50:55 <mikal> So, where to from here? 21:50:59 <mikal> Do we still want that summary email? 21:51:05 <mikal> Or are we dead in the water until newton? 21:51:08 <sdague> ok, so os-brick will never accept another rootwrap change until this is sorted, right? 21:51:34 <gus> "Add os-brick's scsi_id command to rootwrap" - Sep 9 21:51:42 <smcginnis> I don't work in absolutes, but I can strongly discourage it. :) 21:51:51 <sdague> smcginnis: no, this has to be absolutes 21:51:53 <mriedem> mriedem@ubuntu:~/git/os-brick$ git log --oneline --no-merges 0.6.0.. -- etc/os-brick/rootwrap.d/os-brick.filters 21:51:53 <mriedem> d3b9696 os-brick add cinder local_dev lvm code 21:51:53 <mriedem> ebce3c3 DRBD connector class 21:52:09 <mriedem> we kept ebce3c3 out of nova 21:52:14 <mriedem> b/c of the filter add 21:52:25 <mriedem> d3b9696 is not being used in nova i dont think 21:52:31 <mriedem> it was a late add, to prep for newton 21:52:43 <smcginnis> mriedem: Right, not yet. 21:53:01 <mriedem> 0.6.0 is what's in stable/liberty u-c, that's there i get the version from 21:53:53 <mriedem> so i guess let's take this to -nova or -cinder now 21:53:56 <mriedem> and wrap up the nova meeting 21:54:12 <mriedem> i imagine there are some bits of brain and skull on the floor somewhere 21:54:18 <mriedem> anyone have anything else? 21:54:33 <mriedem> ok, ending, thanks everyone 21:54:35 <mriedem> #endmeeting