14:01:19 <russellb> #startmeeting nova
14:01:20 <openstack> Meeting started Thu Mar 20 14:01:19 2014 UTC and is due to finish in 60 minutes.  The chair is russellb. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:01:22 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:01:24 <openstack> The meeting name has been set to 'nova'
14:01:36 <russellb> hello everyone!  who's here to talk about nova?
14:01:38 <alaski> o/
14:01:39 <dansmith> o/
14:01:45 <johnthetubaguy> o/
14:01:54 <russellb> #link https://wiki.openstack.org/wiki/Meetings/Nova
14:02:06 <garyk> hi
14:02:15 <russellb> #topic icehouse-rc1
14:02:22 <russellb> #link https://launchpad.net/nova/+milestone/icehouse-rc1
14:02:28 <russellb> we're aiming to release RC1 next week
14:02:38 <russellb> 9 bugs on the list left to close out
14:02:48 <russellb> let's see if we can merge as much as we can this week
14:03:07 <russellb> come next week i want to start allowing only show stopper and regressions on the list
14:03:26 <russellb> so some of these would get bumped to "nice to have" if they don't make it this week
14:03:45 <russellb> anyone aware of issues not on this list that should be?
14:03:47 <garyk> is the state of the gate a concern?
14:04:02 <russellb> i see the gate time, what's the root cause?
14:04:12 <russellb> i see this patch was promoted to the head of the gate: https://review.openstack.org/79816
14:04:21 <alaski> I think a couple of those might be in the gate already
14:04:49 <russellb> excellent
14:04:58 <garyk> this seems to be the bug that is reported on failures - https://bugs.launchpad.net/neutron/+bug/1283522
14:05:41 <russellb> 138 fails in 24hrs / 956 fails in 14 days
14:05:42 <russellb> on that bug
14:05:52 <russellb> ouch
14:06:21 <russellb> looks like a related patch merged 39 minutes ago
14:06:23 <dansmith> yeah, neutron deadlock and libvirt timeouts are the two issues I think
14:06:33 <russellb> OK
14:06:45 <russellb> so looks like patches going in now related to both
14:06:51 <russellb> so hopefully we'll see some improvement today
14:07:30 <russellb> if something is approved by monday and just fighting through the gate, we can still try to get it in
14:07:32 <russellb> not a huge deal
14:07:51 <russellb> any comments / questions / concerns on icehouse-rc1?
14:08:07 <russellb> one bug was brought up by cyeoh for discussion
14:08:09 <russellb> though he can't be here
14:08:18 <russellb> #link https://bugs.launchpad.net/horizon/+bug/1286297
14:08:25 <russellb> "we made a backwards incompatible change in https://review.openstack.org/#/c/40811/ which caused this"
14:08:32 <russellb> "some details here: http://lists.openstack.org/pipermail/openstack-dev/2014-March/030508.html"
14:08:54 <russellb> question is, even though it's been merged for almost all of icehouse, do we revert it or not?
14:08:59 <johnthetubaguy> api was not very usable before that change though, not by a non-admin anyways
14:09:14 <johnthetubaguy> but breaking our users seems possibly worse
14:09:33 <dansmith> my opinion is that it's been out in the wild for a long time, it's hard to justify reverting it at this point, IMHO
14:09:44 <johnthetubaguy> I wondered about making the flavor auth silently succeed if its a no-op, but maybe thats making things worse
14:09:54 <russellb> ttx's point on list was, if we leave it, we decide to also break everyone going release to release
14:10:17 <russellb> which is honestly a lot more common
14:10:18 <russellb> than CD
14:10:26 <russellb> that's my impression at least
14:10:30 <dansmith> it's a semantic change though, right? not a hard break, and it fails in a reasonable way
14:10:35 <dansmith> I guess I didn't realize there was a thread yet
14:10:37 <PhilD> Sounds more like we fixed a bug than made an incompatible change to me - I'd vote it should stay
14:10:47 <russellb> OK, make your votes on the thread
14:10:54 <russellb> just want to make sure it's a conscious decision
14:11:00 <russellb> my gut says leave it, too
14:11:01 <russellb> FWIW
14:11:23 <russellb> #topic bugs
14:11:26 <russellb> other bugs stuff
14:11:30 <russellb> not sure who added this one to the agenda
14:11:36 <russellb> "https://review.openstack.org/#/c/77524/ bug fix related to cinder and keystone v3, cinderclient is already fixed but nova needs to use keystone v3 API - question is what needs to happen before we are "supporting" keystone v3? What needs to happen in Tempest? What areas of nova do we know need to change?"
14:12:50 <PhilD> I think we do need to be claer on "Bug fix" vs "incompatible change" though - any bug fix in the API is a visible change, and this makes the code match the API documentation
14:13:20 <russellb> agreed with that
14:13:23 <johnthetubaguy> +1
14:13:25 <russellb> #topic blueprints
14:13:30 <russellb> OK, juno blueprints!
14:13:38 <russellb> there's been some progress on an updated process for Juno
14:13:46 <russellb> #link https://wiki.openstack.org/wiki/Blueprints#Nova
14:13:50 <russellb> johnthetubaguy just updated that wiki page
14:13:52 <PhilD> Are we still doing Keystone V3 support as a topic ?
14:14:12 <russellb> PhilD: let's come back to it if you'd like, didn't seem like anyone was around to cover that, but can come back to it
14:14:22 <russellb> so, we have a nova-specs git repo
14:14:28 <russellb> we have a template in the repo for specs
14:14:40 <russellb> and we want to require *all* juno blueprints to go through this repo for review
14:14:46 <russellb> even ones previously reviewed
14:14:49 <johnthetubaguy> I have just unapproved all non-completed blueprints, turns out it is possible
14:14:55 <russellb> johnthetubaguy: great
14:15:14 <russellb> note that we're going to be learning on the go with this, it's a bit of experimentation
14:15:28 <russellb> but i think it's going to result in much better reviews than using the horrible "whiteboard"
14:15:40 <russellb> and we'll have a nice archive of specs in git
14:15:58 <russellb> I think the repo is open for business at this point though
14:16:19 <lcostantino> so, every BP that were approved and had code but deferred will have to go tru this new process right?
14:16:25 <russellb> correct
14:16:39 <russellb> if it was previously approved, hopefully the review will be quick
14:16:44 <lcostantino> great
14:16:47 <russellb> but it does mean we're requiring a much more thorough design document
14:16:55 <russellb> where as before we may have been more lenient
14:17:34 <johnthetubaguy> one big bonus is the use of gerrit should lead to more consistency, in the long run
14:18:07 <russellb> and should address a major complaint i get about the quality of blueprint detail
14:18:24 <russellb> a bit less painful to enforce here
14:18:35 <devoid> so do you place the non-approved blueprint in /juno, what is the naming convention?
14:18:47 <russellb> devoid: it's covered in the tempalte
14:18:57 <russellb> juno/approved/my-blueprint.rst
14:19:12 <russellb> where my-blueprint is https://blueprints.launchpad.net/nova/+spec/my-blueprint
14:19:13 <johnthetubaguy> russellb: do we want to leave the template open to comments till Monday, for those who are interested?
14:19:38 <russellb> johnthetubaguy: i think we should open it now, and say watch for updates
14:19:45 <russellb> i suspect we may have a continuous flow of updates through the cycle
14:19:58 <devoid> I know that the operators group is working to better lay out end-user and deployer impact.
14:20:12 <russellb> devoid: oh?  for this?
14:20:16 <russellb> sure, that'd be great
14:20:22 <devoid> for blueprint approvals
14:20:31 <russellb> and can just be submitted to gerrit as an update to our template
14:20:38 <devoid> yup.
14:20:50 <russellb> any questions or concerns?
14:21:01 <johnthetubaguy> russellb: OK, so I am keen to wait for that feedback? or do we want to go now?
14:21:01 <russellb> otherwise i'll post to the mailing list today drawing more attention to the progress
14:21:11 <devoid> one concern is if blueprints are approved too quickly there's no time for a broad set of people to review.
14:21:44 <russellb> johnthetubaguy: sure, I guess the udpate this week can be "take a look at our template and provide feedback", that's fine with me
14:21:45 <PhilD> Good point, I'd like to see some guildlines on that point.
14:21:56 <russellb> that seems fair
14:22:02 <johnthetubaguy> russellb: yeah, sorry to add a delay
14:22:03 <dansmith> russellb: even with this new system, we're planning not to approve until there is some actual code as we discussed in UT right?
14:22:08 <russellb> johnthetubaguy: no worries
14:22:16 <johnthetubaguy> so, about delays
14:22:21 <russellb> dansmith: good question
14:22:27 <russellb> i think there's two things
14:22:29 <russellb> 1) approving the spec
14:22:38 <russellb> 2) approving a blueprint into a milestone based on that spec
14:22:40 <dansmith> oh, targeting
14:22:41 <dansmith> gotcha
14:22:42 <russellb> maybe we should separate those things
14:22:44 <dansmith> yes
14:22:46 <dansmith> that sounds fine to me
14:22:52 <johnthetubaguy> sometime we need to push little things through, I don't like adding a big delay, but blueprint delays feel better than code.
14:22:54 <russellb> johnthetubaguy: which makes the "originally approved for" thing a bit more difficult
14:22:55 <dansmith> just need to make it clear what "approving the spec" means
14:23:08 <johnthetubaguy> russellb: yeah, true
14:23:25 <russellb> johnthetubaguy: makes me want to go back to removing it ...
14:23:41 <johnthetubaguy> russellb: I wonder about a proposed folder, then move into approved when code goes up, but that feels bad...
14:24:02 <russellb> sounds like tracking work
14:24:06 <russellb> and i'm hoping to keep tracking separate
14:24:10 <johnthetubaguy> russellb: how about just saying approved in Juno-1, but first target might be Juno-2, I am ok with that
14:24:16 <devoid> shouldn't you approve a blueprint separate from code?
14:24:31 <johnthetubaguy> russellb: yeah, tracking in one place, lp, makes most sense
14:24:51 <russellb> johnthetubaguy: OK, but maybe a note to the template and wiki page that clarifies the difference between approving the spec, and targeting to a milestone
14:24:52 <johnthetubaguy> devoid: we sure will, but don't want to approve too much that no one will ever work on
14:25:02 <alaski> I'm +1 on removing the milestone stuff from the template fwiw
14:25:21 <russellb> alaski: yeah +1 i think ... it's confusing
14:25:22 <PhilD> Maybe we could try and capture what would constitute having a wide range of feedback into the spec review - so for example that there should be some review from an operator, etc ?    Feels that's more whats needed than just (has it been open for review X days)
14:25:39 <johnthetubaguy> alaski: I just worry there is no way to track the history, but yeah, it seems simpler to remove it at this point, its just confusing :(
14:25:40 <russellb> PhilD: perhaps, but i think it also depends on the blueprint
14:25:48 <russellb> there's a lot of blueprint stuff ops aren't going to care about
14:25:53 <russellb> or things that are really just not controversial
14:25:59 <russellb> major refactorings
14:26:11 <devoid> agreed, it depends on the bp.
14:26:11 <russellb> maybe case by case, ensure we have sufficient input based on what it is
14:26:21 <johnthetubaguy> PhilD: neutron patches we wait for neutron core, it seems similar to that kind of thing, just do it case by case?
14:26:35 <devoid> posts to mailing list and operators can help for things that clearly need operator input.
14:27:55 <PhilD> Yeah, its hard to get the balance I know.  Just feels that a s John noted a delay here is much better (or at least much less bad) than a delay/rework later on - so we shouldn't be shy of holding off approval in those cases.  I'd rather see this stage lean to slower
14:28:15 <russellb> good poitn
14:28:27 <russellb> it's much less costly to rework a spec than code
14:28:37 <johnthetubaguy> personally, any new process should feel more lightweight than before, I think letting people do the "right thing" and see what breaks is best here
14:28:39 <russellb> so we need to ensure we get it right
14:28:46 <russellb> johnthetubaguy: agreed
14:28:50 <johnthetubaguy> PhilD: +1
14:28:54 <russellb> but i think the things we've talked about are good principles
14:29:14 <johnthetubaguy> yeah, we should evolve those review guidelines
14:29:20 <russellb> indeed
14:29:37 <johnthetubaguy> #link https://wiki.openstack.org/wiki/Blueprints#Blueprint_Review_Criteria
14:29:39 <PhilD> I think its not always easy to devs to judge what does and doesn;t haev an impact on an operator (see some of the recent reverts)  - and part of the point of this is to have BPs expressed to the extent that a non-coder can underdstand what's intended
14:30:23 <russellb> #link http://git.openstack.org/cgit/openstack/nova-specs/tree/template.rst
14:30:23 <PhilD> If they can;t then the BP isn't really complete enough IMO
14:30:25 <johnthetubaguy> PhilD: I would love to see operators join nova-drivers, by participating in lots of nova-specs reviews, just lets see how that goes I think
14:30:44 <PhilD> I'll be there ;-)
14:31:01 <johnthetubaguy> PhilD: the big issue was not having a blueprint that made it clear there was an impact, hopefully we will now be better at that!
14:31:45 <russellb> any more on blueprints?
14:31:54 <russellb> it will be an evolving process i'm sure
14:31:57 <johnthetubaguy> no from me
14:32:05 <russellb> but appreciate willingness to try it out and evolve
14:32:16 <devoid> russellb: +1
14:32:19 <russellb> helps when nobody likes the current situation, heh
14:32:20 <PhilD> +1
14:32:27 <russellb> #topic open discussion
14:32:37 <russellb> plenty of time for other topics if anyone would like
14:32:41 <PhilD> Keystone V3 ?
14:33:08 <russellb> sure
14:33:32 <PhilD> Mostly this is a client issue - I was just wondering what plans were for getting V3 support into the client
14:33:44 <russellb> no plans on my radar
14:34:11 <russellb> there's been a bit more broad hierarchical multi-tenancy discussion
14:34:19 <PhilD> Well V2 becomes deprecited in Icehouse, so we'll need to do something
14:34:21 <russellb> which would require v3
14:34:34 <russellb> i think there's some coordination fail in there
14:34:49 <russellb> i think it's absurd to mark something deprecated when almost every project doesn't support the new thing yet
14:35:28 <sdague> russellb: agreed
14:35:31 * russellb pings to see if dolphm happens to be around
14:35:37 <sdague> perhaps that should be on the project meeting for next week
14:35:44 <mrodden> i think its their way of cracking the whip...
14:35:55 <PhilD> We want to eb abel to use domains in Keystone, which means you only have to be able to use the V3 into Keystone from Horizon say, but other clients on V2 now need to be able to auch users when there name isn;t unique.   You can kind of do this via teh V2 API by going to ID based auth, but that's a bti clutzy (but Ok as a short term move)]
14:36:01 <russellb> sdague: +1
14:36:02 <devoid> ressellb: especially when keystone's cli client doesn't support v3 either.
14:36:17 <PhilD> I have patches up for that in nova and neutron client at the moment
14:36:18 <mrodden> devoid: v3 support comes from openstack client
14:36:40 <PhilD> And I see that some other piecemeal changes seem to be trying to land
14:37:03 <russellb> there's some keystone v3 coordination needed across projects, so let's plan to cover that in the next cross project meeting (tuesday)
14:37:08 <browne> mrodden: yes, but the keystoneclient middleware doesn't support v3 i believe
14:37:08 <russellb> please join if you're interested and able
14:37:19 <PhilD> Ok - what time ?
14:37:38 <PhilD> @browne - I think it does
14:37:46 <russellb> #link https://wiki.openstack.org/wiki/Meetings/ProjectMeeting
14:37:47 <mrodden> browne: yeah i'm not sure about that
14:37:51 <russellb> 2100 UTC
14:38:02 <PhilD> Ok, I'll see what I can do
14:38:08 <russellb> sorry for the rough time
14:38:19 <devoid> browne, mrodden, the middleware supports v3, just not the cli. but openstack client doesn't have packages available yet.
14:38:24 <russellb> #note need to discuss keystone v3 support across projects in the next cross project meeting
14:39:14 <browne> i may be wrong, but i think the auth_token middleware still only supports getting v2 tokens
14:40:14 <russellb> dolph doesn't seem to be around
14:40:19 <russellb> another option would be to start a ML thread
14:40:30 <russellb> anyone interested in doing that?
14:40:38 <PhilD> I thought it could be configured to work with v2 and v3, but I could be wrong too - it would be good to get some clartiy about what the planned migration is for s system running V2 to one running V3
14:41:00 <PhilD> Yeah I could do that
14:41:12 <russellb> ok perfect
14:41:15 <russellb> much appreciated
14:41:22 * johnthetubaguy wonders how long it will be before all clients migrate to v3 keystone
14:41:25 <russellb> yes, need to figure out what the migration is expected to look like
14:42:02 <browne> https://github.com/openstack/python-keystoneclient/blob/master/keystoneclient/middleware/auth_token.py#L764
14:43:09 <PhilD> I'm trying to see if we can get our Keystone folks to tackle the v3 for all clients  - seems like it would make more sense for them to do it that for each project to have to work out what to do.   Most clients don't even include the keystoen client at the moment though
14:43:18 <russellb> PhilD: +1
14:43:47 <PhilD> I said "trying" - I still have some way to go to convince them ;-)
14:44:00 <russellb> well, personally that's what i expect from all projects
14:44:03 <PhilD> Of course if only we had a single converged client .....
14:44:12 <russellb> just like with nova v3, i expect nova devs to reach out and help do the work to migrate users of nova
14:44:24 <johnthetubaguy> PhilD: thats looking closer now right?
14:44:27 <russellb> our client of course, but also help fix horizon, trove, heat
14:45:35 <ndipanov> sorry guys forgot about this
14:45:42 <russellb> ndipanov: so you had an issue you wanted to bring up?
14:45:42 <mriedem> oops
14:45:50 <ndipanov> and my calendar was chillin as well
14:46:17 <ndipanov> so yeah... I looked into https://blueprints.launchpad.net/nova/+spec/graceful-shutdown
14:46:46 <johnthetubaguy> we are a bit closer on that one these days right?
14:46:56 <ndipanov> and apart from a bug in nova that is easy to fix... that makes this dead after switching to oslo
14:47:01 <ndipanov> I still don't think this is done
14:47:28 <russellb> ndipanov: incomplete or fundamentally broken?
14:47:30 <johnthetubaguy> ndipanov: you got that bug?
14:47:42 <ndipanov> russellb, I'd say incomplete
14:48:23 <ndipanov> once the service receives one of these signals - for this to work properly it needs to rally wait for every gt to finish
14:48:39 <ndipanov> and also finish any rpc stuff it has going on
14:48:50 <ndipanov> but not accept any other connections
14:48:57 <ndipanov> does that sound sane?
14:49:06 <ndipanov> johnthetubaguy, yes - it's really tiny
14:49:30 <johnthetubaguy> ndipanov: I thought that was done already
14:49:49 <johnthetubaguy> ndipanov: stops getting new rpc messages at least, and waits for current stuff to finish
14:50:05 <johnthetubaguy> ndipanov: I thought it used to anyway...
14:50:28 <johnthetubaguy> leifz: ping
14:50:44 <dansmith> johnthetubaguy: I think he's saying that post oslo-messaging things are different now
14:50:47 <ndipanov> johnthetubaguy, I don't think so - it just calls (well should) rpc.cleanup() which eventually calls connection.close
14:51:00 <browne> PhilD: http://lists.openstack.org/pipermail/openstack-dev/2014-March/030403.html
14:51:02 <ndipanov> dansmith, I think they aren't fundamentally
14:51:08 <johnthetubaguy> right I got you, so it got regressed by the oslo changes
14:51:12 <dansmith> I think we need to stop listening to compute.$host, but leave everything else until all the GTs die, right?
14:51:29 <ndipanov> dansmith, right
14:51:39 <ndipanov> dansmith, and that's not what's happening
14:51:42 <dansmith> otherwise conductor things will just fail
14:51:44 <dansmith> yeah
14:52:01 <ndipanov> johnthetubaguy, it did regress from not working to not working even more :)
14:52:52 <johnthetubaguy> ndipanov: ok… I got the impression the stop listening to the queue stuff got implemented in olso, then sync across, but I have not looked in detail at that… oops
14:52:52 <ndipanov> well I don't think it is anyway...
14:53:51 <johnthetubaguy> ndipanov: that was the intention at least, so I am certainly agreed with you there
14:54:27 <ndipanov> dansmith, would you say that in order to test this - you could 1) kill the conductor just to cause a call delay
14:54:32 <ndipanov> 2) boot an instance
14:54:45 <ndipanov> 3) send SIGINT to compute and restart conductor.
14:55:03 <ndipanov> 4) see that the boot finishes and then compute dies
14:55:08 <ndipanov> ?
14:55:22 <dansmith> ndipanov: manual testing? I'd say start a tempest largeops run against devstack and then kill your compute and see some logs that show it cleaning things up
14:55:35 <dansmith> ndipanov: because killing conductor will hide whether you're properly still open for rpc replies I think
14:56:00 <ndipanov> hmmm
14:56:25 <dansmith> it's not going to be an easy test
14:56:45 <johnthetubaguy> yeah, just kill compute, see the rpc count rise as you issue terminate commands to all the vms on the compute?
14:56:53 <dansmith> maybe just "nova boot foo; sleep 0.5; killall nova-compute"
14:57:10 <johnthetubaguy> ideally during a snapshot, so you have to wait for the snapshot to finsih, then nova-compute to die
14:57:16 <dansmith> and then check in the db that it got to active
14:57:34 <johnthetubaguy> that bp doesn't do any tidy up
14:57:49 <johnthetubaguy> its just about a kill that waits for current things to finish
14:58:10 <johnthetubaguy> (to avoid the need for any tidy up)
14:58:52 <dansmith> anyway, we're about out of time, but sounds like some thankin' needs doin' on how to make sure this works
14:59:03 <russellb> :)
14:59:06 <russellb> any other topics with 1 minute left?
14:59:31 <ndipanov> johnthetubaguy, afaict it doesn't even do that ... it will call service.stop() which just kills gts
14:59:45 <russellb> k, back to #openstack-nova we go!  :)
14:59:47 <russellb> thanks everyone!
14:59:49 <russellb> #endmeeting