21:01:10 #startmeeting nova 21:01:11 Meeting started Thu Nov 21 21:01:10 2013 UTC and is due to finish in 60 minutes. The chair is russellb. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:01:12 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:01:15 The meeting name has been set to 'nova' 21:01:22 hello everyone! 21:01:23 hi! 21:01:29 hi 21:01:30 hi 21:01:31 hi 21:01:31 o/ 21:01:32 hi 21:01:39 \o 21:01:40 #link https://wiki.openstack.org/wiki/Meetings/Nova 21:01:43 o/ 21:01:52 #topic general announcements 21:01:55 o/ 21:02:07 the compute program needs a mission statement 21:02:10 o/ 21:02:20 i drafted one, if anyone has feedback let me know, before i propose it to the governance repo 21:02:26 #link http://lists.openstack.org/pipermail/openstack-dev/2013-November/019677.html 21:02:35 hi 21:02:41 \o 21:02:42 it's kinda buzzwordy, but ... trying to capture everything we do into a brief statement 21:02:49 hi 21:03:02 anyway, not a huge deal, don't want to bikeshed it, but want to give everyone a chance to provide input if you want to :) 21:03:23 next thing ... mid cycle meetup 21:03:26 we talked about this a bit last week 21:03:29 looks quite good to me, I wonder if we should mention the API, but yeah, looks good 21:03:50 dates are confirmed as Feb 10-12 21:03:51 3 days 21:03:59 in Orem, UT 21:04:02 (right outside SLC) 21:04:08 woohoo! 21:04:16 at Bluehost's facility, thanks to geekinutah :) 21:04:23 I assume there will be a ML announcement 21:04:26 yes 21:04:28 kk 21:04:30 we're very close 21:04:31 and woot 21:04:40 last thing was getting hotel links on our wiki page with all the details 21:04:43 Morning 21:04:46 we'll also have an eventbrite thing to register 21:05:08 but once the details are on the page, i'll post to the ML 21:05:12 just wanted to share the status 21:05:16 That's super awesome, thanks for tweaking the dates 21:05:41 i'm hoping to tack on a day of snowboarding :) 21:05:47 if anyone wants to join ... 21:05:52 and laugh at me 21:05:57 russellb: i'm in! 21:06:00 russellb: sorry, I'm a nerd and can't do that. I would hurt my typing hands. 21:06:01 woot 21:06:09 maybe we could go tubing one night, heh 21:06:19 * n0ano (skier) refuses to comment on knuckle draggers :-) 21:06:24 doesn't anyone ski anymore? 21:06:25 I would like to go skidooing 21:06:25 mikal: I am with you, is there a good pub with a nice view? 21:06:26 hotel should be just under $100/night 21:06:45 johnthetubaguy: we can find one I am sure. Oh, except its Utah. Do they drink there? 21:06:47 so that should be enough to go ask nicely for budget 21:07:07 mikal: every day except sunday (serious) 21:07:15 jaybuff: works for me 21:07:18 next item of business! 21:07:20 Sunday can be liver recovery day 21:07:21 #topic sub-teams 21:07:32 * johnthetubaguy raises hand 21:07:39 johnthetubaguy: k, you can go first 21:07:44 * n0ano scheduler 21:07:58 * hartsocks waves 21:07:59 so, xenapi stuff, making progress with tempest tests 21:08:17 quite a few blueprint bits and bobs going in 21:08:24 I think thats about all from me 21:08:45 cool, smokestack come back? 21:08:55 not that I know of, its a packaging issues 21:09:07 danp is on the case though, I am told 21:09:27 ok 21:09:39 n0ano: scheduler! 21:09:54 lively discussion this week (I should go away more often) 21:09:58 instance groups: hope to complete V3 api's for Icehouse 21:09:59 scalaility: hoping to get info from Boris, goal of ~1000 nodes, simulation vs. real world numbers 21:09:59 sql vs. nosql: discussed a little, google Omega project is worth looking at 21:09:59 scheduler as a service: Nova, Cinder & Neutron all need a scheduler, very early on this 21:10:18 that's pretty much what we went through for now. 21:10:26 ok, and i think there's a thread starting on openstack-dev about now about the common scheduler 21:10:30 so that's something to watch 21:10:33 i'm very keen on that idea 21:10:53 to just bite the bullet and run with that sooner than later ... with a very tightly limited scope to only replace nova's scheduler to start with 21:11:07 maybe, neutron's needs are simple so I'm not totally convined we need a separate service but we'll see 21:11:36 well i think it's more about the information living somewhere common 21:11:42 failure domains, locality 21:11:52 because scheduling instances needs all of that 21:12:03 but anyway, a hot topic in the next week i expect 21:12:10 let's take it to the list :) 21:12:15 there are other proposals to have multiple schedulers, reducing failure domains a little but good discussion 21:12:25 OK 21:12:29 hartsocks: you're up 21:12:32 So.. 21:12:37 * mriedem1 raises hand after hartsocks 21:12:42 We have a couple bugs folks marked "Critical" ... 21:12:42 #link https://bugs.launchpad.net/nova/+bug/1195139 21:12:42 ... kills the vmware drivers for anyone on postgres it turns out. oops. 21:12:43 #link https://bugs.launchpad.net/nova/+bug/1252827 21:12:43 ... kills our CI environment. oops. 21:12:44 Still working on priorities for our BP work. We're excited to do something on the configuration validator that will be broadly usable. *woot* 21:12:45 * lifeless puts up hand for coordinating the common scheduler stuff :) 21:12:45 Launchpad bug 1195139 in nova "vmware Hyper doesn't report hypervisor version correctly to database" [Critical,In progress] 21:12:46 Launchpad bug 1252827 in openstack-vmwareapi-team "VMWARE: Intermittent problem with stats reporting" [Critical,Confirmed] 21:13:02 I typed that up ahead of time :-) 21:13:05 lifeless: awesome 21:13:11 * melwitt raises hand after mriedem1 21:13:13 FWIW, as schedulers go, ironic would probably be involved if there's a general scheduling service, particularly if it starts being aware of failure domains and such 21:13:48 The scheduler stuff is really interesting IMHO. 21:14:06 OK, please be careful when using critical ... 21:14:13 we have weekly meetings on Tues (check the wiki for details), the more the merrier 21:14:15 Failure domain awareness is one of the motivating use case for my work on holistic scheduling. 21:14:25 hartsocks: +1 21:14:28 We really want these two to go back soon. 21:14:36 back port to Havana that is. 21:15:17 Basically our driver kills things if you're on Postgres… and the other bug is killing our CI. :-) 21:15:34 OK, fixes up? 21:15:51 yep. or very soon on the way. 21:15:58 The DB thing is up now. 21:16:18 ok cool 21:16:18 anything else? 21:16:27 that's my pleading for today. 21:16:32 :-) 21:16:38 k! 21:16:42 mriedem1: hi! 21:16:43 zomg zomg zomg 21:16:44 for powervm we were requesting some core eyes on a review: https://review.openstack.org/#/c/57774/ 21:17:01 I call dibs on +A on that 21:17:15 questions? 21:17:18 heh 21:17:41 ha, so we should probably tell dperaza 21:17:42 can you please add a link to that review to the ReleaseNotes/Icehouse wiki page? 21:17:44 i think that's the one 21:17:48 he just pushed a change for config drive 21:17:53 fail. 21:17:53 sure 21:17:53 mriedem1: I just -2d that :D 21:17:56 haha 21:18:10 here now 21:18:14 reading 21:18:14 mriedem1: yeah I just saw that. I no longer need at ask my question either :) 21:18:15 well hello there 21:18:25 dperaza: https://review.openstack.org/#/c/57774/ 21:18:35 * russellb fine with it if you guys are 21:18:39 yup 21:18:41 i'll update the wiki 21:18:43 easy enough 21:18:46 I'm more than fine with it 21:19:01 should we apply something to stable/havana, like a deprecation log message or something? 21:19:06 just ... in case? 21:19:15 I would ask questions about deprecation windows but? 21:19:27 russellb: we can if someone wants us to 21:19:28 russellb: for the principle, yeah, but.. 21:19:37 heh, k 21:19:46 * mrodden was wondering about that too 21:19:58 to my knowledge there aren't any external users of it 21:20:03 if it's not possible that anyone would be using it, meh 21:20:14 what about internal users you don't know about? 21:20:19 anyway, just stuff to consider, no rush 21:20:20 it's us i think 21:20:26 k 21:20:29 i don't need to get into the internal drama here 21:20:37 it's fun, but not here 21:20:38 melwitt: hello! have some novaclient info? 21:20:44 k 21:20:57 I have our weekly report. :) 21:20:57 105 bugs open !(fix released), 78 bugs open !(fix committed), 35 new bugs, no high priority bugs 21:20:57 10 patches up and being actively reviewed and updated or WIP 21:20:57 and I just started going through cleaning the bug list, trying to reproduce the issues and closing old bugs that have since been fixed 21:20:57 that's about it 21:21:24 awesome! 21:21:33 really appreciate the bug list cleanup, long overdue 21:21:41 if anyone wants to help, please talk to melwitt ! 21:21:49 a good opportunity to contribute :) 21:22:07 probably some good bug fix opportunities in that list 21:22:39 one last thing for sub-teams, since a lot are aronud compute drivers 21:22:53 please see dansmith's post and nice wiki page on the driver deprecation plan ... http://lists.openstack.org/pipermail/openstack-dev/2013-November/019355.html 21:23:11 we've talked about how we needed to write down more details about the driver testing requirement and deprecation plan, so there it is 21:23:50 so let us know if you have any questions on any of it! 21:24:04 i think everyone is well aware of the requirements at this point 21:24:21 #topic bugs 21:24:26 lifeless: hello, bug czar! 21:24:35 https://etherpad.openstack.org/p/nova-bug-triage 21:24:46 is my current thoughts 21:25:19 I've now basically finished my massive set of HP paper reviews so I have bandwidth to push this forward 21:25:53 lifeless: so we talked about this a bit, and i added to the agenda, but wondering about how to handle CI/gate breaker bugs special? 21:25:54 https://wiki.openstack.org/wiki/Meetings/Nova 21:25:57 next week I hope to have some metrics, and if we have consensus on the desired properties I will get onto the larger discussion about the exact process to achieve them 21:26:01 i think we should work that into your etherpad somehow 21:26:19 jog0 should have input there 21:26:22 mriedem1: jeblair's email is what I think the answer is: qa get to say 'X is critical' in any gate related project. 21:26:36 lifeless: yup, his email was good timing 21:26:42 if it breaks the gate, why not revert? 21:26:53 it's not always obvious what broke it 21:26:54 jaybuff: sometimes finding what to revert is the problem 21:26:59 jaybuff: thats certainly an option, where feasible. 21:26:59 if it was obvious, it wouldn't have made it in 21:27:07 i see 21:27:12 jaybuff: and folk do try that. The point in the bug context isn't /how/ the bug is fixed. 21:27:22 jaybuff: its 'is such a thing critical'- yes it is. and 21:27:29 jaybuff: how do we get folk working on it quickly 21:27:52 essentially crisis management 21:28:35 mriedem1: So, I don't have a view [yet] on specific tags, but I think having a single tag for recheck issues [which such things are by definition] across all of openstack is desirable 21:28:35 mriedem1: I agree with everything said 21:28:50 i think jog0 did a great job this week of raising awareness 21:29:02 which is a big part of getting people working on it 21:29:06 so when gate starts failing, the first step is to find a fingerprint for the bug ideally that will be close to the root cause, but not always 21:29:10 raising awareness, and tracking all the moving pieces 21:29:15 +1 21:29:20 agreed 21:29:22 anyhow, thats all I have right now 21:29:29 lifeless: cool thanks :) 21:29:41 then we can actaully see how frequent it occurs. And if its bad we mark as critical 21:29:41 such as https://bugs.launchpad.net/nova/+bug/1251920 21:29:41 which isn't fixed, we just disabled the test 21:29:44 Launchpad bug 1251920 in nova "Tempest failures due to failure to return console logs from an instance" [Critical,In progress] 21:29:49 so mikal has some more work on that one i THink 21:29:51 as we all do 21:29:56 jog0: sigh. 21:30:19 going forward we need to be more aggressive about fixing gate bugs 21:30:21 that bug is annoying. 21:30:22 so we never wedge the gate again 21:30:46 jog0: thoughts on how we do that? 21:30:54 if we have a gate critical bug, just lock the gate? 21:30:56 I did look at that test a little, the one that fails was typically running after the test that was skipped, they run in the same test runner and use the same VM 21:30:59 that's a brute force solution.. 21:31:20 fyi for whoever ends up actually looking at it. I believe it is an inter test interaction due to test orderign and shared resources 21:31:46 russellb: well we want to run all gate jobs x times where x > 1 21:32:00 but also what jim said in his email 21:32:20 just do a better job of making sure critical bugs get priority (which why its important to be careful with the critical tag) 21:32:30 clarkb: interesting 21:32:36 jog0: +1 21:33:14 +1 to using critical sparingly 21:33:42 also any critical gate issue should be assigned to the next milestone 21:33:52 at least thats the idea I am going with now 21:34:05 to help with visibility (there may be a better answer to that though) 21:34:27 jog0: so openstackrecheck reports to irc when a patch fails due to a recognized bug - could it also report how many times it's seen that failure? 21:34:49 so we are working on a summery of what happend, in short two bugs took us from unstable gate to wedged in a few days 21:35:05 cyeoh: we have that but hidden a bit status.openstack.org/elastic-recheck/ 21:35:21 jog0: thx 21:35:25 we have so many jobs now, a small error rate on each adds up to a huge error rate I guess 21:35:49 right, and with 50 things trying to go in at once, a failure can be really disruptive 21:35:58 and cause a ton of wasted test time 21:35:59 bingo 21:36:09 also we are exploring killing 'reverify no bug' 21:36:13 as that got us here 21:36:16 jog0: nice 21:36:18 +1 21:36:18 Yeah, I've only worked on that one bug this week 21:36:22 Its been a huge pain in the ass 21:36:23 +1 21:36:26 also we are still in critical bug fix only mode 21:36:30 so no +A yet 21:36:41 getting close to being out ofthis though 21:36:54 mikal: that was a bitch of a bug, and we still don't understand it 21:37:18 but thinking that it might be test interaction? 21:37:28 clarkb's comment a bit ago was very interesting 21:37:30 russellb: all signs point to yes 21:37:31 a good hint... 21:37:41 jog0: I like killing reverify no bug, and recheck no bug 21:37:49 at least we end up classifying things 21:37:50 i do too 21:37:55 it's abused way too much 21:38:02 will be annoying for a few types of failures, but worth it overall i think 21:38:02 which gets us closer to figuring out if they are critical, etc 21:38:07 anyway I don't want to derail the nova meeting, more info coming soon 21:38:16 jog0: ok, thanks a bunch 21:38:28 everyone give jog0 a high five 21:38:30 jog0: ^5 21:38:32 :) 21:38:34 I would personally like to thank mikal for banging his head against the wall for way too long 21:38:38 * bnemec high fives jog0 21:38:39 and mikal! 21:38:41 mikal: ^5 21:38:43 jog0: you're welcome 21:38:48 me ^5 21:38:52 * mspreitz ^5 21:38:54 * bnemec high fives mikal 21:39:04 \o/ to all 21:39:07 jog0, mikal: ^5 21:39:10 * bnemec is starting to feel like a cheerleader 21:39:14 * johnthetubaguy jog0 mikal ^6 21:39:17 be careful with the double high fives; you might fall over 21:39:20 johnthetubaguy: ooh 21:39:22 bnemec: the skirt might have something to do with that 21:39:29 alright, next topic :) 21:39:31 #topic blueprints 21:39:41 * mriedem1 raises hand 21:39:44 http://lists.openstack.org/pipermail/openstack-dev/2013-November/020041.html 21:39:49 we made some really good progress in the last week getting icehouse-1 in shape 21:39:54 roadmap wise 21:40:02 i suspect a bunch won't make it that's there though 21:40:10 merges need to be in by 1.5 weeks fro mnow 21:40:17 and that includes a long US holiday weekend ... 21:40:22 and the gate issues ... so ... 21:40:34 throws a wrench in things 21:40:42 please move things to icehouse-2 that you know won't make it, that will save some paperwork as we approach the milestone 21:40:57 also, here's my latest email to the ML on the Icehouse roadmap status in general 21:40:59 #link http://lists.openstack.org/pipermail/openstack-dev/2013-November/019786.html 21:41:04 i short: icehouse-1 looking good 21:41:06 in* 21:41:16 nova-core folks need to start sponsoring some blueprints 21:41:27 and nova-drivers, we still have some work to do for icehouse-2 and 3 21:41:39 lots of work now early in the cycle, but should calm down in a few weeks 21:41:56 https://blueprints.launchpad.net/nova/icehouse 21:41:56 #link https://blueprints.launchpad.net/nova/icehouse 21:41:58 that's the full icehouse list 21:41:58 90 blueprints earlier today 21:42:10 OH, i think a bunch are missing still too 21:42:20 for things we talked about summit ... so get your blueprints posted :) 21:42:29 i think that's all i had ... now on to specific ones! 21:42:34 mriedem1: ok now you can go 21:42:46 russellb: was just going to point out some relate bps, 21:42:52 about supporting v2 APIs for cinder and glance 21:42:55 #link http://lists.openstack.org/pipermail/openstack-dev/2013-November/020041.html 21:43:05 i wanted to raise attention since the ML thread kind of died 21:43:19 but now there is a patch to do the same for cinder v2 so bringing it back up 21:43:26 since you guys are reviewing blueprints 21:43:53 OK 21:44:00 so, trying to remember where we left that one ... 21:44:08 deployers still want a knob to force it to one or the other 21:44:21 right, but there were thoughts on discovery 21:44:25 in the absense of config, we want some automatic version discovery 21:44:26 and the keystone service catalog 21:44:41 essentially can the service catalog abstract a lot of this garbage from nova 21:44:44 i think the catalog should be to discover the endpoint, not the versions it has 21:44:55 and then you talk to the endpoint to do discovery 21:45:15 and cache it so you don't double up on API requests for everything 21:45:19 IMO 21:45:42 yeah, cache on startup 21:45:48 russellb: +1 it seems a good end goal 21:45:57 cool 21:46:00 i just wasn't sure if jog0 was working something else related to the service catalog that interacted with this 21:46:02 russellb: agreed. There's been a bit of a discussion about version discovery given that catalog entries for specific versions of apis have started to appear 21:46:21 well its cached per user though, its a little tricky 21:47:16 we should make sure keystone agrees with this idea 21:47:22 we also have list of glance servers to round robin, not sure how that fits in the keystone catelog, I guess it might 21:47:28 also we cache the keystone catalog for cinder already 21:47:29 mriedem1: can you ping dolph and get him to weigh in? 21:47:30 in the context 21:47:34 mriedem1: and maybe summarize what we said here? 21:47:44 \o/ 21:47:47 for some apis we might need one version specific catalog entry temporarily as the endpoints often currently point to a specific version rather than the root 21:47:50 russellb: sure 21:48:28 cyeoh: yeah, I agree, we probably need those for the short term 21:48:28 great thanks! 21:49:30 OK, well thanks for raising awareness of this 21:49:31 mriedem1: FYI the only thing I worked on was not saving the entire keystone catalog in the context and that was a while ago 21:49:45 jog0: ok, i'll ping you in nova 21:49:54 mriedem1: ack 21:50:21 #topic open discussion 21:50:26 mriedem1: you're on tire with topics today 21:50:28 fire, too 21:50:36 my other one is open discussion 21:50:37 mriedem1: russellb: +1 for the bits about cached version discovery against unversioned endpoints in the catalog 21:50:50 russellb: so i'll defer for now 21:51:08 dolphm: woot, that was quick 21:51:19 mriedem1: open discussion time now 21:51:23 christ 21:51:28 ok, so oslo-sync 21:51:37 ML: http://lists.openstack.org/pipermail/openstack-dev/2013-November/019098.html 21:51:39 Important because of lifeless's sync for the recent gate breakage: https://review.openstack.org/#/c/57509/ and then this:https://review.openstack.org/#/c/57511/ 21:51:42 * lbragstad lurks... 21:51:42 i just hope that the version discovery mechanism is smart enough to realize when it's handed a versioned endpoint, and happily run with that 21:51:45 there seems to be some debate here 21:51:54 bnemec: lifeless ^ 21:52:00 (by calling that endpoint and doing proper discovery) 21:52:06 o/ 21:52:24 dolphm: yeah, need to handle that gracefully ... 21:52:37 Full Oslo syncs are scary right now, for sure. 21:52:39 not sure we can really tackle the oslo thing now 21:52:42 it's really a cross project issue 21:52:49 and we'd need dhellmann 21:52:57 it might be a good topic for the cross-project weekly meeting on tuesdays 21:53:03 yeah, just another timely thing for the gate issues this week and the ML thread 21:53:06 * russellb nods 21:53:13 definitely important, but bigger than nova i think 21:53:18 totally 21:53:22 I've been trying to slowly sync the things I maintain 21:53:31 I feel its mostly a maintainter thing 21:53:40 last full sync was march 21:53:41 And random people shouldn't be doing super syncs they don't know about 21:53:42 would like to hear what dhellmann thinks is the future here 21:53:45 mikal: yep 21:53:48 and aggressively moving toward libs 21:53:50 russellb: agreed, good tuesday chat 21:54:09 russellb: I feel the interface stability thing is still a good argument for the current way for some things 21:54:09 I think we should not be syncing at all 21:54:12 oslo should be a library[6~ 21:54:16 russellb: libs would be just as bad for changed interfaces 21:54:21 we can solve the interface stability concerns 21:54:22 for the tuesday chat, please ask ttx to put it on the agenda 21:54:46 i'll take that TODO 21:54:49 since i raised it 21:54:52 When is this on Tuesday? I want to learn what's going on there… I'm confused on OSLO. 21:54:53 perfect 21:54:54 lifeless: it would be exactly the same work, just a different mechanism 21:55:02 mikal: no, it wouldn't 21:55:11 hartsocks: we've had a project release status meeting at the same time as this meeting for a long time 21:55:26 but we've changed the format, to be more discussion based for cross-project topics 21:55:28 russellb: okay, I'll lurk that one. 21:55:31 instead of weekly release status syncing 21:55:34 yeah it should be a good one 21:55:44 if you're interested in cross project issues anyway 21:55:57 which everyone should be to some degree :) 21:56:00 recently becoming a problem for us... 21:56:05 * bnemec adds it to his calendar 21:56:07 :-) 21:56:31 i'll be doing the release status sync with ttx in a 1-1 chat 21:56:34 but in a public channel 21:56:45 As performance art 21:56:45 it's 1515 UTC tuesday 21:56:48 if anyone cares :) 21:57:01 right now in #openstack-dev 21:57:26 we go through bugs/blueprint status for the upcoming milestone, and the release roadmap (icehouse) in general 21:57:33 is this the link? https://wiki.openstack.org/wiki/Meetings#OpenStack_Project_.26_Release_Status_meeting 21:57:34 mikal: I will be happy to discuss with you later about this 21:57:35 make sure we have movement on the highest priority things, etc 21:57:40 mriedem1: yes 21:57:50 looks like the wiki has the wrong time 21:58:00 lifeless: sure, more than happy to, but pleae try to understand the historical reasons for the way things are first before assuming we're all wrong 21:58:14 mikal: I was here when the decision was made. 21:58:22 mikal: I know the historical reason :) 21:58:50 mriedem1: why is the time wrong? says 2100 UTC, same as this meeting 21:59:06 russellb: thought you just said it's 1515 21:59:09 i must be confused 21:59:19 oh, no, that's my weekly nova status sync 21:59:25 ah, ok 21:59:26 the above 2100 UTC meeting is the cross-project meeting 21:59:29 yeah 22:00:05 times up 22:00:10 this has been fun 22:00:12 alright, time up for today! 22:00:18 thanks everyone! 22:00:22 #endmeeting