21:00:49 #startmeeting nova 21:00:50 Meeting started Thu Jun 19 21:00:49 2014 UTC and is due to finish in 60 minutes. The chair is mikal. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:00:52 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:00:55 The meeting name has been set to 'nova' 21:00:58 o/ 21:01:00 So, who is around? 21:01:02 o/ 21:01:03 hi 21:01:04 again 21:01:05 hi 21:01:06 hi 21:01:07 o/ 21:01:12 o/ 21:01:12 o/ 21:01:14 o/ 21:01:23 o/ 21:01:28 o/ 21:01:36 o/ 21:01:47 Ok, so let's get started then 21:01:52 o/ 21:01:54 #topic Juno-2 21:02:01 #info Juno-2 is 24 July 21:02:13 o/ 21:02:16 Which is just before our midcycle meetup 21:02:30 #info Proposed spec review day on 26 June 21:02:43 John and I like the idea of trying to do a spec review day on 26 June 21:02:56 People hang out in openstack-nova, and we try to iterate on specs faster than we have so far 21:03:09 * russellb out that day on PTO 21:03:18 We think this is important because we need to get specs approved before very early July if they're going to make it into Juno 21:03:23 PTO? 21:03:28 paid time off 21:03:29 Is that American for vacation? 21:03:31 yes 21:03:38 only some americans :) 21:03:41 Fair enough 21:03:52 So... Is there another day that week that works better for people? 21:04:14 John and I are thinking of a proposal freeze for specs of July 2, so we need to do this before that 21:04:37 25 or 27 are better than 26 of June for me 21:04:39 With the end goal being to try and prioritize spec implementations for review by July 10ish 21:04:46 (These dates to be announced in email soon) 21:05:05 * mikal looks up what days of the week those are 21:05:14 So, Wednesday or Friday work better for people? 21:05:26 I think Friday is bad because its Saturday for Asian devs 21:05:27 friday is a bad idea right 21:05:28 wednesday 21:05:45 Yeah, I think Wednesday might be the winner there 21:05:49 hump day for reviews 21:05:54 #info Spec review day on 25 June, in #openstack-nova 21:06:48 So yeah, just wanting to reinforce the general plan: we do a review day to clean up; have a proposal freeze soon after; and then triage things so we have a prioritized list for review as things land 21:07:01 Any concerns with that plan? 21:07:20 is there a prioritized list of specs that should get hit 21:07:30 there is such a large wall of them at the moment 21:07:35 sdague: not at this point (I assume you mean for the review day) 21:07:42 yeh 21:07:42 right, we do it that day, right? 21:07:47 that's part of the review we need to do I think 21:07:49 24 June for spec triage day? 21:07:54 sounds good 21:08:01 Well, the day is also quite long 21:08:07 i.e. its the night before you guys for me 21:08:18 So early people could have a go at prioritizing as part of reviews 21:08:22 We can track that in an etherpad 21:08:49 But I can pre-create that etherpad so people can try doing that now if they want 21:08:58 the NFV sub-team would love for us to prioritize the specs tracked here: https://wiki.openstack.org/w/index.php?title=Teams/NFV 21:08:58 :) 21:09:06 #action mikal to create etherpad for spec review day where people can try and pre-prioritize 21:09:16 russellb: heh, I'm going to make you talk about nfv in subteams 21:09:24 if you want to review stuff that at least some group this is important 21:09:30 thinks* 21:09:31 I do think prioritization has to include reasons 21:09:43 Not "cause my thing!" 21:09:59 Ok, it sounds like we can move on from here 21:10:21 #topic Bugs 21:10:26 So, we still have a lot of bugs 21:10:35 jogo has some interesting ideas about an initial automated cleanup 21:10:39 jogo: want to talk about that? 21:10:48 mikal: yeah 21:10:53 so this isn't automated cleanup per se 21:11:02 so we have been getting better at doing the initial bug triage 21:11:11 but we are still really bad at following up with existing bugs 21:11:18 https://etherpad.openstack.org/p/eEYO2Fdsuv 21:11:41 so based on a previous meeting I took a look at what infra uses (thanks pleia2) and added a few things to it 21:11:49 http://paste.openstack.org/show/84532 21:12:04 ^ is a link to the first 300 or so bugs (sorted by priority) 21:12:15 generated with https://github.com/jogo/openstack-infra-scripts 21:12:28 So the general idea is to unassign idle bugs, make status match reality, etc. Yes? 21:12:39 with that its fairly easy to follow up on bug statuses and fix them 21:12:41 mikal: correct 21:12:42 i.e. in progress with abandoned patch != in progress 21:12:49 mikal: bingo 21:12:57 yeah the first on that list all patches are merged but it's marked as 'confirmed'? 21:12:58 Cool 21:13:01 mikal: and confirmed ith merged patch may be fixed as well 21:13:26 mriedem: I just fixed that one a few minutes ago in fact 21:13:32 So, if some people want to volunteer to have a grind through some of those that would be appreciated 21:13:39 mikal: yeah 21:13:48 I was hoping the existing bug team can use this 21:13:53 tjones: ^ 21:13:55 Noting that stackalytics now tracks bug closes as contributions, which I am hoping will help too 21:14:00 yeah thanks jogo 21:14:26 I think we should also be reminding people that when they close a bug they should do a quick search for duplicates, or very similar bugs 21:14:29 so paste cut off my list but I have los for about 1k bugs 21:14:38 I've started just picking keywords and searching as I work on a bug 21:14:40 I have a hunch we can close a few hundred of those easily 21:14:45 ooooooooooooooo 21:14:46 And its surprising how often we have very similar bugs 21:15:37 I think this is also a pretty obvious topic for the mid cycle meetup... May be we should carve out some time to try and grid through fixing states etc 21:15:44 We'll wait and see how bad things are at that point I suppose 21:15:59 #info jogo has a script which helps find bugs in inconsistent states 21:16:02 ++ to doing that at mid cycle, but we don't have to wait until then to start working 21:16:12 #link https://etherpad.openstack.org/p/eEYO2Fdsuv 21:16:13 tjones: lets sync after the meeting 21:16:15 jogo: lets sync 21:16:17 lol 21:16:24 i was just saying that 21:16:27 Yes, I very much would like to see some of this done before the mid cycle 21:16:34 Mid cycle is a very expensive way of working on bug state 21:16:37 But we'll do it if we have to 21:16:53 If people feel that bribes would help, then I'd be interested in suggestions 21:16:59 I could buy mars bars for people or something 21:17:06 * mikal flails at incentives 21:17:22 tjones: any interesting bugs for us this week? 21:18:02 re bribes: find a way prioritize review inbox by committer's karma in LP? 21:18:11 yes 2 citical bugs that are not assigned 21:18:16 i think 1 just got fixed 21:18:20 https://bugs.launchpad.net/nova/+bug/1275500 21:18:25 Launchpad bug 1275500 in nova "Cannot reboot instance: Network filter not found: Could not find filter" [Critical,Confirmed] 21:18:48 https://bugs.launchpad.net/nova/+bug/1257626 21:18:50 Launchpad bug 1257626 in nova "Timeout while waiting on RPC response - topic: "network", RPC method: "allocate_for_instance" info: """ [Critical,Invalid] 21:18:52 +1 on bug karma == more reviews 21:19:01 both affecting gate 21:19:20 tjones: 1257626 re-open didn't appear to be valid 21:19:50 I also think that https://bugs.launchpad.net/nova/+bug/1313477 looks interesting 21:19:51 Launchpad bug 1313477 in nova "libvirt driver hang with genisoimage when boot new instance" [High,Confirmed] 21:19:56 (Possible eventlet bug) 21:20:14 And https://bugs.launchpad.net/nova/+bug/1316621 21:20:16 Launchpad bug 1316621 in nova "ebtables calls can race with libvirt" [Medium,Confirmed] 21:20:22 (Race condition with ebtables and libvirt) 21:20:39 Those two are less urgent, but look interesting 21:20:49 jogo: 1257626 - it's probably the general screen race, I've got a retry loop to see if that helps 21:21:05 sdague: ahh that makes sense 21:21:15 yeah was gonna say that sounded like the grenade one that is failing 25% of the time 21:21:17 also we've been trying to triage the untriaged bugs in our meeting. it's really slow going especially since the subject matter experts may not be around 21:21:41 jogo: oh, wait. for large-ops, it shouldn't be using screen. 21:21:46 tjones: is it worth reminding people in nova IRC at the start of that meeting that its on? 21:21:50 tjones: or are you already doing that? 21:21:52 the ares with the mostr untriaged bugs are novaclient, network, testing (surprse), and compute 21:22:10 I do that 21:22:16 OK cool 21:22:34 oh and api 21:22:44 So, we should help tjones more, and fix lots of bugs. Kthxbye. 21:22:50 lol 21:23:00 those tags are super broad, except novaclient 21:23:08 testing is basically anything that fails a compute test in tempest 21:23:13 of which there are many 21:23:24 So, we should probably move on unless there's anything else we can achieve with bugs right now 21:23:25 network could be nova-network or neutronapi 21:23:37 arosen is about the only person that fixes neturonapi bugs 21:23:46 yes - ideas for helping make this bettter? 21:24:10 not really, more specific tags for people to ignore :) 21:24:18 Heh 21:24:20 lol 21:24:29 move on 21:24:42 tjones: perhaps emailing out lists of untriaged bugs? 21:24:50 sure 21:25:09 here is the link by tag https://wiki.openstack.org/wiki/Nova/BugTriage 21:25:11 including a short list of untriaged ones 21:25:14 link bait gets me 21:25:26 like, if i could just click one to triage, and at least have contributed that one 21:25:48 that has a link to untraged bugs per area 21:26:20 i'll send out something to the ML 21:28:58 next topic? 21:29:36 Sorry 21:29:38 DSL dropped 21:29:49 #topic Gate status 21:29:59 So, my understanding is we can now approve things like crazy again 21:30:05 already have been 21:30:08 sdague: is that a fair summary of the state of play? 21:30:08 mikal: yes sir 21:30:16 mikal: ++ 21:30:21 lets break some records https://github.com/openstack/openstack/graphs/commit-activity 21:30:22 Yeah, I guess I'm more after "do you still need help?" 21:30:33 you have 2.5 hrs to try to land another 30 patches to make it another > 100 merge day 21:30:35 fyi that people will probably be rechecking for httpretty failures https://bugs.launchpad.net/openstack-ci/+bug/1332266 21:30:36 Launchpad bug 1332266 in openstack-ci "httpretty 0.8.1 fails to install, causing job failure" [Undecided,Confirmed] 21:30:38 It would be nice to be fixing things so we don't end up wedged again 21:30:58 mriedem: only novaclient, right? 21:31:06 sdague: was more than novaclient in logstash 21:31:13 well, the clients in general 21:31:28 that's where it mostly was being pulled in 21:31:30 Ok, but if people see that failure they can just recheck with that bug number, so its not the end of the world 21:31:31 as for gate status, i still have suspicions about ec2 tests and boto connection pools causing issues with nova-network 21:31:49 the one ssh timeout bug in scenario tests 21:31:49 We have a little more logging in nova-network now, but I think we need more 21:31:54 linux_net specifically could do with more 21:32:21 I'll try and take a look at that in the next few days 21:32:26 Unless someone beats me to it 21:32:34 Nothing else for gate right now, right? 21:32:35 https://bugs.launchpad.net/tempest/+bug/1298472 21:32:37 Launchpad bug 1298472 in tempest "SSHTimeout in tempest scenario tests using nova-network" [Critical,Fix committed] 21:32:38 that's the ssh one 21:32:45 *one of them 21:33:06 no other gate stuff 21:33:09 Ok, let's move on 21:33:16 #topic Subteam reports 21:33:23 Containers people... Got anything to say? 21:33:46 sounds like there has been really nice nova-docker progress this cycle 21:33:49 on features 21:33:55 Cool 21:33:58 ewindisch: you around? 21:34:02 * russellb not a containers person, just hearing updates 21:34:13 It would be interesting to know what their plans are for merging back 21:34:22 i.e. which milestone 21:34:38 But I think we've missed out on containers people today 21:34:56 Ironic people... How goes your nova driver? 21:35:01 devananda: ^-- 21:35:31 hi! 21:35:37 How goes your driver? 21:35:46 I know you'd like to trick us into more spec reviews? 21:35:50 stability is good. testing is good. thanks to the folks who have reviewed the spec! 21:36:06 mikal: i haven't proposed the driver code yet to nova, since the spec hasn't been approved 21:36:18 The gate job ended up not being removed, right? 21:36:21 we're continuing to improve it in our tree until then 21:36:29 the gate job was never removed -- just made non-voting briefly 21:36:32 and we'vce proposed to revert that 21:36:42 it was coupled to oslo-test which put ironic in the integrated gate 21:36:44 So its more reliable now? 21:36:56 it was reliable, outside of network issues 21:37:00 and yes, thsoe have been addressed too now 21:37:02 Ahhh, ok 21:37:14 (the failures were in downlading UCA PPA's) 21:37:14 How is progress going on getting it reporting like a third party CI in nova reviews? 21:37:25 that has stalled in infra 21:37:26 let me find the patch 21:37:32 (sorry, jugglign two meetings at once ) 21:37:38 Oh, sorry 21:37:58 https://review.openstack.org/97411 21:38:44 I can ask jhesketh to chase that with clarkb today 21:38:49 there apparently seems to have been some disconnect in what everyone thought was agreed there. I think it would be good to get mikal, devananda, jeblair into an irc room again to sort out the final bits of confusion 21:39:04 Ok, I'd be happy to do that 21:39:17 ++ 21:39:29 Can someone propose a time and place in email? 21:39:41 Anything else for ironic before we roll on? 21:39:59 mikal: aside from please review the specs -- no :) 21:40:05 So noted 21:40:12 thanks! 21:40:15 Note our spec review day next week, that's your time to iterate quickly 21:40:29 NFV team... russellb, how goes it? 21:40:32 mikal: sorry. which day? 21:40:37 o/ 21:40:39 devananda: Wednesday next week 21:40:39 #link https://wiki.openstack.org/wiki/Teams/NFV 21:40:46 so this isn't nova specific, but highly related to nova 21:40:53 i don't think i have an objection to that patch 21:40:58 ugh. i'll be on a plane // onsite ... 21:40:59 there's a couple things here, 1) what is NFV? and 2) what does that mean for us (OpenStack) ? 21:41:19 russellb: is there a list of features needed from nova forming somewhere? 21:41:20 NFV in short, is a standardization effort in the telco industry on using a cloud platform and VMs to replace hardware network applicances 21:41:23 to be more agile 21:41:28 mikal: yep, will get to that in a sec 21:41:35 Appliances like routers? 21:41:37 yes 21:41:42 sdague, devananda, mikal: ^ 21:41:43 #link https://wiki.openstack.org/wiki/Teams/NFV#What_is_NFV.3F 21:41:50 if nothing else, read our stuff on "What is NFV" 21:41:52 just 2 paragraphs 21:42:03 #info People should read the NFV wiki page 21:42:19 so, the NFV people want to standardize on OPenStack as the platform 21:42:20 which is cool. 21:42:23 and a HUGE opportunity 21:42:29 Are there any proposed specs ready for review now, or are people still frantically typing? 21:42:30 because the industry we're talking about is _really_ bug 21:42:36 Big I hope 21:42:40 big yes 21:42:51 so we want to help make sure OpenStack supports their use cases 21:42:56 and THAT is where the openstack NFV team comes in 21:43:00 Do they have a realistic timeline? 21:43:12 the wiki page includes what NFV is, use cases, and a list of development efforts in support of NFV use cases 21:43:14 I assume its going to take us a while to get all this done? 21:43:18 dev efforts: https://wiki.openstack.org/wiki/Teams/NFV#Development_Efforts 21:43:20 #link https://wiki.openstack.org/wiki/Teams/NFV#Development_Efforts 21:43:22 yes, it'll be a while 21:43:28 but there's several doable for juno, for sure 21:43:32 lots of specs on that list ready for review 21:43:33 Excellent 21:43:36 some with implementations in progress 21:43:47 I think the "what is NFV" thing is a pretty helpful contribution by itself to be honest 21:43:55 so basically, anything on that page is being requested by REALLY big potential openstack users 21:44:06 But it sounds like its relatively on track 21:44:10 yes the #1 goal of that page is to be able to point openstack devs to it 21:44:20 to get quick context on all this 21:44:24 that's one of the key deliverables of the team 21:44:30 context around NFV, and tracking the work 21:44:35 Awesome. Anything else we should know about NFV before we move on? 21:44:43 nope! 21:44:48 Yay! 21:45:02 Scheduler peoples... How goes your scheduler refactor? 21:45:35 still ongoing, not much to report yet 21:45:41 but we think we know what we are doing 21:45:43 Ok 21:45:51 The BP said it was ready for review IIRC 21:45:54 Is that correct? 21:45:57 Or is there more coding to be done? 21:46:12 There is one that should be ready now, yes 21:46:24 Ok, cool 21:46:25 there will be more work but there'll be another BP 21:46:37 Sounds like its in progress and we should get to reviews soonish 21:46:45 Anything else from scheduler peeps? 21:46:55 just a sec and I can give the links to review 21:47:21 they are: https://review.openstack.org/#/c/82778 and https://review.openstack.org/#/c/97232/ 21:47:26 Cool 21:47:35 So... What subteams did I forget? 21:47:39 vmwareapi 21:47:50 And do we like calling them out like this, or should we just pile on like we used to? 21:47:58 i like calling it out 21:48:09 I like it too, but I'm often wrong 21:48:14 :-D 21:48:25 Ok, vmwareapi. Sorry for forgetting such a lovely subteam. Report! 21:48:37 no prob. and i'll be quick as usual 21:48:46 phase 2 refactor is ready for review https://etherpad.openstack.org/p/vmware-subteam-juno 21:49:01 And phase 3 is still a work in progress, right? 21:49:23 the top lists the patches in order. reviews much appreciated! phase 3 is being worked in parallel - it will be posted very soon 21:49:37 Cool 21:49:39 Anything else? 21:49:43 nope that's it 21:49:47 Cool 21:49:54 #topic Open Discussion 21:50:00 o/ 21:50:00 We have 10 minutes for open discussion 21:50:03 Spend it wisely 21:50:10 nova networking 21:50:14 Go 21:50:32 https://bugs.launchpad.net/nova/+bug/1327406 21:50:34 Launchpad bug 1327406 in nova "The One And Only network is variously visible" [Undecided,In progress] 21:50:47 i'll make a few more notes about nova + NFV ... the requests are largely around getting most performance possible out of VMs, so all generally useful stuff as well, but critical for some telco uses 21:50:52 It looks like nova flat networking does not allow a non-administrative user to enumerate networks 21:51:13 ... which looks pretty wrong to me 21:51:27 and, of course, the doc sucks 21:51:30 that's the kind of stuff most doable for juno, so SR-IOV networking, NUMA, large pages, dedicated CPUs ... 21:52:07 I am wondering why gate doesn't object to current breakage. And what people think the right fix is. 21:52:14 And, what were they thinking anyway? 21:52:22 mspreitz: vishy has been doing a bunch of work on nova-network recently, it would be a good idea to draw this bug to his attention 21:52:46 He might have more context about the historical reasons (or lack thereof) 21:52:51 that's a different vish than I already have subscribed? 21:53:03 No, that's him 21:53:08 no, that's not him 21:53:09 But a ping on IRC might be more effective 21:53:13 wrong launchpad vish 21:53:14 Oh, ok 21:53:16 Sigh 21:53:19 for nova api, just wanting more core review on v2.1 on v3 and nova api policy - we still block on those. After those got approved, it will be a lot of patches for them, so hope we can doing on it early. 21:53:30 But I am sure he gets lots of bug email 21:53:30 OK, I'll try to alert the right vish 21:53:34 So, I'd ping him in IRC 21:53:51 angdraug: you had something as well? 21:53:53 rbd backend 21:53:55 In IRC he is "vishy" ? 21:53:58 I have a patch series to address a number of problems with rbd driver, in review since May 1 (one patch since September 2013 really) 21:54:01 https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:bug/1226351,n,z 21:54:01 mspreitz: yes 21:54:05 thanks 21:54:09 blueprint rbd-clone-image-handler, spec: https://review.openstack.org/91486 21:54:17 the first commit in the series ups RPC API, so I have to keep rebasing when other API affecting changes are merged ahead of it (4 times so far) 21:54:25 it didn't have -1 since June 11, a week ago (was addressed on June 12, still a week ago) 21:54:26 The spec is approved? 21:54:36 no, still in review 21:54:43 angdraug: can you ping pbrady (or pixelbeat on irc sometimes) ? i was talking to him today about getting more involved with ceph+nova, so he may be able to help review 21:54:44 first commit is a bugfix, the spec related commits depend on it 21:54:46 So that wont be helping 21:54:51 russellb: I think the integration testing for these feature is something interesting. 21:54:56 I will, thanks 21:55:00 yjiang5: yes 21:55:14 pixelbeat would be a great person to get involved 21:55:45 I have a patch that depends on Barbican, which is now in incubation. The Barbican wrapper spec has been approved https://review.openstack.org/#/c/94918/ 21:55:55 I would really appreciate if this could be given a priority over other API changes long enough to get merged 21:56:12 any chance of that? 21:56:36 What's the review number for the API change? 21:56:39 That's the bugfix one right? 21:56:43 russellb: We should push tempest case to upstream, but also keep them easily be tested by 3rd parties (considering so many vendor interest on it) since gate has no hardware testing. 21:56:49 91722 21:56:52 yes 21:57:11 yjiang5: *nod* ... we'll have to rely heavily on *VERY* good unit testing for the gate for now 21:57:14 the bp commits are pretty well isolated, not a lot of conflicts on rebase 21:57:18 and vendor testing of hw environments 21:57:34 angdraug: I will take a look at that review today 21:57:43 If other people could take a look too that would be great 21:57:52 angdraug: when is rbd/ceph 3rd party CI going to be a thing? :) 21:57:54 angdraug: we don't really have a process for compelling people to review something though 21:58:00 thanks. re testing, fuel team is working on running CI with master branches, we're only running stable/* for now 21:58:06 I would like to see third party CI on this though 21:58:10 ++ 21:58:18 mikal: for rbd? 21:58:20 mriedem: it's still probably a month away at list 21:58:23 at least even 21:58:26 russellb: yes 21:58:31 rbd/ceph 3rd party CI was discussed when things were pulled from FFE in icehouse 21:58:34 russellb: well, kind of for all of these things 21:58:37 when we can have fuel ci for this, it will cover rbd 21:58:39 russellb: lvm, rbd, etc etc 21:58:40 mikal: yeah, i'll go chase that a bit here 21:58:49 russellb: that would be awesome 21:59:40 mikal: do you know how many person register for the meetup till now? 22:00:08 time up! 22:00:09