21:01:23 #startmeeting nova 21:01:24 Meeting started Thu Jul 17 21:01:23 2014 UTC and is due to finish in 60 minutes. The chair is mikal. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:01:25 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:01:28 The meeting name has been set to 'nova' 21:01:28 o/ 21:01:31 o/ 21:01:31 hi 21:01:32 o/ 21:01:40 Heya 21:01:43 hi 21:01:47 So, let's get this show started 21:01:52 #topic Juno-2 21:01:57 o/ 21:02:01 #info j-2 is 24th July 21:02:02 o/ 21:02:04 Which is kind of soon 21:02:09 howdy 21:02:21 o/ 21:02:26 So, there's a few j-2 things we should quickly cover 21:02:34 #info Reviewers, please focus on j-2 targeted things if possible 21:02:40 * russellb is ttx for juno-2 next week, fyi 21:02:56 Yes, and a love ttx you are too 21:03:02 heh <3 21:03:05 The timeline for the next release is at: 21:03:13 #link http://lists.openstack.org/pipermail/openstack-dev/2014-June/038475.html 21:03:31 We also need to talk about specs who have requested a spec freeze exception 21:03:40 :) 21:03:46 * mikal waits for the page to load 21:03:57 damned NBN 21:04:06 dansmith: don't go there 21:04:18 I don't even know if you're on the NBN or waiting for it 21:04:25 No, 2mbit DSL for me 21:04:28 But anyways... 21:04:41 So, we had a list of priority specs based on the summit back in the day 21:04:44 #link https://etherpad.openstack.org/p/nova-juno-spec-priorities 21:04:53 We should use that as a guide for spec freeze exceptions 21:05:08 But nova-drivers can work through that in the spec repo 21:05:10 o/ 21:05:28 Its the first time we've done specs, so I imagine the spec freeze process will involve a little discussion about the process as well as the specs themselves 21:05:37 multi-attach isn't in that list unfortunately 21:05:45 shall we cover requests now? 21:05:55 the -2 comment on specs said we would take requests in the meeting 21:05:56 So, I'm only aware of two 21:06:00 But more might have come up overnight 21:06:03 I just woke up 21:06:05 russellb: sure, go for it 21:06:08 mikal: see "Juno Spec Freeze" section of that etherpad 21:06:20 lots listed as potential, some with sponsors 21:06:30 earlier today here, i was proposing that we require core review sponsors to get something approved 21:06:32 Ugh, that's quite a list 21:06:54 so, if something has -core that cares enough to promise review effort on it, that should be strongly considered for an exception 21:07:00 I was hoping to add multi-attach to the list if possible ? 21:07:03 unless we think it's just way too disruptive to merge this late no matter what 21:07:15 that's how i'd think about it, i think 21:07:58 russellb: I think that's wise. Two cores as sponsors would be even better because then we can point the finger at them for reviews 21:08:05 mikal: indeed 21:08:17 hemna: have you sent an email to openstack-dev requesting one? 21:08:20 so, require at least 2 sponsors, and no major vetos saying it's too disruptive to do late in the cycle? 21:08:23 as the criteria? 21:08:32 hemna: I think we should ask people to make a (brief) case on the mailng list, and then we can take it from there 21:08:34 mikal, not currently. I can do that though. 21:08:35 and if so, maybe do a pass on the ones that have at least 2 already listed now 21:08:41 russellb: that sounds reasonable to me 21:08:44 cool 21:08:47 hemna: please do, ASAP please 21:08:51 will do 21:08:55 so, first one with at least 2 sponsors is 21:08:57 #link https://review.openstack.org/#/c/95025/9/specs/juno/deprecate-baremetal-driver.rst 21:09:03 I am sure we'll tweak the plan as we learn how to make it work, but that's a good start 21:09:06 john, dansmith, and alaski on that one 21:09:08 mikal: I would like to have a exception too 21:09:28 mtanino: email to openstack-dev please? 21:09:32 thoughts on deprecate-baremetal-driver? 21:09:36 mikal: not yet. So I will send a mail too. 21:09:43 We need ironic first for that one 21:09:44 actually 21:09:52 devananda is going to try to join us to talk about that 21:09:57 mikal: so the conversation earlier today was that some folks dont want to acept ironic w/o the deprecation 21:09:59 yeah, no.. I think this should have been part of the ironic spec, 21:10:01 hi! 21:10:01 devananda: o/ 21:10:04 so I think it needs to be tied to the other one 21:10:17 agree with dansmith 21:10:26 but, I also think that it's in reasonable shape so I'm not too worried 21:10:28 devananda: I thought deprecation was part of the plan from the summit, is that not your recollection? 21:10:29 i think being able to deprecate baremetal needs to be tied to merging the ironic driver 21:10:32 just don't want to drop it because it's after the deadline 21:10:45 agreed 21:10:54 i haven't reviewed the spec, but i view it as ironic drive + this, or neither 21:10:56 mikal: yes, both important, and both should happen 21:11:00 devananda: I also believe that upgrade is progressed, but I haven't heard much about it? 21:11:10 there is code proposed for the upgrade 21:11:12 but not yet any testing of it 21:11:20 In your repo or outs? 21:11:22 ours even 21:11:22 yours 21:11:33 Cool 21:11:48 https://review.openstack.org/#/c/101920/ and https://review.openstack.org/#/c/102563/ 21:11:49 I think this is complicated enough that perhaps we should ask for an email describing its status 21:11:51 anyone *not* think this should get an exception? 21:11:57 What's implemented, where its proposed, next steps, etc etc 21:12:01 mikal: want to defer approvals to ML? 21:12:07 or try to sort some now? 21:12:09 russellb: this is a big list to get through 21:12:20 well, was thinking we'd just start with ones that have sponsors listed already 21:12:21 i'm happy to give an email update, but i'd like to sort one question real quick 21:12:25 Let's cherry pick out the most important / obvious ones now and do the rest on the ML 21:12:26 up to you 21:12:29 k 21:12:36 do we (and if so ,how) need grenade testing of the upgrade before we can *land* the driver? 21:12:46 I think we do 21:12:57 that seems like another chiken and egg problem which took us a few days at the ATL summit to detangle 21:13:04 How do we do that if baremetal never supported devstack? 21:13:08 right 21:13:20 there is *no* testing of baremetal right now 21:13:23 so how do we test upgrades from it? 21:13:30 hrm, fair point ... 21:13:37 * dansmith headdesks 21:13:39 Adding baremetal devstack is a pretty big tangent 21:13:40 Hi, can you please consider https://review.openstack.org/#/c/99615/ too? It was submitted in June and looked fine but was not approved. 21:13:46 And probably not something juno has time for 21:13:55 maybe for now we agree this spec should be approved, but let's sort out testing details on ML ? 21:13:57 so, the things we need to test are pretty small, right? 21:14:11 Yeah, let's be brutal here 21:14:17 We're inclined to grant an exception 21:14:28 But we want to see a status email on -dev, and to have a quick discussion about upgrade testing first 21:14:33 dansmith: can baremeteal spin up some instances is the first criteria before we can even do a grenade test of an upgrade 21:14:40 We will then talk about that exception in that thread 21:14:41 dansmith: i have no confidence in the first criteria today 21:14:42 #agreed exception granted for deprecate-baremetal-driver, pending agreement on test requirements 21:14:49 degwea: request an exception on the -dev mailing list 21:15:13 devananda: tripleo uses baremetal though right? 21:15:15 degwea: see above, the process is 1) get email to openstack-dev; 2) add to https://etherpad.openstack.org/p/nova-juno-spec-priorities; 3) get core reviewers to sponsor 21:15:18 devananda: so we know it works at least a little 21:15:22 now the train starts moving ... i'm gonna drop on and offline a bunch 21:15:39 devananda: can you start a thread about test requirements? 21:15:43 #info mikal will send an email about how to request an exception to -dev after this meeting 21:15:44 OK,if I do it tomorrow will it still be consider? Thx 21:15:52 mikal: yes, in tripleo. but it's not tested in devstack 21:15:52 russe ack. feel free to #action me 21:16:02 degwea: that will be fine, but it will need to be tomorrow 21:16:08 hi, this BP can be considered? https://review.openstack.org/#/c/102201/ I think sajeesh can explain it better 21:16:09 #action devananda to start a thread on openstack-dev to nail down test expectations for baremetal -> ironic upgrade 21:16:20 yes 21:16:20 so, the next that has 2 or more sponsors is https://review.openstack.org/#/c/84887/16/specs/juno/use-glance-v2-api.rst 21:16:24 i think this is in the obvious category 21:16:25 raildo: see above 21:16:30 Yeah, approved 21:16:34 Sure it will be tomorrow Thanks 21:16:38 mriedem: ok 21:16:38 Track that in the etherpad? 21:16:39 #agreed glance v2 support exception approved 21:16:45 Or that! 21:16:46 great thanks russellb 21:17:02 Similarly, where did we end up with the cinder API spec? 21:17:08 mikal: it's under review 21:17:10 the code that is 21:17:13 the spec was approved 21:17:14 thought that one was approved already 21:17:14 awhile ago 21:17:15 Oh, good 21:17:20 thingee: ^ 21:17:21 ah right 21:17:34 https://review.openstack.org/#/c/91486/6/specs/juno/rbd-clone-image-handler.rst 21:17:34 I will admit to not having all 60 something approved specs in my head 21:17:37 is the next one with sponsors 21:17:41 has 4 listed, overkill, heh 21:17:57 I guess the other thing we're asking... 21:17:59 cinder api code is just about done. there is one test still failing 21:18:01 i think we have to dig into the rbd test failures a bit 21:18:02 at least 2 people have told me they've already looked over the code and think it's reasonable to land very soon 21:18:02 Do we think that spec could be merged today? 21:18:04 cinder v2 api support* 21:18:06 How close is the spec itself? 21:18:36 mriedem: jaypipes already has a fix in testing for that 21:18:37 mriedem: well the good news is that there weren't *that* many failures 21:18:38 mikal: the rbd specs says 'testing? nah" 21:18:45 yeah 21:18:46 mriedem: le sigh 21:18:47 mriedem: yeah but that's in progress now 21:18:49 there is an inconsistency between v1 and v2 in schema type failing. I'll have to update the tempest test to know about both unfortunately. 21:18:51 So... 21:18:53 spec out of date on the test front 21:18:58 If we approve an exception, it has to be timeboxed, right? 21:19:00 and i'm tracking debugging / fixing test failures 21:19:07 i.e. We will keep reviewing that spec for say a week or something 21:19:15 If we can't agree within a week, its still out? 21:19:22 mikal: agreed 21:19:23 i think the spec's testing section just needs updating 21:19:28 What do we think is a reasonable timeout period? 21:19:33 mikal: was assuming we generally were ready to approve the spec if we give an exception 21:19:38 a week is generous IMO 21:19:40 then the spec is ok, but we can't land the code w/o the test issues resolved 21:20:04 resolved = bugs with possible skips if using rbd maybe 21:20:05 So in this case, we'd be giving a week for someone to come up with a _plan_ for testing that we like 21:20:10 (Not nessesarily an implementation) 21:20:17 mikal: i'm all over it 21:20:21 mikal: russellb has a devstack patch running with rbd 21:20:24 status: https://etherpad.openstack.org/p/LJj1gHesRG 21:20:26 14 failures, might not all be rbd 21:20:30 russellb: sure, but I'm trying to set precident for future exceptions 21:20:34 some are definitely rbd issues 21:20:49 * russellb nods 21:20:52 i'm still not sure how we're going to test glance and cinder v2 21:21:01 or keystone v3 for that matter 21:21:07 #info a spec exception means that you have a week from the exception being granted to reach a concensus and get the spec merged. If that doesn't happen, then exception expires. 21:21:20 #action russellb will update rbd spec's testing section to reflect progress on ceph testing in the gate 21:21:28 can we also look at https://review.openstack.org/#/c/84695/14/specs/juno/v2-on-v3-api.rst (v2.1 on v3) - only has one core sponsor, but I'm pretty sure oomchi would have sponsored too if he knew about the request for core sponsors 21:21:34 Ok, so it sounds like rbd is a spec exception winner 21:21:37 agree on an exception for that one, once the test section is updated? 21:21:39 ok cool 21:21:49 would you guys mind removing the -2 on https://review.openstack.org/#/c/84887/ ? 21:21:58 #agree exception approved for rbd spec, pending update of the test section to reflect gate test progress 21:22:10 arnaud__: i think we already did talk about it, else ML 21:22:22 mikal: that's all that had 2 people confirmed already 21:22:22 Let's go backwards and talk v3 API for a second 21:22:34 yeah, that's what i was going to mention next 21:22:36 I just sneakily added my name there to get it over the line 21:22:38 mriedem, we did, but that doesn't encourage people to review if there is a -2 21:22:43 ah there it is heh 21:22:55 I would like to see us have a really good try at reaching a concensus on that one in the next week 21:22:55 #link https://review.openstack.org/#/c/84695/14/specs/juno/v2-on-v3-api.rst 21:23:02 I think we're painfully close 21:23:02 seems reasonable to me 21:23:05 cyeoh: ^-- 21:23:06 arnaud__: i'd send something to ML so it's not lost 21:23:10 oomichi: ^-- 21:23:18 ok sure mriedem will do 21:23:24 So, I think that means its an exception winner too 21:23:30 i don't think i should commit to review, but happy to support the exception 21:23:36 With a note that we're really close to not having enough time to actually implement it all 21:23:40 #agreed v2-on-v3-api gets an exception 21:23:43 the v2.1 on v3 has all the microversions details merged into it now too so its not dependent on anything else 21:23:55 I think the people I need for reviews are dansmith and alaski based on previous review concerns 21:24:06 sdague too if he has the time 21:24:14 mriedem: from what I understand, no spec will be approved in Juno, only K version, ok? 21:24:19 I'm willing to review it 21:24:33 Thanks man 21:24:35 mikal: yeah, I feel like several of us have said that we're not comfortable trying to get it done by juno, but I'll take another trip through that spec and see if I can convince myself 21:24:43 raildo: exceptions for juno have to go through the openstack-dev mailing list right now 21:24:57 it does seem scary to crunch in 21:25:14 I think it depends where we draw the line 21:25:15 But yes 21:25:18 but i'm not against at least progress in juno if there can be consensus on the spec 21:25:22 mriedem: ok, thanks 21:25:31 I am worried specifically about the proxying code that isn't written yet (cinder, neutron that is) 21:25:31 I agree with that. But progress would be nice so there's more to discuss at a summit 21:25:57 we had the spec initially submitted in April, so we've been ready to go, just waiting on spec approval really (and the v2.1 on v3 spec itself hasn't changed much) 21:26:27 For better or worse this is where we are 21:26:35 But I would like to get a start on it at the least 21:26:47 But yeah, let's let the exception process do its work and see where we end up 21:26:52 yep, agreed 21:27:14 I think we should move on from specs to be honest. We only have 30ish minutes to go 21:27:25 Any other exception requests to the mailing list please 21:27:40 Noting that we already have a really large number of specs approved... 21:27:42 mikal, sent a short quick one for mine. 21:27:47 I am already worried about code review bandwidth 21:27:59 yes. 21:28:01 yep 21:28:07 I'm quite concerned 21:28:12 Well, I'm always worried about it 21:28:17 But I think its getting worse 21:28:23 Anyways 21:28:30 #topic Mid-cycle meetup 21:28:38 did we find chairs? 21:28:42 #info We ran out of seats, mikal has a plan 21:28:44 or will there be lap sitting? 21:28:50 ewwwww 21:28:54 So... I've asked Intel to re-confirm the size of the room 21:28:54 hot seat 21:29:04 Its apparently a training room so we might be able to drag more sets in 21:29:07 it's going to smell like a locker room in there... 21:29:14 I'm sure we're good for 35, >40 might be a problem. 21:29:15 Either way, I'm going to try and sneak in everyone from that mail thread 21:29:25 Some of the containers people ahve moved to their meetup, which helps 21:29:35 ok, was going to ask about that 21:29:45 We will just need to be organized about sending a runner to their room when we start ranting about them 21:29:54 Ditto for the ironic folks 21:30:01 But yeah, we can make it work 21:30:06 the rooms are withint about 30' of each other 21:30:19 n0ano: we're metric for K now. So 10 meters 21:30:21 tin cans and string? 21:30:24 Heh 21:30:32 do we have an agenda? or did i miss it? 21:30:43 So, I had a call with n0ano and the other two meetup organizers about logistics the other day 21:30:46 So I think we're set there 21:31:00 There will be an email with details (emergency phone numbers etc) next week 21:31:09 That will just go to registered attendees 21:31:15 tjones: there's an etherpad with proposed topics 21:31:19 Because I don't want to give out the Intel admin's phone number ot the entire world 21:31:25 Yes, next is the agenda 21:31:28 I do need a final list of attendees names & email aaddresses by this coming Mon. (have to arrange wifi access) 21:31:45 n0ano: ok cool. Can you email me that so I remember? 21:31:55 mikal, already did 21:32:01 So there's an etherpad collecting topics 21:32:03 n0ano: cool 21:32:11 There's also the hot topics from the meeting page 21:32:25 So, let's spend until say Wednesday nailing that down 21:32:33 And then we can try and turn it into a rough agenda 21:32:58 I'm ok with that bit beign a little rubbery -- if somehting needs more time than we think it does, we can keep going on it as long as nothing important misses out 21:33:05 Any questions about the midcycle? 21:33:22 do we know which day we are going to talk about bugs? I want to be there for that but can't be there the whole time 21:33:37 tjones: what day works best for you? 21:33:56 tjones: how about you pick a day and put that on the topics etherpad, we can just work with that 21:34:03 monday would be awesone 21:34:08 or awesome too 21:34:13 I refuse to talk about bugs on monday 21:34:15 Heh 21:34:20 dansmith: i figured 21:34:22 dansmith: mutter 21:34:28 tjones: just trying to be difficult.. monday is fine 21:34:33 dansmith: :-D 21:34:40 Someone bring a blow dart of sedatives for dansmith 21:34:43 tjones: devananda told me I'm always difficult, so I'm trying to live up to that 21:34:45 I think you can't get them at walmart 21:34:58 no, you can. 21:35:01 Oh, good 21:35:06 you can get anything at wal mart 21:35:12 sedatives at least, you may have to bring your own blow darts. 21:35:13 Including freedom? 21:35:17 Anyways 21:35:22 oh wait except music with swearing 21:35:25 So, nothing _serious_ about the midcycle? 21:35:53 * mikal times out 21:36:01 #topic Bugs 21:36:04 no, that's just your DSL 21:36:05 Apparently we have bugs 21:36:12 russellb: sigh 21:36:16 tjones: talk at us 21:36:25 ok so most of you have seen http://54.201.139.117/nova-bugs.html 21:36:32 Yes, its really good 21:36:38 which helps me an others get a better handle on our situation 21:36:53 The bug count seems to have gone down a fair bit since that page appeared 21:37:02 Last I looked we were at 1,000 instead of 1,400 now 21:37:06 jogo: has been very helpful with geting the abandoned and merged bug list down 21:37:09 jogo has been doing things 21:37:09 yeah 21:37:23 yeah things are moving in the right direction! 21:37:40 only 2 criical issues today and both are under control 21:37:55 Even the SSH timeout one? 21:37:56 Wow 21:38:01 i'd like to point out that there are 183 bugs ready for review 21:38:13 ready for code review? 21:38:20 yeah - ssh needs a backport, but last comment was it's ok on master 21:38:27 russellb: yes ready for review 21:38:27 tjones: so... a web page listing bugs for review where the bug is older than 7 days would excite me 21:38:30 tjones: wow 21:38:39 tjones: is that a patch other than the two I did last week to fix that? 21:38:40 I think excluding super new bugs would help 21:38:43 It may be worth removing owners for long un-updated bugs. 21:38:47 mikal: the updated column shows that 21:38:48 Because lots of people file a bug and then immediately fix it 21:38:49 tjones: because I've got all my fixed backported 21:39:13 tjones: oh, I more meant a page listing reviews I should do 21:39:21 tjones: to encourage, well, review of those thigns 21:39:23 mikal: sure i can add that 21:39:36 tjones: perhaps a separate page or somethign? 21:39:39 dansmith: i'm not aware of any others for the ssh thing 21:39:44 tjones: but thanks for keeping hacking on this 21:39:50 mriedem: okay, mine are all backported and committed 21:39:51 dansmith: unless those need to go to stable/havana for some reason 21:39:59 mriedem: yeah, don't think so 21:40:01 check jogo comment on https://bugs.launchpad.net/nova/+bug/1298472 21:40:04 dansmith: logstash can tell us, i'll check 21:40:04 Launchpad bug 1298472 in nova "SSHTimeout in tempest trying to verify that computes are actually functioning" [Critical,Confirmed] 21:40:14 http://status.openstack.org/elastic-recheck/ 21:40:20 we aren't kiling the gate right now 21:40:23 others are 21:40:28 mikal: sure it's been fun actually 21:40:33 On the bug front we should probably be paying attention to the mysql deadlock problems neutron is seeing, because I think we're seeing them too 21:40:50 The fix seems to be lost in pypi land, but we should keep half an eye on it 21:41:04 mriedem: yeah, my last one merged right after jogo's last comment on the bug I think 21:41:09 tjones: so I think that one is good 21:41:16 dansmith: great 21:41:21 dansmith: as in the SSHTimeout bug can be resolved? 21:41:44 Resolved^WMarked fix committed? 21:41:52 mikal: we still hit the problem occasionally, but for a different reason 21:41:57 mikal: this bug is so over-used though, 21:42:09 Can we close it and make a new one for the smaller case then? 21:42:09 yeah the query is very generic 21:42:12 mikal: that I think we should mark it done and open a new one to cover the sporadic other ones 21:42:14 so any ssh timeout will hit this query 21:42:16 mikal: yeah, we should do that 21:42:18 dansmith: +1 21:42:18 dansmith: snap! 21:42:21 i can remove the e-r query 21:42:27 Excellent 21:42:38 dansmith: please go and expense yourself a bottle of champagne 21:42:39 that was a big nasty bug, 21:42:42 dansmith: the cheap stuff please 21:42:52 which was not even gate-related and could be a real problem in production 21:42:56 so glad to get that resolved 21:43:02 dansmith: it could be passionfruit flavoured 21:43:07 mikal: will-do 21:43:20 dansmith: can you get the patch links into the bug report? 21:43:23 not sure they are all there 21:43:29 then we can close it 21:43:38 mriedem: I'll look, but I thought they were there, because I've been referencing it 21:43:47 lp linker doesn't work 21:43:48 always 21:44:13 Ok, we should probably move on again 21:44:14 mriedem: apparently none of them are, so yeah 21:44:17 Anything else on bugs? 21:44:27 not from me 21:44:36 tjones: thanks for doing a great job 21:44:42