14:00:21 #startmeeting Nova 14:00:23 Meeting started Thu Nov 12 14:00:21 2015 UTC and is due to finish in 60 minutes. The chair is johnthetubaguy. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:00:24 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:00:27 The meeting name has been set to 'nova' 14:00:34 \o 14:00:35 o/ 14:00:38 o/ 14:00:39 o/ 14:00:39 \o 14:00:42 o/ 14:00:43 o/ 14:00:43 hi 14:00:44 o/ 14:00:50 o/ 14:00:51 o/ 14:00:58 lets see who survived daylight savings / users a calendar 14:00:58 hello 14:01:03 o/ 14:01:07 o/ 14:01:18 #topic Release Status 14:01:33 so this is mostly a duplicate from the last meeting, but... 14:01:43 o/ 14:01:48 #info November 19: Spec Review Day 14:01:59 #info December 3rd is also non-priority spec and blueprints freeze 14:02:00 o/ 14:02:03 o/ 14:02:11 #link https://wiki.openstack.org/wiki/Nova/Mitaka_Release_Schedule 14:02:18 here is an advert from the API subteam 14:02:29 #info Virtual doc sprint December 8th and 9th 14:02:36 #link http://lists.openstack.org/pipermail/openstack-dev/2015-November/079220.html 14:02:44 johnthetubaguy: thanks 14:02:56 o/ 14:03:01 alex_xu: no problem, its super important we fix those things 14:03:09 so a quick check on blueprints 14:03:13 johnthetubaguy: yea 14:03:48 alex_xu: so I need more understanding, is the docsprint planned to improve http://developer.openstack.org/api-ref-compute-v2.1.html ? 14:03:53 #info 77 blueprints approved for mitaka now, still have 104 specs in review, got 60 ish completed last release 14:04:08 bauzas: several things, thats included 14:04:13 bauzas: yes 14:04:19 bauzas: I want to see the API concept guide make progress too 14:04:31 johnthetubaguy: okay, I guess the etherpad is the source of knowledge gotcha 14:04:32 * mriedem joins late 14:04:44 the swagger thing, I feel, is possible out of scope there, but anyways 14:04:55 yeah, that etherpad is good 14:05:08 ack 14:05:20 cool, so just wanted to talk about reno 14:05:25 bauzas: hows that going 14:05:40 so the reno changes are aligned for review 14:05:55 http://developer.openstack.org/api-ref-compute-v2.1.html 14:06:00 there will be a separate job for the reno docs right? 14:06:03 there are 2 changes per branch (master, liberty) 14:06:30 mriedem: yup, a project-config one https://review.openstack.org/#/c/242027/ 14:06:49 so, like I said, reno is intended for liberty and master 14:07:03 its generating the release notes right? 14:07:07 yup 14:07:21 https://review.openstack.org/242007 is the foundational change 14:07:23 cools, just context for folks there 14:07:34 which deploys reno and setups a way to add relnotes 14:07:51 that one is sharing its changeid with the liberty backport 14:08:00 so we have the odd blueprint complete, so we might need to add something in there after the fact, but we can deal with that later 14:08:08 and then, I provided a relnote example for both branches 14:08:14 like https://review.openstack.org/242007 14:08:15 o/ 14:08:17 so we have to move the liberty release notes in tree? 14:08:26 mriedem: zactly 14:08:30 harumph 14:08:31 ok 14:08:34 http://docs.openstack.org/developer/reno/usage.html#creating-new-release-notes 14:08:57 that's what I like with the new process actually 14:09:09 oh i suppose b/c with stable/liberty we do our own point releases now too 14:09:15 because it means our UpgradeImpact tag becomes less pedantic 14:09:17 yeah 14:09:36 mriedem: right, it's decoupled 14:09:48 so that brings us to the release CPL 14:10:20 johnthetubaguy: like I said in the previous meeting, I can help on that 14:10:41 bauzas: that help would be very welcome 14:10:46 but I welcome any help too :) 14:10:51 heh 14:11:02 mriedem: you are doing good stuff with stable and python-novaclient already, I think it makes sense you keep doing that 14:11:11 so I'm assuming we're going to reno every API change 14:11:11 alright 14:11:22 sdague: that makes sense to me 14:11:25 i had to think awhile to remember what CPL is 14:11:49 ah, yeah, CPL = cross project liaison 14:12:03 sdague: there are some predefined sections for every relnote, but we could certainly ask for an API one http://docs.openstack.org/developer/reno/usage.html#editing-a-release-note 14:12:14 so mriedem, bauzas lets work out how to make sure the right things happen with release stuff 14:12:40 agreed 14:12:50 ack 14:12:55 so python-novaclient, its probably time we cut a release soon? 14:13:12 we were going to try and release a bit like oslo, i.e. more frequently 14:13:16 sdague said mordred had changes to make 14:13:21 one thing about that, I blocked the v1.1 remove 14:13:37 sdague: because too many folks are using it? 14:13:38 because mordred was poking at something else that was going to require a major bump 14:13:45 no, I think we should remove it 14:13:49 I didn't do it 14:13:52 wait 14:13:59 oh, I see just delaying for the major bump 14:14:02 we're just lining up tihngs that cause a 3.0 version 14:14:06 yeah, makes sense 14:14:09 oh - yeah. I'd love to get in on that 14:14:19 sounds like a plan 14:14:28 yeh, we should probably figure out a good way to stack up the stuff we want in that would require a major bump 14:14:29 what the chances of us lining that all up by Tuesday? 14:14:38 zero ish? 14:14:41 I can get my changes up by then 14:14:42 johnthetubaguy: we can cut a release without that one change 14:15:00 I just think we should actually be a little more coordinated on the v3 requiring changes 14:15:03 sdague: thats a not a bad idea, get a minor out, while we line up the other stuff 14:15:17 we need to get better about signaling to the core team when we're going to do a release though, so we don't approve things that we don't want in 14:15:20 * johnthetubaguy wonders about a feature branch for v3, maybe that over kill? 14:15:34 mriedem: very true 14:15:36 johnthetubaguy: honestly, I was thinking a similar thing 14:15:36 summit week was a bit chaotic with novaclient reverts and releases 14:15:41 yeh, it was 14:16:08 we need to fix that test coverage at some point too 14:16:20 but lets focus on the release thing 14:16:22 so - question about how much we can break in v3 14:16:33 oh, wait. nm. 14:16:46 sdague and I already talked about that - I'm still waking up 14:16:53 v3 as in keystone v3? 14:16:53 heh, no worries 14:17:00 mriedem: as in novaclient v3 14:17:04 yeah, that 14:17:08 ok 14:17:13 I think it was mostly the NOVA_ env variables for config 14:17:22 being no longer supported 14:17:30 yah. I'd love to make them die because it makes things eaiser on the occ/ksa front 14:17:31 oh right, good points 14:17:37 but - if we need to keep them, I can keep them 14:17:56 so, anyway, we should huddle up about the non backwards compat changes we want this cycle 14:18:15 yeah, leaving that with sdague and mriedem to report back to the ML with a plan? 14:18:25 oh, if there is a v3 window, maybe that's a good opportunity for seeing what we want to stop supporting - like some contrib shell scripts that I'm questioning :) 14:18:45 in the mean time we do a minor version release anyways? 14:18:48 so, +1 for a feature branch :) 14:18:54 yes, lets do a minor version release 14:19:17 #action mriedem to request a novaclient minor version release for mitaka 14:19:18 I think the project id optional thing is not yet released yet, and we'll need that for project id optional in nova 14:19:23 mriedem: do we want to send an ML note about about a proposed release, just so we know whats happening, thinking thats nice and light weight? 14:19:34 sure 14:19:43 lets try that for size 14:19:46 cools 14:20:03 so I guess one thing I forgot to mention at the summit 14:20:18 we are still planning to do "milestone" releases for Mitaka 14:20:55 we get to choose when now, but I am just going to aim for the usual tuesday releases, split to thursday if we must 14:21:25 wfm 14:21:30 the question is do we want to move to ironic's way of doing actual semver for the milestones, rather than a beta release stuff 14:22:01 so 13.1.0? 14:22:09 yeah, vs 13.0.0.a1 14:22:18 13.0.0a1 rather 14:22:18 well I guess 13.0.0 first 14:22:21 ah snap 14:22:29 bauzas: yeah, thats it 14:22:34 i guess i assumed 13.1.0 would be first stable/mitaka release 14:22:40 but that's just b/c that's how it's always been 14:22:45 13.1 being the stable point release AFAIU 14:22:46 honestly, let's just stick with the old way until there is a reason to change 14:22:58 sdague: +1 14:23:07 sdague: so thats my take, I just wanted to make sure we didn't do that by acciedent 14:23:08 in particular given the many release changes for this cycle :) 14:23:24 yeh, if it ain't broke... 14:23:27 the difference being one signifies beta, and the other stable? 14:23:40 well we allow people to deploy of master 14:23:47 so the milestones mean more than you think 14:23:51 right 14:23:56 it's a pinned release 14:24:08 but honestly, we should move on really 14:24:38 #topic Regular Reminders 14:24:46 so we have reviews highlighted by subteams 14:24:52 there are some changes in there now 14:24:59 #link https://etherpad.openstack.org/p/mitaka-nova-priorities-tracking 14:25:08 priority specs are listed in here: 14:25:18 #link https://etherpad.openstack.org/p/mitaka-nova-spec-review-tracking 14:25:33 #topic Bugs 14:25:36 so, why did we do this as 2 etherpads this time? 14:25:54 that caused some confusion yesterday 14:26:05 yeah 14:26:12 so the old one didn't change, its all code 14:26:16 the first etherpad is the same as last release 14:26:24 the latter is to categorize specs for review 14:26:33 there is some overlap 14:26:33 I though the old one had specs in it as well 14:26:40 we could just mush them together if its easier, it just seemed like it work worth separating 14:26:41 should just be code 14:26:46 we put specs on the old one early on didn't we? 14:26:53 hmm, maybe we did 14:27:06 anyway, it's just confusing to me, 14:27:11 because they look similar, 14:27:18 and you can't tell a thing is a spec by the review url 14:27:32 so I dunno, seems like one place would be fine 14:27:33 so I could cut and past the specs into the code one? 14:27:42 simpler is better 14:27:52 johnthetubaguy: can anyone suggest a feature for this nova priorities etherpad? 14:28:04 i'd actually prefer separataion 14:28:07 raildo: anyone can create a subteam 14:28:12 there are a lot of specs 14:28:28 so the good news is the spec one dies on December 3rd 14:28:31 is the goal of the second one to list every spec? 14:28:36 kill! kill! 14:28:38 johnthetubaguy: nice 14:29:25 so I think its worth trying the separation, if its a flop lets remember not to do that next time? 14:29:29 ok 14:29:44 so turns out we are in the bugs topic, oops 14:29:51 markus_z: whats up? 14:29:55 undo :) 14:30:15 I started the bug shaming thing on the ML 14:30:18 #link http://lists.openstack.org/pipermail/openstack-dev/2015-November/078713.html 14:30:26 Would be cool to get feedback 14:30:43 #info Top 3 subteams with most "New" bugs: untagged: 25, libvirt: 14, volumes: 11 14:30:46 I think there was a call to add low-hanging-fruit into the tracking 14:30:57 but I like the tracking 14:31:13 we know where we are at 14:31:14 It was a call out to tag more bugs with low-hanging-fruit 14:31:26 yeah, just curious about the count 14:31:47 I will add it in the next mail 14:31:48 in general those unassigned are taking in the next following minutes 14:31:54 I'm not that worried 14:31:58 although clearly one million stats and it looks silly 14:32:24 mriedem: how is stable doing, is it time to release any of the branches? 14:32:30 the thing is, we should rather see how many assigned low-hanging-fruits are actually in progress and possibly unassign 14:32:31 stable/juno freeze is today 14:32:42 2014.2.4 is planned for next thursday 14:32:58 #info Thursday Nov 12 stable/juno freeze 14:32:59 https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:stable/juno,n,z 14:32:59 #info Thursday Nov 19 release 2014.2.4 14:33:15 will be happy to eol juno 14:33:19 there are a couple of changes 14:33:25 bauzas: that's the normal step when looking for stale bugs, right? 14:33:34 i think i'm going to recheck babysit the one that's approved and let the rest die on the vine 14:33:41 markus_z: most newcomers don't do that 14:33:47 so the freeze, remind me, does that mean no more merges after today, or no more proposals? 14:33:52 markus_z: but let's bring the convo out of here 14:34:07 it also appears that gate-grenade-dsvm-ironic-sideways is borked on stable/juno 14:34:09 blocking lots of things 14:34:15 trying to get ironic people to look but haven't heard much 14:34:17 jroll: ^ 14:34:31 mriedem: honestly, we should just delete that job 14:34:45 does it actually try an upgrade? 14:34:52 that was about ensuring there was a baremetal -> ironic transition on juno 14:35:00 oh 14:35:01 sideways means bm to ironic yes? 14:35:02 oh... 14:35:05 but if people are waiting until now to do that juno transition 14:35:11 well, they are kindof f'ed 14:35:16 heh, yeah, i'll drop it 14:35:16 as we're about to eol it all 14:36:02 cool, so I guess we need to unblock that, and get the stuff merged, right? 14:36:10 well, fixed up and merged 14:36:15 yeah, i'll take those todos 14:36:21 sweet, thank you 14:36:34 out of curiosity, does ironic still support juno at all themselves? 14:36:40 action mriedem to give stable/juno much love 14:36:48 oops 14:36:50 #action mriedem to give stable/juno much love 14:36:52 ++ to dropping the juno sideways job 14:36:56 OK... 14:36:58 sdague: not really https://review.openstack.org/#/q/status:open+project:openstack/ironic+branch:stable/juno,n,z 14:37:20 #topic Open discussion 14:37:21 so maybe ironic should actually have eoled their branch :) 14:37:46 so... I missed at bit earleir 14:37:49 specless blueprints 14:37:59 I'd like to ask for reviews on the following bugfix: https://review.openstack.org/#/c/215102/ 14:38:07 #link https://etherpad.openstack.org/p/mitaka-nova-spec-review-tracking 14:38:14 there are loads of blueprints in there 14:38:23 I have added comments on them about what I think we should do 14:38:25 It's a libvirt driver change and got a +2 from danpb already two months ago 14:38:44 if there are any objections, do let me know ASAP, and I will get them merged / dropped off the specless list as required 14:39:18 I don't remember any controversial ones, that I didn't just reject to be reviewed as a spec 14:39:35 Ok, any more for any more? 14:39:36 johnthetubaguy, Quick note on the mid cycle 14:39:48 PaulMurray: fire away 14:39:58 I will send out an email in the next couple of days with details about block bookings 14:40:04 and web site for all details 14:40:14 PaulMurray: is it a cheap-ish hotel? 14:40:30 its cheeper than listed 14:40:32 how many people registered so far ? 14:40:47 PaulMurray: well thats an important bit, true 14:40:47 https://review.openstack.org/#/q/topic:grenade_multinode,n,z - grenade multinode changes to make that voting, and drop partial-ncpu from master 14:40:49 don't know - mikal set up the eventbrite ticket 14:40:59 ack 14:41:11 our site will be seperate from eventbrite 14:41:25 I have not heard from mikal on numbers either, worth asking him 14:41:27 unfortunately - but will try to reconcile to avoid mistakes 14:41:35 will do 14:41:38 PaulMurray: I am sure mikal can update that all for you 14:41:39 yeh, need that list from mikal as well, because I think a few of us were thinking about going to the dr who experience on friday post event (given that it's close) 14:42:01 and seemed better to just coord with people that signed up, vs. whole world 14:42:02 doctor who experience ? woah 14:42:06 you know - your not the only one to mention that 14:42:14 yeh, it came up at summit 14:42:16 shoud we organise a trip? :) 14:42:22 PaulMurray: yes 14:42:32 ah, thats work looking at 14:42:50 yeah, most of us are holding to book our flights until the details are sorted out 14:43:09 sdague: the grenade job, do we do the scheme changes with the code running yet? 14:43:10 bauzas, for the dr who experience or for the mid cycle? 14:43:16 I still have to go to the company internal approval process, that's why I didn't yet enrolled for the midcycle 14:43:22 PaulMurray: for your details 14:43:25 Would be cool to get the information too 14:43:25 johnthetubaguy: can you rephrase the question? 14:43:38 PaulMurray: for the midcycle 14:43:45 johnthetubaguy: no, we don't 14:43:54 johnthetubaguy: I'd like to have a job to do that, but it's a bit wasteful 14:43:55 sdague: sorry, sure, I am thinking about online schema migrations, and how we check they are online, thats probably not the right approach 14:44:14 dansmith: experimental queue :) 14:44:15 johnthetubaguy: perhaps an experimental job to do that and we can run it periodically 14:44:21 dansmith: maybe we could do some DB functional tests against while running it, or something 14:44:29 mriedem: yeah, thats a good point 14:44:34 johnthetubaguy: we can't really 14:44:52 dansmith: we can't with a conductor and an old schema? 14:44:55 dansmith: run stable/liberty tests or something 14:44:58 johnthetubaguy: I think an grenade job in experimental that just applies then and never upgrades anything is the way to go 14:45:04 sdague: opposite 14:45:16 sdague: mriedem: do you still need something from me re: juno or did y'all figure it out 14:45:23 jroll: ignore 14:45:25 jroll: https://review.openstack.org/#/c/244688/ 14:45:35 dansmith: ok, let's ponder offline 14:45:43 yep 14:45:43 mriedem: cool, thanks 14:45:44 dansmith: oh, gotcha, thats a good point, could we not just do the schema migrations first on all genade jobs 14:45:49 sdague: yeah, I got distracted 14:45:59 so... 14:46:06 anyway, on the grenade multinode change it's mostly infra / test changes to get us going, however it would be good for people to review 14:46:12 I think we have wondered off into the weeds, so we are probably done now? 14:46:23 maybe one last thing 14:46:24 the big last piece is a test to ensure we actually can run computes on all the nodes 14:46:31 which is in that stack 14:46:42 https://review.openstack.org/#/c/244330/ - an experimental queue job for cells v1 + neutron 14:46:52 to mitigate against the worker not being able to communicate with the controller post upgrade 14:46:56 and us passing anyway 14:47:25 cells v1 + neutron seems like an epic distraction from moving forward with cells v2 14:48:10 I'm really afraid it means we lose another cycle or two getting to a model we want 14:48:32 i think that's a bit premature 14:48:58 it's not like this is a voting job that everyone sees daily now 14:49:08 maybe, I know I've spent at least 4 hours in the last week having to care about cells because of this effort 14:49:09 so its tempting if we could turn off the nova-network job 14:49:19 sdague: that wasn't b/c of this 14:49:28 johnthetubaguy: no we aren't doing that 14:49:30 johnthetubaguy: we have to implement event callbacks for cells at a minimum for being able to gate on this 14:49:31 mriedem: no, it was because of cells v1 in general 14:49:34 cells v1 doesn't have external events support 14:49:36 dansmith: true 14:49:44 dansmith: which there is a change up for 14:49:48 but still 14:49:51 yeah 14:49:53 we want a cells v1 job 14:49:58 we have nova-net, this isn't replacing that 14:50:03 if this is never going further than an experimental job then fine, whatever 14:50:08 bdm stuff blew up last week and is still being worked 14:50:25 thats the alternative, keep it experimental 14:50:33 mriedem: can you link me that change? 14:50:40 sec 14:50:52 https://review.openstack.org/#/c/184155/ 14:50:53 having two cells v1 jobs all the time seems like the wrong call, somehow 14:50:54 right, but the point is that every time we take our eye of the cells v2 stuff to duct tape back cells v1 which keeps slowing us down with blow ups like the bdm thing, we push cells v2 out further 14:51:02 johnthetubaguy: it wouldn't be all the time 14:51:09 nova-net is gating and voting 14:51:16 neutron is experimental queue (on demand) 14:51:18 mriedem: yeah, I think its fine in experimental 14:51:55 so I am working hard to get more effort behind cells v2, internally 14:52:05 in the hope to hurry that along more 14:52:13 its critical we get there ASAP 14:52:22 I was going to work on cellsv2 more this cycle 14:52:45 but I've spent a lot of time on cellsv1 bugs already, and plan to go review this cellsv1 event patch (which is wrong) 14:53:09 right, that's the issue, this keeps being messaged as "it's not a big deal" and then it's death by a thousand paper cuts 14:53:20 right 14:53:39 also, 14:53:45 so should we deprecate it now? to make it clear upstream will not maintain that job next release 14:53:48 mriedem's single cell job will show that this event patch works 14:53:54 but it doesn't for multiple cells properly 14:53:54 its up to third party CI if we want it? 14:54:01 I got dragged into cells stuff because I couldn't land devstack patches until it got reasonably passing again 14:55:30 personally, I think the important thing is to make it clear that the focus is moving forward. A lot of things are broken by design in cells v1 (like the locked mechanism) 14:55:31 i wouldn't be on board with deprecating a cells job (the one we have voting today) until v2 is ready 14:55:48 and cellsv1 will get no new features 14:55:51 so I was originally of the opinion that landing some of the code that operators are holding would be a good thing, but I do agree that this is taking up valuable time from v2 14:56:10 we have to be firm on the no new features front 14:56:15 otherwise, we're never going to get to v2 14:56:39 so I am really starting to lean that way 14:56:44 the options are: 14:56:54 several of us have spent serious time on the exisiting bdm problem with v1 in the last week and a half 14:56:56 so, only regressions then? like the bdm failures 14:57:05 1) drop cells v1 job, and be clear its dead 14:57:23 2) add a few missing bits from cells v1 14:57:26 alaski: the bdm thing wasn't actually a regression, it was just a new test that exposed the problem 14:57:33 I don't think we should drop the job. I think we should decide do we fix failures or skip them 14:57:36 3) feature freeze v1 hard, but keep on top of regressions 14:57:42 the bdm failure originally came from something which was skipped and then enabled 14:57:52 I think #3 14:57:56 without having properly been tested with cells 14:57:58 definitely #3 14:57:59 right 14:58:05 #3 +1 14:58:07 mriedem: okay, wasn't quite aware of that 14:58:09 regressions only, not even fixing things that are known broken 14:58:22 and realize that because new tests are getting added, and cells is massively under tested, fixing in #3 might mean skipping a test 14:58:24 so the bdm stuff was even not a regression 14:58:28 I just want a way to message its expense with regressions only, that means feature freeze to make that possible 14:58:33 yeh, like dansmith said, only fixing actual regressions 14:58:39 not new bugs that were exposed 14:58:42 +1 14:58:45 that were always there 14:59:04