15:00:32 <bswartz> #startmeeting manila 15:00:33 <openstack> Meeting started Thu Feb 1 15:00:32 2018 UTC and is due to finish in 60 minutes. The chair is bswartz. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:00:34 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:00:36 <openstack> The meeting name has been set to 'manila' 15:00:40 <bswartz> hello all 15:00:42 <amito> hello o/ 15:00:44 <ganso> hello 15:00:45 <tpsilva> hey 15:00:49 <zhongjun> Hi 15:00:57 <tbarron> hi 15:01:11 <toabctl> hi 15:01:23 <bswartz> courtesy ping: gouthamr xyang markstur vponomaryov cknight 15:01:35 <markstur> hi 15:02:21 <vkmc> o/ 15:02:27 <bswartz> I'll skip announcements today and go right into topics because the topics cover the announcements (if that makes sense) 15:02:38 <bswartz> #agenda https://wiki.openstack.org/wiki/Manila/Meetings 15:02:55 <bswartz> #topic Rocky PTL 15:03:03 <bswartz> #link http://lists.openstack.org/pipermail/openstack-dev/2018-January/126767.html 15:03:21 <bswartz> So all of you who read the ML know that I'm stepping down at PTL 15:03:38 <bswartz> I'm not a candidate for Rocky 15:03:58 <bswartz> I think I've been PTL 7 (maybe 8) terms and it's long enough 15:05:30 <bswartz> I guess the most important thing I can say is thank you to everyone in the community who has supported the project, and hopefully my leadership has been okay 15:05:42 <bswartz> :-) 15:06:09 <tbarron> well your leadership has been awesome 15:06:14 <zhongjun> But you still will continue to work on manila? 15:06:48 <bswartz> So my term should continue until the end of Queens, and I'll keep running the meetings, finish the release, and make sure the PTG is organized 15:06:55 <xyang> hi 15:06:59 <zhongjun> Thanks for all your work over 5 years 15:07:04 <bswartz> And I'd like to help teach the new PTL the ropes, whoever that may be 15:07:21 <bswartz> zhongjun: yes I intend to remain involved 15:07:50 <toabctl> bswartz, thanks for being the PTL so long. Well done!! 15:08:15 <bswartz> The main think I wanted to mention at this meeting today is that I encourage anyone interested in leadership to nominate themselves for the Rocky term 15:08:21 <toabctl> bswartz, will manila still be you fulltime job what you get paid for? 15:08:57 <bswartz> Last time I checked, there weren't any nominees, and if the nominating period ends with no nominees, we enter a murky territory where the TC chooses someone 15:09:26 <markstur> tc can make bswartz lifetime appointment? 15:09:29 <bswartz> toabctl: I'm still paid to work on manila, but not full time -- I'm also doing kubernetes work now 15:09:42 * dustins slides in late 15:09:50 <ganso> bswartz: the TC may choose anybody or only manila contributors? 15:10:21 <bswartz> To nominate yourself, add a file to this directory: https://github.com/openstack/election/tree/master/candidates/rocky/Manila 15:10:48 <bswartz> ganso: honestly I'm not sure how the TC makes that decision -- in the cases I've been in the past, it's been messy 15:11:21 <bswartz> s/been/seen/ 15:11:35 <bswartz> Better to avoid that and follow the normal nomination/election process 15:11:57 <bswartz> I'm very committed to helping the new PTL come up to speed, because I care about the fate of the project 15:12:33 <bswartz> But I'm not a huge fan of the lifetime appointment thing 15:12:52 <xyang> bswartz: thank you for being a great PTL for so many years! 15:13:13 <amito> bswartz: thank you! 15:13:16 <bswartz> Thanks for the kind words 15:13:48 <bswartz> Please PM me offline if you're interested in leading the project but have any questions or reservations 15:13:58 <bswartz> Getting back to business... 15:14:06 <bswartz> #topic RC1 15:14:15 <bswartz> #link https://launchpad.net/manila/+milestone/queens-rc1 15:14:47 <bswartz> We're a week away from the RC1 target date and we don't really have many bugs to speak of 15:15:27 <bswartz> Is anyone working on bugs, and they simply haven't been targeted yet? 15:15:55 <ganso> bswartz: I am 15:16:00 <bswartz> If not, we could cut the RC as soon as ganso's bug is fixed 15:16:07 <zhongjun> I have a simple one 15:16:09 <bswartz> ganso: I see 1 in your name -- are there others? 15:16:19 <dustins> ganso: They must be the two I added to the etherpad this morning :) 15:16:27 <ganso> yes, but I would like to discuss the controversy of one before targetting the others 15:16:34 <bswartz> Okay we can cover specific bugs in the next topic 15:16:39 <tbarron> https://review.openstack.org/#/q/owner:junbo85.li%2540gmail.com+status:open 15:16:48 <tbarron> ^^^ soem of those are in master 15:16:50 <zhongjun> We also have the bug that it can let ci break 15:17:22 <bswartz> But I wanted to make it clear that this page is how I decide when to cut the RC, and if there are no open bugs on it, I have to assume it's safe to cut the RC immediately 15:17:23 <amito> I work on one related to our driver (infinidat), don't know if that counts 15:17:49 <bswartz> amito: yes, driver bugs are covered by the RC deadline 15:18:06 <bswartz> either the bug must be fixed before RC1, or we punt it to Rocky 15:18:35 <bswartz> No bugs may be fixed between RC1 and final release except for bugs discovered after the RC1 tag 15:19:09 <amito> bswartz: ok. the patch is already out there 15:19:25 <bswartz> amito: what matters is the milestone target in LP 15:19:36 <bswartz> Post a link to the LP bug, and I'll target it right now 15:20:05 <bswartz> Obviously every change we merge must have a bug number in it at this point (except docs) 15:20:22 <amito> https://bugs.launchpad.net/manila/+bug/1746439 15:20:23 <openstack> Launchpad bug 1746439 in Manila "INFINIDAT share driver fails to delete shares with snapshots" [Undecided,In progress] - Assigned to Amit Oren (amito) 15:20:37 <bswartz> Any others to target right now? 15:21:41 <bswartz> Okay 15:21:46 <zhongjun> https://review.openstack.org/#/c/539406/ ? 15:22:06 <bswartz> zhongjun: thanks 15:22:31 <tbarron> https://review.openstack.org/#/c/535611/ 15:22:53 <bswartz> So reviewers should focus review on the bugs on that list, and as we get closer to the deadline, we either need to get the fix merged, or untarget the bug if the fix needs more time for whatever reason 15:23:15 <bswartz> tbarron: already got that one 15:23:26 <tbarron> bswartz: cool, sorry 15:23:29 <bswartz> The client bugs will be trickier 15:23:36 <bswartz> I have yet to create a rc1 milestone for the client 15:23:52 <bswartz> but there is time to merge fixes to our client too 15:24:33 <bswartz> Anyone who owns a bug targeted at RC1 can expect pings from me next week as long the bugfixes aren't merged 15:24:59 <bswartz> Based on history, this list doesn't look very bad 15:25:38 <bswartz> I'm optimistic we can get everything reviewed and merged by early next week, as long as the authors are responsive to review feedback 15:25:57 <bswartz> Let's move on to dustins' list in case he has some bugs the required discussion 15:26:09 <bswartz> #topic Let's Go Over New Bugs 15:26:17 <bswartz> #link https://etherpad.openstack.org/p/manila-bug-triage-pad 15:26:22 <dustins> It's a short one this week, actually, both ones opened in the last day or so 15:26:33 <dustins> #link https://bugs.launchpad.net/manila/+bug/1746725 15:26:34 <openstack> Launchpad bug 1746725 in Manila "LVM driver is unable to remove addresses in different IP versions belonging to the same interface properly" [Undecided,New] 15:26:40 <ganso> dustins: I opened them both today 15:26:48 <ganso> and one of them is the one I would like to discuss 15:26:54 <bswartz> Ooh they're in red 15:26:58 <dustins> ganso: Is it this one or the other one? 15:27:00 <bswartz> They must be very important 15:27:10 <bswartz> lol 15:27:10 <ganso> it is the first one 15:27:27 <dustins> Okay, cool, take it away, ganso :) 15:27:46 <ganso> ok so let's talk about https://bugs.launchpad.net/manila/+bug/1746725 15:27:48 <openstack> Launchpad bug 1746725 in Manila "LVM driver is unable to remove addresses in different IP versions belonging to the same interface properly" [Undecided,New] 15:28:04 <ganso> I identified a problem in exportfs -u command from nfs library 15:28:18 <bswartz> Oh yes 15:28:19 <ganso> or if it is isn't a problem, it is something that the LVM driver is not prepared to handle 15:28:27 <bswartz> s/nfs library/nfs-utils/ 15:28:33 <ganso> as it can leave the manila DB in an inconsistent state 15:29:02 <ganso> in summary: the driver uses exportfs -u to remove rules 15:29:16 <bswartz> I've been aware of problems with the NFS helper used by the generic and LVM drivers since ocata 15:29:33 <bswartz> That's the reason I proposed a new helper (NfsExportsHelper) 15:29:35 <ganso> and when there are 2 rules, one being IPv4 and the other being IPv6, and both relate to the same physical network interface, the exportfs -u removes both rules with when attempting to remove one or the other 15:29:55 <bswartz> #link https://review.openstack.org/#/c/476239/ 15:30:20 <ganso> so, I am already working on a fix, and the approach I am taking to fix this bug is pretty much the same as bswartz in his NFSHelper 15:30:39 <ganso> I am using https://github.com/openstack/manila/blob/81e708281f6bdeea2dba12eb08cf9c33265e7050/manila/utils.py#L700 15:30:39 <bswartz> The main issue with my new helper is that I never found a way to migrate users from one helper to another, making upgrades a possible nightmare 15:31:15 <ganso> to write back the file with all the rules, except the one removed, after reading it with a cat 15:32:00 <ganso> bswartz pointed out a possible disadvantage 15:32:05 <ganso> of using this approach 15:32:29 <ganso> if you look closely, this command processes input from the execute command 15:32:36 <bswartz> ganso's fix might work, but it builds on an unstable foundation 15:32:47 <ganso> bswartz: precisely 15:33:11 <bswartz> I'd like to work out a solution to the upgrade problem and then move everyone to the new helper I proposed 15:33:23 <bswartz> Unfortunately that can't really happen until rocky 15:34:11 <ganso> I don't have enough field experience to know all possible consequences of this approach, bswartz feels uneasy about it. I would like to know other community members' take on this or if someone can share more knowledge on the caveats of this approach 15:34:12 <bswartz> So do we want to commit ganso's fix for queens, or leave this bug unfixed (it only occurs in pretty rare circumstances) 15:34:14 <bswartz> ? 15:34:53 <ganso> if we as a community agree this is not the right solution, and I cannot come up with a better one, I will prefer to punt this to Rocky 15:35:02 <tbarron> +1 15:35:23 <bswartz> tbarron: +1 for not fixing? 15:35:27 <tbarron> if we had lvm driver users broken right now complaining I'd think differently 15:35:30 <tbarron> bswartz: ack 15:35:32 <bswartz> k 15:35:57 <bswartz> ganso: can you live with that? 15:36:21 <ganso> bswartz: yes, then I would like to discuss the fix for the share migration bug that we wouldn't be able to test in the gate 15:36:41 <bswartz> ganso: is that the next bug in the list? 15:36:55 <ganso> bswartz: no, but this decision impacts that bug 15:37:10 <dustins> bswartz: Doesn't look like it, I only grabbed bugs that were new and unassigned 15:37:25 <bswartz> ganso: can you link that bug? 15:37:36 <ganso> https://bugs.launchpad.net/manila/+bug/1745436 15:37:38 <openstack> Launchpad bug 1745436 in Manila "Host assisted Share Migration breaks on IPv4+IPv6 backends" [High,In progress] - Assigned to Rodrigo Barbieri (rodrigo-barbieri2010) 15:37:53 <bswartz> Oh this one that's already targeted 15:37:59 <ganso> so, while fixing that bug, I stumbled across the LVM bug because that's what we use to test in the gate 15:38:25 <bswartz> As long as the core code works with non-buggy drivers, I think that's good enough for the release 15:38:50 <bswartz> If we have to mark some tests as skip temporarily, that's okay with me 15:39:08 <bswartz> (only skip on the buggy drivers) 15:39:24 <ganso> bswartz: I'd prefer to not take that approach 15:39:59 <bswartz> ganso: what do you have in mind? 15:40:32 <ganso> bswartz: I like any other alternative as well, such as having an ipv6-only job, which would still not allow to test IPv4+IPv6 configuration 15:40:59 <ganso> bswartz: but it would improve coverage and allow this bug to be fixed and 3rd party vendors to be able to use IPv6 in migration (which is blocked right now) 15:41:32 <bswartz> Okay I'm confused 15:41:44 <ganso> right now vendors can only use share migration with IPv4 only. They could use it with IPv6-only before we blocked it 15:41:56 <bswartz> I want to fix the core issue 15:42:07 <ganso> bswartz: the LVM driver cannot handle IPv4+IPv6 as it is, so we cannot have a gate job for it 15:42:30 <bswartz> That doesn't stop us from fixing the underlying problem 15:42:45 <ganso> bswartz: I don't feel happy about disabling all upstream share migration tests so we can commit this fix, I'd rather stay as we are than do that 15:43:13 <bswartz> Me neither -- why couldn't we disable just the versions of the test that will run into trouble? 15:43:26 <bswartz> Like only the mixed v4/v6 flavors of migration 15:43:55 <ganso> bswartz: ok, we would only need to disable it in the LVM driver job 15:44:01 <ganso> bswartz: scenario job and generic would still run it 15:44:09 <ganso> (I need to double check that) 15:44:11 <bswartz> would generic pass or fail? 15:44:30 <ganso> in theory will pass, as the job is already configured with IPv4 only 15:44:32 <bswartz> Generic uses the same nfs helper code 15:44:35 <bswartz> Ok 15:44:36 <ganso> but I need to double check if it is not disabled 15:45:07 <ganso> because, when we moved all the important backend jobs to run on the LVM job, we may have disabled them in the other jobs for performance reasons 15:45:17 <bswartz> We can always verify the fix using yours (or my) fix to the helper, even if neither of those changes are ready to merge for queens 15:45:52 <ganso> ok 15:45:58 <ganso> seems like we reached an agreement 15:46:04 <bswartz> Okay 15:46:11 <bswartz> there's another bug on dustins' list 15:46:40 <ganso> https://bugs.launchpad.net/manila/+bug/1746723 15:46:41 <openstack> Launchpad bug 1746723 in Manila "LVM driver does not handle IPv6 addresses in recovery mode" [Undecided,New] 15:46:44 <ganso> that is an easy one 15:46:57 <ganso> but there is no point in working on that we if don't address the previous one 15:47:06 <ganso> the recovery mode path is used only by share migration at this moment 15:47:19 <bswartz> Okay but what is the impact for not fixing this? 15:47:29 <ganso> and it is not handling IPv6 well. We don't see it in the gate because we are dodging that 15:47:33 <bswartz> Does it cause problems in ipv6-only scenarios (which should work in theory)? 15:48:06 <ganso> bswartz: that IPv6-only scenario has been blocked by our previous IPv6 patch, which allows only IPv4 export locations to be used in share migration 15:48:32 <bswartz> But that one will be fixed won't it? 15:48:32 <ganso> bswartz: so unless we unblock that, nobody will hit be able to run IPv6-only migration and will never hit that bug on LVM 15:48:39 <bswartz> Ugh 15:49:09 <bswartz> I wish we'd identified these issues earlier, and we could have spend time sorting out the upgrade issues around my new helper patch 15:49:56 <bswartz> Does anyone else have heartburn about potentially not testing a bunch of ipv6 code paths in the gate? 15:50:39 <dustins> I mean, IPv6 isn't going away anytime soon, we'll have to test it sooner than later 15:50:43 <ganso> bswartz: I can fix that bug and unblock IPv6, which will allow IPv6-only scenarios, and we wouldn't be running the tests in the LVM job. So all should work fine in a production cloud, in theory. We wouldn't be preventing anyone from running IPv6-only jobs anymore 15:50:58 <ganso> s/IPv6-only jobs/IPv6-only migrations 15:51:11 <bswartz> Okay I like that idea 15:51:26 <ganso> ok 15:51:30 <ganso> sounds good 15:51:54 <bswartz> Any other bugs? 15:52:07 <ganso> not from me 15:52:16 <bswartz> #topic Open Discussion 15:52:17 <dustins> None from me today 15:52:33 <bswartz> Anything else for today? 15:52:36 <markstur> bswartz Thank you, Ben! for all you've done as PTL and the making of manila 15:52:47 <tbarron> +1000 15:53:00 <bswartz> I should mention that this is the time in each release when we usually update docs related to changes during the release 15:53:57 <bswartz> We should make sure that the various API changes that went in have corresponding docs changes 15:54:19 <bswartz> It's mostly small stuff 15:54:37 <bswartz> We can revisit that again next week 15:55:00 <bswartz> Oh and please keep adding topics to the Manila PTG etherpad 15:55:10 <bswartz> I'll add the topic about the NFS export helper and upgrade issues 15:55:28 <bswartz> That's all for today 15:55:37 <bswartz> Thanks everyone 15:55:41 <bswartz> #endmeeting