15:00:37 <bswartz> #startmeeting manila
15:00:38 <openstack> Meeting started Thu Aug 25 15:00:37 2016 UTC and is due to finish in 60 minutes.  The chair is bswartz. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:39 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:41 <openstack> The meeting name has been set to 'manila'
15:00:44 <bswartz> hello all
15:00:44 <cknight> Hi
15:00:45 <tbarron> hi
15:00:46 <gouthamr> hey o/
15:00:46 <jseiler> hi
15:00:47 <dgonzalez> hi
15:00:47 <toabctl> hi
15:00:48 <aovchinnikov> hi
15:00:51 <tovchinnikova> \\//
15:00:52 <markstur> hi
15:00:53 <dustins> \o
15:01:00 <xyang1> hi
15:01:01 <zengyingzhe_> Hi
15:01:07 <zhongjun_> hi
15:01:18 <bswartz> #topic announcements
15:01:26 <Yogi1> hi
15:01:27 <ganso> hello
15:01:40 <bswartz> okay feature freeze is next week!
15:01:57 <bswartz> that means we're down to only 3 days or so to merge features for newton
15:02:35 <bswartz> the gate isn't cooperating as usual, and everything is also very slow as usual
15:03:03 <bswartz> that means everything not already merged is at risk of not making it
15:03:29 <bswartz> please prioritize gate fix issues and please don't abuse jenkins this coming week
15:03:52 <bswartz> also, non-client library feature freeze is TODAY
15:04:08 <bswartz> so we can consider manila-ui frozen for newton
15:05:04 <bswartz> there's more I can say about the state of automated tests and the feature freeze but I'll save that for later
15:05:32 <bswartz> #topic Huawei share replication and CI
15:05:54 <bswartz> zengyingzhe_, zhongjun_: who wants to take this topic?
15:06:25 <zhongjun_> zengyingzhe working on it
15:06:29 <zengyingzhe_> I'll talk about this.
15:06:39 <vkmc> o/
15:06:41 <vkmc> hi hi
15:06:57 * bswartz marks vkmc tardy
15:07:02 <bswartz> :-)
15:07:48 <zengyingzhe_> Here's the situation. For the replication of huawei driver, we need two huawei arrays to consist of the replication relaitonship.
15:08:31 <zengyingzhe_> But now, our CI env only got one array. so it can't do the replication tests.
15:09:34 <zengyingzhe_> we've already arranged a new array, but it'll take some time to setup, and integrate with CI env.
15:10:07 <gouthamr> zengyingzhe_: were you able to run these tests locally?
15:10:26 <gouthamr> zengyingzhe_: last i recall you found a bug in one of the tests..
15:10:42 <zengyingzhe_> gouthamr, yes, that's what I've been doing these days.
15:11:10 <gouthamr> finding bugs in replication? thanks! :D
15:11:19 <bswartz> based on my earlier conversation with zengyingzhe_, he was going to qualify the driver's replication support in his dev environment, where there are 2 arrays
15:11:25 <zengyingzhe_> I've passed some tests, but some of them failed, so I'm still working on it.
15:11:27 <bswartz> zengyingzhe_: were you able to do that?
15:11:48 <bswartz> zengyingzhe_: are the failures driver bugs or tests bugs?
15:12:27 <zengyingzhe_> mostly are driver bugs
15:13:07 <zengyingzhe_> bswartz, I think I can finish all of these tests dignose tomorrow.
15:13:27 <zengyingzhe_> and paste the test logs in my patch as you said.
15:13:49 <bswartz> okay
15:14:46 <bswartz> so the question for us then is are we okay merging the feature, without CI from huawei, as long as we can see that it does pass tempest (from the logs zengyingzhe_ will provide) and that CI will start covering this before the newton final release?
15:15:16 <cknight> bswartz: yes, given your caveat
15:15:25 <bswartz> personally I'm okay with this but it's a deviation from our normal requirement that new driver features need CI coverage to merge
15:15:49 <cknight> bswartz: a little grace goes a long way
15:15:51 <bswartz> anyone not okay with that proposal?
15:16:05 <markstur> ok
15:16:16 <zhongjun_> I am ok
15:16:52 <bswartz> okay I don't hear anyone disagreeing
15:16:56 <gouthamr> +1 zengyingzhe_: if you find bugs in the tests or the core feature, could you make a separate patch to fix these
15:17:13 <tbarron> +1
15:17:15 <bswartz> so zengyingzhe_ we're just waiting for you to post the logs of the driver passing all the tests
15:17:20 <zengyingzhe_> gouthamr, sure
15:17:31 <bswartz> of course the gate also has to cooperate if we're going to merge this
15:17:46 <bswartz> #topic IPv6
15:17:50 <zengyingzhe_> bswartz, thanks, I'll try my best to finish it soon.
15:17:53 <bswartz> cknight: you're up
15:17:57 <cknight> There has been a proposal up for a while to enable IPv6 in Manila, driven by our Huawei friends.  The areas affected most are API (access rules) and in the drivers (access rules & export locations).  DHSS=True drivers also have to care about network allocations.
15:18:07 <cknight> The original patch won't fly.  #link https://review.openstack.org/#/c/312321/
15:18:19 <cknight> But zhongjun_ and gouthamr have been working on a patch to enable simple IPv4 and IPv6 address validation in the Manila API layer for IP access rules.  #link https://review.openstack.org/#/c/359316/  Technically this patch missed FPF, but overlooking that...
15:18:35 <cknight> The patch replaces the cheesy homegrown IPv4 validation, and everything IPv4-related should still just work.  And some drivers work fine with IPv6, while others need changes.
15:18:46 <bswartz> which one? it looks like the one you linked was poasted May 3
15:18:49 <bswartz> posted
15:18:51 <cknight> The question is, do we go ahead with the later patch and allow IPv6 addresses to pass the API checks and flow into Newton Manila, or does the community need more time to assess the impacts, if any, in the rest of the core and all the drivers?
15:19:14 <cknight> If more time is needed, we could merge the first patch early in Ocata and have the whole release for everyone to support it.  I lean towards waiting, but I figured we should get a sense of the greater community.
15:19:20 <bswartz> oh zhongjun's
15:19:24 <zhongjun_> this code have been submit to gerrit since May 4. https://review.openstack.org/#/c/312321/
15:19:56 <bswartz> IPv6 support has been something we've wanted for a long time but nobody has made it a priority until huawei did
15:19:59 <zhongjun_> https://review.openstack.org/#/c/359316/ this patch just used for test
15:20:34 <bswartz> how much work is left to do?
15:20:48 <gouthamr> some drivers in tree make the assumption that both ipv4 and ipv6 addresses can be sent while asking for access
15:21:04 <gouthamr> so, i think it's a bug that the API disqualifies IPv6
15:21:11 <gouthamr> and another bug that we didn't have these tests
15:21:28 <gouthamr> and driver bugs if they made the assumption that ipv6 isn't supported
15:21:39 <zhongjun_> It will be a simple change when we need to support ipv6 format in ip access_type
15:21:50 <gouthamr> i don't recall any documentation that only v4 addresses are supportedd..
15:21:52 <bswartz> if it was done today I'd be willing to give it an FFE, but I suspect there's work left to be done and we can't be churning on API changes later in the release
15:22:16 <cknight> gouthamr:  I would call IPv6 a feature, and there may be other places in the core that need tweaks.
15:22:25 <zhongjun_> Does this feature need to be versioned?
15:23:06 <tbarron> zhongjun_: yeah, I was wondering about that ...
15:23:09 <bswartz> zhongjun_: that depends on the nature of the change -- if it's a "bugfix" because IPv6 was working in some cases and it stopped working then not really
15:23:30 <ganso> if adding support breaks anything, then I recommend not doing it... and it seams it does break
15:23:31 <zhongjun_> bswartz: Does we need to save the old validation code?
15:23:32 <bswartz> if IPv6 was never working and the patch makes it work, then it feels to end users like a feature and should be versioned
15:23:44 <cknight> bswartz: The access rules API rejected all except IPv4.
15:24:02 <bswartz> so why were some drivers allowing ipv6?
15:24:14 <bswartz> that could couldn't have been tested....
15:24:27 <ganso> bswartz: some drivers are "compatible"... they accept both because the backend accepts both
15:24:50 <cknight> ganso: +1  Ideally, the IP version doesn't matter to the driver.
15:24:51 <bswartz> do any drivers return IPv6 export locations?
15:25:18 <bswartz> because v6 access rules don't make much sense if you don't have v6 export locations
15:26:17 <ganso> still, that's still for some drivers. Other drivers are coded to accept only IPv4 because they know the API blocks IPv6. If all of a sudden it stops blocking, then drivers can suffer from errors they were not expecting
15:26:26 <zhongjun_> currently, no driver return IPv6 export locations
15:26:28 <ganso> and we need to avoid those ^
15:26:42 <cknight> ganso: It sounds like you want the feature to wait.
15:26:44 <bswartz> ganso: I see your point
15:26:48 <ganso> cknight: I do
15:26:54 <cknight> ganso: +1
15:27:10 <bswartz> well more than that issue, it sounds like there are some undecided issues around IPv6
15:27:42 <bswartz> as much as I'd love to have this feature yesterday, I don't think it's a good idea to merge it in newton -- we need to reach agreement on the design
15:27:46 * bswartz goes hunting for a spec
15:27:59 <ganso> it is too late in Newton to change this. The patch is not late per se, but no attention has been given to this, and this is very important and I believe it needs to be coordinated more carefully, just like "update_access". We could even have an internal extra_spec to prevent drivers from breaking
15:27:59 <cknight> bswartz: It sounds like we need an ML post telling everyone to get IPv6 ready in Ocata.
15:28:14 <zhongjun_> but, it is up to driver, it can simple support export locations with driver configure. and driver could not need to change any driver code.
15:28:28 <bswartz> I can't find a spec for ipv6 and I feel that we need one
15:28:52 <markstur> bswartz loves specs
15:28:54 <cknight> ganso: I thought about the extra spec, too, but it would be better to just know all the drivers are ready for it.  I wonder if we have any backends that just don't do IPv6.
15:28:57 <bswartz> if there are drivers that really can't support ipv6 then manila needs to know that and the end user needs to know that
15:29:19 <gouthamr> i was thinking we can note this in documentation
15:29:19 <bswartz> because the rest of the world does support ipv6
15:29:44 <zhongjun_> Because this bp submit before the specs
15:29:46 <bswartz> yeah I'd prefer to make the statement that all manila drivers MUST support ipv6 and therefor no capability bits are needed
15:30:16 <bswartz> however first we need to find out if that's going to be a problem for anyone
15:30:24 <cknight> *notes that bswartz has a data center in his attic that has been running IPv6 for nearly a decade.*
15:30:25 <bswartz> markstur: I'm coming around
15:30:36 <bswartz> cknight: +1
15:31:17 * bswartz remembers the days of the 6bone and the bad old days of 6in4 tunneling
15:31:45 <markstur> now us kids have to google 6bone.  Is that safe?
15:31:46 <zhongjun_> All manila drivers MUST support ipv6, Sounds it need a long time :)
15:32:03 <bswartz> yeah 6bone is SFW
15:32:31 <bswartz> zhongjun_: well we can find that out with a ML post and some test code
15:32:56 <bswartz> just write the IPv6 support into the API and driver layers, then add tempest tests and we see who fails, then we talk to those maintainers
15:33:01 <gouthamr> zhongjun_'s test code pointed a bug in the NetApp driver and teh fix was real simple
15:33:38 <zhongjun_> All manila drivers support ipv6 feature in one patch?
15:33:38 <gouthamr> but i agree we can dig into all our gate drivers and fix any issues before we claim support
15:34:08 <bswartz> zhongjun_: no not all at once, but possibly by end of ocata
15:34:31 <bswartz> what we want is to be able to tell users that IPv6 "just works" and that there aren't any limitations related to it
15:34:58 <bswartz> adding it in such a way that only some drivers support it or only some features support it will lead to bad user experiences
15:34:58 <zhongjun_> +1
15:35:40 <cknight> bswartz: Thanks, it sounds like there is consensus here.
15:36:02 <bswartz> so we should form a plan for ocata, including a brief spec, and a set of patches to fix the API limitations for ocata
15:36:34 <bswartz> then we need tests and then we need to hunt down driver maintainers and get them to fix their drivers
15:37:03 <bswartz> It occurs to me that properly testing IPv6 will REQUIRE scenario tests which is another thing I've been wanting to add to 3rd party CI tests
15:37:29 <zhongjun_> but it will be hard to hunt all of the driver maintainers
15:37:40 <bswartz> I'd love to create an IPv6-only neutron subnet and put a VM on it and have it mount the backend share
15:37:48 <bswartz> zhongjun_: leave that part to me
15:38:12 <zhongjun_> bswartz: ok
15:38:27 <zhongjun_> bswartz: :)
15:38:30 <bswartz> the key here will be getting all the infrastructure in place early in ocata do driver maintainers have time to fix their drivers
15:38:43 <bswartz> so let's not slow down on these ipv6 patches
15:39:07 <bswartz> let's plan to merge them during/before barcelona if possible
15:39:51 <bswartz> cknight: did we reach agreement on the usage of "ip" instead of "ipv4 and ipv6" for access rule types?
15:40:05 <cknight> bswartz: yes.  it's just 'ip'
15:40:28 <bswartz> okay good
15:40:34 <zhongjun_> How about microversion
15:41:01 <cknight> zhongjun_: yes, when the API begins accepting IPv6 addresses
15:41:04 <bswartz> zhongjun_: it sounds like there's no way it can work today and if we change that users will want to know, so we should microversion the API change
15:41:24 <bswartz> that way a users who depends on IPv6 support can set their minimum microversion to the version where we fixed this
15:41:43 <cknight> zhongjun_: Ideally there is a single core patch to enable IPv6, and that change bumps the microversion.
15:41:51 <bswartz> cknight: yes
15:42:05 <cknight> zhongjun_: I really don't think that will be a large patch.
15:42:19 <cknight> zhongjun_: But it will have a lot of tests.
15:42:34 <zhongjun_> cknight: yes, it is not a large patch
15:43:13 <bswartz> #topic open discussion
15:43:23 <dgonzalez> I would like to ask for reviews of https://review.openstack.org/#/c/283494/ Its the HPB base patch
15:43:27 <bswartz> alright anything else for today?
15:43:29 <dgonzalez> Now that the container driver has merged we were able to test this at the gate. So it would be great if we could merge it asap
15:43:36 <bswartz> dgonzalez: that was fast!
15:43:54 <dgonzalez> bswartz: Yeah I already typed it :D
15:44:01 <dgonzalez> I was prepared!
15:44:34 <bswartz> dgonzalez: k I will review
15:44:45 <tovchinnikova> :) Speaking about reviews: one high priority manila UI bug fix waiting for the reviews too: https://review.openstack.org/#/c/352889/
15:44:56 <dgonzalez> bswartz: great thanks
15:45:00 <bswartz> tovchinnikova: okay
15:45:20 <tovchinnikova> bswartz, thanks!
15:45:36 <bswartz> so there's a side conversation back in the channel that touches on the other thing I didn't say at the beginning
15:45:51 <bswartz> we have a lot of tests which "randomly" fail
15:46:05 <bswartz> and we have a culture here of just rechecking
15:46:15 <bswartz> that's not the right thing to do!
15:46:29 <bswartz> when tests fail, we should file bugs and investigate them, even if it's not 100% of the time
15:46:44 <bswartz> sometimes the investigation turns up an issue we have no control over, but that's very rare
15:47:04 <aovchinnikov> bswartz: file bugs against which project?
15:47:17 <bswartz> aovchinnikov: wherever the bug lies
15:47:56 <bswartz> we have some issues related to concurrency -- the fix for those is to implement better separation between concurrent things
15:48:18 <bswartz> we have some issues related to timing problem -- the fix for those is to add more state checking and add wait loops
15:48:47 <bswartz> we have some issues related to stuff being slow as hell -- the fix for those is to optimize or do less
15:49:20 <bswartz> many of these issues are hard to fix but if we just recheck around them they won't go away and they'll pile up until the gate is so bad we can't get any real work done
15:49:41 <bswartz> We've started a few initiatives in the past to try to solve some of the biggest problems we know about
15:49:53 <bswartz> some of those have succeeded, some have failed, and some are ongoing
15:50:25 <bswartz> I guess my only point is that these problems are everyone's responsibility
15:51:38 <bswartz> also these last 2 weeks are especially bad because our bug-fixer-in-chief vponomaryov is on vacation
15:52:08 <bswartz> vponomaryov come back and save us! :-D
15:52:29 <bswartz> alright that's all I had
15:52:42 <bswartz> let;s get back to reviewing so we can land at least a few things before FF
15:52:54 <bswartz> #endmeeting