15:00:19 <tbarron> #startmeeting manila
15:00:20 <openstack> Meeting started Thu Feb 14 15:00:19 2019 UTC and is due to finish in 60 minutes.  The chair is tbarron. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:22 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:24 <openstack> The meeting name has been set to 'manila'
15:00:30 <tbarron> ping ganso
15:00:34 <tbarron> ping vkmc
15:00:36 <gouthamr> o/
15:00:48 <tbarron> ping xyang
15:00:53 <tbarron> ping toabctl
15:00:54 <ganso> hello
15:00:59 <tbarron> ping bswartz
15:01:01 <toabctl> hi
15:01:05 <xyang> hi
15:01:07 <tbarron> ping erlon
15:01:11 <tbarron> tping tpsilva
15:01:14 <vkmc> o/
15:01:18 <tbarron> ping tpsilva
15:01:24 <tbarron> ping amito
15:01:37 <tbarron> that's our ping list, minus a couple not online
15:01:50 * tbarron waits 2 minutes
15:02:31 <amito> Hey
15:03:09 <tbarron> OK, let's get started.
15:03:13 <tbarron> Hello all!
15:03:22 <tbarron> #topic Announcements
15:03:37 <tbarron> I don't have any.  Anyone else?
15:04:28 <tbarron> #topic Vision Reflection
15:04:36 <bswartz> .o/
15:04:46 <tbarron> hi bswartz
15:05:13 <tbarron> If you recall Zane Bitter met with us last November regarding the
15:05:41 <tbarron> (then draft) OpenStack Vision Statement up before the Technical Committee
15:06:01 <tbarron> Since then it merged as a governance doc and
15:06:17 <tbarron> each project was asked to read it over and produce a
15:06:33 <tbarron> "Reflection" document noting gaps relative to the vision.
15:06:56 <tbarron> These gaps would then be the basis for work plans going forwards.
15:07:23 <tbarron> Here's a reference to the relevant email thread:
15:07:34 <tbarron> #link http://lists.openstack.org/pipermail/openstack-discuss/2019-February/002678.html
15:07:55 <tbarron> So I took a first stab at this:
15:08:09 <tbarron> #link https://review.openstack.org/#/c/636770/
15:08:55 <tbarron> Please look it over and use the review to note disagreements, stuff missing, and so on.
15:09:46 <tbarron> Let's use the review for substantive remarks, but any questions or remarks now about what we're trying to do with this document?
15:10:13 <tbarron> ok, hearing none ...
15:10:21 <tbarron> #topic Schedule reminders
15:10:42 <tbarron> #link https://releases.openstack.org/stein/schedule.html
15:11:06 <tbarron> feature proposal freeze is 21 February
15:11:22 <tbarron> feature freeze and client library freeze is 7 March
15:11:34 <tbarron> just reminders.
15:11:40 <tbarron> questions? remarks?
15:11:53 <gouthamr> we have a new driver proposed :s
15:12:01 <tbarron> #topic Inspur Instorage driver
15:12:18 <tbarron> new driver submission deadline as 10 January for Stein
15:12:35 <tbarron> so I put a -2 on the review but added
15:12:40 <tbarron> -- I hope --
15:13:01 <vkmc> shall we submit a spec for openstackclient before march 7th?
15:13:05 <vkmc> it's a community goal for train
15:13:09 <tbarron> a nice note indicating that they can work the review cycle
15:13:24 <tbarron> now and get CI working now and we'll merge them asap in Train.
15:13:37 <tbarron> vkmc: do we need a spec?
15:13:52 <tbarron> gouthamr: ....^?
15:13:55 <vkmc> tbarron, well, yes
15:14:09 <tbarron> vkmc: for trying to recruit interns?
15:14:29 <vkmc> tbarron, no, I'm thinking more on leaving a record of how the new cli would look like
15:14:52 <bswartz> It's a bit late for new drivers isn't it?
15:15:00 <gouthamr> we kinda agreed we don't need a spec at the PTG, but i'm not opposed to it..
15:15:11 <tbarron> bswartz: that's why I put a -2 on the review
15:15:27 <tbarron> bswartz: deadline was 10 January
15:15:31 <bswartz> Is anyone asking for an exception for that driver?
15:15:40 <bswartz> Or everyone is okay with targeting next release?
15:15:47 <tbarron> bswartz: no exception requested so far, just a review posted
15:16:06 <tbarron> bswartz: I didn't see CI accompanying it
15:16:20 <tbarron> bswartz: if they get CI going and ask then we can talk about it
15:17:13 <tbarron> vkmc: gouthamr: who is going to do the spec?
15:17:25 <vkmc> tbarron, I can work on that
15:17:39 <tbarron> vkmc: gouthamr: it would be a Train spec I think but getting it started now would be great
15:17:47 <vkmc> sounds good
15:18:11 <tbarron> vkmc: cool.  gouthamr has done some initial work there.  I think you know how to find him.
15:18:13 <vkmc> I know gouthamr already have some stuff advanced, so we can work together
15:18:17 <vkmc> yes
15:18:36 * gouthamr moves to different undisclosed location
15:18:39 * tbarron enjoys seeing his kids work on projects together
15:18:43 <gouthamr> :D
15:19:18 <tbarron> Anything else on schedule or new drivers or the openstack client?
15:19:44 <gouthamr> nope..
15:20:02 <tbarron> I'll note that the osc gap and UI api microversion backlog are mentioned in the draft Vision Reflection doc
15:20:13 <tbarron> #topic tracking our work
15:20:35 <tbarron> the AZ work is progressing well so the main thing
15:20:42 <vkmc> :D
15:20:50 <tbarron> I want to call people's attention to given our
15:21:11 <tbarron> deadlines is ganso's work on manage-unmanage-with-share-servers
15:21:30 <tbarron> he's posted a number of patches recently
15:21:37 <tbarron> #link https://review.openstack.org/#/q/topic:bp/manage-unmanage-with-share-servers+(status:open+OR+status:merged)
15:21:39 <ganso> tbarron: thanks Tom
15:21:59 <tbarron> Please put your review eyeballs on and look at them.
15:22:19 <tbarron> Give them an initial review even if it's clear there's more work to do.
15:23:01 <tbarron> ganso: anything you want to say on this topic?
15:23:19 <ganso> tbarron: the only patch remaining is the manila-tempest-plugin one, the other patches I don't have anything else to work on, unless there are review comments or bugs found on the tempest patch
15:23:24 <gouthamr> ganso: i don't see a driver implementation
15:23:30 <ganso> gouthamr: there are 2
15:23:36 <ganso> gouthamr: it is in that wiki page
15:24:11 <ganso> #LINK https://wiki.openstack.org/wiki/Manila/SteinCycle#Manage.2FUnmanage_Share_Servers
15:24:20 <gouthamr> ah thanks!
15:24:39 <tbarron> sorry, I should have pasted that link instead of just that topic
15:24:55 <gouthamr> ganso: you can set a common topic across all of them.. makes them easier to find
15:25:37 <ganso> gouthamr: will do
15:25:43 <tbarron> ganso: thanks
15:25:50 <tbarron> ok ....
15:26:00 <tbarron> #topic Bugs
15:26:15 <ganso> gouthamr: just did =)
15:26:20 <gouthamr> thanks ganso
15:26:24 <tbarron> jgrosso isn't here today but ...
15:26:37 <tbarron> gouthamr has been working with him I hear
15:26:50 <tbarron> gouthamr: do you have any bugs for us to consider today?
15:27:14 <tbarron> if not, that's cool, I just sprung this on you ...
15:27:22 <gouthamr> tbarron: yes, dunno if you guys can see this google doc: https://docs.google.com/spreadsheets/d/1oaXEgo_BEkY2KleISN3M58waqw9U5W7xTR_O1jQmQ74/edit#gid=758082340
15:27:47 <tbarron> gouthamr: I don't think so, that's why I didn't link it in the agenda
15:28:00 <tbarron> gouthamr: was going to talk with jgrosso about that ....
15:28:32 <tbarron> we'll get it moved to a public place
15:28:35 <gouthamr> yes
15:28:39 <gouthamr> we should use https://ethercalc.openstack.org/uc8b4567fpf4 perhaps
15:28:45 <tbarron> yup
15:28:59 <tbarron> but those who can't see it, it's awesome :)
15:30:31 <gouthamr> Let's start with triaging this one: https://bugs.launchpad.net/manila/+bug/1815532
15:30:36 <openstack> Launchpad bug 1815532 in Manila "manila does not return the request ID when using shrink and extend APIs" [High,New]
15:31:29 <gouthamr> was found while reviewing a scenario test that nirg (Nir Galboa) was writing
15:31:32 <gouthamr> #LINK https://review.openstack.org/#/c/531615/
15:31:46 <bswartz> I can't remember where request IDs appear in the API
15:31:49 <bswartz> Do you have an example?
15:32:15 <gouthamr> bswartz: in my brief experimentation, it was whenever there was a response object of any kind being returned
15:32:44 <gouthamr> bswartz: these APIs are some of the few that only return a code on success
15:32:48 <bswartz> How do those get persisted?
15:33:10 <gouthamr> they don't, the middleware just sets the response header from the request header
15:33:34 <gouthamr> i.e, manila makes one up afaiu
15:35:41 <tbarron> just so I understand, why is this High?
15:36:13 <tbarron> it's admin only, right?  what will the admin not be able to do, track the request across services?
15:36:43 <gouthamr> tbarron: true, coupled with another bug affecting shrink share, it felt like a significant bug
15:36:46 * tbarron observes that many bugs seem to be set to High w/o explanation of why
15:37:25 <gouthamr> #LINK https://bugs.launchpad.net/manila/+bug/1810315
15:37:26 <openstack> Launchpad bug 1810315 in Manila "Error status on File Share Creation" [Undecided,Invalid]
15:37:32 <tbarron> gouthamr: I'm not so much arguing that it shouldn't be, just trying to make sure we understand the impact
15:37:40 <gouthamr> ouch, wrong link
15:38:21 <ganso> looks like a generic driver bug ^
15:38:44 <gouthamr> #LINK https://bugs.launchpad.net/manila/+bug/1802424
15:38:45 <openstack> Launchpad bug 1802424 in Manila "No user message when share shrinking fails at the driver" [Undecided,New]
15:38:46 <bswartz> Oh THAT request ID
15:38:58 <bswartz> The ephemeral one that's used for logging
15:39:04 <gouthamr> yes
15:40:01 <bswartz> Must be an API bug
15:40:09 <tbarron> so no mystery about RCA, just need to fix, right?
15:40:33 <tbarron> "root cause analysis"
15:40:46 * tbarron turns off his jargon button
15:41:14 <gouthamr> yes, i suppose we need someone who has some cycles to investigate and fix - don't expect it to be a large effort
15:41:59 <gouthamr> but i think we ought to backport this, even if it has APIImpact, because this is useful for support purposes
15:42:04 <bswartz> I'm not sure that's a nigh priority
15:42:08 <bswartz> High even
15:42:25 <bswartz> The impact would mostly be on logging and troubleshooting
15:43:08 <tbarron> inclined to agree with bswartz but think it's probably pretty low hanging fruit.  Worth fixing though.
15:43:26 <bswartz> Easy to fix doesn't mean high priority
15:43:27 <gouthamr> sure, I'll bump down the priority
15:43:31 <tbarron> and probably one can just copy the way it's done for the other APIs
15:43:37 <tbarron> bswartz: +1
15:44:01 <tbarron> ideally we'd target to fix by M3 and lower the prio
15:44:59 <tbarron> anyone want this one?
15:45:18 <gouthamr> nope, moving on
15:45:21 <gouthamr> #LINK https://bugs.launchpad.net/manila/+bug/1808127
15:45:22 <openstack> Launchpad bug 1808127 in Manila "intermittent tempest duplicate access rule failure" [Undecided,New]
15:45:38 <tbarron> this one is very annoying
15:46:35 <tbarron> I see it with the ceph back ends but so far as I can tell it shouldn't have anything to do with which back end is being used.
15:46:50 <tbarron> or am I confused about that?
15:47:35 <tbarron> the api is supposed to detect duplicates, right?
15:48:18 <gouthamr> the error seems to be with the timeout
15:48:28 <gouthamr> #LINK https://github.com/openstack/manila-tempest-plugin/blob/b96188/manila_tempest_tests/tests/api/test_rules_negative.py#L138
15:48:35 <tbarron> it sets more than one duplicate in the same test, could there be a teardown cleanup issue?
15:48:36 <bswartz> Duplicates should be detected and rejected
15:48:43 <gouthamr> that's the test the bug references ^
15:48:44 <bswartz> Right?
15:48:56 <tbarron> bswartz: yeah, that's the point of the test
15:49:26 <gouthamr> "Share's access_rules_status failed to transition to active within the required time 500" and this issue is intermittent?
15:50:03 <tbarron> gouthamr: to transition to active the backend has to say it did the job, right?
15:50:29 <gouthamr> tbarron: did you happen to see this with IPv4 rules, or has it been only with IPv6 - asking because we do have the ignore-ipv6-rule behavior if the driver doesn't support IPv6
15:50:39 <gouthamr> tbarron: yeah, not always
15:50:59 <tbarron> gouthamr: dunno if it was only ipv6 or not
15:51:21 <gouthamr> tbarron: we'll probably have to start debugging down that path, if the manager doesn't change the state back to "active" that may be our bug...
15:51:31 <tbarron> cephfs with nfs currently doesn't support ipv6 -- really only b/c we haven't tested it, I think it will work
15:51:37 <gouthamr> tbarron: after ignoring the ipv6 rules*
15:51:57 <tbarron> gouthamr: I can take this one
15:52:20 <tbarron> gouthamr: will likely need to consult with you though
15:52:25 <gouthamr> tbarron: cool, clicking the buttons...
15:53:13 <gouthamr> next one:
15:53:15 <gouthamr> #LINK https://bugs.launchpad.net/manila/+bug/1807126
15:53:16 <openstack> Launchpad bug 1807126 in Manila "Cannot create shares using generic driver" [High,Confirmed]
15:54:32 <gouthamr> so the bug has notes, but is still valid/confirmed
15:54:48 <gouthamr> have we discussed this one before?
15:55:01 <tbarron> well I don't think the underlying issues for ssh have been fixed, just mitigated
15:55:16 <tbarron> I suspect some kind of neutron-ovs issue
15:55:37 <tbarron> we've set the paramiko timeouts now though so that the initial ssh connection
15:55:44 <tbarron> doesn't fail
15:55:50 <tbarron> it just takes too long
15:56:01 <tbarron> and then that connection gets re-used
15:56:08 <tbarron> so carlos is probably ok
15:56:32 <tbarron> our gate jobs seem to work in this aspect now and they were broken
15:56:53 <gouthamr> tbarron: ah, okay.. i recall you mentioning that ML2/OVN doesn't have this behavior
15:57:01 <bswartz> What's the problem with the re-use?
15:57:01 <tbarron> there's still issues with the tempest ssh client where we don't have control over all the paramiko knobs
15:57:16 <bswartz> Should be not be using ssh connection pools?
15:57:21 <bswartz> Should we*
15:57:28 <tbarron> bswartz: the re-use is good, we don't have to wait all over again, and that part is working
15:58:10 <tbarron> it's something about the initial ssh header exchange.  I suspect an mtu issue but I wasn't able to "fix" the problem with my mtu experiments.
15:58:13 <bswartz> Okay
15:58:37 <bswartz> Oh MTU issues are devilish
15:58:38 <tbarron> and as goutham mentioned I was experimenting with OVN for other reasons and
15:58:41 <bswartz> Yes
15:58:47 <tbarron> the issue seems not to appear there
15:58:47 <bswartz> What did you experiment with?
15:58:59 <bswartz> Can we hack the tap device that we create to have a lower MTU?
15:59:02 <bswartz> That might make a huge difference
15:59:12 <tbarron> OVN, the next gen thing that will replace OVS
15:59:30 <tbarron> the ovs-vswitch commands are still there but there are additional capabilities
15:59:59 <tbarron> I was using developer version of OVN and it might be that dev version of OVS would also work
16:00:30 <tbarron> bswartz: I tried setting the mtu down on tap and on the VM's interface and it wasn't sufficient
16:00:47 <gouthamr> time check
16:00:50 <tbarron> like down to 512 or lower
16:00:50 <bswartz> Down how low?
16:00:55 <bswartz> Oh wow
16:00:58 <tbarron> gouthamr: thanks
16:01:09 <bswartz> I would have guessed 1300 was low enough
16:01:09 <tbarron> we can continue in #openstack-manila
16:01:14 <tbarron> Thanks everyone!
16:01:19 <tbarron> #endmeeting