15:00:28 <tbarron> #startmeeting manila
15:00:29 <openstack> Meeting started Thu Jan 17 15:00:28 2019 UTC and is due to finish in 60 minutes.  The chair is tbarron. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:30 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:32 <openstack> The meeting name has been set to 'manila'
15:00:35 <bswartz> .o/
15:01:00 <ganso> hello
15:01:16 <carthaca> hi
15:01:26 <tbarron> ping amito
15:01:33 <tbarron> ping xyang
15:01:44 <tbarron> ping toabctl
15:01:47 <xyang> hi
15:01:52 <tbarron> ping vkmc
15:02:12 <tbarron> gouthamr is on PTO till Tuesday
15:02:25 <tbarron> Hi folks
15:02:29 <tbarron> small group today
15:02:36 <bswartz> No trout slapping gouthamr today :-(
15:02:40 <tbarron> I'll wait a couple more minutes
15:03:15 <tbarron> bswartz: hopefully he's fishing or doing something fun in the Pacific NW besides watching Clemson football
15:03:40 <tbarron> ok, lets' start
15:04:01 <tbarron> #topic Agenda https://wiki.openstack.org/wiki/Manila/Meetings
15:04:17 <tbarron> it's short, so if you have updates please make them
15:04:48 <tbarron> and tell me to reload my browser
15:05:04 <tbarron> #topic Announcements
15:05:37 <tbarron> I indicated previously that I planned to do library releases (manila-ui, python-manilaclient) around M2
15:05:56 <tbarron> M2 was last week and I still haven't done them yet b/c
15:06:15 <tbarron> I want to get the functional job for py3 in the client merged first
15:06:20 <tbarron> it's imminent
15:06:42 <tbarron> The other announcement is that we didn't have a (virtual) mid-cycle this week even
15:06:54 <tbarron> though we had discussed doing so earlier.
15:07:17 <tbarron> Everyone I know was quite busy, not lacking for direction either.
15:07:47 <tbarron> Is anyone direly missing the mid-cycle this time?
15:08:25 <tbarron> OK, if you really want to do this, ping me and we'll talk about it.
15:08:29 <vkmc> o/
15:08:31 <bswartz> What is the plan for sync ups going forward?
15:08:43 <tbarron> Otherwise I'm going to assume no need this cycle.
15:08:44 <bswartz> Are they going to be towards the middle of cycles or around release time?
15:08:50 <tbarron> bswartz: glad you asked.
15:09:08 <tbarron> PTG is scheduled for right after the Denver summit.
15:09:18 <bswartz> For this cycle I don't feel like the lack of a sync up is hurting me
15:09:40 <tbarron> I was polled asking if manila wanted space and replied yes.
15:09:47 <bswartz> And are the summits offset from the release by 4 weeks like before?
15:10:03 <tbarron> bswartz: umm, let's look.
15:10:10 <bswartz> Wait you said after the summit
15:10:36 <tbarron> summit is the first part of the last week in April, going into May
15:11:10 <bswartz> And release date is?
15:11:25 <tbarron> PTG is the last three days of that week, including Saturday
15:11:50 <bswartz> Stein releases 10 April
15:11:56 <bswartz> #link https://releases.openstack.org/stein/index.html
15:12:21 <tbarron> yeah, that's the one I was going to grab, ty
15:12:21 <bswartz> So summit Monday-Wednesday, PTG Thurs-Sat?
15:13:19 <tbarron> #link https://www.openstack.org/summit/denver-2019/
15:13:43 <tbarron> bswartz: right
15:14:09 <bswartz> Okay so it's once again looking like the old days
15:14:21 <tbarron> and release is April 8-12
15:14:30 <tbarron> so a couple weeks after release
15:14:49 <tbarron> bswartz: so it may well make sense to do a mid-cycle in Train
15:15:04 <tbarron> cause as you say the cadence is like we used to have
15:15:11 <bswartz> Right
15:15:15 <tbarron> but let's deal with that then
15:15:24 <bswartz> If we want anything special, it would be the midcycle
15:15:30 <tbarron> No one is crying out for one in Stein.
15:15:32 <bswartz> But the PTG may be enough
15:15:37 <tbarron> right
15:15:48 <tbarron> ok
15:15:59 <tbarron> Anyone else have announcements?
15:16:23 <tbarron> #topic Python 3 jobs
15:16:57 <tbarron> All projects are supposed to have voting *functional* jobs for M2.
15:17:14 <tbarron> There's a "report card" that I've been filling in.
15:17:23 <tbarron> We're OK for manila proper now.
15:17:43 <tbarron> dummy driver job is running under python3
15:18:07 <tbarron> and really doing it, not running with a pastiche of py2 and py3 services :)
15:18:21 <tbarron> I'd like to get lvm job next
15:18:45 <tbarron> #link https://review.openstack.org/#/c/623061/ still isn't working quite right yet
15:19:02 <tbarron> manila-ui functional job is now py3
15:19:18 <tbarron> python-manilaclient was giving us trouble
15:19:45 <tbarron> I was going to merge changes to run it under python3 but skip a couple unicode tests because
15:20:12 <tbarron> so far as I could tell they were failing b/c of an issue in tempest/lib that we couldn't fix in manila itself
15:20:19 <tbarron> That distressed gouthamr
15:20:21 <tbarron> :)
15:21:02 <tbarron> The issue is that manila tempest uses tempestlib to open a pipe to a subprocess to run 'manila blah blah' commands
15:21:19 <tbarron> The tests put mandaring description and name fields in the command
15:21:41 <tbarron> and inside tempest lib (not in our code) the strings were being byte-encoded to ascii
15:21:55 <tbarron> so the >128 code points caused exceptions
15:22:32 <tbarron> gouthamr was starting PTO so he stayed up late and figured out that the problem was due to our tox environment
15:22:38 <tbarron> So we're ready to go.
15:22:44 <tbarron> Please review:
15:23:00 <tbarron> #link https://review.openstack.org/#/c/629536/
15:23:02 <tbarron> and
15:23:07 <bswartz> Not string problems!
15:23:19 <tbarron> #link https://review.openstack.org/#/c/630462/
15:23:22 <bswartz> Those always burn people moving from py2 to py3
15:23:30 <tbarron> I think we can merge these today.
15:23:39 <amito> (here, sorry, was afk for a while)
15:23:45 <tbarron> They fix the string probs and do so compatibly with py2
15:24:03 <tbarron> bswartz: yeah, and this one was particularly nasty
15:24:42 <tbarron> general rule is byte-encode at the edges and decode to strings (unicode types in py2) for all internal processing
15:25:02 <bswartz> six.text_type actually
15:25:04 <tbarron> but the popen.subprocess acts like an edge interface as far as that rule goes
15:25:08 <tbarron> bswartz: right
15:25:19 * bswartz dreams of the day when six isn't needed anymore
15:26:06 <bswartz> I think openstack and ceph might be the last 2 python projects anybody cares about still using py2
15:26:28 <bswartz> Once they move to py3, py2 can be left in the dustbin of history
15:26:33 <tbarron> ceph is right now talking about dropping py2 support in nautilus
15:26:48 <amito> tbarron: did you use 2to3 for the conversions?
15:26:55 <amito> or a similar tool?
15:27:02 <tbarron> amito: that was run on the codebase long ago
15:27:12 <amito> tbarron: oh ok
15:27:16 <tbarron> amito: and all the unit tests were made to work
15:27:31 <tbarron> amito: but running functional tests revealed some other issues
15:28:08 <tbarron> amito: like that our tox env was setting locale in a way that (subtly) broke the tempest.lib client calls under py3
15:28:53 <tbarron> Anyways, lets review and merge those two patches (unless someone sees something wrong) today
15:29:14 <tbarron> and I'll cut python-manilaclient and manila-ui releases tomorrow
15:29:28 <tbarron> Anything else on this topic?
15:29:40 <jgrosso> tbarron: I have to drop unexpected meeting
15:30:02 <tbarron> jgrosso: ack, we'll see you here next week !
15:30:18 <jgrosso> yes I will make sure I have no other meetings
15:30:18 <tbarron> #topic Gate Issues
15:30:27 <bswartz> Wait
15:30:38 <bswartz> Just 1 changes needs reviewing for py3, or more than 1?
15:30:45 * tbarron screeches to a halt
15:30:46 <tbarron> bswartz: two
15:31:00 <tbarron> #link https://review.openstack.org/#/c/629536/
15:31:02 <tbarron> and
15:31:16 <tbarron> #link https://review.openstack.org/#/c/630462/
15:31:30 <bswartz> Okay thx
15:31:37 <tbarron> bswartz: ty!
15:31:49 <tbarron> #topic Gate Issues
15:32:02 <tbarron> I haven't done much on this front this week.
15:32:13 <tbarron> Nor so far as I can tell has anyone else but
15:32:32 <tbarron> next up is getting the manila-image-elements job to actually
15:32:47 <tbarron> publish to tarballs.o.o when a change merges
15:33:10 <tbarron> We merged a change to allow ssh login to the service image by key w/o password but
15:33:16 <tbarron> it didn't show up there.
15:33:19 <bswartz> tbarron: do you know how to do that?
15:33:34 <bswartz> I looked at it long ago and I thought I got that working, but it was hard to understand
15:33:45 <tbarron> bswartz: not sure, maybe just define the publish job in the post pipeline in zuul?
15:33:46 <bswartz> I might have done it wrong
15:33:58 <tbarron> bswartz: no  you had it right before, I think we
15:34:01 <bswartz> Well there is a publish job
15:34:08 <tbarron> broke it when we moved the jobs in-tree
15:34:11 <bswartz> But it could be fragile or easy to break
15:34:14 <bswartz> Oh I see
15:34:34 <tbarron> I'll reach out to zuul/infra folks if it's not obvkous.
15:34:38 <tbarron> obvious.
15:35:09 <tbarron> I'm picking that as the next task because I *think* tempest ssh client tries to login w/o password.
15:35:13 <tbarron> Sometimes.
15:35:33 <bswartz> I would hope that would be configurable
15:35:35 <tbarron> Anyways, fixing it should help me test that conjecture.
15:35:54 <tbarron> bswartz: I *think* there are some places it isn't.
15:36:23 <tbarron> bswartz: so we might want to fix that too but if we can get pw/less login working that's great in its own right.
15:36:34 <tbarron> Other known gate issues:
15:36:51 <tbarron> 1) intermittent db migration failures
15:37:15 <tbarron> 2) host assisted migration - we discussed last week, let's see how bad it is after
15:37:23 <tbarron> getting the tempest ssh issues fixed
15:37:55 <tbarron> 3) snapshot races in lvm driver?  toabctl made some fixes in this area, maybe it's better
15:38:19 <tbarron> 4) intermittent access-rule and replication issues in the dummy job
15:38:26 <tbarron> #4 means races I think
15:38:35 <tbarron> probably other stuff
15:38:47 <tbarron> There's a rich area for people to work here!!
15:39:26 <tbarron> Any other remakrs on this topic today?
15:39:56 <tbarron> #Topic Bugs
15:40:22 <tbarron> jgrosso got called to another meeting but he's working on a spreadsheet/tooling for our bug backlog
15:40:42 <tbarron> instead of midcycle we may schedule some synchrnous time for bug smashing
15:40:55 <bswartz> what could be more important than manila meeting?
15:40:56 <tbarron> but there's a week or two of organizational work to do still.
15:41:06 * bswartz slaps jgrosso around a bit with a large trout
15:41:15 <tbarron> bswartz: :)
15:41:21 <tbarron> managers
15:41:42 <tbarron> carthaca: do you have some favorite bugs to tell us about today?
15:42:18 <tbarron> trout-slapping leads tbarron to get people to lob bug-grenades towards ntap
15:42:48 <bswartz> ganso: do we have bugs?
15:42:50 <tbarron> k, I guess there aren't any netapp back end bugs then :)
15:43:12 <tbarron> except if you do replication or work at SAP scale
15:43:22 <tbarron> :)
15:43:42 <carthaca> thanks for asking - not this week :)
15:43:42 <tbarron> alright, we can move on
15:43:48 <ganso> bswartz: not critical ones as far as I know
15:43:52 <tbarron> #topic Open Discussion
15:44:10 * tbarron makes a note to go mark more bugs Critical :)
15:44:30 <ganso> tbarron: :P
15:44:41 <bswartz> Well only mark them as critical if they're critical
15:44:54 <tbarron> bswartz: of course, I was just messing around
15:45:40 <tbarron> Last week after our meeting xyang and I had an interesting discussion about snapshot capabilities in manila and in the CSI world
15:45:43 <bswartz> My rule was always that vendor bugs could never be higher than medium priority
15:45:58 <tbarron> bswartz: prob. makes sense
15:46:23 <xyang> tbarron:  yes:)
15:46:36 <tbarron> In manila we distinguish among (1) bare snap capability, (2) create from snap, (3) mountable snap, (4) revert to snap
15:46:58 <tbarron> in csi there is just "snapshot capability" and it implies "create from snap"
15:47:00 <tbarron> but
15:47:08 <tbarron> correct me xyang if I misspeak
15:47:11 <bswartz> Very much like cinder
15:47:24 <bswartz> In cinder create from snap was assumed
15:47:26 <tbarron> the create-from-snap doesn't have to be efficient
15:47:47 <xyang> csi combined (1) and (2) into (1), but that was because there were no use cases at that time
15:47:53 <bswartz> IIRC nothing ever has to be efficient
15:47:55 <xyang> (3) and (4) are not supported yet
15:48:02 <tbarron> that is, the create doesn't have to be a fixed-time operation not proportional to the amoiunt of data in the volume/share
15:48:12 <bswartz> CSI snapshots are allowed to be dog slow
15:48:17 <tbarron> right
15:48:19 <ganso> xyang: but if (3) and (4) are ever to be supported, won't (2) need to split from (1)?
15:48:23 <tbarron> and I didn't understand why
15:48:29 <tbarron> but
15:48:33 <tbarron> maybe I do now
15:48:36 <xyang> no one says "cut snapshot" should be slow though
15:48:45 <tbarron> xyang: right
15:48:48 <tbarron> because
15:48:55 <tbarron> again, correct if I misspeak
15:49:07 <tbarron> in CSI the whole game is application portability
15:49:17 <xyang> ganso: I am going to propose to separate (2) from (1)
15:49:18 <bswartz> Well however the snapshot is implemented, there's an assumption that it represents a single point in time copy
15:49:26 <tbarron> so snapshots are useful for porting the storage an app needs across clouds
15:49:31 <xyang> It was in my original proposal, but got removed after reviews
15:49:34 <tbarron> and that is never going to be efficient
15:49:41 <xyang> no object for removal back then
15:49:49 <tbarron> your new volume may be ready tomorrow
15:49:49 <bswartz> You can't have a snapshot that's smeared across time (like just tarring up the whole filesystem)
15:50:31 <tbarron> bswartz: right, but you could use tar or rsync to move the snapshot data to a new destination, right?
15:50:47 <tbarron> the snapshot itself would have to be a single point in time
15:50:56 <bswartz> Depends what you're tarring
15:51:38 <tbarron> bswartz: tar might be the wrong tool but lets say you could put the snapshot on a tape and carry it to the new cloud and build a new volume/share from it there
15:51:40 <bswartz> I guess the other assumption worth pointing out is that taking a snapshot can't interrupt I/O for a noticeable amount of time
15:52:14 <tbarron> bswartz: because as I understand it CSI snaphsots quiesce the application around the snapshot operation
15:52:16 <bswartz> Because there are implementation that could just remount the FS readonly and tar that, but we wouldn't accept such a solution
15:52:52 <tbarron> bswartz: b/c the remounted FS is once again "live"?
15:53:16 <bswartz> No, because you'd be stuck in readonly mode until the tar completed, which could be minutes or hours
15:53:29 <tbarron> bswartz: i see, makes sense
15:53:41 <xyang> tbarron: the quiesce part is still under development.  Also it depends on what quiesce command user defines.
15:54:39 <tbarron> xyang: well if the goal is to be able to port an app from AWS to on-prem or vice versa with its required storage then app consistency rather than just crash consistency would seem necessary
15:55:12 <tbarron> or AWS to Azure, GCE, etc
15:55:36 <xyang> Yes, but each application requires different way to do quiesce.  So K8S is not going to provide the scripts for every application in the world
15:55:54 <tbarron> xyang: I see I've got a lot to learn :)
15:55:55 <xyang> we will just provide a hook for user to add those commands
15:56:43 <tbarron> ah, maybe we need smart "operators" so poor applicaiton developers don't have to know what all to put in those hooks :)
15:57:02 <bswartz> Yeah people will need help
15:57:05 <tbarron> xyang: bswartz: thanks for explaining.  Seems like
15:57:09 <bswartz> But it's a hard problem
15:57:32 <tbarron> we don't have to yell at CSI for not learning the lessons of manila w.r.t. snapshots then
15:57:33 <bswartz> It's always been a hard problem, and cloud infrastructure does nothing to make it easier
15:58:41 <tbarron> Well we're short on time.  Anything else today?
15:59:22 <tbarron> Thanks everyone, see you in #openstack-manila !!
15:59:26 <tbarron> #endmeeting