15:00:28 #startmeeting manila 15:00:29 Meeting started Thu Jan 17 15:00:28 2019 UTC and is due to finish in 60 minutes. The chair is tbarron. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:00:30 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:00:32 The meeting name has been set to 'manila' 15:00:35 .o/ 15:01:00 hello 15:01:16 hi 15:01:26 ping amito 15:01:33 ping xyang 15:01:44 ping toabctl 15:01:47 hi 15:01:52 ping vkmc 15:02:12 gouthamr is on PTO till Tuesday 15:02:25 Hi folks 15:02:29 small group today 15:02:36 No trout slapping gouthamr today :-( 15:02:40 I'll wait a couple more minutes 15:03:15 bswartz: hopefully he's fishing or doing something fun in the Pacific NW besides watching Clemson football 15:03:40 ok, lets' start 15:04:01 #topic Agenda https://wiki.openstack.org/wiki/Manila/Meetings 15:04:17 it's short, so if you have updates please make them 15:04:48 and tell me to reload my browser 15:05:04 #topic Announcements 15:05:37 I indicated previously that I planned to do library releases (manila-ui, python-manilaclient) around M2 15:05:56 M2 was last week and I still haven't done them yet b/c 15:06:15 I want to get the functional job for py3 in the client merged first 15:06:20 it's imminent 15:06:42 The other announcement is that we didn't have a (virtual) mid-cycle this week even 15:06:54 though we had discussed doing so earlier. 15:07:17 Everyone I know was quite busy, not lacking for direction either. 15:07:47 Is anyone direly missing the mid-cycle this time? 15:08:25 OK, if you really want to do this, ping me and we'll talk about it. 15:08:29 o/ 15:08:31 What is the plan for sync ups going forward? 15:08:43 Otherwise I'm going to assume no need this cycle. 15:08:44 Are they going to be towards the middle of cycles or around release time? 15:08:50 bswartz: glad you asked. 15:09:08 PTG is scheduled for right after the Denver summit. 15:09:18 For this cycle I don't feel like the lack of a sync up is hurting me 15:09:40 I was polled asking if manila wanted space and replied yes. 15:09:47 And are the summits offset from the release by 4 weeks like before? 15:10:03 bswartz: umm, let's look. 15:10:10 Wait you said after the summit 15:10:36 summit is the first part of the last week in April, going into May 15:11:10 And release date is? 15:11:25 PTG is the last three days of that week, including Saturday 15:11:50 Stein releases 10 April 15:11:56 #link https://releases.openstack.org/stein/index.html 15:12:21 yeah, that's the one I was going to grab, ty 15:12:21 So summit Monday-Wednesday, PTG Thurs-Sat? 15:13:19 #link https://www.openstack.org/summit/denver-2019/ 15:13:43 bswartz: right 15:14:09 Okay so it's once again looking like the old days 15:14:21 and release is April 8-12 15:14:30 so a couple weeks after release 15:14:49 bswartz: so it may well make sense to do a mid-cycle in Train 15:15:04 cause as you say the cadence is like we used to have 15:15:11 Right 15:15:15 but let's deal with that then 15:15:24 If we want anything special, it would be the midcycle 15:15:30 No one is crying out for one in Stein. 15:15:32 But the PTG may be enough 15:15:37 right 15:15:48 ok 15:15:59 Anyone else have announcements? 15:16:23 #topic Python 3 jobs 15:16:57 All projects are supposed to have voting *functional* jobs for M2. 15:17:14 There's a "report card" that I've been filling in. 15:17:23 We're OK for manila proper now. 15:17:43 dummy driver job is running under python3 15:18:07 and really doing it, not running with a pastiche of py2 and py3 services :) 15:18:21 I'd like to get lvm job next 15:18:45 #link https://review.openstack.org/#/c/623061/ still isn't working quite right yet 15:19:02 manila-ui functional job is now py3 15:19:18 python-manilaclient was giving us trouble 15:19:45 I was going to merge changes to run it under python3 but skip a couple unicode tests because 15:20:12 so far as I could tell they were failing b/c of an issue in tempest/lib that we couldn't fix in manila itself 15:20:19 That distressed gouthamr 15:20:21 :) 15:21:02 The issue is that manila tempest uses tempestlib to open a pipe to a subprocess to run 'manila blah blah' commands 15:21:19 The tests put mandaring description and name fields in the command 15:21:41 and inside tempest lib (not in our code) the strings were being byte-encoded to ascii 15:21:55 so the >128 code points caused exceptions 15:22:32 gouthamr was starting PTO so he stayed up late and figured out that the problem was due to our tox environment 15:22:38 So we're ready to go. 15:22:44 Please review: 15:23:00 #link https://review.openstack.org/#/c/629536/ 15:23:02 and 15:23:07 Not string problems! 15:23:19 #link https://review.openstack.org/#/c/630462/ 15:23:22 Those always burn people moving from py2 to py3 15:23:30 I think we can merge these today. 15:23:39 (here, sorry, was afk for a while) 15:23:45 They fix the string probs and do so compatibly with py2 15:24:03 bswartz: yeah, and this one was particularly nasty 15:24:42 general rule is byte-encode at the edges and decode to strings (unicode types in py2) for all internal processing 15:25:02 six.text_type actually 15:25:04 but the popen.subprocess acts like an edge interface as far as that rule goes 15:25:08 bswartz: right 15:25:19 * bswartz dreams of the day when six isn't needed anymore 15:26:06 I think openstack and ceph might be the last 2 python projects anybody cares about still using py2 15:26:28 Once they move to py3, py2 can be left in the dustbin of history 15:26:33 ceph is right now talking about dropping py2 support in nautilus 15:26:48 tbarron: did you use 2to3 for the conversions? 15:26:55 or a similar tool? 15:27:02 amito: that was run on the codebase long ago 15:27:12 tbarron: oh ok 15:27:16 amito: and all the unit tests were made to work 15:27:31 amito: but running functional tests revealed some other issues 15:28:08 amito: like that our tox env was setting locale in a way that (subtly) broke the tempest.lib client calls under py3 15:28:53 Anyways, lets review and merge those two patches (unless someone sees something wrong) today 15:29:14 and I'll cut python-manilaclient and manila-ui releases tomorrow 15:29:28 Anything else on this topic? 15:29:40 tbarron: I have to drop unexpected meeting 15:30:02 jgrosso: ack, we'll see you here next week ! 15:30:18 yes I will make sure I have no other meetings 15:30:18 #topic Gate Issues 15:30:27 Wait 15:30:38 Just 1 changes needs reviewing for py3, or more than 1? 15:30:45 * tbarron screeches to a halt 15:30:46 bswartz: two 15:31:00 #link https://review.openstack.org/#/c/629536/ 15:31:02 and 15:31:16 #link https://review.openstack.org/#/c/630462/ 15:31:30 Okay thx 15:31:37 bswartz: ty! 15:31:49 #topic Gate Issues 15:32:02 I haven't done much on this front this week. 15:32:13 Nor so far as I can tell has anyone else but 15:32:32 next up is getting the manila-image-elements job to actually 15:32:47 publish to tarballs.o.o when a change merges 15:33:10 We merged a change to allow ssh login to the service image by key w/o password but 15:33:16 it didn't show up there. 15:33:19 tbarron: do you know how to do that? 15:33:34 I looked at it long ago and I thought I got that working, but it was hard to understand 15:33:45 bswartz: not sure, maybe just define the publish job in the post pipeline in zuul? 15:33:46 I might have done it wrong 15:33:58 bswartz: no you had it right before, I think we 15:34:01 Well there is a publish job 15:34:08 broke it when we moved the jobs in-tree 15:34:11 But it could be fragile or easy to break 15:34:14 Oh I see 15:34:34 I'll reach out to zuul/infra folks if it's not obvkous. 15:34:38 obvious. 15:35:09 I'm picking that as the next task because I *think* tempest ssh client tries to login w/o password. 15:35:13 Sometimes. 15:35:33 I would hope that would be configurable 15:35:35 Anyways, fixing it should help me test that conjecture. 15:35:54 bswartz: I *think* there are some places it isn't. 15:36:23 bswartz: so we might want to fix that too but if we can get pw/less login working that's great in its own right. 15:36:34 Other known gate issues: 15:36:51 1) intermittent db migration failures 15:37:15 2) host assisted migration - we discussed last week, let's see how bad it is after 15:37:23 getting the tempest ssh issues fixed 15:37:55 3) snapshot races in lvm driver? toabctl made some fixes in this area, maybe it's better 15:38:19 4) intermittent access-rule and replication issues in the dummy job 15:38:26 #4 means races I think 15:38:35 probably other stuff 15:38:47 There's a rich area for people to work here!! 15:39:26 Any other remakrs on this topic today? 15:39:56 #Topic Bugs 15:40:22 jgrosso got called to another meeting but he's working on a spreadsheet/tooling for our bug backlog 15:40:42 instead of midcycle we may schedule some synchrnous time for bug smashing 15:40:55 what could be more important than manila meeting? 15:40:56 but there's a week or two of organizational work to do still. 15:41:06 * bswartz slaps jgrosso around a bit with a large trout 15:41:15 bswartz: :) 15:41:21 managers 15:41:42 carthaca: do you have some favorite bugs to tell us about today? 15:42:18 trout-slapping leads tbarron to get people to lob bug-grenades towards ntap 15:42:48 ganso: do we have bugs? 15:42:50 k, I guess there aren't any netapp back end bugs then :) 15:43:12 except if you do replication or work at SAP scale 15:43:22 :) 15:43:42 thanks for asking - not this week :) 15:43:42 alright, we can move on 15:43:48 bswartz: not critical ones as far as I know 15:43:52 #topic Open Discussion 15:44:10 * tbarron makes a note to go mark more bugs Critical :) 15:44:30 tbarron: :P 15:44:41 Well only mark them as critical if they're critical 15:44:54 bswartz: of course, I was just messing around 15:45:40 Last week after our meeting xyang and I had an interesting discussion about snapshot capabilities in manila and in the CSI world 15:45:43 My rule was always that vendor bugs could never be higher than medium priority 15:45:58 bswartz: prob. makes sense 15:46:23 tbarron: yes:) 15:46:36 In manila we distinguish among (1) bare snap capability, (2) create from snap, (3) mountable snap, (4) revert to snap 15:46:58 in csi there is just "snapshot capability" and it implies "create from snap" 15:47:00 but 15:47:08 correct me xyang if I misspeak 15:47:11 Very much like cinder 15:47:24 In cinder create from snap was assumed 15:47:26 the create-from-snap doesn't have to be efficient 15:47:47 csi combined (1) and (2) into (1), but that was because there were no use cases at that time 15:47:53 IIRC nothing ever has to be efficient 15:47:55 (3) and (4) are not supported yet 15:48:02 that is, the create doesn't have to be a fixed-time operation not proportional to the amoiunt of data in the volume/share 15:48:12 CSI snapshots are allowed to be dog slow 15:48:17 right 15:48:19 xyang: but if (3) and (4) are ever to be supported, won't (2) need to split from (1)? 15:48:23 and I didn't understand why 15:48:29 but 15:48:33 maybe I do now 15:48:36 no one says "cut snapshot" should be slow though 15:48:45 xyang: right 15:48:48 because 15:48:55 again, correct if I misspeak 15:49:07 in CSI the whole game is application portability 15:49:17 ganso: I am going to propose to separate (2) from (1) 15:49:18 Well however the snapshot is implemented, there's an assumption that it represents a single point in time copy 15:49:26 so snapshots are useful for porting the storage an app needs across clouds 15:49:31 It was in my original proposal, but got removed after reviews 15:49:34 and that is never going to be efficient 15:49:41 no object for removal back then 15:49:49 your new volume may be ready tomorrow 15:49:49 You can't have a snapshot that's smeared across time (like just tarring up the whole filesystem) 15:50:31 bswartz: right, but you could use tar or rsync to move the snapshot data to a new destination, right? 15:50:47 the snapshot itself would have to be a single point in time 15:50:56 Depends what you're tarring 15:51:38 bswartz: tar might be the wrong tool but lets say you could put the snapshot on a tape and carry it to the new cloud and build a new volume/share from it there 15:51:40 I guess the other assumption worth pointing out is that taking a snapshot can't interrupt I/O for a noticeable amount of time 15:52:14 bswartz: because as I understand it CSI snaphsots quiesce the application around the snapshot operation 15:52:16 Because there are implementation that could just remount the FS readonly and tar that, but we wouldn't accept such a solution 15:52:52 bswartz: b/c the remounted FS is once again "live"? 15:53:16 No, because you'd be stuck in readonly mode until the tar completed, which could be minutes or hours 15:53:29 bswartz: i see, makes sense 15:53:41 tbarron: the quiesce part is still under development. Also it depends on what quiesce command user defines. 15:54:39 xyang: well if the goal is to be able to port an app from AWS to on-prem or vice versa with its required storage then app consistency rather than just crash consistency would seem necessary 15:55:12 or AWS to Azure, GCE, etc 15:55:36 Yes, but each application requires different way to do quiesce. So K8S is not going to provide the scripts for every application in the world 15:55:54 xyang: I see I've got a lot to learn :) 15:55:55 we will just provide a hook for user to add those commands 15:56:43 ah, maybe we need smart "operators" so poor applicaiton developers don't have to know what all to put in those hooks :) 15:57:02 Yeah people will need help 15:57:05 xyang: bswartz: thanks for explaining. Seems like 15:57:09 But it's a hard problem 15:57:32 we don't have to yell at CSI for not learning the lessons of manila w.r.t. snapshots then 15:57:33 It's always been a hard problem, and cloud infrastructure does nothing to make it easier 15:58:41 Well we're short on time. Anything else today? 15:59:22 Thanks everyone, see you in #openstack-manila !! 15:59:26 #endmeeting