16:00:56 <jungleboyj> #startmeeting cinder
16:00:57 <openstack> Meeting started Wed Jan 30 16:00:56 2019 UTC and is due to finish in 60 minutes.  The chair is jungleboyj. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:58 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:01:00 <openstack> The meeting name has been set to 'cinder'
16:01:08 <yikun> o/
16:01:12 <davidsha> o/
16:01:13 <whoami-rajat> Hi
16:01:19 <jungleboyj> courtesy ping:  jungleboyj diablo_rojo, diablo_rojo_phon, rajinir tbarron xyang xyang1 e0ne gouthamr thingee erlon tpsilva ganso patrickeast tommylikehu eharney geguileo smcginnis lhx_ lhx__ aspiers jgriffith moshele hwalsh felipemonteiro lpetrut lseki _alastor_ whoami-rajat yikun rosmaita enriquetaso
16:01:19 <avishay> hello
16:01:21 <woojay> Hello.
16:01:23 <jungleboyj> @!
16:01:23 <_pewp_> jungleboyj ( ・_・)ノ
16:01:27 <eharney> hi
16:01:28 <smcginnis> o/
16:01:30 <rosmaita> o/
16:01:31 <geguileo> hi! o/
16:01:34 <erlon> hey
16:01:41 <ganso> hello
16:02:11 <e0ne> hi
16:02:17 * jungleboyj can't respond as my eyes are frozen right now.
16:02:39 <rosmaita> how bad is it up there?
16:02:57 <jungleboyj> -22 in my backyard feels like -30
16:03:14 <rosmaita> ouch
16:03:17 <jungleboyj> That is an improvement from -28 when I woke up with windchills in the -40s
16:03:31 <erlon> jungleboyj, celcius?
16:03:41 <smcginnis> For comparison, it's currently -4F at the south pole.
16:03:51 <jungleboyj> No, Farenheit
16:04:02 <smcginnis> It's almost to the point where C and F match. :)
16:04:14 <jungleboyj> What is that -52 ?
16:04:21 <smcginnis> -40
16:04:24 <whoami-rajat> jungleboyj: stay safe!
16:04:42 <jungleboyj> :-)  I will leave the house to get my boys and that is about it.
16:04:46 <e0ne> jungleboyj: -22F or -22C?
16:04:55 <jungleboyj> -22F
16:05:15 <jungleboyj> Anyway, looks like we have the usual suspects.
16:05:18 <e0ne> oh.. it's really cold
16:05:32 <jungleboyj> #topic announcements
16:05:51 <jungleboyj> So, a reminder that we have the mid-cycle happening next week.
16:06:43 <jungleboyj> Based on the etherpad it looks like it will be Sean, Walt, Eric, Brian, Jon and I on site?
16:07:07 <smcginnis> jungleboyj: Are you going to bring your A/V setup for streaming?
16:07:17 <jungleboyj> Others will try to join remotely.
16:07:21 <jungleboyj> smcginnis:  Yes, I will be doing that.
16:07:38 <smcginnis> Great, hopefully we can have a decent online experience then.
16:07:48 <jungleboyj> *Fingers crossed*
16:08:04 <jungleboyj> We will be in a quiet room and it isn't too big.
16:08:14 <jungleboyj> hemna:  Are you around?
16:08:57 <jungleboyj> There will be a good phone in the room so it would be nice if we could figure out how to do that trick where he linked a phone call to the meetings.
16:09:20 <smcginnis> I think I still have notes on how we did that in Ft Collins.
16:09:35 <jungleboyj> Ok.  Cool.  If you could get those to me it would be appreciated.
16:09:55 <jungleboyj> Security is super tight at Lenovo *wink wink, nudge nudge*
16:10:15 <smcginnis> Although, now that zoom is popular, that might end up being a better tool. Any idea if that works out of there?
16:10:24 <jungleboyj> The badge machine in the building is down so you guys just need to write your names on the badges.
16:10:25 <smcginnis> Or do we need to go over your phone data? :)
16:10:38 <smcginnis> I'll bring my crayons.
16:10:50 <enriquetaso> o/
16:11:02 <jungleboyj> I will meet  you guys at the front door.  Putting my phone number in the etherpad now so you can text me if necessary.
16:11:45 <jungleboyj> rosmaita:  Are you still planning to come?
16:12:04 <rosmaita> i will be there, calling 1-800-jungleboy now
16:12:13 <smcginnis> :)
16:12:14 <jungleboyj> :-)  Ha ha ha.
16:12:30 <jungleboyj> smcginnis:  Zoom would be nice to use.  I don't have an account though.
16:12:55 <erlon> smcginnis, zoom works as a charm on Linux.
16:13:11 <jungleboyj> The guest Wifi has been tested and works.  Shouldn't need to use my phone.
16:13:17 <avishay> zoom is great on Android as well
16:13:40 <jungleboyj> smcginnis:  Do you have zoom?
16:14:29 <smcginnis> The basic account is free. Just limits meetings to 40 minutes.
16:14:54 <jungleboyj> Hmmm.  Yuck.
16:15:09 <smcginnis> $15/month for 24hr meetings.
16:15:35 <rosmaita> is there a limit to a consecutive number of 40min meetings?
16:15:44 <jungleboyj> The other option, depending on the phone in the room is that we use Lync.  That has been working really well in Lenovo.
16:17:08 <smcginnis> rosmaita: No, it says unlimited number of meetings.
16:17:08 <jungleboyj> Anyway, smcginnis lets talk about this after the meeting.  Don't need everyone for that decision.
16:17:14 <smcginnis> Yeah
16:17:52 <jungleboyj> Anyway, with the midcycle happening this particular meeting won't then happen.
16:18:48 <jungleboyj> I think that is all I had for announcements.
16:18:57 <jungleboyj> smcginnis:  You have anything there?
16:19:11 <smcginnis> Nope
16:19:17 <jungleboyj> Ok.  Moving on then.
16:19:19 <whoami-rajat> jungleboyj: what are the timings ?
16:20:19 <jungleboyj> whoami-rajat:  Oh, good question.  I was going to integrate that into the next topic.  :-)
16:20:35 <jungleboyj> #topic review mid-cycle topics and do some planning
16:20:53 <jungleboyj> #link https://etherpad.openstack.org/p/cinder-stein-mid-cycle-planning
16:21:05 <jungleboyj> So, we have a good list of topics.
16:21:32 <jungleboyj> The meetings will be happening from roughly 9 am to 5 pm Eastern Time.
16:22:07 <whoami-rajat> jungleboyj:  similar as of PTG, right?
16:22:27 <whoami-rajat> previous*
16:22:45 <jungleboyj> No.  We are actually two timezones closer to India.
16:22:59 <jungleboyj> So, Eastern time is UTC-5 I believe.
16:23:23 <jungleboyj> #link https://greenwichmeantime.com/time-zone/usa/eastern-time/
16:23:48 <whoami-rajat> jungleboyj:  ok, will do the conversions.
16:24:11 <jungleboyj> So we are 6 hours behind Spain.
16:24:49 <jungleboyj> So, for instance geguileo If there are topics that you want to make sure you are in on please indicate those so we can schedule them for the morning.
16:24:58 <whoami-rajat> 7:30 PM - 3:30 AM for India.
16:25:04 <geguileo> jungleboyj: thanks, will do
16:25:12 <jungleboyj> whoami-rajat:  Same for you.  :-)
16:25:18 <e0ne> jungleboyj: could you please add timezone to the etherpad?
16:25:27 <whoami-rajat> jungleboyj: sure, thanks!
16:26:15 <jungleboyj> Done.
16:26:38 <e0ne> jungleboyj: thanks a lot!
16:26:47 <jungleboyj> e0ne:  No problem.
16:27:19 <jungleboyj> If those of you who are in Europe and Asia can indicate which topics you want scheduled earlier in the day I will work on putting a schedule together for the topics later this week.
16:27:52 <jungleboyj> Have that ready by Monday so people can take a look at it and plan accordingly.
16:28:31 <jungleboyj> Also, please add topics in there.
16:28:55 <jungleboyj> This is going to be a rare time that we can actually work all together so the more we can discuss the better.
16:30:24 <jungleboyj> I added a couple other ways we can use the time to clean up bugs and patches.
16:30:50 <jungleboyj> Any other questions/concerns about the current list of topics?
16:30:51 <smcginnis> That would be time well spent.
16:30:56 <jungleboyj> smcginnis:  Yeah.
16:32:05 <jungleboyj> Ok.  If there aren't any more topics right now, I know that we will think of them as we are in the room working together.
16:32:36 <jungleboyj> Next topic:
16:32:50 <jungleboyj> #topic Are we ok with allowing changes to things other than just the drivers in driverfixes/newton
16:33:08 <jungleboyj> #link https://review.openstack.org/633031
16:33:31 <jungleboyj> So, the patch above brought up the question of what we were ok letting in to driverfixes/newton
16:33:46 <e0ne> it could become a bit uncontrollable if we allow all fixes to driverfixes/* branches
16:33:47 <eharney> hrmm
16:33:52 <jungleboyj> At the time we created the branch we had said we would only allow driverfixes.
16:34:03 <jungleboyj> e0ne:  That was the concern
16:34:13 <smcginnis> We had originally talked about deleting everything but drivers to make sure no one used those as stable branches.
16:34:19 <jungleboyj> Doesn't look like Keith Mnemonic is here.
16:34:20 <eharney> we also said nobody was supposed to be running this branch directly
16:34:29 <e0ne> eharney: +1
16:34:33 <jungleboyj> eharney:  That is right.
16:34:38 <eharney> but i'm not sure whether that's a pro or con for landing this :)
16:35:10 <smcginnis> I've gone back and forth on this in light of extended maintenance changes. But I think I would feel better if we kept the policy that these are only for drivers patches.
16:35:37 <smcginnis> The rest is out of scope for the purpose of these. And things ocata and later change because we actually do have extended maintenance on them.
16:35:57 <smcginnis> If someone needs something in those older branches, that's on the vendors that are still supporting running releases that old.
16:36:05 <eharney> i'm inclined to agree mostly because it only applies to newton and mitaka so it hopefully isn't a big detriment to just not do it
16:36:24 <jungleboyj> smcginnis:  Yeah, that is consistent with the conversations we originally had.
16:36:35 <jungleboyj> The extended maint only went back to Ocata.
16:36:58 <eharney> also, we don't run tempest on this branch
16:37:49 <smcginnis> And the original concern still stands that we don't want anyone to mistake these for something they can pull down and run. Especially without tempest, we have no way of knowing how broken that would be.
16:38:44 <jungleboyj> Ok, so it sounds like we are still at the same point we were at when we originally made the branch.
16:38:58 <jungleboyj> That we shouldn't merge anything that isn't for just drivers.
16:39:13 <rosmaita> that seems to make the most sense
16:39:49 <rosmaita> someone really desperate can fork on github and good luck to them
16:39:50 <jungleboyj> Ok.  That sounds like a decision then.
16:40:07 <jungleboyj> rosmaita:  Ah, no.  Don't even say that.  ;-)
16:40:40 <rosmaita> i take it back!
16:40:46 <jungleboyj> Ok, so lets move to the next subject:
16:41:00 <jungleboyj> #topic os-brick to validate WWN upon connection
16:41:18 <jungleboyj> #link https://bugs.launchpad.net/os-brick/+bug/1813609
16:41:19 <openstack> Launchpad bug 1813609 in os-brick "Data corruption when os-brick uses defunct devices for new volume connections" [Undecided,In progress] - Assigned to Avishay Traeger (avishay-il)
16:41:24 <jungleboyj> avishay_: ...
16:41:25 <avishay_> Yo
16:41:31 <jungleboyj> Yo yo
16:41:35 <jungleboyj> @!
16:41:35 <_pewp_> jungleboyj (=ﾟωﾟ)ﾉ
16:42:04 <avishay_> So basically we saw a case where devices on a host weren't cleaned properly, and a new connection reused those devices
16:42:20 <avishay_> All kinds of badness
16:42:48 <avishay_> So the proposed solution is for drivers to return the expected WWN with the initialize connection data, and for os-brick to validate it
16:42:57 <avishay_> I have patches up for:
16:43:08 <avishay_> os-brick: https://review.openstack.org/#/c/633676/2
16:43:14 <avishay_> and Pure driver: https://review.openstack.org/#/c/633680/2
16:43:34 <avishay_> I guess the main question that keeps coming up is tracking which drivers still need to support it
16:43:40 <avishay_> Questions? :)
16:44:08 <eharney> how do we determine which drivers need to support it?
16:44:20 <jungleboyj> So, we had talked about creating an etherpad to track who had implemented the fix.
16:44:44 <whoami-rajat> jungleboyj: maybe we can consider storyboard
16:44:45 <avishay_> I guess it's any driver that reuses LUNs
16:44:47 <jungleboyj> Maybe use a coordinated topic on the patches to organize them a bit.
16:44:53 <avishay_> eharney: ^
16:45:03 <erlon> avishay_, isn't that related to the concurrency problem that geguileo fixed some time ago?
16:45:03 <eharney> ah, ok
16:45:25 <avishay_> Also, my proposed os-brick fix addresses iSCSI.  FC is also affected in the same way, but I don't have hardware to test a fix.
16:45:25 <jungleboyj> whoami-rajat:  Good point.  That is also a topic for the Mid-cycle.
16:45:27 <erlon> where hosts were leaving left over devices
16:46:15 <avishay_> geguileo did great work on cleanup, but I believe there are still corner cases where devices can be left over
16:46:19 <whoami-rajat> jungleboyj: ok
16:46:45 <avishay_> For example network issues during disconnect, or when nova just "forgets" to disconnect
16:46:47 <geguileo> avishay_: there shouldn't if Nova does things right (which probably doesn't)
16:47:01 <geguileo> and if one installs the latest code from the iSCSI initiator
16:47:14 <geguileo> at least not on iSCSI, I haven't thoroughly tested FC
16:47:31 <avishay_> So we were running an older version of CentOS and os-brick when it happened (now upgraded all to latest), but still good to have a validation I believe
16:47:42 <geguileo> avishay_: issues during disconnect are fine (if Nova doesn't just ignore them)
16:48:09 <geguileo> avishay_: it's not only about a newer version of os-brick
16:48:18 <geguileo> you also need the right iscsi initiator
16:48:30 <avishay_> geguileo: yes, new CentOS brought new iSCSI initiator with it
16:48:39 <geguileo> and Nova is probably still ignoring when we fail on disconnect and just proceed as if it went fine
16:48:59 <geguileo> then we'll have to check the Nova code
16:49:06 <geguileo> I haven't tested it in a while...
16:49:11 <geguileo> with error injection I mean
16:49:54 <avishay_> I agree with all of that, but you are still in favor of this patch, correct?
16:50:17 <geguileo> avishay_: the one that is merging?
16:50:42 <jungleboyj> geguileo:  Yeah, I merged that based on discussion the other day.  Hope that was ok.
16:50:54 <geguileo> the patch looks OK
16:50:57 <jungleboyj> I mean, I could stop it.
16:51:13 <geguileo> though I'm not sure if we'll forcefully disconnect the leftover device when doing it this way
16:51:29 <avishay_> I haven't submitted a patch in so long I didn't even realize :P
16:51:30 <geguileo> avishay_: have you tried that scenario?
16:51:43 <geguileo> the failure one?
16:52:33 <avishay_> I saw that the attach fails, but honestly didn't notice if the leftover device got cleaned up
16:52:45 <avishay_> I can check that first thing tomorrow (already night time here)
16:53:12 <geguileo> avishay_: that would be great
16:53:18 <geguileo> at least to know what we can expect
16:53:38 <avishay_> OK cool, will report back to you on that
16:53:44 <geguileo> avishay_: thanks
16:54:36 <avishay_> And I guess you all discuss how to track which drivers need to make the change next week in the mid-cycle and I'll try to assist with that
16:54:40 <jungleboyj> #action avishay_  to test if the leftover device is cleaned up.
16:54:51 <jungleboyj> avishay_:  That sounds like a good plan.
16:55:02 <avishay_> Cool cool
16:55:10 <jungleboyj> #action Discuss how to track what drivers need the change in the midcycle next week.
16:55:55 <jungleboyj> So, there may be a follow-up patch if the device doesn't get cleaned up as hoped?
16:56:14 <avishay_> Indeed
16:56:22 <jungleboyj> Also, do we have anyone that can help with the FC aspect?
16:56:39 <jungleboyj> geguileo:  Do you have FC hardware?
16:56:52 <geguileo> jungleboyj: no, I don't :-(
16:57:10 <geguileo> so if anybody wants to give me a gift...
16:57:12 <jungleboyj> avishay_:  Otherwise you may need to push it back at patrickeast and see if he has FC hardware to test with.  He used to.
16:57:14 <avishay_> Who uses FC anyway... :P
16:57:22 <jungleboyj> :-)
16:57:45 <jungleboyj> Anyway, lets get the iSCSI case taken care of and then discuss the FC problem.
16:58:09 <jungleboyj> Lets try to get the last topic quickly.
16:58:21 <jungleboyj> #topic Move 'attach.end' notify to attachment_complete
16:58:24 <yikun> OK, thanks.
16:58:26 <jungleboyj> yikun:
16:58:31 <yikun> This patch we try to move the 'attach.end' notification into attachment_complete (new added in microversion 3.44, the end of new style attach) rather than the end of attachemnt_update (the end of old style attach).
16:58:38 <yikun> and @hemna had some question about should we send the notification in API, but the fact is we had the precedent that on [1], and I also could not see any other potential problem on it. So I think send the notification in API is OK for me.
16:58:45 <yikun> and some more background I write in [2], you could take a look.
16:58:53 <yikun> [1] https://github.com/openstack/cinder/blob/ee1ac54/cinder/api/v2/volumes.py#L302
16:58:53 <yikun> [2] https://review.openstack.org/#/c/625856/3/cinder/volume/manager.py@4452
16:59:01 <yikun> So, I just wanna get more feedback on it. :)
16:59:25 <jungleboyj> Ok.  We will try to get some eyes on that then.
16:59:28 <eharney> if the operation isn't totally done until that point in the API, don't we have to send it from the API?
17:00:16 <jungleboyj> Sounds right.
17:00:32 <jungleboyj> If we can get more eyes on that it would be good given the sensitivity of that code.
17:00:38 <whoami-rajat> #link https://review.openstack.org/#/c/625856
17:00:49 <yikun> yes, we should send the end notification in the real end.
17:01:14 <yikun> whoami-rajat: ha, thanks for your link
17:01:24 <jungleboyj> Ok.  We are over time.
17:01:32 <whoami-rajat> yikun: :)
17:01:36 <jungleboyj> Thanks for meeting.
17:01:44 <jungleboyj> Look forward to seeing some of you next week.
17:01:49 <jungleboyj> #endmeeting