Wednesday, 2018-09-19

*** dmacpher has joined #openstack-meeting00:05
*** janki has quit IRC00:09
*** janki has joined #openstack-meeting00:09
*** janki has quit IRC00:10
*** janki has joined #openstack-meeting00:11
*** b1air has quit IRC00:17
*** gyee has quit IRC00:19
*** jamesmcarthur has joined #openstack-meeting00:26
*** dmacpher_ has joined #openstack-meeting00:34
*** dmacpher has quit IRC00:36
*** diablo_rojo has quit IRC00:36
*** longkb has joined #openstack-meeting00:46
*** jamesmcarthur has quit IRC00:59
*** diablo_rojo has joined #openstack-meeting01:11
*** slaweq has joined #openstack-meeting01:11
*** evrardjp has quit IRC01:12
*** slaweq has quit IRC01:15
*** jamesmcarthur has joined #openstack-meeting01:16
*** alpha_sh has quit IRC01:33
*** litao has joined #openstack-meeting01:35
*** tetsuro has quit IRC01:35
*** hongbin has joined #openstack-meeting01:37
*** diablo_rojo has quit IRC01:43
*** jamesmcarthur has quit IRC02:12
*** armax has quit IRC02:15
*** jamesmcarthur has joined #openstack-meeting02:18
*** iyamahat_ has quit IRC02:21
*** yamahata has quit IRC02:22
*** jamesmcarthur has quit IRC02:22
*** ijw has joined #openstack-meeting02:31
*** ijw has quit IRC02:36
*** tetsuro has joined #openstack-meeting02:43
*** jamesmcarthur has joined #openstack-meeting02:45
*** apetrich has quit IRC02:51
*** armax has joined #openstack-meeting02:52
*** armax has quit IRC02:56
*** slaweq has joined #openstack-meeting03:11
*** slaweq has quit IRC03:16
*** tobiash has left #openstack-meeting03:32
*** jamesmcarthur has quit IRC03:34
*** jamesmcarthur has joined #openstack-meeting03:40
*** jamesmcarthur has quit IRC03:50
*** jamesmcarthur has joined #openstack-meeting03:56
*** ykatabam has quit IRC03:57
*** tetsuro has quit IRC03:58
*** jamesmcarthur has quit IRC04:00
*** jamesmcarthur has joined #openstack-meeting04:00
*** jamesmcarthur has quit IRC04:05
*** yamamoto has joined #openstack-meeting04:07
*** rfolco has quit IRC04:08
*** jamesmcarthur has joined #openstack-meeting04:09
*** jamesmcarthur has quit IRC04:13
*** jamesmcarthur has joined #openstack-meeting04:23
*** jamesmcarthur has quit IRC04:27
*** diablo_rojo has joined #openstack-meeting04:28
*** bobh has quit IRC04:31
*** bobh has joined #openstack-meeting04:37
*** ykatabam has joined #openstack-meeting04:40
*** bobh has quit IRC04:42
*** bobh has joined #openstack-meeting04:51
*** bobh has quit IRC04:55
*** hongbin has quit IRC05:06
*** diablo_rojo has quit IRC05:07
*** slaweq has joined #openstack-meeting05:11
*** slaweq has quit IRC05:15
*** tetsuro has joined #openstack-meeting05:20
*** rcernin has quit IRC05:30
*** rcernin has joined #openstack-meeting05:30
*** ykatabam has quit IRC05:36
*** rcernin_ has joined #openstack-meeting06:04
*** rcernin has quit IRC06:06
*** Luzi has joined #openstack-meeting06:16
*** janki has quit IRC06:17
*** slaweq has joined #openstack-meeting06:18
*** slaweq has quit IRC06:23
*** rcernin has joined #openstack-meeting06:34
*** rcernin_ has quit IRC06:36
*** arne_wiebalck_ has joined #openstack-meeting06:51
*** slaweq has joined #openstack-meeting06:55
*** janki has joined #openstack-meeting06:55
*** apetrich has joined #openstack-meeting06:57
*** slaweq has quit IRC07:00
*** rcernin has quit IRC07:06
*** cloudrancher has joined #openstack-meeting07:06
*** jamesmcarthur has joined #openstack-meeting07:13
*** slaweq has joined #openstack-meeting07:16
*** jamesmcarthur has quit IRC07:17
*** e0ne has joined #openstack-meeting07:36
*** kopecmartin has joined #openstack-meeting07:37
*** jesusaur has quit IRC07:39
*** jesusaur has joined #openstack-meeting07:44
*** e0ne has quit IRC07:48
*** Emine has joined #openstack-meeting08:06
*** dmacpher_ has quit IRC08:09
*** tetsuro has quit IRC08:12
*** rossella_s has quit IRC08:14
*** rossella_s has joined #openstack-meeting08:15
*** e0ne has joined #openstack-meeting08:25
*** e0ne has quit IRC08:26
*** e0ne has joined #openstack-meeting08:33
*** alexchadin has joined #openstack-meeting08:43
*** iyamahat has joined #openstack-meeting08:55
*** dmacpher_ has joined #openstack-meeting08:58
*** iyamahat has quit IRC09:01
*** e0ne has quit IRC09:02
*** AndreyS has joined #openstack-meeting09:05
*** ykatabam has joined #openstack-meeting09:24
*** markvoelker has quit IRC09:25
*** ijw has joined #openstack-meeting09:54
*** ykatabam has quit IRC09:57
*** ijw has quit IRC09:58
*** rbudden has joined #openstack-meeting10:22
*** cloudrancher has quit IRC10:25
*** markvoelker has joined #openstack-meeting10:30
*** longkb has quit IRC10:34
*** longkb has joined #openstack-meeting10:38
*** longkb has quit IRC10:39
*** longkb has joined #openstack-meeting10:40
*** longkb has quit IRC10:42
*** e0ne has joined #openstack-meeting10:47
*** markvoelker has quit IRC11:00
*** erlon has joined #openstack-meeting11:02
*** dtrainor has quit IRC11:13
*** dtrainor has joined #openstack-meeting11:13
*** priteau has joined #openstack-meeting11:16
*** imacdonn has quit IRC11:19
*** cloudrancher has joined #openstack-meeting11:19
*** imacdonn has joined #openstack-meeting11:20
*** sambetts|afk is now known as sambetts11:35
*** ssbarnea has quit IRC11:39
*** tssurya has joined #openstack-meeting11:48
*** cloudrancher has quit IRC11:55
*** cloudran_ has joined #openstack-meeting11:55
*** ssbarnea has joined #openstack-meeting11:56
*** raildo has joined #openstack-meeting11:56
*** markvoelker has joined #openstack-meeting11:57
*** ijw has joined #openstack-meeting12:02
*** ijw has quit IRC12:07
*** rfolco has joined #openstack-meeting12:16
*** jamesmcarthur has joined #openstack-meeting12:18
*** e0ne has quit IRC12:20
*** ijw has joined #openstack-meeting12:26
*** ijw has quit IRC12:30
*** jamesmcarthur has quit IRC12:31
*** markvoelker has quit IRC12:31
*** finix has joined #openstack-meeting12:43
*** jamesmcarthur has joined #openstack-meeting12:51
*** dustins has joined #openstack-meeting12:52
*** alexchadin has quit IRC12:53
*** alexchadin has joined #openstack-meeting12:54
*** alexchadin has quit IRC13:11
*** alexchadin has joined #openstack-meeting13:15
*** janki has quit IRC13:16
*** jamesmcarthur has quit IRC13:18
*** jamesmcarthur has joined #openstack-meeting13:19
*** mriedem has joined #openstack-meeting13:21
*** dkrol has joined #openstack-meeting13:23
*** tpsilva has joined #openstack-meeting13:36
*** longkb has joined #openstack-meeting13:37
*** munimeha1 has joined #openstack-meeting13:42
*** bobh has joined #openstack-meeting13:46
*** jamesmcarthur has quit IRC13:49
*** jamesmcarthur has joined #openstack-meeting13:50
*** bobh has quit IRC13:51
*** annabelleB has joined #openstack-meeting13:56
*** bobh has joined #openstack-meeting13:58
*** janki has joined #openstack-meeting13:59
*** mpiwowarczy has joined #openstack-meeting14:00
*** bobh has quit IRC14:02
*** janki has quit IRC14:04
*** bobh has joined #openstack-meeting14:08
*** munimeha1 has quit IRC14:09
*** awaugama has joined #openstack-meeting14:12
*** bobh has quit IRC14:12
*** hongbin has joined #openstack-meeting14:18
*** bobh has joined #openstack-meeting14:19
*** Luzi has quit IRC14:19
*** aagate has joined #openstack-meeting14:24
*** Leo_m has joined #openstack-meeting14:24
*** alexchadin has quit IRC14:25
*** bobh has quit IRC14:27
*** alexchadin has joined #openstack-meeting14:30
*** bobh has joined #openstack-meeting14:32
*** gouthamr_ is now known as gouthamr14:35
*** bobh has quit IRC14:36
*** longkb has quit IRC14:40
*** bobh has joined #openstack-meeting14:40
*** alexchadin has quit IRC14:43
*** bobh has quit IRC14:45
*** bobh has joined #openstack-meeting14:48
*** munimeha1 has joined #openstack-meeting14:49
*** bobh has quit IRC14:53
*** alexchadin has joined #openstack-meeting14:57
*** cloudran_ has quit IRC14:59
*** bobh has joined #openstack-meeting14:59
*** bobh has quit IRC15:03
*** AndreyS has quit IRC15:04
*** bobh has joined #openstack-meeting15:10
*** cloudrancher has joined #openstack-meeting15:10
*** Emine has quit IRC15:10
*** bobh has quit IRC15:14
*** munimeha1 has quit IRC15:16
*** iyamahat has joined #openstack-meeting15:16
*** alexchadin has quit IRC15:19
*** bobh has joined #openstack-meeting15:22
*** armax has joined #openstack-meeting15:25
*** bobh has quit IRC15:27
*** jamesmcarthur has quit IRC15:30
*** bobh has joined #openstack-meeting15:31
*** jamesmcarthur has joined #openstack-meeting15:32
*** bobh has quit IRC15:36
*** gyee has joined #openstack-meeting15:37
*** bobh has joined #openstack-meeting15:37
*** bobh has quit IRC15:42
*** iyamahat has quit IRC15:46
*** pcaruana has joined #openstack-meeting15:47
*** bobh has joined #openstack-meeting15:47
*** bobh has quit IRC15:47
*** bobh has joined #openstack-meeting15:47
*** annabelleB has quit IRC15:50
*** yamamoto has quit IRC15:53
*** annabelleB has joined #openstack-meeting15:53
*** woojay has joined #openstack-meeting15:54
*** kopecmartin has quit IRC15:56
*** _alastor_ has joined #openstack-meeting15:58
*** whoami-rajat has joined #openstack-meeting16:00
*** diablo_rojo has joined #openstack-meeting16:00
jungleboyj#startmeeting Cinder16:00
openstackMeeting started Wed Sep 19 16:00:23 2018 UTC and is due to finish in 60 minutes.  The chair is jungleboyj. Information about MeetBot at
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:00
*** openstack changes topic to " (Meeting topic: Cinder)"16:00
openstackThe meeting name has been set to 'cinder'16:00
*** walshh_ has joined #openstack-meeting16:00
_pewp_jungleboyj (=゚ω゚)ノ16:01
geguileohi! o/16:02
jungleboyjOkie dokie.16:03
*** annabelleB has quit IRC16:03
jungleboyjLooks like we have a decent number of people here.16:03
jungleboyj#topic announcements16:04
*** openstack changes topic to "announcements (Meeting topic: Cinder)"16:04
jungleboyjSo, we have the etherpad for the Berlin Forum.16:04
jungleboyjUnfortunately I hadn't seen that one was already created by geguileo16:04
jungleboyjgeguileo:  Did you ever send your's to the mailing list?16:04
geguileobecause I didn't tell anyone  lol16:04
geguileojungleboyj: no I didn't16:04
jungleboyjOk.  Good.  So it wasn't me then.  :-)16:04
geguileonop, it was me16:05
geguileoI assumed the next one would check the wiki16:05
geguileomy bad  :-(16:05
jungleboyjOk, so are you ok with moving your content over to the one I created and I will update the link in the Wiki to point to the one I created?16:05
geguileojungleboyj: I moved the contents and updated the wiki16:05
geguileoso we are good now16:05
jungleboyjgeguileo:  Oh sweet.  Thanks buddy!16:06
jungleboyjSo, please add your topics in there.  I have at least one to add based on PTG discussions.16:06
*** e0ne has joined #openstack-meeting16:07
jungleboyjAlso, wanted to announce that the driverfixes/ocata branch has been removed.16:08
jungleboyjsmcginnis:  Thanks for the clean-up there.16:08
jungleboyjIf you all remember we did work to get things from driverfixes/ocata to stable/ocata16:09
geguileosmcginnis: thanks for synchronizing the branches16:09
jungleboyjWe verified that everything had been moved.16:09
jungleboyjSo, determined it was safe to remove driverfixes and avoid the chances for future confusion.16:09
smcginnisScripting FTW. :)16:09
jungleboyjsmcginnis:  Anything you want to add on that topic?16:10
smcginnisI added something for later related to ocata, but that can wait.16:10
jungleboyjOk.  Sounds good.16:10
jungleboyjLast announcement is that I have gotten the summary of discussions from the Stein PTG put together in our Wiki:16:11
jungleboyjI hope I got the action item assignments correct for everyone.16:11
jungleboyjThe video links should be correct.  :-)16:11
jungleboyjAnd I recovered the lost Thursday etherpad.  Thank goodness.16:11
jungleboyjWell, I didn't, the awesome Infra team did, but I created a new version of that etherpad with the content they recovered.16:12
jungleboyjSo, please review and let me know if there is anything I got wrong in there.16:13
erlonjungleboyj, thanks for that Jay, helps a lot16:13
jungleboyjerlon:  No problem.  Made yesterday a long day but I know it is helpful for everyone.16:13
jungleboyjLast note of follow up from the PTG.  I have started reaching out to some of the Core team members that have been less active.  Watch for an e-mail to the mailing list as follow up to that.16:14
jungleboyjSo, that was all I had for announcements.16:15
jungleboyj#topic stable/ocata maintenance mode16:15
*** openstack changes topic to "stable/ocata maintenance mode (Meeting topic: Cinder)"16:15
smcginnisSo we passed the EM date for ocata.16:15
smcginnisWhich means teams have to decide if they want to continue maintaining stable/ocata and means no more official releases will be done.16:16
smcginnisMatt wrote up some of this changes on the ML:16:16
smcginnis#link Extended maintenance discussion16:16
*** davidsha has joined #openstack-meeting16:16
smcginnisI haven't checked, but I think we probably have unreleased commits in stable/ocata.16:16
*** e0ne has quit IRC16:17
smcginnisWe should quick do a final stable release with whatever we have.16:17
smcginnisWe can still backport fixes, but in extended maintenance there will not be any official releases.16:17
smcginnisJust a place for distros to pick up anything they want to roll into their packaging.16:17
jungleboyjsmcginnis: ++16:18
jungleboyjI can propose a release.16:18
jaypipeserlon: you're welcome.16:19
smcginnisYeah, looks like a few commits since the last release -
jungleboyjjaypipes: :-p16:19
jungleboyjI never get thanked for you.  :-(16:19
jaypipesBTW, while I'm here, I should mention that I still owe geguileo and erlon an email detailing the strategy for taking advantage of a "record generation marker" DB query approach. Just noting that's still on my to-do list.16:20
erlonjaypipes, hey np :p16:20
geguileojaypipes: thanks16:20
jungleboyjsmcginnis:  Ok, will propose that yet today.16:20
jungleboyjjaypipes:  Cool.  Thanks.  erlon and geguileo were asking about that earlier this week.16:21
erlonjaypipes, that would be great thanks16:21
jungleboyj#action jungleboyj to propose one last stable/ocata release.16:22
jungleboyjsmcginnis:  Anything else to propose/discuss on that topic?16:23
*** annabelleB has joined #openstack-meeting16:23
smcginnisNo, I think that's it. Just raising awareness of the change there mostly.16:23
jungleboyjsmcginnis:  Cool.  Thank you.16:23
jungleboyjThat was all we had on the agenda.16:24
jungleboyjwhoami-rajat:  You had a todo to pick out a few bugs for triage that came out of the PTG.16:24
jungleboyjDid you get a chance to do that?16:24
whoami-rajatjungleboyj: yes, i picked some16:25
jungleboyjwhoami-rajat:  Cool.16:25
jungleboyj#topic Bug Triage16:25
*** openstack changes topic to "Bug Triage (Meeting topic: Cinder)"16:25
jungleboyjwhoami-rajat:  All yours.16:25
*** yamamoto has joined #openstack-meeting16:26
whoami-rajatso the first bug i found was this
openstackLaunchpad bug 1790840 in Cinder "Multiattach - multiattach is set to false after retype operation" [Undecided,New] - Assigned to Rajat Dhasmana (whoami-rajat)16:26
whoami-rajatthere had been a long chain of commits but still none of them fixed it, still reproducible.16:27
*** pcaruana has quit IRC16:27
jungleboyjwhoami-rajat:  Good catch.  That looks like an important one to fix  given the interest in multi-attach.16:29
jungleboyjMarking it confirmed if you have been able to reproduce.16:30
jungleboyjLooks like you have assigned that to yourself.16:30
walshh_The manage use case was addressed
openstackLaunchpad bug 1783790 in Cinder "Multiattach - the multiattach is still set to False after a Manage and Retype operation" [Undecided,Fix released] - Assigned to TommyLike (hu-husheng)16:30
whoami-rajatjungleboyj: thanks, thought will give it a try so assigned myself.16:30
*** e0ne has joined #openstack-meeting16:31
jungleboyjwalshh_:  Saw that.16:31
walshh_the retype usecase still remains unfixed hense
openstackLaunchpad bug 1790840 in Cinder "Multiattach - multiattach is set to false after retype operation" [High,Confirmed] - Assigned to Rajat Dhasmana (whoami-rajat)16:31
jungleboyjwhoami-rajat:  Ok.  Sounds good.  If you need help pull in tommylike since he fixed the manage case.16:32
whoami-rajatwalshh_: yea there are multiple bugs reported for different scenarios.16:32
jungleboyjwhoami-rajat:  What are the other bugs?16:32
whoami-rajatjungleboyj: ok sure.16:32
openstackLaunchpad bug 1788619 in OpenStack Compute (nova) "disk cachemodes should be restricted with multiattached volumes" [Undecided,New]16:32
whoami-rajatno updates since a month on this16:33
jungleboyjLooks like this was marked as a duplicate:
openstackLaunchpad bug 1770671 in Cinder "duplicate for #1778525 multiattach is not enabled consistently when the default volume type supports multiattach" [Undecided,Fix released] - Assigned to TommyLike (hu-husheng)16:33
*** arne_wiebalck_ is now known as arne_wiebalck16:34
jungleboyjwhoami-rajat:  Yuck.  the cacemodes issue doesn't look good.16:34
*** e0ne has quit IRC16:35
jungleboyjI didn't know that there was a way to set cachemodes.16:35
jungleboyjAh, it is a nova configuration option.16:36
whoami-rajatjungleboyj: i also don't have deep understanding of the bug, eric doesn't seem to be here to address it.16:36
jungleboyjwhoami-rajat:  Right.  He is out for the next two weeks.16:37
jungleboyjLet me update that bug and save that one for discussion when he returns.16:37
whoami-rajatwhoami-rajat: ok, seems like we should move on the next bug?16:37
jungleboyjAgree that is one that should be given some attention though.16:38
jungleboyjwhoami-rajat: Sure.16:39
whoami-rajatjungleboyj: yes.16:39
jungleboyjwhoami-rajat:  What else you got for us?16:39
whoami-rajatso i've an old one but the change seems to be abandoned and was never fixed, or maybe addressed by some other bug i may not be aware of,16:40
openstackLaunchpad bug 1648885 in Cinder "ceph: TestVolumeBootPattern.test_create_ebs_image_and_check_boot times out failing to delete volume due to snapshot is busy" [Undecided,In progress]16:40
jungleboyjWell that is an old one.16:42
jungleboyjsmcginnis:  unassigned it and the change was abandoned due to inactivity.16:43
jungleboyjjbernard isn't here.  :-(16:43
*** bobh has quit IRC16:43
jungleboyjGiven that really hasn't been touched in over a year and a half is there any reason to not close this?16:43
smcginnisProbably good to close at this point.16:44
whoami-rajatyeah, i guess.16:44
jungleboyjIf we start seeing failures again there is no reason we can't open a new bug or re-open.16:44
jungleboyjThere was hit on that in the last 4 months.16:47
jungleboyjOk.  I am going to close that one out then.16:48
jungleboyjAny other bugs to discuss?16:49
whoami-rajati've a last one, didn't get time to reproduce the issue but seems quite bad,
openstackLaunchpad bug 1782714 in OpenStack Compute (nova) "Properties of an attached volume are lost after live migration" [Undecided,New]16:50
*** annabelleB has quit IRC16:50
jungleboyjHmmm.  That looks like one that is worth investigating further.16:52
jungleboyjAnyone able to try to recreate that one?16:54
whoami-rajatthe author has provided most of the logged details for the issue about the point when the metadata seems to be lost.16:55
tbarronjaypipes: would you copy '' on that "record generation marker" DB mail?  we need to use the same technique in manila16:55
jaypipestbarron: yes sir!16:55
jungleboyjtbarron: ++16:56
tbarronjaypipes: thanks!16:56
jungleboyjsmcginnis:  geguileo Thoughts on this last bug?16:56
jungleboyjI wonder if it is specific to Ceph.16:56
*** davidsha has quit IRC16:57
jungleboyjI seem to have lost the room.16:57
jungleboyjSo, I will put this on the follow up list for next week and we will go from there.16:58
smcginnisLooks like an issue, but don't really have much more on it than that.16:58
jungleboyjsmcginnis:  Ok.  Not sure if I have a place to recreate it right now.16:58
* jungleboyj has never done live migration. :-)16:58
jungleboyjMaybe time to get my other systems set up.16:59
geguileojungleboyj: I haven't looked at the bug, but that's weird...16:59
*** annabelleB has joined #openstack-meeting16:59
jungleboyjgeguileo:  Agreed.16:59
jungleboyjOk.  Well, we should wrap up.  I will keep this in follow up.16:59
jungleboyjwhoami-rajat:  Good first stab a bug triage.  Thank you!17:00
geguileoI think I know where that issue could be...17:00
jungleboyjClose one and bring attention to a couple others.17:00
geguileoit's probably in the swap of the DB entries17:00
jungleboyjgeguileo:  Cool.  Ok.17:00
jungleboyjYep.  Thanks for a good meeting team.17:00
*** openstack changes topic to "OpenStack Meetings ||"17:01
openstackMeeting ended Wed Sep 19 17:01:01 2018 UTC.  Information about MeetBot at . (v 0.1.4)17:01
openstackMinutes (text):
*** woojay has left #openstack-meeting17:01
*** radeks has joined #openstack-meeting17:07
*** iyamahat has joined #openstack-meeting17:09
*** radeks has quit IRC17:18
*** sambetts is now known as sambetts|afk17:27
*** cloudrancher has quit IRC17:27
*** aagate has quit IRC17:30
*** iyamahat has quit IRC17:33
*** jamesmcarthur has quit IRC17:34
*** cloudrancher has joined #openstack-meeting17:43
*** cloudrancher has quit IRC17:44
*** iyamahat has joined #openstack-meeting17:47
*** munimeha1 has joined #openstack-meeting17:53
*** yamahata has joined #openstack-meeting18:02
*** mpiwowarczy has quit IRC18:12
*** walshh_ has quit IRC18:12
*** annabelleB has quit IRC18:17
*** annabelleB has joined #openstack-meeting18:21
*** fatema_ has joined #openstack-meeting18:35
*** ijw has joined #openstack-meeting18:36
*** ijw has quit IRC18:42
*** vkmc is now known as vkmc|afk18:49
*** ijw has joined #openstack-meeting18:59
*** jamesmcarthur has joined #openstack-meeting18:59
diablo_rojoThe storyboard team is not meeting this week so if you have any questions, please join us in #storyboard!19:03
*** munimeha1 has quit IRC19:05
*** annabelleB has quit IRC19:15
*** annabelleB has joined #openstack-meeting19:22
*** awaugama has quit IRC19:38
*** dustins has quit IRC19:39
*** tssurya has quit IRC19:51
*** pfallenop has joined #openstack-meeting19:56
*** jamesmcarthur has quit IRC20:10
*** pfallenop has quit IRC20:16
*** pfallenop has joined #openstack-meeting20:16
*** dustins has joined #openstack-meeting20:18
*** patchbot has joined #openstack-meeting20:23
*** erlon has quit IRC20:24
*** armax has quit IRC20:27
*** timburke has joined #openstack-meeting20:29
*** jamesmcarthur has joined #openstack-meeting20:30
*** hemna_ has quit IRC20:36
*** priteau has quit IRC20:39
*** dklyle has quit IRC20:47
*** raildo has quit IRC20:54
*** m_kazuhiro has joined #openstack-meeting20:55
timburketdasilva: is it... time?21:00
tdasilva#startmeeting swift21:00
openstackMeeting started Wed Sep 19 21:00:37 2018 UTC and is due to finish in 60 minutes.  The chair is tdasilva. Information about MeetBot at
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.21:00
*** openstack changes topic to " (Meeting topic: swift)"21:00
openstackThe meeting name has been set to 'swift'21:00
timburkei'm so excited! or maybe i just had too much coffee this morning21:00
tdasilvahello, notmyname won't be avaiable to run meeting today, so he asked me to run21:01
tdasilvawho is here for swift meeting?21:01
rledisezhi o/21:01
*** zaitcev has joined #openstack-meeting21:02
*** clayg has joined #openstack-meeting21:02
claygdid not realize I was not in here21:02
tdasilvahehe, no worries21:02
tdasilvameeting agenda is here:
tdasilva#topic PTG summary and review21:03
*** openstack changes topic to "PTG summary and review (Meeting topic: swift)"21:03
tdasilvahopefully everyone had a safe flight home21:03
mattoliverauSo how'd did the ptg go?21:03
tdasilvait was really cool to see everyone there, and we missed those that couldn' t make it21:03
mattoliverauGive me all the goss :)21:03
timburkemattoliverau: well, we missed you :-(21:03
tdasilvai was looking at the etherpad earlier:
mattoliverauSure sure :p21:04
timburkebut i *guess* you oughta spend time with the new baby...21:04
*** Leo_m_ has joined #openstack-meeting21:04
tdasilvaand I think we almost all topics there21:04
mattoliverauTrip or marrage and i chose marrage, Zoe is very cute too which helps21:04
tdasilvagood choice21:05
tdasilvathere was some discussion on PUT+POST, general task queue, multinode testing, py3, what else?21:05
kota_sort of s3 things?21:06
tdasilvawe had some informal conversations about 1space21:06
*** Leo_m has quit IRC21:06
kota_and losfs.21:06
timburkeon s3 things -- including getting ceph tests going again! thanks, kota_!21:06
tdasilvaright, I saw an a patch earlier today about s3api and keystone enhancements, i think it came out of conversations there, right21:06
timburkeyep, got it starred21:07
timburkeand i think i fixed up the gate check? tried, anyway21:07
kota_yeah, it sounds great performance improvement.21:07
tdasilvaany to share in terms of PTG process? do we need to do anything different next time?21:08
tdasilvaI thought it was quite the same as usual, a bit more focused on hacking on code directly...21:08
tdasilvaok, if nothing else to share about PTG we can move on to next topic?21:09
tdasilva#topic PUT+POST patch discovered issues21:09
*** openstack changes topic to "PUT+POST patch discovered issues (Meeting topic: swift)"21:09
claygHere's the 100ft view of where we're at:
mattoliverau there was an etherpad topic on sharding in prod and how thats going and no note howd it go?21:10
claygmattoliverau: I remember speaking briefly about container-sharding to the room21:10
timburkesorry mattoliverau -- i think the gist of it it, "so far, so good -- we think?"21:11
claygmattoliverau: basically we're really happy with how it's going - but we have a ton of tooling downstream to make it manageable for our professional services team21:11
claygwe're just using it when the alternative is "live with horribly horribly big and broken giant 100+ GB containers" and waiting for some issue to show up?21:11
timburkelike, containers get broken up, and (so far as i know) nobody's complaining about data going missing, so that's good21:11
claygEvery day someone is sharding a container somewhere!  No show stopping bugs found!21:12
*** Leo_m_ has quit IRC21:12
claygFrom there it seemed no one else is using it yet (probably because the manual api/script stuff is non-trivial)21:12
*** Leo_m has joined #openstack-meeting21:12
tdasilvazaitcev: just for context, on Friday morning, there were some more discussions on PUT+POST, so clayg wrote up the etherpad to summarize what we 'found out'21:12
claygwe probably should have encouraged other operators to *at least* start running the sharding deamon and monitoring their "largest container" alerts21:13
zaitcevtdasilva: is that the eventlet link above?21:13
mattoliverauWell thanks for running it in prod and getting valuable feedback21:13
mattoliverauNow we cam go fox things and make it better21:14
timburkezaitcev: in particular, was some new information that fell out of trying to get latest eventlet installed for probe tests21:14
openstackLaunchpad bug 1792615 in OpenStack Object Storage (swift) "Eventlet 0.22.0+ changed how graceful shutdowns work" [Undecided,New]21:14
mattoliverauWow, thanks autocomplete on my tablet21:14
tdasilvawhat are the next steps re: PUT+POST?21:15
timburkethat was coming up in, and i kicked out to fix up the test but need to rebase it21:16
patchbotpatch 564700 - swift - Add ceph-s3 test non-voting job (MERGED) - 35 patch sets21:16
patchbotpatch 602526 - swift - Use latest eventlet in probe tests - 6 patch sets21:16
claygzaitcev: executive summary is "double continue garbage is not portable" so the thinking was "PUT+POST[+POST]" should be about the same on the wire and more portable!!  But "on the wire" isn't the same as "semantically equivalent" and having the object-server carry state in memory between "PUT+<footers>" is a bad idea, because HTTP says you can close a pipelined request at anytime.21:16
claygin fact eventlet recently decided it would start more aggressively kicking out clients - so it may not even be uncommon in production to have an object server go away between the data and the metadata - in which case the proxy is kinda screwed.21:17
claygSoo... next step could be "don't throw the baby out with the bath, PUT+second-http-request-for-footers may not work, but we could still fix the double continue madness as long as we were prepared to retry"21:18
claygwhich looks like a hybrid of MIMEPut+POST21:18
*** _pewp_ has quit IRC21:18
zaitcevclayg: closing of the pipelined requests changes nothing to PUT+POST, except adding SYN/SYN+ACK/ACK packets, so performance is lower. But we aren't using the connection state for anything.21:18
claygor *maybe just maybe* - the whole idea was bonkers and we need to re-think our plan to get the hell off eventlet21:19
*** _pewp_ has joined #openstack-meeting21:19
zaitcevclayg: it's a bit hard to believe, but I forgot to filter out Connection: close at some point and everything worked21:19
claygzaitcev: yeah but if the process dies you loose the df cache21:19
claygwe haven't commited anything to disk - there's no way to find the link_at file descriptor to add the metadata too21:19
zaitcevclayg: True. And is this any different from the process dying that received the body but not MIME trailer?21:20
claygthat's another option I considered - go ahead and fsync the .data.tmp file w/o the metadata so if we have to reconnect/recontinue we can "find it"21:20
claygzaitcev: not different than the process *dying* (which should be rare) - but unfortunately it *is* different from the process being asked to gracefully shut down - which will not be rare :'(21:21
claygthere was some consideration if maybe we could patch eventlet - but then what happens if we which to another web server that also likes to tell clients trying to do pipelined requests that they can't have their POST after PUT21:21
claygthe reality is trying to do stateful stuff on a stateless protocol was a bad idea - I probably should have seen it ages ago - but the ramifications of HOW the object-server would track resources from one request to the next wasn't obvious to me until I was reviewing your code - and even then I wasn't sure it was wrong enough to need to abandon21:23
zaitcevokay, I see where this is going21:23
zaitcevnext step, an RPC encoded with ASN.1 over WebSockets21:24
zaitcevThat should be statefull enough for you.21:24
claygI sure am getting tired of beating my head against HTTP!21:24
*** jamesmcarthur has quit IRC21:24
claygall I want is some data with metadata afterwards!?21:24
zaitcevDid you look at HTTP/2?21:24
claygzaitcev: see - now you're thinking!21:25
zaitcevThat one has channelized requests, maybe they can help.21:25
claygI just want off eventlet 😭21:25
*** armax has joined #openstack-meeting21:25
zaitcevHey, no cheating. You said you wanted stateful, that's more than just "off eventlet".21:25
claygI think the protocol for http/2 is amazing absolutely!  there's no good way to swap it into a functioning wsgi application implementation21:25
*** jamesmcarthur has joined #openstack-meeting21:26
zaitcevYeah... That's why I wasn't joking about WebSockets21:26
claygthat's the whole problem - we keep wanting to not have to "throw away all that stuff that already works" 🤮21:26
claygzaitcev: I'm going to push up what I have soon (today or tomorrow)21:27
claygzaitcev: I want to reconsider your change from a tactical level21:27
claygbut I think they both get -1'd for the same issue "if you hup the object-server on latest eventlet the proxy can't finish the upload"21:27
claygso... we'll have to figure out where to go next21:28
claygI was cussin on Friday21:28
claygbut I'm over it now - because I hate eventlet - and humming bird is free for the taking - and all I really want is data followed by metadata and nothing is really worse, my clusters are running the same today with MIME+eventlet that they were the week before21:29
zaitcevI still don't understand how PUT+POST is any worse than MIME in this regard.21:29
claygthe new eventlet behavior is that if you HUP a server (which we do plenty on config push etc) the object-server will let the PUT finish then kick the connection to the floor21:30
zaitcevAh, okay.21:30
claygwith MIME it would let the PUT+footers+commit *all* finish - which is super good and exactly what we want21:30
*** jamesmcarthur has quit IRC21:30
*** armax has quit IRC21:30
claygI would be even more mad at eventlet (I still kinda am really) - but in this case they're right - what I had suggested was brain-dead and I now I kinda hate http too21:31
claygbut I'm over it!21:31
claygtdasilva: i think that's all I've got on that subject for now21:31
claygdid everyone else doze off 😬21:32
tdasilvazaitcev: any thing else?21:32
zaitcevOK, fine. So why don't you create a PUT2 that takes our good old MIME, only encoded in such a way that there aren't any zero-size bodies? Can eventlet take at least that much?21:32
claygzaitcev: FWIW I don't have an obviously better plan - so if you want to high bandwidth about this later I'd be down for that.  Or we can just let it stew and work on py3 or whatever and maybe an door will present it self21:32
*** radeks has joined #openstack-meeting21:33
zaitcevI see. You want to avoid the tree to accept another undercooked solution like ssync.21:33
claygzaitcev: yeah, I think MIME/PUT + POST commit would work - and we end with something that is not intimately coupled with eventlet - but it still doesn't feel super portable to me - it's a hodge podge!  and YES - EXACTLY - we still have to un$%^& ssync too!21:34
claygwe need a better option than HTTP I'm afraid - our requirements don't match what it can offer us it seems!21:35
zaitcevI'm willing to wait until the day we have an external event that forces us to accept something that is good enough, like PUT+POST or an equivalent STAGE+MERGE+COMMIT. Maybe py3, unless we find a way to make MIMEPutter just work there.21:35
*** dustins has quit IRC21:35
tdasilvazaitcev, clayg: maybe we could continue the conversation in either #openstack-swift or schedule a video call or something?21:36
claygtdasilva: yup, I think that's what we needed to get out today21:36
*** armax has joined #openstack-meeting21:36
timburkeif we could find a way to get the zero-byte chunk out of the mime stuff, that might work. the trouble (IIRC) is correctly identifying end-of-data. i think we wouldn't even necessarily need to invent a new verb. but if we wanted to use another framework/language, i think we'd still have some degree of trouble with wanting to send/receive headers with the 100 Continue response, and wanting to send more than one 100 Continue21:36
claygtimburke: gets it21:37
zaitcevOne thing that makes me vary of rolling our own RPC (over WebSockets or even over ssh) is my experience with Ceph, where they do have their project-specific RPC. That thing is a disaster. Firstly, it's underdocumented, so writing implementations is next to impossible. Secondly, the strong binding seeps in, so now you MUST have proxies (monitors in Ceph) and storage (OSD in Ceph) to have matching versions.21:37
claygHTTP is not the right protocol for what we're doing21:37
zaitcevPlease be aware that RPC has these hazards21:38
claygzaitcev: gets it21:38
timburkezaitcev: are we sure that MIMEPutter *doesn't* work with py3? i thought i'd gotten some EC stuff working on py3...21:38
zaitcevtimburke: it might. I dimly remember some problem with it, but maybe it's resolved. I had many problems in the beginning of py3.21:39
claygour requirements are that off the wall - there's a better foundation for what we need to do than HTTP - no one could have known our requirements would over time out grow HTTP - is was is21:39
claygare *not* that off the wall21:39
tdasilvacool...let's move on and we can continue on #openstack-swift21:40
claygthe problem we've been scared of is how to move to something better w/o having to re-due everything at once. that problem hasn't gone away.21:40
tdasilva#topic Configure DiskFileClass21:40
*** openstack changes topic to "Configure DiskFileClass (Meeting topic: swift)"21:40
tdasilvaI'm not sure if it was notmyname or kota_ that added that topic21:41
kota_that's what i added21:41
tdasilvakota_: what's up?21:41
kota_just a re-minder for mattoliverau and it looks he noticed my irc message in the swift channel.21:41
kota_we had some discussion at the PTG with tdasilva then, we don't think any issues so for.21:42
patchbotpatch 447129 - swift - Configure diskfile per storage policy - 20 patch sets21:42
kota_so far.21:42
claygtimburke: kota_: thanks for reviewing that21:42
claygand m_kazuhiro too!21:42
*** dklyle has joined #openstack-meeting21:42
tdasilvaand it already has one +2, so should be close21:43
kota_but it's an incremental improvement so that, i may want to wait other eyes before landed.21:43
timburkeclayg: i wouldn't say i "reviewed it" -- but i at least started thinking about it a bit21:43
claygall the tests seem to be passing - we know it's not horribly broken21:43
kota_but also i know rledisez want to get it merged *really* so I'm now thinking if can i make it with single +2 or not21:44
claygI don't think I'm going to be re-spinning kinetic anytime soon21:44
claygso unless I have some other use-case for an out-of-tree diskfile I'm happy to let the entrypoints in and if we find an issue later we'll resolve it21:45
claygprobably with a bug like "can't use my out of tree disk file with new entrypoints because wah wah wah"21:45
timburketdasilva: so does it seem like swiftonfile would be able to use this instead of needing to provide its own object-server?21:45
claygthat would be pretty cool21:45
mattoliverauIll try and get some kind of review in this week, but definitly a better one in before next meeting. (/me is kinda on leave this week)21:46
tdasilvatimburke: IIRC swiftonfile overwrites some methods of the object-server, need to reload why21:46
kota_alright, thanks mattoliverau21:46
timburkenot that it *must* be done to get the patch landed -- just thinking it might make a good data point to think about whether it *could* be done21:46
claygmattoliverau: unless you'd be spinning testing with an out-of-tree disk file I'm not sure what  we'd cover new beyond what kota_ has seen21:46
tdasilvatimburke: yep! agreed21:47
claygkota_: is there anything specifically about how it interacts with the *in-tree* diskfiles that gives you pause?  Or you *only* reservation was out-of-tree diskfiles interop?21:47
mattoliverauYeah,  i won't be doin that21:47
claygmaybe another reviewer could leverage and give swift-on-file a whirl21:48
timburkei still wanna know what happens when you mis-configure the entry points -- do a replicated diskfile with an ec policy, say21:48
claygshould we do that before we merge?  do we owe that to diskfile consumers (cc zaitcev tdasilva)21:48
tdasilvatimburke had a good question on whether the option should be in object-server21:48
*** Leo_m_ has joined #openstack-meeting21:48
rlediseztimburke: it should raise an error and the process won't start21:48
timburkecool! thanks rledisez21:49
tdasilvaclayg: i'm really not concerned about swiftonfile in this case21:49
kota_clayg: i think it works with in-tree21:49
claygkota_:  then I think you did the right thing bringing to the meeting and I think we can merge it and probably should21:49
zaitcevclayg: Sadly I'm not that in touch with Gluster people who run swift-on-file. Not heard from them in years.21:49
kota_basically, what i concerns is with out-of-tree but for sure, another couple of eyes always fine to find something I missed.21:50
claygWFM!  I'm not trying to break anybody on purpose - this is just an issues of resources and needing to move our ball forward21:50
claygI don't think anyone is going to hate on us too long - we're doing our best21:50
tdasilvaclayg: I can reach out to gluster folks and see if they want to have a say...21:51
kota_yup, so that, i was thinking to do if we may have a long time.21:51
*** Leo_m has quit IRC21:51
claygtdasilva: ok, I'd be willing to hold on that response until next week21:51
tdasilvasounds good, i'll reach out to them tomorrow21:51
claygtdasilva: and that's probably pretty freaking amicable of you to offer that21:51
claygand if anyone did show up and ask if we could do xyz to make their lives easier - we'd be all for it - patches welcome!21:52
tdasilvaanything else on this topic?21:52
kota_summarizing that, mattoliverau will review it in this week, then tdasilva will ask to gluster folks, probably we can land it in the next week21:52
timburkerledisez: so with another diskfile implementation -- would you expect to switch over one policy at a time, or all policies for a single node?21:53
rlediseztimburke: as you prefer. we only manage one policy per node, so it's the same for us. but you can switch one policy at a time21:54
timburkeok. just trying to get a model in my head for how things are connected21:55
kota_thx timburke21:55
tdasilva#topic open discussion21:55
*** openstack changes topic to "open discussion (Meeting topic: swift)"21:55
tdasilvaanything else to discuss ?21:56
tdasilvarledisez, m_kazuhiro anything on 'general task'?21:56
tdasilvathere are still some patches on that need review, so I'd encourage to look at that21:57
zaitcevBTW, why are you running the meeting and what has happened to notmyname?21:58
tdasilvanotmyname had an appointment so he couldn't join21:58
kota_oic, s3 and swift cross compatibilities, we should discuss, probably it's in the next meeting. it will take longer time.21:58
tdasilvaso he asked me to run it today21:58
zaitcevgot it21:59
tdasilvahe should be back running it next week21:59
kota_just a minute left.21:59
tdasilvakota_: can you add to the agenda21:59
tdasilvathe rolling upgrade test job got merged on monday but there is still a ton of work/cleanup to there, checkout this etherpad if you would like to discuss help:
mattoliverauNice Thanks tdasilva22:00
clayg+1 tdasilva is awesome!!!22:00
tdasilvaok, going to call it here and we can continue on #openstack-swift if there's anything else to discuss22:00
claygsorry I took up so much time on the PUT+POST MIME :'(22:00
m_kazuhirotdasilva: we discussed updatiing process of general task, so I updating patch for that now.22:00
tdasilvaclayg: no problem, thanks for sharing all that, and zaitcev thanks for your time on it too22:00
kota_clayg: no worries, that's awesome.22:00
tdasilvam_kazuhiro: awesome!22:01
*** openstack changes topic to "OpenStack Meetings ||"22:01
openstackMeeting ended Wed Sep 19 22:01:21 2018 UTC.  Information about MeetBot at . (v 0.1.4)22:01
openstackMinutes (text):
mattoliverauThanks for chairing tdasilva, nice work!22:01
*** zaitcev has left #openstack-meeting22:01
*** timburke has left #openstack-meeting22:03
*** m_kazuhiro has quit IRC22:03
*** fatema_ has left #openstack-meeting22:04
*** patchbot has left #openstack-meeting22:07
*** rbudden has quit IRC22:41
*** ykatabam has joined #openstack-meeting22:42
*** rcernin has joined #openstack-meeting22:44
*** tpsilva has quit IRC22:54
*** Leo_m_ has quit IRC22:54
*** Leo_m has joined #openstack-meeting22:55
*** Leo_m has quit IRC22:55
*** armax has quit IRC22:59
*** armax has joined #openstack-meeting23:07
*** armax has quit IRC23:08
*** bobh has joined #openstack-meeting23:11
*** jamesmcarthur has joined #openstack-meeting23:14
*** rbudden has joined #openstack-meeting23:15
*** annabelleB has quit IRC23:15
*** jamesmcarthur has quit IRC23:19
*** slaweq has quit IRC23:19
*** rbudden has quit IRC23:24
*** dklyle has quit IRC23:29
*** rbudden has joined #openstack-meeting23:29
*** finix has quit IRC23:31
*** rbudden has quit IRC23:33
*** radeks has quit IRC23:42
*** mriedem is now known as mriedem_away23:44
*** aagate has joined #openstack-meeting23:58

Generated by 2.15.3 by Marius Gedminas - find it at!