| *** gyee has quit IRC | 00:17 | |
| *** raildo has quit IRC | 00:17 | |
| *** hongbin has quit IRC | 00:36 | |
| *** yamamoto has joined #openstack-meeting-alt | 00:45 | |
| *** Swami has quit IRC | 00:52 | |
| *** yamamoto has quit IRC | 01:18 | |
| *** yamamoto has joined #openstack-meeting-alt | 01:18 | |
| *** markvoelker has quit IRC | 01:33 | |
| *** betherly has joined #openstack-meeting-alt | 01:40 | |
| *** betherly has quit IRC | 01:44 | |
| *** iyamahat_ has joined #openstack-meeting-alt | 01:48 | |
| *** david-lyle has joined #openstack-meeting-alt | 01:48 | |
| *** iyamahat has quit IRC | 01:50 | |
| *** dklyle has quit IRC | 01:51 | |
| *** iyamahat__ has joined #openstack-meeting-alt | 01:57 | |
| *** iyamahat_ has quit IRC | 02:01 | |
| *** hongbin has joined #openstack-meeting-alt | 02:35 | |
| *** bhavikdbavishi has joined #openstack-meeting-alt | 02:41 | |
| *** bhavikdbavishi1 has joined #openstack-meeting-alt | 02:44 | |
| *** yamamoto has quit IRC | 02:45 | |
| *** bhavikdbavishi has quit IRC | 02:45 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 02:45 | |
| *** betherly has joined #openstack-meeting-alt | 02:51 | |
| *** betherly has quit IRC | 02:55 | |
| *** yamamoto has joined #openstack-meeting-alt | 03:24 | |
| *** iyamahat_ has joined #openstack-meeting-alt | 03:27 | |
| *** iyamahat__ has quit IRC | 03:30 | |
| *** yamahata has quit IRC | 03:30 | |
| *** iyamahat_ has quit IRC | 03:33 | |
| *** yamamoto has quit IRC | 03:34 | |
| *** tssurya has quit IRC | 03:42 | |
| *** pcarver has quit IRC | 03:56 | |
| *** diablo_rojo has quit IRC | 04:06 | |
| *** pcarver has joined #openstack-meeting-alt | 04:11 | |
| *** yamamoto has joined #openstack-meeting-alt | 04:29 | |
| *** betherly has joined #openstack-meeting-alt | 04:32 | |
| *** hongbin has quit IRC | 04:33 | |
| *** janki has joined #openstack-meeting-alt | 04:34 | |
| *** betherly has quit IRC | 04:37 | |
| *** lpetrut has joined #openstack-meeting-alt | 04:52 | |
| *** markstur has joined #openstack-meeting-alt | 05:03 | |
| *** apetrich has quit IRC | 05:07 | |
| *** markstur has quit IRC | 05:07 | |
| *** markstur has joined #openstack-meeting-alt | 05:12 | |
| *** lpetrut has quit IRC | 05:36 | |
| *** yamahata has joined #openstack-meeting-alt | 05:40 | |
| *** apetrich has joined #openstack-meeting-alt | 05:48 | |
| *** masahito has joined #openstack-meeting-alt | 06:02 | |
| *** markstur has quit IRC | 06:18 | |
| *** sridharg has joined #openstack-meeting-alt | 06:20 | |
| *** radeks__ has joined #openstack-meeting-alt | 06:49 | |
| *** kopecmartin|off is now known as kopecmartin | 07:02 | |
| *** bhavikdbavishi has quit IRC | 07:13 | |
| *** radeks_ has joined #openstack-meeting-alt | 07:22 | |
| *** jtomasek has joined #openstack-meeting-alt | 07:23 | |
| *** markstur has joined #openstack-meeting-alt | 07:23 | |
| *** radeks__ has quit IRC | 07:25 | |
| *** dpawlik has joined #openstack-meeting-alt | 07:28 | |
| *** markstur has quit IRC | 07:28 | |
| *** e0ne has joined #openstack-meeting-alt | 07:38 | |
| *** radeks_ has quit IRC | 07:50 | |
| *** rcernin has quit IRC | 07:56 | |
| *** rcernin has joined #openstack-meeting-alt | 08:03 | |
| *** rdopiera has joined #openstack-meeting-alt | 08:04 | |
| *** lpetrut has joined #openstack-meeting-alt | 08:06 | |
| *** slaweq has joined #openstack-meeting-alt | 08:06 | |
| *** slaweq has quit IRC | 08:15 | |
| *** markstur has joined #openstack-meeting-alt | 08:25 | |
| *** markstur has quit IRC | 08:30 | |
| *** rcernin has quit IRC | 08:33 | |
| *** bhavikdbavishi has joined #openstack-meeting-alt | 08:55 | |
| *** lpetrut has quit IRC | 08:57 | |
| *** markvoelker has joined #openstack-meeting-alt | 09:01 | |
| *** ttsiouts has joined #openstack-meeting-alt | 09:27 | |
| *** markvoelker has quit IRC | 09:34 | |
| *** derekh has joined #openstack-meeting-alt | 09:44 | |
| *** bhavikdbavishi has quit IRC | 09:49 | |
| *** iyamahat has joined #openstack-meeting-alt | 09:52 | |
| *** masahito has quit IRC | 09:56 | |
| *** tssurya has joined #openstack-meeting-alt | 09:59 | |
| *** ttsiouts has quit IRC | 10:00 | |
| *** ttsiouts has joined #openstack-meeting-alt | 10:01 | |
| *** ttsiouts has quit IRC | 10:05 | |
| *** e0ne has quit IRC | 10:24 | |
| *** e0ne has joined #openstack-meeting-alt | 10:27 | |
| *** dtrainor has quit IRC | 10:29 | |
| *** erlon has joined #openstack-meeting-alt | 10:31 | |
| *** markvoelker has joined #openstack-meeting-alt | 10:31 | |
| *** ttsiouts has joined #openstack-meeting-alt | 10:34 | |
| *** markstur has joined #openstack-meeting-alt | 10:45 | |
| *** ttsiouts has quit IRC | 10:50 | |
| *** ttsiouts has joined #openstack-meeting-alt | 10:50 | |
| *** markstur has quit IRC | 10:50 | |
| *** ttsiouts has quit IRC | 10:55 | |
| *** markvoelker has quit IRC | 11:05 | |
| *** dtrainor has joined #openstack-meeting-alt | 11:06 | |
| *** kopecmartin is now known as kopecmartin|pto | 11:06 | |
| *** ttsiouts has joined #openstack-meeting-alt | 11:20 | |
| *** jesusaur has quit IRC | 11:27 | |
| *** jesusaur has joined #openstack-meeting-alt | 11:31 | |
| *** cloudrancher has joined #openstack-meeting-alt | 11:35 | |
| *** cloudrancher has quit IRC | 11:42 | |
| *** ttsiouts has quit IRC | 11:43 | |
| *** ttsiouts has joined #openstack-meeting-alt | 11:44 | |
| *** bhavikdbavishi has joined #openstack-meeting-alt | 11:48 | |
| *** ttsiouts has quit IRC | 11:48 | |
| *** sambetts_ has quit IRC | 12:13 | |
| *** sambetts_ has joined #openstack-meeting-alt | 12:15 | |
| *** yamamoto has quit IRC | 12:22 | |
| *** sambetts_ has quit IRC | 12:26 | |
| *** ttsiouts has joined #openstack-meeting-alt | 12:28 | |
| *** sambetts_ has joined #openstack-meeting-alt | 12:28 | |
| *** erlon has quit IRC | 12:29 | |
| *** yamamoto has joined #openstack-meeting-alt | 12:31 | |
| *** markstur has joined #openstack-meeting-alt | 12:36 | |
| *** pbourke has quit IRC | 12:40 | |
| *** pbourke has joined #openstack-meeting-alt | 12:40 | |
| *** markstur has quit IRC | 12:40 | |
| *** ttsiouts has quit IRC | 12:40 | |
| *** ttsiouts has joined #openstack-meeting-alt | 12:41 | |
| *** erlon has joined #openstack-meeting-alt | 12:43 | |
| *** ttsiouts has quit IRC | 12:46 | |
| *** tpsilva has joined #openstack-meeting-alt | 12:48 | |
| *** betherly has joined #openstack-meeting-alt | 12:52 | |
| *** ttsiouts has joined #openstack-meeting-alt | 12:58 | |
| *** betherly has quit IRC | 13:01 | |
| *** markstur has joined #openstack-meeting-alt | 13:07 | |
| *** markstur has quit IRC | 13:12 | |
| *** rcernin has joined #openstack-meeting-alt | 13:17 | |
| *** rcernin has quit IRC | 13:29 | |
| *** liuyulong has joined #openstack-meeting-alt | 13:32 | |
| *** markstur has joined #openstack-meeting-alt | 13:39 | |
| *** rfolco has quit IRC | 13:41 | |
| *** rfolco has joined #openstack-meeting-alt | 13:41 | |
| *** tssurya has quit IRC | 13:41 | |
| *** yamamoto has quit IRC | 13:41 | |
| *** tssurya has joined #openstack-meeting-alt | 13:41 | |
| *** markstur has quit IRC | 13:44 | |
| *** dpawlik has quit IRC | 13:50 | |
| *** bhavikdbavishi has quit IRC | 13:53 | |
| *** yikun has quit IRC | 13:56 | |
| *** smyers_ has joined #openstack-meeting-alt | 13:57 | |
| *** smyers has quit IRC | 13:57 | |
| *** smyers_ is now known as smyers | 13:57 | |
| *** ttsiouts has quit IRC | 14:01 | |
| *** ttsiouts has joined #openstack-meeting-alt | 14:01 | |
| *** ttsiouts has quit IRC | 14:06 | |
| *** jcoufal has joined #openstack-meeting-alt | 14:10 | |
| *** markstur has joined #openstack-meeting-alt | 14:11 | |
| *** raildo has joined #openstack-meeting-alt | 14:11 | |
| *** markstur has quit IRC | 14:16 | |
| *** yamamoto has joined #openstack-meeting-alt | 14:24 | |
| *** ttsiouts has joined #openstack-meeting-alt | 14:24 | |
| *** cloudrancher has joined #openstack-meeting-alt | 14:30 | |
| *** cloudrancher has quit IRC | 14:30 | |
| *** smyers has quit IRC | 14:32 | |
| *** smyers has joined #openstack-meeting-alt | 14:32 | |
| *** janki has quit IRC | 14:33 | |
| *** yamamoto has quit IRC | 14:37 | |
| *** liuyulong has quit IRC | 14:40 | |
| *** ttsiouts has quit IRC | 14:41 | |
| *** ttsiouts has joined #openstack-meeting-alt | 14:42 | |
| *** markstur has joined #openstack-meeting-alt | 14:42 | |
| *** Swami has joined #openstack-meeting-alt | 14:44 | |
| *** markstur has quit IRC | 14:47 | |
| *** ttsiouts has quit IRC | 14:47 | |
| *** ttsiouts has joined #openstack-meeting-alt | 14:53 | |
| *** janki has joined #openstack-meeting-alt | 14:54 | |
| *** Swami has left #openstack-meeting-alt | 14:55 | |
| *** carthaca has joined #openstack-meeting-alt | 14:56 | |
| *** Swami has joined #openstack-meeting-alt | 14:56 | |
| *** ganso has joined #openstack-meeting-alt | 14:59 | |
| *** AlanClark has joined #openstack-meeting-alt | 15:00 | |
| tbarron | #startmeeting manilia | 15:01 |
|---|---|---|
| openstack | Meeting started Thu Dec 6 15:01:24 2018 UTC and is due to finish in 60 minutes. The chair is tbarron. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:01 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:01 |
| *** openstack changes topic to " (Meeting topic: manilia)" | 15:01 | |
| openstack | The meeting name has been set to 'manilia' | 15:01 |
| gouthamr | o/ | 15:01 |
| bswartz | .o/ | 15:01 |
| ganso | hello | 15:01 |
| tbarron | gouthamr: top of the morning to ya | 15:01 |
| tbarron | ping xyang | 15:02 |
| tbarron | ping markstur | 15:02 |
| tbarron | ping erlon | 15:02 |
| tbarron | ping tpsilva | 15:02 |
| tbarron | ;ping vkmc | 15:02 |
| tbarron | ping vkmc | 15:02 |
| tbarron | ping amito | 15:02 |
| tbarron | that's our ping list, | 15:02 |
| * tbarron waits a minute | 15:02 | |
| bswartz | tbarron: not worried about auto bans anymore? | 15:02 |
| ganso | tbarron: The meeting name has been set to 'manilia' | 15:03 |
| xyang | hi | 15:03 |
| tbarron | I'll end the meeting and restart | 15:03 |
| tbarron | thanks ganso | 15:03 |
| tbarron | #endmeeting | 15:03 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 15:03 | |
| openstack | Meeting ended Thu Dec 6 15:03:46 2018 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:03 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/manilia/2018/manilia.2018-12-06-15.01.html | 15:03 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/manilia/2018/manilia.2018-12-06-15.01.txt | 15:03 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/manilia/2018/manilia.2018-12-06-15.01.log.html | 15:03 |
| tbarron | #startmeeting manila | 15:03 |
| openstack | Meeting started Thu Dec 6 15:03:55 2018 UTC and is due to finish in 60 minutes. The chair is tbarron. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:03 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:03 |
| *** openstack changes topic to " (Meeting topic: manila)" | 15:03 | |
| openstack | The meeting name has been set to 'manila' | 15:04 |
| tbarron | for the record, I set the meeting name wrong and just reatarted | 15:04 |
| bswartz | .o/ | 15:04 |
| tbarron | please say hi again for the record | 15:04 |
| ganso | hello | 15:04 |
| tbarron | restarted | 15:04 |
| gouthamr | o/ :) | 15:04 |
| *** hongbin has joined #openstack-meeting-alt | 15:05 | |
| * tbarron notes that xyang said hi | 15:05 | |
| xyang | :) | 15:05 |
| carthaca | Hello | 15:05 |
| tbarron | bswartz: I don't think the bot will kick me off for pinging unicasts | 15:06 |
| tbarron | carthaca: hello! | 15:06 |
| tbarron | we have the guy who fixes more netapp bugs than netapp here :) | 15:06 |
| tbarron | Welcome Carthaca. | 15:06 |
| bswartz | >_< | 15:06 |
| tbarron | OK, we don't have big attendance but let's get started. | 15:07 |
| carthaca | gouthamr kinda invited me in https://bugs.launchpad.net/manila/+bug/1804659 :) | 15:07 |
| openstack | Launchpad bug 1804659 in Manila "speed up list storage pools" [High,In progress] - Assigned to Maurice Schreiber (maurice-schreiber) | 15:07 |
| tbarron | carthaca: k, we'll get to that pretty soon | 15:07 |
| *** alexchadin has quit IRC | 15:07 | |
| tbarron | Agenda: https://wiki.openstack.org/wiki/Manila/Meetings#Next_meeting | 15:07 |
| tbarron | #topic Announcements | 15:07 |
| *** openstack changes topic to "Announcements (Meeting topic: manila)" | 15:07 | |
| tbarron | I just want to note that the end-of-year holidays for the "western world" are coming up and that | 15:08 |
| tbarron | M2 milestone will be soon after | 15:08 |
| tbarron | M2 is the week of Jan. 7 | 15:08 |
| tbarron | We have bugs targeted for M2. | 15:09 |
| tbarron | And it's the new driver deadline. | 15:09 |
| tbarron | I think the only outstanding new driver is Pure, from last cycle. | 15:09 |
| *** jgrosso has joined #openstack-meeting-alt | 15:09 | |
| tbarron | Anyone know others? | 15:09 |
| ganso | tbarron: deadline only for brand new drivers, correct? | 15:10 |
| ganso | tbarron: driver changes could be proposed up to FPF, correct? | 15:10 |
| tbarron | Well we've got about 30 back ends now so I dunno we're desparate for more :) | 15:10 |
| *** dpawlik has joined #openstack-meeting-alt | 15:10 | |
| tbarron | ganso: correct | 15:10 |
| tbarron | ok, any more announcements? | 15:11 |
| tbarron | hearing none, .... | 15:11 |
| tbarron | #topic new bug czar | 15:11 |
| *** openstack changes topic to "new bug czar (Meeting topic: manila)" | 15:11 | |
| tbarron | gouthamr: your name is on this one | 15:11 |
| gouthamr | yep... ty tbarron | 15:11 |
| gouthamr | so, our last bug czar left Red Hat, and in the process has little to no time to dedicate to upstream development... :( | 15:12 |
| gouthamr | so we need a new bug czar, or a bug subteam even :) | 15:12 |
| gouthamr | do we have any volunteers? | 15:13 |
| tbarron | gouthamr: what would a subteam look like? | 15:13 |
| jgrosso | I will volunteer | 15:13 |
| gouthamr | jgrosso++ | 15:14 |
| *** markstur has joined #openstack-meeting-alt | 15:14 | |
| tbarron | terrific | 15:14 |
| gouthamr | tbarron: we'd probably rotate from week to week or split up responsibility between projects | 15:14 |
| *** jcoufal_ has joined #openstack-meeting-alt | 15:14 | |
| *** jcoufal_ has quit IRC | 15:15 | |
| *** dpawlik has quit IRC | 15:15 | |
| gouthamr | i was combing through bugs and right now we have a lot of cruft | 15:15 |
| gouthamr | i suspect we'd need a lot of initial workload of weeding out things we've already fixed | 15:15 |
| gouthamr | s/need/have | 15:15 |
| tbarron | gouthamr: ok, well we'll let jgrosso decide whether he wants a subteam to whom he can delegate some of the work and return to the topic in another week or two, sound good? | 15:15 |
| *** jcoufal_ has joined #openstack-meeting-alt | 15:16 | |
| gouthamr | tbarron: sure thing.. i can work with jgrosso to bring him up to speed | 15:16 |
| tbarron | gouthamr: you can get that "weeding" info to jgrosso and maybe spend a couple hours together cleaning out the garden | 15:17 |
| *** dpawlik has joined #openstack-meeting-alt | 15:17 | |
| tbarron | ok, let's move on for now | 15:17 |
| gouthamr | we'd need more than a couple of hours on manila's LP | 15:17 |
| tbarron | #topic Our Work for this cycle | 15:17 |
| jgrosso | :) | 15:17 |
| *** openstack changes topic to "Our Work for this cycle (Meeting topic: manila)" | 15:17 | |
| *** dpawlik has quit IRC | 15:17 | |
| *** jcoufal has quit IRC | 15:17 | |
| * tbarron is being ridiculously optimistic, PTL perogative | 15:17 | |
| *** dpawlik has joined #openstack-meeting-alt | 15:18 | |
| tbarron | #link https://review.openstack.org/#/c/572283/ | 15:18 |
| tbarron | This is the main priority for access rules review | 15:18 |
| tbarron | We carried it over from the last cycle and it has recently been refreshed. | 15:19 |
| *** markstur has quit IRC | 15:19 | |
| tbarron | I wonder if we should set up a call to workk on this one together and see if we can get it advanced. | 15:19 |
| gouthamr | so zhengzhenyu hasn't proposed any driver changes | 15:20 |
| bswartz | Is it feature complete? | 15:20 |
| gouthamr | ganso do you plan on fixing up the netapp driver? | 15:20 |
| ganso | gouthamr: as per we've agreed, it doesn't need fixing | 15:20 |
| *** sridharg has quit IRC | 15:21 | |
| ganso | gouthamr: actually, just a minor change to stop reordering when there are no priority conflicts | 15:21 |
| gouthamr | instinct says it does, because the driver reorders rules | 15:21 |
| ganso | gouthamr: we've agreed that drivers can still reorder if they want when there is a priority conflict between rules | 15:21 |
| gouthamr | ganso: oh, sure - that's what i meant | 15:22 |
| tbarron | but the driver needs to change to *not* re-order otherwise, i.e. to respect the order set by the manager | 15:22 |
| ganso | gouthamr: oh, then yes, I thought you implied that our driver would *break*. I meant that it will not break | 15:22 |
| ganso | tbarron: yes | 15:23 |
| *** slaweq has joined #openstack-meeting-alt | 15:23 | |
| tbarron | so bswartz I think this core patch is feature complete, we still would need driver work, client work, ui work, tempest-plugin work | 15:23 |
| ganso | tbarron: minor change though | 15:23 |
| gouthamr | ganso: i think we should move that reordering into the share manager for rules that still have the same priority | 15:24 |
| gouthamr | ganso: so we can do it consistently across drivers | 15:24 |
| tbarron | OK I don't want to get stuck, please review this one by next week. | 15:24 |
| ganso | gouthamr: I suggested that in the past, because I am personally against "undeterminate behavior" | 15:25 |
| tbarron | gouthamr: you are disagreeiing with the result of a previous discussion. I'm not saying we can't revisit it, just am observing | 15:25 |
| ganso | gouthamr: but we've agreed to not to, after a lot of discussion | 15:25 |
| gouthamr | okay i am probably forgetting the previous discussion | 15:25 |
| tbarron | previous discussion said let's maintain backwards compatability when the specified priority still leaves "ties" | 15:26 |
| tbarron | Please review this one by next week and have an opinion on whether we can safely merge it while the other work is pending. | 15:26 |
| gouthamr | ack | 15:27 |
| tbarron | like if we expose the changes via the client last | 15:27 |
| tbarron | We need to make progress and not get totally stuck on this one. | 15:27 |
| tbarron | #topic testing global change to devstack for py3 first by default | 15:27 |
| *** openstack changes topic to "testing global change to devstack for py3 first by default (Meeting topic: manila)" | 15:27 | |
| tbarron | #link https://review.openstack.org/#/c/623061/ | 15:28 |
| tbarron | gouthamr: anything we need to do on this? | 15:28 |
| gouthamr | tbarron: yes, the test failed.. i'll investigate and propose any fixes | 15:28 |
| tbarron | recheck? fix? | 15:28 |
| tbarron | gouthamr: ack | 15:29 |
| tbarron | I put this one in mostly for awareness. | 15:29 |
| *** slaweq has quit IRC | 15:29 | |
| tbarron | gouthamr: ask others to help on this work :) | 15:29 |
| gouthamr | http://logs.openstack.org/61/623061/2/check/manila-tempest-minimal-dsvm-lvm/d190cc1/logs/devstacklog.txt.gz#_2018-12-05_21_36_42_366 | 15:29 |
| * gouthamr :P manila server throws a 500, no big deal | 15:29 | |
| tbarron | ok, stuff to figure out | 15:30 |
| tbarron | good that we're doing this test | 15:30 |
| *** carlos_silva has joined #openstack-meeting-alt | 15:30 | |
| tbarron | #topic Bugs | 15:30 |
| *** openstack changes topic to "Bugs (Meeting topic: manila)" | 15:30 | |
| tbarron | #link https://etherpad.openstack.org/p/manila-bug-triage-pad | 15:31 |
| tbarron | let's take the bug carthaca mentioned first | 15:31 |
| gouthamr | #LINK https://bugs.launchpad.net/manila/+bug/1804659 | 15:32 |
| openstack | Launchpad bug 1804659 in Manila "speed up list storage pools" [High,In progress] - Assigned to Maurice Schreiber (maurice-schreiber) | 15:32 |
| gouthamr | thanks for being here carthaca! | 15:32 |
| tbarron | +1000 | 15:32 |
| *** slaweq has joined #openstack-meeting-alt | 15:32 | |
| tbarron | Note that with the "Edge" infra work discussed in the last couple of Summits and PTGs customers are contemplating | 15:33 |
| gouthamr | i believe this is a regression because of the way the host map is constructed post https://review.openstack.org/#/c/351034/ | 15:33 |
| tbarron | running with ~100 back ends | 15:33 |
| gouthamr | ^ that too | 15:34 |
| *** jhesketh has quit IRC | 15:34 | |
| gouthamr | i believe carthaca's direction is correct, but wanted to discuss it here first.. | 15:34 |
| tbarron | So we need to flush out scale/performance issues in the scheduler and api services when they interact with lots of back ends | 15:34 |
| gouthamr | #LINK https://review.openstack.org/#/c/619576/ | 15:35 |
| * tbarron notes, also races ... | 15:35 | |
| *** jhesketh has joined #openstack-meeting-alt | 15:35 | |
| carthaca | I already observed the pool detail list to take over 5 minutes with just 1000 shares in total and 5 netapp backends | 15:35 |
| carthaca | my cache really helps, but ideally I would like to find the source, why it takes so long :( | 15:36 |
| *** slaweq has quit IRC | 15:37 | |
| bswartz | wow | 15:37 |
| bswartz | Storage pool list is the one that's not served from the DB right? | 15:38 |
| gouthamr | carthaca: i think this is your culprit: https://github.com/openstack/manila/blob/fb17422/manila/scheduler/host_manager.py#L398 | 15:38 |
| gouthamr | bswartz: yes, we extract it from the scheduler | 15:39 |
| gouthamr | but teh scheduler is counting up the sizes of the shares to calculate the provisioned_capacity_gb | 15:39 |
| tbarron | gouthamr: but the share sizes are in the db, why would that be slow? | 15:40 |
| gouthamr | tbarron: if you have thousands of shares, that get all by host call can be slowing things up | 15:41 |
| carthaca | Maybe this is the right time to wish for osprofiler https://blueprints.launchpad.net/manila/+spec/manila-os-profiler ? :) | 15:42 |
| gouthamr | carthaca: huh, dunno how that's used in nova, glance, cinder, but is worth investigating | 15:44 |
| gouthamr | i'll test carthaca's code today and review, can others please take a look too? | 15:46 |
| tbarron | carthaca: thanks for working on this one, it seems important. | 15:46 |
| tbarron | gouthamr +1 | 15:46 |
| tbarron | Any more on this one now? | 15:47 |
| carthaca | gouthamr: thanks | 15:47 |
| tbarron | ok, I promised that we'd talk about carlos_silva's generic driver ssh bug | 15:48 |
| tbarron | #link https://bugs.launchpad.net/manila/+bug/1807126 | 15:48 |
| openstack | Launchpad bug 1807126 in Manila "Cannot create shares using generic driver" [Undecided,New] | 15:48 |
| tbarron | carlos_silva: are you around by any chance? | 15:48 |
| bswartz | Who is currently looking into this one? | 15:49 |
| tbarron | bswartz: it was just raised yesterday | 15:49 |
| bswartz | It seems like the classic SSH problem we've seen for years in different flavors | 15:49 |
| tbarron | I noticed that the same error that carlos_silva was reporting with xenial is showing up in gate with bionic | 15:49 |
| carlos_silva | tbarron: yeah, i'm here | 15:49 |
| tbarron | gouthamr asked carlos_silva to try the ssh connection manually and | 15:50 |
| tbarron | hi carlos_silva !!~ | 15:50 |
| bswartz | One workaround is to use SSH with password instead of keys | 15:50 |
| tbarron | carlos_silva: did you try that? ^^ | 15:50 |
| bswartz | The underlying problem is very hard to solve | 15:50 |
| carlos_silva | bswartz: yeah, i tried | 15:51 |
| carlos_silva | but it didn't works | 15:51 |
| tbarron | bswartz: I think we're using p/w rather than keys in CI and we still are hitting the issue | 15:51 |
| bswartz | tbarron: could be another issue then | 15:52 |
| tbarron | carlos_silva: didn't you report that ssh works manaully, but only after a very long time? | 15:52 |
| tbarron | I think paramiko times out after 15s | 15:52 |
| bswartz | The network tricks we play with the generic driver seem to interfere with nova doing the stuff it's supposed to do | 15:52 |
| tbarron | manually | 15:52 |
| bswartz | Could be we need to revisit the networking approach used by generic | 15:53 |
| bswartz | Too bad vponomaryov isn't still here | 15:53 |
| bswartz | He was the only one with a deep understanding of that code | 15:53 |
| tbarron | yeah | 15:53 |
| tbarron | carlos_silva: so as you see we no longer have experts in this area and | 15:53 |
| carlos_silva | tbarron: yes | 15:54 |
| gouthamr | so we have a config opt: "ssh_conn_timeout" | 15:54 |
| tbarron | part of the problem is that it's complicated and no one is paid to work on that driver | 15:54 |
| bswartz | It doesn't take a huge amount of skill to debug the problem | 15:54 |
| bswartz | But it does take a huge amount of time | 15:54 |
| tbarron | but University professors like it and we still document it as a reference because conceptually it is nice :) | 15:55 |
| bswartz | Because you have to read all the code for generic and linux networking, and painstakingly step through the code in a debugger to see what's going on | 15:55 |
| bswartz | I found horrifying things in paramiko by doing this ^ | 15:55 |
| *** dpawlik has quit IRC | 15:55 | |
| tbarron | carlos_silva: is your interest in this back end using it for production? | 15:55 |
| ganso | carlos_silva: maybe it would be useful to include in the bug report that ssh still works manually after a very long time | 15:55 |
| tbarron | ganso: +1 | 15:55 |
| gouthamr | carlos_silva: it's weird that the ssh client takes a long time, and it presents some issue with the underlying network? | 15:56 |
| gouthamr | https://github.com/openstack/manila/blob/0fd1b8f9fa40bdd504c9402dd5c43e86387671bd/manila/share/driver.py#L139 | 15:56 |
| gouthamr | ^ if you toggle that timeout, manila should allow waiting until timeout | 15:56 |
| carlos_silva | tbarron: no | 15:56 |
| bswartz | gouthamr: it could be a network issue, or a metadata server issue, or a paramiko issue | 15:56 |
| *** jaypipes has quit IRC | 15:56 | |
| carlos_silva | ganso: ok, i'll do it | 15:56 |
| tbarron | gouthamr: how do you know it's the network rather than someting in the service VM that makes it slow? | 15:56 |
| bswartz | That's not even counting possible new problems introduced by new version of nova/neutron | 15:57 |
| gouthamr | we haven't changed the service image over a couple of releases | 15:57 |
| *** jaypipes has joined #openstack-meeting-alt | 15:58 | |
| *** markstur has joined #openstack-meeting-alt | 15:58 | |
| tbarron | gouthamr: ack but we've had lots of indeterministic failures with timeouts and the log replaying the boot up sequence, suggesting to me that we were close to the threshold | 15:59 |
| * tbarron admits he doesn't know this area well thogh | 15:59 | |
| ganso | tbarron: I don't think increasing the timeout is a good alternative for our upstream CI | 15:59 |
| ganso | tbarron: this kind of thing shouldn't take a very long time | 15:59 |
| ganso | tbarron: last time I used it, it was instantenously | 15:59 |
| tbarron | anecdotes :) | 16:00 |
| tbarron | there are lots of timeouts in gate jobs, for a logn time | 16:00 |
| tbarron | ok, we're out of time | 16:00 |
| ganso | tbarron: s/instantenously/instantaneously | 16:00 |
| gouthamr | "Error reading SSH protocol banner" is a weird paramiko response - iirc it could happen for any number of reasons, a recent one that we saw was because of a wrong ssh key encryption algorithm | 16:00 |
| tbarron | see you in #openstack-manila | 16:00 |
| tbarron | Thanks everyone! | 16:01 |
| tbarron | #endmeeting | 16:01 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 16:01 | |
| *** jcoufal_ has quit IRC | 16:01 | |
| openstack | Meeting ended Thu Dec 6 16:01:13 2018 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:01 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/manila/2018/manila.2018-12-06-15.03.html | 16:01 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/manila/2018/manila.2018-12-06-15.03.txt | 16:01 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/manila/2018/manila.2018-12-06-15.03.log.html | 16:01 |
| *** ganso has left #openstack-meeting-alt | 16:01 | |
| *** carlos_silva has left #openstack-meeting-alt | 16:01 | |
| *** janki is now known as janki|dinner | 16:02 | |
| *** markstur has quit IRC | 16:03 | |
| *** jcoufal has joined #openstack-meeting-alt | 16:20 | |
| *** rdopiera has quit IRC | 16:28 | |
| *** e0ne has quit IRC | 16:30 | |
| *** carthaca has left #openstack-meeting-alt | 16:34 | |
| *** rossella_s has quit IRC | 16:38 | |
| *** ttsiouts has quit IRC | 16:49 | |
| *** ttsiouts has joined #openstack-meeting-alt | 16:50 | |
| *** ttsiouts has quit IRC | 16:55 | |
| *** Swami has quit IRC | 17:14 | |
| *** janki|dinner has quit IRC | 17:16 | |
| *** tssurya has quit IRC | 17:18 | |
| *** jtomasek has quit IRC | 17:32 | |
| *** AlanClark has quit IRC | 17:54 | |
| *** gyee has joined #openstack-meeting-alt | 17:55 | |
| *** AlanClark has joined #openstack-meeting-alt | 17:56 | |
| *** macza has joined #openstack-meeting-alt | 17:57 | |
| *** derekh has quit IRC | 17:59 | |
| *** Swami has joined #openstack-meeting-alt | 18:05 | |
| *** e0ne has joined #openstack-meeting-alt | 18:08 | |
| *** e0ne has quit IRC | 18:09 | |
| *** dave-mccowan has joined #openstack-meeting-alt | 18:29 | |
| *** slaweq has joined #openstack-meeting-alt | 18:30 | |
| *** yamamoto has joined #openstack-meeting-alt | 18:34 | |
| *** diablo_rojo has joined #openstack-meeting-alt | 18:35 | |
| *** slaweq has quit IRC | 18:36 | |
| *** yamamoto has quit IRC | 18:38 | |
| *** jcoufal has quit IRC | 18:40 | |
| *** jcoufal has joined #openstack-meeting-alt | 18:40 | |
| *** slaweq has joined #openstack-meeting-alt | 18:52 | |
| *** slaweq has quit IRC | 18:57 | |
| *** dpawlik has joined #openstack-meeting-alt | 19:22 | |
| *** dpawlik has quit IRC | 19:24 | |
| *** yamahata has quit IRC | 19:25 | |
| *** iyamahat has quit IRC | 19:26 | |
| *** alex_xu has quit IRC | 19:32 | |
| *** dpawlik has joined #openstack-meeting-alt | 19:39 | |
| *** iyamahat has joined #openstack-meeting-alt | 19:43 | |
| *** dpawlik has quit IRC | 19:44 | |
| *** iyamahat_ has joined #openstack-meeting-alt | 19:46 | |
| *** iyamahat has quit IRC | 19:49 | |
| *** yamahata has joined #openstack-meeting-alt | 20:02 | |
| *** rcernin has joined #openstack-meeting-alt | 20:12 | |
| *** david-lyle is now known as dklyle | 20:24 | |
| *** AlanClark has quit IRC | 20:34 | |
| *** rdopiera has joined #openstack-meeting-alt | 20:38 | |
| *** jcoufal has quit IRC | 21:19 | |
| *** rdopiera has quit IRC | 21:28 | |
| *** jgrosso has quit IRC | 21:47 | |
| *** raildo has quit IRC | 21:53 | |
| *** lbragstad has quit IRC | 23:08 | |
| *** lbragstad has joined #openstack-meeting-alt | 23:09 | |
| *** yamamoto has joined #openstack-meeting-alt | 23:56 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!