Tuesday, 2019-01-22

*** macza has joined #openstack-meeting00:00
*** bobh has joined #openstack-meeting00:06
*** tetsuro has joined #openstack-meeting00:10
*** igordc has joined #openstack-meeting00:22
*** macza has quit IRC00:24
*** erlon has joined #openstack-meeting00:42
*** Liang__ has joined #openstack-meeting00:44
*** jamesmcarthur has joined #openstack-meeting00:50
*** erlon has quit IRC01:05
*** _alastor_ has quit IRC01:05
*** igordc has quit IRC01:05
*** markvoelker has quit IRC01:08
*** slaweq has joined #openstack-meeting01:11
*** slaweq has quit IRC01:15
*** weshay is now known as weshay_PTO01:26
*** bobh has quit IRC01:40
*** jamesmcarthur has quit IRC01:52
*** markvoelker has joined #openstack-meeting02:09
*** hongbin has joined #openstack-meeting02:24
*** bbowen has quit IRC02:25
*** lbragstad_503 has quit IRC02:38
*** psachin has joined #openstack-meeting02:42
*** markvoelker has quit IRC02:43
*** bobh has joined #openstack-meeting02:44
*** hongbin has quit IRC02:58
*** hongbin has joined #openstack-meeting03:01
*** slaweq has joined #openstack-meeting03:11
*** apetrich has quit IRC03:14
*** slaweq has quit IRC03:15
*** hongbin has quit IRC03:16
*** hongbin has joined #openstack-meeting03:17
*** hongbin has quit IRC03:17
*** hongbin has joined #openstack-meeting03:17
*** hongbin has quit IRC03:20
*** hongbin has joined #openstack-meeting03:20
*** bobh has quit IRC03:21
*** hongbin has quit IRC03:21
*** bbowen has joined #openstack-meeting03:21
*** hongbin has joined #openstack-meeting03:21
*** hongbin has quit IRC03:22
*** hongbin has joined #openstack-meeting03:22
*** armax has joined #openstack-meeting03:23
*** rcernin has quit IRC03:35
*** markvoelker has joined #openstack-meeting03:41
*** baojg has joined #openstack-meeting03:48
*** tpatil has joined #openstack-meeting03:50
*** rcernin has joined #openstack-meeting03:50
*** lbragstad_503 has joined #openstack-meeting03:54
*** imsurit_ofc has joined #openstack-meeting03:54
*** rcernin has quit IRC03:57
*** rcernin has joined #openstack-meeting03:58
*** sagara has joined #openstack-meeting03:59
tpatil#startmeeting Masakari04:00
openstackMeeting started Tue Jan 22 04:00:44 2019 UTC and is due to finish in 60 minutes.  The chair is tpatil. Information about MeetBot at http://wiki.debian.org/MeetBot.04:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.04:00
*** openstack changes topic to " (Meeting topic: Masakari)"04:00
tpatilHi All04:00
openstackThe meeting name has been set to 'masakari'04:00
tpatilsagara: Hi04:01
*** ShilpaSD has joined #openstack-meeting04:01
tpatil#topic Critical Bugs04:01
*** openstack changes topic to "Critical Bugs (Meeting topic: Masakari)"04:01
tpatil#link https://review.openstack.org/#/c/627097/04:02
tpatilI have voted +2 on this patch04:02
tpatilsagara: Can you please review this patch?04:02
sagaratpatil: Yes, I will review it later04:03
tpatilIt's a critical issue so we should merge this patch as early as possible04:03
tpatilsagara: Thank you04:03
tpatilI don't see any other critical bug in LP04:05
tpatilMoving ahead04:05
tpatil#topic Stein workitems04:05
*** openstack changes topic to "Stein workitems (Meeting topic: Masakari)"04:05
*** Bhujay has joined #openstack-meeting04:05
tpatil Add event notification feature to masakari04:06
*** Bhujay has quit IRC04:06
tpatilWe should merge the specs soon04:06
tpatilI can see the implementation patches are ready for review as well04:07
*** Bhujay has joined #openstack-meeting04:07
ShilpaSDtpatil: requesting you to review specs https://review.openstack.org/#/c/473057/6/04:07
tpatil#link https://review.openstack.org/#/q/status:open+project:openstack/masakari+branch:master+topic:bp/notifications-in-masakari04:07
ShilpaSDtpatil: and patches related to it https://review.openstack.org/#/q/status:open+project:openstack/masakari+branch:master+topic:bp/notifications-in-masakari04:07
*** tashiromt has joined #openstack-meeting04:07
tpatilShilpaSD: We will review these patches04:08
*** Bhujay has quit IRC04:08
ShilpaSDtpatil: thank you04:08
*** Bhujay has joined #openstack-meeting04:08
*** bobh has joined #openstack-meeting04:09
ShilpaSDtpatil: also propsed BP and patch for04:09
ShilpaSDAdding progress details for recovery workflow, kindly review the same04:09
*** tetsuro_ has joined #openstack-meeting04:09
tpatilI have already reviewed these patches once, just waiting for other members to review  as well04:09
*** Bhujay has quit IRC04:09
ShilpaSDtpatil: okay, thank you04:09
tpatilShilpaSD: Will review this specs as well04:09
*** Bhujay has joined #openstack-meeting04:10
ShilpaSDtpatil: thank you04:10
tpatilI think we should target these two specs in Stein cycle04:10
*** Bhujay has quit IRC04:11
*** tetsuro has quit IRC04:11
*** Bhujay has joined #openstack-meeting04:11
tpatilFunctional Tests04:12
*** Bhujay has quit IRC04:12
*** baojg has quit IRC04:12
*** Bhujay has joined #openstack-meeting04:13
ShilpaSDtpatil: we are in the process of making list of functional tests so that all code will get cover, and then will start implementing the same using masakari client04:13
*** markvoelker has quit IRC04:13
*** Bhujay has quit IRC04:14
*** bobh has quit IRC04:14
tpatilShilpaSD: Ok,let's first add a functional CI job for running functional tests04:14
*** Bhujay has joined #openstack-meeting04:14
ShilpaSDtpatil: okay, will focus on adding CI job first04:15
tpatilThere are already some functional tests written in openstacksdk which can be easily added in masakari quickly04:15
tpatilAfter adding those tests, then you can start identifying what other tests can be added04:15
ShilpaSDtpatil: okay, thats sounds good start04:16
ShilpaSDtpatil: thank you for this input04:17
*** baojg has joined #openstack-meeting04:18
tpatilAdd masakari-monitors in devstack plugin04:18
tpatilslow progress, but plan is to add this support in stein cycle04:19
tpatilexcept running masakarihost-monitors which requires you to install pacemaker04:19
tpatilif pacemaker is not installed, it fails to start masakarihost-monitors04:20
ShilpaSDtpatil: ok, here we need to run masakari monitors process as a service similar to the devstack@n-cpu.service, we are looking into the same.04:23
tpatilShilpaSD: Ok04:23
tpatilDoes anyone wants to update about any Stein work items?04:23
*** baojg has quit IRC04:23
tashiromtI do not have update04:24
tpatilOk, Let's end this meeting early then04:25
ShilpaSDone more from my side, pl review https://review.openstack.org/#/c/626541/04:25
tpatilShilpaSD: Will do04:26
ShilpaSDtpatil: thank you04:26
tpatilOk, then ending this meeting early, Thank you all for joining.04:27
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"04:27
openstackMeeting ended Tue Jan 22 04:27:17 2019 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)04:27
openstackMinutes:        http://eavesdrop.openstack.org/meetings/masakari/2019/masakari.2019-01-22-04.00.html04:27
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/masakari/2019/masakari.2019-01-22-04.00.txt04:27
openstackLog:            http://eavesdrop.openstack.org/meetings/masakari/2019/masakari.2019-01-22-04.00.log.html04:27
*** tetsuro has joined #openstack-meeting04:31
*** tetsuro_ has quit IRC04:31
*** sagara has quit IRC04:36
*** snowgoggles has quit IRC04:51
*** imsurit_ofc has quit IRC04:53
*** Bhujay has quit IRC04:58
*** tpatil has quit IRC04:59
*** whoami-rajat has joined #openstack-meeting05:04
*** _alastor_ has joined #openstack-meeting05:05
*** Bhujay has joined #openstack-meeting05:09
*** hongbin has quit IRC05:10
*** markvoelker has joined #openstack-meeting05:10
*** _alastor_ has quit IRC05:10
*** slaweq has joined #openstack-meeting05:11
*** baojg has joined #openstack-meeting05:11
*** slaweq has quit IRC05:15
*** baojg has quit IRC05:22
*** sridharg has joined #openstack-meeting05:23
*** baojg has joined #openstack-meeting05:23
*** macza has joined #openstack-meeting05:24
*** macza has quit IRC05:29
*** slaweq has joined #openstack-meeting05:35
*** Bhujay has quit IRC05:37
*** ekcs has quit IRC05:38
*** markvoelker has quit IRC05:43
*** igordc has joined #openstack-meeting05:45
*** radeks has joined #openstack-meeting05:51
*** _alastor_ has joined #openstack-meeting05:52
*** e0ne has joined #openstack-meeting05:52
*** e0ne has quit IRC05:53
*** _alastor_ has quit IRC05:57
*** lbragstad_503 has quit IRC05:57
*** vishakha has joined #openstack-meeting06:06
*** imsurit_ofc has joined #openstack-meeting06:16
*** ijw has quit IRC06:21
*** dkushwaha has joined #openstack-meeting06:33
*** vishalmanchanda has joined #openstack-meeting06:34
*** markvoelker has joined #openstack-meeting06:40
*** Luzi has joined #openstack-meeting06:49
*** imsurit_ofc has quit IRC06:59
*** a-pugachev has joined #openstack-meeting07:08
*** lpetrut has joined #openstack-meeting07:12
*** markvoelker has quit IRC07:14
*** a-pugachev has quit IRC07:24
*** aojea has joined #openstack-meeting07:27
*** igordc has quit IRC07:27
*** bobh has joined #openstack-meeting07:29
*** bhagyashris_ has joined #openstack-meeting07:31
*** Liang__ has quit IRC07:34
*** bobh has quit IRC07:35
*** Liang__ has joined #openstack-meeting07:36
*** baojg has quit IRC07:36
*** tssurya has joined #openstack-meeting07:41
*** pcaruana has joined #openstack-meeting07:41
*** rtjure has quit IRC07:42
*** rtjure has joined #openstack-meeting07:44
*** hyunsikyang has joined #openstack-meeting07:46
*** tpatil has joined #openstack-meeting07:52
*** jaewookoh has joined #openstack-meeting07:55
*** JangwonLee has joined #openstack-meeting07:55
*** jaewook_oh has joined #openstack-meeting07:55
*** phuoc__ has joined #openstack-meeting07:56
*** hjwon has joined #openstack-meeting08:02
*** baojg has joined #openstack-meeting08:03
dkushwaha#startmeeting tacker08:03
openstackMeeting started Tue Jan 22 08:03:08 2019 UTC and is due to finish in 60 minutes.  The chair is dkushwaha. Information about MeetBot at http://wiki.debian.org/MeetBot.08:03
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.08:03
*** openstack changes topic to " (Meeting topic: tacker)"08:03
openstackThe meeting name has been set to 'tacker'08:03
dkushwaha#startmeeting tacker08:03
openstackdkushwaha: Error: Can't start another meeting, one is in progress.  Use #endmeeting first.08:03
dkushwaha#topic Roll Call08:04
*** openstack changes topic to "Roll Call (Meeting topic: tacker)"08:04
dkushwahawho is here for weekly meeting?08:04
phuoc__hi dkushwaha08:04
phuoc__I think we should end the old meeting first08:04
phuoc__lest me try08:04
*** macza has joined #openstack-meeting08:05
phuoc__nothing happens :)08:05
dkushwahaphuoc__, seems no another meeting started as I got error that:  Can't start another meeting, one is in progress08:06
phuoc__I think it's fine08:07
dkushwahahello phuoc__ bhagyashris_08:07
bhagyashris_dkushwaha, phuoc___: Hi08:07
dkushwahaok lets start..08:08
*** kopecmartin|off is now known as kopecmartin08:08
dkushwaha#chair phuoc__08:08
openstackCurrent chairs: dkushwaha phuoc__08:08
dkushwaha#topic Denever CFP08:08
*** openstack changes topic to "Denever CFP (Meeting topic: tacker)"08:08
*** macza has quit IRC08:09
dkushwahaTomorrow is last date for submitting proposal08:09
*** apetrich has joined #openstack-meeting08:10
dkushwahaI have prepared some initial topics for session08:10
dkushwaha#link https://etherpad.openstack.org/p/Tacker-CFP--Denever-summit08:10
*** markvoelker has joined #openstack-meeting08:10
dkushwahaplease feel free to add your name there08:11
dkushwahaif you are adding talk, you can add that on link too08:11
dkushwahaphuoc__, bhagyashris_ if you have some more/better idea, please suggest08:13
phuoc__I think one session about auto healing and monitoring VNF is good08:13
phuoc__dkushwaha and bhagyashris_, you guys can make a presentation about it I think08:14
bhagyashris_dkushwaha, phuoc__:  sure08:15
hyunsikyangHi:) I am late!08:16
dkushwahaphuoc__, I have 2 points on that: 1: we can add your both points in Topic #1 with existing points(as I missed it to add them).08:17
dkushwaha2: prepare a separate session for only those topics in details08:17
bhagyashris_phuoc__, I will prepare presentation for vdu autohealing . and are you asking about mistral monitoring ?08:18
dkushwahahello hyunsikyang08:19
phuoc__because VNF is important object, with monitoring and auto healing VNFs08:20
phuoc__we can support managing VNFs better,08:20
phuoc__IMO, it can be a separate session08:21
dkushwahaphuoc__, I am agree to keep a separate session on that. bhagyashris_ would you like to join that tession?08:22
dkushwahaor you plan to submit alone?08:23
bhagyashris_dkushwaha: Not decided but someone fron NTT will join08:24
dkushwahaok, So I just added the topic there08:26
hyunsikyangWhen is the deadline for this?08:27
dkushwahaPlease feel free to add names and kindly help on abstract drafting08:27
dkushwahahyunsikyang, deadline is by tomorrow08:27
dkushwahawe needs to be hurry08:27
hyunsikyangAh ok  Actually , i am working with phuoc. so we also consider for this.08:29
*** ralonsoh has joined #openstack-meeting08:30
dkushwahathanks hyunsikyang . so please add your name08:30
dkushwahaif interested08:31
dkushwahamoving next..08:31
dkushwaha#topic BPs08:31
*** openstack changes topic to "BPs (Meeting topic: tacker)"08:31
dkushwahaMistral workflow monitoring08:32
dkushwaha#link https://review.openstack.org/#/c/486924/08:32
*** hokeen has joined #openstack-meeting08:33
dkushwahafunctional test are failing on multinoe, so I am working on fixing that08:33
phuoc__I hope that will be working soon :)08:34
dkushwahaphuoc__, hoping same :)08:34
dkushwahanothing else update on this patch08:34
dkushwaha#topic vdu_autoheal08:35
*** openstack changes topic to "vdu_autoheal (Meeting topic: tacker)"08:35
dkushwaha#link https://review.openstack.org/#/c/612595/08:36
dkushwahabhagyashris_, thanks for your great work on this spec.08:36
dkushwahaHope test case issue will be fixed soon08:37
bhagyashris_dkushwaha: thank you :)08:37
phuoc__Should we merge this patch first, https://review.openstack.org/#/c/63143908:37
bhagyashris_dkushwaha: yes i am working on fixing the test case issue08:37
phuoc__And remove it in https://review.openstack.org/#/c/61259508:37
dkushwahaphuoc__, It seems https://review.openstack.org/#/c/631439 is not fixing the issue08:39
bhagyashris_dkushwaha, phuoc__:  as on master the _hosting_vnfs is defined as the class level attribute but it is stired as instance level attribute in every place so i will prospse the change that the class level attribute will be stored as class level attribute only08:39
phuoc__I am looking for the patch that can fix the issue08:41
*** markvoelker has quit IRC08:43
*** tashiromt has quit IRC08:43
dkushwahamoving next..08:44
bhagyashris_I have just pushed the patch https://review.openstack.org/63143908:45
dkushwahabhagyashris_, great, will review it08:46
bhagyashris_I have one doubt from the begging tacker used _hosting_vnfs as class level attribute but every where that is stored as instance level attribute so _hostingPvnfs should be class level or instance level attribute08:47
dkushwahabhagyashris_, I didn't observed any issue related to that in existing code earlier, will check it again08:51
bhagyashris_dkushwaha: ok08:52
dkushwahaphuoc__, any update from you?08:52
phuoc__about force delete VNFs, because my server has problem so I will push patches this week08:53
dkushwahaphuoc__, cool08:54
dkushwaha#topic Open Discussion08:55
*** openstack changes topic to "Open Discussion (Meeting topic: tacker)"08:55
phuoc__I see there is a guy he want to have demo about rolling upgrade08:55
phuoc__OpenStack rolling upgrade with zero downtime to application08:55
*** _JangwonLee_ has joined #openstack-meeting08:55
phuoc__he want to integrate Fenix and VNFM08:55
phuoc__if you guy interested in this demo, we can do it :)08:56
phuoc__it is for ONS North America Demo08:56
*** electrofelix has joined #openstack-meeting08:56
*** rcernin has quit IRC08:56
dkushwahaphuoc__, any link ?08:56
*** priteau has quit IRC08:56
phuoc__it's OPNFV discuss08:57
*** priteau has joined #openstack-meeting08:57
dkushwahaphuoc__, seems interesting.08:57
phuoc__I will find the mail, or forward it to you guys08:57
dkushwahaphuoc__, yes, please share08:58
hyunsikyangplease share it thanks.08:58
*** JangwonLee has quit IRC08:58
dkushwahahyunsikyang, do you have something to discuss?08:58
hyunsikyangActually i didn't submit blueprint yet.08:59
hyunsikyang but, I want to extend networking features for C-VNF08:59
hyunsikyangSo I will upload blueprint ASAP09:00
dkushwahahyunsikyang, IMO there is already an existing BP for that09:00
hyunsikyangi didn't see it09:00
hyunsikyangi will check it again. could give me the info for that BP?09:00
dkushwahahyunsikyang, ok let me check again09:01
*** a-pugachev has joined #openstack-meeting09:01
dkushwahawe are running out of time so closing this meeting here and we can continue on our IRC channel.09:02
hyunsikyangok thanks09:02
dkushwahaThanks Fols09:02
dkushwahaThanks Folks..09:02
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"09:02
phuoc__thank, see you guys09:02
openstackMeeting ended Tue Jan 22 09:02:39 2019 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)09:02
openstackMinutes:        http://eavesdrop.openstack.org/meetings/tacker/2019/tacker.2019-01-22-08.03.html09:02
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/tacker/2019/tacker.2019-01-22-08.03.txt09:02
openstackLog:            http://eavesdrop.openstack.org/meetings/tacker/2019/tacker.2019-01-22-08.03.log.html09:02
*** hokeen has quit IRC09:03
*** _JangwonLee_ has quit IRC09:04
*** JangwonLee has joined #openstack-meeting09:04
*** yahrens has joined #openstack-meeting09:04
*** tpatil has quit IRC09:04
*** armax has quit IRC09:23
*** tssurya has quit IRC09:25
*** tetsuro has quit IRC09:36
*** iyamahat_ has quit IRC09:40
*** markvoelker has joined #openstack-meeting09:40
*** Liang__ has quit IRC09:42
*** bhagyashris_ has quit IRC09:44
*** imsurit_ofc has joined #openstack-meeting09:46
*** _alastor_ has joined #openstack-meeting09:53
*** baojg has quit IRC09:57
*** baojg has joined #openstack-meeting09:58
*** _alastor_ has quit IRC09:58
*** iyamahat has joined #openstack-meeting10:01
*** erlon has joined #openstack-meeting10:08
*** baojg has quit IRC10:09
*** markvoelker has quit IRC10:13
*** erlon_ has joined #openstack-meeting10:23
*** e0ne has joined #openstack-meeting10:24
*** erlon has quit IRC10:26
*** imsurit_ofc has quit IRC10:32
*** hjwon has quit IRC10:34
*** tetsuro has joined #openstack-meeting10:34
*** davidsha has joined #openstack-meeting10:42
*** mahatic has joined #openstack-meeting10:51
*** imsurit_ofc has joined #openstack-meeting10:55
*** markvoelker has joined #openstack-meeting11:10
*** apetrich has quit IRC11:13
*** imsurit_ofc has quit IRC11:14
*** phuoc__ has left #openstack-meeting11:32
*** sridharg has quit IRC11:44
*** markvoelker has quit IRC11:44
*** bdperkin has joined #openstack-meeting11:46
*** sridharg has joined #openstack-meeting11:47
*** radeks_ has joined #openstack-meeting11:52
*** radeks has quit IRC11:55
*** apetrich has joined #openstack-meeting11:57
*** bbowen has quit IRC11:58
*** bobh has joined #openstack-meeting12:03
*** bobh has quit IRC12:08
*** macza has joined #openstack-meeting12:21
*** jaewook_oh has quit IRC12:21
*** ttsiouts has joined #openstack-meeting12:22
*** gary_perkins has quit IRC12:24
*** e0ne has quit IRC12:25
*** macza has quit IRC12:26
*** gary_perkins has joined #openstack-meeting12:26
*** e0ne has joined #openstack-meeting12:30
*** gary_perkins has quit IRC12:38
*** tssurya has joined #openstack-meeting12:38
*** markvoelker has joined #openstack-meeting12:41
*** gary_perkins has joined #openstack-meeting12:41
*** gary_perkins has quit IRC12:44
*** gary_perkins has joined #openstack-meeting12:46
*** tetsuro has quit IRC12:53
*** baojg has joined #openstack-meeting13:00
*** radeks_ has quit IRC13:06
*** radeks_ has joined #openstack-meeting13:06
*** markvoelker has quit IRC13:09
*** bbowen has joined #openstack-meeting13:11
*** mmethot has joined #openstack-meeting13:15
*** psachin has quit IRC13:16
*** mriedem has joined #openstack-meeting13:18
*** lpetrut has quit IRC13:20
*** lpetrut has joined #openstack-meeting13:28
*** mmethot has quit IRC13:39
*** mmethot has joined #openstack-meeting13:40
*** rtjure has quit IRC13:42
*** mjturek has joined #openstack-meeting13:42
*** rtjure has joined #openstack-meeting13:45
*** priteau has quit IRC13:46
*** arne_wiebalck_ has joined #openstack-meeting13:56
*** lbragstad_503 has joined #openstack-meeting13:56
*** armstrong has joined #openstack-meeting14:00
*** lbragstad_503 is now known as lbragstad14:03
*** e0ne has quit IRC14:05
*** e0ne has joined #openstack-meeting14:08
*** njohnston has joined #openstack-meeting14:15
*** mmethot has quit IRC14:15
*** mmethot has joined #openstack-meeting14:17
*** bbowen_ has joined #openstack-meeting14:18
*** TxGirlGeek has joined #openstack-meeting14:19
*** mriedem has quit IRC14:19
*** mmethot has quit IRC14:19
*** mmethot has joined #openstack-meeting14:20
*** bbowen has quit IRC14:21
*** mriedem has joined #openstack-meeting14:21
*** priteau has joined #openstack-meeting14:21
*** lbragstad has quit IRC14:23
*** mmethot has quit IRC14:23
*** mmethot has joined #openstack-meeting14:24
*** mmethot has quit IRC14:25
*** mmethot has joined #openstack-meeting14:25
*** lbragstad has joined #openstack-meeting14:26
*** mmethot_ has joined #openstack-meeting14:28
*** mmethot has quit IRC14:31
*** mmethot_ has quit IRC14:31
*** mmethot_ has joined #openstack-meeting14:32
*** ttsiouts has quit IRC14:39
*** TxGirlGeek has quit IRC14:39
*** ttsiouts has joined #openstack-meeting14:39
*** ttsiouts has quit IRC14:41
*** ttsiouts has joined #openstack-meeting14:41
*** mmethot_ has quit IRC14:43
*** mmethot_ has joined #openstack-meeting14:45
*** efried_mlk is now known as efried14:45
*** mmethot_ has quit IRC14:46
*** mmethot_ has joined #openstack-meeting14:47
*** mmethot_ has quit IRC14:48
*** mmethot_ has joined #openstack-meeting14:48
*** mmethot_ has quit IRC14:53
*** mmethot has joined #openstack-meeting14:54
*** mmethot has quit IRC14:59
*** mmethot has joined #openstack-meeting15:00
*** efried1 has joined #openstack-meeting15:00
*** efried has quit IRC15:01
*** efried1 is now known as efried15:01
*** awaugama has joined #openstack-meeting15:03
*** eharney has joined #openstack-meeting15:05
*** wwriverrat has quit IRC15:06
*** hongbin has joined #openstack-meeting15:09
*** Luzi has quit IRC15:10
*** liuyulong has joined #openstack-meeting15:13
*** _alastor_ has joined #openstack-meeting15:20
*** bobh has joined #openstack-meeting15:23
*** _alastor_ has quit IRC15:24
*** bobh has quit IRC15:26
*** ijw has joined #openstack-meeting15:28
*** eharney has quit IRC15:32
*** jamesmcarthur has joined #openstack-meeting15:42
*** sridharg has quit IRC15:43
*** lpetrut has quit IRC15:46
*** jamesmcarthur_ has joined #openstack-meeting15:46
*** eharney has joined #openstack-meeting15:46
*** jamesmcarthur has quit IRC15:50
*** _alastor_ has joined #openstack-meeting15:50
*** TxGirlGeek has joined #openstack-meeting15:52
*** dklyle has joined #openstack-meeting15:53
*** TxGirlGe_ has joined #openstack-meeting15:55
*** TxGirlGeek has quit IRC15:57
*** mmethot has quit IRC15:57
*** jamesmcarthur_ has quit IRC15:57
*** armax has joined #openstack-meeting15:57
*** yahrens has quit IRC15:58
*** mlavalle has joined #openstack-meeting15:59
*** arne_wiebalck_ has quit IRC15:59
*** efried has quit IRC16:00
slaweq#startmeeting neutron_ci16:00
openstackMeeting started Tue Jan 22 16:00:32 2019 UTC and is due to finish in 60 minutes.  The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot.16:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:00
*** openstack changes topic to " (Meeting topic: neutron_ci)"16:00
openstackThe meeting name has been set to 'neutron_ci'16:00
*** TxGirlGe_ has quit IRC16:01
slaweqhaleyb: njohnston hongbin: are You around for CI meeting?16:01
haleybhi, i might have to leave early though16:01
slaweqok, lets start then16:02
slaweq#topic Actions from previous meetings16:02
*** openstack changes topic to "Actions from previous meetings (Meeting topic: neutron_ci)"16:02
slaweqthere wasn't too many actions from last week16:02
slaweqslaweq to make e-r query for bug 181151516:02
openstackbug 1811515 in neutron "pyroute2.NetNS don't work properly with concurrency in oslo.privsep" [Critical,Fix released] https://launchpad.net/bugs/1811515 - Assigned to Slawek Kaplonski (slaweq)16:02
*** dmacpher has quit IRC16:03
slaweqI didn't write this query but bug is now fixed (workarounded) so it's not necessary anymore16:03
slaweqnext one:16:03
slaweqslaweq to check if oslo.privsep < 1.31.0 will help to workaround issue with SSH to FIP16:03
mlavalleit did16:03
slaweqas above, this is now fixed16:03
slaweqit did help but we workarounded it in different way16:03
slaweqand the last one was:16:04
bcafarelo/ too16:04
slaweqslaweq to post more examples of failiures in bug 181151516:04
openstackbug 1811515 in neutron "pyroute2.NetNS don't work properly with concurrency in oslo.privsep" [Critical,Fix released] https://launchpad.net/bugs/1811515 - Assigned to Slawek Kaplonski (slaweq)16:04
slaweqI think I added link to logstash query from where more example could be found16:04
slaweqbut it's already fixed16:04
slaweqso that was all from last week :)16:04
mlavalleyou did16:04
mlavallenot quite16:04
mlavalleI had an action item16:04
slaweqok, so I somehow missed it16:05
slaweqsorry mlavalle16:05
slaweqgo on16:05
mlavallecontinue working on https://bugs.launchpad.net/neutron/+bug/179587016:05
openstackLaunchpad bug 1795870 in neutron "Trunk scenario test test_trunk_subport_lifecycle fails from time to time" [High,In progress] - Assigned to Miguel Lavalle (minsel)16:05
mlavallewhich I did16:05
mlavallesubmitted DNM patch https://review.openstack.org/#/c/630778/16:05
mlavallewhich I rechecked a few times and I got lucky16:06
mlavalleI got one succesful run and one failure right afterwards16:06
mlavallethat is allowing me to compare sucess / failure:16:06
mlavalle1) When the test passes, the router is being hosted both in the copntroller and the compute16:07
mlavalleso we see the router in the logs of both L3 agents16:07
mlavalle2) When the test fails, the router is never scheduled in the controller and it doesn't show up in its L3 agent16:08
*** jamesmcarthur has joined #openstack-meeting16:08
slaweqhmm, it should be scheduled to controller always as only there is dhcp port, right?16:08
mlavalleThis is an example of L3 agent log in a failed execution in the controller: http://logs.openstack.org/78/630778/1/check/neutron-tempest-plugin-dvr-multinode-scenario/02391e0/controller/logs/screen-q-svc.txt.gz?level=TRACE16:08
*** TxGirlGeek has joined #openstack-meeting16:09
*** efried has joined #openstack-meeting16:09
mlavalleplease note that the L3 agent is down according to the neutron server16:09
mlavalleI am seeing the same pattern in at least two cases16:09
slaweqwhy agent is down?16:10
slaweqdo You know?16:10
mlavallethe agent is not actually down16:10
mlavalleit is running16:10
mlavallebut the server thinks is down16:11
*** jamesmcarthur has quit IRC16:11
mlavalleand my next step is to investigate why16:11
mlavalleso please give me an action item for next week16:11
slaweq#action mlavalle to continue investigate why L3 agent is considered as down and cause trunk tests fail16:12
slaweqthx mlavalle for working on this16:12
slaweqthat is interesting why this agent is treated as down16:12
mlavalleyeah, in the same node16:12
*** jamesmcarthur has joined #openstack-meeting16:13
slaweqI wonder if agent is not sending heartbeat or neutron-server is not processing it properly16:13
slaweqIMHO it's more likely that agent is not sending it16:14
slaweqas heartbeats from other agents are ok on server16:14
slaweqbut we will see when You will check it :)16:14
slaweqok, so now I think we can move on to the next topic16:15
slaweq#topic Python 316:15
*** openstack changes topic to "Python 3 (Meeting topic: neutron_ci)"16:15
*** dims has quit IRC16:15
slaweqEtherpad: https://etherpad.openstack.org/p/neutron_ci_python316:15
slaweqin last week we merged 2 patches and now neutron-tempest-linuxbridge and neutron-tempest-dvr jobs are running on python 3 and using zuulv3 syntax already16:16
slaweqI plan to do the same with neutron-tempest-dvr-ha-multinode-full this week16:16
slaweqthere is also this neutron-functional job which needs to be switched16:17
slaweqand that is still problematic16:17
slaweqThis week I sent email to ML: http://lists.openstack.org/pipermail/openstack-discuss/2019-January/001904.html16:17
slaweqmaybe someone more familiar with ostestr/subunit and python 3 will be able to help us with it16:18
mlavalledid you get any responses so far?16:18
slaweqnope :/16:18
slaweqI will ask tomorrow on openstack-qa channel - maybe e.g. gmann will know who can help us with it16:19
bcafarelcrossing fingers16:20
njohnstongood idea16:20
*** dims has joined #openstack-meeting16:20
slawequnfortunatelly except that I have no any other idea how to deal with this issue currently :/16:20
slaweqand that's all related to python 3 from me16:21
slaweqnjohnston: do You have anything else to add?16:21
slaweqhow it's going with grenade jobs switch?16:21
bcafarelyeah pushing to add other limits to log output does not sound good as a viable fix :/ hopefully someone will fix the root cause and we get all our jobs py3-green16:21
njohnstonNo, nothing for this week.  I have been focused on bulk ports, but I should be ablke to move forward on the grenade work this week16:21
slaweqnjohnston: great, thx16:22
*** factor has joined #openstack-meeting16:22
njohnston#action njohnston Work on grenade job transition16:22
slaweqthx :)16:22
slaweqok, so lets go to the next topic16:22
slaweq#topic Grafana16:22
*** openstack changes topic to "Grafana (Meeting topic: neutron_ci)"16:22
slaweqthere is peak on many jobs today, I don't know exactly why it is like that but:16:24
slaweq1. I didn't found in today's jobs anything very bad,16:25
slaweq2. as it is even in pep8 job, I think that it could be some "generic" issue, not related to neutron bug directly16:25
*** ianychoi has joined #openstack-meeting16:26
slaweqfrom other things I think that we again have one major issue16:26
slaweqand this week it is tempest-slow job :/16:26
mlavalleit is evident in Grafana16:26
slaweqso, if You don't have anything else related to grafana, lets move to talk about this tempest jobs now16:27
slaweqok, I get it as yes :)16:28
slaweq#topic Tempest/Scenario16:28
*** openstack changes topic to "Tempest/Scenario (Meeting topic: neutron_ci)"16:28
slaweqWe have one major issue with tempest-slow job, it is described in bug report https://bugs.launchpad.net/neutron/+bug/181255216:29
openstackLaunchpad bug 1812552 in neutron "tempest-slow tests fails often" [Critical,In progress] - Assigned to Slawek Kaplonski (slaweq)16:29
slaweqToday I was talking with seon-k-mooney about that16:29
mlavalledid he help?16:29
bcafarelslaweq: for today's jobs there was some general post failure ~20h ago that probably did not help in the graph (sorry lagging as usual)16:29
slaweqbcafarel: yes, that might be the reason :)16:30
mlavallethanks anyway bcafarel16:30
mlavallemuch appreciated16:30
slaweqok, so tempest-slow job issue in details:16:30
slaweq1. from grafana and logstash it looks that this issue was caused somehow by https://review.openstack.org/#/c/631584/16:30
slaweq2. I proposed revert of this patch https://review.openstack.org/#/c/631944/ and indeed this job passed 5 or 6 times already16:31
slaweqin fact it didn't failed even once with this revert16:31
slaweqwhich confirms somehow that this patch is culprit16:31
slaweq3. I talked with sean today and we though that maybe this my patch introduced some additional race in ovsdb monitor and how it handles ports events on ovs bridges16:32
mlavalleso we are leaving interfaces un-plugged, right?16:32
slaweqbut, what is strange for me is fact that it cause issues only in this job, and only (at least where I was checking it) with two tests which shelve/unshelve instance16:33
slaweqmlavalle: not exactly16:33
slaweqtoday I invesigated logs from one of such jobs carefully16:33
slaweq(please read my last comment in bug report)16:33
slaweqand it looks for me that port was configured properly16:34
slaweqand communication, at least from VM to dhcp server was fine16:34
slaweqwhich puzzled me even more16:34
mlavallebut the VM doesn't get the dhcp offer, right?16:35
slaweqI have no idea what is wrong there and how (if) this mentioned patch could break it16:35
slaweqmlavalle: it looks so16:35
slaweqbut dnsmasq get DHCP Request from VM16:35
slaweqand sends this DHCP offer16:35
mlavalleso the the DHCP request gets to dnsmasq16:35
slaweqbut this is somehow missed somewhere16:35
*** ociuhandu has joined #openstack-meeting16:35
mlavallecould it be a problem with the flows?16:36
slaweqpossibly yes16:36
slaweqespecially that we are using openvswitch fw driver there16:36
mlavallewe are dropping the offer16:36
mlavalleperhapds, that is16:37
slaweqit can be16:37
slaweqbut basically I think that we should revert this patch to make tempest-slow into better shape16:37
slaweq*to get16:37
mlavalleyes, let's pull the trigger16:37
mlavalleworst case, we revert the revert ;-)16:38
*** e0ne has quit IRC16:38
bcafarelit's been some times since I saw a "revert revert revert revert revert ..." review16:38
slaweqI know that sean is going to release new os-vif version soon and it will have his revert of patches which caused that port was created twice during booting vm16:38
slaweqso maybe this my patch will not be really needed16:39
*** e0ne has joined #openstack-meeting16:39
slaweqbut as a sidenote I want to mention that I think that I saw some similar issues from time to time already16:39
*** mriedem is now known as mriedem_away16:39
slaweqI mean issues that VM didn't have configured IP address and because of that was not reachable16:40
slaweqmaybe it is same issue but just happens less often without this my patch16:40
slaweqplease keep it in mind just :)16:40
mlavalleok, thanks for the clarification16:40
slaweqok, thats all from my side about tempest tests16:41
slaweqanything else You want to add?16:41
mlavallenot me16:42
slaweqok, lets move on then16:42
slaweqnext topic16:42
slaweq#topic fullstack/functional16:42
*** openstack changes topic to "fullstack/functional (Meeting topic: neutron_ci)"16:42
slaweqtoday I found new bug in functional tests \o/16:42
openstackLaunchpad bug 1812872 in neutron "Trunk functional tests can interact with each other " [Medium,Confirmed] - Assigned to Slawek Kaplonski (slaweq)16:42
slaweqfortunately so far we are lucky and it don't hit us in gate16:43
slaweqbut I can reproduce it locally16:43
slaweqand I hit it in my patch where I want to switch functional job to python316:43
slaweqbasically it is race between two trunk related tests16:44
slaweqin bug description there are details16:44
slaweqI have one idea how to "fix" it in ugly but fast way16:44
slaweqwe can add lockutils.synchronized() decorator for those 2 tests and it should works fine IMHO16:44
slaweqwhat do You think about such solution?16:45
*** jamesmcarthur has quit IRC16:45
mlavalleso I assume they share a resource16:45
mlavallethat is why you need the lock16:45
slaweqother than that is probably doing something similar to fullstack tests where we will need to monkey patch some functions16:45
slaweqthey don't share resources in fact16:45
slaweqbut each of them is listing ovsdb monitor events16:46
slaweqand such events aren't distingueshed between tests16:46
mlavallewell, that's the sahred resource, the stream of events16:46
slaweqso one test have got mocked handle_trunk_remove method to not clean trunk bridge16:47
slaweqbut then second test is cleaning this bridge as it gets event from ovsdb monitor16:47
slaweqmlavalle: yes, so in that way they share resources16:47
slaweqso is such lock (with comment) acceptable for You?16:48
slaweqwhat You think about it?16:48
njohnstonare trunk bridges vlan-specific?  Would there be a way to plumb another fake vlan, thus making the trunk bridges in each test different from each other?16:48
mlavalleI don't see ehy not16:48
slaweqnjohnston: trunk bridges are different16:48
slaweqthey are created in setUp() method for each test16:49
slaweqbut one of them can remove trunk bridge for another one because it is triggered by ovsdb event16:49
mlavalleso esentially another alternative is to come up with a way for each test to have iot's own stream events16:49
slaweqthat was discussed some time ago I think in context of fullstack tests16:50
njohnstonso the ovsdb event does not include info on which bridge it is related to?16:50
mlavallewould that be a lot of work?16:50
slaweqand we don't have (then we didn't have at least) easy way to tell - listen only events from this bridge16:50
mlavallewe have a lot of work to do16:51
slaweqbut tbh now, when we switched ovsdb monitors to native implementation thx to ralonsoh work, maybe that would be possible16:51
slaweqI can check it16:51
slaweqif this will be too much to do, I will go for now with lock and TODO note how it might be fixed in fututr16:51
slaweqok for You?16:51
mlavalleif fixing this without locks is too labor intensive, let's go for the locks and leave a todo comment16:51
slaweqmlavalle: ++ :)16:52
slaweq#action slaweq to check if new ovsdb monitor implementation can allow to listen only for events from specific bridge16:52
slaweqok, anything else You want to mention related to functiona/fullstack tests?16:53
mlavallenot me16:53
slaweqok, lets move to next topic then16:53
slaweq#topic Periodic16:53
*** openstack changes topic to "Periodic (Meeting topic: neutron_ci)"16:53
slaweqsince few days at least we have issue with jobs openstack-tox-py27-with-oslo-master and openstack-tox-py35-with-oslo-master16:53
slaweqExample: http://logs.openstack.org/periodic/git.openstack.org/openstack/neutron/master/openstack-tox-py27-with-oslo-master/c787964/testr_results.html.gz16:54
slaweqit's only link to one python27 job16:54
slaweqbut error is exactly the same in python 3516:54
slaweqI didn't report this bug in launchpad yet16:54
slaweqis there any volunteer to report it on launchpad and fix it? :)16:54
njohnstonnever seen one like thta before16:54
*** ttsiouts has quit IRC16:55
njohnstonI'll take a look16:55
*** ttsiouts has joined #openstack-meeting16:55
slaweqnjohnston: it's probably some change in newest oslo_service lib and we need to adjust our code to it16:55
slaweqthx njohnston16:55
njohnstonthat's what I am thinking too16:56
slaweq#action njohnston to take care of periodic UT jobs failures16:56
slaweqThank You :)16:56
slaweqok, and that's all from me for today :)16:56
slaweqdo You want to talk about anythin else quickly?16:56
mlavallenot me16:57
slaweqok, so thanks for attending guys16:57
slaweqhave a great week16:57
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"16:57
openstackMeeting ended Tue Jan 22 16:57:49 2019 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:57
mlavalleThanks, you too16:57
openstackMinutes:        http://eavesdrop.openstack.org/meetings/neutron_ci/2019/neutron_ci.2019-01-22-16.00.html16:57
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/neutron_ci/2019/neutron_ci.2019-01-22-16.00.txt16:57
openstackLog:            http://eavesdrop.openstack.org/meetings/neutron_ci/2019/neutron_ci.2019-01-22-16.00.log.html16:57
*** ttsiouts has quit IRC17:00
*** efried has quit IRC17:00
*** pcaruana has quit IRC17:02
*** macza has joined #openstack-meeting17:02
*** e0ne has quit IRC17:02
*** jamesmcarthur has joined #openstack-meeting17:05
*** macza_ has joined #openstack-meeting17:06
*** macza has quit IRC17:07
*** radeks__ has joined #openstack-meeting17:08
*** mlavalle has quit IRC17:08
*** radeks_ has quit IRC17:10
*** aojea has quit IRC17:15
*** jamesmcarthur has quit IRC17:20
*** a-pugachev has quit IRC17:21
*** igordc has joined #openstack-meeting17:28
*** davidsha has quit IRC17:31
*** radeks__ has quit IRC17:33
*** mattw4 has joined #openstack-meeting17:34
*** igordc has quit IRC17:39
*** radeks has joined #openstack-meeting17:47
*** efried has joined #openstack-meeting17:49
*** erlon_ has quit IRC17:49
*** radeks has quit IRC17:50
*** armax has quit IRC17:53
*** dmacpher has joined #openstack-meeting17:53
*** ekcs has joined #openstack-meeting17:57
*** pcaruana has joined #openstack-meeting18:01
*** jamesmcarthur has joined #openstack-meeting18:02
*** ssbarnea|rover has joined #openstack-meeting18:02
*** jamesmcarthur has quit IRC18:03
*** jamesmcarthur has joined #openstack-meeting18:04
*** ssbarnea|bkp2 has quit IRC18:04
*** priteau has quit IRC18:05
*** ociuhandu has quit IRC18:06
*** bbowen_ is now known as bbowen18:08
*** mmethot has joined #openstack-meeting18:12
*** jamesmcarthur has quit IRC18:14
*** snowgoggles has joined #openstack-meeting18:15
*** jamesmcarthur has joined #openstack-meeting18:18
*** mmethot has quit IRC18:21
*** mmethot has joined #openstack-meeting18:21
*** yamahata has quit IRC18:22
*** iyamahat has quit IRC18:22
*** pcaruana has quit IRC18:23
*** mmethot has quit IRC18:25
*** mmethot has joined #openstack-meeting18:25
*** jamesmcarthur has quit IRC18:28
*** iyamahat has joined #openstack-meeting18:36
*** kopecmartin is now known as kopecmartin|off18:38
*** TxGirlGeek has quit IRC18:48
*** mmethot has quit IRC18:48
*** mmethot has joined #openstack-meeting18:49
*** armstrong has quit IRC18:50
*** armax has joined #openstack-meeting18:51
*** jamesmcarthur has joined #openstack-meeting18:52
*** mmethot has quit IRC18:55
*** yamahata has joined #openstack-meeting18:55
*** mmethot has joined #openstack-meeting18:56
*** baojg has quit IRC18:57
*** baojg has joined #openstack-meeting18:57
*** baojg has quit IRC18:58
*** baojg has joined #openstack-meeting18:58
*** baojg has quit IRC18:58
*** baojg has joined #openstack-meeting18:59
*** jesusaur has quit IRC18:59
*** baojg has quit IRC18:59
*** baojg has joined #openstack-meeting18:59
*** TxGirlGeek has joined #openstack-meeting19:00
*** baojg has quit IRC19:00
*** baojg has joined #openstack-meeting19:00
*** baojg has quit IRC19:01
*** baojg has joined #openstack-meeting19:01
*** baojg has quit IRC19:02
*** ianw_pto is now known as ianw19:02
*** baojg has joined #openstack-meeting19:02
*** baojg has quit IRC19:02
*** baojg has joined #openstack-meeting19:03
*** baojg has quit IRC19:04
*** baojg has joined #openstack-meeting19:04
*** baojg has quit IRC19:05
*** baojg has joined #openstack-meeting19:05
*** mmethot has quit IRC19:05
*** baojg has quit IRC19:06
*** iyamahat has quit IRC19:06
*** baojg has joined #openstack-meeting19:06
*** iyamahat has joined #openstack-meeting19:06
*** baojg has quit IRC19:06
*** ociuhandu has joined #openstack-meeting19:07
*** e0ne has joined #openstack-meeting19:07
*** baojg has joined #openstack-meeting19:07
*** baojg has quit IRC19:08
*** baojg has joined #openstack-meeting19:08
*** jamesmcarthur has quit IRC19:09
*** baojg has quit IRC19:09
*** baojg has joined #openstack-meeting19:09
*** baojg has quit IRC19:09
*** baojg has joined #openstack-meeting19:10
*** baojg has quit IRC19:10
*** baojg has joined #openstack-meeting19:11
*** baojg has quit IRC19:11
*** TxGirlGeek has quit IRC19:11
*** baojg has joined #openstack-meeting19:11
*** baojg has quit IRC19:12
*** baojg has joined #openstack-meeting19:13
*** baojg has quit IRC19:13
*** baojg has joined #openstack-meeting19:14
*** TxGirlGeek has joined #openstack-meeting19:14
*** baojg has quit IRC19:14
*** baojg has joined #openstack-meeting19:15
*** baojg has quit IRC19:15
*** baojg has joined #openstack-meeting19:15
*** baojg has quit IRC19:16
*** baojg has joined #openstack-meeting19:16
*** baojg has quit IRC19:17
*** baojg has joined #openstack-meeting19:17
*** baojg has quit IRC19:17
*** jamesmcarthur has joined #openstack-meeting19:18
*** baojg has joined #openstack-meeting19:19
*** baojg has quit IRC19:20
*** jamesmcarthur has quit IRC19:21
*** jamesmcarthur has joined #openstack-meeting19:24
*** jesusaur has joined #openstack-meeting19:25
*** takamatsu has quit IRC19:27
*** tssurya has quit IRC19:36
*** e0ne has quit IRC19:42
*** iyamahat_ has joined #openstack-meeting19:43
*** TxGirlGeek has quit IRC19:46
*** iyamahat has quit IRC19:46
*** iyamahat__ has joined #openstack-meeting19:48
*** pcaruana has joined #openstack-meeting19:50
*** iyamahat_ has quit IRC19:51
*** whoami-rajat has quit IRC19:52
*** jamesmcarthur has quit IRC19:53
*** ralonsoh has quit IRC19:53
*** diablo_rojo has joined #openstack-meeting19:56
*** TxGirlGeek has joined #openstack-meeting19:58
*** jamesmcarthur has joined #openstack-meeting19:58
*** jamesmcarthur has quit IRC20:02
*** diablo_rojo has quit IRC20:05
*** pcaruana has quit IRC20:11
*** mriedem_away is now known as mriedem20:16
*** mjturek has quit IRC20:24
*** pcaruana has joined #openstack-meeting20:24
*** jamesmcarthur has joined #openstack-meeting20:36
*** martial__ has joined #openstack-meeting20:39
*** martial__ is now known as martial20:39
*** dims has quit IRC20:40
*** dims has joined #openstack-meeting20:42
*** oneswig has joined #openstack-meeting20:50
*** dims has quit IRC20:50
*** janders has joined #openstack-meeting20:50
*** isq has joined #openstack-meeting20:52
*** dims has joined #openstack-meeting20:52
martialoneswig: I see blair in the unregistered channel ... hopefully fix it soon20:58
oneswigok thanks martial20:59
oneswig#startmeeting scientific-sig21:00
openstackMeeting started Tue Jan 22 21:00:27 2019 UTC and is due to finish in 60 minutes.  The chair is oneswig. Information about MeetBot at http://wiki.debian.org/MeetBot.21:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.21:00
*** openstack changes topic to " (Meeting topic: scientific-sig)"21:00
openstackThe meeting name has been set to 'scientific_sig'21:00
oneswigLet us get the show on the road!21:00
oneswig#link agenda for today https://wiki.openstack.org/wiki/Scientific_SIG#IRC_Meeting_January_22nd_201921:00
oneswig#chair martial21:01
openstackCurrent chairs: martial oneswig21:01
*** trandles has joined #openstack-meeting21:02
oneswigWhile we are gathering, I'll do the reminder - CFP ends tomorrow, midnight pacific time!21:02
oneswig#link CFP link for Open Infra Denver https://www.openstack.org/summit/denver-2019/call-for-presentations/21:03
oneswigjanders: you ready?21:03
jandersI am21:03
oneswigExcellent, lets start with that.21:04
oneswig#topic Long-tail latency of SR-IOV Infiniband21:04
*** openstack changes topic to "Long-tail latency of SR-IOV Infiniband (Meeting topic: scientific-sig)"21:04
oneswigSo what's been going on?21:04
*** b1air has joined #openstack-meeting21:04
jandersHere are some slides I prepared - these cover  IB microbenchmarks across bare-metal and SRIOV/IB21:04
*** artom has quit IRC21:04
oneswigExcellent, thanks21:04
b1airo/ hello! (slight issue with NickServ here...)21:05
jandersAs per the Slide 2 we were aiming to get a better picture of low-level causes of the performance disparity we were seeing (or mostly hearing about)21:05
oneswigHi b1air, you made it21:05
oneswig#chair b1air21:05
openstackCurrent chairs: b1air martial oneswig21:05
jandersWe were also thinking this will help decide what's best running on bare-metal and what can run happily in a (HPC)VM21:05
martialbad nick serve21:05
jandershey Blair!21:06
martialwelcome Blair :)21:06
jandersSlide 3 has the details of the lab setup. We tried to stay as generic as possible so the numbers aren't super-optimised, however we've done all the ususal reasonable things to do for benchmarking (performance mode, CPU passthru)21:07
jandersSimilar "generic" approach was applied to the microbenchmarks - we ran with the default parameters which are captured in Slide 421:07
*** b1airo has joined #openstack-meeting21:07
jandersOn that - the results once again prove what many of us here have seen - virtualised RDMA (or IB in this case) can match the bandwidth of bare-metal RDMA21:08
martialI asked Rion to join as well (since it is his scripts for Kubespray)21:08
jandersWe're at around 99% efficiency bandwidth wise21:09
jandersWith latency, however things things are different21:09
oneswigThis is just excellent. :-)21:09
jandersVirtualisation seems to add a fair bit there, about .4 us off from bare-metal21:09
janders(or ~24%)21:10
b1airook, i've managed to log in twice it seems :-)21:10
martialb1airo: cool :)21:10
jandersthe more the merrier :)21:10
oneswigDoes your VM have NUMA passthrough?21:10
*** b1air has quit IRC21:10
oneswig#chair b1airo21:11
openstackCurrent chairs: b1air b1airo martial oneswig21:11
jandersI haven't explicitly tweaked NUMA, however the CPUs are in passthru mode21:11
b1airothanks oneswig21:11
jandersI'm happy to check later in the meeting and report back - just run "numactl --show"?21:11
b1airoi note the hardware is relatively old now, wonder how that impacts these results as compared to say Skylake with CX-421:12
*** jamesmcarthur has quit IRC21:12
oneswigThat ought to do it, or if you have multiple sockets in lscpu, I think that also means it's on21:12
jandersI was considering applying NFV style tuning, but thought this would make it less generic21:12
oneswigjanders: what kind of things?21:12
jandersgood point on hugepages though - just enabling that shouldn't have any negative side effects21:12
jandersNFV style approach would involve CPU pinning21:13
jandershowever my worry there is we'll likely improve IB microbenchmark numbers at the expense on Linpack21:13
b1airothis could be a nice start to some work to come up with a standard set of microbenchmarks we could use...21:13
*** deardooley has joined #openstack-meeting21:13
martialoh here is Rion21:13
oneswigHello Rion, welcome21:14
jandersI hope to get some newer kit soon and I'm intending to retest on Skylake/CX5 and maybe newer21:14
b1airosurely if you were going to be running MPI code you'd want pinning though janders ?21:14
deardooleyHi all21:14
* b1airo waves to Rion21:14
oneswigYour page 6, plotting sigmas against one another.  Are you able to plot histograms instead?  I'd be interested to see the two bell curves superimposed.21:15
b1airoit's an interesting consideration though - how much tuning is reasonable and what are the implications...21:16
jandersoneswig: thank you for the suggestion - I will include that when I work on this again soon (with more hardware)21:17
oneswigOne potential microbenchmark for capturing the effect of long-tail jitter on a parallel job would be to time multi-node barriers21:17
oneswigI'm not sure there's an ib_* benchmark for that but there might be something in IMB.21:17
jandersThe second interesting observation is the impact of virtualisation on standard deviation in results21:17
jandersinterestingly this is seen across the board. In these slides I'm mostly focusing on ib_write_* but I put Linpack there for reference too21:18
janderswhether it's Linpack, bandwidth or latency, baremetals are heaps more consistent21:18
oneswigIs that a single-node Linpack configuration?21:18
jandersin absolute numbers the fluctuation isn't huge, but in relative numbers it's an order of magnitude21:19
jandersyes, it's single node21:19
*** pcaruana has quit IRC21:19
jandersI considered multinode but was getting mixed messages about the potential impact of interconnect virtualisation on the results from different people, so thought I better keep things simple21:20
jandersthis way we have the overheads measured separately21:20
oneswigI'd love to know more on the root causes (wouldn't we all)21:20
jandersI think the variability could likely be addressed with NFV style tyning21:21
jandersat least to some degree21:21
*** baojg has joined #openstack-meeting21:21
janderswith the latency impact, I think the core of it might have to do with the way IB virtualisation is done, however I've never heard the official explanation21:21
jandersI feel it likely gets better for larger message sizes21:22
b1airoseems to me that once you start doing cpu and/or numa pinning and/or static hugepage backing, then you really need to commit to isolating specific hardware for that job and you create just a few instance-types that fit together on your specific hardware nicely. then you probably also have some other nodes for general purpose. so perhaps there are at least three interesting scenarios to benchmark: 1) VM tuned for21:22
b1airogeneral purpose hosts; 2) VM tuned for high-performance dedicated hosts; 3) bare-metal21:22
*** janders_ has joined #openstack-meeting21:23
janders_sorry got kicked out :(21:23
janders_1 2 3 testing21:23
oneswigDid you catch b1airo's message with 3 cases?21:23
janders_unfortunatly not, I lost everything past "08:22] <janders> given the bandwidth numbers are quite good"21:24
*** bbowen has quit IRC21:24
janders_b1airo: can you re-post please?21:24
janders_sorry about that21:24
martialhelping b1airo 16:23:00 <b1airo> seems to me that once you start doing cpu and/or numa pinning and/or static hugepage backing, then you really need to commit to isolating specific hardware for that job and you create just a few instance-types that fit together on your specific hardware nicely. then you probably also have some other nodes for general purpose. so perhaps there are at least three interesting scenarios21:25
martialto benchmark: 1) VM tuned for21:25
martial16:23:00 <b1airo> general purpose hosts; 2) VM tuned for high-performance dedicated hosts; 3) bare-metal21:25
janders_have you guys had a chance to benchmark CPU pinned configurations in similar ways?21:26
oneswigThis test might be interesting, across multiple nodes: IMB-MPI1 Barrier21:26
janders_I wonder if pinning helps with consistency21:26
*** janders has quit IRC21:26
janders_and what would be the impact of the pinned configuration on local Linpack?21:26
*** baojg has quit IRC21:27
oneswigjanders_: yes, I did some stuff using OpenFOAM (with paravirtualised networking)21:27
b1airothanks martial - i was in the bathroom briefly21:27
janders_(I suppose we would  likely lose some cores for the host OS - and I wonder to what degree the performance improvement on the pinned cores would compensate that)21:27
oneswigYou can see the impact of pinning in simple test cases like iperf - I found it didn't increase TCP performance much but it certainly did help with jitter21:28
b1airoyes, i believe pinning does help with consistency21:28
janders_I think the config proposed by Blair is a good way forward - my worry is if they users will know if they need max_cores configuration (20 core VM on a 20 core node) or the NFV configuration21:28
b1airoi think the question of reserved host cores is another interesting one for exploration...21:29
janders_I tried Linpack in 18 core and 20 core VMs in the past and 20 core was still faster21:29
*** efried has quit IRC21:30
janders_despite the potential of scheduling issues between the host and the guest21:30
b1airoi would contend that if most of your network traffic is happening via SR-IOV interface then reserving host cores is unnecessary21:30
*** efried has joined #openstack-meeting21:30
oneswigb1airo: makes sense unless they are working for the libvirt storage too21:30
oneswigOK, we should move on.  Any more questions for janders?21:31
oneswigjanders_: one final thought - did you disable hyperthreading?21:31
b1airoah yes, good point oneswig - i guess i was thinking of storage via SR-IOV too, i.e., parallel filesystem21:31
janders_yes I did21:32
janders_no HT21:32
oneswigWe found that makes a significant difference.21:32
janders_I typically work with node-local SSD/NVMe for scratch21:32
b1airoto Linpack oneswig ?21:33
janders_and a parallel fs indeed mounted via SRIOV interface21:33
oneswigb1airo: haven't tried that.  Other benchmarks.21:33
janders_on the local scratch it would be interesting to look at the impact of qcow2 vs lvm21:33
janders_lvm helps with IOPS a lot, but in a scenario like the one we're discussing where there's little CPU for host OS, this might be even more useful21:34
janders_so - b1airo - do you think in the configuration you proposed, would we need two "performance VM" flavors?21:34
janders_max_cpu and low_latency (pinned)?21:35
b1airojanders_: we could talk more on this offline perhaps, i'd be keen to try and get some comparable benchmarks together as well21:35
*** eharney has quit IRC21:35
oneswigjanders_: you need Ironic deploy templates... let's revisit that21:36
janders_OK! being mindful of time, let's move on to the next topic. Thank you for your attention and great insights!21:36
oneswigOK time is pressing21:36
*** wwriverrat has joined #openstack-meeting21:36
oneswig#topic Terraform and Kubespray21:36
*** openstack changes topic to "Terraform and Kubespray (Meeting topic: scientific-sig)"21:36
oneswigdeardooley: martial: take it away!21:36
martialso I invited Rion to this convesrsation21:36
martialbut the idea is simple, I needed to deploy a small K8s cluster for testing on top of OpenStack21:37
martialinternally we have used Kubespray to do so21:37
martialto deploy a Kubernetes (one master and two minion nodes) in an pre-configured OpenStack project titled nist-ace, using Kubespray and Terraform.21:37
martialthe default Kubespray install requires the creation of pre-configured VMs21:38
martial#link https://github.com/kubernetes-sigs/kubespray/blob/master/docs/openstack.md21:38
martialTerraform has the advantage to pre-configures the OpenStack project for running the ansible playbook given information about networking, users, and the OpenStack project itself. Then Terraform handles the VM configurations and creations.21:38
oneswigGiven Kubespray is Ansible, why the need for preconfiguration?21:38
martial#link https://github.com/kubernetes-sigs/kubespray/blob/master/contrib/terraform/openstack/README.md21:39
martialthat was also my question, but the ansible script did not create the OpenStack instances21:39
martialterraform will21:39
*** dklyle has quit IRC21:39
deardooley@oneswig it's a pretty common pattern. terraform is much faster at pure provisioning that ansible, but config management is not it's strong suite. ansible is a good complement once the infrastructure is in place.21:40
martialyou obviously need the OpenStack project's RC file21:40
martialonce you have this sourced you are able to create the terraform configuration file to include master/minion number and names21:41
martialimages, flavors, IP pools21:41
*** priteau has joined #openstack-meeting21:41
*** mmethot has joined #openstack-meeting21:42
oneswigHow do you find Terraform compares to expressing the equivalent in Heat?21:42
*** trandles has quit IRC21:42
*** mmethot has quit IRC21:43
martialgiven that kubespray has its own ways of spawning on top of OpenStack, I did not try heat for this purpose21:43
janders_in the context of a Private Cloud, would it make sense to disable port-security so that we don't need to worry about address pairs?21:44
janders_or do you see value in having this extra layer of protection?21:44
*** mmethot has joined #openstack-meeting21:44
martialsupposedely terraforms create the private network needed for indeed k8s communication21:44
martialon top of the OpenStack project21:44
martial's own network21:44
martialonce the configuration is done, calling terraform init21:45
martialfollowed by terraform apply21:45
*** mmethot has quit IRC21:45
oneswigWhat do you configure for Kubernetes networking?  Does it use Weave, Calico, ?21:46
deardooleywithin the context of kubespray, you the terraform provisioner will handle all security group creation and managmeent for you as part of the process. You will need to implement any further security at the edge of your k8s apiservers on your own.21:46
martial(with a little extra obviously) creates the openstack21:47
martialI was checking in my configuration file and I do not see the k8s networking set21:48
deardooleyit's pluggable. defaults to calico. there are some tweaks you need to make in your specific inventory depending on the existence of different openstack components in your particular cloud.21:48
oneswigSeems like a lot of interesting things are happening around blending the layers of abstraction, and interfacing with OpenStack to provide resources (storage, networking, etc) for Kubernetes - eg https://github.com/kubernetes/cloud-provider-openstack - does Kubespray support enabling that kind of stuff?21:48
martialthe default seems to be Callico21:48
oneswig... would be cool if it did21:49
deardooleyfor example, to plug into external loadbalancers, dynamically configure upstream dns, use cinder as persistent volume provisioner, etc.21:49
oneswigdeardooley: that kind of stuff21:49
deardooleyyeah. it's all pluggable with the usual caveates.21:50
martial(I am kind of tempting Rion here to think about a presentation at the summit on the topic of Kubespray deployment on top of OpenStack)21:51
oneswigHave you been using those cloud-provider integrations and is it working?21:51
martialI have not21:53
deardooleyI use them on a couple different openstack installs.21:53
deardooleythey work, but there are ways to get the job done, and there are ways to get the job done and keep all your hair and staff in tact21:54
oneswigdeardooley: sounds familiar :-)21:54
deardooleyit's likely anyone on this channel could pull it off in a day or two by pinging this list, but once you do, you'll appreciate the "Hey Google, create a 5 node Kubernes cluster" demo in a whole new way.21:55
*** rbudden has joined #openstack-meeting21:55
deardooleythat being said, once you get your config down, it really is push button to spin up another dozen in the same environment.21:56
oneswigdeardooley: in your experience, what does Kubespray do wrong / badly?  Does it have shortcomings?21:56
martialafter Terraform pre-configures everything is it simply the steps or running the ansible playbook21:56
*** jamesmcarthur has joined #openstack-meeting21:56
b1airocertainly sounds like it could be an interesting presentation topic21:57
deardooleyit can build a production scale cluster for you. it can't do much to help you manage it.21:57
b1airoi think i need a diagram of the openstack - kubespray - terraform interactions21:57
martialsee Rion "it could be an interesting presentation topic" :)21:57
deardooleyas long as you treat nodes idempotently and get sufficient master quorem defined up front, it's not a huge issue. when something goes weird, just delete it, remove the node, and rerujn the cluster.yml playbook with a new vm.21:58
oneswigWe are nearly at time - final thoughts?21:58
b1airoone other motivation type question? why not Magnum?21:58
martialin my particular case, the person I was working with wanted K8s to test containers ... truth is given how many containers they want, docker swarm might be enough21:59
deardooleyflexibility, portability across cloud providers, secruity across multiple layers of the infrastructure and application stack, logging, monitoring, etc...21:59
oneswigOK, we are at time22:00
martialand I will second Rion's comment of "remove" "rerun", that was very useful for testing things22:00
oneswigThanks deardooley martial - interesting to hear about your work.22:00
*** jamesmcarthur has quit IRC22:00
janders_great work, thanks guys!22:00
oneswigUntil next time22:00
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"22:00
openstackMeeting ended Tue Jan 22 22:00:50 2019 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)22:00
martialhappy to share :)22:00
b1airothanks guys!22:00
openstackMinutes:        http://eavesdrop.openstack.org/meetings/scientific_sig/2019/scientific_sig.2019-01-22-21.00.html22:00
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/scientific_sig/2019/scientific_sig.2019-01-22-21.00.txt22:00
openstackLog:            http://eavesdrop.openstack.org/meetings/scientific_sig/2019/scientific_sig.2019-01-22-21.00.log.html22:00
*** janders_ has quit IRC22:01
*** wwriverrat has quit IRC22:01
*** oneswig has quit IRC22:01
*** priteau has quit IRC22:08
*** rcernin has joined #openstack-meeting22:10
*** slaweq has quit IRC22:12
*** jamesmcarthur has joined #openstack-meeting22:14
*** armax has quit IRC22:17
*** rcernin has quit IRC22:17
*** rcernin has joined #openstack-meeting22:19
*** jamesmcarthur has quit IRC22:20
*** TxGirlGeek has quit IRC22:24
*** armax has joined #openstack-meeting22:24
*** TxGirlGe_ has joined #openstack-meeting22:25
*** jamesmcarthur has joined #openstack-meeting22:28
*** slaweq has joined #openstack-meeting22:29
*** TxGirlGe_ has quit IRC22:31
*** devananda has joined #openstack-meeting22:31
*** bbowen has joined #openstack-meeting22:33
*** slaweq has quit IRC22:33
*** lifeless_ is now known as lifeless22:34
*** TxGirlGeek has joined #openstack-meeting22:40
*** imacdonn_ has quit IRC22:48
*** imacdonn_ has joined #openstack-meeting22:49
*** baojg has joined #openstack-meeting22:52
*** efried has quit IRC22:53
*** TxGirlGeek has quit IRC22:59
*** slaweq has joined #openstack-meeting23:00
*** TxGirlGeek has joined #openstack-meeting23:04
*** slaweq has quit IRC23:05
*** mattw4 has quit IRC23:07
*** a-pugachev has joined #openstack-meeting23:08
*** armax has quit IRC23:31
*** ociuhandu_ has joined #openstack-meeting23:32
*** ociuhandu has quit IRC23:33
*** TxGirlGeek has quit IRC23:35
*** rbudden has quit IRC23:37
*** _alastor_ has quit IRC23:53
*** mattw4 has joined #openstack-meeting23:56

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!