Tuesday, 2018-01-09

*** claudiub has quit IRC00:00
*** david-lyle has quit IRC00:07
*** dklyle has joined #openstack-meeting00:07
*** dklyle has quit IRC00:07
*** wanghao has quit IRC00:15
*** wanghao has joined #openstack-meeting00:17
*** wanghao has quit IRC00:18
*** wanghao has joined #openstack-meeting00:23
*** wanghao has quit IRC00:28
*** sdague has quit IRC00:29
*** edmondsw has joined #openstack-meeting00:36
*** wanghao has joined #openstack-meeting00:36
*** esberglu has quit IRC00:37
*** chyka_ has quit IRC00:38
*** felipemonteiro_ has joined #openstack-meeting00:44
*** emagana has joined #openstack-meeting00:48
*** tovin07_ has joined #openstack-meeting00:52
*** emagana has quit IRC00:52
*** TxGirlGeek has quit IRC01:01
*** emagana has joined #openstack-meeting01:03
*** emagana has quit IRC01:07
*** emagana has joined #openstack-meeting01:08
*** TxGirlGeek has joined #openstack-meeting01:11
*** felipemonteiro_ has quit IRC01:12
*** TxGirlGeek has quit IRC01:12
*** emagana has quit IRC01:13
*** markvoelker has joined #openstack-meeting01:17
*** kiennt26 has joined #openstack-meeting01:18
*** zhurong has joined #openstack-meeting01:21
*** armax has joined #openstack-meeting01:25
*** frank_wang has joined #openstack-meeting01:27
*** amodi has joined #openstack-meeting01:31
*** Dinesh_Bhor has joined #openstack-meeting01:34
*** aeng has quit IRC01:34
*** mriedem1 has joined #openstack-meeting01:40
*** mriedem has quit IRC01:42
*** Tom-Tom has quit IRC01:43
*** Tom-Tom has joined #openstack-meeting01:43
*** gongysh has joined #openstack-meeting01:44
*** Dinesh_Bhor has quit IRC01:45
*** aeng has joined #openstack-meeting01:47
*** markstur_ has joined #openstack-meeting01:51
*** zhongjun has joined #openstack-meeting01:52
*** Tom-Tom has quit IRC01:53
*** markstur has quit IRC01:54
*** mriedem has joined #openstack-meeting01:56
*** mriedem1 has quit IRC01:58
*** felipemonteiro_ has joined #openstack-meeting02:02
*** Dinesh_Bhor has joined #openstack-meeting02:05
*** Dinesh_Bhor has quit IRC02:07
*** jamesmcarthur has joined #openstack-meeting02:07
*** Tom-Tom has joined #openstack-meeting02:09
*** Tom-Tom has quit IRC02:09
*** Tom-Tom has joined #openstack-meeting02:09
*** haint has joined #openstack-meeting02:20
*** hazmat has quit IRC02:22
*** hazmat has joined #openstack-meeting02:24
*** nguyentrihai has quit IRC02:24
*** Dinesh_Bhor has joined #openstack-meeting02:26
*** ekcs has joined #openstack-meeting02:29
*** edmondsw has quit IRC02:30
*** Apoorva has quit IRC02:32
*** harlowja has quit IRC02:32
*** Dinesh_Bhor has quit IRC02:33
*** simonmcc has quit IRC02:36
*** simonmcc has joined #openstack-meeting02:37
*** julim has joined #openstack-meeting02:46
*** julim has quit IRC02:47
*** jamesmcarthur has quit IRC02:50
*** jamesmcarthur has joined #openstack-meeting02:51
*** hongbin has joined #openstack-meeting02:57
hongbin#startmeeting zun03:00
openstackMeeting started Tue Jan  9 03:00:04 2018 UTC and is due to finish in 60 minutes.  The chair is hongbin. Information about MeetBot at http://wiki.debian.org/MeetBot.03:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.03:00
*** openstack changes topic to " (Meeting topic: zun)"03:00
openstackThe meeting name has been set to 'zun'03:00
hongbin#link https://wiki.openstack.org/wiki/Zun#Agenda_for_2017-01-09_0300_UTC Today's agenda03:00
hongbin#topic Roll Call03:00
*** openstack changes topic to "Roll Call (Meeting topic: zun)"03:00
*** julim has joined #openstack-meeting03:02
hongbinhi mkrai03:03
hongbinit looks just you and me today03:03
mkraihongbin (IRC): hi03:03
hongbinmkrai: btw, thanks for serving the ptl role while i was away03:03
*** bkopilov has quit IRC03:04
mkraihongbin (IRC): my pleasure03:04
kiennt26hi all o/03:04
hongbinkiennt26: hi kien03:04
hongbinmkrai: kiennt26 there are three of us03:04
hongbini guess each of us can give a short update, then end the meeting03:04
mkraikiennt26 (IRC): hi03:05
hongbinmkrai: anything to update from your side?03:05
mkraiOk sounds good03:05
mkraihongbin (IRC): I didn’t do much last week03:05
hongbinmkrai: ack, that is fine03:05
mkraiI will update the Clear container devstack patch today03:05
mkraiAnd then it should be fine to merge03:06
*** markvoelker has quit IRC03:06
mkraiAfter that we can mark it as complete for now03:06
mkraiAnd later work on Kata containers03:06
mkraiThat’s all 😊03:06
hongbinyes, i am waiting for kata containers to be ready03:06
hongbinthanks mkrai03:06
hongbinkiennt26: anything from your side?03:07
*** adreznec has quit IRC03:07
kiennt26From my side, this week i'm working on Zuulv3 native jobs03:07
kiennt26but in zuulv3 native, multinode jobs stills failed. Seems that Devstack-multinode job doesn't ready yet03:08
hongbini see03:08
*** adreznec has joined #openstack-meeting03:08
*** anilvenkata has quit IRC03:08
kiennt262nd, I'm investigating with miao in the bellow patch.03:09
kiennt26#link https://review.openstack.org/#/c/531646/03:09
*** Tom-Tom has quit IRC03:09
*** Tom-Tom has joined #openstack-meeting03:09
kiennt26All tempest jobs were failed.03:09
hongbinhe submitted another patch for tempest-plugin03:09
hongbinhe managed to pass all jobs on that patch03:09
hongbin#link https://review.openstack.org/#/c/531655/03:10
kiennt26hongbin: yeah, zun-tempest-plugin patch depends on zun side patch. It was passes but Zun side was failed.03:10
hongbinif these two patches can merge together, i think everything would be fine03:11
hongbinhowever, that is impossible03:11
hongbini think he needs to do it this way:03:11
hongbin1. add 'project' in server03:11
hongbin2. convert 'tenant' to 'project' in tempest plugin03:12
hongbin3. remove 'tenant' in server03:12
hongbinthen, it should manage to pass the gate03:12
hongbini will comment on his patch about that03:12
hongbinkiennt26: anything else from your side?03:13
kiennt26hongbin: thank you. I was about to do it.03:13
*** iyamahat has quit IRC03:13
kiennt26hongbin: That's all from me this week03:13
hongbinkiennt26: ack, thanks kiennt2603:13
hongbinfrom my side, i was working with enterprice working group to define a reference architecture for zun03:14
hongbin#link https://review.openstack.org/#/c/531613/03:14
hongbini will continue to work on that in this week03:14
hongbinin addition, i am working on a few heat patches to make heat to pick up the latest version of zun03:15
mkraihongbin (IRC): that’s great. I would like to look into it03:15
hongbinmkrai: :)03:15
hongbinthat is all from my side03:15
*** yamahata has quit IRC03:15
hongbinok, any question ?03:15
hongbinif no, i will end hte meeting soon03:15
mkraiNone from me03:16
kiennt26hongbin: nothing from me.03:16
hongbinok, all, thanks for joining hte meting, see you next week03:16
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"03:16
*** adreznec has quit IRC03:16
openstackMeeting ended Tue Jan  9 03:16:33 2018 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)03:16
openstackMinutes:        http://eavesdrop.openstack.org/meetings/zun/2018/zun.2018-01-09-03.00.html03:16
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/zun/2018/zun.2018-01-09-03.00.txt03:16
openstackLog:            http://eavesdrop.openstack.org/meetings/zun/2018/zun.2018-01-09-03.00.log.html03:16
*** adreznec has joined #openstack-meeting03:21
*** ijw has quit IRC03:21
*** hongbin has quit IRC03:24
*** hongbin has joined #openstack-meeting03:25
*** amodi has quit IRC03:31
*** ricolin has joined #openstack-meeting03:37
*** jamesmcarthur has quit IRC03:40
*** yamamoto has joined #openstack-meeting03:40
*** pengdake has joined #openstack-meeting03:41
*** tw0x88 has joined #openstack-meeting03:41
*** Tom-Tom has quit IRC03:42
*** Tom-Tom has joined #openstack-meeting03:42
*** Tom-Tom has quit IRC03:47
*** gyee has quit IRC03:48
*** abhishekk has joined #openstack-meeting03:55
*** zhurong has quit IRC03:55
*** tw0x88 has left #openstack-meeting03:56
*** frank_wang has quit IRC03:56
*** frank_wang has joined #openstack-meeting03:56
*** shaohe_feng has quit IRC03:58
*** shaohe_feng has joined #openstack-meeting03:58
*** sridharg has joined #openstack-meeting03:58
*** sagara has joined #openstack-meeting03:59
*** rkmrHonjo has joined #openstack-meeting03:59
*** Dinesh_Bhor has joined #openstack-meeting04:01
rkmrHonjosagara: hi04:01
rkmrHonjoDinesh_Bhor: hi04:01
Dinesh_BhorrkmrHonjo, sagara: Hi04:01
rkmrHonjoSampath can't attend this meeting at first. So I conduct this meeting.04:01
sagaraI heard today Tushar has still vacation04:02
rkmrHonjosagara: ok04:02
openstackrkmrHonjo: Error: A meeting name is required, e.g., '#startmeeting Marketing Committee'04:02
rkmrHonjo#startmeeting Masakari04:02
openstackMeeting started Tue Jan  9 04:02:43 2018 UTC and is due to finish in 60 minutes.  The chair is rkmrHonjo. Information about MeetBot at http://wiki.debian.org/MeetBot.04:02
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.04:02
*** openstack changes topic to " (Meeting topic: Masakari)"04:02
openstackThe meeting name has been set to 'masakari'04:02
rkmrHonjo#topic Bugs04:03
*** openstack changes topic to "Bugs (Meeting topic: Masakari)"04:03
rkmrHonjoDo you have any updates for bugs?04:03
rkmrHonjo#link https://bugs.launchpad.net/masakari/+bug/174011104:03
openstackLaunchpad bug 1740111 in oslo.log "masakari-api failed to launch due to setting of watch_log_file and log_file" [Undecided,New]04:03
rkmrHonjoThis bug was caused by oslo.log. So I sent a mail to devML about this issue. As a result, Doug Hellmann replied to me.04:04
rkmrHonjo#link http://lists.openstack.org/pipermail/openstack-dev/2018-January/125966.html04:04
rkmrHonjoIn short, pyinotify is unmaintained. We should use other libraries or deprecate watch_log_file option.04:04
rkmrHonjoI write above information on launchpad after that.04:05
Dinesh_BhorrkmrHonjo: nice. I think we should wait for communities decision and follow the same04:06
rkmrHonjoDinesh_Bhor: Yeah.04:06
rkmrHonjo#link https://bugs.launchpad.net/masakari/+bug/173938304:07
openstackLaunchpad bug 1739383 in masakari "'409 Conflict' occurred when adding reserved_host to aggregate" [Undecided,In progress] - Assigned to takahara.kengo (takahara.kengo)04:07
rkmrHonjoTakahara pushed a patch for this issue. Please review it if you can.04:08
rkmrHonjo#link https://review.openstack.org/#/c/531310/04:08
Dinesh_BhorrkmrHonjo: Thank you for the patch. I will review it.04:09
rkmrHonjoDinesh_Bhor: thanks a lot!04:09
rkmrHonjoOf course, I'll review it, too.04:09
rkmrHonjoAny other updates?04:10
Dinesh_BhorI think its time to make py35 job voting04:10
rkmrHonjoDinesh_Bhor: +1. But we should resolve py35 ut issue before that.04:12
Dinesh_BhorrkmrHonjo:  yes04:13
rkmrHonjoSorry, my patch(https://review.openstack.org/#/c/486576/) is late. I push a new patch next week.04:13
rkmrHonjoI think that I should remove a message attribute from masakari.exception.04:13
Dinesh_BhorOkay, np04:13
rkmrHonjoDinesh_Bhor: sorry.04:14
*** Apoorva has joined #openstack-meeting04:15
*** rbudden has quit IRC04:15
*** pengdake has quit IRC04:15
rkmrHonjoIf there is not other updates, let's go to next topic.04:15
Dinesh_BhorrkmrHonjo:  yes please04:15
rkmrHonjo#topic Discussion points04:16
*** openstack changes topic to "Discussion points (Meeting topic: Masakari)"04:16
*** esberglu has joined #openstack-meeting04:16
rkmrHonjo<Horizon Plugin>04:16
*** esberglu has quit IRC04:17
Dinesh_BhorThis needs one more reviewer: https://review.openstack.org/#/c/528647/04:17
*** gongysh has quit IRC04:17
rkmrHonjoDinesh_Bhor: ok, I'll reivew it.04:17
Dinesh_BhorNiraj will soon submit a patch for adding segments panel.04:18
rkmrHonjoDinesh_Bhor: Is it other patch? or new patch set for the above patch?04:18
Dinesh_BhorNo, The Cookiecutter patch is different04:19
rkmrHonjoDinesh_Bhor: OK, I got it.04:19
rkmrHonjo#action rkmrHonjo reviews https://review.openstack.org/#/c/528647/04:20
rkmrHonjoNext, <Install guide document>04:20
rkmrHonjoI heard that tpatil was checking about how to publish documents.04:21
rkmrHonjoDinesh_Bhor, sagara: Do you know about the status of tpatil's researching?04:22
Dinesh_BhorrkmrHonjo: I think its in progress. I will take update from him04:23
*** bkopilov has joined #openstack-meeting04:23
rkmrHonjoDinesh_Bhor: thank you for sharing status.04:23
*** jamesmcarthur has joined #openstack-meeting04:23
*** jamesmcarthur has quit IRC04:23
*** jamesmcarthur has joined #openstack-meeting04:23
rkmrHonjoNext, <recovery method customization>04:24
rkmrHonjosampath's review is required for this, isn't it?04:25
*** zhhuabj has quit IRC04:26
*** hongbin has quit IRC04:26
rkmrHonjook, let's go to next item because he isn't here today.04:26
rkmrHonjo<Ansible support for Masakari>04:27
Dinesh_BhorI am working on this04:27
rkmrHonjoDinesh_Bhor: great. Do you want to talk about it today?04:27
*** beekhof_mb has joined #openstack-meeting04:28
*** epico has joined #openstack-meeting04:28
Dinesh_BhorSo far I don't have much update. If I find any difficulties I will bring them in meeting.04:28
rkmrHonjoDinesh_Bhor: I got it. Let's go to next item.04:29
rkmrHonjo<Call Force down API when host-failure will be notified>04:29
rkmrHonjoI talked with tpatil about this issue by offline.04:29
rkmrHonjoAnd, I expect that he cancel his -1 on gerrit.04:30
rkmrHonjoI wait the end of his vacation.04:30
*** yangyapeng has quit IRC04:31
*** yangyapeng has joined #openstack-meeting04:31
rkmrHonjoDinesh_Bhor: thanks. If he has other concerns, we discuss about it.04:31
rkmrHonjook, let's go to next item.04:31
rkmrHonjooh, sorry, this is the last item.04:32
*** liuyulong has joined #openstack-meeting04:32
rkmrHonjoNext topic is BM-HA, but I think that we can't talk about it without sampath.04:32
rkmrHonjoDo you want to talk about it now?04:33
rkmrHonjo#topic AOB04:33
*** openstack changes topic to "AOB (Meeting topic: Masakari)"04:33
*** felipemonteiro_ has quit IRC04:34
rkmrHonjoDo you have any other topics?04:34
Dinesh_BhorLooks like Gregs is facing come issues with masakari-monitors and masakari integration04:35
Dinesh_BhorI am helping over the mail04:35
rkmrHonjoDinesh_Bhor: Thanks. I'd like to help him, too.04:36
Dinesh_BhorOkay, Thats it from my side04:37
*** zhhuabj has joined #openstack-meeting04:38
rkmrHonjoDinesh_Bhor: sorry...04:38
*** frank_wang has quit IRC04:38
Dinesh_BhorrkmrHonjo:  Np,04:38
rkmrHonjoBy the way, do you know about the preparing status of masakari's IRC channel?04:39
Dinesh_BhorrkmrHonjo: No,04:39
Dinesh_BhorI will figure it out04:40
rkmrHonjoDinesh_Bhor: thanks.04:40
rkmrHonjook, is there any topics? If it is nothing, we can finish this meeting.04:41
Dinesh_BhorNothing from my side04:42
rkmrHonjosagara: nothing?04:42
rkmrHonjook. thank you all.04:43
*** zhhuabj has quit IRC04:43
Dinesh_BhorThank you all04:43
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"04:44
openstackMeeting ended Tue Jan  9 04:44:08 2018 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)04:44
openstackMinutes:        http://eavesdrop.openstack.org/meetings/masakari/2018/masakari.2018-01-09-04.02.html04:44
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/masakari/2018/masakari.2018-01-09-04.02.txt04:44
openstackLog:            http://eavesdrop.openstack.org/meetings/masakari/2018/masakari.2018-01-09-04.02.log.html04:44
sagarathank you, bye04:44
*** sagara has quit IRC04:44
*** mriedem has quit IRC04:47
*** janki has joined #openstack-meeting04:47
*** rkmrHonjo has quit IRC04:48
*** Dinesh_Bhor has quit IRC04:50
*** Dinesh_Bhor has joined #openstack-meeting04:51
*** Dinesh_Bhor has quit IRC04:52
*** Dinesh_Bhor has joined #openstack-meeting04:55
*** markvoelker has joined #openstack-meeting04:56
*** abhishekk has left #openstack-meeting04:57
*** pengdake has joined #openstack-meeting05:01
*** zhhuabj has joined #openstack-meeting05:01
*** felipemonteiro_ has joined #openstack-meeting05:01
*** coolsvap has joined #openstack-meeting05:02
*** psachin has joined #openstack-meeting05:04
*** links has joined #openstack-meeting05:07
*** ijw has joined #openstack-meeting05:19
*** anilvenkata has joined #openstack-meeting05:22
*** arnewiebalck_ has quit IRC05:24
*** Tom-Tom has joined #openstack-meeting05:28
*** markstur_ has quit IRC05:29
*** markstur has joined #openstack-meeting05:29
*** claudiub has joined #openstack-meeting05:29
*** markstur has quit IRC05:29
*** frank_wang has joined #openstack-meeting05:37
*** frank_wang has quit IRC05:42
*** frank_wang has joined #openstack-meeting05:43
*** caowei has joined #openstack-meeting05:44
*** Apoorva has quit IRC05:49
*** markstur has joined #openstack-meeting05:50
*** emagana has joined #openstack-meeting05:51
*** Tom-Tom has quit IRC05:53
*** markstur has quit IRC05:55
*** iyamahat has joined #openstack-meeting05:56
*** Tom-Tom has joined #openstack-meeting06:02
*** jamesmcarthur has quit IRC06:02
*** jamesmcarthur has joined #openstack-meeting06:03
*** sandanar has joined #openstack-meeting06:03
*** jamesmcarthur has quit IRC06:07
*** Dinesh_Bhor has quit IRC06:07
*** felipemonteiro_ has quit IRC06:08
*** Dinesh_Bhor has joined #openstack-meeting06:10
*** wanghao_ has joined #openstack-meeting06:12
*** yamahata has joined #openstack-meeting06:13
*** ekcs has quit IRC06:14
*** wanghao has quit IRC06:15
*** aeng has quit IRC06:16
*** gouthamr has quit IRC06:17
*** marios has joined #openstack-meeting06:18
*** dbecker has quit IRC06:31
*** markstur has joined #openstack-meeting06:32
*** links has quit IRC06:35
*** markstur has quit IRC06:37
*** rcernin has quit IRC06:38
*** niraj_singh has joined #openstack-meeting06:38
*** niraj_singh has left #openstack-meeting06:39
*** dbecker has joined #openstack-meeting06:45
*** jamesmcarthur has joined #openstack-meeting06:46
*** zhurong has joined #openstack-meeting06:50
*** jamesmcarthur has quit IRC06:51
*** links has joined #openstack-meeting06:52
*** ad_rien_ has quit IRC06:52
*** ykatabam has quit IRC06:52
*** markstur has joined #openstack-meeting06:53
*** pcaruana has joined #openstack-meeting06:56
*** markstur has quit IRC06:57
*** gcb has joined #openstack-meeting06:58
*** liuyulong has quit IRC07:02
*** links has quit IRC07:03
*** Dinesh_Bhor has quit IRC07:05
*** egallen has joined #openstack-meeting07:05
*** numans has quit IRC07:11
*** emagana has quit IRC07:11
*** numans has joined #openstack-meeting07:13
*** markstur has joined #openstack-meeting07:14
*** links has joined #openstack-meeting07:14
*** Dinesh_Bhor has joined #openstack-meeting07:15
*** markstur has quit IRC07:18
*** Dinesh_Bhor has quit IRC07:27
*** felipemonteiro_ has joined #openstack-meeting07:27
*** links has quit IRC07:27
*** liyi has joined #openstack-meeting07:32
*** ad_rien_ has joined #openstack-meeting07:35
*** egallen has quit IRC07:37
*** makowals has joined #openstack-meeting07:39
*** felipemonteiro_ has quit IRC07:39
*** slaweq has joined #openstack-meeting07:41
*** wanghao_ has quit IRC07:43
*** slaweq has quit IRC07:44
*** wanghao has joined #openstack-meeting07:44
*** e0ne has joined #openstack-meeting07:44
*** links has joined #openstack-meeting07:45
*** slaweq has joined #openstack-meeting07:47
*** Dinesh_Bhor has joined #openstack-meeting07:48
*** notmyname has quit IRC07:49
*** rhagarty has quit IRC07:49
*** d0ugal has joined #openstack-meeting07:51
*** slaweq has quit IRC07:52
*** notmyname has joined #openstack-meeting07:52
*** egallen has joined #openstack-meeting07:52
*** rhagarty has joined #openstack-meeting07:52
*** slaweq has joined #openstack-meeting08:00
*** armax has quit IRC08:08
*** slaweq has quit IRC08:08
*** rbartal has joined #openstack-meeting08:08
*** egallen has quit IRC08:10
*** oidgar has joined #openstack-meeting08:19
*** tesseract has joined #openstack-meeting08:20
*** egallen has joined #openstack-meeting08:21
*** alexchadin has joined #openstack-meeting08:22
*** slaweq has joined #openstack-meeting08:23
*** Alex_Staf has joined #openstack-meeting08:25
*** markstur has joined #openstack-meeting08:26
*** psachin has quit IRC08:30
*** markstur has quit IRC08:30
*** phil_ has joined #openstack-meeting08:32
*** phil_ is now known as Guest2417808:33
*** egallen_ has joined #openstack-meeting08:33
*** egallen has quit IRC08:34
*** egallen_ is now known as egallen08:34
*** yamamoto_ has joined #openstack-meeting08:36
*** daidv has quit IRC08:38
*** tovin07_ has quit IRC08:38
*** kiennt26 has quit IRC08:38
*** tovin07_ has joined #openstack-meeting08:38
*** kiennt26 has joined #openstack-meeting08:39
*** daidv has joined #openstack-meeting08:39
*** yamamoto has quit IRC08:39
*** psachin has joined #openstack-meeting08:40
*** fzdarsky has joined #openstack-meeting08:40
*** dmacpher has joined #openstack-meeting08:47
*** Dinesh_Bhor has quit IRC08:49
*** markstur has joined #openstack-meeting08:52
*** Sandeep has joined #openstack-meeting08:55
*** Sandeep is now known as Guest1774908:55
*** priteau has joined #openstack-meeting08:56
*** chenying_ has joined #openstack-meeting08:56
*** markstur has quit IRC08:57
chenying_#startmeeting karbor09:00
openstackMeeting started Tue Jan  9 09:00:32 2018 UTC and is due to finish in 60 minutes.  The chair is chenying_. Information about MeetBot at http://wiki.debian.org/MeetBot.09:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.09:00
*** openstack changes topic to " (Meeting topic: karbor)"09:00
openstackThe meeting name has been set to 'karbor'09:00
chenying_There is no topic today. If there are some topics about karbor, we can discuss them in the karbor's irc meeting.09:06
*** Dinesh_Bhor has joined #openstack-meeting09:08
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"09:12
openstackMeeting ended Tue Jan  9 09:12:50 2018 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)09:12
openstackMinutes:        http://eavesdrop.openstack.org/meetings/karbor/2018/karbor.2018-01-09-09.00.html09:12
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/karbor/2018/karbor.2018-01-09-09.00.txt09:12
openstackLog:            http://eavesdrop.openstack.org/meetings/karbor/2018/karbor.2018-01-09-09.00.log.html09:12
*** markstur has joined #openstack-meeting09:13
*** iyamahat has quit IRC09:14
*** liyi has quit IRC09:15
*** markstur has quit IRC09:18
*** wanghao has quit IRC09:25
*** wanghao has joined #openstack-meeting09:26
*** wanghao has quit IRC09:26
*** wanghao has joined #openstack-meeting09:26
*** wanghao has quit IRC09:26
*** wanghao has joined #openstack-meeting09:27
*** wanghao has quit IRC09:27
*** wanghao has joined #openstack-meeting09:27
*** jamesmcarthur has joined #openstack-meeting09:28
*** wanghao has quit IRC09:28
*** wanghao has joined #openstack-meeting09:28
*** wanghao has quit IRC09:29
*** wanghao has joined #openstack-meeting09:29
*** wanghao has quit IRC09:29
*** wanghao has joined #openstack-meeting09:30
*** wanghao has quit IRC09:30
*** wanghao has joined #openstack-meeting09:31
*** wanghao has quit IRC09:31
*** wanghao has joined #openstack-meeting09:31
*** wanghao has quit IRC09:32
*** wanghao has joined #openstack-meeting09:32
*** jamesmcarthur has quit IRC09:32
*** wanghao has quit IRC09:33
*** wanghao has joined #openstack-meeting09:33
*** Dinesh_Bhor has quit IRC09:38
*** iyamahat has joined #openstack-meeting09:42
*** iyamahat has quit IRC09:42
*** iyamahat has joined #openstack-meeting09:47
*** yamamoto_ has quit IRC09:52
*** erlon has joined #openstack-meeting09:54
*** iyamahat has quit IRC09:56
*** lpetrut has joined #openstack-meeting10:01
*** Tom-Tom has quit IRC10:07
*** apetrich has quit IRC10:09
*** Guest17749 has quit IRC10:09
*** epico has quit IRC10:14
*** kaliya has quit IRC10:14
*** frank_wang has quit IRC10:15
*** yamamoto has joined #openstack-meeting10:16
*** sambetts|afk is now known as sambetts10:16
*** epico has joined #openstack-meeting10:16
*** d0ugal has quit IRC10:19
*** d0ugal has joined #openstack-meeting10:21
*** caowei has quit IRC10:32
*** janki has quit IRC10:38
*** Tom-Tom has joined #openstack-meeting10:38
*** sdague has joined #openstack-meeting10:40
*** liyi has joined #openstack-meeting10:41
*** Tom-Tom has quit IRC10:42
*** priteau has quit IRC10:43
*** tovin07_ has quit IRC10:43
*** priteau has joined #openstack-meeting10:44
*** liyi has quit IRC10:46
*** hemna_ has quit IRC10:48
*** priteau has quit IRC10:48
*** kopecmartin has joined #openstack-meeting10:50
*** anilvenkata_ has joined #openstack-meeting10:50
*** chyka has joined #openstack-meeting10:51
*** janki has joined #openstack-meeting10:51
*** yamahata has quit IRC10:53
*** alexchadin has quit IRC10:53
*** anilvenkata has quit IRC10:54
*** chyka has quit IRC10:55
*** epico has quit IRC10:56
*** alexchadin has joined #openstack-meeting10:59
*** anilvenkata_ has quit IRC11:03
*** fzdarsky is now known as fzdarsky|afk11:08
*** priteau has joined #openstack-meeting11:11
*** janki has quit IRC11:13
*** janki has joined #openstack-meeting11:13
*** priteau has quit IRC11:15
*** jchhatbar has joined #openstack-meeting11:19
*** janki has quit IRC11:22
*** gmann_ has joined #openstack-meeting11:22
*** pengdake has quit IRC11:35
*** bkopilov has quit IRC11:36
*** egallen has quit IRC11:37
*** egallen has joined #openstack-meeting11:37
*** priteau has joined #openstack-meeting11:38
*** egallen has quit IRC11:41
*** apetrich has joined #openstack-meeting11:41
*** egallen has joined #openstack-meeting11:42
*** jamesmcarthur has joined #openstack-meeting11:49
*** liyi has joined #openstack-meeting11:49
*** jamesmcarthur has quit IRC11:53
*** liyi has quit IRC11:54
*** apetrich has quit IRC12:01
*** yamamoto has quit IRC12:06
*** yamamoto has joined #openstack-meeting12:06
*** ijw has quit IRC12:08
*** alexchadin has quit IRC12:14
*** alexchadin has joined #openstack-meeting12:14
*** jkilpatr has quit IRC12:18
*** raildo has joined #openstack-meeting12:19
*** hareesh has joined #openstack-meeting12:23
*** tlaxkit has joined #openstack-meeting12:24
*** rbartal has quit IRC12:29
*** rbartal has joined #openstack-meeting12:32
*** hareesh has quit IRC12:34
*** ijw has joined #openstack-meeting12:40
*** liyi has joined #openstack-meeting12:49
*** liyi has quit IRC12:53
*** jkilpatr has joined #openstack-meeting12:54
*** liyi has joined #openstack-meeting12:54
*** zhurong_ has joined #openstack-meeting12:56
*** pchavva has joined #openstack-meeting12:57
*** bobh has joined #openstack-meeting12:58
*** liyi has quit IRC12:58
*** makowals has quit IRC13:00
*** makowals has joined #openstack-meeting13:01
*** chenyb4 has joined #openstack-meeting13:03
*** zhurong_ has quit IRC13:07
*** zhurong has quit IRC13:07
*** liyi has joined #openstack-meeting13:08
*** frank_wang has joined #openstack-meeting13:11
*** liyi has quit IRC13:12
*** frank_wang has quit IRC13:15
*** bkopilov has joined #openstack-meeting13:17
*** bobh has quit IRC13:17
*** edmondsw has joined #openstack-meeting13:21
*** alexchadin has quit IRC13:23
*** alexchadin has joined #openstack-meeting13:24
*** yangyapeng has quit IRC13:27
*** gmann_ has quit IRC13:31
*** Tom-Tom has joined #openstack-meeting13:35
*** jamesmcarthur has joined #openstack-meeting13:38
*** chenyb4 has quit IRC13:41
*** markstur has joined #openstack-meeting13:42
*** anilvenkata_ has joined #openstack-meeting13:43
*** davidsha has joined #openstack-meeting13:44
*** markstur has quit IRC13:47
*** kiennt26_ has joined #openstack-meeting13:47
*** esberglu has joined #openstack-meeting13:48
*** dprince has joined #openstack-meeting13:48
*** apetrich has joined #openstack-meeting13:49
*** liyi has joined #openstack-meeting13:51
*** coolsvap has quit IRC13:52
*** mriedem has joined #openstack-meeting13:54
*** jchhatbar has quit IRC13:54
*** yangyapeng has joined #openstack-meeting13:54
*** lhx_ has joined #openstack-meeting13:58
*** shintaro has joined #openstack-meeting14:00
*** markstur has joined #openstack-meeting14:03
*** liyi has quit IRC14:03
*** alexchadin has quit IRC14:06
*** markstur has quit IRC14:07
*** jamesmcarthur has quit IRC14:07
*** alexchadin has joined #openstack-meeting14:10
*** amodi has joined #openstack-meeting14:13
*** jroll has quit IRC14:19
*** jroll has joined #openstack-meeting14:19
*** makowals has quit IRC14:21
*** VW has joined #openstack-meeting14:22
*** makowals has joined #openstack-meeting14:22
*** markstur has joined #openstack-meeting14:24
*** rbudden has joined #openstack-meeting14:24
*** psachin has quit IRC14:28
*** markstur has quit IRC14:28
*** makowals has quit IRC14:29
*** cleong has joined #openstack-meeting14:33
*** shintaro has quit IRC14:33
*** makowals has joined #openstack-meeting14:34
*** felipemonteiro_ has joined #openstack-meeting14:36
*** yamamoto_ has joined #openstack-meeting14:37
*** jamesmcarthur has joined #openstack-meeting14:39
*** yamamoto has quit IRC14:40
*** markvoelker has quit IRC14:40
*** eharney has joined #openstack-meeting14:41
*** jamesmcarthur has quit IRC14:43
*** spn has quit IRC14:43
*** markstur has joined #openstack-meeting14:45
*** kaisers has quit IRC14:45
*** markstur has quit IRC14:49
*** spn has joined #openstack-meeting14:50
*** spn has quit IRC14:50
*** spn has joined #openstack-meeting14:50
*** kaisers has joined #openstack-meeting14:51
*** ttrifonov has quit IRC14:55
*** alexchadin has quit IRC14:56
*** ttrifonov has joined #openstack-meeting14:57
*** awaugama has joined #openstack-meeting14:59
*** gouthamr has joined #openstack-meeting15:00
*** slaweq has quit IRC15:02
*** slaweq has joined #openstack-meeting15:04
*** oidgar has quit IRC15:05
*** slaweq has quit IRC15:09
*** oidgar has joined #openstack-meeting15:11
*** bobh has joined #openstack-meeting15:12
*** markvoelker has joined #openstack-meeting15:13
*** wanghao_ has joined #openstack-meeting15:15
*** markvoelker has quit IRC15:16
*** wanghao has quit IRC15:17
*** armax has joined #openstack-meeting15:21
*** egallen has quit IRC15:24
*** egallen has joined #openstack-meeting15:24
*** markvoelker has joined #openstack-meeting15:25
*** longkb1 has left #openstack-meeting15:31
*** makowals has quit IRC15:32
*** Apoorva has joined #openstack-meeting15:33
*** ssathaye has quit IRC15:34
*** ssathaye has joined #openstack-meeting15:34
*** links has quit IRC15:36
*** slaweq has joined #openstack-meeting15:38
*** slaweq has quit IRC15:42
*** longkb has joined #openstack-meeting15:42
*** kiennt26_ has quit IRC15:43
*** david-lyle has joined #openstack-meeting15:44
*** makowals has joined #openstack-meeting15:45
*** felipemonteiro__ has joined #openstack-meeting15:45
*** felipemonteiro_ has quit IRC15:49
*** jlibosva has joined #openstack-meeting15:55
*** markstur has joined #openstack-meeting15:55
*** slaweq has joined #openstack-meeting15:56
*** mlavalle has joined #openstack-meeting15:56
*** fabian_ has joined #openstack-meeting15:58
*** jamesmcarthur has joined #openstack-meeting15:59
*** SerenaFeng has joined #openstack-meeting15:59
*** jamesmcarthur has quit IRC15:59
*** jamesmcarthur has joined #openstack-meeting15:59
*** ihrachys has joined #openstack-meeting16:00
*** apetrich has quit IRC16:00
ihrachys#startmeeting neutron_ci16:00
openstackMeeting started Tue Jan  9 16:00:33 2018 UTC and is due to finish in 60 minutes.  The chair is ihrachys. Information about MeetBot at http://wiki.debian.org/MeetBot.16:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:00
*** openstack changes topic to " (Meeting topic: neutron_ci)"16:00
openstackThe meeting name has been set to 'neutron_ci'16:00
ihrachysI think I happily missed the last time we were supposed to have a meeting. sorry for that. :)16:01
ihrachyspost-holiday recovery is hard!16:01
ihrachys#topic Actions from prev meeting16:01
*** openstack changes topic to "Actions from prev meeting (Meeting topic: neutron_ci)"16:01
ihrachysfirst is "mlavalle to send patch(es) removing duplicate jobs from neutron gate"16:01
mlavalleI did16:01
ihrachysI believe it's https://review.openstack.org/530500 and https://review.openstack.org/53149616:02
ihrachysand of course we should land the last bit adding new jobs to stable: https://review.openstack.org/53107016:02
*** felipemonteiro__ has quit IRC16:03
ihrachysmlavalle, speaking of your concerns in https://review.openstack.org/#/c/531496/16:03
*** felipemonteiro__ has joined #openstack-meeting16:03
ihrachysmlavalle, I would think that we replace old jobs with in-tree ones for those other projects16:03
mlavallewill work on that16:03
ihrachysyou are too agreeable today! come on! :)16:03
*** janki has joined #openstack-meeting16:03
*** sandanar has quit IRC16:03
*** yamahata has joined #openstack-meeting16:04
mlavalleyou speak with reason, so why not :-)16:04
ihrachysmlavalle, I am not saying that's the right path; I just *assume*16:04
ihrachysstill makes sense to check with infra first16:04
mlavalleyes I will do that16:04
mlavallebut we don't need to rush on that patch16:04
mlavalleas long as 530500 lands16:04
ihrachysmlavalle, the project-config one will already remove duplicates from gate for us?16:04
mlavalle530500 will remove the duplication16:05
ihrachysok good. yeah, we can leave the second one for infra take.16:05
ihrachysok next item was "mlavalle to report back about result of drivers discussion on unified tempest plugin for all stadium projects"16:05
ihrachysmlavalle, could you please give tl;dr of the discussion?16:06
mlavallewe discussed it in the drivers meeting before the Holidays16:06
mlavallewe will let the stadium projects join the unified tempest plugin or keep their tests in their repos16:06
mlavallewe don't want to stretch them with more work16:06
mlavallesent a message to the ML last week16:07
mlavalleand discussed it in the Neutron meeting last week16:07
mlavallewas well received16:07
ihrachysok cool. are there candidates who are willing to go through the exercise?16:08
mlavalleI haven't heard16:08
ihrachysok, fine by me!16:08
*** rbartal has quit IRC16:08
ihrachysnext is "frickler to post patch updating neutron grafana board to include designate scenario job"16:08
ihrachysI believe this was merged https://review.openstack.org/#/c/529822/16:08
ihrachysI just saw the job in grafana before the meeting, so it works16:08
ihrachysnext one is "jlibosva to close bug 1722644 and open a new one for trunk connectivity failures in dvr and linuxbridge scenario jobs"16:09
openstackbug 1722644 in neutron "TrunkTest fails for OVS/DVR scenario job" [High,Confirmed] https://launchpad.net/bugs/172264416:09
*** markvoelker has quit IRC16:09
jlibosvaI did16:09
openstackLaunchpad bug 1740885 in neutron "Security group updates fail when port hasn't been initialized yet" [High,In progress] - Assigned to Jakub Libosvar (libosvar)16:09
ihrachysthe other bug doesn't seem closed16:09
jlibosvafix is here: https://review.openstack.org/#/c/531414/16:09
jlibosvaoops, I just wrote a comment I'm gonna close it and I acutally didn't16:10
jlibosvaclosed now16:10
ihrachysjlibosva, right. I started looking at the patch yesterday, and was wondering if the fix makes it ignore some updates from the server. can we miss an update and leave a port in old state with the fix?16:10
jlibosvaI guess so, the update stores e.g. IP in case a new port is added to remote security group. So we should still receive an update but let the implementation of flow rules deferred16:11
ihrachysto clarify, you mean there is a potential race now?16:12
jlibosvathere has been a race and the 531414 fixes it16:13
jlibosvait caches the new data and once the port initializes, it already has the correct data in cache16:13
ihrachysoh I see, so there is cache involved so we should be good16:14
jlibosvaI saw the ovsfw job failing, reporting the issue, so I still need to investigate why it fails16:15
jlibosvaI'm not able to reproduce it, I'll need to run tempest in the loop overnight and put a breakpoint once it fails so I'll be able to debug16:15
jlibosvait's likely a different issue16:15
ihrachysthe ovsfw job was never too stable. but yeah, it's good to look into it.16:15
ihrachysok we have way forward here.16:16
ihrachysnext item was "jlibosva to disable trunk scenario connectivity tests"16:16
ihrachysI believe it's https://review.openstack.org/53076016:16
ihrachysand it's merged16:16
jlibosvaso now next step should be to enable it back :)16:16
jlibosvaperhaps it should be part of 531414 patch16:17
*** hongbin has joined #openstack-meeting16:17
*** tobiash has joined #openstack-meeting16:18
ihrachysyeah it makes sense to have it incorporated. we can recheck it a bunch before merging.16:18
*** lpetrut has quit IRC16:18
ihrachysok next was "ihrachys to report sec group fullstack failure"16:18
ihrachysI am ashamed but I forgot about it16:18
* haleyb wanders in late16:19
ihrachysI will follow up after the meeting16:19
mlavallethe Holidays joy I guess16:19
ihrachys#action ihrachys to report sec group fullstack failure16:19
ihrachysnext item was "slaweq to debug qos fullstack failure https://bugs.launchpad.net/neutron/+bug/1737892"16:19
openstackLaunchpad bug 1737892 in neutron "Fullstack test test_qos.TestBwLimitQoSOvs.test_bw_limit_qos_port_removed failing many times" [High,Confirmed] - Assigned to Slawek Kaplonski (slaweq)16:19
slaweqso i was debugging it for some time16:20
slaweqand I have no idea why it is failing16:20
slaweqI was even able to reproduce it locally once for every 20-30 runs but I don't know what is happening there16:20
slaweqI will have to check it more still16:21
ihrachysI see. "Also strange thing is that after test is finished, those records are deleted from ovs." <- don't we recycle ovsdb on each run?16:21
slaweqbut maybe for now I will propose patch to mark this test as unstable16:21
slaweqihrachys: maybe, but I'm not sure about that16:21
ihrachysah wait, we didn't have it isolated properly yet16:21
slaweqno, it's same ovsdb for all tests IMO16:22
ihrachysso if I read it right, ovsdb command is executed but structures are still in place?16:22
jlibosvacould it be related to the ovsdb issue we have been hitting? that commands return TRY_AGAIN and never succeed?16:23
slaweqand ovsdb commands are finished with success every time as I was checking it locally16:23
jlibosvaor they timeout?16:23
slaweqno, there wasn't any retry or timeout on it16:23
jlibosvait doesn't sound like it then16:23
slaweqtransactions was finished fine always16:24
ihrachyscould it be something recreates them after destroy?16:24
*** jkilpatr_ has joined #openstack-meeting16:24
*** SerenaFeng has quit IRC16:24
slaweqihrachys: I don't think so because I was also locally watching for them with watch command and it didn't flap or something like that16:24
slaweqbut I will check it once again maybe16:25
*** jkilpatr has quit IRC16:26
ihrachysok. I think disabling the test is a good idea while we are looking into it.16:26
slaweqihrachys: ok, so I will do such patch today16:26
ihrachysand those were all action items we had16:26
ihrachys#topic Tempest plugin16:26
*** openstack changes topic to "Tempest plugin (Meeting topic: neutron_ci)"16:26
ihrachysI think we are mostly done with tempest plugin per se16:26
ihrachysthough there are some tempest bits in-tree that required stadium projects to switch to new repo first16:27
ihrachysand those were not moving quick enough16:27
ihrachysf.e. vpnaas: https://review.openstack.org/#/c/521341/16:27
ihrachysI recollect that mlavalle was going to talk to their representatives about moving those patches forward16:28
ihrachysmlavalle, any news on this front?16:28
mlavallewell, now it is my turn to say that I forgot about that16:28
mlavalleI will follow up this week16:28
mlavallesorry :-(16:28
ihrachys#action mlavalle to follow up with stadium projects on switching to new tempest repo16:29
ihrachysthat's fine it happens :)16:29
ihrachysthe list was at https://etherpad.openstack.org/p/neutron-tempest-plugin-job-move line 27+16:29
ihrachysafaiu we should cover vpnaas dynamic-routing and midonet16:29
*** Guest24178 has quit IRC16:30
*** Tom-Tom has quit IRC16:30
ihrachysafair there were concerns about how to install the new repo, and devstack plugin was added to neutron-tempest-plugin, so stadium projects should consume it16:30
ihrachys#topic Grafana16:31
*** openstack changes topic to "Grafana (Meeting topic: neutron_ci)"16:31
ihrachyswe actually have some good news there16:31
ihrachyslinuxbridge scenario job seems to be in decent shape after recent tests disabled and security group fix16:31
ihrachysit's at ~10% right now16:31
*** ad_rien_ has quit IRC16:32
ihrachyswhich I would say a regular failure rate one could expect from a tempest job16:32
ihrachysso that's good. we should monitor and eventually make it voting if it will keep the level.16:32
slaweqnice :)16:32
jlibosvakudos to those who fixed it :)16:33
ihrachysdvr flavor is not as great though also down from 90% level it stayed in for months16:33
ihrachyscurrently at ~40% on my chart16:33
ihrachysand fullstack is in same bad shape. so mixed news but definitely progress on scenario side.16:34
ihrachyskudos to everyone who was and is pushing those forward16:34
jlibosvaI see functional is now ~10%, yesterday EMEA morning, it was up to 50%16:34
*** kopecmartin has quit IRC16:34
ihrachysI saw a lot of weird errors in gate yesterday16:35
*** lhx_ has quit IRC16:35
ihrachysRETRY_TIMEOUTS and stuff16:35
ihrachyscould be that the gate was just unstable in general16:35
*** e0ne has quit IRC16:35
*** jamesmcarthur has quit IRC16:35
ihrachysbut we'll keep an eye16:35
ihrachysf.e. I see a similar spike for unit tests around same time16:36
ihrachys#topic Scenarios16:36
*** openstack changes topic to "Scenarios (Meeting topic: neutron_ci)"16:36
ihrachysso since linuxbridge seems good now, let's have a look at a latest failure for dvr16:36
*** makowals has quit IRC16:37
*** markvoelker has joined #openstack-meeting16:37
ihrachysok I took http://logs.openstack.org/98/513398/10/check/neutron-tempest-plugin-dvr-multinode-scenario/568c685/job-output.txt.gz16:37
ihrachysbut it seems like a timeout for the job16:37
ihrachystook almost 3h and failed16:37
ihrachyshere is another one: http://logs.openstack.org/51/529551/4/check/neutron-tempest-plugin-dvr-multinode-scenario/ee927b8/job-output.txt.gz16:38
ihrachyssame story16:38
jlibosvaperhaps we should try now to increase the concurrency? let me check the load16:39
ihrachysso for what I see, it either times out or it passes16:39
ihrachysI suspect it may be related to meltdown16:39
ihrachyswe were looking yesterday in neutron channel at slowdown for rally scenarios16:39
haleybhttps://bugs.launchpad.net/neutron/+bug/1717302 still isn't fixed either16:39
openstackLaunchpad bug 1717302 in neutron "Tempest floatingip scenario tests failing on DVR Multinode setup with HA" [High,Confirmed] - Assigned to Brian Haley (brian-haley)16:39
ihrachysx2-x3 slowdown for some scenarios16:39
ihrachysand figured it depends on cloud and whether they are patched...16:40
*** yamamoto_ has quit IRC16:40
jlibosvahaleyb: don't we skip the fip test cases?16:41
jlibosvaI mean that they are tagged as unstable, so skipped if they fail16:41
haleyboh yeah, didn't see a link in the bug16:41
ihrachysjlibosva, we do skip them, yes16:41
*** cdub has quit IRC16:41
*** cdub has joined #openstack-meeting16:41
ihrachysjlibosva, you mentioned concurrency. currently we seem to run with =216:42
ihrachyswould the suggestion be to run with eg. number of cpus?16:42
jlibosvajust remove the concurrency and let the runner decide16:42
ihrachysI see most scenarios already taking like 5minutes+ each in good run16:42
*** markstur_ has joined #openstack-meeting16:42
jlibosvaI remember I added the concurrency there because we thought the server gets choked on slow machines16:42
ihrachysI suspect there may be hardcoded timeouts in some of scenarios we have, like waiting for a resource to come back16:43
slaweqbut wouldn't it be problem with e.g. memory consumption if You will have more treads?16:43
slaweq(just asking)16:43
ihrachysslaweq, there may be. especially where scenarios spin up instances.16:43
ihrachysI guess it's easier to post a patch and recheck it to death16:44
ihrachysand see how it fares16:44
ihrachysI can have a look at it. need to report a bug for timeouts too.16:44
ihrachys#action ihrachys to report bug for dvr scenario job timeouts and try concurrency increase16:44
ihrachysit's interesting that linuxbridge doesn't seem to trigger it.16:45
ihrachysis it because maybe more tests are executed for dvr?16:45
*** felipemonteiro_ has joined #openstack-meeting16:45
*** markstur has quit IRC16:46
ihrachysit's 36 tests in dvr and 28 in linuxbridge16:46
ihrachyssome of those are dvr migration tests so that's good16:47
ihrachysI also noticed that NetworkMtuBaseTest is not executed for linuxbridge because apparently we assume gre type driver enabled and it's not supported by linuxbridge16:47
ihrachyswe can probably configure the plugin for vxlan and have it executed for linuxbridge too then16:47
jlibosvamakes sense, let's make linuxbridge suffer too :)16:48
ihrachysactually, it's 31 vs 21 tests16:48
ihrachysI read the output incorrectly before16:48
ihrachysjlibosva, huh. well it shouldn't affect much and would add coverage16:48
jlibosvaI totally agree :)16:49
ihrachysI will report a bug for that at least. it's not critical to fix it right away.16:49
*** felipemonteiro__ has quit IRC16:49
ihrachys#action ihrachys to report bug for mtu scenario not executed for linuxbridge job16:49
ihrachysanyway, the diff in total time spent for tests is ~3k seconds16:50
ihrachyswhich is like 50 minutes?16:50
ihrachysthat's kinda significant16:50
jlibosvathat could be it, average test is like 390 seconds? so if you add 10 more, with concurrency two, it adds almost 30 minutes16:51
ihrachysyeah. and linuxbridge is already 2h:20m in good run16:51
jlibosvawith bad luck using some slow cloud, it could start hitting timeouts too. That's why I think it's good to make linuxbridge suffering too, to confirm our theory :)16:51
ihrachysadd 30-50 on top and you have timeout triggered16:51
ihrachysok anyway, we have way forward. let's switch topics16:52
*** chyka has joined #openstack-meeting16:52
ihrachys#topic Fullstack16:52
*** openstack changes topic to "Fullstack (Meeting topic: neutron_ci)"16:52
jlibosvaI remember in tempest repo, we have the slow tag so maybe we will need to start using it in the future16:52
ihrachysjlibosva, yeah but ideally we still want to have all those executed in some way in gate16:52
ihrachysso the best you can do is then split the job into pieces16:52
ihrachysok fullstack. same exercise, looking at latest runs.16:53
jlibosvayeah, looking at times of particular tests, they all are slow16:53
ihrachysexample: http://logs.openstack.org/43/529143/3/check/neutron-fullstack/d031a6b/logs/testr_results.html.gz16:53
*** StefanPaetowJisc has joined #openstack-meeting16:53
ihrachysthis is I believe the failure I should have reported but failed to.16:53
ihrachysjlibosva, afaiu we don't have accelerated virtualization in infra clouds. so starting an instance takes ages.16:53
ihrachyslet's see if other fullstack runs are same16:54
ihrachysok this one is different: http://logs.openstack.org/98/513398/10/check/neutron-fullstack/b62a726/logs/testr_results.html.gz16:54
ihrachysour old friend "Commands [...] exceeded timeout 10 seconds"16:54
*** StefanPaetowJisc has quit IRC16:54
haleybnot a very nice friend16:55
ihrachysI recollect jlibosva reported https://bugs.launchpad.net/neutron/+bug/1741889 lately16:55
openstackLaunchpad bug 1741889 in neutron "functional: DbAddCommand sometimes times out after 10 seconds" [Critical,New]16:55
jlibosvaI thought it's more sever as by the time I reported the functional job seemed busted16:55
jlibosvaso it's not that hot anymore16:55
mlavalleyou mean 1741889?16:56
mlavalleshould we lower the importance then?16:56
ihrachysjlibosva, do you suggest that may be intermittent and same and we may no longer experience it?16:56
jlibosvawell, I mean we talked about it before, you said UT had a peak at about that time, so the bug might not be the reason for functional failure rate peak16:57
jlibosvaalthough it's not a nice back as it still keeps coming back in weird intervals, like twice a year :) and then it goes away16:58
ihrachysok I lowered to High it for now16:58
ihrachysand added details about fullstack16:58
jlibosvacould be related to slow hw used16:58
jlibosvajust thinking out loud16:58
ihrachysyeah I think it's ok to just monitor16:58
ihrachysthat other sec group failure seems more important to have a look.16:59
ihrachysand we have some work to do for scenarios anyway till next week16:59
slaweqI can check on this failure with SG16:59
slaweq*check this failure16:59
ihrachysslaweq, if you like, I would be grateful of course.16:59
ihrachyshave some other pressing things on my plate so I don't insist :)16:59
*** jamesmcarthur has joined #openstack-meeting16:59
ihrachys#action slaweq to take over sec group failure in fullstack (report bug / triage / fix)17:00
ihrachyswe are out of time17:00
slaweqjust to be sure, we are talking about issue liek in http://logs.openstack.org/43/529143/3/check/neutron-fullstack/d031a6b/logs/testr_results.html.gz17:00
ihrachysslaweq, yes17:00
*** gyee has joined #openstack-meeting17:01
ihrachysok. thanks everyone for being so helpful and showing initiative. it's good it's just me slagging here, otherwise we wouldn't achieve all we did.17:01
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"17:01
openstackMeeting ended Tue Jan  9 17:01:11 2018 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)17:01
openstackMinutes:        http://eavesdrop.openstack.org/meetings/neutron_ci/2018/neutron_ci.2018-01-09-16.00.html17:01
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/neutron_ci/2018/neutron_ci.2018-01-09-16.00.txt17:01
openstackLog:            http://eavesdrop.openstack.org/meetings/neutron_ci/2018/neutron_ci.2018-01-09-16.00.log.html17:01
*** slaweq has quit IRC17:02
*** dmacpher has quit IRC17:03
*** sridharg has quit IRC17:03
*** Leo_m has joined #openstack-meeting17:04
*** Alex_Staf has quit IRC17:04
*** armax has quit IRC17:05
*** cdub has quit IRC17:05
*** janki has quit IRC17:06
*** alex_xu_ has quit IRC17:07
*** tlaxkit has quit IRC17:07
*** hongbin has quit IRC17:08
*** hongbin has joined #openstack-meeting17:08
*** cdub has joined #openstack-meeting17:08
*** alex_xu has joined #openstack-meeting17:08
*** jkilpatr has joined #openstack-meeting17:09
*** jkilpatr_ has quit IRC17:09
*** rderose has joined #openstack-meeting17:10
*** felipemonteiro_ has quit IRC17:13
*** fabian_ has quit IRC17:15
*** fabian_ has joined #openstack-meeting17:16
*** julim has quit IRC17:16
*** julim has joined #openstack-meeting17:17
*** fabian_ has quit IRC17:20
*** julim has quit IRC17:22
*** david-lyle has quit IRC17:23
*** ricolin has quit IRC17:24
*** jlibosva has quit IRC17:25
*** david-lyle has joined #openstack-meeting17:26
*** rossella_s has quit IRC17:27
*** marios has quit IRC17:27
*** gagehugo has joined #openstack-meeting17:28
*** rossella_s has joined #openstack-meeting17:29
*** Apoorva has quit IRC17:31
*** makowals has joined #openstack-meeting17:33
*** jamesmcarthur has quit IRC17:36
*** egallen has quit IRC17:36
*** StefanPaetowJisc has joined #openstack-meeting17:37
*** yamahata has quit IRC17:38
*** sgrasley1 has quit IRC17:38
*** sgrasley1 has joined #openstack-meeting17:40
*** StefanPaetowJisc has quit IRC17:40
*** yamamoto has joined #openstack-meeting17:41
*** jdennis has joined #openstack-meeting17:45
*** egallen has joined #openstack-meeting17:46
*** jamesmcarthur has joined #openstack-meeting17:47
*** markvoelker has quit IRC17:47
*** yamamoto has quit IRC17:49
*** davidsha has quit IRC17:52
*** tesseract has quit IRC17:57
*** jessegler has joined #openstack-meeting17:58
*** edmondsw has quit IRC17:58
*** lamt` has joined #openstack-meeting17:58
lbragstad#startmeeting keystone18:00
openstackMeeting started Tue Jan  9 18:00:17 2018 UTC and is due to finish in 60 minutes.  The chair is lbragstad. Information about MeetBot at http://wiki.debian.org/MeetBot.18:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.18:00
*** openstack changes topic to " (Meeting topic: keystone)"18:00
openstackThe meeting name has been set to 'keystone'18:00
lbragstadping ayoung, breton, cmurphy, dstanek, edmondsw, gagehugo, henrynash, hrybacki, knikolla, lamt, lbragstad, lwanderley, kmalloc, rderose, rodrigods, samueldmq, spilla, aselius, dpar, jdennis, ruan_he18:00
lbragstad#link https://etherpad.openstack.org/p/keystone-weekly-meeting18:00
kmallocman, is it meeting time already? I just finished my coffee...18:00
kmalloci need more.18:00
*** spilla has joined #openstack-meeting18:00
lbragstadbetter refill18:00
kmallocit will take like 6-10m to do so... so, i'll just wait.18:01
*** edmondsw has joined #openstack-meeting18:01
lbragstadwe'll give folks a minute to join18:01
*** Apoorva has joined #openstack-meeting18:02
samueldmqhi o/18:02
spilla o/18:02
*** Apoorva has quit IRC18:02
lbragstad#topic announcements: PTG planning18:04
*** openstack changes topic to "announcements: PTG planning (Meeting topic: keystone)"18:04
kmallocoh my ;)18:04
lbragstadnow that we're past the holidays - i've started prepping for the PTG18:04
lbragstad#link https://etherpad.openstack.org/p/keystone-rocky-ptg18:04
samueldmqPTG? When and where?18:04
lbragstadlast week of february18:05
lbragstadin dublin18:05
*** rmcall has joined #openstack-meeting18:05
lbragstad#link https://www.openstack.org/ptg18:05
lbragstadso - per our usual process18:05
lbragstadplease don't hesitate to throw topics on the etherpad18:05
lbragstadi bootstrapped it with some content, but it certainly isn't full18:05
*** edmondsw has quit IRC18:06
jdennisdublin, sounds like a good venue for someone named Colleen Murphy :-)18:06
*** david-lyle has quit IRC18:06
lbragstadonce we get closer to the PTG - we'll finalize a schedule18:06
cmurphythey get confused by my accent there18:06
lbragstadall in favor of cmurphy adopting an irish accent for the week?18:07
lbragstadi also have a list of attendance at the top18:08
lbragstadif you're planning on being in dublin, please add your name18:08
lbragstadI'd like to organize a team dinner of some kind prior to the PTG18:08
lbragstadno big rush there if you're still waiting on approval18:09
lbragstad(also, if there is anything i can do to help you get approval, just DM me!)18:09
lbragstad#topic announcement: reviews18:10
*** openstack changes topic to "announcement: reviews (Meeting topic: keystone)"18:10
lbragstadjust a friendly reminder that feature freeze is in 16 days (not factoring in feature freeze exceptions)18:10
lbragstadbut given the rc window is only a couple week, i expect very little, if any time, for feature freeze exceptions18:11
lbragstadwe're going to have to be on the ball with feature work before that deadline18:11
lbragstad#link https://releases.openstack.org/queens/schedule.html18:11
*** oidgar has quit IRC18:11
lbragstadwe have four big efforts underway18:11
lbragstadunified limits18:11
lbragstad#link https://review.openstack.org/#/q/topic:bp/unified-limits+status:open18:12
lbragstadapplication credentials #link https://review.openstack.org/#/q/status:open+project:openstack/keystone+branch:master+topic:bp/application-credentials18:12
lbragstadtons of work to reorganize our documentation18:12
lbragstad#link https://review.openstack.org/#/q/status:open+project:openstack/keystone+branch:master+topic:api-ref-reorganization18:12
lbragstadand system scope work18:12
lbragstad#link https://review.openstack.org/#/q/status:open+project:openstack/keystone+branch:master+topic:bp/system-scope18:12
cmurphydoes the docs reorg have the same feature freeze deadline?18:12
*** ekcs has joined #openstack-meeting18:12
lbragstadit doesn't, we can land that into the RC period18:13
lbragstadgood point cmurphy18:13
cmurphyi guess it's mostly all done/easy reviews anyway18:13
lbragstadi did add it to the list because of the volume of patches it generates18:13
lbragstadthe reorganization is the first box in this epic https://trello.com/c/5WCMxrLg/12-docs-goals-organization18:14
lbragstadSuramya has been doing a good job tending to review feedback18:14
*** jamesmcarthur has quit IRC18:14
*** jamesmcarthur has joined #openstack-meeting18:14
lbragstadand thanks to samueldmq for the reorganization template18:15
samueldmqglad it helped!18:15
lbragstad#topic announcement: office hours bugs18:15
*** openstack changes topic to "announcement: office hours bugs (Meeting topic: keystone)"18:15
lbragstadi went through a bunch of bugs and reworked the office-hours list18:16
lbragstad#link https://goo.gl/PTd8bp18:16
lbragstadfeel free to check ^ out if you're interested in picking something up for office hours18:16
lbragstad#topic questions about scope_types18:16
*** openstack changes topic to "questions about scope_types (Meeting topic: keystone)"18:16
*** yamahata has joined #openstack-meeting18:16
*** jkilpatr has quit IRC18:17
lbragstadalright - if you haven't noticed, i've been spamming our review queue with a slew of patches that add scope_types to each keystone resource18:17
lbragstadthis is essentially the other half of the equation for system-scope18:17
lbragstadbut as i worked through them, i started coming up with more questions and fixmes18:18
lbragstad#link https://review.openstack.org/#/q/status:open+project:openstack/keystone+branch:master+topic:add-scope-types18:18
lbragstad^ reviews18:18
lbragstad#link https://review.openstack.org/#/c/526159/3/keystone/common/policies/project.py@3218:18
lbragstad^ that is an example of a question i have concerning both scopes18:18
* lbragstad gives folks a minute to read18:18
*** jamesmcarthur has quit IRC18:19
*** julim has joined #openstack-meeting18:19
cmurphyoof hard questions18:20
lbragstadwhich is why i'd like to discuss/get advice from you smart people :)18:20
lbragstadideally - keystone is going to have to start doing "scope" checks of its own18:20
lbragstadmost likely in code18:20
cmurphywhat does "relevant to that project's position in the hierarchy" mean?18:21
lbragstadcmurphy: for example; if A is a project and has a child B which has a child C18:21
lbragstadA -> B -> C18:21
lbragstadand you're an administrator on project B18:22
*** jamesmcarthur has joined #openstack-meeting18:22
lbragstadwhat should happen when you call identity:list_projects with a token scoped to project B?18:22
gagehugowould you just see B and C?18:22
cmurphyI don't think you can assume the user has a role assignment on C18:22
lbragstad^ that begs another question, but you'll have to check for role inheritance18:23
lbragstadif we don't assume a user who is admin on project B has a role on project C, then does that make identity:list_projects a system-only policy?18:24
*** jamesmcarthur has quit IRC18:25
*** jamesmcarthur has joined #openstack-meeting18:25
cmurphyI think it is18:25
cmurphyif you want just your own projects you use /v3/auth/projects18:26
lbragstadif you're a member of a project though, should you be able to get a project?18:26
lbragstadthat's true18:26
lbragstadso - we have a similar relationship elsewhere18:26
lbragstadand that is the service catalog18:26
lbragstad(the /v3/endpoint API is system-only but users can still get service catalog information via their token or GET /auth/catalog)18:27
*** Leo_m has quit IRC18:27
*** Leo_m has joined #openstack-meeting18:28
cmurphyyou can get /v3/projects/{project_id} on your own project18:28
cmurphyyou just can't get /v3/projects?project_id={project_id}18:28
lbragstadso scope_types for GET /v3/projects/{project_id} is going to have to ['system', 'project']18:29
gagehugothat makes sense18:29
*** jamesmcarthur has quit IRC18:30
lbragstadif we want to refactor scope checks out of the policy definition (or check_str) and into code, then we'll have to encode that check into keystone18:30
*** esberglu has quit IRC18:31
*** esberglu has joined #openstack-meeting18:31
gagehugoso keystone would be responsible for that check then?18:32
*** Apoorva has joined #openstack-meeting18:32
kmallocfwiw, back to the would you see b+c bit, we never claimed heirarchy is private info18:32
kmallocdon't try and make it so.18:32
kmallocit is a big big big headache to do so18:32
*** Leo_m has quit IRC18:32
kmallocin the a->b->c, you'd see a,b,c18:33
kmallocit just has to be the way it works the current implementation and behaviors18:33
lbragstadif you're an admin on A?18:33
kmallocif you're an admin on b, you still see it18:33
kmallocif you're and admin on a you see everything below you18:33
kmallocyou can't obscure that without breaking api behavior18:34
gagehugoso if you're admin on B you'll see A? or am I misreading this18:34
kmallocwe at one point planned to implement it, but that was set aside due to the volume of headaches and issues with name being  unique within a tree, etc18:34
kmallocgagehugo: the hierarchy above/below you is not priviledged info18:34
kmallocso you would see A.18:35
gagehugoah ok18:35
kmallocand C18:35
*** Tom-Tom has joined #openstack-meeting18:35
lbragstadso - what about identity:get_project18:36
*** armax has joined #openstack-meeting18:36
lbragstadthe policy just above that one18:36
kmalloci think we have a mis-match in behaviors between list and get18:36
kmalloclist has historically been admin18:36
lbragstadif you call it with a system-scoped token and you're an admin, you can get any project18:37
lbragstadwhich makes sense18:37
kmallocthat being said, i'd be ok with anyone in the hierarchy being avble to get any project within it (as a project/domain admin)18:37
lbragstadbut if you're a member of a project and call it with a project-scoped token, what happens?18:37
kmallocsystem scope is *anything*18:37
kmallocand everything18:37
kmallocif its in your hierarchy, it should work, there is no reason to treat the project structure as priv data if it's above or directly below you.18:38
*** jkilpatr has joined #openstack-meeting18:38
kmallocmostly likely if you're a project admin anything below you is game for you to see anyway18:38
kmallocand the way the heierarchy works for auth etc, it is important for folks to know where in the tree they are18:38
kmalloci'd let get_project fly in those casesd.18:38
lbragstadshould a user be able to get /v3/projects/{project_id} if they aren't a member of that project?18:39
lbragstador don't have a role assignment on it?18:39
kmallocif it is above/below the project they're in.18:39
lbragstadwhat if we're talking about flat structures18:39
kmallocpeers are not covered18:40
kmallocso, flat is no.18:40
kmallocbasically a->b a->q18:40
kmallocb can't see q18:40
kmallocq cant get b18:40
lbragstad(all of this is going to have to be coded into keystone somewhere)18:40
kmallocbut a can get ba and q18:40
lbragstadso getting back to my original question, where do we want this type of stuff to live18:40
kmallocyou're going to have to add the logic into the resource manager18:41
kmallocor we'd break behavior(s)18:41
kmallocbecause mving to systemscope, you still have to support the current mode of operations18:41
lbragstadbecause it's business logic?18:41
kmallocand because it's business logic18:41
lbragstadok - that makes sense18:41
kmallocit's not driver and it's not REQ<->SerializedOutput18:42
lbragstadso - that means we're going to need some level of request information in the manager18:42
lbragstadand the policy operation, too18:42
kmallocwhich is doable via the threadlocal request data18:42
lbragstadso the managers can reason about it, right?18:42
kmallocor should be if we're using oslo.context correctly18:42
kmallocthis is why it's important to break apart the @protected decorator18:43
lbragstadand do policy enforcement inline18:43
kmallocbecause there are more Policy Enforcement Points because of the system scope and added business logic18:43
kmallocyeah, you need to do some of this in-line. the callback system, etc wont cover it18:44
*** slaweq has joined #openstack-meeting18:44
kmallocbut managers can/should be able to look at thread local data18:44
lbragstadright - which either bloats the @protected decorator or creates more business logic in the manager18:44
kmallocnoooow. we also need to make sure to handle cases where thread local data isn't populated18:44
kmallocaka, keystone-manage bootstrap18:44
kmallocbut in 99% of the cases context should be populated from a request18:45
*** dprince has quit IRC18:45
*** gouthamr has quit IRC18:46
*** slaweq has quit IRC18:46
kmallocit's a simple edge cased, and the thread local could be populated by keystone-manage bootstrap with something like is_bootstrap=True18:46
kmallocso we can dodge it w/o breaking the security model by just passing if context isn't populated sanely18:46
lbragstadso - i imagine most reviews in the scope_types topic to generate some discussion like this18:47
lbragstad(i've highlighted the bits i see with FIXMEs)18:47
lbragstadbut this is going to be a pretty significant amount of work18:47
lbragstadthoughts on how we should track it?18:48
lbragstadbug per FIXME? bug per a resource's FIXMEs?18:48
kmalloceither works18:48
*** irenab has quit IRC18:48
kmallocas long as we're tracking it18:48
kmalloci like fewer bugs, but wouldn't argue for per fixme18:48
*** oanson has quit IRC18:48
*** gouthamr has joined #openstack-meeting18:48
kmallocargue against*18:49
lbragstadso something like "correct system scope and project scope for the project resource":18:49
lbragstadand the fix for it will correct all FIXMEs in https://review.openstack.org/#/c/526159/3/keystone/common/policies/project.py18:49
*** rderose has quit IRC18:50
*** felipemonteiro has joined #openstack-meeting18:51
kmalloci like that18:52
lbragstadok - so it sounds like we can discuss the intended behavior of the each resource in review18:53
lbragstaddocument the outcome in FIXMEs18:53
lbragstadand then i'll open bugs when they merge pointing to the FIXMEs so that we can track the work18:53
*** Shrews has joined #openstack-meeting18:54
lbragstadi think this gives me enough to get started18:54
lbragstad#topic open discussion18:54
*** openstack changes topic to "open discussion (Meeting topic: keystone)"18:54
lbragstadfloor is open if anyone has anything18:55
kmallocyeah i have a question18:56
kmallocwhat if you get scared half to death twice?18:56
gagehugowould you be scared to 3/4th death then total?18:56
lbragstadkmalloc: you're not factoring in health regen18:56
kmallocok ok...18:57
lbragstadbecause gagehugo's factors that in18:57
*** amodi has quit IRC18:57
kmallocif one syncronised swimmer drowns... do they all follow in suit?18:57
*** sshnaidm has joined #openstack-meeting18:57
kmallocok ok i'm done18:57
lbragstadalright - time to refill your beverage before office hours!18:58
lbragstadthanks for the time folks18:58
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"18:58
openstackMeeting ended Tue Jan  9 18:58:26 2018 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)18:58
openstackMinutes:        http://eavesdrop.openstack.org/meetings/keystone/2018/keystone.2018-01-09-18.00.html18:58
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/keystone/2018/keystone.2018-01-09-18.00.txt18:58
openstackLog:            http://eavesdrop.openstack.org/meetings/keystone/2018/keystone.2018-01-09-18.00.log.html18:58
*** spilla has left #openstack-meeting18:58
*** jdennis has left #openstack-meeting18:59
clarkbHello infra, anyone here for the meeting?18:59
clarkbianw: good morning!19:00
clarkband welcome back19:00
clarkb#startmeeting infra19:01
openstackMeeting started Tue Jan  9 19:01:15 2018 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.19:01
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.19:01
*** openstack changes topic to " (Meeting topic: infra)"19:01
openstackThe meeting name has been set to 'infra'19:01
clarkb#link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting19:01
clarkb#topic Announcements19:01
*** openstack changes topic to "Announcements (Meeting topic: infra)"19:01
* mordred waves19:01
clarkbI will be traveling for meetings on January 23rd so will not be able to run or attend the infra meeting19:02
fungii'm back, just in the nick of time19:02
clarkbthat is the meeting after next19:02
clarkbI think fungi is in this same boat too.19:02
mordredfungi: I thought you were back in the nick of fungi19:02
fungii'll be on the same trip, so can't volunteer to chair19:02
clarkbwould be great if someone else was able to run that assumed we aren't all traveling at the same time19:02
clarkblet me know if you are interested19:02
fungipabelanger and ianw have chaired meetings in my stead in the past, fwiw19:03
ianwhappy to if nobody else wants to19:03
pabelangeryah, happy to help, unless somebody else wants to try :)19:03
fungiwho ever _wants_ to chair a meeting? ;)19:03
clarkb#topic Actions from last meeting19:03
*** openstack changes topic to "Actions from last meeting (Meeting topic: infra)"19:03
clarkb#link http://eavesdrop.openstack.org/meetings/infra/2018/infra.2018-01-02-19.06.txt minutes from last meeting19:04
clarkbthere were none \o/19:04
clarkb#topic Specs approval19:04
*** openstack changes topic to "Specs approval (Meeting topic: infra)"19:04
clarkbI don't think there are any specs up for approval right now. But I do still intend on cleaning up our specs list after fungi's jenkins vote removal bubbled a bunch of old ones to the front of the list19:04
* fungi is happy to tale the blame for people noticing old changes in review19:05
corvusi didn't put it on the agenda, but i'd like to issue a request for collaboration on https://review.openstack.org/53124219:05
clarkbAlso worth mentioning there is an irc bot improvement spec that we should be reviewing which is related to handling the irc spam freenode has been seeing recently19:05
corvusclarkb: ya that :)19:05
fungier, s/tale/take/19:05
clarkbit would be helpful if you could rebase your old specs if they are still valid and abandon them otherwise19:05
clarkbI will be going through and doing this for specs whose owners are less active now19:06
clarkb#link https://review.openstack.org/531242 irc bot improvement spec19:06
pabelangerYah, I have a spec or 2 to rebase19:06
corvusplease get me feedback on that soon, and i'll put it on the agenda for approval19:06
fungithanks for the reminder on the irc spec19:06
dmsimard\o sorry I'm late19:07
clarkb#topic Priority Efforts19:07
*** openstack changes topic to "Priority Efforts (Meeting topic: infra)"19:07
clarkb#topic Storyboard19:07
*** openstack changes topic to "Storyboard (Meeting topic: infra)"19:07
dmsimardThere is also the ARA spec, I was waiting on one more review or two before spinning up a new draft: https://review.openstack.org/#/c/527500/19:07
openstackRemoving item from minutes: #topic Storyboard19:08
openstackRemoving item from minutes: #topic Priority Efforts19:08
clarkb#link https://review.openstack.org/#/c/527500/ ARA dashboard spec19:08
clarkbdmsimard: you might want to rebase it and push a new patchset anyways as it is merge conflicted which a lot of people filter out19:08
dmsimardI can do that.19:09
clarkb#topic Priority Efforts19:09
*** openstack changes topic to "Priority Efforts (Meeting topic: infra)"19:09
clarkb#topic Storyboard19:09
*** openstack changes topic to "Storyboard (Meeting topic: infra)"19:09
clarkbWe don't often have storyboard topics in this meeting but I wanted to mention that I have nominated diablo_rojo as storyboard core19:10
clarkbdiablo_rojo has been actively reviewing and shepherding storyboard changes recently and want ot encourage more of that.19:10
*** david-lyle has joined #openstack-meeting19:10
clarkb(this in addition to helping projects work through the migration to storyboard)19:10
clarkbthere is an email thread on the infra list if you want to chime in19:10
clarkb#topic Zuul v319:11
*** openstack changes topic to "Zuul v3 (Meeting topic: infra)"19:11
corvusSee my email about mailing lists19:11
clarkbI've almost got the zuulv3-issues etherpad cleared out by either tracking down things as fixed, fixing them, or creating bugs.19:11
corvus#link zuul mailing list email http://lists.openstack.org/pipermail/openstack-infra/2018-January/005758.html19:11
fungizuul mailing lists double-plus good19:12
corvusclarkb: thanks!19:12
corvusnote that clarkb added two 3.0 release blocking bugs to storyboard19:12
corvusso there's work available there19:12
diablo_rojothanks clarkb :)19:13
fungis puppet changes are looking good19:13
corvusfungi: oh great!19:13
clarkbfungi: is there a topic or other gerrit query we can use to find those?19:13
* fungi apologizes for having his return key adjacent to his apostrophe)19:13
corvusfungi: i fixed that, it's next to my19:13
*** markvoelker has joined #openstack-meeting19:13
clarkb(they are on my list of things to followup on to make sure we are progressing towards a merged feature branch)19:13
*** markvoelker has quit IRC19:13
mordredfyi for folks, we're working on turning on testing ansible PRs from their github repo - https://review.openstack.org/#/q/topic:turn-on-ansible is the stack ATM19:14
fungiclarkb: good question. i simply found them looking in openstack-infra/puppet-openstackci open changes19:14
corvusfungi: are they just ready for a +3 or are we going to need to be careful?19:14
clarkbfungi: thanks19:14
corvus#link puppet-openstack changes https://review.openstack.org/52395119:14
fungii think they're probably ready for +3 since they default to zuul 2.x code paths, but eatra reviews are of course strongly recommended19:14
fungier, extra19:14
funginot eatra19:15
fungibut that should totally be a word19:15
corvusmordred: that's so cool.  i'm hoping to follow that up shortly with cross-source deps which will be nice to have in place before we make any big announcements (that way we can show people a change in ansible depending on a change in shade!)19:15
clarkbmordred: corvus will/are we reporting to github yet?19:15
clarkbor is it currently just commenting on gerrit changes?19:15
mordredwe are not yet reporting to github but will be by the end of that stack19:16
* fungi wants a change in nodepool depending on a change in ansible depending on a change in shade19:16
clarkband ansible knows that this is happening and won't hunt us down?19:16
mordredcurrently iterating on the test job to make sure it's actually, you know, testing the right thing19:16
fungii recently had a change in pbr which depended on a change in setuptools19:16
fungibut alas, no zuul support for me19:16
corvusso we'll start reporting... poke at things a bit, make sure it looks good, add in cross-source deps, test that out, then show people how cool it all is19:16
mordredclarkb: yup!19:16
*** Leo_m has joined #openstack-meeting19:16
pabelangerexciting times19:17
fungiand not even in the ancient chinese proverb sense19:17
clarkbmmedvede: other than reviews is there anything we need to be doing or aware of in updating puppet-openstackci?19:17
fungiclarkb: luckily, we have robyn to prevent ansible people from _actually_ hunting us down19:18
corvusincidentally, the implementation path i've chosen for cross-source deps should let us leave error messages on changes when there are dependency cycles.  that's a long-requested feature.19:18
mmedvedeclarkb: well, it is more work than I initially thought it would be :)19:18
fungicorvus: i missed that tidbit... amazing!19:18
corvusfungi: it's not there yet -- will probably happen in a followup.  but i'm laying groundwork.19:19
mordredcorvus: \o/19:19
clarkbcorvus: I filed two bugs related to that that are not marked 3.0 just 319:19
fungicolor me excited19:19
clarkbcorvus: one is lack of reporting the other is detecting cycles when there isn't a proper cycle19:19
clarkbalright any other zuul v3 related items?19:20
mmedvedeclarkb: I am restarting work on puppet-openstackci, would follow up outside of meeting with you if you got a moment19:21
clarkbmmedvede: sure19:21
clarkb#topic General Topics19:21
*** openstack changes topic to "General Topics (Meeting topic: infra)"19:21
corvusit's the 9th is ubuntu updated?19:21
fungii've melted down at least a few spectres already19:21
clarkbcorvus: they have not19:21
clarkbcorvus: there are kernels in testing though, but they haven't listed one for trusty unless you use the xenial kernel on trusty :/19:22
fungidebian was pretty johnny-on-the-spot, but it's hard to beat openbsd getting patches in more than a decade ago19:22
clarkbfungi: debian still doesn't have kernel for jessie for me :/19:22
corvus#link ubuntu meltdown/spectre https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/SpectreAndMeltdown19:22
*** jessegler has quit IRC19:22
fungiclarkb: it's called 'apt dist-upgrade' to stretch :/19:22
clarkbfungi: ya I should probably upgrade the entire OS19:23
fungilts security is not handled by the debian security team, alas19:23
corvusmeanwhile, the meltdown POC was published today...19:23
*** markvoelker has joined #openstack-meeting19:23
dmsimardIn the meantime, I've (for another purpose) made what should be a cross-distro playbook that allows to see what hosts are patched and which aren't19:23
clarkbanyways tl;dr for us is this:19:23
mordredfungi: so we should be migrating to openbsd, yeah?19:23
dmsimardI'll clean it up and we can use it to keep an inventory of which host needs to be patched still19:23
fungimordred: i'm running a bunch of it at home since long ago19:23
*** sambetts is now known as sambetts|afk19:23
clarkbOur CentOS machines (git*.openstack.org) are all patched and running with page table isolation. We are waiting for ubuntu to publish kernel updates for the rest of our control plane.19:23
corvusdmsimard: sounds cool, thx19:23
fungiand i gather you can run openbsd virtual machines as long as you don't need pv guests, but i still need to try that myself19:24
clarkbThe test node VMs will automagically pick up new kernels as part of daily rebuilds19:24
clarkbRackspace as a Xen user is for the most part not affected by meltdown19:24
clarkbso I think concern about side channel attacks attacking our control plane is relatively minor19:24
dmsimardWe don't upload any secrets from the executor to the test nodes, do we ?19:24
fungitheir hypervisor layer not particularly impacted still doesn't change that we need to upgrade our guest layer kernels though19:24
dmsimard(since we don't control the state of each nodepool provider)19:25
pabelangerdmsimard: yes, AFS credentials19:25
clarkbfungi: correct19:25
*** timcline has joined #openstack-meeting19:25
ianwclarkb: i'm assuming etherpad/ethercalc are pretty much where we left them before break?19:25
clarkbalso everyone should use firefox and make sure it is up to date as it has meltdown mitigation built in. Chrom* will get updatedon the 23rd19:25
clarkbianw: ya19:25
ianwok, should get back to that, as part of this too19:26
corvuspabelanger: what job uploads afs creds to nodepool nodes?19:26
dmsimardpabelanger: we mount the AFS on the test nodes ?19:26
dmsimardactually, creds should only be required for writing I would guess ? since read only is unauthenticated19:26
pabelangercorvus: dmsimard our wheel builders19:26
pabelangerlet me get jobs19:26
dmsimardpabelanger: the contents aren't recovered by the executor like logs or other artifacts would be ?19:27
* dmsimard is totally unfamiliar with the wheel build process19:27
fungithe bigger concern is whether we put decrypted secrets on systems where we run untrusted code. to the extent that our (config-core) reviews are thorough, that shouldn't be the case19:27
clarkbas soon as Ubuntu does publish kernels (which is supposed to be today), it would be much appreciated if we could have a semi organized patching "sprint"19:28
corvusthat's not running proposed code though, that's a job we control centrally.  however, it may be running code in python packages that build wheels.  but that's not a new attact vector.19:28
dmsimardyeah, I thought secrets were meant to stay on the executor -- but that tells me there's nothing preventing a project from sending the secret(s) to the test nodes19:28
pabelangerpublish-wheel-mirror-centos-7 for example will use secrets on nodepool node19:28
clarkbmaybe we can all hop in the incident channel and start working through it and partition out servers19:28
corvusdmsimard: secrets are meant to be used however needed19:28
clarkbcorvus: +119:28
clarkber +1 to not new attack vector19:29
fungiagreed, ping infra-root in #-incident as soon as someone notices ubuntu trusty and/or xenial kernel updates addressing meltdown19:29
fungii'm happy to pitch in on upgrading/rebooting if i'm at a computer19:29
clarkbI will likely end up making that my priority once we have packages19:30
corvusas with anything, each piece of privileged information needs to be thought about and risk evaluated.  there's no hard rule we can make about secrets.19:30
dmsimardcorvus: what I am saying is that if secrets are used in test nodes, then they are vulnerable if the test node is running in an unpatched nodepool cloud provider19:30
corvusdmsimard: i agree.19:30
*** apetrich has joined #openstack-meeting19:30
clarkbyup, but those creds have alway sbeen vulnerable to attacks through the package builds19:30
fungidmsimard: to the extent that the jobs/code running there aren't approved by the same people who know the secret19:30
clarkbfungi: no, just running the job opens it to attack19:31
clarkbor maybe thats what you mean by approved19:31
fungiattack from...19:31
dmsimardfungi: meltdown19:31
dmsimardin $vulnerablecloud19:31
clarkbvia other cloud users19:31
fungiusing secrets in jobs which run proposed patches is already a huge no-no19:31
clarkbfungi: the attack vector is ovh customers19:31
dmsimardregardless if it's proposed, post, or release19:31
fungiohm, in providers who haven't secured their hypervisor hosts? yes indeed19:31
clarkbcompeltely unrelated to us19:32
clarkbso this increases the risk of exposure of the credentials but I don't think it does so greatly19:32
corvusso that job is now vulnerable to evil package maintainers, and other customers on unpatched nodepool cloud providers19:32
*** priteau has quit IRC19:32
corvus(whereas it used to only be the first -- a risk we deemed acceptable)19:32
dmsimardIt's probably not that big of a risk, but it's worth keeping in mind and generally try to keep secrets off of test nodes where possible19:33
clarkbdmsimard: yup19:33
*** harlowja has joined #openstack-meeting19:33
corvusdmsimard: i'm not going to agree with we should try to keep secrets off test nodes where possible19:33
corvusi'm only going to go as far as we need to keep that risk in mind when we evaluate where we use secrets19:34
fungiyep, if a job using a secret doesn't need to run arbitrary code and can leverage the bwrap environment on the executor, that's best19:34
clarkbcorvus: I think it is part of the risk assessment, and then we decide if we can accept possible insecure_cpu/cloud bugs19:34
clarkbcorvus: ++19:34
dmsimardGiven the choice of using the secret on the executor or on the test node, I'd keep it on the executor if it doesn't mean jumping through 10 hoops19:34
pabelangerwhat about jobs that are using docker credentials, should we post something to warn about exposure? Or hope they understand it19:34
fungibut secrets on executors are only actually safe if the other rackspace guests on the same hosts aren't 64-bit pv flavor19:34
corvus(and to be really specific, i would like to eventually put a secrets on the test node so that we don't have to route logs through the executor)19:34
fungiso, it's still up to the provider to provide proper security19:35
dmsimardcorvus: an approach we have for RDO is that we have the logserver pull the logs instead of having the test nodes push the logs19:35
corvusso i'm not going to climb on board any rule that precludes that.  i'm happy to have a discussion about the risks when we do.19:35
dmsimardi.e, the log server has a ssh key that grants access to the test nodes19:35
pabelangerWe talked about that before, but didn't implement it for some reason19:35
corvuslet's not turn this into a discussion about that.  i don't have a proposal.19:36
clarkbpabelanger: it may be worth a note, maybe when the dust settles and we can speak with a bit more concrete info (eg find out when $clouds say they are patched, after we are patched and so on)19:36
corvusi merely cite it as an example of why i disagree with the blanket statement.19:36
fungiyeah, this is not the venue for redesigning test log publishing19:36
clarkbpabelanger: right now I don't think we have enough concrete info other than "please don't panic but be aware of this"19:36
pabelangerclarkb: agree19:36
clarkbother than the general situation update adn call for help once we have packages I'19:37
clarkbm not sure there is much more to talk about here19:37
clarkb#topic Project Renames19:37
*** openstack changes topic to "Project Renames (Meeting topic: infra)"19:37
dmsimardclarkb: you missed a topic19:37
fungior he's reordering them19:38
clarkbdmsimard: refresh confirms19:38
clarkbno I had missed it19:38
openstackRemoving item from minutes: #topic Project Renames19:38
clarkbwas added late19:38
dmsimardadded it shortly before meeting, sorry :)19:38
clarkbsshnaidm: are you around?19:38
sshnaidmclarkb, yep19:38
clarkb"Allow jobs to send data to graphite.openstack.org"19:39
sshnaidmthe question I wanted to ask - is it possible to do in general?19:39
sshnaidmI know now it's firewalled19:39
corvusi'd like to make sure everyone is aware of and has read this spec:19:39
corvus#link qa counter spec https://specs.openstack.org/openstack/qa-specs/specs/devstack/counter-inspection.html19:39
clarkbsshnaidm: the spec corvus linked has a specific way to accomplish this despite the firewall19:40
dmsimarda target of mitaka-2, nice19:40
clarkbI think that approach is still valid19:40
corvusit is partially implemented, and probably needs a little bit of updating because it predates zuulv3 and the changes we made to subunit processing, but the approach is still largely viable and is important prior work19:40
sshnaidmclarkb, yeah, I ready it briefly19:40
*** harlowja_ has joined #openstack-meeting19:41
dmsimardfrom skimming the spec, it's the same approach as openstack-health and logstash-workers ?19:41
sshnaidmclarkb, it seems close, but not really what is done in jobs19:41
clarkbthe appraoch should also be generally applicable to all jobs19:41
clarkbsshnaidm: meaning it doesn't meet your usecase?19:41
sshnaidmclarkb, maybe I miss something in docs, but what I need actually it's to send some my custom statistics to some statistics server (graphite/influx/whatever)19:42
clarkbsshnaidm: yes that is what that spec is for19:42
sshnaidmclarkb, during the job run19:42
fungii haven't looked closely at the spec, but is there no sane way to ship the stats back to the executor and let it emit them?19:42
dmsimardsshnaidm: why would it be a problem if you send the data after the job is complete ?19:42
clarkbsshnaidm: jobs will collect the data then as part of the log publishing and processing we write the info to graphite19:42
*** markvoelker has quit IRC19:42
sshnaidmclarkb, ok, just maybe confused by subunit, mysql and other stuff there..19:43
clarkbsshnaidm: it is just pointing out that subunit2sql and logstash already use a similar data flow19:43
sshnaidmdmsimard, post playbook is ok, of course19:43
*** markvoelker has joined #openstack-meeting19:43
clarkbsshnaidm: and SpamapS was explicitly interested in mysql stats19:43
*** harlowja has quit IRC19:43
fungiahh, so basically what i was asking. i should definitely look over the spec19:43
sshnaidmclarkb, ok, so we need just to wait afaiu19:43
clarkbProbably the best approach here is to work to implement that spec and update it where it needs to be updated19:43
dmsimardsshnaidm: yeah, the approach in how we currently ask logstash and openstack-health to index data involves a post playbook19:43
clarkbsshnaidm: I don't think anyone is actively working it.19:44
corvussshnaidm: oh, no.  SpamapS is not working on that spec.  if you want this to happen, someone will need to work on it.19:44
fungiwaiting isn't likely to get you much unless there is someone else actively implementing19:44
ianwsshnaidm: you might be interested in something like https://review.openstack.org/#/c/434625/ ; that is an example of adding coverage output to the subunit stream19:44
sshnaidmcorvus, clarkb ok.. it it tracked anywhere?19:44
corvussshnaidm: i'd recommend talking to SpamapS to find out what's completed and what maybe should be changed.19:44
clarkbsshnaidm: you'd have to ask the qa team probably, or SpamapS19:44
SpamapSIndeed, I don't, unfortunately, have bandwidth to finish the perf stats spec. :-/19:45
*** timcline has quit IRC19:45
SpamapSHappy to advise anyone who wants to pick it up.19:45
sshnaidmok, thanks, will try to poke there19:45
clarkbsshnaidm: ^ does that work as a starting point?19:45
dmsimardI'm sure a lot of projects could benefit from that spec -- I've seen kolla implement a self-contained collectd recently19:45
dmsimard(for the purpose of tracking gate job resource usage and performance data)19:45
SpamapSI wanted to do that too. :)19:46
*** amodi has joined #openstack-meeting19:46
dmsimardsshnaidm: I can help point you in the right direction19:46
clarkbcool sounds like we have a rough starting point/plan19:46
sshnaidmdmsimard, great19:46
corvusdmsimard: wait -- they implemented it just for kolla?19:46
clarkbcorvus: I think its more that kolla deployed clouds need that tooling anyways19:46
clarkbcorvus: so they implemented it for their users (and may also take advantage of it in jobs)19:47
dmsimardcorvus: it's a collectd implementation that generates a html page at the end and the html page is stored in the logs19:47
sshnaidmdmsimard, well, in tripleo we also run dstat that collect data.. just not sending it anywhere19:47
corvusdmsimard: oh that makes sense... i thought you were saying they stood up a private collectd19:47
ianwsshnaidm: https://review.openstack.org/#/c/436768/5/specs/coverage-tracking.rst also has a bunch of info on subunit streams and how they plug in19:47
ianwto openstack-health, etc19:47
corvusdmsimard: i agree, that's a great first step that can feed into what sshnaidm wants to accomplish and make things better for everyone19:47
sshnaidmianw, thanks, will look19:47
clarkb#topic Project Renames19:48
*** openstack changes topic to "Project Renames (Meeting topic: infra)"19:48
sshnaidmanother question I have is - how can I inject credentials to some 3d-party server in jobs? for example to authenticate with some http api in 3party to send data there19:48
*** markvoelker has quit IRC19:48
clarkbI don't want to derail the previous discussion but we are running out of time and would like to talk about project renames before the meeting ends19:48
corvussshnaidm: why not send the data to a first party server?19:48
sshnaidmok, let's talk later about it19:48
dmsimardsshnaidm: it's a question for #openstack-infra, we can help there19:48
sshnaidmgreat, thanks19:49
fungishort answer...19:49
fungi#link https://docs.openstack.org/infra/zuul/feature/zuulv3/user/encryption.html#encryption Zuul v3 secrets19:49
clarkbfungi: mordred ianw I know everyone has been afk and snowed in or under a heat wave, but curious if any progress has been made on sorting out a project rename plan19:49
fungiclarkb: AJaeger has been asking that we discuss the plan for pypi registration for new projects. is that still in need of discussion?19:49
*** jdandrea has joined #openstack-meeting19:50
clarkbfungi: I thought there was rough consensus on the plan there to register empty projects? I may have missed some subtle details that need ironing out19:50
fungii'm not aware of any progress on redesigning the renames process yet, unfortunately19:50
mordredclarkb: the project name not needed patch landed to zuul, so that part is good, but we've still got required-projects entries to consider19:50
clarkbmordred: oh cool19:51
corvusoh, and i guess we've restarted zuul since then19:51
corvusi'll start pushing up patches to remove project names19:51
clarkbsounds like good progress for everyone being on holiday19:51
fungimordred: clarkb: also we had talked about switching to implicit required jobs where desired, and dropping teh system-required project-template?19:51
corvus#action corvus patchbomb project name removals from zuul.yaml19:51
*** trandles has joined #openstack-meeting19:51
mordredclarkb, fungi: on pypi - I've got patches up to verify that a project either doesn't exist in pypi or exists but has openstackci properly as the owner ...19:51
fungimordred: cool, saw those, still need to find a moment to review, but seems like a solution is in progress anyway19:52
clarkbfor require-projects maybe we can have project aliases19:52
clarkbwhere we map old to new and zuul will attempt to use both19:52
clarkbthen after a rename is compelte we can remove the aliases19:52
corvusor we could remove the jobs, rename, then readd.19:52
*** gouthamr has quit IRC19:53
corvusthough really, required-projects failures should probably soft-fail -- they should not fail at startup, but should still error on proposed reconfigurations.19:54
mordredclarkb, fungi: https://review.openstack.org/#/q/topic:check-pypi ... it does not contain anything to actually auto-register yet (figure let's get verification working solidly first)19:54
*** hearnsj has joined #openstack-meeting19:54
corvussince we should protect zuul from external renames happening, but still prevent folks from botching a config change19:54
clarkbya external renames would be annoying19:54
fungisounds right19:54
clarkbespecuially if you couldn't start zuul as a result19:54
corvusi think that would cover this case?  the job would just be 'broken' till someone updated its' required projects...19:54
clarkbcorvus: ya I think it would. Job would be broken but zuul would work for everyone else19:55
*** harlowja_ has quit IRC19:55
fungilast thing we want is for zuul to stop being able to start due to a change in external data19:55
clarkband renames already tend to put projects in a half working state anyways19:55
corvusof course, if it's an important job, fallout could be large, but hey.19:55
*** harlowja has joined #openstack-meeting19:55
clarkb#topic open discussion19:56
*** openstack changes topic to "open discussion (Meeting topic: infra)"19:56
clarkbwe have a few minutes for whatever else we missed :)19:56
fungia wild Zara appears19:56
ZaraI'm late but congrats diablo_rojo!19:56
ZaraI arrived *right* after the storyboard topic and have been lurking haha19:56
corvusdiablo_rojo: ++ :)19:56
ZaraI'm very happy about the nomination :)19:56
diablo_rojoThanks :)19:56
diablo_rojoHappy to share the load ;)19:56
fungihopefully there are more new sb contributors on their way too19:57
fungiseems like it's gaining visibility19:57
fungiand mordred's been hacking on it some again19:57
* diablo_rojo writes a dozen 'You are eligible for migration' emails19:58
clarkboh! PTG is coming up19:58
*** rbudden has quit IRC19:58
fungikrotscheck has even been resurfacing on occasion19:58
pabelangerThere are a few patches up to add puppetlabs to AFS mirrors, if anybody else wants some hands on XP on AFS changes: https://review.openstack.org/#/q/topic:puppetlabs/mirrors19:58
clarkbI'll probably start soliciting rough planning thoughts for PTG once we get past meltdown19:58
pabelangerhappy to walk people thought it or will likely do it in a day or so19:59
clarkbpabelanger: you might want to check with frickler directly as the timezones don't make it easy to see pings19:59
clarkbI expect frickler will be interested19:59
clarkbok we are out of time. Thank you everyone20:00
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"20:00
openstackMeeting ended Tue Jan  9 20:00:08 2018 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)20:00
openstackMinutes:        http://eavesdrop.openstack.org/meetings/infra/2018/infra.2018-01-09-19.01.html20:00
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/infra/2018/infra.2018-01-09-19.01.txt20:00
openstackLog:            http://eavesdrop.openstack.org/meetings/infra/2018/infra.2018-01-09-19.01.log.html20:00
Zarathanks! :)20:00
fungithanks clarkb!20:00
*** rbudden has joined #openstack-meeting20:00
*** hearnsj has quit IRC20:01
*** hearnsj has joined #openstack-meeting20:02
*** gouthamr has joined #openstack-meeting20:02
*** amodi has quit IRC20:03
*** amodi has joined #openstack-meeting20:06
*** hearn_ has joined #openstack-meeting20:06
*** hearn_ has quit IRC20:06
*** hearnsj has quit IRC20:06
*** abrito has joined #openstack-meeting20:07
*** hearnsj has joined #openstack-meeting20:07
hearnsj#startmeeting scientific_sig20:11
openstackMeeting started Tue Jan  9 20:11:38 2018 UTC and is due to finish in 60 minutes.  The chair is hearnsj. Information about MeetBot at http://wiki.debian.org/MeetBot.20:11
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.20:11
*** openstack changes topic to " (Meeting topic: scientific_sig)"20:11
openstackThe meeting name has been set to 'scientific_sig'20:11
*** yamamoto has joined #openstack-meeting20:11
*** markstur_ has quit IRC20:13
hearnsj#endmeeting scientific_sig20:14
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"20:14
openstackMeeting ended Tue Jan  9 20:14:19 2018 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)20:14
openstackMinutes:        http://eavesdrop.openstack.org/meetings/scientific_sig/2018/scientific_sig.2018-01-09-20.11.html20:14
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/scientific_sig/2018/scientific_sig.2018-01-09-20.11.txt20:14
openstackLog:            http://eavesdrop.openstack.org/meetings/scientific_sig/2018/scientific_sig.2018-01-09-20.11.log.html20:14
*** markstur has joined #openstack-meeting20:14
*** Shrews has left #openstack-meeting20:14
*** dtrainor has joined #openstack-meeting20:15
*** dustins has joined #openstack-meeting20:15
*** slaweq has joined #openstack-meeting20:17
*** jamesmcarthur has joined #openstack-meeting20:18
*** ijw has quit IRC20:20
*** VW has quit IRC20:21
*** eharney has quit IRC20:24
*** hearnsj has quit IRC20:25
*** eharney has joined #openstack-meeting20:26
*** amodi has quit IRC20:26
*** markvoelker has joined #openstack-meeting20:33
*** dprince has joined #openstack-meeting20:37
*** cleong has quit IRC20:38
*** dbecker has quit IRC20:45
*** dbecker_ has joined #openstack-meeting20:45
*** yamamoto has quit IRC20:48
*** yamamoto has joined #openstack-meeting20:49
*** armax has quit IRC20:53
*** reedip has quit IRC20:53
*** yamamoto has quit IRC20:54
*** priteau has joined #openstack-meeting20:55
*** oneswig has joined #openstack-meeting20:55
*** belmoreira has joined #openstack-meeting20:57
*** ykatabam has joined #openstack-meeting20:58
oneswig#startmeeting scientific-sig21:00
openstackMeeting started Tue Jan  9 21:00:35 2018 UTC and is due to finish in 60 minutes.  The chair is oneswig. Information about MeetBot at http://wiki.debian.org/MeetBot.21:00
*** VW has joined #openstack-meeting21:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.21:00
*** openstack changes topic to " (Meeting topic: scientific-sig)"21:00
openstackThe meeting name has been set to 'scientific_sig'21:00
oneswigHello hello hello21:00
oneswig#link Agenda for today https://wiki.openstack.org/wiki/Scientific_SIG#IRC_Meeting_January_9th_201821:01
oneswigAndrey tells me he's stuck in a taxi somewhere in Delhi, hoping to join us shortly21:02
*** aembrito has joined #openstack-meeting21:02
*** powerd has joined #openstack-meeting21:03
oneswigI had a discussion with some colleagues from Cambridge University earlier.21:03
*** abrito has quit IRC21:03
oneswigThey have been benchmarking the effect of the spectre/meltdown fixes21:03
oneswigOn a lustre router, apparently there is a 40% hit!21:04
oneswigAre there other people testing their platforms?21:04
trandlesfilesystems/IO is bad post-patch21:05
trandlestoo much context switching :(21:05
oneswigHi Tim, so it seems - the worst case21:05
bolligwe saw 30% IO loss on VMs, 2% cpu loss. one sec and I’ll paste in my query earlier today on #scientific-wg21:05
trandlesHello Stig. David Daniel says hello.  Had a meeting with him this morning. :)21:05
oneswigPlease do21:05
*** reedip has joined #openstack-meeting21:05
oneswigDDD - fantastic!  You've made my day21:06
*** martial_ has joined #openstack-meeting21:06
belmoreirawe started testing as well. Our compute workloads don't seem affected in the initial benchmarks. I will have more info next week21:06
rbuddenwe’ve started baremetal testing, but I don’t have any result at the moment21:07
oneswigThere's been some discussion around whether there is an impact for RDMA, and in which modes of usage21:07
*** jmlowe has joined #openstack-meeting21:07
jmloweHi everybody21:08
oneswig#link Here's an intriguing coincidence form a couple of months ago https://www.fool.com/investing/2017/12/19/intels-ceo-just-sold-a-lot-of-stock.aspx21:08
martial_Hi Mike, Bob, Stig21:08
oneswigHi jmlowe rbudden belmoreira et al21:08
oneswigHey martial_21:08
oneswig#chair martial_21:08
openstackCurrent chairs: martial_ oneswig21:08
oneswigSo, it does appear that the worst case scenarios can be readily hit.21:09
belmoreirahumm... not a coincidence :)21:09
bolligWe have broadwell arch (compute: E5-2680v4, storage: E5-2630v4). Ceph Luminous, Openstack Newton, qemu+kvm virtualization. First, after patching our hypervisors we saw a 2% CPU perf-loss in HPL benchmark running inside an unpatched centos 6.5 VM, plus 30% I/O perf-loss in the FIO benchmark inside the same VM. No further loss from patching VMs. Finally, we patched the storage servers and again saw no further degredation. Better than I expected,21:10
bolligI’m curious if it plays out that way for others. We have NVMe in our storage, which might be amortizing the cost of I/O operations at the storage servers.21:10
bolligThe numbers were very close to redhat’s predictions21:11
oneswigbollig: I'd guess, if it's a performance penalty incurred on every context switch, then it'll be more painful for nvme than for other devices simply because they achieve more context switches, but the penalty is constant for each.  Perhaps21:12
*** makowals has quit IRC21:12
*** markvoelker has quit IRC21:13
*** markvoelker has joined #openstack-meeting21:13
oneswigWhat was the extent to which an unpatched guest could read data from the hypervisor?21:13
oneswigThis is one of those rare circumstances where bare metal looks like the security-conscious option21:15
*** archit has joined #openstack-meeting21:15
martial_(too used to slack, I want to +1 Stig's last comment)21:15
oneswigha, slack is too easy!21:16
oneswigOK, well interesting to hear people's experiences.  I'm sure it's just early days.21:17
jmloweI really want to know the definitive answer to that question as well, do I need to make sure my guests are patched or is qemu and hypervisor patching sufficient to ensure guests can't read more than their own memory21:17
*** markvoelker has quit IRC21:17
bolligthat I dont know. We’re rebuilding all of our base images, and looking for that same answer for existing VMs21:18
jmloweI had some linpac numbers from Jonathan Mills at NASA, worst case was %50 best case was %5, seemed to vary linearly with N21:18
oneswigjmlowe: if you find out, will you let us know - sure it'll be widely applicable to just about everyone in OpenStack21:19
jmlowewe are also patching all of our images as per ususal21:19
*** egallen_ has joined #openstack-meeting21:19
*** egallen has quit IRC21:20
*** egallen_ is now known as egallen21:20
oneswigAndrey is ready, or thereabouts.  He's sent ahead a presentation to share21:20
oneswig#link SGX and OpenStack https://drive.google.com/file/d/1wBXVrd9v8GjyreFLET5nW7IhROaOf7A6/view21:20
jmloweI have a more urgent need to patch, it seems that either my 2.1.26 i40e driver is leaking or it's the rhel/centos 3.10.0-693.5 kernel is leaking about 20GB/month, it's starting to trigger oom killer on my instances21:22
aembritoHi everyone, I was offline on a plane and had much less time than I expected, so please, consider it a first discussion21:22
aembritoI will then come back and give more details, including on how we are using it with OpenStack (Ironic, LXD, KVM, and Magnum+Kubernetes)21:23
*** aembrito is now known as abrito21:23
oneswigHi Andrey, thanks for joining us today21:24
oneswig#topic SGX on OpenStack21:24
*** openstack changes topic to "SGX on OpenStack (Meeting topic: scientific-sig)"21:24
oneswigIs this specific to Skylake?  I've seen previous articles on it that appear to date from 201321:25
abritoyes, it is to Skylake21:26
abritoprevious discussion was based on simulations21:26
oneswigHow much of a limitation is it that the code in the enclave can't make system calls?21:27
*** egallen_ has joined #openstack-meeting21:28
*** egallen has quit IRC21:28
*** egallen_ is now known as egallen21:28
abritothere are tools to help circumvent this21:30
oneswigabrito: Are there uses for this as protection for bare metal infrastructure from malicious users?21:30
abritofor example, SCONE is a tool that places one thread inside the enclave and another outside the enclave, the one outside does the syscalls21:30
abritothe one inside takes care that there is no leak (e.g., encrypts data going to the disk operation)21:31
oneswigIt seems to be targeted as an application-level tool rather than a system-level tool.  Is that accurate?21:31
abritoone application in baremetal would be to store certificates and other secrets in the enclave21:31
abritoIntel has just released an POC for doing that21:32
oneswigOooh, got a link?21:32
abritojust a sec21:32
abritothis one is from a project partner: https://github.com/lsds/TaLoS21:33
*** armax has joined #openstack-meeting21:34
oneswigIs there a performance penalty for accessing memory within the enclave, or executing code within the enclave?21:36
*** julim has quit IRC21:36
abritoin the graph in slide 8, you can see something about this21:38
abritoif the memory footprint is small21:38
abritoyou see no penalty21:38
abritothis would be the case if you are, for example, streaming the data through the protected application21:38
*** pcaruana has quit IRC21:38
oneswigAh, the y axis is relative slowdown of running in an enclave?21:39
abritoif you exceed the EPC size (e.g, the 128 mb) then it needs to decrypt and re-encrypt the data21:39
*** slaweq has quit IRC21:39
abritoadding a huge overhead21:39
*** david-lyle has quit IRC21:40
abritoyes, the Y axis is the overhead compared to regular C code running outside an enclave21:40
oneswigWhat is the difference in the code generated?21:40
oneswigHave you found it easy to work with?21:41
abritoit allocated a piece of the "enclave memory" and the secure functions and its data are allocated inside it21:41
*** eharney has quit IRC21:41
oneswigJust curious, what's a secure process for loading code into the enclave?21:42
*** egallen_ has joined #openstack-meeting21:42
*** gagehugo has left #openstack-meeting21:42
abritothere is some learning curve if your are using intel SDK directly, but if you do not need syscalls for the confidential algorithms/transformations, then it is mostly boilerplate code21:42
abritocan you rephrase that last question?21:43
*** egallen has quit IRC21:43
*** egallen_ is now known as egallen21:43
oneswigJust wondering how we trust the code as it is transferred in.  I guess there is some code signing or similar?21:43
abritoOnce the code is executed you can do a remote attestation21:43
abritothe remote attestation starts by an external participant asking the application to get a "quote" of itself21:44
abritothe quote is produced by the processor where the application is running in21:44
abritothen the application gives you the signed quote and if you have never trusted that processor before21:45
abritothen you go to intel attestation service (IAS) for it to confirm that the quote was emitted from a sgx-supporting processor21:45
abritousing the current firmware21:45
abritothat has not been blacklisted and that is running in the correct mode (e.g., non-debug or simulated)21:46
abritoif you already trusted the processor, you do not need to go to the intel service again21:46
oneswiginteresting - so if you trust Intel then you can also trust the cpu21:47
*** pchavva has quit IRC21:47
martial_did I understand properly: it creates a hardware memory map in the enclave ?21:48
abritoyes, for this version of SGX you have to trust intel to tell you that the code is actually running in the correct mode and processor21:48
abritomartial_: during boot it separates a piece of memory to be used by the enclaves21:48
oneswigI can see it being useful in apps where secrets are held.  Do you think it will succeed for future cpu generations?21:49
abritothat piece of memory cannot be accessed by code other then the code from the enclave that allocated it on creation21:49
abritooneswig: yes, I am optimistic21:49
oneswiggood to hear it.21:50
abritoone this is that recently, Azure and IBM have mentioned that they are making test services available that use SGX21:50
oneswigWhat will you do next with it?21:50
abritoe.g.: SGX capable VMs21:50
abritoI answer heard that enclave memory is likely to become larger in the short term21:51
abritomy next step is to run kubernetes jobs on it21:51
abritousing code in python running inside the enclaves21:51
oneswigwith the enclave holding something for the containerised app, or something for kubernetes itself?21:51
abritothere is not much to be done with kubernetes itself21:52
abritomonitoring needs to be done differently21:52
abritoso that you consider the EPC usage21:52
*** rmcall has quit IRC21:52
*** yamamoto has joined #openstack-meeting21:52
abritootherwise you can hit the 128 MB and suffer the performance hit21:52
abritobut once you have the code running, it mostly a matter of configuring the right tools21:53
abritoyou also want, for example, that the tasks in the tasks queues are encrypted21:53
abritoand only workers that have been attested hold the keys21:53
*** armstrong has joined #openstack-meeting21:54
abritowe (not only UFCG, but the securecloud consortium) are also working on monitoring and scheduling tools21:54
martial_so how different is it from a HSM?21:55
oneswigI'd be interested to know where you take it21:55
abritoit is a HSM, the advantage is that you already have it on your table21:55
abritoit is not a separate hardware piece21:56
abritothe downside is that not many Xeon have it21:56
martial_thank you, that helps21:56
oneswigOK - anything more for Andrey - we are close to time21:56
abritothere are people also looking at sgx for barbican21:57
oneswigI was wondering about that...21:57
abritoexactly because of its easier availability21:57
oneswigwould be great to use it for holding secrets 'at rest'21:57
martial_abrito: I was wondering about this, in 2006 the Barbican team did a Hands on during the Barcelona summit and they had a HSM setup21:58
*** jamesmcarthur has quit IRC21:58
martial_(can not remember the hardware now)21:58
oneswigOK, we must press on21:58
oneswigthank you Andrey - really interesting to hear about your work21:58
abritoSo, I would like to thank you for the invitation21:58
*** jamesmcarthur has joined #openstack-meeting21:59
abritoand apologize for the terrible slides21:59
*** yamamoto has quit IRC21:59
martial_really cool indeed, thank you for explaining this to us21:59
abritoI should had been more pessimistic about the time21:59
oneswig#topic AOB21:59
oneswigI had one item to raise - PTG21:59
oneswigThe Scientific SIG have been invited to have a slot at the PTG in Dublin, and I'm planning to go as there'll be at least 5-6 members present21:59
*** openstack changes topic to "AOB (Meeting topic: scientific-sig)"21:59
abritoI will do a second round, and explain details21:59
jmlowehe possibility of using a $200 nuc as a backing store for barbican is really exciting21:59
trandlesoneswig: thanks for carrying the torch for configurable deployment steps in Ironic at the Dublin PTG22:00
trandlesif you need anything documenting our use case let me know offline22:00
martial_Mike: yes sounds interesting indeed :)22:00
oneswigAnything development-centric on people's wish lists, let's have it before then22:01
oneswigIronic deployment steps - check22:01
oneswigWe'll also aim to cover some of the CERN/SKA subject areas22:01
oneswigbut anything else to cover - have a think and do follow up before the PTG, which is late February22:01
oneswigwe are out of time - anything else to raise?22:01
martial_good for me22:02
*** yamahata has quit IRC22:02
martial_thanks Stig22:02
trandlesthx for the topic today, very interesting22:03
*** jamesmcarthur has quit IRC22:03
rbuddenthanks everyone!22:04
*** dprince has quit IRC22:04
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"22:04
openstackMeeting ended Tue Jan  9 22:04:43 2018 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)22:04
openstackMinutes:        http://eavesdrop.openstack.org/meetings/scientific_sig/2018/scientific_sig.2018-01-09-21.00.html22:04
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/scientific_sig/2018/scientific_sig.2018-01-09-21.00.txt22:04
openstackLog:            http://eavesdrop.openstack.org/meetings/scientific_sig/2018/scientific_sig.2018-01-09-21.00.log.html22:04
*** oneswig has quit IRC22:06
*** rcernin has joined #openstack-meeting22:06
*** jmlowe has left #openstack-meeting22:06
*** oneswig has joined #openstack-meeting22:08
*** belmoreira has quit IRC22:10
*** oneswig has quit IRC22:11
*** trandles has left #openstack-meeting22:12
*** jamesmcarthur has joined #openstack-meeting22:13
*** egallen has quit IRC22:14
*** egallen_ has joined #openstack-meeting22:14
*** egallen_ has quit IRC22:14
*** abrito is now known as aembrito22:16
*** aeng has joined #openstack-meeting22:17
*** aembrito has quit IRC22:18
*** jamesmcarthur has quit IRC22:18
*** aeng has quit IRC22:19
*** dustins has quit IRC22:24
*** rfolco has quit IRC22:25
*** artom_ has joined #openstack-meeting22:26
*** jamesmcarthur has joined #openstack-meeting22:28
*** markvoelker has joined #openstack-meeting22:29
*** artom has quit IRC22:30
*** markvoelker has quit IRC22:30
*** jamesmcarthur has quit IRC22:33
*** zerick_ has quit IRC22:33
*** zerick has joined #openstack-meeting22:34
*** powerd has quit IRC22:34
*** gouthamr has quit IRC22:35
*** priteau has quit IRC22:40
*** bobh has quit IRC22:41
*** yamahata has joined #openstack-meeting22:48
*** aeng has joined #openstack-meeting22:48
*** zerick has quit IRC22:49
*** archit has quit IRC22:52
*** gouthamr has joined #openstack-meeting22:53
*** _pewp_ has quit IRC22:56
*** aeng has quit IRC22:56
*** _pewp_ has joined #openstack-meeting22:57
*** hongbin has quit IRC22:57
*** e0ne has joined #openstack-meeting22:58
*** sdague has quit IRC23:04
*** aeng has joined #openstack-meeting23:05
*** markvoelker has joined #openstack-meeting23:11
*** markvoelker has quit IRC23:11
*** aeng has quit IRC23:16
*** VW_ has joined #openstack-meeting23:27
*** Leo_m has quit IRC23:27
*** VW has quit IRC23:30
*** gmann_ has joined #openstack-meeting23:30
*** aeng has joined #openstack-meeting23:31
*** VW_ has quit IRC23:31
*** awaugama has quit IRC23:33
*** sdague has joined #openstack-meeting23:34
*** ykatabam has quit IRC23:34
*** ykatabam has joined #openstack-meeting23:35
*** julim has joined #openstack-meeting23:37
*** gmann has joined #openstack-meeting23:37
*** pengdake has joined #openstack-meeting23:37
*** markvoelker has joined #openstack-meeting23:43
*** markvoelker has quit IRC23:43
*** ianw has quit IRC23:50
*** pengdake has quit IRC23:51
*** ianw has joined #openstack-meeting23:51
*** ijw has joined #openstack-meeting23:54
*** netapp has quit IRC23:54
*** netapp has joined #openstack-meeting23:58

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!