Tuesday, 2018-03-20

*** liyi has joined #openstack-meeting00:04
*** aeng has joined #openstack-meeting00:05
*** liyi has joined #openstack-meeting00:05
*** markstur has quit IRC00:08
*** cloudrancher has joined #openstack-meeting00:11
*** ircuser-1 has joined #openstack-meeting00:12
*** wanghao has quit IRC00:21
*** wanghao has joined #openstack-meeting00:21
*** yamamoto has quit IRC00:26
*** cloudrancher has quit IRC00:27
*** yamamoto has joined #openstack-meeting00:28
*** yamamoto has quit IRC00:28
*** aeng has quit IRC00:35
*** aeng has joined #openstack-meeting00:35
*** TxGirlGeek has quit IRC00:36
*** fzdarsky__ is now known as fzdarsky|afk00:38
*** hongbin has joined #openstack-meeting00:40
*** amodi has quit IRC00:44
*** tovin07_ has joined #openstack-meeting00:45
*** lifeless_ has joined #openstack-meeting00:46
*** lifeless has quit IRC00:46
*** tovin07_ has quit IRC00:48
*** tovin07_ has joined #openstack-meeting00:49
*** ykatabam1 has quit IRC00:53
*** ykatabam has joined #openstack-meeting00:54
*** diman has joined #openstack-meeting01:00
*** aeng has quit IRC01:02
*** diman has quit IRC01:04
*** yamamoto has joined #openstack-meeting01:07
*** aeng has joined #openstack-meeting01:10
*** wxy has joined #openstack-meeting01:10
*** annabelleB has quit IRC01:11
*** Kevin_Zheng has joined #openstack-meeting01:14
*** fnaval has joined #openstack-meeting01:22
*** yamamoto has quit IRC01:27
*** tovin07_ has quit IRC01:29
*** tovin07_ has joined #openstack-meeting01:29
*** Li_Jiale has joined #openstack-meeting01:31
*** gman-tx has joined #openstack-meeting01:31
*** galstrom_zzz is now known as galstrom01:35
*** gman-tx has quit IRC01:36
*** kevzha01 has joined #openstack-meeting01:43
*** kevzha01 is now known as kevinz01:44
*** artom_ has joined #openstack-meeting02:01
*** caowei has joined #openstack-meeting02:03
*** artom has quit IRC02:03
*** ykatabam has quit IRC02:05
*** ykatabam has joined #openstack-meeting02:05
*** Nil_ has quit IRC02:12
*** galstrom is now known as galstrom_zzz02:12
*** chyka_ has joined #openstack-meeting02:15
*** chyka has quit IRC02:18
*** iyamahat has quit IRC02:20
*** yamahata has quit IRC02:21
*** chyka_ has quit IRC02:22
*** gongysh has joined #openstack-meeting02:25
*** kaisers_ has joined #openstack-meeting02:27
*** jbadiapa has quit IRC02:29
*** kaisers has quit IRC02:29
*** fzdarsky_ has joined #openstack-meeting02:36
*** links has joined #openstack-meeting02:40
*** fzdarsky|afk has quit IRC02:40
*** rfolco has quit IRC02:40
*** galstrom_zzz is now known as galstrom02:44
*** salv-orl_ has joined #openstack-meeting02:50
*** salv-orlando has quit IRC02:52
*** psachin has joined #openstack-meeting03:01
*** bkopilov has quit IRC03:08
*** jamesmcarthur has joined #openstack-meeting03:13
*** zerick has quit IRC03:14
*** dangtrinhnt_ has joined #openstack-meeting03:15
*** Guest66700 has joined #openstack-meeting03:17
*** zerick has joined #openstack-meeting03:18
*** Li_Jiale has quit IRC03:18
*** zerick has quit IRC03:20
*** chyka has joined #openstack-meeting03:23
*** dangtrinhnt_ has quit IRC03:25
*** chyka has quit IRC03:28
*** Dinesh_Bhor has joined #openstack-meeting03:41
*** gongysh has quit IRC03:42
*** aeng has quit IRC03:43
*** aeng has joined #openstack-meeting03:44
*** galstrom is now known as galstrom_zzz03:51
*** yamamoto has joined #openstack-meeting03:52
*** ricolin has joined #openstack-meeting03:55
*** tpatil has joined #openstack-meeting03:55
*** samP has joined #openstack-meeting03:59
samPHi all for masakari04:00
tpatilHi04:00
*** rkmrHonjo has joined #openstack-meeting04:00
samP#startmeeting masakari04:00
openstackMeeting started Tue Mar 20 04:00:25 2018 UTC and is due to finish in 60 minutes.  The chair is samP. Information about MeetBot at http://wiki.debian.org/MeetBot.04:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.04:00
*** openstack changes topic to " (Meeting topic: masakari)"04:00
openstackThe meeting name has been set to 'masakari'04:00
samPHi all04:00
Dinesh_BhorHi all04:00
rkmrHonjohi04:00
*** hongbin has quit IRC04:00
samPToday's main items is discuss Rocky work items.04:00
samPAny other items need to discuss before that?04:01
*** yamamoto has quit IRC04:01
samPCritical bugs/patches etc..?04:01
rkmrHonjoCan I talk about a bug?04:01
samPrkmrHonjo: sure04:01
rkmrHonjook. I'd like to talk about #link https://bugs.launchpad.net/python-masakariclient/+bug/175604704:01
openstackLaunchpad bug 1756047 in python-masakariclient "masakari command failed due to os-service-type1.2.0" [Undecided,New]04:02
rkmrHonjomasakariclient uses os-service-type, but it doesn't work with os-service-type 1.2.0.04:02
rkmrHonjo(In queen, the upper version is 1.1.0.)04:03
*** sagara has joined #openstack-meeting04:03
rkmrHonjoThe cause of this bug was written in Launcpad:04:03
*** niraj_singh has joined #openstack-meeting04:03
rkmrHonjo> In os-service-types 1.2.0, "ha" service has been added to os_service_type/data/service-types.json.04:03
rkmrHonjo> So, if masakariclient tries add_service to openstack.connection.Connection object, then the error occurr.04:03
rkmrHonjo> However, if masakariclient don't add_service, it can not get the correct proxy class for masakari's operation.04:03
rkmrHonjo> Probably, the "ha" service of service-types.json is for use by masakari.04:03
Dinesh_BhorrkmrHonjo: I was able to reproduce this issue on my machine with "openstacksdk-0.11.3" and "os_service_types-1.2.0".04:03
rkmrHonjoDinesh_Bhor: thank you for reproducing.04:04
rkmrHonjoSo, I think we should do 4 steps.04:04
rkmrHonjo1. Add a class about masakari to openstack-sdk.04:04
rkmrHonjo2. Bump openstack-sdk version in global requierments.04:04
rkmrHonjo3. Fix masakariclient to use the new version of openstack-sdk.04:04
rkmrHonjo4. Merge the patches changing service-type from "ha" to "instance-ha" #link http://lists.openstack.org/pipermail/openstack-dev/2018-January/126476.html04:04
*** yamamoto has joined #openstack-meeting04:04
rkmrHonjoI and Takahara succeeded to modify openstack-sdk and masakariclient in our environment.04:05
tpatilDoes you think #1 will be accepted by openstacksdk team?04:05
tpatils/does/Do04:05
rkmrHonjotpatil: I think that is no problem. Because it accept many official projects.04:05
rkmrHonjoSo we are preparing to submit the patch to openstack-sdk after that.04:06
tpatilbut in that case, we will need to move all code of sdk from masakariclient to openstacksdk, correct?04:06
tpatilAnyone tried to test masakari using stable/queens branch?04:07
samPI think Monty mention about (1) in some were..04:07
rkmrHonjotpatil: No, we shouldn't do it in stable/queens branch.04:07
rkmrHonjoBecause upper constraint version of os_service_type is 1.1.0 in stable/queens.04:07
*** yamamoto has quit IRC04:08
Dinesh_BhorsamP: yes, http://lists.openstack.org/pipermail/openstack-dev/2018-January/126421.html04:08
tpatilrkmrHonjo: That's another question, nothing to do with this bug04:08
samPDinesh_Bhor: yes..this was the first mail in the thread04:09
rkmrHonjotpatil: Oh, sorry.04:09
Dinesh_BhorThe question is "Is stable/queens branch of all masakari projects is working properly?"04:09
*** yamamoto has joined #openstack-meeting04:09
tpatilLet's address stable/queens branch question later04:10
samPIn stable/queens. except python-masakariclient, others are OK04:10
tpatilrkmrHonjo: Please continue04:10
rkmrHonjotpatil: Takahara tried to use masakariclient with #link https://review.openstack.org/#/c/553319/ . And it worked.04:11
rkmrHonjoAnd, Louie Kwan said that this patch worked, too.04:11
rkmrHonjoI'm glad if you try this patch and review it. (Ofcourse I review it.)04:13
*** yamamoto has quit IRC04:14
tpatilok, Let's review this patch and merge it in stable/queens branch04:14
rkmrHonjotpatil: thanks.04:15
samPrkmrHonjo: Thanks for the proposal. #553319 is to fix stable/queens and your proposed steps 1-5 is to fix master.04:15
*** yamamoto has joined #openstack-meeting04:16
rkmrHonjosamP: ok.04:16
tpatilrkmrHonjo: There is no other way to fix this issue other than moving the code to openstacksdk04:17
samPrkmrHonjo: Agree to review and merge #553319 to stable/queens. Let's discuss your proposed steps to fix master separately (in ML).04:17
samPtpatil: agree, even if we fixed it, then we will face same problem again in next release or sdk updates.04:18
*** jamesmcarthur has quit IRC04:18
tpatilsamP: I agree to move the code to openstacksdk but that's going to take lot of time. In the mean time, I was checking if there is any way to fix the problem in masakariclient04:19
samPtpatil: agree, thanks.04:19
rkmrHonjotpatil: thanks.04:19
samPAny other issues?04:20
samPIf any please bring them after next topic04:21
samP#topic Rocky Work Items04:21
*** openstack changes topic to "Rocky Work Items (Meeting topic: masakari)"04:21
*** liyi has quit IRC04:21
samP#link https://etherpad.openstack.org/p/masakari-rocky-work-items04:21
*** liyi has joined #openstack-meeting04:22
tpatilRecovery method customization04:22
samPtpatil: go ahead04:22
samPAny changes on plan?04:23
tpatilInstead of supporting mistral driver to run recovery action workflow, I was thinking of adding recovery method customization support in the existing task flow driver04:23
tpatilwe will add specs to add this support in task flow driver first04:23
samPtpatil: No objection on this. However, I would like to be this more configurable.04:25
tpatilsamP: Yes, the existing workflow for each notifications will be broken down into separate tasks and operator can configure which tasks to be included to run a particular workflow (instance/host/process notifications)04:26
tpatilAnyone can write a new tasks in a separate library and include it in any workflow. This library should be installed on the host where masakari-engine is running04:27
*** yamamoto has quit IRC04:27
samPtpatil: Do you still need this lib, even if you only use task flow driver?04:28
*** diman has joined #openstack-meeting04:28
*** epico has joined #openstack-meeting04:28
*** yamamoto has joined #openstack-meeting04:28
*** liyi has quit IRC04:29
*** liyi has joined #openstack-meeting04:29
*** andreas_s has joined #openstack-meeting04:29
*** tpatil has quit IRC04:31
samPtpatil: I think we had this discussion in earlier, please update the spec for this. We can discuss in details on the spec.04:32
samP(2) Develop Masakari Horizon plugin04:32
*** tpatil has joined #openstack-meeting04:32
samPtpatil: came back :)04:32
tpatilsorry I lost my connection04:33
samPtpatil: NP04:33
samPmy question was; Do you still need this lib, even if you only use task flow driver?04:33
*** radeks_ has joined #openstack-meeting04:33
*** diman has quit IRC04:33
tpatilThe default tasks will be part of masakari code.04:33
samPtpatil: got it.04:33
tpatilbut if operator wants customization, then the new tasks can be written in library04:34
tpatilfor example sending SMS after disabling compute host04:34
*** andreas_s has quit IRC04:34
*** radeks__ has joined #openstack-meeting04:34
*** liyi has quit IRC04:34
tpatilthese new tasks will not be part of masakari code but it will reside in library which needs to be installed on masakari-engine node and then operator can configure it for host notification workflow04:35
*** wanghao has quit IRC04:35
samPtpatil: understood.04:36
tpatilAny questions?04:36
tpatilsamP: Ok04:36
*** radeks_ has quit IRC04:37
tpatilcan we move to the next item ?04:37
samPNo questions for now. please add this details to spec, I would like to discuss more about how Operator would add this new methods.04:37
tpatilsamP: sure04:37
samPlet's move to next item04:38
samP(2) Develop Masakari Horizon plugin04:38
niraj_singhUnit test cases are failing because of service descriptor bug. Other than that all good.04:38
tpatilIt depends on https://bugs.launchpad.net/python-masakariclient/+bug/175604704:38
openstackLaunchpad bug 1756047 in python-masakariclient "masakari command failed due to os-service-type1.2.0" [Undecided,New]04:38
samPjust approved the https://review.openstack.org/#/c/52864704:40
niraj_singhsamP:thanks04:40
*** bkopilov has joined #openstack-meeting04:41
samPtpatil: niraj_singh: Thanks for doing this. This is heavy work. please ask if you need any help on this.04:41
niraj_singhsamP: yes sure. Thanks04:42
tpatilsamP: Most of the work has been completed by niraj_singh. Waiting for that bug to be fixed and he will push the patches for review04:43
samPOK, that's great...sorry for the blocker.04:43
*** yamamoto has quit IRC04:43
samP(3) Intrusive Instance Monitoring through QEMU Guest Agent04:44
samPThanks for the review.04:44
samPLouie and Greg are working on this. Let's review the spec and code and support them to merge this feature in early Rocky.04:46
Dinesh_Bhoryes04:46
*** yamamoto has joined #openstack-meeting04:46
samPCongratulations!! they got it on Lightning talk at Vancouver04:47
Dinesh_BhorGreat!!!04:47
rkmrHonjocheers04:48
tpatilThat's amazing!!04:48
samPIts better we can finish this before Summit.. and it will be great support to them.04:48
samP(4) Add database purge support04:48
tpatilNeeds code review04:48
samPtpatil: I will look into this. Sorry for the delay.04:49
tpatilsamP: Thanks04:49
samPAll members: please review this patch..04:50
samP#link https://review.openstack.org/#/c/487430/04:50
samPNew Items04:51
samP#topic New Items for Rocky04:51
*** openstack changes topic to "New Items for Rocky (Meeting topic: masakari)"04:51
samPSorry only have, 8 mins.04:51
*** wanghao has joined #openstack-meeting04:51
samPif we run out of time, will carry over to next week IRC meeting.04:51
*** liyi has joined #openstack-meeting04:52
samP(1) Update notification state API04:52
samPWho will proceed this item?04:52
rkmrHonjoI wrote it.04:52
samPrkmrHonjo: ok, Thanks.04:52
samPhow far we done on this?04:53
rkmrHonjoSorry, "how far"?04:54
*** armax has quit IRC04:54
rkmrHonjoAh, time span?04:54
samPrkmrHonjo: sorry, this is a new item04:54
samP;904:54
samPthis is some thing like force change the state?04:55
rkmrHonjosamP: yes.04:55
rkmrHonjoIf operator recovers host manually while masakari-engine is crashed, I want to force disable notification.04:55
samPOK, we need new API for this, and I would like to have spc for this.04:56
samPis it possible to write a spec for this?04:56
rkmrHonjosamP: Yes. I start to write the spec.04:56
samPrkmrHonjo: great.04:57
samPrkmrHonjo: thanks04:57
samP(2) Move some code from masakariclient to openstack-sdk04:57
samPrkmrHonjo: I guess this also you.04:57
rkmrHonjosamP: thanks.04:58
samPAnd we discussed part of this above.04:58
*** Dinesh__Bhor has joined #openstack-meeting04:58
*** Dinesh_Bhor has quit IRC04:58
samPWe are running out of time. Let's discuss other items in next meeting.04:58
tpatilsamP: Ok, one quick question04:59
samPIf you have any questions or comment on discussed items, please send them to ML04:59
samPtpatil: yes?04:59
tpatilthe openstack sensible plugin blueprint should be created in openstack-ansible project and not masakari, correct?04:59
tpatils/sensible/ansible04:59
samPtpatil: In Openstack ansible project04:59
tpatilsamP: Ok, thanks04:59
samP#endmeeting05:00
*** openstack changes topic to "Sending periodic "unreleased" reports (Meeting topic: releaseteam)"05:00
openstackMeeting ended Tue Mar 20 05:00:03 2018 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)05:00
openstackMinutes:        http://eavesdrop.openstack.org/meetings/masakari/2018/masakari.2018-03-20-04.00.html05:00
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/masakari/2018/masakari.2018-03-20-04.00.txt05:00
openstackLog:            http://eavesdrop.openstack.org/meetings/masakari/2018/masakari.2018-03-20-04.00.log.html05:00
samPThnks all05:00
*** lpetrut has joined #openstack-meeting05:02
*** liyi has quit IRC05:03
*** liyi has joined #openstack-meeting05:04
*** sagara has quit IRC05:04
*** rkmrHonjo has quit IRC05:12
*** liyi has quit IRC05:21
*** liyi has joined #openstack-meeting05:22
*** b1airo has joined #openstack-meeting05:25
*** Zames has joined #openstack-meeting05:31
*** sridharg has joined #openstack-meeting05:38
*** Zames has quit IRC05:39
*** beekhof_mb is now known as beekhof05:43
*** claudiub has joined #openstack-meeting05:45
*** Guest66700 has quit IRC05:45
*** liyi has quit IRC05:46
*** sidx64 has joined #openstack-meeting05:51
*** sidx64 has quit IRC05:57
*** lpetrut has quit IRC06:00
*** sidx64 has joined #openstack-meeting06:03
*** sidx64 has quit IRC06:04
*** aeng has quit IRC06:05
*** dineshbhor__ has joined #openstack-meeting06:07
*** tpatil has quit IRC06:07
*** Dinesh__Bhor has quit IRC06:09
*** wanghao has quit IRC06:09
*** sidx64 has joined #openstack-meeting06:11
*** lpetrut has joined #openstack-meeting06:12
*** masber has quit IRC06:12
*** sidx64 has quit IRC06:13
*** trinaths has joined #openstack-meeting06:21
*** e0ne has joined #openstack-meeting06:21
*** psachin has quit IRC06:27
*** sidx64 has joined #openstack-meeting06:38
*** sidx64 has quit IRC06:44
*** masber has joined #openstack-meeting06:44
*** psachin has joined #openstack-meeting06:47
*** masber has quit IRC06:47
*** masber has joined #openstack-meeting06:47
*** jbadiapa has joined #openstack-meeting06:53
*** armax has joined #openstack-meeting06:55
*** lpetrut has quit IRC06:57
*** marios has joined #openstack-meeting06:57
*** lhx_ has joined #openstack-meeting06:57
*** chyka has joined #openstack-meeting07:00
*** sidx64 has joined #openstack-meeting07:00
*** dineshbhor__ has quit IRC07:00
*** ykatabam has quit IRC07:00
*** sidx64 has quit IRC07:01
*** radeks_ has joined #openstack-meeting07:02
*** chyka has quit IRC07:04
*** radeks__ has quit IRC07:05
*** d0ugal has joined #openstack-meeting07:12
*** d0ugal has quit IRC07:12
*** d0ugal has joined #openstack-meeting07:12
*** rbartal has joined #openstack-meeting07:15
*** jamesmcarthur has joined #openstack-meeting07:18
*** d0ugal has quit IRC07:21
*** jamesmcarthur has quit IRC07:23
*** rcernin has quit IRC07:23
*** rcernin has joined #openstack-meeting07:24
*** rcernin has quit IRC07:24
*** lhx_ has quit IRC07:26
*** epico has quit IRC07:26
*** alexchadin has joined #openstack-meeting07:30
*** trinaths has quit IRC07:31
*** samP has quit IRC07:31
*** salv-orl_ has quit IRC07:32
*** salv-orlando has joined #openstack-meeting07:33
*** salv-orlando has quit IRC07:33
*** salv-orlando has joined #openstack-meeting07:33
*** lhx_ has joined #openstack-meeting07:34
*** pcaruana has joined #openstack-meeting07:34
*** dmacpher has quit IRC07:36
*** alexchadin has quit IRC07:36
*** alexchadin has joined #openstack-meeting07:37
*** masaito has joined #openstack-meeting07:39
*** epico has joined #openstack-meeting07:41
*** d0ugal has joined #openstack-meeting07:48
*** sidx64 has joined #openstack-meeting07:52
*** erlon has quit IRC07:52
*** masaito has left #openstack-meeting07:53
*** sidx64 has quit IRC07:58
*** tesseract has joined #openstack-meeting08:00
*** andreas_s has joined #openstack-meeting08:02
*** sidx64 has joined #openstack-meeting08:03
*** egonzalez has joined #openstack-meeting08:04
*** sidx64 has quit IRC08:06
*** ykatabam has joined #openstack-meeting08:06
*** sidx64 has joined #openstack-meeting08:08
*** priteau has joined #openstack-meeting08:08
*** _pewp_ has quit IRC08:11
*** _pewp_ has joined #openstack-meeting08:15
*** Zames has joined #openstack-meeting08:19
*** mardim has joined #openstack-meeting08:19
*** sidx64 has quit IRC08:22
*** dangtrinhnt has quit IRC08:23
*** trinaths has joined #openstack-meeting08:24
*** Kevin_Zheng has quit IRC08:30
*** _alastor_ has quit IRC08:30
*** zxiiro has quit IRC08:30
*** kopecmartin has joined #openstack-meeting08:30
*** _alastor_ has joined #openstack-meeting08:31
*** Kevin_Zheng has joined #openstack-meeting08:31
*** zxiiro has joined #openstack-meeting08:32
*** e0ne has quit IRC08:33
*** electrofelix has joined #openstack-meeting08:38
*** HeOS has joined #openstack-meeting08:38
*** _pewp_ has quit IRC08:41
*** Zames has quit IRC08:41
*** _pewp_ has joined #openstack-meeting08:47
*** lpetrut has joined #openstack-meeting08:49
*** sidx64 has joined #openstack-meeting08:49
*** lpetrut_ has joined #openstack-meeting08:50
*** lpetrut has quit IRC08:50
*** Zames has joined #openstack-meeting08:54
*** dbecker has joined #openstack-meeting08:56
*** sidx64 has quit IRC08:58
*** alexchadin has quit IRC08:58
*** Zames has quit IRC08:58
*** alexchadin has joined #openstack-meeting08:59
*** andreas_s has quit IRC09:01
*** andreas_s has joined #openstack-meeting09:01
*** d0ugal has quit IRC09:02
*** tssurya has joined #openstack-meeting09:02
*** jiaopengju_ is now known as jiaopengju09:02
chenying#startmeeting karbor09:08
openstackMeeting started Tue Mar 20 09:08:25 2018 UTC and is due to finish in 60 minutes.  The chair is chenying. Information about MeetBot at http://wiki.debian.org/MeetBot.09:08
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.09:08
*** openstack changes topic to " (Meeting topic: karbor)"09:08
openstackThe meeting name has been set to 'karbor'09:08
chenyingweclome to karbor irc meeting.09:08
chenyingping jiaopengju09:09
*** alexchadin has quit IRC09:09
jiaopengjuhi chenying09:10
*** phil has joined #openstack-meeting09:10
chenyingGood afernoon.09:10
*** phil is now known as Guest9422209:10
jiaopengju:)09:10
*** andreas_s has quit IRC09:10
*** alexchadin has joined #openstack-meeting09:10
chenyingwe don't have any fixed topic today. Do you have anything want to discuss?09:11
chenyinghttps://ethercalc.openstack.org/karbor-rocky09:11
jiaopengjunot yet09:12
jiaopengjuoh, I forget it, and will add it ASAP09:13
*** andreas_s has joined #openstack-meeting09:13
*** andreas_s has quit IRC09:22
*** andreas_s has joined #openstack-meeting09:22
*** fzdarsky_ is now known as fzdarsky09:23
*** _pewp_ has quit IRC09:24
chenyingSo if we have any idea and features about karbor in rocky, we can add it to this links.09:24
*** sidx64 has joined #openstack-meeting09:24
*** _pewp_ has joined #openstack-meeting09:26
*** beekhof has quit IRC09:28
jiaopengjuok09:29
chenyingSo I will end this meeting?09:30
*** _pewp_ has quit IRC09:33
*** ykatabam has quit IRC09:33
*** andreas_s has quit IRC09:36
*** andreas_s has joined #openstack-meeting09:37
chenying#endmeeting09:40
*** openstack changes topic to "Sending periodic "unreleased" reports (Meeting topic: releaseteam)"09:40
openstackMeeting ended Tue Mar 20 09:40:07 2018 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)09:40
openstackMinutes:        http://eavesdrop.openstack.org/meetings/karbor/2018/karbor.2018-03-20-09.08.html09:40
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/karbor/2018/karbor.2018-03-20-09.08.txt09:40
openstackLog:            http://eavesdrop.openstack.org/meetings/karbor/2018/karbor.2018-03-20-09.08.log.html09:40
*** _pewp_ has joined #openstack-meeting09:40
*** b1airo has quit IRC09:43
*** kevinz has quit IRC09:52
*** kaisers_ has quit IRC09:53
*** ricolin has quit IRC09:54
*** e0ne has joined #openstack-meeting09:55
*** andreas_s has quit IRC09:56
*** andreas_s has joined #openstack-meeting09:57
*** tovin07_ has quit IRC10:01
*** yamamoto has quit IRC10:04
*** alexchadin has quit IRC10:04
*** andreas_s has quit IRC10:07
*** andreas_s has joined #openstack-meeting10:07
*** Zames has joined #openstack-meeting10:08
*** yamamoto has joined #openstack-meeting10:09
*** chyka has joined #openstack-meeting10:13
*** yamamoto has quit IRC10:14
*** Zames has quit IRC10:18
*** chyka has quit IRC10:19
*** kaisers has joined #openstack-meeting10:21
*** Zames has joined #openstack-meeting10:23
*** Zames has quit IRC10:25
*** beekhof has joined #openstack-meeting10:31
*** sidx64 has quit IRC10:35
*** sambetts|afk is now known as sambetts10:36
*** ykatabam has joined #openstack-meeting10:37
*** masber has quit IRC10:43
*** caowei has quit IRC10:49
*** beekhof has quit IRC10:50
*** sidx64 has joined #openstack-meeting10:51
*** sidx64 has quit IRC11:02
*** rcernin has joined #openstack-meeting11:02
*** rcernin has quit IRC11:06
*** masber has joined #openstack-meeting11:08
*** yamamoto has joined #openstack-meeting11:10
*** masuberu has joined #openstack-meeting11:12
*** yamamoto has quit IRC11:12
*** yamamoto has joined #openstack-meeting11:12
*** masber has quit IRC11:14
*** ansmith has quit IRC11:17
*** tpsilva has joined #openstack-meeting11:20
*** pcaruana has quit IRC11:23
*** psachin has quit IRC11:28
*** fzdarsky has quit IRC11:29
*** psachin has joined #openstack-meeting11:33
*** chyka has joined #openstack-meeting11:35
*** sidx64 has joined #openstack-meeting11:38
*** chyka has quit IRC11:39
*** yamamoto has quit IRC11:40
*** yamamoto has joined #openstack-meeting11:41
*** ricolin has joined #openstack-meeting11:44
*** beekhof has joined #openstack-meeting11:44
*** beekhof has quit IRC11:48
*** pchavva has joined #openstack-meeting11:52
*** beekhof has joined #openstack-meeting11:52
*** sidx64 has quit IRC11:53
*** sidx64 has joined #openstack-meeting11:54
*** pcaruana has joined #openstack-meeting11:55
*** caowei has joined #openstack-meeting11:55
*** sidx64 has quit IRC11:57
*** sidx64 has joined #openstack-meeting11:58
*** sidx64 has quit IRC11:59
*** krenczewski has joined #openstack-meeting12:01
*** dangtrinhnt has joined #openstack-meeting12:04
*** sidx64 has joined #openstack-meeting12:07
*** raildo has joined #openstack-meeting12:08
*** sidx64 has quit IRC12:11
*** rfolco has joined #openstack-meeting12:14
*** artom_ has quit IRC12:15
*** edmondsw has joined #openstack-meeting12:17
*** sidx64 has joined #openstack-meeting12:18
*** trinaths has quit IRC12:20
*** epico has quit IRC12:21
*** sidx64 has quit IRC12:21
*** sidx64 has joined #openstack-meeting12:27
*** ricolin has quit IRC12:29
*** ricolin has joined #openstack-meeting12:29
*** dprince has joined #openstack-meeting12:30
*** bkopilov has quit IRC12:32
*** sidx64 has quit IRC12:34
*** Zames has joined #openstack-meeting12:38
*** sidx64 has joined #openstack-meeting12:40
*** Zames has quit IRC12:40
*** d0ugal has joined #openstack-meeting12:41
*** d0ugal has quit IRC12:41
*** d0ugal has joined #openstack-meeting12:41
*** VW has joined #openstack-meeting12:43
*** ricolin has quit IRC12:45
*** mjturek has joined #openstack-meeting12:46
*** sidx64 has quit IRC12:52
*** gman-tx has joined #openstack-meeting12:55
*** lhx_ has quit IRC13:02
*** ansmith has joined #openstack-meeting13:03
*** noslzzp has joined #openstack-meeting13:05
*** sidx64 has joined #openstack-meeting13:06
*** dangtrinhnt has quit IRC13:06
*** mjturek has quit IRC13:11
*** mriedem has joined #openstack-meeting13:18
*** iyamahat has joined #openstack-meeting13:18
*** iyamahat_ has joined #openstack-meeting13:19
*** psachin has quit IRC13:21
*** dustins has joined #openstack-meeting13:22
*** sidx64 has quit IRC13:22
*** iyamahat has quit IRC13:22
*** sidx64 has joined #openstack-meeting13:29
*** hongbin has joined #openstack-meeting13:30
*** chenyb4 has joined #openstack-meeting13:38
*** esberglu has quit IRC13:39
*** esberglu has joined #openstack-meeting13:42
*** Spaz-Home has quit IRC13:44
*** annabelleB has joined #openstack-meeting13:46
*** jamesmcarthur has joined #openstack-meeting13:47
*** jamesmcarthur has quit IRC13:47
*** jamesmcarthur has joined #openstack-meeting13:48
*** fnaval has quit IRC13:48
*** bkopilov has joined #openstack-meeting13:51
*** links has quit IRC13:51
*** rbudden has joined #openstack-meeting13:54
*** dmellado has quit IRC13:55
*** gouthamr has joined #openstack-meeting14:00
*** reedip_ has joined #openstack-meeting14:00
*** gman-tx has quit IRC14:01
*** shintaro has joined #openstack-meeting14:02
*** gman-tx has joined #openstack-meeting14:03
*** eharney has joined #openstack-meeting14:03
*** stevebaker has quit IRC14:04
*** ykatabam has quit IRC14:04
*** fnaval has joined #openstack-meeting14:06
*** finucannot is now known as stephenfin14:07
*** bobh has joined #openstack-meeting14:07
*** egallen has joined #openstack-meeting14:11
*** sidx64 has quit IRC14:11
*** awaugama has joined #openstack-meeting14:12
*** yamamoto_ has joined #openstack-meeting14:12
*** yamamoto has quit IRC14:12
*** chenyb4 has quit IRC14:14
*** reedip_ has quit IRC14:16
*** chenyb4 has joined #openstack-meeting14:17
*** chyka has joined #openstack-meeting14:18
*** cloudrancher has joined #openstack-meeting14:20
*** pcaruana has quit IRC14:21
*** eharney_ has joined #openstack-meeting14:23
*** egallen_ has joined #openstack-meeting14:23
*** egallen has quit IRC14:23
*** egallen_ is now known as egallen14:23
*** chyka has quit IRC14:23
*** eharney has quit IRC14:24
*** felipemonteiro has joined #openstack-meeting14:27
*** pcaruana has joined #openstack-meeting14:28
*** galstrom_zzz is now known as galstrom14:28
*** felipemonteiro_ has joined #openstack-meeting14:29
*** felipemonteiro has quit IRC14:32
*** dustins_ has joined #openstack-meeting14:33
*** eharney_ is now known as eharnye14:34
*** eharnye is now known as eharney14:34
*** egallen has quit IRC14:34
*** gouthamr has quit IRC14:35
*** Nil_ has joined #openstack-meeting14:35
*** dustins has quit IRC14:35
*** hemna_ has joined #openstack-meeting14:36
*** d0ugal has quit IRC14:38
*** annabelleB has quit IRC14:42
*** stevebaker has joined #openstack-meeting14:42
*** artom_ has joined #openstack-meeting14:46
*** browny__ has joined #openstack-meeting14:46
*** browny has quit IRC14:46
*** chenyb4 has quit IRC14:48
*** cloudrancher has quit IRC14:48
*** fnaval has quit IRC14:49
*** fnaval has joined #openstack-meeting14:49
*** annabelleB has joined #openstack-meeting14:50
*** chenyb4 has joined #openstack-meeting14:50
*** eharney_ has joined #openstack-meeting14:50
*** gouthamr has joined #openstack-meeting14:50
*** eharney has quit IRC14:51
*** wxy| has joined #openstack-meeting14:54
*** dmellado has joined #openstack-meeting14:54
*** electrofelix has quit IRC14:54
*** artom_ is now known as artom14:55
*** dmellado has quit IRC14:56
*** mjturek has joined #openstack-meeting14:57
*** amodi has joined #openstack-meeting15:00
*** shintaro has quit IRC15:00
*** dmellado has joined #openstack-meeting15:01
*** dprince has quit IRC15:03
*** dmellado has quit IRC15:04
*** gagehugo has joined #openstack-meeting15:04
*** dmellado has joined #openstack-meeting15:05
*** stevebaker has quit IRC15:06
*** TxGirlGeek has joined #openstack-meeting15:06
*** browny has joined #openstack-meeting15:07
*** browny__ has quit IRC15:08
*** Swanson has quit IRC15:09
*** stevebaker has joined #openstack-meeting15:09
*** d0ugal has joined #openstack-meeting15:10
*** Swanson has joined #openstack-meeting15:10
*** dmellado has quit IRC15:16
*** eharney_ is now known as eharney15:16
*** stevebaker has quit IRC15:17
*** julim has quit IRC15:22
*** VW_ has joined #openstack-meeting15:25
*** cloudrancher has joined #openstack-meeting15:26
*** VW_ has quit IRC15:27
*** VW_ has joined #openstack-meeting15:27
*** Leo_m has joined #openstack-meeting15:27
*** yamamoto_ has quit IRC15:27
*** galstrom is now known as galstrom_zzz15:28
*** VW has quit IRC15:29
*** chenyb4 has quit IRC15:30
*** galstrom_zzz is now known as galstrom15:32
*** stevebaker has joined #openstack-meeting15:36
*** ayoung has joined #openstack-meeting15:37
*** galstrom is now known as galstrom_zzz15:37
*** longkb has joined #openstack-meeting15:38
*** trinaths has joined #openstack-meeting15:39
*** trinaths has quit IRC15:43
*** trinaths has joined #openstack-meeting15:44
*** TxGirlGeek has quit IRC15:45
*** trinaths has quit IRC15:45
*** stevebaker has quit IRC15:46
*** TxGirlGeek has joined #openstack-meeting15:47
*** felipemonteiro_ has quit IRC15:48
*** spilla has joined #openstack-meeting15:48
*** chyka has joined #openstack-meeting15:49
*** longkb has quit IRC15:49
*** felipemonteiro_ has joined #openstack-meeting15:49
*** trinaths has joined #openstack-meeting15:51
*** gagehugo has left #openstack-meeting15:51
*** TxGirlGeek has quit IRC15:53
*** TxGirlGeek has joined #openstack-meeting15:53
*** stevebaker has joined #openstack-meeting15:53
*** salv-orlando has quit IRC15:56
*** salv-orlando has joined #openstack-meeting15:57
*** salv-orlando has quit IRC15:57
*** salv-orlando has joined #openstack-meeting15:57
*** spilla has left #openstack-meeting15:58
*** rbartal has quit IRC15:59
*** ihrachys has joined #openstack-meeting16:00
ihrachys#startmeeting neutron_ci16:00
openstackMeeting started Tue Mar 20 16:00:57 2018 UTC and is due to finish in 60 minutes.  The chair is ihrachys. Information about MeetBot at http://wiki.debian.org/MeetBot.16:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:00
*** openstack changes topic to " (Meeting topic: neutron_ci)"16:01
openstackThe meeting name has been set to 'neutron_ci'16:01
slaweqhi16:01
ihrachyso/16:01
ihrachysmlavalle won't be able to join us today16:01
ihrachyshaleyb, jlibosva16:01
*** jlibosva has joined #openstack-meeting16:01
jlibosvao16:01
jlibosva/16:01
*** TxGirlGeek has quit IRC16:02
ihrachysok let's get going16:02
*** stevebaker has quit IRC16:02
ihrachys#topic Actions from prev meeting16:02
*** openstack changes topic to "Actions from prev meeting (Meeting topic: neutron_ci)"16:02
ihrachysbtw thanks slaweq for taking it over the prev time, appreciate it16:02
ihrachysslaweq, you should probably become the chair for the meeting16:03
slaweqihrachys: no problem :)16:03
ihrachyssince you are the one who actually does most of the work :) you and jlibosva16:03
*** jessegler has joined #openstack-meeting16:03
ihrachysslaweq, I mean in generally16:03
slaweqmaybe jlibosva then :)16:03
slaweqI don't think I'm experienced enought to do it all the time16:03
*** TxGirlGeek has joined #openstack-meeting16:04
jlibosvaYou did really well last time :)16:04
ihrachysnah I think you are a better long term shot ;) things I know... :)16:04
ihrachysok as for action items16:04
ihrachys"slaweq to enable voting for fullstack"16:04
slaweqdone16:04
ihrachyshttps://review.openstack.org/55268616:04
*** stevebaker has joined #openstack-meeting16:04
*** VW_ has quit IRC16:05
ihrachyswas the change global for all queues? https://review.openstack.org/#/c/552686/1/.zuul.yaml16:05
ihrachysor is it blocked from voting in gate somewhere else?16:05
*** dtrainor has quit IRC16:05
slaweqit is only for master branch and for check queue for now16:05
slaweqthat is what we decided last time16:05
*** felipemonteiro_ has quit IRC16:06
slaweqif all will be fine we will add it to gate queue also, am I right?16:06
ihrachysoh I get it, it's not in gate jobs list16:06
*** VW has joined #openstack-meeting16:06
slaweqyes16:06
ihrachysyeah you are right, I am just bad at tech16:06
*** TxGirlGeek has quit IRC16:06
ihrachys"slaweq to enable voting for linuxbridge scenarios"16:06
slaweqdone also: https://review.openstack.org/#/c/552689/16:07
*** julim has joined #openstack-meeting16:07
ihrachysdemocracy, everyone gets a vote16:07
slaweqand also only in check queue for now16:07
ihrachys+16:07
ihrachys"jlibosva to take a look on dvr trunk tests issue"16:07
jlibosvaehm16:07
jlibosvaI didn't16:07
ihrachyscontext: http://logs.openstack.org/32/550832/2/check/neutron-tempest-plugin-dvr-multinode-scenario/6d09fd8/logs/testr_results.html.gz16:07
jlibosvaI'll try this week16:08
ihrachys#action jlibosva to take a look on dvr trunk tests issue: http://logs.openstack.org/32/550832/2/check/neutron-tempest-plugin-dvr-multinode-scenario/6d09fd8/logs/testr_results.html.gz16:08
ihrachysgreat16:08
ihrachys"slaweq check reasons of failures of neutron-tempest-dvr-ha-multinode-full"16:08
slaweqI checked it and I created bug https://bugs.launchpad.net/neutron/+bug/175630116:08
openstackLaunchpad bug 1756301 in neutron "Tempest DVR HA multimode tests fails due to no FIP connectivity" [High,Confirmed]16:08
ihrachyscontext http://logs.openstack.org/14/529814/5/check/neutron-tempest-dvr-ha-multinode-full/d8cfbdf/logs/testr_results.html.gz16:08
slaweqit looks that in all those cases failure is the same16:09
slaweqbut I didn't do any debugging on this issue16:09
slaweqall of other cases which I checked issues were related to cinder volumes16:10
*** TxGirlGeek has joined #openstack-meeting16:10
ihrachysslaweq, that's weird. I would understand it if e.g. instance would crash because of volume attachment.16:10
ihrachysbut it's clear it's something net related16:11
slaweqyes, In all those cases which are pointed in this bug report errors are related to neutron16:11
ihrachysat least one failure of those listed is in tempest.scenario.test_server_basic_ops.TestServerBasicOps16:11
ihrachysI don't think it's vol specific16:11
slaweqI was just saying that I found many other failures of this job with issues not related to network but related to cinder16:11
slaweqsorry if I wasn't clear :)16:12
ihrachysI see.16:12
*** stevebaker has quit IRC16:12
ihrachyswell, the bug is reported. is someone will have cycles they can take it16:13
ihrachys"slaweq switch scenario jobs to lib/neutron"16:13
ihrachysslaweq, I am missing context on that one. while it's good to move there, I don't think we did for gate or any other jobs.16:13
ihrachysso what's special about scenarios?16:13
slaweqnothing, I just wanted to start with something :)16:13
slaweqso I send a patch https://review.openstack.org/#/c/552689/ even16:14
*** stevebaker has joined #openstack-meeting16:14
slaweqto switch one scenario job to zuul v316:14
slaweqsorry, wrong link, this one is good: https://review.openstack.org/#/c/552846/1316:14
*** ricolin has joined #openstack-meeting16:15
slaweqbut in meantime devstack-tempest job was switched back to lib/neutron-legacy so this will not switch our job to using lib/neutron for now16:15
*** TxGirlGeek has quit IRC16:15
slaweqbut I still think that it could be good to start moving jobs to zuul v3 definitions16:15
slaweqwhat You think about that?16:15
*** kopecmartin has quit IRC16:16
ihrachysyeah that's good. I am not sure if that helps with lib/neutron target but we should move to the new format regardless.16:16
jlibosvawe talked with slaweq and we agreed it would be good to migrate and then once devstack-tempest is switched, we'll get it for free16:16
slaweqexactly, thx jlibosva :)16:16
slaweqso please review it if You will have some time :)16:17
ihrachysfair16:18
*** masuberu has quit IRC16:18
ihrachys#topic Grafana16:18
*** openstack changes topic to "Grafana (Meeting topic: neutron_ci)"16:18
jlibosvaI looked but I don't feel like I understand the zuul thingy completely so I'm not comfortable +2'ing16:18
ihrachyshttp://grafana.openstack.org/dashboard/db/neutron-failure-rate16:18
*** egonzalez has quit IRC16:20
ihrachysthere was some suspicious fullstack spike a day ago. while some other jobs also spikes at the same time, the fullstack chart doesn't seem to have recovered completely.16:20
ihrachysany ideas what's the spike about?16:20
slaweqtoday I spotted 2 or 3 times issue like described in https://bugs.launchpad.net/neutron/+bug/175708916:21
openstackLaunchpad bug 1757089 in neutron "Fullstack test_port_shut_down(Linux bridge agent) fails quite often" [High,Confirmed] - Assigned to Slawek Kaplonski (slaweq)16:21
slaweqI started debugging it even but haven't found anything yet16:21
slaweqI think it is some kind of race between finishing configuration of port and setting it to admin_state_up = False16:21
slaweqbut I'm not sure yet16:22
ihrachysyeah I haven't seen this test case failing before16:22
*** wxy|_ has joined #openstack-meeting16:22
*** andreas_s has quit IRC16:22
*** wxy| has quit IRC16:22
*** stevebaker has quit IRC16:22
ihrachysslaweq, are you planning to dig it further or we should have someone else on it?16:22
slaweqyes, I will take care of this one16:22
ihrachysthe job just moved to voting so it would be nice to tackle that before people start complaining :)16:22
ihrachysgreat!16:22
ihrachysanother change in grafana worth discussion is that dvr-scenarios seem to be at 100% now16:23
slaweqbut I don't know if this is the only issue with fullstack16:23
ihrachyssince 4 days ago16:23
*** TxGirlGeek has joined #openstack-meeting16:23
ihrachysfailure example: http://logs.openstack.org/76/550676/10/check/neutron-tempest-plugin-dvr-multinode-scenario/1c3297a/logs/testr_results.html.gz16:24
*** stevebaker has joined #openstack-meeting16:24
ihrachyshaleyb, ideas what's broken in dvr scenarios job?16:25
*** Guest94222 has quit IRC16:25
ihrachysok here are errors: http://logs.openstack.org/76/550676/10/check/neutron-tempest-plugin-dvr-multinode-scenario/1c3297a/logs/screen-q-l3.txt.gz?level=TRACE16:26
ihrachys"NetlinkError: (99, 'Cannot assign requested address')"16:26
*** gyee has joined #openstack-meeting16:26
ihrachysslaweq, I believe that's your pyroute2 patch16:26
*** andreas_s has joined #openstack-meeting16:26
slaweqihrachys: it can be16:27
slaweqI will check that one ASAP16:28
*** yamamoto has joined #openstack-meeting16:28
*** salv-orlando has quit IRC16:28
*** jlibosva has quit IRC16:28
*** salv-orlando has joined #openstack-meeting16:28
ihrachys#action slaweq to check why dvr-scenario job is broken with netlink errors in l3 agent log16:28
ihrachysinteresting that it doesn't happen in linuxbridge16:29
ihrachysslaweq, btw jlibosva left so we are two now16:30
slaweqihrachys: ok16:30
ihrachys#topic Fullstack16:30
*** openstack changes topic to "Fullstack (Meeting topic: neutron_ci)"16:30
ihrachysthere are some existing bug reports that we could take a look at since we have half an hour16:31
ihrachyshttps://bugs.launchpad.net/neutron/+bugs?field.tag=fullstack&orderby=status&start=016:31
ihrachysI will ignore wishlist bugs16:31
slaweqsure16:31
*** cloudrancher has quit IRC16:31
ihrachyshttps://bugs.launchpad.net/neutron/+bug/175708916:32
openstackLaunchpad bug 1757089 in neutron "Fullstack test_port_shut_down(Linux bridge agent) fails quite often" [High,Confirmed] - Assigned to Slawek Kaplonski (slaweq)16:32
ihrachysthis one we already discussed, it's the new failure you just reported16:32
slaweqyep16:32
*** ricolin has quit IRC16:32
*** cloudrancher has joined #openstack-meeting16:32
*** salv-orlando has quit IRC16:33
ihrachysand you are going to look into it16:33
ihrachyshttps://bugs.launchpad.net/neutron/+bug/174440216:33
openstackLaunchpad bug 1744402 in neutron "fullstack security groups test fails because ncat process don't starts" [High,Confirmed] - Assigned to Slawek Kaplonski (slaweq)16:33
slaweqyes16:33
ihrachysthis one resulted in test disabled16:33
*** yamamoto has quit IRC16:34
ihrachysdo we have patches tackling the original issue?16:34
*** marios has quit IRC16:34
ihrachysdoesn't seem like it16:34
slaweqno, we don't have any patches for this one16:34
ihrachyschecking on logstash how often it happens16:35
*** stevebaker has quit IRC16:35
slaweqand to be honest I'm not sure if this is still issue or maybe it is fixed by https://review.openstack.org/#/c/545820/16:35
slaweqas it could be related to problem with not configured IP address in fake machine namespace16:36
ihrachysslaweq, you think ncat failed to start because no ip set?16:36
*** andreas_s has quit IRC16:36
slaweqit could be IMO16:36
slaweqif on destination fake machine there wasn't IP address, it can't start16:36
*** armax has quit IRC16:37
slaweqbut I'm not 100% sure, maybe there is different problem still16:37
ihrachyswe could close it and see if it reemerges16:37
ihrachysboy logstash is slow16:37
slaweqwe can do that, and I will remove this unstable test decorator16:38
slaweqso if it will fail we will easier find it16:38
ihrachysyeah. no hits in logstash in a month.16:38
slaweqso I will remove this decorator and we will see, fine?16:39
*** cloudrancher has quit IRC16:39
ihrachys#action slaweq to revert https://review.openstack.org/#/c/541242/ because the bug is probably fixed by https://review.openstack.org/#/c/545820/416:39
slaweqthx16:39
*** stevebaker has joined #openstack-meeting16:39
ihrachyshttps://bugs.launchpad.net/neutron/+bug/168707416:39
openstackLaunchpad bug 1687074 in neutron "Sometimes ovsdb fails with "tcp:127.0.0.1:6640: error parsing stream"" [High,Confirmed]16:39
ihrachysthat's an old one.16:40
ihrachysI don't think Terry made any progress since it was initially looked into16:40
slaweqI didn't saw any patch related to this one for sure16:40
*** sshank has joined #openstack-meeting16:40
slaweqbut it's also not so often now IMO16:41
ihrachysat some point, this was present in lots of failures. I don't think we have those lately.16:41
ihrachysright. it seems like maybe something was fixed in the middle.16:41
ihrachysand I recollect we sometimes saw those errors in successful runs too16:41
ihrachysso it could as well be a red herring, and what helped is a set of unrelated patches that you carefully merged into for legit issues.16:42
slaweqso maybe we can close it for now and reopen if will be necessary?16:42
ihrachysmaybe we were blaiming the failures on the error message16:42
slaweqit can be like You are saying16:42
ihrachysyes that's my inclination. I will leave ovsdbapp part open in LP so that they can decide on their own.16:42
*** rfolco is now known as rfolco_brb16:42
slaweqok16:43
*** dprince has joined #openstack-meeting16:44
ihrachysdid just that16:44
*** gman-tx has quit IRC16:44
slaweqthx16:44
ihrachyshttps://bugs.launchpad.net/neutron/+bug/167353116:44
openstackLaunchpad bug 1673531 in neutron "fullstack test_controller_timeout_does_not_break_connectivity_sigkill(GRE and l2pop,openflow-native_ovsdb-cli) failure" [High,Confirmed] - Assigned to Ihar Hrachyshka (ihar-hrachyshka)16:44
slaweqI also didn't saw this one lately16:47
ihrachyscrawling logstash in background. I will close if no hits.16:47
slaweqcould be maybe fixed by https://review.openstack.org/#/c/547345/16:47
*** jamesmcarthur has quit IRC16:47
slaweqbut I don't remember exactly in which moment this test was failing usually so I'm not sure16:47
ihrachysmaybe. it was slow port active state propagation16:47
ihrachysok no hits in logstash16:47
ihrachysclosed16:47
ihrachysthe rest are wishlist bugs16:47
slaweqFYI: I just marked https://bugs.launchpad.net/neutron/+bug/1744396 as dubplicate16:48
openstackLaunchpad bug 1723912 in neutron "duplicate for #1744396 Refactor securitygroup fullstack tests" [Wishlist,Confirmed] - Assigned to Dongcan Ye (hellochosen)16:48
ihrachys(moved on Low -> Wishlist just now that talks about signal we use to tear down resources)16:48
slaweqbecause there was two about same thing16:48
ihrachysslaweq, great16:48
slaweqonly two not whishlist, that is good :)16:49
ihrachysok we are done with this list and still have 10 more minutes. so let's also clean up gate-failure bugs16:49
ihrachyshttps://bugs.launchpad.net/neutron/+bugs?field.tag=gate-failure&orderby=status&start=016:49
ihrachyshttps://bugs.launchpad.net/neutron/+bug/174188916:49
openstackLaunchpad bug 1741889 in neutron "functional: DbAddCommand sometimes times out after 10 seconds" [High,New]16:49
slaweqis it still an issue?16:50
ihrachysnot sure. ovsdbapp release that claims it's fixed was issued.16:50
slaweqfunctional tests are quite stable last days16:50
ihrachysthere are 16 hits of the timeout message in logstash for a month16:51
ihrachysbut no functional it seems16:52
ihrachysexample: http://logs.openstack.org/99/538699/3/gate/tempest-full-py3/71b6729/controller/logs/screen-q-agt.txt?level=TRACE#_Mar_20_08_57_06_77565316:52
slaweqbut this is oslo messaging timeout16:52
ihrachysbut there is also queue.Empty above16:52
slaweqrigh16:52
slaweqright16:52
ihrachysslaweq, oslo messaging?16:53
slaweqhttp://logs.openstack.org/99/538699/3/gate/tempest-full-py3/71b6729/controller/logs/screen-q-agt.txt?level=TRACE#_Mar_20_08_57_07_66363916:53
ihrachysoh I see below16:53
*** timsim has left #openstack-meeting16:53
*** kaisers has quit IRC16:53
slaweqI was talking about this one just below "yours" :)16:53
*** kaisers has joined #openstack-meeting16:53
ihrachysthat's timeout on reporting state16:53
ihrachyslooks like both rpc state thread and ovsdbapp was hanging16:54
slaweqmaybe it's  another issue with load on host16:54
ihrachysyeah16:54
ihrachysok let's close the functional bug16:54
slaweq++16:54
*** felipemonteiro has joined #openstack-meeting16:55
ihrachyshttps://bugs.launchpad.net/neutron/+bug/173847516:55
openstackLaunchpad bug 1738475 in neutron "test_notify_port_ready_after_enable_dhcp fails with RuntimeError: 'dhcp_ready_on_ports' not be called" [Low,New]16:55
ihrachyshaven't seen it honestly16:56
slaweqIMO we can close it - I didn't found anything like that since then16:56
ihrachysok16:56
ihrachyshttps://bugs.launchpad.net/neutron/+bug/166434716:57
openstackLaunchpad bug 1664347 in OpenStack Compute (nova) "test_volume_boot_pattern failed to get an instance into ACTIVE state" [Undecided,Confirmed]16:57
ihrachysit's Incomplete for us16:57
slaweqyep16:57
ihrachyshow to get rid off it from the list?16:58
ihrachysI thought incomplete bugs eventually expire16:58
ihrachysbut probably the nova component holds it16:58
slaweqI don't know16:58
ihrachysmaybe I will just kill the neutron link16:58
slaweq++16:58
ihrachyshttps://bugs.launchpad.net/neutron/+bug/175033416:59
openstackLaunchpad bug 1750334 in neutron "ovsdb commands timeouts cause fullstack tests failures" [High,Confirmed]16:59
ihrachysisn't it the same as other timeout bugs?16:59
*** masuberu has joined #openstack-meeting16:59
*** HeOS has quit IRC16:59
slaweqyes, AFAIR it was same issue like with functional16:59
ihrachysmarking as dup16:59
ihrachysok we are at the top of the hour17:00
*** cschwede_ has quit IRC17:00
ihrachysgreat, nice to do some cleanup once in a while :)17:00
ihrachysslaweq, thanks a lot!17:00
slaweqthx17:00
ihrachys#endmeeting17:00
*** openstack changes topic to "Sending periodic "unreleased" reports (Meeting topic: releaseteam)"17:00
slaweqI like such cleanup :)17:00
openstackMeeting ended Tue Mar 20 17:00:29 2018 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)17:00
openstackMinutes:        http://eavesdrop.openstack.org/meetings/neutron_ci/2018/neutron_ci.2018-03-20-16.00.html17:00
slaweqhave a nice day17:00
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/neutron_ci/2018/neutron_ci.2018-03-20-16.00.txt17:00
openstackLog:            http://eavesdrop.openstack.org/meetings/neutron_ci/2018/neutron_ci.2018-03-20-16.00.log.html17:00
*** stevebaker has quit IRC17:00
*** gman-tx has joined #openstack-meeting17:03
*** caowei has quit IRC17:03
*** dtrainor has joined #openstack-meeting17:04
*** sridharg has quit IRC17:04
*** felipemonteiro_ has joined #openstack-meeting17:04
*** mjturek has quit IRC17:05
*** mjturek has joined #openstack-meeting17:07
*** felipemonteiro has quit IRC17:08
*** jessegler has quit IRC17:08
*** felipemonteiro has joined #openstack-meeting17:08
*** fzdarsky has joined #openstack-meeting17:09
*** stevebaker has joined #openstack-meeting17:09
*** felipemonteiro_ has quit IRC17:11
*** bkero- has quit IRC17:11
*** wxy|_ has quit IRC17:12
*** bobh has quit IRC17:13
*** annabelleB has quit IRC17:18
*** annabelleB has joined #openstack-meeting17:19
*** stevebaker has quit IRC17:19
*** stevebaker has joined #openstack-meeting17:28
*** salv-orlando has joined #openstack-meeting17:29
*** lpetrut_ has quit IRC17:30
*** priteau has quit IRC17:30
*** yamamoto has joined #openstack-meeting17:30
*** masuberu has quit IRC17:33
*** HeOS has joined #openstack-meeting17:33
*** salv-orlando has quit IRC17:34
*** e0ne has quit IRC17:35
*** yamamoto has quit IRC17:36
*** galstrom_zzz is now known as galstrom17:36
*** baoli has joined #openstack-meeting17:39
*** baoli has quit IRC17:40
*** bkero1 has joined #openstack-meeting17:42
*** stevebaker has quit IRC17:44
*** bkero1 has quit IRC17:46
*** bkero1 has joined #openstack-meeting17:47
*** HeOS has quit IRC17:47
*** trinaths has quit IRC17:51
*** jessegler has joined #openstack-meeting17:53
*** fzdarsky has quit IRC17:54
*** stevebaker has joined #openstack-meeting17:54
*** dustins has joined #openstack-meeting17:58
*** dustins_ has quit IRC17:58
*** jessegler_ has joined #openstack-meeting18:01
*** rfolco_brb is now known as rfolco18:03
*** salv-orlando has joined #openstack-meeting18:07
*** stevebaker has quit IRC18:07
*** cloudrancher has joined #openstack-meeting18:11
*** jamesmcarthur has joined #openstack-meeting18:12
*** cloudrancher has quit IRC18:12
*** sambetts is now known as sambetts|afk18:13
*** cloudrancher has joined #openstack-meeting18:13
*** lpetrut has joined #openstack-meeting18:13
*** stevebaker has joined #openstack-meeting18:16
*** bkero1 has quit IRC18:19
*** diablo_rojo has quit IRC18:22
*** harlowja has joined #openstack-meeting18:23
*** stevebaker has quit IRC18:29
*** gyee has quit IRC18:30
*** apetrich has quit IRC18:30
*** yamamoto has joined #openstack-meeting18:32
*** apetrich has joined #openstack-meeting18:33
*** cloudrancher has quit IRC18:33
*** jessegler has quit IRC18:35
*** galstrom is now known as galstrom_zzz18:36
*** stevebaker has joined #openstack-meeting18:36
*** diablo_rojo has joined #openstack-meeting18:37
*** yamamoto has quit IRC18:37
*** tssurya has quit IRC18:37
*** sshank has quit IRC18:41
*** andreas_s has joined #openstack-meeting18:45
*** stevebaker has quit IRC18:46
*** stevebaker has joined #openstack-meeting18:48
*** andreas_s has quit IRC18:50
*** arxcruz is now known as arxcruz|off18:53
*** tobiash has joined #openstack-meeting18:53
*** jessegler_ has quit IRC18:54
*** AJaeger has joined #openstack-meeting18:56
*** stevebaker has quit IRC18:57
dmsimard\o19:00
dmsimardodd, looks like we're int he middle of a meeting if we look at the topic19:00
dmsimardoh, someone started a meeting from a leaked meeting topic19:00
clarkbmeetbot will set the topic properly for us then after our meeting we can correct the bad topic19:01
clarkbhello infra19:01
clarkb#startmeeting infra19:01
openstackMeeting started Tue Mar 20 19:01:18 2018 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.19:01
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.19:01
*** openstack changes topic to " (Meeting topic: infra)"19:01
openstackThe meeting name has been set to 'infra'19:01
fricklero/19:01
* jesusaur lurks19:01
ianwo/19:01
clarkb#link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting19:02
* fungi has the topic correction queued up for after the meeting, just remind me19:02
clarkb#topic Announcements19:02
*** openstack changes topic to "Announcements (Meeting topic: infra)"19:02
clarkbinfra-root should go and sign the rocky release gpg key19:02
mordredo/19:02
clarkbthe directions for doing this are on the system-config docs19:02
dmsimardWhat is the link to see the key on the server again ?19:02
*** cloudrancher has joined #openstack-meeting19:02
* clarkb makes a note to do it himself19:02
dmsimardfungi gave that to me the other day but didn't bookmark it19:02
mordredI seem to remember signing it already19:02
fungi#link https://docs.openstack.org/infra/system-config/signing.html#attestation Attestation instructions19:03
*** cloudrancher has quit IRC19:03
fungi#link https://sks-keyservers.net/pks/lookup?op=vindex&search=0xc31292066be772022438222c184fd3e1edf21a78&fingerprint=on Rocky Cycle key19:03
fungialso find them at the bottom of the releases site when in production19:04
dmsimardI ended up finding http://sks.spodhuis.org/pks/lookup?op=vindex&longkeyid=on&search=0x184FD3E1EDF21A78 but I guess that's the same thing19:04
fungi#link https://releases.openstack.org/#cryptographic-signatures19:04
mordredbut nope - I have not19:04
fungifor the record, it went into production late yesterday, but you can still attest to it any time you like19:04
clarkbI'm going to be traveling at the tail end of this week and early next week. Which means it would probably best to have someone else chair next weeks meeting19:05
clarkbIf you'd like to do that let me know in the infra channel later or via email?19:05
fungii can volunteer, but happy for someone else to take a turn19:05
* fungi got his fill19:06
clarkbI'll likely go quiet thursday or friday while I pack and prep for conferency things19:06
*** stevebaker has joined #openstack-meeting19:06
clarkbAnd finally since this group includes people that straddle the ops and dev roles you may be interested in a proposal to on the ops mailing list to merge the ops midcycle with the ptg19:07
*** cloudrancher has joined #openstack-meeting19:07
fungioh, last minute but you can announce that zuul &co are no longer infra deliverables as of today19:08
*** cloudrancher has quit IRC19:08
clarkbya was going to bring that up later too. We've had a long standing zuulv3 priority effort meeting agenda item that maybe we need to reevaluate in this split out? We can talk about it in a short while19:08
fungi#link http://lists.openstack.org/pipermail/openstack-operators/2018-March/014994.html Ops Meetup, Co-Location options, and User Feedback19:08
fungi(start of the colocation thread)19:09
clarkb#topic Specs approval19:09
*** openstack changes topic to "Specs approval (Meeting topic: infra)"19:09
clarkb#link https://review.openstack.org/#/c/550550/ Improve IRC discoverability19:09
clarkbthat change could still use some reviews19:09
clarkbI also wonder if the slack kills irc gateway plan affects our other irc related spec19:10
clarkbIt doesn't really affect us directly as we don't use slack but there have been a non zero number of people wanting to use our bots with slack19:10
corvusclarkb: "the slack kills irc gateway plan" ?19:11
*** jamesmcarthur has quit IRC19:11
mordredclarkb: it makes me less inclined to be accomodating, tbh19:11
clarkbcorvus: slack recently announced that mid may their irc gateway would be shutdown19:11
clarkbcorvus: basically forcing all bots to use their applications interface. Which is quite limited on the free tier19:11
clarkb(I'm sure that that isn't related at all to killing the gateway)19:11
corvusclarkb: you mean http://specs.openstack.org/openstack-infra/infra-specs/specs/irc.html ?19:12
corvusclarkb: that doesn't actually mention slack19:12
corvusin fact, i'd say a key characteristic of that spec is that we've de-prioritized non-irc support for our bots19:12
clarkbcorvus: it doesn't but one of the other alternatives we had considered in the past was using that one lib that talks all the protocols.19:12
clarkbcorvus: yes19:12
*** jamesmcarthur has joined #openstack-meeting19:13
corvusyeah, so i guess that's now *even less* important19:13
fungi#link https://review.openstack.org/319506 [abandoned] Add spec for hosted IRC client19:13
fungiis also potentially related19:13
clarkbfwiw I'm happy to continue using irc. I think it would be difficult for us to supprot slack natively19:13
clarkbbut I've also suggested in the past that "hey all the bots work with slack right now because irc"19:13
fungii'm increasingly in favor of helping people find ways to make irc fit their needs rather than support additional protocols which merely fracture the community further19:14
clarkb++19:14
corvusyep, sign me up for that19:14
dmsimardI know for a fact that several projects are using a slack <-> irc bridge (with something like https://github.com/ekmartin/slack-irc )19:14
dmsimardKolla does it, ARA does as well -- not sure if there's others19:15
clarkbok sounds like we don't need to take a major shift here as we are happy pushing towards irc19:15
dmsimardI mean, as much as I dislike Slack.. there's people who dislike IRC just as much and that shouldn't prevent them from contributing and discussing things19:15
corvussure it should19:15
corvusit's not free software19:15
corvusand we don't require folks to use free software to work with our systems19:15
dmsimardI never said it was a requirement to use SLack19:16
corvussee https://governance.openstack.org/tc/reference/irc.html19:16
dmsimardI said there's people who prefer Slack to IRC and if we can let them use Slack to talk to us, why not ?19:16
clarkbthat I think is an entirely separate conversation19:16
dmsimardYeah it is, sorry for sidetracking19:16
clarkb(I was mostly scopign this to the specs we have related to changes to our bots)19:16
clarkb(and sounds like there is agreement that wedon't need to update any specs related to that)19:17
clarkbwe can talk about the other thing post meeting if we really want to19:17
clarkb#topic Priority Efforts19:17
fungitranslating: "if people prefer proprietary software we should go out of our way to support it" (i can't personally agree with that)19:17
*** openstack changes topic to "Priority Efforts (Meeting topic: infra)"19:17
clarkb#topic Zuul v319:17
*** openstack changes topic to "Zuul v3 (Meeting topic: infra)"19:18
corvusi had an obvious typo in a previous remark, but i'll leave it as an exercise for the reader so we can move on19:18
clarkbcorvus: I parsed it :)19:18
corvusso about dropping this item --19:18
clarkbas mentioned earlier Zuul and friends are now independent projects no longer tied to OpenStack (and consequently technically infra)19:19
corvusi think we're really close to the v3 release at this point19:19
fungito your earlier foreshadowing, i think we should still have a zuul v3 priority effort until v3 is released (possibly longer). there's plenty of work to be done on the implementing for openstack infra side still19:19
clarkbThis has been a long standing agenda item and ya I was going to say maybe we just keep it going through the release?19:19
corvusand i think we've covered everything in the spec (with the exception of some CD stuff -- but arguably, you can accomplish that with secrets now anyway)19:19
fungithere's still a fair amount of job conversion long tail which _could_ justify the priority effort extending past 3.0.0 getting tagged19:20
corvusthe last several meetings this has mostly been an operational topic anyway; less about development.19:20
clarkbthen any operational concerns would be normal agenda items. Going forward. I think the consequence of this is that we should probably reevaluate our priority efforts and see if anything needs to be prioritized.19:20
corvuskeeping this on the list until release and then just adding operational items as they come up seems reasonable.  but also, if we wanted to go ahead and drop it, that works too.19:21
*** VW_ has joined #openstack-meeting19:21
clarkbThis is neat because it means we are almost done with a long standing item and zuul gets to be free and hopefully used by far more people.19:21
AJaegerI think it will come up - somehow. Either with new features or new ways of doing jobs.19:22
fungiyeah, it will be nice to cross it off the in-progress list. i suppose v3 release is a good enough milestone we can call it no longer transitional19:22
AJaegerJust as a heads-up: tox-siblings is current topic that mordred is digging into, this caused a bit more havoc than expected ;(19:22
clarkbAJaeger: ya I think it will remain on topic to talk about how zuul affects infra and openstack. I think bringing this up is more about fungis point19:22
clarkbso rather than standing priority agenda item we'll move to as needed agenda items19:22
AJaegerclarkb: works for me19:23
mordredAJaeger, clarkb: I think we've got tox-siblings fixes almost ready19:23
AJaegermordred: Hope so - just wanted to share with team so that they are aware of it19:23
mordredincidentally, one of the tox-siblings bugs was also present in the tox_install.sh scripts - but nobody had noticed it wasn't doing the right thing19:23
clarkbwhich is good I think that means infra has accomplished the majority of why this was a priority effort in the first place. Which is a major milestone19:23
mordredclarkb: ++19:24
clarkband it is also a transitional step for zuul itself19:24
clarkbwe'll all have to take a drink of our favorite beverage once that tag is cut19:24
* fungi toasts another major success19:24
*** VW has quit IRC19:25
fungioh, i'm early i guess ;)19:25
*** VW_ has quit IRC19:25
clarkbas for zuulv3 topics proper I think the biggest one is pointing out that if you are running a zuulv3 you want to update all your executors to run latest master19:25
clarkbthere were several security bug fixes last week. Infra has updated its executors19:25
clarkbanything else we want to go over re Zuulv3?19:26
corvusand there will be at least one more security fix, hopefully this week19:26
clarkbLook out for that on the zuul mailing list19:27
*** bobh has joined #openstack-meeting19:27
fungiat least one more security fix before 3.0.0 presumably19:27
clarkbya19:27
fungii expect plenty more security fixes in the coming months/years because it's software19:27
clarkb#topic Project Renames19:28
*** openstack changes topic to "Project Renames (Meeting topic: infra)"19:28
* fungi should clearly not seek a side job in public relations19:28
*** psachin has joined #openstack-meeting19:28
clarkbfungi: ha19:28
clarkbfungi has written up a new project rename process. I suppose we should all review that (do we want to wait before merging it after we perform renames?) and mordred has provided specifics for his project rename19:28
*** mjturek has quit IRC19:28
clarkbI think this means we are basically ready to give it a go19:29
mordred\o/19:29
*** awaugama has quit IRC19:29
fungiit's not so much written up ans written out. i just deleted things ;)19:29
clarkbI'm probably not going to be around to help thursday-wednesday if we want to go and do it real quickly but that shouldn't stop us I don't think19:29
corvusoh what's the link?19:29
fungi#link https://review.openstack.org/554261 Update Gerrit project renaming for Zuul v319:29
fungiwas just fishing it out19:30
corvuswill review19:30
fungithanks!19:30
clarkblooking at the release schedule next week should be safe too19:30
*** stevebaker has quit IRC19:30
fungiit's primarily an attempt at encoding what we discussed in the meeting last week and subsequently over the course of the week in #openstack-infra, so hopefully no surprises19:30
clarkband probably the week after that19:30
*** VW has joined #openstack-meeting19:31
clarkbthough I'm somewhat hopefully we'll do it sooner than that just to get past it19:31
corvusi can assist fri, but would prefer not to drive19:31
mordredoh jeez. I have a lot of patches to prepare19:31
fungii've been asked to also help with some storyboard imports friday, so am willing to help but would also rather not be in the hot seat19:31
corvusmordred: should we wait a bit then?19:32
clarkbalso I think mordred is also traveling late this week?19:32
mordredcorvus: no - it's fine - I just need to make rename patches for all the repos listed in fungi's patch19:32
funginext week is starting to sound better19:32
clarkbya maybe March 29 or 3019:32
fungimordred: hopefully most of those don't need patches, it's more a list of places to check whether they need patches19:33
corvusdmsimard: have you done a rename yet?19:33
dmsimardI have not19:33
* mordred is not travelling friday - but is travelling monday - 29 and 30 are both great19:33
fricklerthat would be the easter weekend, not sure who has holidays there19:33
clarkbmordred: oh we'll miss you on the weekend then?19:33
fungidmsimard: [sales pitch] it uses ansible!19:33
* clarkb wonders if he needs to brush up on his zuul19:33
mordredclarkb: no - I'll be in LA on the weekend19:33
mordredclarkb: I'm just flying to LA first thing saturday morning19:33
clarkbmordred: ah just not friday gotcha19:33
clarkbfrickler: arg19:33
* mordred has symphony tickets for friday for the brandenburg concertos19:34
*** yamamoto has joined #openstack-meeting19:34
clarkbmordred: if we wanted to do it friday would you be able to drive it?19:34
* corvus now has bach stuck in his head19:34
clarkb(to avoid easter weekend conflict)19:34
mordredclarkb: yes, I can drive it on friday if need be19:34
fungii have the utmost respect for people of religion, but must admit i have a hard time remembering when religious holidays fall19:34
clarkbfungi: easter is a hard one because it moves around19:35
mordredalso - the followup for openstacksdk for the rename is doing the storyboard transition - so if fungi is already in storyboard transition brainspace, that could be handy19:35
mordredeaster is easy ...19:35
AJaegerclarkb: there's school vacation starting a week earlier for my kids, helps remembering ;)19:35
fungimordred: sure, happy to19:35
mordredit's the first sunday after the first full moon after the vernal equinox19:35
fungimordred: though also, renames in sb are super easy19:35
mordredfungi: nod- I figured I'd just do it after the rename to keep churn low19:36
*** stevebaker has joined #openstack-meeting19:36
clarkbif corvus and fungi are able to help out and mordred can drive on friday I think that gives us enough humans to give it a go19:36
fungiit's, like, one sql update query19:36
clarkband then any other infra-roots wanting to be involved can help19:36
clarkb(I'll likely be floating around too just not persistently)19:36
mordredclarkb: are we past needing 3 hour reindexes now?19:36
clarkbmordred: yes it should happen all online19:36
fungiif people review my updated process draft, i'll happily put together a maintenance plan based on it19:36
mordred\o/19:36
fungimordred: the reindex may take 12 hours, but doesn't require gerrit downtime (though does tend to block git replication)19:37
*** d34dh0r53 has quit IRC19:37
*** d34dh0r53 has joined #openstack-meeting19:37
clarkbits a far less expensive process19:37
pabelangerI'm able to help too19:38
clarkbfriday is weeked for ianw so we should probably let ianw weekend. frickler dmsimard any interest? If so maybe we aim for an early start?19:38
fungidmsimard is in the same tx i am, i think19:38
fungier, same tz19:38
ianwi can be around if earlyish, but not sure i have any special insight for this one19:38
dmsimardI'm on eastern (15:39 currently) and I'm leaving in one minute to pick up the kids from school :p19:38
fungiyeah, that's my tz too19:39
dmsimardI could watch and participate in the rename ? I'm not sure I could drive it on my own19:39
clarkb1500UTC is 8am PDT and early enough that frickler can join in if interested19:39
dmsimardbrb19:39
*** yamamoto has quit IRC19:39
*** mjturek has joined #openstack-meeting19:39
pabelangerwfm19:40
fricklerclarkb: well on a friday thats pretty late for me already, so I'd rather skip that19:40
clarkbproposal: 1500UTC Friday if that works for mordred (as driver) and others that volunteered to help19:40
clarkbfrickler: ah ok19:40
mordredwith only two repos to name - it SHOULD be easy - even with github clicking19:40
fungiif the revised plan is good, it doesn't actually require zuul downtime just a very brief (automated( shutdown and startup for gerrit19:40
clarkbfrickler: I'm thinking it would be good to do it when corvus is awake since it is our first with zuulv319:40
fungiso hopefully people won't be impacted too much19:40
clarkbfungi: which means probably no earlier than 1500UTC19:40
frickleryeah, just plan it according to your needs19:40
clarkbyay tab complete failures19:40
mordredclarkb: 1500UTC wfm19:41
corvus1500utc is fine for me19:41
fungii'm good with whatever time you like. i have no life19:41
clarkbok lets say 1500UTC Friday then19:41
clarkband I'll go ping release team after this meeting19:41
clarkbthanks everyone it will be nice to put this behind us19:41
fungisounds good, thanks!19:41
clarkb#topic General Topics19:41
*** openstack changes topic to "General Topics (Meeting topic: infra)"19:41
*** tssurya has joined #openstack-meeting19:42
clarkbianw: it would probably be good to get a recap on all of the afs changes that have happened19:42
ianwok19:42
fungioh, yeah, i'm sure i missed at least some of the details19:42
ianwfirst thing was i updated mirror-update to use backported bionic era 1.8~pre5 packages19:43
ianwthat is in https://launchpad.net/~openstack-ci-core/+archive/ubuntu/openafs-1.8-xenial19:43
ianwthat actually seemed to get things very stable.  all reprepro's ran for about 3 days without failure19:43
*** diablo_rojo has quit IRC19:44
ianwwe found one weird warning, which after discussion seems harmless, and i've proposed https://gerrit.openafs.org/#/c/12964/19:44
*** psachin has quit IRC19:44
*** stevebaker has quit IRC19:45
ianwafter that, i updated all our fileservers to be running with the settings as suggested by auristor and documented in https://review.openstack.org/54019819:45
pabelangernice19:45
*** stevebaker has joined #openstack-meeting19:45
clarkboh good that did merge19:45
fungiauristor also mentioned that we should be possibly concerned that our afs servers aren't running as new a version as the client?19:45
fungior maybe i misread19:46
ianwyeah, as i understand it, there's just a mismatch between resources in terms of threads and callbacks, with 1.8 having more, that may result in lower performance19:47
clarkbwhich is why we increased resources per that docs change right?19:47
fungioh, right, that the 1.8 client can overrun the number of requests the server is able to accept19:47
fungiokay, so the config change does address that concern?19:47
*** psachin has joined #openstack-meeting19:47
*** Leo_m has quit IRC19:48
ianwclarkb: i think we were just really underspecced to start with, using the defaults.  we may want to tweak things as we move on, now we have the idea that we should19:48
clarkbfungi: that was my understanding. Basically increased the number of server side resources to match the client side increases19:48
fungiexcellent19:48
clarkbianw: gotcha19:48
ianwi've been fiddling with getting some of the stats out of various tools and possibly sticking them into graphite, so we can at least see where we might have issues19:48
*** Leo_m has joined #openstack-meeting19:48
fungithat's super helpful, thanks!19:48
ianwand the last thing is the afs docs jobs, which AJaeger pointed out can be quite unstable, seeming to hang19:49
clarkbianw: those jobs will use the older afs packages on the base distro too right?19:49
ianwwe had a bit of back and forth over what this might be, simultaneous vos releases which corvus pointed out wasn't an issue, etc19:49
ianwit's writing from the executor i think, and it seems the rsync hangs?19:50
ianwit's wrapped under a significant amount of layers of ansible, etc19:50
clarkboh right and we updated the afs and kernel on the executors19:50
clarkbmaybe we need to roll them all the way up to your new 1.8 packages?19:51
ianwanyway ... i will keep an eye on this.  maybe new settings will help.  otherwise i think we need to keep digging to understand the timeout a bit better19:51
clarkbsounds good19:51
*** salv-orlando has quit IRC19:51
*** salv-orlando has joined #openstack-meeting19:52
clarkbAnother topic is cloud changes. dmsimard is in the process of bringing up limestone networks' cloud for use as test resources and we are testing out new flavors on vexxhost which require boot from volume19:52
clarkbpabelanger: ^ vexxhost seems stable now ya? we just had to switch to using raw images?19:52
pabelangerit is!19:52
pabelangerwe have 10 nodes, and boot-from-volume with 80GB volumes19:52
clarkbpabelanger: any indication if the jobs are happier with the new flavors yet?19:52
pabelangerclarkb: they seem to be, I haven't seen any timeouts yet19:53
clarkbnice19:53
pabelangerI also think the new mirror is helping with that too19:53
fungii haven't heard anything else from packethost since we briefed them on our consumption model, but presumably we have them on the way soonish as well19:53
pabelangersince we no longer have BW constraints19:53
clarkband I think the change on how ipv6 works in limestone will help us bootstrap more quickly there19:53
*** e0ne has joined #openstack-meeting19:53
clarkbdmsimard: I'd be curious to know if you have further mirror booting issues with ipv6 there19:54
pabelangeryah, I plan to confirm with mnaser again how many nodes we can get in vexxhost again19:54
dmsimardclarkb: (back from school), yes, I'll give it another try19:54
clarkbianw: any linaro cloud updates?19:54
clarkbianw: I guess at this point its adding limited set of jobs to use the arm nodes?19:54
ianwi released dib 2.12 with all the required changes, and released nb03 from the emergency file19:54
ianwnot sure if builds have happened yet, but will keep an eye there too19:55
ianwother than that, no, i think use as people want.  i have some things i want to do with devstack support, like etcd versions etc19:55
ianws/no/yes/19:55
*** tesseract has quit IRC19:56
pabelangerianw: might want to double check I used the right username / password for nodepool clouds.yaml. I think I did it correctly19:56
*** salv-orlando has quit IRC19:56
clarkbreally quickly before our hour is up19:57
clarkb#topic Open Discussion19:57
*** openstack changes topic to "Open Discussion (Meeting topic: infra)"19:57
ianwpabelanger: yeah, i think i did yesterday, lgtm19:57
fricklerI have a patch up to restore the nodepool runtime graphs https://review.openstack.org/55371819:58
clarkbanything else?19:58
pabelangerI think https://review.openstack.org/554624 will allow us to 2 stage a gerrit install on review-dev01.o.o19:58
fricklerbut failing to find per-provider metrics19:58
fricklerhelp appreciated19:58
clarkb#link https://review.openstack.org/553718 restore nodepool runtime graphs19:58
pabelangerhopefully will pass tests19:58
*** annabelleB has quit IRC19:58
*** stevebaker has quit IRC19:58
*** stevebaker has joined #openstack-meeting19:59
clarkb#link https://review.openstack.org/554624 prereq for booting review-dev0119:59
clarkbcorvus may know about finding per provider metrics. frickler I often find reading the source is the easiest way to sort that out:/20:00
clarkband with that we are at time. Thank you everyone20:00
clarkb#endmeeting20:00
*** openstack changes topic to "Sending periodic "unreleased" reports (Meeting topic: releaseteam)"20:00
openstackMeeting ended Tue Mar 20 20:00:22 2018 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)20:00
openstackMinutes:        http://eavesdrop.openstack.org/meetings/infra/2018/infra.2018-03-20-19.01.html20:00
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/infra/2018/infra.2018-03-20-19.01.txt20:00
openstackLog:            http://eavesdrop.openstack.org/meetings/infra/2018/infra.2018-03-20-19.01.log.html20:00
clarkbfungi: you had the topic fix queued up20:00
fungithanks clarkb!20:00
*** ChanServ changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"20:00
fungiand done20:00
*** AJaeger has left #openstack-meeting20:00
*** TxGirlGeek has quit IRC20:01
*** masber has joined #openstack-meeting20:02
*** Leo_m has quit IRC20:03
*** Leo_m_ has joined #openstack-meeting20:03
*** artom has quit IRC20:03
*** sshank has joined #openstack-meeting20:03
*** TxGirlGeek has joined #openstack-meeting20:03
*** HeOS has joined #openstack-meeting20:04
*** stevebaker has quit IRC20:08
*** stevebaker has joined #openstack-meeting20:08
*** psachin has quit IRC20:08
*** psachin has joined #openstack-meeting20:09
*** Leo_m_ has quit IRC20:13
*** stevebaker has quit IRC20:13
*** Leo_m has joined #openstack-meeting20:14
*** psachin has quit IRC20:14
*** awaugama has joined #openstack-meeting20:15
*** Leo_m has quit IRC20:15
*** sshank has quit IRC20:16
*** Leo_m has joined #openstack-meeting20:16
*** radeks_ has quit IRC20:18
*** iyamahat_ has quit IRC20:20
*** iyamahat_ has joined #openstack-meeting20:20
*** cloudrancher has joined #openstack-meeting20:21
*** stevebaker has joined #openstack-meeting20:23
*** dprince has quit IRC20:25
*** priteau has joined #openstack-meeting20:29
*** stevebaker has quit IRC20:30
*** esberglu has quit IRC20:31
*** eharney has quit IRC20:31
*** masber has quit IRC20:32
*** esberglu has joined #openstack-meeting20:34
*** TxGirlGeek has quit IRC20:34
*** yamamoto has joined #openstack-meeting20:35
*** stevebaker has joined #openstack-meeting20:36
*** TxGirlGeek has joined #openstack-meeting20:36
*** TxGirlGeek has quit IRC20:38
*** ansmith has quit IRC20:41
*** yamamoto has quit IRC20:41
*** stevebaker has quit IRC20:44
*** julim has quit IRC20:47
*** TxGirlGeek has joined #openstack-meeting20:48
*** oneswig has joined #openstack-meeting20:50
*** tssurya has quit IRC20:50
*** edmondsw has quit IRC20:51
*** tssurya has joined #openstack-meeting20:51
*** felipemonteiro_ has joined #openstack-meeting20:51
*** stevebaker has joined #openstack-meeting20:51
*** salv-orlando has joined #openstack-meeting20:52
*** TxGirlGeek has quit IRC20:53
*** felipemonteiro has quit IRC20:54
*** tssurya has quit IRC20:55
*** jamesmcarthur has quit IRC20:55
*** tssurya has joined #openstack-meeting20:55
*** dmellado has joined #openstack-meeting20:56
*** salv-orlando has quit IRC20:57
*** dmellado has quit IRC20:58
*** kaisers has quit IRC20:58
*** TxGirlGeek has joined #openstack-meeting21:01
*** pchavva has quit IRC21:02
*** oneswig has quit IRC21:02
*** dmellado has joined #openstack-meeting21:02
*** b1airo has joined #openstack-meeting21:03
*** jamesmcarthur has joined #openstack-meeting21:03
*** oneswig has joined #openstack-meeting21:03
b1airo#startmeeting scientific-sig21:03
openstackMeeting started Tue Mar 20 21:03:50 2018 UTC and is due to finish in 60 minutes.  The chair is b1airo. Information about MeetBot at http://wiki.debian.org/MeetBot.21:03
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.21:03
*** openstack changes topic to " (Meeting topic: scientific-sig)"21:03
openstackThe meeting name has been set to 'scientific_sig'21:03
b1airomorning!21:04
oneswigHi b1airo - apologies, I am on a train, in a tunnel21:04
*** martial has joined #openstack-meeting21:04
*** jamesmcarthur has quit IRC21:04
b1airo#chair oneswig21:04
openstackCurrent chairs: b1airo oneswig21:04
oneswighighly likely to be intermittently present :-)21:04
*** jamesmcarthur has joined #openstack-meeting21:04
martialsame as stig21:04
oneswighey martial21:05
b1airooh come now oneswig, in Australia the wireless broadband service is generally better than fixed line!21:05
martialHi Stig, Blair21:05
b1airo#chair martial21:05
openstackCurrent chairs: b1airo martial oneswig21:05
oneswigI am not in a train in Australia...21:05
oneswig... alas21:05
b1airo:-)21:05
martialwe are in the "Network Challenges" session of the conference with a presentation from Internet 221:05
*** raildo has quit IRC21:06
b1airoha21:06
oneswigI've heard of such challenges21:06
b1airomartial: did you get anything back from Wilfred/Paul ?21:06
*** sshank has joined #openstack-meeting21:07
martialnothing unfortunately21:07
martialor Enol for that matter21:07
b1airoi've been up until past 3am this morning dealing with storage "challenges", so if no-one else shows up to this meeting in the next 5 mins i'll be happy to cut it short21:07
b1airo*grumble*21:08
martialbut I got the slide deck from Zhipeng about Blockchain use in the Passport program21:08
oneswigI'll be dropping off around then due to continuing my journey on bicyle21:08
oneswigWhat's the news on storage?21:08
*** dmellado has quit IRC21:08
*** mjturek has quit IRC21:08
martialb1airo if interested check goo.gl/nPXeMR21:09
*** stevebaker has quit IRC21:09
b1airojust the delicacy of a Ceph pool with a couple hundred OSDs over 95% full21:09
martialoh Mike would have tons of things to say about that, he had that glitch21:09
*** stevebaker has joined #openstack-meeting21:10
oneswigThere's a chewy problem21:10
b1airowe've added some more OSDs for the moment, but the iffy CRUSH distribution in our present setup means there's quite a period of yoyo-ing before things start to trend down for the most full OSDs21:11
b1airoi saw one at 96.82%21:11
b1airosomewhat too close to the 97% failsafe full ratio21:11
b1airowe tried to copy a 100TB RBD into a new pool on different hardware, but a few OSDs in the source pool didn't seem to like the idea and stopped processing other work for some reason21:12
*** dmellado has joined #openstack-meeting21:12
*** dmellado has quit IRC21:13
*** bobh has quit IRC21:13
b1airoalright, i'll run through the agenda for posterity21:13
oneswigSomeone I saw today was reporting a 30 minute commit latency for IO to their glusterfs. yikes21:13
oneswigBTW b1airo: I thought your roundup on GPU passthrough was really helpful.  Thanks for taking the trouble.21:13
b1airocool, i you thought so then it was probably ok :-)21:14
*** annabelleB has joined #openstack-meeting21:15
*** dmellado has joined #openstack-meeting21:15
*** haleyb has quit IRC21:15
b1airoi was reminded about NVIDIA's HGX-1 architecture afterwards too, but that's not generally available outside the big customers afaik21:15
b1airoapparently they will have some big feature announcements around a 2nd gen of that platform soon21:16
oneswigdon't know it, sorry - not an nvidia fanboi21:16
*** VW has quit IRC21:16
oneswigok, train's in, gotta go21:16
oneswiguntil next time21:16
*** oneswig has quit IRC21:16
martialwe are having a private meeting today indeed :)21:17
*** sshank has quit IRC21:17
martialb1airo: end of meeting?21:19
*** tobberydberg_ has joined #openstack-meeting21:19
*** TxGirlGeek has quit IRC21:20
b1airosry, distracted by a small person21:20
*** TxGirlGeek has joined #openstack-meeting21:20
*** stevebaker has quit IRC21:20
martialnot too much happening today, so I would say let's skip?21:21
b1airo#topic Forum Brainstorming21:21
*** openstack changes topic to "Forum Brainstorming (Meeting topic: scientific-sig)"21:21
*** diablo_rojo has joined #openstack-meeting21:21
b1airomartial: yeah, i'll just write these things up for the logs but we can revisit next week21:21
martialcool21:21
b1airoThe Forum process has started for Vancouver'1821:21
b1airowe're are the brainstorming stage where the community starts throwing out ideas for topics and sessions21:22
martialwhat is the link for the etherpad?21:22
b1airoi've started an etherpad here:21:22
*** tobberydberg_ has quit IRC21:22
b1airo#link https://etherpad.openstack.org/p/YVR18-scientific-sig-brainstorming21:22
b1airowhich is also linked from the Forum wiki page, #link: https://wiki.openstack.org/wiki/Forum/Vancouver2018#Etherpads_from_Teams.2C_Working_Groups.2C_and_SIGs21:23
b1airoso if you have ideas for topics of interest to the Scientific OpenStack ops/user community then please throw them in21:24
b1airoi've taken the liberty of kicking things off with a state of the art/practice on GPUs with OpenStack21:24
b1airo#topic Discussion regarding state of GPU passthrough and interest in collaborating to resolve security issues21:24
*** openstack changes topic to "Discussion regarding state of GPU passthrough and interest in collaborating to resolve security issues (Meeting topic: scientific-sig)"21:24
martialI see that21:24
martialis that a follow up from the conversation you talked to us in Sydney?21:25
b1airopartly, yes21:25
b1aironot really much more to say about this right now beyond what I summarised on-list yesterday, happy to talk more if people have specific questions21:26
b1airoi also know some scant technical details (for specific devices) about a couple of pci config space windows that should be closed to guests, e.g., VBIOS flash21:27
*** kaisers has joined #openstack-meeting21:27
b1airoso that could be a start to adding and testing pci-quirks in QEMU21:27
b1airoi'd really like to find a register or something that does "a bad thing" (tm) that could be used as a demonstrable exploit - that would probably garner some more attention21:28
b1airoand would be a useful test-case for quirks21:28
b1airo#topic AOB21:29
*** openstack changes topic to "AOB (Meeting topic: scientific-sig)"21:29
b1airoAnything else to mention?21:29
martialnot much from my side, the workshop is going very well21:30
martialvery busy21:30
b1airogreat21:30
martialbut we will have the pleasure of doing the second day partially on gotomeeting due to a fair amount of snow21:30
martialthe follow up from this meeting is expected to be at the Vancouver Summit21:31
*** dmellado has quit IRC21:32
*** TxGirlGeek has quit IRC21:32
b1airolook forward to hearing the round-up from it then21:32
*** kaisers has quit IRC21:32
martialyes Sir, will provide some information21:32
b1airowe might have to defer that till next fortnight if we have Zhipeng next week21:32
*** haleyb has joined #openstack-meeting21:32
martialseems to be the case for next time indeed21:32
martialgood news still21:33
*** e0ne has quit IRC21:33
b1airook all done then21:35
b1airocatch you next week!21:35
*** TxGirlGeek has joined #openstack-meeting21:35
b1airo#endmeeting21:35
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"21:35
openstackMeeting ended Tue Mar 20 21:35:24 2018 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)21:35
openstackMinutes:        http://eavesdrop.openstack.org/meetings/scientific_sig/2018/scientific_sig.2018-03-20-21.03.html21:35
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/scientific_sig/2018/scientific_sig.2018-03-20-21.03.txt21:35
openstackLog:            http://eavesdrop.openstack.org/meetings/scientific_sig/2018/scientific_sig.2018-03-20-21.03.log.html21:35
*** martial has quit IRC21:35
*** esberglu has quit IRC21:35
*** TxGirlGeek has quit IRC21:36
*** b1airo has quit IRC21:36
*** yamamoto has joined #openstack-meeting21:37
*** jamesmcarthur has quit IRC21:39
*** kaisers has joined #openstack-meeting21:39
*** esberglu_ has joined #openstack-meeting21:40
*** VW has joined #openstack-meeting21:41
*** yamamoto has quit IRC21:43
*** lpetrut has quit IRC21:44
*** esberglu_ has quit IRC21:45
*** ssbarnea has quit IRC21:46
*** wznoinsk_ has quit IRC21:46
*** HeOS has quit IRC21:46
*** nsingh has joined #openstack-meeting21:47
*** HeOS has joined #openstack-meeting21:48
*** gouthamr has quit IRC21:49
*** niraj_singh has quit IRC21:50
*** VW has quit IRC21:51
*** ssbarnea has joined #openstack-meeting21:52
*** wznoinsk has joined #openstack-meeting21:53
*** salv-orlando has joined #openstack-meeting21:53
*** TxGirlGeek has joined #openstack-meeting21:55
*** ykatabam has joined #openstack-meeting21:58
*** tssurya has quit IRC22:00
*** dmellado has joined #openstack-meeting22:04
*** browny_ has quit IRC22:07
*** jbadiapa has quit IRC22:07
*** rcernin has joined #openstack-meeting22:13
*** bollig has quit IRC22:13
*** diablo_rojo has quit IRC22:13
*** bollig has joined #openstack-meeting22:14
*** diablo_rojo has joined #openstack-meeting22:14
*** priteau has quit IRC22:15
*** priteau has joined #openstack-meeting22:15
*** slagle has quit IRC22:16
*** slagle has joined #openstack-meeting22:17
*** jamesmcarthur has joined #openstack-meeting22:18
*** felipemonteiro_ has quit IRC22:18
*** browny_ has joined #openstack-meeting22:19
*** priteau has quit IRC22:20
*** awaugama has quit IRC22:20
*** dustins has quit IRC22:24
*** TxGirlGeek has quit IRC22:26
*** stevebaker has joined #openstack-meeting22:27
*** TxGirlGeek has joined #openstack-meeting22:28
*** ianychoi__ is now known as ianychoi22:32
*** browny_ has quit IRC22:36
*** TxGirlGeek has quit IRC22:38
*** yamamoto has joined #openstack-meeting22:39
*** jamesmcarthur has quit IRC22:41
*** TxGirlGeek has joined #openstack-meeting22:42
*** ansmith has joined #openstack-meeting22:42
*** sshank has joined #openstack-meeting22:43
*** hongbin has quit IRC22:44
*** yamamoto has quit IRC22:45
*** andreas_s has joined #openstack-meeting22:47
*** Leo_m has quit IRC22:51
*** andreas_s has quit IRC22:51
*** TxGirlGeek has quit IRC22:52
*** TxGirlGeek has joined #openstack-meeting22:55
*** harlowja has quit IRC22:57
*** tpsilva has quit IRC22:57
*** chyka has quit IRC23:00
*** chyka has joined #openstack-meeting23:01
*** julim has joined #openstack-meeting23:01
*** TxGirlGeek has quit IRC23:02
*** fnaval has quit IRC23:02
*** julim has quit IRC23:02
*** masber has joined #openstack-meeting23:03
*** masuberu has joined #openstack-meeting23:04
*** TxGirlGeek has joined #openstack-meeting23:06
*** browny_ has joined #openstack-meeting23:06
*** fnaval has joined #openstack-meeting23:08
*** masber has quit IRC23:08
*** fnaval has quit IRC23:09
*** fnaval has joined #openstack-meeting23:09
*** gman-tx has quit IRC23:12
*** rfolco has quit IRC23:13
*** HeOS has quit IRC23:19
*** jamesmcarthur has joined #openstack-meeting23:24
*** Zames has joined #openstack-meeting23:25
*** jamesmcarthur has quit IRC23:29
*** Anticime1 is now known as Anticimex23:29
*** Zames has quit IRC23:30
*** mriedem has quit IRC23:34
*** yamamoto has joined #openstack-meeting23:41
*** browny_ has quit IRC23:41
*** ansmith has quit IRC23:43
*** chyka has quit IRC23:46
*** yamamoto has quit IRC23:46
*** amodi has quit IRC23:46
*** harlowja has joined #openstack-meeting23:49
*** mriedem has joined #openstack-meeting23:52
*** cloudrancher has quit IRC23:54
*** cloudrancher has joined #openstack-meeting23:54

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!