Tuesday, 2018-08-21

*** gyee has quit IRC00:00
*** ykatabam has quit IRC00:01
*** longkb has joined #openstack-meeting00:05
*** jessegler has quit IRC00:05
*** hongbin has joined #openstack-meeting00:26
*** macza has quit IRC00:41
*** erlon has quit IRC00:41
*** imacdonn has quit IRC00:49
*** imacdonn has joined #openstack-meeting00:49
*** njohnston has quit IRC01:03
*** felipemonteiro has joined #openstack-meeting01:04
*** ykatabam has joined #openstack-meeting01:08
*** bzhao__ has joined #openstack-meeting01:10
*** yamahata has quit IRC01:22
*** felipemonteiro has quit IRC01:27
*** janki has joined #openstack-meeting01:28
*** yamahata has joined #openstack-meeting01:33
*** ricolin has joined #openstack-meeting01:33
*** cloudrancher has quit IRC01:34
*** cloudrancher has joined #openstack-meeting01:35
*** isq has quit IRC01:54
*** Bhujay has joined #openstack-meeting02:00
*** felipemonteiro has joined #openstack-meeting02:00
*** isq has joined #openstack-meeting02:02
*** gcb_ has joined #openstack-meeting02:07
*** yamahata has quit IRC02:15
*** apetrich has quit IRC02:27
*** markstur has joined #openstack-meeting02:28
*** felipemonteiro has quit IRC02:43
*** tpatil has joined #openstack-meeting02:44
*** markvoelker has joined #openstack-meeting03:01
*** psachin has joined #openstack-meeting03:06
*** janki has quit IRC03:06
*** jamesmcarthur has joined #openstack-meeting03:15
*** tpatil has quit IRC03:17
*** jamesmcarthur has quit IRC03:19
*** larainema has joined #openstack-meeting03:21
*** diablo_rojo has quit IRC03:21
*** Bhujay has quit IRC03:24
*** tpatil has joined #openstack-meeting03:33
*** Nel1x has quit IRC03:34
*** markstur has quit IRC03:38
*** samP has joined #openstack-meeting03:56
*** ykatabam has quit IRC03:59
samPhi all for masakari04:01
samP#startmeeting masakari04:01
openstackMeeting started Tue Aug 21 04:01:26 2018 UTC and is due to finish in 60 minutes.  The chair is samP. Information about MeetBot at http://wiki.debian.org/MeetBot.04:01
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.04:01
*** openstack changes topic to " (Meeting topic: masakari)"04:01
openstackThe meeting name has been set to 'masakari'04:01
samPtpatil: hi04:01
samPLet's start the meeting04:01
samP#topic bugs&patches04:02
*** openstack changes topic to "bugs&patches (Meeting topic: masakari)"04:02
samPAny bugs or patches to discuss?04:02
openstackLaunchpad bug 1779752 in masakari-monitors "masakari segment-list returns error (openstack queens)" [Critical,Confirmed] - Assigned to Shilpa Devharakar (shilpasd)04:02
tpatilin the last meeting, we have discussed about patches that needs attention04:02
tpatil^^  Pass region_name and interface parameters during connection initialization04:03
tpatilThis patch is pushed against master branch04:03
samPtpatil: yep, sorry. I will review asap04:03
tpatilafter it's merged, we need to take the decision to back port it to stable/rocky branch04:04
tpatilTo resolve above LP bug, we have proposed fix in patch : https://review.openstack.org/#/c/590735/04:04
tpatilthis is proposed for stable/queens04:05
samPtpatil: got it.04:05
tpatilsame issue also needs to be fixed in masakariclient in stable/queens branch, this is that patch : https://review.openstack.org/#/c/588157/04:05
tpatilRequest you to please take a look at these 3 patches04:06
samPtpatil: sure, I will and sorry for the delay.04:06
tpatilsamP: NP04:07
samPThere are some patches to run python 3 by default, which we need to review and merge04:08
tpatilok, will take a look at it04:09
samPmost of them are LGTM, I will put my review comments.04:09
tpatilis it a community goal to use python3 by default?04:09
samPtpatil: yes04:10
*** hongbin has quit IRC04:10
tpatilTargeted for rocky or stein release?04:10
samPI think Stein, let me ckeck04:10
samP#link https://governance.openstack.org/tc/goals/stein/python3-first.html04:11
samPIt is for Stein04:11
tpatilGot it04:11
*** markstur has joined #openstack-meeting04:11
samPThanks Nguyễn for those patches !!04:12
samPAny other patches need to discuss?04:13
*** ijw has joined #openstack-meeting04:13
samPOtherwise, let move to next topic04:13
samP#topic PTG and Stein Dev items04:14
*** openstack changes topic to "PTG and Stein Dev items (Meeting topic: masakari)"04:14
samPReally sorry that I could not complete the etherpad. Shall we discuss this in next week meeting?04:15
samP#link https://etherpad.openstack.org/p/masakari-ptg-stein04:15
samP^^ still empty ;)04:15
tpatilOK, by that time, I will also add topic from my end that I want to see in stein release04:15
samPtpatil: Thanks04:16
samP#topic Rocky Release04:17
*** openstack changes topic to "Rocky Release (Meeting topic: masakari)"04:17
samPalready have RC1 and stable/rocky cut off04:17
samPSince masakari release job failed, I will discuss with release team and cut RC2 tag.04:18
samPOther than that we are fine for Final Rocky release04:18
samPThat's all from my side.04:19
samPAny other item to discuss today?04:19
samPOK then. Let's finish today's meeting04:20
samPThank you.04:20
samPPlease use openstack-dev ML with [masakari] or #openstack-masakari on IRC for further discussion.04:21
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"04:21
openstackMeeting ended Tue Aug 21 04:21:22 2018 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)04:21
openstackMinutes:        http://eavesdrop.openstack.org/meetings/masakari/2018/masakari.2018-08-21-04.01.html04:21
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/masakari/2018/masakari.2018-08-21-04.01.txt04:21
openstackLog:            http://eavesdrop.openstack.org/meetings/masakari/2018/masakari.2018-08-21-04.01.log.html04:21
samPtpatil: thanks04:21
tpatilsamP: Thank you04:21
*** ykatabam has joined #openstack-meeting04:22
*** felipemonteiro has joined #openstack-meeting04:25
*** tpatil has left #openstack-meeting04:26
*** janki has joined #openstack-meeting04:27
*** psachin has quit IRC04:27
*** markvoelker has quit IRC04:32
*** markvoelker has joined #openstack-meeting04:43
*** Bhujay has joined #openstack-meeting04:45
*** Bhujay has quit IRC04:51
*** markstur has quit IRC04:51
*** markvoelker has quit IRC04:52
*** markstur has joined #openstack-meeting04:52
*** Bhujay has joined #openstack-meeting04:53
*** markstur has quit IRC04:57
*** psachin has joined #openstack-meeting05:14
*** ekcs has quit IRC05:16
*** ekcs has joined #openstack-meeting05:17
*** ekcs has quit IRC05:23
*** ekcs has joined #openstack-meeting05:24
*** ijw has quit IRC05:35
*** janki has quit IRC06:06
*** yamahata has joined #openstack-meeting06:07
*** hyunsikyang has joined #openstack-meeting06:26
*** alexchadin has joined #openstack-meeting06:27
*** rcernin has quit IRC06:38
*** rcernin has joined #openstack-meeting06:41
*** pcaruana has joined #openstack-meeting06:42
*** rcernin has quit IRC06:51
*** ykatabam has quit IRC06:51
*** tssurya has joined #openstack-meeting06:52
*** qwebirc21340 has quit IRC07:08
*** jamesmcarthur has joined #openstack-meeting07:15
*** joxyuki has joined #openstack-meeting07:20
*** jamesmcarthur has quit IRC07:20
*** alexchadin has quit IRC07:21
*** alexchadin has joined #openstack-meeting07:32
*** tetsuro has quit IRC07:36
*** li_jia_le has joined #openstack-meeting07:46
*** dkushwaha_ has joined #openstack-meeting07:55
*** felipemonteiro has quit IRC07:57
*** hadai has joined #openstack-meeting08:00
dkushwaha_#startmeeting tacker08:06
openstackMeeting started Tue Aug 21 08:06:26 2018 UTC and is due to finish in 60 minutes.  The chair is dkushwaha_. Information about MeetBot at http://wiki.debian.org/MeetBot.08:06
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.08:06
*** openstack changes topic to " (Meeting topic: tacker)"08:06
openstackThe meeting name has been set to 'tacker'08:06
*** macza has joined #openstack-meeting08:06
dkushwaha_#topic Roll Call08:07
*** openstack changes topic to "Roll Call (Meeting topic: tacker)"08:07
dkushwaha_who is here for Tacker meeting08:07
li_jia_lehi ,08:07
dkushwaha_li_jia_le, hi08:08
dkushwaha_joxyuki, hi08:08
*** tetsuro has joined #openstack-meeting08:08
*** YanXing_an has joined #openstack-meeting08:08
dkushwaha_YanXing_an, hi08:09
dkushwaha_ok let´s start..08:10
*** e0ne has joined #openstack-meeting08:10
dkushwaha_#chair YanXing_an08:10
openstackCurrent chairs: YanXing_an dkushwaha_08:10
dkushwaha_#topic Announcements08:10
*** openstack changes topic to "Announcements (Meeting topic: tacker)"08:10
*** macza has quit IRC08:11
dkushwaha_Stable/Rocky released for Tacker08:11
dkushwaha_Stable brach cut for Tacker-horion, Tacker server and client08:12
dkushwaha_#topic Rocky Blueprint Status08:12
*** openstack changes topic to "Rocky Blueprint Status (Meeting topic: tacker)"08:12
YanXing_annice, it's time to commit more patches and features08:13
dkushwaha_One of long pending BP could not be completed in Rocky https://review.openstack.org/#/c/486924/08:13
dkushwaha_I will plan it on priority.08:14
dkushwaha_YanXing_an, yes we can commit now in master branch08:15
YanXing_andkushwaha_, when will we have a vPTG?08:15
dkushwaha_YanXing_an, not planned yet08:16
dkushwaha_i have just created etherpad link08:16
dkushwaha_Dear all, please start putting your ideas, topic on link08:17
YanXing_anyou can use #info08:17
dkushwaha_YanXing_an, ok08:17
joxyukidkushwaha_, thanks for the link08:18
dkushwaha_#info  created etherpad link for Stein vPTG08:18
dkushwaha_#link https://etherpad.openstack.org/p/Tacker-PTG-Stein08:18
dkushwaha_joxyuki, do you have something to talk about reservation BP ?08:20
joxyukidkushwaha_, yea, I am planning to retry the BP during Stein cycle.08:20
dkushwaha_joxyuki, great.08:21
dkushwaha_if required i also can join that08:21
dkushwaha_#topic open discussion08:22
*** openstack changes topic to "open discussion (Meeting topic: tacker)"08:22
YanXing_anli_jia_le, you can consider some BPs that can be realized in Stein08:22
dkushwaha_it will be great for me if somebody can help me to re-spin this bp  https://review.openstack.org/#/c/486924/08:24
li_jia_leYanXing_an: OK08:25
li_jia_lehttps://review.openstack.org/#/c/486924/  I can work on it. dkushwaha_08:25
dkushwaha_li_jia_le, Thanks08:26
dkushwaha_joxyuki, YanXing_an li_jia_le Do you have something to talk?08:27
dkushwaha_Thanks Folks , closing this meeting08:29
*** joxyuki has left #openstack-meeting08:29
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"08:29
openstackMeeting ended Tue Aug 21 08:29:41 2018 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)08:29
openstackMinutes:        http://eavesdrop.openstack.org/meetings/tacker/2018/tacker.2018-08-21-08.06.html08:29
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/tacker/2018/tacker.2018-08-21-08.06.txt08:29
openstackLog:            http://eavesdrop.openstack.org/meetings/tacker/2018/tacker.2018-08-21-08.06.log.html08:29
*** hadai has quit IRC08:30
*** electrofelix has joined #openstack-meeting08:33
*** dbecker has quit IRC08:34
*** li_jia_le has quit IRC08:36
*** li_jia_le has joined #openstack-meeting08:41
*** macza has joined #openstack-meeting08:48
*** alexchadin has quit IRC08:53
*** macza has quit IRC08:53
*** macza has joined #openstack-meeting09:09
*** macza has quit IRC09:14
*** alexchadin has joined #openstack-meeting09:14
*** apetrich has joined #openstack-meeting09:23
*** macza has joined #openstack-meeting09:30
*** dkushwaha_ has left #openstack-meeting09:31
*** macza has quit IRC09:35
*** YanXing_an has quit IRC09:40
*** mriedem has joined #openstack-meeting09:42
*** li_jia_le has quit IRC10:03
*** macza has joined #openstack-meeting10:12
*** jamesmcarthur has joined #openstack-meeting10:15
*** macza has quit IRC10:16
*** jamesmcarthur has quit IRC10:20
*** alexchadin has quit IRC10:25
*** Bhujay has quit IRC10:26
*** macza has joined #openstack-meeting10:33
*** alexchadin has joined #openstack-meeting10:36
*** dbecker has joined #openstack-meeting10:37
*** macza has quit IRC10:38
*** macza has joined #openstack-meeting10:54
*** macza has quit IRC10:59
*** Bhujay has joined #openstack-meeting11:13
*** macza has joined #openstack-meeting11:35
*** e0ne has quit IRC11:36
*** macza has quit IRC11:40
*** tpsilva has joined #openstack-meeting11:44
*** longkb has quit IRC11:50
*** larainema has quit IRC11:54
*** macza has joined #openstack-meeting12:04
*** numans_ has joined #openstack-meeting12:07
*** macza has quit IRC12:08
*** numans has quit IRC12:10
*** macza has joined #openstack-meeting12:11
*** macza has quit IRC12:12
*** raildo has joined #openstack-meeting12:17
*** macza has joined #openstack-meeting12:35
*** ssbarnea is now known as ssbarnea|ruck12:35
*** macza has quit IRC12:40
*** jamesmcarthur has joined #openstack-meeting12:47
*** efried is now known as efried_goatin13:03
*** jamesmcarthur has quit IRC13:04
*** r-mibu has joined #openstack-meeting13:15
*** psachin has quit IRC13:17
*** arne_wiebalck has quit IRC13:17
*** arne_wiebalck has joined #openstack-meeting13:19
*** rossella_s has joined #openstack-meeting13:19
*** dustins has joined #openstack-meeting13:22
*** eharney has quit IRC13:26
*** erlon has joined #openstack-meeting13:28
*** apetrich has quit IRC13:33
*** efried_goatin is now known as efried13:37
*** e0ne has joined #openstack-meeting13:40
*** jamesmcarthur has joined #openstack-meeting13:44
*** awaugama has joined #openstack-meeting13:49
*** alexchadin has quit IRC13:55
*** longkb has joined #openstack-meeting13:57
*** alexchadin has joined #openstack-meeting13:58
*** felipemonteiro has joined #openstack-meeting13:59
*** felipemonteiro has quit IRC14:07
*** jbadiapa has joined #openstack-meeting14:11
*** alexchadin has quit IRC14:13
*** alexchadin has joined #openstack-meeting14:13
*** annabelleB has joined #openstack-meeting14:23
*** eharney has joined #openstack-meeting14:23
*** felipemonteiro has joined #openstack-meeting14:26
*** munimeha1 has joined #openstack-meeting14:26
*** felipemonteiro has quit IRC14:32
*** annabelleB has quit IRC14:33
*** annabelleB has joined #openstack-meeting14:41
*** Leo_m has joined #openstack-meeting14:54
*** felipemonteiro has joined #openstack-meeting14:56
*** alexchadin has quit IRC14:56
*** hongbin has joined #openstack-meeting14:56
*** yamahata has quit IRC14:57
*** jamesmcarthur has quit IRC14:57
*** alexchadin has joined #openstack-meeting14:57
*** felipemonteiro has quit IRC15:02
*** felipemonteiro has joined #openstack-meeting15:03
*** alexchadin has quit IRC15:03
*** felipemonteiro has quit IRC15:05
*** jamesmcarthur has joined #openstack-meeting15:07
*** pcaruana has quit IRC15:09
*** priteau has joined #openstack-meeting15:13
*** sambetts|afk is now known as sambetts15:16
*** markstur has joined #openstack-meeting15:23
*** Bhujay has quit IRC15:27
*** felipemonteiro has joined #openstack-meeting15:30
*** felipemonteiro has quit IRC15:32
*** jbadiapa has quit IRC15:33
*** electrofelix has quit IRC15:37
*** electrofelix has joined #openstack-meeting15:37
*** yamahata has joined #openstack-meeting15:38
*** dklyle has quit IRC15:41
*** armax has joined #openstack-meeting15:42
*** dklyle has joined #openstack-meeting15:44
*** e0ne has quit IRC15:44
*** strigazi has quit IRC15:46
*** strigazi has joined #openstack-meeting15:46
*** longkb has quit IRC15:49
*** gyee has joined #openstack-meeting15:56
*** mlavalle has joined #openstack-meeting15:59
slaweq#startmeeting neutron_ci16:00
openstackMeeting started Tue Aug 21 16:00:18 2018 UTC and is due to finish in 60 minutes.  The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot.16:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:00
*** openstack changes topic to " (Meeting topic: neutron_ci)"16:00
openstackThe meeting name has been set to 'neutron_ci'16:00
*** macza has joined #openstack-meeting16:00
* mlavalle has to leave in 40 minutes for an appointement16:00
slaweqok, mlavalle so we will try to do it fast :)16:01
slaweqlets start then16:01
slaweq#topic Actions from previous meetings16:01
*** openstack changes topic to "Actions from previous meetings (Meeting topic: neutron_ci)"16:01
slaweqnjohnston to tweak stable branches dashboards16:01
slaweqI don't know if he did something about that but I don't think so16:02
slaweqI pinged njohnston on neutron channel, maybe he will join16:03
*** njohnston has joined #openstack-meeting16:03
slaweqhi njohnston16:03
slaweqwe are talking about actions from previous meeting16:03
mlavallehe always shows up16:03
njohnstonsorry I'm late, was engrossed in code :-)16:04
slaweqand we have:16:04
slaweqnjohnston to tweak stable branches dashboards16:04
slaweqno problem :)16:04
njohnstonGah, I forgot all about that.  My apologies.  I'll do it today.16:04
*** r-mibu has quit IRC16:04
slaweqsure, fine16:04
slaweqI will add it for next week then16:04
slaweq#action njohnston to tweak stable branches dashboards16:04
slaweqnext action was:16:05
slaweqslaweq add data about slow API to etherpad for PTG16:05
slaweqI added it to etherpad but I still need to add some data about what tests and what API calls are slowest16:05
*** markstur has quit IRC16:05
slaweqthere are also patches to improve that in progress so I will also point to them there16:06
mlavallethat's good, thanks16:06
*** ricolin has quit IRC16:06
slaweqok, next one was:16:06
slaweqslaweq will add number of failures to graphana16:06
slaweqI added such graphs to grafana16:06
*** markstur has joined #openstack-meeting16:06
slaweqit shows summarize of jobs from last 24h16:06
slaweqlets use it for a while and tweak if that will be necessary16:07
mlavalleyeah, we need to give it some time16:07
slaweqI also did small reorganization of graphs there16:08
slaweqand moved integration tests to one graph and scenario jobs to separate graph16:08
slaweqI wanted to make same graphs for "check" and "gate" queue and I think that now it is like that16:09
mlavalleyes, good that you are tweaking it16:09
slaweqmaybe would be better to move gate queue to separate dashboard even because there is a lot of graphs on it now16:09
slaweqbut lets see how it will work like that for few weeks16:09
njohnstonit definitely looks nicer +116:10
mlavalleI have always found the dasboard and some panels too loaded with data16:10
mlavalleI'm not that smart16:10
mlavalleso I like that you are simplifying it16:10
slaweqI'm trying but I'm not the best person for tweaking UX :P16:11
mlavallewell, as long as we get something we all understand easily, don't worry about UX orthodoxies16:11
slaweqok, thx :)16:11
slaweqI will remember that16:11
*** markstur has quit IRC16:11
slaweqok, last action from last week was:16:12
slaweqslaweq to report new bug about fullstack test_securitygroup(linuxbridge-iptables) issue and investigate this16:12
slaweqBug reported already: https://bugs.launchpad.net/neutron/+bug/1779328 I just updated it16:12
openstackLaunchpad bug 1779328 in neutron "Fullstack tests neutron.tests.fullstack.test_securitygroup.TestSecurityGroupsSameNetwork fails" [High,Confirmed]16:12
slaweqI also did small patch to enable debug_iptables_rules in L2 agent config16:12
slaweqit was merged today or yesterday16:12
slaweqso if it will fail again, I hope that this debug option will help me to figure out what is wrong there16:13
mlavalleso are you planning to work on it?16:13
slaweqyes, I have an eye on that for now16:13
mlavalleok, I'll assign you to it and mark it as in progress16:13
slaweqok, thx16:13
slaweqI forgot about that16:13
slaweqit's always iptables driver which is failing, never openvswitch firewall driver16:14
slaweqI suppose that it's some race condition again but have no idea what exactly16:14
slaweqok, lets talk about grafana now16:15
slaweq#topic Grafana16:15
*** openstack changes topic to "Grafana (Meeting topic: neutron_ci)"16:15
mlavalleyaay, the new and improved16:15
slaweqAs I said, I reorganized it a bit recently16:15
slaweqand I wanted to ask about one thing also16:15
slaweqI found that in available metrics are also metrics like jobs TIMED_OUT and POST_FAILURE16:16
slaweqshould we include them in our graphs?16:16
slaweqfor now we have it calculated as (SUCCESS / (FAILURE+SUCCESS))16:16
mlavallewell the danger there is that we overload the panels with data16:16
slaweqso in graphs we don't see jobs which had TIMEOUT or POST_FAILURE16:16
njohnstonCan we use wildcards in the selection of metric names?  If so we could use a mid-string wildcard to count all fo the TIMED_OUT and POST_FAILURE for all the jobs and just count the number of them occurring as a sort of "healthcheck on zuul" graph16:17
mlavallenow, timed out and port failures are infra issues, aren't they?16:17
mlavallepost failures ^^^^16:17
slaweqbut I was thinking about somethink like ((FAILURE + TIME_OUT + POST_FAILURE) / (FAILURE + TIME_OUT + POST_FAILURE + SUCCESS))16:17
mlavallefailures are our problem16:18
slaweqpost failure is infra issue usually16:18
mlavalleTIME_OUT + POST_FAILURE are infra's, or am I wrong?16:18
slaweqbut timeout is mostly our issue (slow api for example)16:18
mlavalleok, I buy that16:18
slaweqand time_out is what we hit more often16:19
njohnstonwe should still have the tests get killed by the internal timer and register as a normal FAILURE instead of a TIMED_OUT16:19
mlavalleso if we can organize this in such a way that we can easily discriminate what we need to worry about and what we need to communicate to infra, I am all for it16:19
slaweqnjohnston: yes, but if job is taking long time it reaches some kind of "global" timeout and job is killed16:19
slaweqso updating to something like ((FAILURE + TIME_OUT) / (FAILURE + TIME_OUT + SUCCESS)) [in %]16:20
mlavallethat formula seems right to indicate "what we need to worryt about"16:21
slaweqwe will then have percentage of failures and timeouts for all our jobs which isn't post_failure (so percentage of our issues)16:21
slaweqok, so I will update it in grafana16:21
slaweq#action slaweq to update grafana dashboard to ((FAILURE + TIME_OUT) / (FAILURE + TIME_OUT + SUCCESS))16:21
mlavallenjohnston: you agree with that formula?16:21
mlavallewell, let's move on16:23
slaweqI think we lost njohnston :/16:23
njohnstonyes that is good16:23
slaweqspeaking about grafana there is no new "big" issues IMO16:24
slaweqso let's talk about some specific jobs now16:24
slaweq#topic functional16:24
*** openstack changes topic to "functional (Meeting topic: neutron_ci)"16:24
slaweqI want to start with functional because there is one urgent issue with it16:24
slaweqin stable/queens it looks it's failing 100% times since few days16:25
slaweqbug reported https://bugs.launchpad.net/neutron/+bug/178818516:25
openstackLaunchpad bug 1788185 in neutron "[Stable/Queens] Functional tests neutron.tests.functional.agent.l3.test_ha_router failing 100% times " [Critical,Confirmed]16:25
slaweqexample of failure: http://logs.openstack.org/78/593078/1/check/neutron-functional/28fe681/logs/testr_results.html.gz16:25
slaweqlast patch merged to queens branch was https://review.openstack.org/#/c/584276/16:27
slaweqwhich IMO can be potential culprit16:28
slaweqbut it has to be checked16:28
mlavallenobody working on it?16:29
mlavallethe bug I mean16:29
mlavalleit has no assignee16:29
slaweqI reported it today, few hours ago16:30
slaweqand I hadn't got time to work on it yet16:30
mlavalledo you have bandwidth? I can help if you don't16:30
slaweqwould be great if You could check that16:31
mlavalleok, I'll take a stab at it16:31
mlavallejust assigned it to me16:31
mlavalleI'll yell for help if I get myself in trouble16:31
slaweqok, let's quickly move for other issues16:33
*** annabelleB has quit IRC16:33
slaweqI recently found few times issues in neutron.tests.functional.agent.l3.test_metadata_proxy.UnprivilegedUserGroupMetadataL3AgentTestCase16:34
slaweqbut this should be "fixed" by patch https://review.openstack.org/#/c/586341/4 :)16:34
slaweqso please review it16:34
mlavalleadded it to my pile16:35
mlavallewill look at it when I come back after my appointment16:35
njohnstonI've been seeing issues with neutron_tempest_plugin.scenario.test_dns_integration.DNSIntegrationTests.test_server_with_fip but I haven't been able to track it to a source16:35
njohnstonIt's happened about 40 times in the last week; I added an elastic-recheck change to look for it https://review.openstack.org/59372216:36
slaweqnjohnston: I know, it's even reported on https://bugs.launchpad.net/bugs/1788006 :)16:36
openstackLaunchpad bug 1788006 in neutron "neutron_tempest_plugin DNS integration tests fail with "Server [UUID] failed to reach ACTIVE status and task state "None" within the required time ([INTEGER] s). Current status: BUILD. Current task state: spawning."" [Undecided,New]16:36
njohnstonyeah, I dropped that bug when I exceeded my timebox on tracking down the source of the issue16:36
mlavallemhhh the description sound like a nova issue16:36
mlavallenot saying it is not our issue16:37
mlavallebut seems odd16:37
slaweqyep, it looks so at first glance16:37
slaweqbut we should check it16:37
slaweqanyone has bandwidth for it?16:37
mlavalledon't have much, but I'll try to take a look16:38
njohnstonI could ping the nova-neutron liaison16:38
njohnstonI believe that is sean-k-mooney16:38
mlavalleI'll takle a quick look before pinging sean-k-mooney16:38
njohnstonthanks much16:38
slaweqthx mlavalle16:38
slaweqand thx njohnston :)16:38
mlavalleok, I have to leave guys16:39
slaweq#action mlavalle to check neutron_tempest_plugin.scenario.test_dns_integration.DNSIntegrationTests.test_server_with_fip issue16:39
slaweqok, thx mlavalle16:39
slaweqit's basically all what I had for today from important things16:39
slaweqso njohnston if You don't have anything to talk about I think we can finish earlier today16:40
njohnstonnothing from me, it just seems like we have a lot of timeouts of late16:40
njohnstonanyhow, have a good evening slaweq, and thanks as always!16:41
slaweqok, thx njohnston16:41
slaweqand have a nice day :)16:41
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"16:42
openstackMeeting ended Tue Aug 21 16:42:04 2018 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:42
openstackMinutes:        http://eavesdrop.openstack.org/meetings/neutron_ci/2018/neutron_ci.2018-08-21-16.00.html16:42
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/neutron_ci/2018/neutron_ci.2018-08-21-16.00.txt16:42
openstackLog:            http://eavesdrop.openstack.org/meetings/neutron_ci/2018/neutron_ci.2018-08-21-16.00.log.html16:42
slaweqsee You16:42
*** annabelleB has joined #openstack-meeting16:43
*** felipemonteiro has joined #openstack-meeting16:48
*** markvoelker has joined #openstack-meeting16:50
*** sambetts is now known as sambetts|afk16:52
*** felipemonteiro has quit IRC16:52
*** jamesmcarthur has quit IRC17:14
*** yamahata has quit IRC17:17
*** Namrata has joined #openstack-meeting17:39
*** Namrata has quit IRC17:39
*** electrofelix has quit IRC17:39
*** e0ne has joined #openstack-meeting17:42
*** cloudrancher has quit IRC17:48
*** jamesmcarthur has joined #openstack-meeting17:49
*** cloudrancher has joined #openstack-meeting17:49
*** cloudrancher has quit IRC17:50
*** tssurya has quit IRC17:50
*** cloudrancher has joined #openstack-meeting17:51
*** cloudrancher has quit IRC17:52
*** cloudrancher has joined #openstack-meeting17:53
*** cloudrancher has quit IRC17:53
*** cloudrancher has joined #openstack-meeting17:53
*** cloudrancher has quit IRC17:54
*** portdirect has quit IRC17:54
*** numans_ has quit IRC17:54
*** cloudrancher has joined #openstack-meeting17:54
*** cloudrancher has quit IRC17:55
*** diablo_rojo has joined #openstack-meeting17:56
*** cloudrancher has joined #openstack-meeting17:56
*** cloudrancher has quit IRC17:56
*** yamahata has joined #openstack-meeting17:58
*** apetrich has joined #openstack-meeting18:02
*** markstur has joined #openstack-meeting18:07
*** jamesmcarthur has quit IRC18:08
*** jbadiapa has joined #openstack-meeting18:09
*** mjturek has joined #openstack-meeting18:10
*** markstur has quit IRC18:12
*** Shrews has joined #openstack-meeting18:26
*** rledisez has quit IRC18:37
*** lamt has quit IRC18:37
*** Adri2000 has quit IRC18:37
*** leifz has quit IRC18:37
*** vkmc has quit IRC18:37
*** ttx has quit IRC18:37
*** kencjohnston has quit IRC18:37
*** fungi has quit IRC18:37
*** kencjohnston_ has joined #openstack-meeting18:38
*** rledisez has joined #openstack-meeting18:38
*** vkmc has joined #openstack-meeting18:40
*** kota_ has quit IRC18:42
*** TheJulia has quit IRC18:42
*** tobberydberg has quit IRC18:42
*** hrybacki has quit IRC18:42
*** lsell__ has quit IRC18:42
*** mnaser has quit IRC18:42
*** AJaeger has joined #openstack-meeting18:42
*** kota_ has joined #openstack-meeting18:46
*** fungi has joined #openstack-meeting18:48
*** eharney has quit IRC18:52
*** isq_ has joined #openstack-meeting18:54
*** e0ne has quit IRC18:57
*** isq has quit IRC18:57
*** sdatko has joined #openstack-meeting18:57
*** isq_ has quit IRC18:58
*** isq has joined #openstack-meeting18:58
*** eharney has joined #openstack-meeting18:59
*** priteau has quit IRC19:00
fungiinfra team, assemble!19:00
fungisince it looks like nobody volunteered to chair, i'm falling on that grenade. bear with me as i'm wholly unprepared19:01
fungilooks like i'm also the only one to propose a topic this week (one which i added just now)19:01
ianwoh sorry, could have done it :/  thought no news was good news :)19:01
*** TheJulia has joined #openstack-meeting19:01
fungias i'm the only one with a general topic, i suppose it's only fair that i chair19:02
sdatkohello o/19:02
fungihi sdatko!19:02
fungi#startmeeting infra19:02
openstackMeeting started Tue Aug 21 19:02:25 2018 UTC and is due to finish in 60 minutes.  The chair is fungi. Information about MeetBot at http://wiki.debian.org/MeetBot.19:02
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.19:02
*** openstack changes topic to " (Meeting topic: infra)"19:02
openstackThe meeting name has been set to 'infra'19:02
fungi#link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting19:02
fungi#topic Announcements19:02
*** openstack changes topic to "Announcements (Meeting topic: infra)"19:02
fungi#info clarkb (our peerless feeder, i mean fearless leader) is on vacation August 20-24, this week19:03
fungiwhich is why you get to deal with your illustrious substitute meeting chair19:03
* mordred waves19:03
fungi#topic Actions from last meeting19:04
*** openstack changes topic to "Actions from last meeting (Meeting topic: infra)"19:04
fungi#link http://eavesdrop.openstack.org/meetings/infra/2018/infra.2018-08-14-19.01.html Minutes from last meeting19:04
fungi1. (none)19:05
fungiwell, that's exciting. i guess we did all those ;)19:05
fungi#topic Specs approval19:05
*** openstack changes topic to "Specs approval (Meeting topic: infra)"19:05
fungiwe don't seem to have any specs up this week on the agenda for infra council rollcall, though there are a couple active under review at present19:06
fungi#link https://review.openstack.org/563849 Direction setting for 3rd Party CI (ianw)19:06
fungi#link https://review.openstack.org/587283 A spec for exploring the specifics of letsencrypt support (ianw)19:07
ianwi haven't received any particular comments on the section about the goals of 3rd party ci19:07
fungiianw: i know i still owe you a rough implementation writeup of my alternative for that second one19:07
fungiin the process of drafting it as a review comment so as not to get in the way of you addressing the other comments on that most recent patchset19:08
ianwfungi: ok, i can update and am happy for you to upload directly19:08
fungithat's fine too. if you want to address the other comments in there however and then i'll do a patchset on top of that with the alternative as a cohesive bit under the alternatives section (which is now remarkably sparse anyway)19:09
*** mnaser has joined #openstack-meeting19:09
fungianyway, unless anyone wants to bring up anything specs related right now, i'll move on since i think the priority efforts discussion is likely to take up some time19:10
fungi#topic Priority Efforts - A Task Tracker for OpenStack19:11
*** openstack changes topic to "Priority Efforts - A Task Tracker for OpenStack (Meeting topic: infra)"19:11
fungi#link http://specs.openstack.org/openstack-infra/infra-specs/specs/task-tracker.html19:11
*** jamesmcarthur has joined #openstack-meeting19:11
diablo_rojoToday on the ML I suggested that now since everything else is in flux for Searchlight and Trove, might be a good time to migrate.19:12
fungithe fix for 404 errors from the "Jump to..." box landed last week in the webclient19:13
diablo_rojoYes! That is exciting too19:13
fungias did one for non-array team_id and user_id parameters and something about disabling closing modals when you click the backdrop19:13
fungii think the fontawesome update is also close to landing19:13
diablo_rojoWill be wonderful to have all that landed.19:14
fungiwhich is the bottom of a fairly large stack SotK has up for review to improve the webclient19:14
*** dmsimard has joined #openstack-meeting19:14
diablo_rojoLooks like Fatema has two patches ready for review too- she also said that she would try to stick around as much as she can through school- she doesn't plan on disappearing now that her internship is over.19:15
fungithat's excellent news19:15
diablo_rojoI think she likes our community or something ;)19:15
fungiusually interns can't wait to get away from us (just kidding, usually they simply get really busy with their studies, which is entirely understandable)19:15
*** jamesmcarthur has quit IRC19:15
diablo_rojoYeah she has till like the end of Sept before school starts19:16
diablo_rojoand she will try to stick around past then too but time will tell19:16
fungipressure is also mounting to implement some swift integration or something like that to handle story attachments19:16
diablo_rojoYeah I think that is the next big target. We can talk about that at the PTG.19:17
fungiso if anybody's interested, reviewing open changes and maybe hacking on attachments support would be great places to contribute to the effort19:17
fungi(start with writing a spec for attachments, please)19:17
fungiwe good on sb updates for the week?19:17
diablo_rojoYeah I think so19:18
* diablo_rojo sits down at the back of the room19:18
fungistay standing, we're almost sure to volunteer you for things! ;)19:18
*** mlavalle has left #openstack-meeting19:18
fungi#topic Priority Efforts - Update Config Management19:18
*** openstack changes topic to "Priority Efforts - Update Config Management (Meeting topic: infra)"19:18
fungi#link http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html19:18
* diablo_rojo sinks lower in her seat19:18
fungihere's where i expect we'll spend a good portion of our hour today19:19
mordredfungi: it's done!19:19
fungithere has been a continued flurry of activity on this19:19
mordredwe're completely migrated and everything is perfect19:19
corvusit's done depending on exactly how you dereference "it"19:19
cmurphyoh that's good news19:19
* fungi waits for the other shoe to drop19:19
* cmurphy goes on vacation19:19
mordredcorvus: I think life is easier with vague antecedents to our pronouns19:19
AJaegerso, anything we should not merge currently? Any caveats for reviewers?19:19
mordredok - seriously ...19:20
*** bnemec has quit IRC19:20
mordredthe cronjob on bridge.openstack.org isn't live yet - but I think we're mostly good and should be able to land it post-meeting19:20
*** bnemec has joined #openstack-meeting19:20
fungiyeah, i was +2 on it but didn't want to single-core merge that one particularly19:21
mordredAJaeger: I'd still wait until the cron patch has gone live for new projects - but otherwise, no19:21
mordredfungi: ++19:21
mordredif people want to review a GIANT patch related to iptables rules ...19:21
cmurphythat sounds very fun and not at all terrible19:22
fungii've unfortunately had to save looking at that one until my evening. today has gotten away from me rather quickly19:22
mordredhttps://review.openstack.org/#/c/593973 and https://review.openstack.org/#/c/594340 and https://review.openstack.org/#/c/594437 and https://review.openstack.org/#/c/594438 are fun19:22
AJaegermordred: ok, just tell once ready...19:22
mordredcmurphy: ^^ there you go - all the fun you've ever wanted19:22
mordredAJaeger: will do19:22
cmurphyre puppet there are a few lingering topic:puppet-4 patches (the ones verified+1 and without workflow-1) that are just adding tests that should be easy reviews19:23
corvusmordred: after snmpd and iptables, are there any other base puppet things to move to ansible?19:23
cmurphyand if anyone is brave there is a series to turn on the future parser on several more nodes19:23
mordredalso - the groups.yaml file is starting to become a bit unwieldy - so we might need to make a new inventory plugin to let us express some groups more easier19:23
ianwmordred: is there any reason why that can't be tested?19:24
mordredianw: nope - and in fact, at least some of it is being tested19:24
ianwI have a moderately large stack starting https://review.openstack.org/#/c/593487/ that uses roles out of system-config19:24
mordredcorvus: afs (which ianw's work should do for us)19:25
mordredcorvus: and unbound19:25
ianwthere are a few things that need doing to get the roles documented and usable19:25
corvusmordred: afs isn't used on every server19:25
corvusbut unbound is19:25
mordredyah- oh - and unattended-upgrades19:25
fungibup might be nice to do early on as well?19:26
corvusi think my question is this -- the next milestone we should be working toward is "bridge configured as a standard host and everything on bridge configured by ansible", yeah?19:26
mordredso unbound/unattended-upgrades/timezone/ntp are the last every-server things - afs isn't on every server but is curently plumbed through openstack_project::server so getting it transitioned will get us off of openstack_project::server19:26
fungior did bup get done already (is bridge being backed up)?19:26
mordredfungi: no - I don't think bup has been done- and I don't see it in openstack_project::server - so yeah, we should put that up on the list19:27
mordredcorvus: I like your milestone19:27
fungifwiw, backing up puppetmaster had been a very-long-standing todo anyway19:27
mordredI think 'bridge configured as standard host' is one milestone and 'everything on bridge configured by ansible' is a good followup19:28
fungimostly because we wanted to avoid backing up cleartext sensitive file content (passwords, keys...)19:28
corvusmordred: oh are we running puppet on bridge?19:28
mordredsince there are some things that bridge+others all need, like ntp or unbound - and there are things thare are specific to bridge that got done by hand19:28
mordredcorvus: nope - there is no puppet on bridge - there are just some things that got copied over from puppetmaster - like the clouds.yaml file19:29
corvusmordred: oh, i thought all the stuff that was done by hand was captured.19:29
corvusi see19:29
ianwmordred: sorry, crossing streams ... if the iptables rule was in a top level role/ directory i feel like it could be tested with our usual integration jobs (like what i've proposed for afs+kerberos)19:29
fungi#agreed next milestone is bridge configured as standard host, with everything on bridge configured by ansible (including anything which was initially configured manually)19:29
fungii can #undo that if we don't actually agree19:29
mordredianw: yup. I agree - I think we can land the move-roles-dir patch pretty much any time19:29
corvusmordred: what's "timezone"?19:30
ianwmordred: well, see everything i've stacked ontop of it first, having actually used it :)19:30
mordredcorvus: setting the system timezone to utc19:30
corvuswow we really have a puppet-timezone module19:30
corvusi didn't know/remember that19:30
mordredclass { 'timezone': timezone => 'Etc/UTC', } ... yeah19:30
corvus#info fundamental modules remaining to port to ansible: iptables (in progress), snmp (in progress), unbound, unattended-upgrades, timezone, ntp19:31
fungithat does indeed seem like overkill19:31
corvusthat look right ^?19:31
fungiwfm, unless we also want bup in there on principle19:31
*** armax has quit IRC19:31
corvusfungi: i'd like to draw the line before bup since it's not strictly required for every server19:31
fungibut i also am good with considering backups of bridge.o.o as a separate milestone, since we need to be careful about _what_ we back up from there19:32
corvus(i'm mostly trying to come up with work units for the proposed milestone)19:32
fungiyeah, i think that's perfectly fine19:32
*** armax has joined #openstack-meeting19:33
corvusi think getting this work done gets us to the point where the tree can branch and we can do more interesting things at the PTG19:33
mordredtotally agre19:33
fungithat will be awesome19:33
*** melwitt has quit IRC19:33
*** sdake has quit IRC19:33
mordredonce the base stuff is done, hacking on migrating individual services is much more parallelizable19:34
*** melwitt has joined #openstack-meeting19:34
*** sdake has joined #openstack-meeting19:34
fungiany specific items on that set of work units we want to dive into in the meeting?19:35
fungianything we should be on the lookout for?19:36
corvusi'll finish working on snmp/iptables; i think the others are up for grabs19:36
fungiianw: were your testing concerns addressed above?19:36
corvusi'd like to talk more about testing19:36
fungiplease do19:36
ianwIMO we should start out moving these to top level roles/ and test them from zuul as much as possible, and also have things like the readme written with an eye to them being generic-ish.  I know we can race ahead and leave all that for later19:37
corvuswell, actually i'd like to listen more about testing :)19:37
ianwbut later is a thing that tends to never come19:37
pabelangerI've been testing with molecule for ansible recently, works well. same idea as beaker testing for puppet19:37
corvusianw: yeah, i think we've got exim and (at the end of the series) iptables and snmp all done generically and with a readme19:37
ianwcorvus: right; i've got patches up to actually make the documentation too, which fixes at least one issue in the exim readme19:38
fungipabelanger: bending beaker to our model of already-provisioned test environments ended up being sort of a challenge, if my memory isn't failing me. does molecule present similar base assumptions we have to work around?19:38
corvusdoes anything exercise these roles for real?19:38
mordredthe roles are being run in our current puppet tests19:38
corvusmordred: the apply test?19:39
corvusor the beaker test?19:39
*** ahale has quit IRC19:39
*** evrardjp has quit IRC19:39
*** pcarver has quit IRC19:39
*** timothyb89 has quit IRC19:39
*** aarents has quit IRC19:39
*** Neptu has quit IRC19:39
*** rodrigods has quit IRC19:39
*** scas has quit IRC19:39
corvusoh it's the beaker tests19:39
mordredat least the beaker test19:39
corvusmy snmp role failed beaker but passed apply19:39
mordredwe run base.yaml as a setup step in the beaker test19:39
pabelangerfungi: yea, molecule has a delegated backend, where you can say the node has already been provisioned. Then either use localhost or loopback ssh19:39
*** mriedem is now known as mriedem_afk19:39
ianwpabelanger: link for molecule?19:40
pabelangerfungi: the main bonus for molecule, would be to say allow local iteration against docker / vagrant, then when in zuul just assume node is already setup and skip the provision19:40
corvusthe spec says we should start using testinfra19:40
mordredyah - testinfra is a way to express the actual tests19:40
mordredmy understanding is that molecule is a test runner and can use testinfra tests19:40
corvusokay, so are we at a place where we should start using testinfra?19:41
mordredso kind of like beaker and serverspec19:41
mordredcorvus: I think we're definitely at a place where we should figure out the molecule/testinfra situation and come up with a testing pattern19:41
pabelangerYah, I've also switched some roles to testinfra recently too:  http://git.openstack.org/cgit/openstack/ansible-role-nodepool/tree/tests/test_role.py19:41
pabelangerdefault is pytest, but does support python unittests19:41
pabelangerbut haven't tested19:41
*** sambetts|afk has quit IRC19:42
corvuswell, the other thing we can do is, and this is maybe what ianw was getting at, is have zuul run the ansible for the tests19:43
fungirather than an indirection layer19:43
*** Neptu has joined #openstack-meeting19:43
corvusso, basically, run the base.yaml playbook from zuul, with zuul's own inventory19:43
mordredyes. the main issue with that is that zuul is using 2.5 and were using 2.6 on bridge - it's PROBABLY not a problem - but it's a thing to be aware of19:43
pabelangerif we don't expect people to use sytem-config roles, that sounds like a good start19:43
corvusthen validate with testinfra19:43
fungilike a dry-run of what we might do in the future to configure servers directly from zuul jobs?19:43
mordredalso - we have some plugins19:44
mordredthat zuul will disallow us from using19:44
corvusmordred: what plugins?19:44
pabelangerzuul ansible, run tox ansible still is valid for testing too19:44
mordredthe iptables patch adds a jinja filter_plugin19:44
ianwmordred/corvus: hrm, yeah i hit that with https://review.openstack.org/#/c/593998/19:45
mordredwe also have an edited copy of module_utils for the zypper/apt detection19:45
corvusmordred: oh, so does exim19:45
*** sambetts_ has joined #openstack-meeting19:45
ianwif that's not ok, system-config may need to be a trusted project?19:45
mordredand I think it's correct for us to have those - so I think running ansible with ansible is maybe a better bet19:45
corvusianw: oh you're right, we don't need true anymore19:46
pabelangerianw: if trusted, then we get no per merge testing19:46
mordredalso - we're going to run ansible with ansible for CD - so testing that way would be closer to producciton19:46
ianwpabelanger: yes, which is why I didn't want that! :)19:46
*** samueldmq has quit IRC19:46
fungicould this be our first use case for a post-review pipeline?19:46
*** _hemna has joined #openstack-meeting19:47
pabelangerI still think having ansible-playbook from zuul, run tox jobs, which then does ansible-playbook is a good way to testing too19:47
mordredzuul's ansible running a shell task called "ansible-playbook" on a node called "bridge" with an inventory consisting of the other nodes created by zuul would be closer to testing production than having zuul's ansible run directly against nodes19:47
*** ahale has joined #openstack-meeting19:47
corvusokay, so it sounds like ansible->ansible is what we need for this.  kind of a bummer; i'd prefer to use this as a test case for how much we can push into zuul directly, but that's going to be slow and it doesn't sound like we have much appetite for that.19:47
*** onovy has quit IRC19:48
mordredcorvus: maybe we can also do direct-zuul for some of the roles19:48
corvusand yeah, if we've crossed the bridge of adding the indirection for production, it probably doesn't make too much sense to stretch the paradigm for test.19:48
mordredcorvus: yah19:49
*** samueldmq has joined #openstack-meeting19:49
mordredI'm pondering writing an ansible ansible module19:49
mordredsimilar to the ansible puppet module19:49
ianwmordred: the problem i hit with 593998 was that if anything has plugins, zuul stops?19:49
* fungi sees it's ansibles all the way down19:50
mordredianw: yes, that's right19:50
corvusianw: yes.  so two things: 1) we can proceed with 998 because it's removing dead code.  but 2) the general solution is that we should not use these roles directly in zuul.  we should use them indirectly with zuul-ansible running a separate ansible as part of the test process.19:50
corvuspabelanger: what does molecule add to this?19:51
mordredso, I've been arguing that some of them might be useful to use directly in zuul19:51
ianwright, just saying "maybe we can also do direct-zuul for some of the roles" doesn't really work unless we start splitting things into two repos?19:51
mordred(like ianw's afs and krb roles)19:51
pabelangercorvus: just saying if you wanted to iterate on local infra, you could use molecule19:51
pabelangervia docker or vagrant19:51
ianwmordred: more that might be useful, that's really the point of them, to be used in the wheel jobs19:52
mordredmaybe we need a more nuanced story then - we could have a roles/ and also a playbooks/roles/ and put things that are generally useful in both jobs and production into roles/  and things that are not useful directly from jobs into playbooks/roles ?19:53
mordredor is that too convoluted?19:53
corvusi can follow that19:53
ianwmordred: that can work, and with a small update https://review.openstack.org/#/c/593477/ you can even have them documented19:54
*** onovy has joined #openstack-meeting19:54
mordredneat. I'll update the move-roles patch to only move _some_ of the roles19:54
ianwat least until we have the complete ansible-turtles-all-the-way-down testing story19:55
*** njohnston has left #openstack-meeting19:55
corvusso what does that look like?19:56
corvusi'd like to get that running asap19:56
corvusbut i don't have enough context to write it myself from scratch19:56
ianwthe only theoretical problem might be if people start relying on the roles, and then we have to go, say, adding a plugin and have to essentially move it to a "private" role19:56
corvusi don't think anyone should rely on anything in the system-config repository :)19:57
fungiwe've been saying that for a very long time19:57
corvusfungi: well, actually we haven't19:57
mordredwell - except for other zuul jobs19:57
corvuswe've said that the system-config repository is in fact suitable for anyone to use19:57
fungii thought we'd long said system-config was full of openstackisms and that was why we extracted the reusable bits out to other modules19:58
mordredlike with the afs-client role19:58
corvusi would like to revert back to what we used to say, which is that it is *merely* for operating the actual systems we run for openstack19:58
fungiahh, i thought we'd been saying that all along19:58
pabelangerI think if a role is system-config, it is limited to system-config. if we want others to use said role, we should create the role under openstack-infra, like we did with puppet-foo19:59
corvusfungi: there's an extra level of abstraction in the sytem-config puppet designed to allow other people to "run an etherpad the way we run an etherpad"19:59
*** eharney has quit IRC19:59
mordredpabelanger: eventually I tihn that's appropriate for some of them19:59
pabelangermordred: yup, some for sure19:59
pabelangerdon't think we want all19:59
ianwpabelanger: yeah, i mean ... do you think puppet-timezone etc has really worked that well?  i think collecting them makes it much more discoverable and maintainble19:59
mordredbut it's a lot of extra churn and effort here at the beginning while we're sorting thigns out and reorganizing as we learn new things19:59
corvusianw: to be honest, puppet-timezone has worked perfectly for me :)20:00
ianwcorvus: but as a community building and reusable component20:00
corvus(i didn't even remember it existed)20:00
mordredI think there are three types of roles here ... roles that are very infra specific, roles that we want to use in infra but also want to use in jobs, and roles that are at the level of wanting to be legit standalone20:00
fungiokay, we're a minute over time. we should probably take this to the #openstack-infra channel. my general topic isn't super urgent anyway20:01
corvusianw: yep, it has failed to accrue a community to support it, but i'm not certain that was (or should be) a goal :)20:01
pabelangerif the roles are small, and compact, and say timezone only modifies timezone. I think it works, the grey aready is when a role does things with apache, then package repos, then etc.  Much harder to suppport for other users20:01
corvusftr, i do not know how to proceed in setting up a real deployment test20:01
mordred(also, to be clear - we did not write puppet-timezone - it's a puppet community module we're consuming)20:01
corvusperhaps we can continue to discuss how to set up that test, either here or in infra?20:02
corvusif this channel isn't reserved, can we just continue?20:02
ianwif nobody is kicking us out, would be good to be logged here20:02
mordredI have no issues with either choice20:02
corvusokay, so let me sketch out a strawman20:03
fungii'm fine continuing, though i may need to #chair someone else if it goes on much longer (need to start cooking dinner shortly)20:03
corvusfungi: at this point we can all endmeeting anyway :)20:03
fungiyup, i don't suppose the topic needs to change again ;)20:03
fungijust to be clear, if anyone is in here waiting for another meeting to start, please let us know and we'll scram20:04
corvusa job which requests a xenial node and runs ansible-playbook with the supplied nodepool inventory and the base playbook.  then it runs testinfra to validate that things were set up correctly?20:05
*** yankcrime has joined #openstack-meeting20:05
pabelangerseems reasonable to me20:06
mordredcorvus: yes. or perhaps a 2-node job, with a bionic 'bridge' node on which ansible-playbook is run and a second node to run against?20:06
corvusmordred: do we bootstrap the bridge node by running the base playbook on it?20:07
mordredcorvus: I don't think we need the base playbook for that - there is a smaller bridge playbook that should get the bridge things installed20:08
mordredand which is probably much more suitable for running directly from zuul20:08
mordredalthough even that isn't quite right - since it's going to install the openstack inventory plugin20:09
corvusokay, so zuul uses its own ansible to run the bridge playbook against the bridge node, then zuul copies the inventory over, and runs ansible-playbook base.yaml on bridge.20:10
mordredso we might want to construct a playbook specifically for that purpose20:10
ianware there any secrets that need to be set for the playbooks to work?20:10
pabelangeras an example: http://git.openstack.org/cgit/openstack/windmill/tree/tests/playbooks/run.yaml is a crud want to run ansible->ansible on a single node. But it is currently not using the inventory file from nodepool.20:10
mordredcorvus: yes, that sounds correct20:10
pabelangerbut could be a something to work from20:10
*** yankcrime has left #openstack-meeting20:10
*** geguileo has quit IRC20:11
corvusokay, is that correct up to that point in the job?20:11
mordredyes. I think so20:12
corvusso how's this testinfra thing work? :)20:12
*** beisner_ has joined #openstack-meeting20:12
*** gmann_ has joined #openstack-meeting20:13
pabelangercould be a tox entry, testinfra will use the ansible inventory file for the remote nodes20:13
corvussomething like:     zuul runs tox which runs testinfra using zuul's inventory20:14
pabelangeryah, that is possible20:14
corvusmordred: is that like what you were thinking?20:14
*** jgriffit1 has joined #openstack-meeting20:14
mordredyes - I think so20:15
corvusianw: i can't think of any secrets needed for the base playbooks right now20:15
mordredwe should also ponder what per-service playbooks looks like -I'm guessing they each want a specific zuul job20:15
corvusyeah, i'll continue the etherpad sketch20:15
sdatkofungi: just FYI, i am not sure how github.com/openstack-infra/irc-meetings is accurate, but it seems like for today there is only one more meeting left: scientific-sig-meeting (grep -nri 'Tuesday' --before-context 1 *.yaml | grep -i 'time' | sort -k 4 | column -t)20:16
fungisdatko: yes, there's also a calendar file linked from http://eavesdrop.openstack.org/20:17
corvussdatko: yes, my calendar has that starting in 43 mins20:17
*** samP_ has joined #openstack-meeting20:18
sdatkoach, ok :-D20:18
*** edmondsw_ has joined #openstack-meeting20:18
corvusianw: can you type something in the etherpad?20:18
corvusianw: preferably your name20:18
*** samP has quit IRC20:19
*** edmondsw has quit IRC20:19
*** beisner has quit IRC20:19
*** AJaeger has quit IRC20:19
*** jgriffith has quit IRC20:19
*** gmann has quit IRC20:19
*** hugokuo has quit IRC20:19
*** edmondsw_ is now known as edmondsw20:19
*** samP_ is now known as samP20:19
*** gmann_ is now known as gmann20:19
*** beisner_ is now known as beisner20:19
corvusokay.  weird.  i clicked the thing to set my name, and it was pre-filled with ianw20:19
mordredcorvus: maybe you are actually the same person as ianw20:19
corvusother than that, things seem to be working.20:19
corvusokay, this makes sense to me, and seems like something we should be able to get going pretty quickly20:20
pabelangerare we planning in running ansible-playbook in virtualenv or use distro version20:20
corvus#link etherpad sketching out system-config ansible testing https://etherpad.openstack.org/p/CJUWmx62al20:21
corvuspabelanger: whatever bridge.yaml installs; currently i think it's pip installed globally?  or something?20:21
mordredpip installed globally20:21
mordredusing the python3.6 that came with bionic20:22
*** jamesmcarthur has joined #openstack-meeting20:22
corvusmordred: is it the case that we do not want the openstack inventory plugin installed?  if there's no clouds.yaml file, will it no-op or break?20:23
mordredcorvus: it doesn't matter - what will matter is the ansible.cfg we install20:23
*** AJaeger has joined #openstack-meeting20:24
corvusmordred: ok, so we might be able to run the bridge playbook directly, then a second playbook which modifies the config20:24
mordredcorvus: yah20:24
mordredthat's probably the cleanest thing to do20:24
mordredand gets us coverage of the bridge playbook20:25
mordredwe're going to need to think about host_vars/group_vars and how they interact with this - someof them are essential qualities of a given host, so we'd want them to get pulled in20:25
mordredbut some of them are deployment specific - like the IP address of a remote host to enable in iptables20:25
mordredI don't have a solution for that to propose yet - just a thing to keep in mind while we keep working on this20:25
corvusmordred: yeah.  maybe we can overlay test values on top of the normal values20:25
*** jamesmcarthur has quit IRC20:26
corvusmordred: if our jobs are based on functional units, we might be able to use inventory values for that.  eg, a multinode job with all the zuul components as hosts20:27
corvusanyway, i think i have a good enough idea of what this should look like for us to get started; anyone have anything else?20:28
mordredI think that's great!20:29
fungisounds like a solid start on a plan20:29
corvusfungi: since you're around, i'll leave the endmeeting honors to you :)20:29
fungiawesome, i'll cover my general topic real fast since we have the channel anyway...20:29
fungi#topic Continued discussion of Winterscale naming (fungi)20:30
*** openstack changes topic to "Continued discussion of Winterscale naming (fungi) (Meeting topic: infra)"20:30
fungi#link http://lists.openstack.org/pipermail/openstack-infra/2018-August/006075.html20:30
fungifor those of you who aren't following the infra ml, hoping we can find some consensus on a name for the winterscale effort, ideally before the ptg20:30
fungithat's all20:30
corvuswhat should we do if we like that name and want to move forward with it? :)20:30
fungisilence or thumbs up or have a beer20:30
corvus(the email was clear on what to do if we *don't* like it, less clear on how to express support)20:30
corvusi don't know how to do the first 2 things so i'll do the last20:31
fungiif there are additional items added to the short list, we can discuss them. of there aren't, then i guess we have one to move forward on20:31
fungiany other quick questions? i'll leave it on the agenda for next week when clarkb's back too20:32
fungithanks everyone! find us in #openstack-infra for further discussion20:33
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"20:33
openstackMeeting ended Tue Aug 21 20:33:19 2018 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)20:33
openstackMinutes:        http://eavesdrop.openstack.org/meetings/infra/2018/infra.2018-08-21-19.02.html20:33
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/infra/2018/infra.2018-08-21-19.02.txt20:33
openstackLog:            http://eavesdrop.openstack.org/meetings/infra/2018/infra.2018-08-21-19.02.log.html20:33
*** Shrews has left #openstack-meeting20:33
*** sdatko has quit IRC20:35
*** erlon has quit IRC20:39
*** annabelleB has quit IRC20:41
*** raildo has quit IRC20:46
*** awaugama has quit IRC20:51
*** annabelleB has joined #openstack-meeting20:55
*** rbudden has joined #openstack-meeting20:56
*** oneswig has joined #openstack-meeting20:56
*** jamesmcarthur has joined #openstack-meeting20:58
oneswig#startmeeting scientific-sig21:00
openstackMeeting started Tue Aug 21 21:00:15 2018 UTC and is due to finish in 60 minutes.  The chair is oneswig. Information about MeetBot at http://wiki.debian.org/MeetBot.21:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.21:00
*** openstack changes topic to " (Meeting topic: scientific-sig)"21:00
openstackThe meeting name has been set to 'scientific_sig'21:00
oneswig#link agenda for today https://wiki.openstack.org/wiki/Scientific_SIG#IRC_Meeting_August_21st_201821:00
oneswigMartial sends his apologies as he is travelling21:01
rbuddenhello everyone21:01
oneswigBlair's hoping to join us later - overlapping meeting21:01
oneswigHey rbudden! :-)21:01
rbuddenhello hello :)21:01
*** ildikov has joined #openstack-meeting21:01
oneswigHow's BRIDGES?21:01
rbuddenit’s doing well21:02
*** harlowja has joined #openstack-meeting21:02
rbuddenOur Queens HA setup is about to get some friendly users/staff for testing :)21:02
oneswigI've been having almost too much fun working with these guys on a Queens system: https://www.euclid-ec.org21:02
oneswigMapping the dark matter of OpenStack...21:03
oneswigIt boils down to a bunch of heat stacks and 453 nodes in a Slurm partition with some whistles and bells attached21:03
*** mriedem_afk is now known as mriedem21:04
rbuddenSounds interesting21:04
oneswigFound the scaling limits of Heat on our system surprisingly quickly!21:04
rbuddenHehe, I haven’t used Heat much21:05
oneswigrbudden: did you ever get your work on Slurm into the public domain?21:05
rbuddenI took the lazy programmer approach and wrote it all in bash originally, then converted it mostly to Ansible21:05
rbuddenActually, I did not21:05
rbuddenI have piles that need to be pushed to our public GitHub21:05
oneswigThere's some interest from various places in common Ansible for OpenHPC+Slurm21:06
rbuddenIt’s on my never ending todo list somewhere in there21:06
oneswigRight, somewhere beyond the event horizon...21:06
oneswigAre you going to the PTG rbudden?  It's merged with the Ops meetup too this time21:07
*** mjturek has quit IRC21:08
rbuddenI am not21:08
rbuddenUnfortunately I don’t have much funding for travel right now21:08
oneswigNo, me neither - Martial's leading a session there21:08
rbuddenI’d have loved to get more involved with the PTG and Ops, but time and funding have always kept me away unfortunately!21:09
oneswigSome of our team will be there but I'm of limited use being less hands-on than I'd like to be nowadays21:09
oneswigWhat's new on BRIDGES, what have you been working on?21:10
rbuddenAs long as we have someone there that’s good21:10
rbuddenLargely the Queens HA and looking into scaling Ironic better21:10
rbuddenWe’ve also had a large influx of requests for Windows VMs from up on CMU campus21:10
oneswigbare metal windows?21:11
rbuddenso we’ve been trying to patch together some infra to make those easier21:11
rbuddenno, virtual ATM21:11
rbuddenI’m starting to plan for our next major upgrade as well21:11
oneswigwhat are you fighting with Ironic scaling?21:11
rbuddenso getting some multiconductor testing and attempting boot over OPA are high on the list21:12
oneswigIPoIB interface on OPA?21:12
rbuddeni had read the 7.4 centos kernel has stock drivers that could be used for the deploy21:13
rbuddenI’m not sure if anyone has tried it though21:13
oneswigThe OPA system I've been working on has the Intel HFI driver packages21:13
rbuddeni’d love to deploy nodes via IPoIB vs our lonely GigE private network ;)21:13
oneswigI'd be surprised if the in-box driver is new enough.21:13
rbuddenour last upgrade to 7.4 took longer than expect pushing out updates via puppet/scripts21:14
oneswigLast time I looked was end of last year though21:14
rbuddenso we’d like to attempt to reimage all 900+ nodes as fast as possible via Ironic this time21:14
rbuddenyeah, I admit I haven’t done much looking other than finding the pieces are there21:14
rbuddeni.e. PXE boot via OPA has been done21:14
oneswigThis might interest you - proposal for Ironic native RBD boot from Ceph volume - https://storyboard.openstack.org/#!/story/200351521:15
rbuddengeneric OPA IPoIB drivers should be in the mainline kernel now (whether they work is anotehr story)21:15
rbuddencool, thx, i’ll check it out21:15
oneswigJust a framework for an idea right now but one we've already tried to explore ourselves, because it's a natural fit21:16
rbuddenwe’ve only had a limited experience with Ceph21:16
rbuddenand so far we’ve seen incredibly bad performance21:16
rbuddenI’ve been meaning to solicit some advice/stats from the group21:16
rbuddenon setups/performance numbers21:17
oneswigCeph's single-node performance is lame in comparison but the scaling's excellent, particularly for object and block.  File is still a little doubtful.21:17
oneswigby single-node I mean single-client21:17
rbuddenok, interesting21:18
rbuddenthat’s what we were curious about21:18
rbuddenespecially given this:21:18
oneswigOur team has recently been working on BeeGFS deployed via ansible21:18
rbuddenit looks like single client peformance is a small fraction of what the hardware is capable of21:18
rbuddenbut i was curious if that was due to scaling/sustaining those numbers in parallel to a large portion of clients21:19
oneswigThat matches my experience, but 20 clients will get 75% of the raw performance of the hardware21:19
rbuddenwe have a very small amount of hardware to dedicate21:19
oneswigI bet I have less :-)21:19
rbuddensay 3-4 servers for Ceph21:19
oneswigyup, won hands down :-)21:19
rbuddenso far the best we’ve seen was 15 MB/s writes, though reads were maybe between 50-100 MB/s21:20
rbuddenpretty attrocious21:20
oneswigThe EUCLID deploy's home dir is CephFS, served from 2 OSDs.  It's a bit ad-hoc, all we had available at the time.21:20
oneswigrbudden: wow, that's not good!21:20
oneswigby any measure21:20
rbuddenthere must be some configuration issues21:20
rbuddenyeah, not sure what’s wrong just yet21:21
*** munimeha1 has quit IRC21:21
rbuddenwe pulled in Mike from IU try and spot something obvious ;)21:21
oneswigOn performance, one of the team at Cambridge has been benchmarking BeeGFS vs Lustre on the same OPA hardware.21:21
rbuddenoh yeah?21:22
rbuddenwe have both at PSC as well21:22
oneswigMike's a great call :-)!21:22
oneswigWrite performance was equivalent.21:22
rbuddenour Olympus cluster has both Lustre and BeeGFS21:22
oneswigBeeGFS ~20% faster for reads21:22
rbuddenvery interesting21:22
oneswigQuite a surprise21:22
rbuddenwe’ve been toying with using BeeGFS on Bridges21:22
rbuddensince we already mount Olympus filesystems on both21:23
oneswigThere seems to be good momentum behind it currently.  It's pretty easy to set up and use21:23
rbuddenI’ll have to relay the performance numbers to ppl here21:23
rbuddenit’s a small cluster from our Public Health group21:23
oneswigI'll see if I can get graphs for you.21:23
rbuddenexact hardware as Bridges21:24
rbuddenjust a separate Slurm scheduler to manage them21:24
rbuddenwe’ve even folded their cluster under Ironic now21:24
oneswigNice work.  Hopefully not twice the effort for you now!21:25
rbuddenI think that might put us over 1000 nodes on single conductor :P21:25
rbuddennot that i’d normally advise that… but we only image a few nodes a week at this point21:25
oneswigAny additional concerns taken into account for public health data on that system?21:25
rbuddennot to my knowledge. we don’t do any classified work, etc.21:26
rbuddenfor nothing fancy with network isolation or anything of that nature21:26
oneswigGot it.21:26
oneswigIt's a requirement that is proliferating though21:27
rbuddenwe’ve done some small projects that required HIPPA with local medical community21:27
oneswigon Bridges?21:28
rbuddeni’m not sure if it was Bridges or a small subproject on our DXC hardware21:28
rbuddeni’d have to check and make sure i’m not mixing things up ;)21:28
rbuddenI know being compliant for securing big data is on our radar21:28
oneswighipaa compliance?21:29
rbuddenthat’s one, but i believe there are others21:29
rbuddenfamiliar with HIPPA?21:29
oneswigfaintly at best..21:30
rbuddenit’s a health care act to help protect sensitive patient data21:30
rbuddenor should i say regulation ;)21:30
*** dustins has quit IRC21:31
rbuddenoneswig: sorry to jet, but i need to step offline for daycare pickup!21:31
rbuddenwe should follow up offline, it’s been awhile since we chatted in Vancouver21:31
oneswigNP, good to talk21:31
oneswigI'll mail re: cambridge data21:31
rbuddeni’ve been meaning to reach out21:31
rbuddensounds good21:32
oneswigrbudden: always a pleasure21:32
rbuddentty soon21:32
*** b1air has joined #openstack-meeting21:38
b1airam i too late?21:38
oneswigb1air: howzit?21:39
b1aira'ight, you?21:39
oneswigWas just chatting with rbudden, he's dropped off21:39
oneswigGood thanks.  How's the new role going?21:39
b1airi'm in the thick of having my head exploded by all the context i'm trying to soak up21:40
b1airbut otherwise good21:40
oneswigAre you going to be hands-on with the Cray?21:40
b1airat least i now have a vague idea of what preconceptions the organisation has about my role, i honestly didn't know what i was really signing up for to start off with, just that it would probably be quite different from what i was doing at Monash21:41
b1airi will be unlikely to have root access there anytime soon21:41
b1airNIWA seem to like to keep systems stuff pretty closely guarded21:42
b1airhow was your holiday?21:43
oneswigVery good indeed.  We hiked in the Pyrenees and it was ruddy hot21:43
oneswigPerhaps we should close the meeting, and then I'll reply to your email :-)21:45
b1airi need to come up with a hiking/biking/sailing itinerary for summer here!21:45
oneswigb1air: in NZ, it's not hard, surely mate21:45
b1airthe poor boat needs a bit of love first21:46
oneswigsandpaper = tough love ?21:46
b1airlol, water blaster might be enough i hope! anyway, sure, let's continue to be penpals :-)21:46
oneswigok bud, have a good day down there21:47
b1aircheers, over and out21:47
oneswiguntil next time21:47
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"21:47
openstackMeeting ended Tue Aug 21 21:47:30 2018 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)21:47
openstackMinutes:        http://eavesdrop.openstack.org/meetings/scientific_sig/2018/scientific_sig.2018-08-21-21.00.html21:47
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/scientific_sig/2018/scientific_sig.2018-08-21-21.00.txt21:47
openstackLog:            http://eavesdrop.openstack.org/meetings/scientific_sig/2018/scientific_sig.2018-08-21-21.00.log.html21:47
*** jamesmcarthur has quit IRC21:48
*** oneswig has quit IRC21:49
*** jamesmcarthur has joined #openstack-meeting21:50
*** jamesmcarthur_ has joined #openstack-meeting21:52
*** jamesmcarthur_ has quit IRC21:55
*** jamesmcarthur has quit IRC21:55
*** jamesmcarthur has joined #openstack-meeting21:56
*** cloudrancher has joined #openstack-meeting21:57
*** jamesmcarthur has quit IRC22:01
*** jamesmcarthur has joined #openstack-meeting22:04
*** jamesmcarthur has quit IRC22:08
*** rcernin has joined #openstack-meeting22:10
*** ykatabam has joined #openstack-meeting22:12
*** annabelleB has quit IRC22:21
*** mriedem is now known as mriedem_away22:22
*** annabelleB has joined #openstack-meeting22:27
*** rbudden has quit IRC22:32
*** Leo_m has quit IRC22:44
*** diablo_rojo has quit IRC22:44
*** hongbin has quit IRC22:44
*** diablo_rojo has joined #openstack-meeting22:48
*** tpsilva has quit IRC22:58
*** erlon has joined #openstack-meeting23:06
*** erlon has quit IRC23:22
*** macza has quit IRC23:37
*** jamesmcarthur has joined #openstack-meeting23:39
*** jamesmcarthur has quit IRC23:44
*** annabelleB has quit IRC23:51
*** annabelleB has joined #openstack-meeting23:52

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!