Tuesday, 2019-06-04

*** yamamoto has joined #openstack-meeting00:02
*** markvoelker has quit IRC00:24
*** Liang__ has joined #openstack-meeting00:27
*** _erlon_ has quit IRC00:29
*** Liang__ is now known as LiangFang00:30
*** ekcs has quit IRC00:38
*** Adri2000 has quit IRC00:40
*** mriedem has quit IRC00:44
*** Adri2000 has joined #openstack-meeting00:45
*** bobh has joined #openstack-meeting00:58
*** ttsiouts has joined #openstack-meeting01:00
*** yamamoto has quit IRC01:03
*** ttsiouts has quit IRC01:06
*** lseki has quit IRC01:10
*** gyee has quit IRC01:19
*** jamesmcarthur has quit IRC01:22
*** njohnston_ has quit IRC01:27
*** jamesmcarthur has joined #openstack-meeting01:31
*** jamesmcarthur has quit IRC01:43
*** jamesmcarthur has joined #openstack-meeting01:44
*** _hemna has joined #openstack-meeting01:53
*** bobh has quit IRC01:54
*** jamesmcarthur has quit IRC02:00
*** jamesmcarthur has joined #openstack-meeting02:00
*** hyunsikyang has quit IRC02:16
*** hyunsikyang has joined #openstack-meeting02:17
*** hongbin has joined #openstack-meeting02:17
*** _hemna has quit IRC02:17
*** yamahata has quit IRC02:24
*** jamesmcarthur has quit IRC02:39
*** jamesmcarthur has joined #openstack-meeting02:40
*** jamesmcarthur has quit IRC02:43
*** jamesmcarthur has joined #openstack-meeting02:43
*** yamamoto has joined #openstack-meeting02:51
*** _hemna has joined #openstack-meeting02:52
*** whoami-rajat has joined #openstack-meeting03:14
*** _hemna has quit IRC03:25
*** hyunsikyang has quit IRC03:37
*** igordc has quit IRC03:39
*** jamesmcarthur has quit IRC03:45
*** jamesmcarthur has joined #openstack-meeting03:46
*** jamesmcarthur has quit IRC03:49
*** jamesmcarthur_ has joined #openstack-meeting03:49
*** tashiromt has joined #openstack-meeting03:55
*** LiangFang has quit IRC03:59
samP#startmeeting masakari04:00
openstackMeeting started Tue Jun  4 04:00:05 2019 UTC and is due to finish in 60 minutes.  The chair is samP. Information about MeetBot at http://wiki.debian.org/MeetBot.04:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.04:00
*** openstack changes topic to " (Meeting topic: masakari)"04:00
openstackThe meeting name has been set to 'masakari'04:00
samPHi all for Masakari04:00
*** Liang__ has joined #openstack-meeting04:01
samP#topic Critical bugs and patches04:01
*** openstack changes topic to "Critical bugs and patches (Meeting topic: masakari)"04:01
*** imsurit has joined #openstack-meeting04:01
samPPlease share any items need to discuss here.04:02
*** yamahata has joined #openstack-meeting04:02
samPtashiromt: Hi04:02
samPSeems like we do not have any item to discuss today.04:03
*** tpatil has joined #openstack-meeting04:04
tpatilSorry to join late04:04
samPtpatil: no problem04:04
*** shilpasd has joined #openstack-meeting04:04
samPNow in the topic of Critical bugs and patches04:05
samPAny issues to share?04:05
tpatilsamP: No critical bugs to be discussed from my side04:05
samPtpatil: thanks04:05
samP#topic Train work items04:06
*** openstack changes topic to "Train work items (Meeting topic: masakari)"04:06
samPPlease share if you have any update on Train work items04:06
*** igordc has joined #openstack-meeting04:06
tpatil1) Add devstack support to install host-monitors04:07
tpatilWe are working on understanding how to configure STONITH resource04:07
tpatilI think in host-monitor, it's possible to use STONITH_TYPE ssh and external/ipmi04:07
tpatilAny idea, how does ssh works?04:08
tpatilsamP: Also, in our dev env. we don't have support for IPMI, so we couldn't make any progress. I would like to discuss with you about the dev. env offline.04:09
samPtpatil: Sure, I will make time to do that. And I think we have IPMI in for bare metal nodes. I will share those details later04:11
tpatil#link : https://github.com/openstack/masakari-monitors/blob/master/masakarimonitors/hostmonitor/hostmonitor.sh#L70704:11
tpatilabove link for ssh STONITH_TYPE04:11
tpatilI have one question about STONITH_TYPE: ssh, if one compute node is down, how is it possible to interact with that node using SSH from other compute node?04:13
samPtpatil: Thanks for the link. Before use that STONITH method, we need to configure pacemaker cluster with ssh settings04:13
samPtpatil: you can't. STONITH is use to kill a node.04:13
tpatilsamP: Kill a node proactively or after pacemaker detects an issue in that node?04:15
samPtpatil: As an Ex: if operator (or Pacemaker) find something wrong in node, then he (it) try to kill that node and isolate it from the other nodes04:16
samPtpatil:  correct.04:16
*** hongbin has quit IRC04:17
tpatilsamP: The job of stonith resource device is to kill that node completely, correct?04:18
samPtpatil: correct04:19
tpatilsamP: in that sense, how does STONITH_TYPE ssh works is my main question? becoz it's possible that failed node may fail to response to ssh04:19
tpatilsamP: We can discuss about supported STONITH_TYPE in host-monitors and how it works separately as I'm completely new in this area04:21
samPtpatil: yes, there is a possibility. Normally we configure several type of STONITH_TYPS such as IPMI, ssh .. etc.04:21
samPif ssh failed, then it will try to use the IPMI or any other methods04:22
tpatilsamP: Ok04:22
samPdifferent stonith method has its own advantages and disadvantages04:22
*** haleyb has quit IRC04:22
*** armax has quit IRC04:22
samPin ssh, you execute commands such as take the logs, wait for core dump.04:23
samPon the other hand it will take time.04:23
samPin IPMI, you can kill a node very quick. However, you may loose core dump in servers with large RAM04:24
samPtpatil: I will make detail doc for this and make time to explain about this.04:25
tpatilsamP: That will help, Thank you.04:25
samPI will contact you offline for this.04:25
tpatilsamP: Ok, Thank you04:26
tpatil2) Real time rendering of recovery workflow in masakari-dashboard04:26
tpatilas explained in last weekly meeting, fastener commit will be reverted : https://github.com/harlowja/fasteners/issues/3604:26
tpatilSo it's a blocker to implement real time rendering as we need to add more progress details during the evacuation of instances during execution of workflow04:28
tpatilWe haven't yet tried the workaround solution to monkey patch method from fastener to make this thing work in masakari04:28
tpatil3) Improve documentation04:31
samPtpatil: Thanks for update.04:31
tpatilUnfortunately, not yet started. Will take up this task sometime in this week04:31
samPtpatil: no problem04:31
tpatilThat's all update about Train items from my end04:32
samPtpatil: thanks04:32
samPI have no update from my side.04:33
samPI will make time to discuss about pacemaker and stonith settings. I will let you know the details04:33
samP#topic AOB04:33
*** openstack changes topic to "AOB (Meeting topic: masakari)"04:33
samPAny other issues need to discuss04:34
tpatilsam: OK04:34
tpatilsamP: OK04:34
tpatilsamP: No other topics for discussion from my side04:35
samPtpatil: thanks04:35
*** zaneb has quit IRC04:35
samPOK then, let's finish today's meeting here04:35
*** baojg has quit IRC04:35
samPThank you all!04:35
tpatilsamP: Ok, Thanks04:36
samPplease use ML or #openstack-masakari IRC@freenode for further discussions.04:36
samPThank you all again..04:36
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"04:36
openstackMeeting ended Tue Jun  4 04:36:37 2019 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)04:36
openstackMinutes:        http://eavesdrop.openstack.org/meetings/masakari/2019/masakari.2019-06-04-04.00.html04:36
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/masakari/2019/masakari.2019-06-04-04.00.txt04:36
openstackLog:            http://eavesdrop.openstack.org/meetings/masakari/2019/masakari.2019-06-04-04.00.log.html04:36
*** baojg has joined #openstack-meeting04:41
*** zaneb has joined #openstack-meeting04:47
*** pwu has joined #openstack-meeting04:57
*** tpatil has quit IRC04:58
*** lbragstad has quit IRC05:09
*** pcaruana has joined #openstack-meeting05:12
*** radeks has joined #openstack-meeting05:18
*** _hemna has joined #openstack-meeting05:22
*** pcaruana has quit IRC05:23
*** pwu_ has joined #openstack-meeting05:24
*** pwu has quit IRC05:24
*** pcaruana has joined #openstack-meeting05:28
*** jamesmcarthur_ has quit IRC05:29
*** e0ne has joined #openstack-meeting05:43
*** belmoreira has joined #openstack-meeting05:49
*** igordc has quit IRC05:51
*** Liang__ has quit IRC05:53
*** Liang__ has joined #openstack-meeting05:54
*** _hemna has quit IRC05:55
*** kopecmartin|off is now known as kopecmartin06:07
*** lpetrut has joined #openstack-meeting06:10
*** bbowen has joined #openstack-meeting06:12
*** slaweq has joined #openstack-meeting06:13
*** bbowen_ has quit IRC06:14
*** zaneb has quit IRC06:16
*** wwriverrat has joined #openstack-meeting06:17
*** zaneb has joined #openstack-meeting06:23
*** _hemna has joined #openstack-meeting06:25
*** hyunsikyang has joined #openstack-meeting06:34
*** dkushwaha has joined #openstack-meeting06:36
*** jbadiapa has joined #openstack-meeting06:41
*** yamamoto has quit IRC06:41
*** belmoreira has quit IRC06:43
*** ianychoi has quit IRC06:57
*** ianychoi has joined #openstack-meeting06:58
*** pwu_ has quit IRC06:58
*** _hemna has quit IRC06:58
*** yamamoto has joined #openstack-meeting06:59
*** ttsiouts has joined #openstack-meeting07:09
*** e0ne has quit IRC07:09
*** _hemna has joined #openstack-meeting07:11
*** ykatabam has quit IRC07:15
*** _pewp_ has quit IRC07:16
*** _hemna has quit IRC07:16
*** _pewp_ has joined #openstack-meeting07:16
*** slaweq has quit IRC07:16
*** rossella_s has joined #openstack-meeting07:17
*** yamamoto has quit IRC07:20
*** rubasov has joined #openstack-meeting07:21
*** yamamoto has joined #openstack-meeting07:21
*** yamamoto has quit IRC07:30
*** zbr has joined #openstack-meeting07:31
*** markvoelker has joined #openstack-meeting07:32
*** yamamoto has joined #openstack-meeting07:33
*** ttsiouts has quit IRC07:34
*** ttsiouts has joined #openstack-meeting07:35
*** yamamoto has quit IRC07:39
*** ttsiouts has quit IRC07:39
*** slaweq has joined #openstack-meeting07:40
*** yamamoto has joined #openstack-meeting07:42
*** jaewook_lee has joined #openstack-meeting07:55
*** boxiang has joined #openstack-meeting07:56
*** pwu has joined #openstack-meeting07:56
*** takahashi-tsc has joined #openstack-meeting08:01
*** ttsiouts has joined #openstack-meeting08:02
*** markvoelker has quit IRC08:05
*** tetsuro has joined #openstack-meeting08:07
*** takahashi-tsc has quit IRC08:10
*** bhagyashris has joined #openstack-meeting08:10
*** tpatil has joined #openstack-meeting08:11
*** imsurit has quit IRC08:13
*** jaewook_lee has quit IRC08:13
*** boxiang has quit IRC08:14
*** boxiang has joined #openstack-meeting08:14
*** imsurit has joined #openstack-meeting08:14
*** priteau has joined #openstack-meeting08:23
*** tpatil has quit IRC08:28
*** e0ne has joined #openstack-meeting08:32
*** janki has joined #openstack-meeting08:39
*** tssurya has joined #openstack-meeting08:40
*** tashiromt has quit IRC08:50
*** lennyb has quit IRC08:53
*** ayoung has quit IRC08:54
*** ykatabam has joined #openstack-meeting08:55
*** tesseract has joined #openstack-meeting08:56
*** ttsiouts has quit IRC09:01
*** ttsiouts has joined #openstack-meeting09:02
*** Liang__ has quit IRC09:03
*** Liang__ has joined #openstack-meeting09:05
*** ttsiouts has quit IRC09:06
*** rcernin has quit IRC09:07
*** imsurit has quit IRC09:08
*** _hemna has joined #openstack-meeting09:12
*** lennyb has joined #openstack-meeting09:12
*** panda is now known as panda|ruck09:38
*** tesseract has quit IRC09:40
*** bhagyashris has quit IRC09:41
*** _hemna has quit IRC09:46
*** mugsie_ is now known as mugsie09:52
*** ociuhandu has joined #openstack-meeting09:54
*** markvoelker has joined #openstack-meeting10:02
*** Liang__ has quit IRC10:03
*** ykatabam has quit IRC10:17
*** slaweq has quit IRC10:30
*** baojg has quit IRC10:33
*** markvoelker has quit IRC10:36
*** yamamoto has quit IRC10:36
*** yamamoto has joined #openstack-meeting10:37
*** yamamoto has quit IRC10:38
*** yoctozepto_ is now known as yoctozepto10:42
*** bbowen has quit IRC10:49
*** baojg has joined #openstack-meeting10:59
*** slaweq has joined #openstack-meeting11:01
*** beagles has quit IRC11:03
*** dkushwaha has quit IRC11:06
*** carloss has joined #openstack-meeting11:09
*** pwu has quit IRC11:11
*** yamamoto has joined #openstack-meeting11:21
*** ttsiouts has joined #openstack-meeting11:22
*** raildo has joined #openstack-meeting11:26
*** janki has quit IRC11:30
*** ttsiouts has quit IRC11:33
*** markvoelker has joined #openstack-meeting11:33
*** ttsiouts has joined #openstack-meeting11:34
*** ttsiouts_ has joined #openstack-meeting11:37
*** ttsiouts has quit IRC11:38
*** tetsuro has quit IRC11:39
*** _hemna has joined #openstack-meeting11:42
*** panda|ruck is now known as panda|ruck|eat11:45
*** panda|ruck|eat is now known as panda|ruck11:51
*** bbowen has joined #openstack-meeting11:52
*** yamahata has quit IRC11:56
*** yamamoto has quit IRC11:58
*** yamamoto has joined #openstack-meeting12:00
*** b3nt_pin has joined #openstack-meeting12:02
*** dmacpher has joined #openstack-meeting12:02
*** baojg has quit IRC12:02
*** baojg has joined #openstack-meeting12:04
*** markvoelker has quit IRC12:05
*** yamamoto has quit IRC12:13
*** _hemna has quit IRC12:17
*** mugsie is now known as mugsie_12:18
*** mugsie_ is now known as mugsie12:18
*** njohnston has joined #openstack-meeting12:24
*** priteau has quit IRC12:34
*** panda|ruck is now known as panda|ruck|eat12:37
*** davidsha has joined #openstack-meeting12:48
*** yamamoto has joined #openstack-meeting12:51
*** panda|ruck|eat is now known as panda|ruck12:51
*** plutoinmii has joined #openstack-meeting12:53
*** priteau has joined #openstack-meeting12:58
*** yamamoto has quit IRC12:59
*** yamamoto has joined #openstack-meeting12:59
*** jamesmcarthur has joined #openstack-meeting13:00
*** Luzi has joined #openstack-meeting13:00
*** priteau has quit IRC13:04
*** baojg has quit IRC13:09
*** baojg has joined #openstack-meeting13:10
*** ttsiouts_ has quit IRC13:10
*** ttsiouts has joined #openstack-meeting13:11
*** rubasov has quit IRC13:12
*** Luzi has quit IRC13:13
*** baojg has quit IRC13:14
*** ttsiouts has quit IRC13:15
*** mriedem has joined #openstack-meeting13:17
*** diablo_rojo has joined #openstack-meeting13:18
*** b3nt_pin is now known as beagles13:19
*** jamesmcarthur has quit IRC13:20
*** jamesmcarthur has joined #openstack-meeting13:20
*** diablo_rojo has quit IRC13:22
*** Liang__ has joined #openstack-meeting13:23
*** diablo_rojo has joined #openstack-meeting13:23
*** jamesmcarthur has quit IRC13:25
*** ttsiouts has joined #openstack-meeting13:26
*** enriquetaso has joined #openstack-meeting13:28
*** lbragstad has joined #openstack-meeting13:32
*** rubasov has joined #openstack-meeting13:38
*** lpetrut has quit IRC13:43
*** boxiang has quit IRC13:49
*** boxiang has joined #openstack-meeting13:52
*** liuyulong has joined #openstack-meeting13:53
*** mlavalle has joined #openstack-meeting13:54
*** boxiang has quit IRC13:55
*** boxiang has joined #openstack-meeting13:55
*** lajoskatona has joined #openstack-meeting13:56
*** jrbalderrama has joined #openstack-meeting13:58
*** eharney has joined #openstack-meeting14:00
slaweq#startmeeting networking14:01
openstackMeeting started Tue Jun  4 14:01:07 2019 UTC and is due to finish in 60 minutes.  The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot.14:01
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.14:01
*** openstack changes topic to " (Meeting topic: networking)"14:01
openstackThe meeting name has been set to 'networking'14:01
*** lseki has joined #openstack-meeting14:01
slaweq#topic Announcements14:02
*** openstack changes topic to "Announcements (Meeting topic: networking)"14:02
slaweqMilestone Train-1 is this week14:02
slaweq#link https://releases.openstack.org/train/schedule.html14:02
slaweqmlavalle: do we need to release new versions this week then?14:03
mlavalleyes, that's the agreement we did last ccyle14:03
*** haleyb has joined #openstack-meeting14:04
mlavallerelease stable branches14:04
*** munimeha1 has joined #openstack-meeting14:04
slaweqthat's what I thought14:04
slaweqso amotoki and mlavalle You will do this, right?14:04
mlavallehaleyb did that last time14:04
* haleyb arrives after being in nickserv detention14:04
haleybi actually started stable patches, but think there are one or two more things to merge in each branch14:05
mlavallehaleyb: will you create the patches for stable branches this time around?14:05
mlavallecool, thanks!14:05
*** shintaro has joined #openstack-meeting14:05
haleybso yes, i can do it, i'll ping people about getting reviews done on stragglers14:05
slaweqhaleyb: thx14:05
slaweqnext one then14:06
slaweqJust a reminder: Open Infrastructure Summit CFP Open - Deadline: July 214:06
slaweqlink: http://lists.openstack.org/pipermail/openstack-discuss/2019-May/006262.html14:06
slaweqso if You want to have propose some talk for next Summit, it's time to prepare proposal :)14:07
slaweqanother reminder, is someone missed it on previous meeting: Photos from recent Denver PTG are available at:14:07
slaweq#link https://www.dropbox.com/sh/fydqjehy9h5y728/AAC1gIc5bJwwNd5JkcQ6Pqtra/Neutron?dl=0&subfolder_nav_tracking=114:07
slaweqand last reminder14:08
slaweqTrain PTG summary:14:08
slaweq#link http://lists.openstack.org/pipermail/openstack-discuss/2019-May/006408.html14:08
slaweqany other announcements?14:08
mlavallenone from me14:09
slaweqok, lets move on then14:09
slaweq#topic Blueprints14:10
*** openstack changes topic to "Blueprints (Meeting topic: networking)"14:10
slaweqThis is what we have planned for T-1:14:10
slaweq#link https://launchpad.net/neutron/+milestone/train-114:10
mlavalleon this topic I have to say that last week we completed https://blueprints.launchpad.net/neutron/+spec/smart-nic-ovs14:11
slaweqI just wanted to ask if we need anything else for that one14:11
slaweqthx mlavalle14:11
mlavallethis is the patch that completed the feature: https://review.opendev.org/#/c/586252/14:11
mlavalleit merged last week14:12
mlavalleon https://blueprints.launchpad.net/neutron/+spec/enginefacade-switch14:12
*** _hemna has joined #openstack-meeting14:13
mlavallewe are making good progress14:13
*** shintaro has quit IRC14:13
mlavallewe are merging patches14:13
munimeha1can we add this one to train-1 also https://bugs.launchpad.net/neutron/+bug/145889014:13
openstackLaunchpad bug 1458890 in neutron "[RFE] Add segment support to Neutron" [Wishlist,Triaged] - Assigned to Carl Baldwin (carl-baldwin)14:13
mlavalleand there are several more in the pipeline14:13
munimeha1based on changes for routed networks to support multiple segments14:14
*** mriedem is now known as mriedem_away14:14
munimeha1we can also levereage same to enhance sriov14:14
mlavallethanks to ralonsoh and njohnston for their hard work on this14:14
slaweqmlavalle: yes, I saw couple of patches merged/close to merge related to this one14:15
slaweqand it looks that it's going pretty easy so far :)14:15
slaweq(too easy :P)14:15
njohnstonwe're plucking the low hanging fruit first :-)14:15
mlavallewell, I think we are starting to hit the difficult ones14:15
mlavallehere's an example for last night: https://review.opendev.org/#/c/662869/14:16
mlavalleso don't let your guard down and stay ready for a long fight14:16
njohnstonamen brother14:16
slaweqthx You all for working on this14:17
mlavallemunimeha1: regarding multiple segments in routed networks, we already have that as a target: https://blueprints.launchpad.net/neutron/+spec/multiple-segment-per-network-per-host14:17
slaweqmunimeha1: according to Your question, I think that we can only add Your rfe to T-2 as T-1 is this week :)14:17
slaweqmlavalle: what do You think?14:17
munimeha1ok T-2 is fine14:18
mlavallemunimeha1: but look at the blueprint that I just mentioned ^^^^14:18
mlavallethat is what you are talking about, right?14:18
mlavalleso it is already targeted for T-1. it is going to slip to T-214:19
mlavallebt you can help revieiwing the spec: https://review.opendev.org/#/c/65717014:19
mlavallewhich I ask others to do as well14:19
slaweqspeaking about https://blueprints.launchpad.net/neutron/+spec/multiple-segment-per-network-per-host there is patch https://review.opendev.org/#/c/657170 waiting for review also14:19
mlavalleit is my goal for this week14:19
mlavalleslaweq: right. thanks for bringing it up14:19
slaweqYou were faster mlavalle :)14:20
munimeha1Did you notice this one is related https://review.opendev.org/#/c/656885/14:20
mlavallemunimeha1: yes, that one is a debt that Nova has owed to us for a long time14:21
lajoskatonaand don't forget the bug for routed networks: https://bugs.launchpad.net/neutron/+bug/182854314:21
openstackLaunchpad bug 1828543 in neutron "Routed provider networks: placement API handling errors" [Medium,New] - Assigned to Lajos Katona (lajos-katona)14:21
*** zbr has quit IRC14:21
mlavallelajoskatona: thanks. I owe you a response to an email. Haven't forgotten14:21
* mlavalle was feeling bad this morning while walking the dog14:22
lajoskatonamlavalle: thanks, now in keystoneauth there is a fix for exception handling14:22
slaweqso I will update this BP https://blueprints.launchpad.net/neutron/+spec/multiple-segment-per-network-per-host to include links to all those things there, ok for You?14:22
*** zbr has joined #openstack-meeting14:22
lajoskatonamlavalle: so I continue to see what is missing to make this working again and add tests14:22
mlavallecool, thanks14:23
lajoskatonaslaweq: ok for me,  thanks14:23
slaweqok, I will do it after the meeting14:23
slaweqlets move on then14:23
slaweqwe have also BP https://blueprints.launchpad.net/neutron/+spec/event-notifier-ironic14:23
slaweqand we have already merged https://review.opendev.org/#/c/658787/14:23
slaweqwhich looks like the only needed thing on our side14:23
slaweqdo You know if we can consieder it as done maybe?14:24
slaweqor what else is missing on our side for it14:24
mlavallehang on...14:24
mlavalleI think that's it14:25
mlavallewe finished that one14:25
slaweqgreat, so mlavalle can You update BP and set proper status for it?14:25
mlavalleyes I will14:25
slaweqthx a lot mlavalle14:26
slaweqok, lets move on14:26
mlavalleand I will also go over the PTG summary this week14:26
mlavalle there are other BPs in there that we need to add to the dashboard14:26
slaweqthx mlavalle14:26
slaweq#topic Bugs14:27
*** openstack changes topic to "Bugs (Meeting topic: networking)"14:27
slaweqlast week amotoki was our bug deputy14:27
slaweqhe sent summary http://lists.openstack.org/pipermail/openstack-discuss/2019-June/006850.html14:27
slaweqI would especially want to ask db experts (looking at You njohnston :)) to look at https://bugs.launchpad.net/neutron/+bug/183100914:28
openstackLaunchpad bug 1831009 in neutron "Improper close connection to database leading to mysql/mariadb block connection." [Undecided,New]14:28
mlavalleis this reported against master branch?14:29
mlavalleoh, I see he is reporting the neutron version14:29
slaweqno, it's Neutron-server: openstack-neutron-13.0.2-1.el7.noarch14:29
mlavallethat's rocky14:30
slaweqyes, I think so :)14:30
slaweqwe also have 2 bugs related to api performance: https://bugs.launchpad.net/neutron/+bug/1830679 and https://bugs.launchpad.net/neutron/+bug/183063014:31
openstackLaunchpad bug 1830679 in neutron "Security groups RBAC cause a major performance degradation" [High,New]14:31
openstackLaunchpad bug 1830630 in neutron "Get external networks too slowly because it would join subnet and rbac" [Medium,New]14:31
slaweqboth are not assigned so would be good if someone would have time to take a look14:32
slaweqany other bugs You want to talk about today?14:32
*** boden has joined #openstack-meeting14:33
slaweqok, I get it as no :)14:34
slaweqthis week our bug deputy is haleyb14:34
* mlavalle will try to take a look at some of the bugs mentioned above ^^^^14:34
slaweqand next week it will be manjeets14:34
haleybslaweq: thanks for the reminder14:34
slaweqmlavalle: will You contact manjeets or should I ask him if he will be able to do it?14:35
slaweqhaleyb: yw :)14:35
*** armax has joined #openstack-meeting14:35
mlavalleslaweq: go ahead and ping him, please14:35
slaweqsure, I will do14:35
slaweqok, next topic then14:35
slaweq#topic neutron-lib14:35
*** openstack changes topic to "neutron-lib (Meeting topic: networking)"14:35
slaweqboden: hi :)14:36
bodenthere's a neutron-lib release patch out for review https://review.opendev.org/#/c/661839/   as far as I'm concerned it's good to go14:36
* mlavalle just +1ed it14:37
bodenwould be nice to get that out as I think some folks are waiting for it14:37
bodenok thanks!14:37
mlavalleyeah prometheanfire was very vocal last week14:37
bodenthe other topic is in regards to reviews for non-neutron networking projects (sfc, l2gw, etc.)14:37
bodenwondering if we can look to expand the +2 powers in some of these14:37
bodenperhaps the neutron core team14:37
slaweqboden: that was added to "on demand" section :)14:38
slaweqbut we can talk about it now14:38
slaweqthx for bringing this up14:38
lajoskatonaboden, mlavalle, slaweq: that was discussed during ptg as well, and with mlavalle we planned to have a meeting with l2gw ptl14:39
mlavalleis this for neutron-lib consumption patches?14:39
liuyulongI have a question, we have decided to use neutron-lib master branch for neutron CI, or not?14:39
bodenmlavalle yes and no... in general its not easy to land patches in some gates due to lack of reviews.. these could be neutron-lib related or not14:39
slaweqyes, I agree with boden here14:40
mlavalledo me a favor. Send me an email with the list of projects14:41
mlavalleand I'll make sure all the core team members have +2 rights in them14:41
slaweqsometimes we have some "trivial" changes e.g. related to jobs definitions or some "mechanical" patches in stadium projects and there is no many people who can +2 those patches14:41
bodenmlavalle cool thanks14:42
bodenliuyulong we never got anywhere on that discussion14:42
slaweqmlavalle: IMHO all stadium projects should be in this list14:42
bcafarelsounds good, recent example is the rehoming of tempest plugins14:42
mlavalleI will also start a personal routines to go over the pending patches in those projects14:42
mlavalleyeah, but it's more than stadium14:42
mlavallelajoskatona: yes, I will ping ricardo today14:43
lajoskatonamlavalle: thanks,14:43
lajoskatonamlavalle: otherwise the idea to add +2 right for cores is good.14:44
lajoskatonaI mean for stadium projects14:44
bodenwe can start with stadium I guess and go from there... if other projects crop up we can discuss expanding +2 rights I suppose14:44
njohnstonI think you can just include the neutron-cores group in the list of cores for each of the stadium projects... IIRC that is how the stable cores work for stadium projects (at least fwaas)14:45
amotokiin my understanding, neutron-release team members have an appripriate right for stadium projects, but it is not true for third-party prijects.14:45
mlavallesounds good boden14:45
bodennothing else from me... just business as usual14:45
slaweqok, thx boden for updates14:45
-amotoki- just got home and is scrolling back the logs14:46
slaweqanything else regarding neutron-lib?14:46
slaweqor we can move on?14:46
bodennothing from me14:47
slaweqok, lets move on then14:47
slaweqlast topic14:47
slaweq#topic On demand agenda14:47
*** openstack changes topic to "On demand agenda (Meeting topic: networking)"14:47
slaweqwe already discussed one of things from the list14:47
slaweqthere is also one from njohnston14:47
slaweq"Cleaning old OVO compatibility code."14:47
*** yamamoto has quit IRC14:48
slaweqgo on njohnston14:48
njohnstonoh, I think we talked about that last week; my apologies for not cleaning it off the agenda14:48
slaweqnjohnston: no problem :)14:48
slaweqso I will clean it today to not forget next time14:48
slaweqanything else You want to talk about today?14:48
slaweqif not, I will give You couple of minutes back :)14:49
mlavalleI don't have anything else14:49
liuyulongHold on14:49
amotokiI have nothing special. I still thinks njohnston's topic is worth checked by everyone https://review.opendev.org/#/c/661995/14:49
liuyulongOne minute about that neutron-lib dev14:50
tidwellrnjohnston: I owe you an update on https://review.opendev.org/#/c/661995/14:50
njohnstonthanks amotoki and tidwellr for great feedback on that14:50
njohnstonthe change is to define a policy as regards OVO object compatibility code in such a way that we respect the upgrade windows for distributions; please take a look and let us know what you thing14:51
tidwellrnjohnston: I can add this to the review, but as long as we suport N->N+2 or greater I think that will be acceptable14:51
amotokinjohnston: you're welcome. thanks for leading this coordination :)14:52
liuyulongEverytime if neutron touch a constant or a utils function, we will try to move it to neutron-lib, and then neutron dev was blocked by such procedure.14:52
liuyulongSo it neutron CI does not run on master neutron-lib, this will happen again.14:52
slaweqliuyulong: we aren't blocked but a bit slowered down I would say :)14:52
amotokiliuyulong: it is true if it affects other stadium projects other than neturon. is my understanding right?14:53
*** yamamoto has joined #openstack-meeting14:53
*** yamamoto has quit IRC14:53
liuyulongamotoki, not exactly, event a constant is only used by neutron itself.14:53
bodenat one point we were discussing something like that topic here https://review.opendev.org/#/c/602748/.  but no one really had any interest14:54
*** yamamoto has joined #openstack-meeting14:54
bodenthere are pros and cons to using neutron-lib master vs. released... if someone wants to kick the tires on this topic again go ahead, but I probably don't have the bandwidth14:55
mlavalleliuyulong: if a constant or function is used only by neutron, it doesn't have to be sent to the lib14:55
liuyulongSo, I'd like to say, if we should ensure neutron happy first, then do that move.14:55
*** yamamoto has quit IRC14:55
amotokiliuyulong: if the neutron repo is an only consumer of some constant, there is no need for neutron-lib in pricinpal.14:55
*** yamamoto has joined #openstack-meeting14:55
boden^ true14:55
mlavalleamotoki is right14:55
*** yamamoto has quit IRC14:55
mlavallein fact, if that util function or constant is only used by neutron, it shouldn't go to the lib14:56
liuyulongYes, agress14:56
bodenyep, that was the idea14:56
*** lpetrut has joined #openstack-meeting14:57
amotokiAPI definitions are exceptions as we would like to define the neutron API as a whole of the neutron stadium. All others are to share something in stadium projects.14:57
*** jrbalderrama has quit IRC14:57
*** Liang__ has quit IRC14:58
liuyulongHowever, if a function is only used by neutron alone temporarily, IMO, it should implement in neutron first.14:58
*** shilpasd has quit IRC14:58
amotokiliuyulong: you'ree right14:59
slaweqok, we are running out of time here14:59
slaweqthx for attending14:59
slaweqand have a great week!14:59
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"14:59
openstackMeeting ended Tue Jun  4 14:59:38 2019 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)14:59
openstackMinutes:        http://eavesdrop.openstack.org/meetings/networking/2019/networking.2019-06-04-14.01.html14:59
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/networking/2019/networking.2019-06-04-14.01.txt14:59
openstackLog:            http://eavesdrop.openstack.org/meetings/networking/2019/networking.2019-06-04-14.01.log.html14:59
*** boden has left #openstack-meeting14:59
*** cheng1__ has joined #openstack-meeting15:00
*** mlavalle has left #openstack-meeting15:01
*** ttsiouts has quit IRC15:04
*** lajoskatona has left #openstack-meeting15:04
*** ttsiouts has joined #openstack-meeting15:05
*** ttsiouts has quit IRC15:09
*** jamesmcarthur has joined #openstack-meeting15:11
*** yamamoto has joined #openstack-meeting15:15
*** yamamoto has quit IRC15:15
*** yamamoto has joined #openstack-meeting15:15
*** _hemna has quit IRC15:18
*** yamamoto has quit IRC15:20
*** lpetrut has quit IRC15:29
*** e0ne has quit IRC15:32
*** armax has quit IRC15:34
*** gyee has joined #openstack-meeting15:35
*** yamahata has joined #openstack-meeting15:39
*** bnemec has quit IRC15:44
*** bnemec has joined #openstack-meeting15:46
*** iyamahat has joined #openstack-meeting15:46
*** cheng1__ has quit IRC15:55
slaweq#startmeeting neutron_ci16:01
openstackMeeting started Tue Jun  4 16:01:05 2019 UTC and is due to finish in 60 minutes.  The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot.16:01
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:01
*** openstack changes topic to " (Meeting topic: neutron_ci)"16:01
openstackThe meeting name has been set to 'neutron_ci'16:01
bcafarelhi again16:01
slaweqI know that mlavalle will not be able to join this meeting today16:02
slaweqso I think we can start now16:02
slaweqGrafana dashboard: http://grafana.openstack.org/dashboard/db/neutron-failure-rate16:02
*** davidsha has quit IRC16:02
slaweq#topic Actions from previous meetings16:02
*** openstack changes topic to "Actions from previous meetings (Meeting topic: neutron_ci)"16:02
slaweqslaweq to continue work on fetch-journal-log zuul role16:03
slaweqI did patch https://review.opendev.org/#/c/643733/ and in https://review.opendev.org/#/c/661915/ it's tested that it works16:03
slaweqbut I'm not sure if this will be needed in fact16:03
slaweqI noticed recently that full journal log is already dumped in devstack.journal.xz.gz file in job's logs16:03
slaweqbut it's in binary journal format so You need to download it and use journalctl to examine it16:04
njohnstongood to know!16:04
slaweqyeah, I had no idea about that before16:04
bcafarelmaybe we could use a role to translate that file in job definition?16:05
slaweqbcafarel: You mean to text file?16:05
bcafarelyes, to skip the download step16:06
bcafarelfor lazy folks like me :)16:06
slaweqit's basically what role proposed by me in https://review.opendev.org/#/c/643733/ is doing16:06
slaweqwe can try to convice infra-root people that it may be useful :)16:07
clarkbslaweq: I'm still of the opinion that we should capture the serialized version in its entirety and compress it well as it is a very large file16:08
fungiit ends up taking up a lot of additional disk space in some jobs to keep two copies of the log, and the native version is useful for more flexible local filtering16:08
clarkbslaweq: it gives you way more flexibility that way, with a small amount of setup16:08
clarkbif we know we need specific service log files we can pull those out along with the openstack service logs16:09
clarkbbut for the global capture I think what we've got now with the export format works really well16:09
slaweqclarkb: yes, but e.g. for neutron it is many different services, like keepalived, haproxy, dnsmasq which are spawned per router or network16:09
clarkbyup which is why the current setup is great. You can do journalctl -u keepalived -u haproxy -u dnsmasq -u devstack@q-agt and get only those logs interleaved16:10
clarkbyou can also set date ranges to narrow down what you are looking at16:11
clarkbyou cannot easily do that with the chnage you have proposed16:11
slaweqclarkb: so as I said, I can just abandon this my patch as now I know that there is this log available and how to get it16:11
slaweqI think that this will be the best approach, what do You think?16:11
clarkboh I was going to suggest you update the role so that you don't have to use devstack to get that functionality16:11
clarkbbasically do what devstack does but in a reconsumable role in zuul-jobs16:12
clarkbthen we can apply it to non devstack jobs16:12
slaweqclarkb: tbh I need it to devstack based jobs so it's enough for us what is now :)16:12
njohnstonI bet we could easily create a grabjournal script that could fetch the journal for a specific job of a specific change and run that journalctl command on it, if our issue is just that we want to make it more accessible for developers16:12
clarkbslaweq: ok16:13
slaweqnjohnston: initially I wasn't aware of that this log exists already in job's logs, so I proposed patch16:13
slaweqbut now IMO the only "issue" is this accessibility of log and I think that this is not something which we should spent a lot of time on :)16:14
njohnston+1 sounds good16:14
bcafarelyes, I can survive a download+parse :)16:15
slaweqso my proposal is: lets for now use what is already there16:15
*** Shrews has joined #openstack-meeting16:15
slaweqand we will see if this will have to be improved somehow :)16:15
slaweqok, so lets move on to the next action16:16
slaweqslaweq to remove neutron-tempest-plugin-bgpvpn-bagpipe from "neutron-tempest-plugin-jobs" template16:16
slaweqDone: https://review.opendev.org/#/c/661899/16:16
*** Shrews has left #openstack-meeting16:16
slaweqand the last one was;16:16
slaweqmlavalle to debug neutron-tempest-plugin-dvr-multinode-scenario failures (bug 1830763)16:16
openstackbug 1830763 in neutron "Debug neutron-tempest-plugin-dvr-multinode-scenario failures" [High,Confirmed] https://launchpad.net/bugs/1830763 - Assigned to Miguel Lavalle (minsel)16:16
slaweqbut as mlavalle is not here now, I think we can assign it to him for next week to not forget about it16:17
slaweq#action mlavalle to debug neutron-tempest-plugin-dvr-multinode-scenario failures (bug 1830763)16:17
slaweqdo You have anything else to add regarding actions from last week?16:17
bcafarelall good here16:18
slaweqok, lets move on16:18
slaweq#topic Stadium projects16:18
*** openstack changes topic to "Stadium projects (Meeting topic: neutron_ci)"16:18
slaweqPython 3 migration16:18
slaweqStadium projects etherpad: https://etherpad.openstack.org/p/neutron_stadium_python3_status16:18
slaweqany updates on it?16:18
bcafarelnot from me sorry no progress here16:19
njohnstonme neither16:19
slaweqsame from my side16:19
slaweqso next thing16:19
slaweqtempest-plugins migration16:19
*** wwriverrat has quit IRC16:19
slaweqEtherpad: https://etherpad.openstack.org/p/neutron_stadium_move_to_tempest_plugin_repo16:19
slaweqfor networking-bgpvpn both main patches are merged16:20
*** wwriverrat has joined #openstack-meeting16:20
slaweqbut there was need to do some follow up cleanup which I forgot to do16:20
slaweqso there is also https://review.opendev.org/#/c/662231/ waiting for review16:20
njohnstonoh, what needed to be cleaned up?16:20
*** mriedem_away is now known as mriedem16:20
njohnstonah I see16:20
slaweqand there was https://review.opendev.org/#/c/662142/ from masayukig but this one is already merged16:20
slaweqso please also remember about that in Your patches :)16:21
slaweqand that's all from my side about this16:21
slaweqany updates on Your side?16:21
bcafarelfor sfc https://review.opendev.org/#/c/653747 is close to merge (second patch), pending on some gate fixes16:21
bcafarelso mostly gerrit work left to do :)16:22
slaweqbcafarel: great :)16:22
njohnstonI've been letting fwaas sit, but I will probably be able to dedicate some time to it about a week from now16:22
slaweqnjohnston: great, if You would need any help, ping me :)16:23
slaweqok, so lets move on to the next topic then16:24
slaweq#topic Grafana16:24
*** openstack changes topic to "Grafana (Meeting topic: neutron_ci)"16:24
slaweq#link http://grafana.openstack.org/dashboard/db/neutron-failure-rate16:24
slaweqstill IMO things looks relatively good16:25
slaweqthe biggest problem which we have are some failures in functional/fullstack tests16:25
slaweqand those ssh failure in tempest jobs16:25
slaweqbut I think it happens less often recently :)16:25
njohnstonI did have one general grafana question; in the "Number of Functional/Fullstack job runs (Gate queue)" graph I see that the lines diverge.  Does that mean we lost data somewhere?  I can't imagine a scenario where neutron-functional runs but not neutron-functional-python2716:26
njohnstonthat's just one example; it happens from time to time elsewhere as well16:27
*** lpetrut has joined #openstack-meeting16:27
slaweqgood question njohnston but I don't know an answer16:29
haleybthat is interesting, if it's only +/- 1 i could just see it being one had finished at that point in time but the other hadn't16:29
slaweqIMHO it may be lack of some data collected by infra - we have some gaps from time to time on graphs so maybe it's also something like that16:29
njohnstonthat's why I picked that one: neutron-functional = 10; neutron-functional-python27 = 7; neutron-fullstack = 616:29
njohnstonyeah, it's just weird, something to be aware of16:30
slaweqI agree, thx njohnston for pointing this16:31
slaweqanything else regarding grafana in general?16:32
haleybspeaking of failure rates, i did update the neutron-lib dashboard last week, not that we talk about it much here, just an FYI16:32
slaweqhaleyb: thx16:32
haleybit looks similar to this one and ovn now16:32
slaweqI should take a look at it from time to time too16:32
*** mattw4 has joined #openstack-meeting16:33
slaweqnot too much data there yet :)16:33
haleybnope, not many failures or patches16:33
slaweqsome periodic jobs are failing constantly16:35
slaweqwhich is maybe worth to check16:35
haleybi think they are known failures, but will look!16:36
slaweqhaleyb: thx a lot16:36
slaweqok, lets move on then16:36
slaweq#topic fullstack/functional16:36
*** openstack changes topic to "fullstack/functional (Meeting topic: neutron_ci)"16:36
slaweqI was looking at results of some failed jobs from last couple of days16:37
slaweqand I found 2 new failed tests in functional job16:37
slaweqhttp://logs.openstack.org/78/653378/7/check/neutron-functional/c5ac6a3/testr_results.html.gz and16:37
*** mattw4 has quit IRC16:37
slaweqbut each of those I saw only once16:38
slaweqdid You maybe see something like that before?16:38
haleybi haven't seen it, but failed running sysctl in the first?16:39
slaweqhere are logs from this first failed test: http://logs.openstack.org/78/653378/7/check/neutron-functional/c5ac6a3/controller/logs/dsvm-functional-logs/neutron.tests.functional.agent.l3.test_dvr_router.TestDvrRouter.test_dvr_ha_router_unbound_from_agents.txt.gz#_2019-06-02_14_15_09_38816:39
haleyboh, rtnetlink error, that could be a bug?16:40
haleybi.e. delete_gateway should deal with it16:41
slaweqyes, it's possible16:42
slaweqI will try to look deeper into this log this week16:42
slaweqmaybe I will find something obvious to change :)16:42
njohnstonfor the second one nothing jumps out at me but this one error: http://logs.openstack.org/82/659882/3/check/neutron-functional-python27/5e30908/controller/logs/dsvm-functional-logs/neutron.tests.functional.agent.test_l2_ovs_agent.TestOVSAgent.test_assert_br_phys_patch_port_ofports_dont_change.txt.gz#_2019-05-30_08_52_08_87316:42
slaweq#action slaweq to check logs with failed test_dvr_ha_router_unbound_from_agents functional test16:42
haleybslaweq: i know rodolfo had a patchset up to change this area to use privsep, so might want to start there16:43
slaweqhaleyb: good to know, thx16:43
slaweqI will ask him when he will back16:43
haleyberrors will change after that, into some pyroute2 one perhaps?16:44
slaweqok, I will keep it in mind, thx :)16:45
slaweqnjohnston: for the second one, this error can be maybe related16:46
slaweqor maybe not :)16:47
slaweqnjohnston: this test is stoping agent: https://github.com/openstack/neutron/blob/86139658efdc739c6cc330304bdf4455613df78d/neutron/tests/functional/agent/test_l2_ovs_agent.py#L26116:48
slaweqand IMO this "error" in log is related to this agent's stop16:48
slaweqso for me it don't look like possible problem at first glance16:48
njohnstonI agree16:49
slaweqso lets keep this issue in mind, if it will happen more often we will investigate it :)16:50
slaweqdo You agree?16:50
slaweqok, I guess it means yes :)16:51
slaweqso that is all what I had for today16:51
slaweqdo You have anything else You want to talk about today?16:52
haleyb-1 from me16:52
bcafarelnothing from me either16:52
haleybi'm hungry16:52
slaweqhaleyb: I'm tired16:52
slaweqso let's finish it bit earlier today16:53
slaweqthx for attending :)16:53
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"16:53
openstackMeeting ended Tue Jun  4 16:53:18 2019 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:53
openstackMinutes:        http://eavesdrop.openstack.org/meetings/neutron_ci/2019/neutron_ci.2019-06-04-16.01.html16:53
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/neutron_ci/2019/neutron_ci.2019-06-04-16.01.txt16:53
openstackLog:            http://eavesdrop.openstack.org/meetings/neutron_ci/2019/neutron_ci.2019-06-04-16.01.log.html16:53
slaweqbonne appetite haleyb :)16:53
*** lpetrut has quit IRC16:55
*** mattw4 has joined #openstack-meeting16:58
*** _hemna has joined #openstack-meeting17:14
*** kopecmartin is now known as kopecmartin|off17:18
*** whoami-rajat has quit IRC17:24
*** yamahata has quit IRC17:27
*** iyamahat has quit IRC17:27
*** ociuhandu has quit IRC17:28
*** ekcs has joined #openstack-meeting17:35
*** tssurya has quit IRC17:38
*** ricolin has joined #openstack-meeting17:38
*** iyamahat has joined #openstack-meeting17:38
*** gyee has quit IRC17:41
*** iyamahat_ has joined #openstack-meeting17:45
*** jamesmcarthur has quit IRC17:46
*** yamamoto has joined #openstack-meeting17:47
*** _hemna has quit IRC17:48
*** iyamahat has quit IRC17:49
*** yamamoto has quit IRC17:52
*** yamahata has joined #openstack-meeting17:58
*** Shrews has joined #openstack-meeting18:01
*** brault has joined #openstack-meeting18:05
*** brault has quit IRC18:09
*** diablo_rojo has quit IRC18:12
*** whoami-rajat has joined #openstack-meeting18:25
*** diablo_rojo has joined #openstack-meeting18:31
*** ociuhandu has joined #openstack-meeting18:34
*** ociuhandu has quit IRC18:43
*** e0ne has joined #openstack-meeting18:54
*** e0ne has quit IRC18:55
clarkbanyone else here for the infra meeting18:59
clarkbwe'll get started in a minute or two18:59
clarkbmust be a good sanwich18:59
fungisounds like it19:00
clarkb#startmeeting infra19:01
openstackMeeting started Tue Jun  4 19:01:16 2019 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.19:01
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.19:01
*** openstack changes topic to " (Meeting topic: infra)"19:01
openstackThe meeting name has been set to 'infra'19:01
clarkb#link http://lists.openstack.org/pipermail/openstack-infra/2019-June/006396.html19:01
clarkb#topic Announcements19:01
*** openstack changes topic to "Announcements (Meeting topic: infra)"19:01
clarkbThese didn't make it to the wiki agenda but I should mention them anyway. I'll be digging up rocks with family June 17-20 in the desert. Don't expect I'll have much of an internet connection19:02
clarkbas a result of that I plan to do ssl cert refreshes late next week19:02
clarkb(they expire on like the 20th or 21st)19:02
fungii hear that desert has too many rocks anyway, they won't miss a few19:02
Shrewsi'm sure it won't be hot at all19:03
fungihe's going before summer!19:03
fungi(a day before summer anyway)19:03
clarkbhigh of 83F today. Only 59F friday19:03
clarkbso could be scorching or cool19:03
clarkb#topic Actions from last meeting19:04
*** openstack changes topic to "Actions from last meeting (Meeting topic: infra)"19:04
clarkb#link http://eavesdrop.openstack.org/meetings/infra/2019/infra.2019-05-28-19.01.txt minutes from last meeting19:04
clarkbianw volunteered to look into what is necessary to ansible our backup server (instead of updating puppet 3 to 4 on it)19:04
clarkbianw: ^ anything to report on that yet?19:04
ianwi've started on https://review.opendev.org/#/c/662657 although it's in progress, but i think it will work19:05
ianwi think it should allow us to stop with all the manual user adds etc too (something i've had on my todo list for a long time!) ... so next week maybe something to review19:05
clarkb#link https://review.opendev.org/#/c/662657 backup server and backup ansible stuff19:05
clarkbthank you for kicking that off19:06
ianwalso slightly related is https://review.opendev.org/#/c/662662 if people could look to use ansible 2.8 on bridge ... it has some features helpful for deploying ssh keys19:06
clarkbthat will tie into the gitea mysql backups work I plan to start today19:06
clarkb#topic Specs approval19:07
*** openstack changes topic to "Specs approval (Meeting topic: infra)"19:07
clarkbNo specs up for approval but a call out for reviews. https://review.opendev.org/#/q/project:opendev/infra-specs+status:open is worth a look if you haev a moment (its on my todo list somewhere after followups on gitea things)19:07
clarkbcorvus: ^ you may be interested in the irc spec (involves email and irc)19:08
corvusack, thx19:08
clarkb#topic Priority Efforts19:08
*** openstack changes topic to "Priority Efforts (Meeting topic: infra)"19:08
clarkb#topic Update Config Management19:09
*** openstack changes topic to "Update Config Management (Meeting topic: infra)"19:09
clarkbAny news on this topic other than ianw's ansible backup work and my plan to do mysql backups on gitea servers?19:09
clarkbSeems we're starting to transition from concrete targets to organic changes on this item19:09
corvusi think we're also waiting on the gitea-project-creation-in-python work19:09
funginone i'm aware of19:09
clarkbcorvus: oh right19:10
corvusthat's a blocker for finishing up the independent zuul playbook19:10
clarkbright so the concrete target we have is the CD'ing end goal19:10
corvusmordred: now that you're back, do you think you might be able to get to that soon?  if not, i should be able to do it19:10
* mordred waves19:10
*** ricolin has quit IRC19:10
clarkbwhich deps on a few things like independent zuul playbook and gitea project management in python19:10
mordredI can work on that soon19:10
corvuscool, thx19:11
mordredI believe I have no outstanding higher-priority tasks to clear first19:11
corvusgreat, then once that's in place, we can run the zuul service playbook in test, and then in cd19:12
mordredluckily the logic is in ansible already, so that's awesome19:12
clarkbwe may want to consider this item as "done" (enough) and drop it from priority topics once we 've got CDing happening. Then we can transition to more organic updates like the mirror update and backup server update. Then decide if we want to have a concerted effort for replacing puppet19:13
clarkbbut should probably consider that once we get to the CDing state19:14
clarkbalright lets move on to the next topic19:15
clarkb#topic OpenDev19:15
*** openstack changes topic to "OpenDev (Meeting topic: infra)"19:15
fungithat's a mighty fine topic you have there19:15
clarkbLast friday we did the project renaming thing. We learned we had a few more gaps in the renaming playbooks but I think we fixed them all (gerrit start timeout, lack of var loading in play, missing grab of existing orgs to check if we need to create new orgs)19:16
clarkbwe've also got groups other than ara using zuul based git replication to github19:16
clarkband openstack is talking about how to manage their github orgs now too19:17
clarkball that to say I think we are in really good shape for managing new projects and renames (with the caveat that we still need gerrit downtime)19:17
clarkbthank you to everyone who helped with that19:17
corvuswe did miss one rename we intended to do19:18
corvusi forget which19:18
clarkbyup, it wasn't in the gerrit projects.yaml so didn't make it into my repos.yaml input for the playbook19:18
clarkbit was ansible-role-cloud-launcher19:18
clarkbajaeger has pushed changes up to get it on the list for next time19:18
clarkbI believe the spec for the migration is still open, but at this point I'm comfortable marking it complete. Any objections?19:20
clarkbopendev-gerrit specifically19:21
corvusi think we've done all the worke items19:21
corvusso yeah, i think it's done :)19:21
corvus#link opendev-gerrit spec https://docs.opendev.org/opendev/infra-specs/latest/specs/opendev-gerrit.html19:21
fungiwe also had a last-minute rename request... did we end up folding it into the batch? i don't recall now19:21
clarkbfungi: we did19:22
clarkbwe did not fold mnasers last minute request in due to people awnting more discussion on whether or not those belong in openstack/19:22
fungiokay, so it's just x/ansible-role-cloud-launcher which we're currently missing19:22
fungioh, right, i was confusing two last-minute requests19:22
fungiseems likely the ones mnaser brought up may move into the openstack org, so would still be renamed19:23
clarkbya but to a different target19:23
mnaseryes, sorry, I will -w that for now19:23
fungibut i'm expecting them to be in the list for the next maintenance, whenever that winds up getting scheduled19:23
fungias the logistics will likely be worked out by then19:23
clarkbAnything else opendev related worth bringin up?19:24
clarkbcorvus: ^ did you want to talk about tarballs.opendev.org/19:24
corvusoh, yeah...19:25
corvuszuul needs a place to put branch-tip tarballs before it can move out of the openstack zuul tenant19:25
corvus(presumably, other opendev projects will want the same)19:26
corvussome opendev admins and zuul maint folks had a quick chat in #zuul about it, and we figured making tarballs.opendev.org, similar to tarballs.openstack.org, is the best way forward for now19:26
corvusi don't think that's controversial, so i plan on just doing it19:27
corvuschanges are in progress and nearly complete19:27
clarkbthere are two big differences to the old oepnstack tarballs ost which is it will be using afs instead of "local" disk and projects are expected to be fully namespaced (not just using shortnames)19:27
corvusmain differences between this and existing tarballs site are: 1) new site will be in afs; 2) it will be scoped by full project name rather than the short name19:27
clarkbyay I didn't say inaccurate things :)19:28
corvus#link opendev tarball publish jobs https://review.opendev.org/66287719:28
corvusthat's the current tip of development (follow depends-on for the rabbit hole19:28
corvusi'm currently testing its dependencies19:28
clarkbthanks for the update on that.19:28
corvusnp.  eol.19:29
clarkbany other opendev items before we move on to storyboard?19:29
*** e0ne has joined #openstack-meeting19:29
fungialso we have the option to migrate tarballs.openstack.org into that afs volume if we update the publishing jobs and adjust teh vhost config for it, i think?19:30
fungi(well, amd move the vhost to somewhere like files.o.o or add an afs client on static.o.o)19:30
clarkbfungi: should be possible. basically have new runs of jobs publish to afs then rsync the old stuff over19:30
corvusyeah.  we should schedule that since we're changing publication locations19:30
*** e0ne has quit IRC19:30
fungiyeah, would like an eventual result where we're not indefinitely maintaining two different solutions for it19:30
corvusbut it's branch-tip tarballs -- i don't think it's going to be a big deal.19:31
corvuswell, it's also our release tarballs19:31
ianwone thought popped up ... we've had issues before with parallel vos release's ... i haven't looked but is good locking around this explicitly handled?19:31
corvusianw: i plan on adding it to the periodic vos release cron19:31
corvusso publication to the new site will be delayed up to 5m.19:31
fungii think that's fine if we set that expectation19:32
corvus(and that's only one actor)19:32
clarkbthat cron runs on afsdb01 iirc19:32
clarkbso ya only one actor for that19:32
fungiif jobs need a tighter timeline to reuse those artifacts, then they can use paused jobs or something19:32
fungilike the image jobs do it with their own ephemeral repository19:33
clarkbfungi: can also read from the RW afs mount19:33
corvuscould the pypi publication jobs be affected?19:33
fungii suppose, but probably better we just warn against reconsuming those artifacts in dependent jobs where you may race a vos release19:33
corvusoff the top of my head, i don't remember the sequence for that19:34
fungithe pypi publication jobs don't reconsume what's pushed onto the tarballs site any longer19:34
clarkbthat change happened with the zuulv3 transition19:34
fungithey used to, but the zuul v3 refactor elimniated teh push/pull/push model19:34
fungiyeah, what clarkb said19:34
corvusok we're probably good then19:34
fungiwe only did it back then to be able to have signing keys on a different (static) host for added protection19:34
fungizuul v3 solves that problem more cleanly for us19:35
fungiwell, and also pypi creds19:35
clarkbalright we have ~25 minutes left in our meeting so lets continue on19:35
clarkb#topic Storyboard19:35
*** openstack changes topic to "Storyboard (Meeting topic: infra)"19:35
clarkbfungi: diablo_rojo how are storyboard things?19:36
corvusoh, i set up the slow query log but have not pulled in the net to see what it's caught yet19:36
corvusi need to do that19:36
diablo_rojoNo negative feedback fromt he swift team yet19:36
diablo_rojopost test migration19:36
diablo_rojoso thats cool19:36
fungicorvus: i skimmed it the day after you turned it on and there are some egregious examples19:36
corvusfungi: great news!19:36
diablo_rojoWe should definitely pull that.19:37
diablo_rojoNow that our outreachy intern is getting started19:37
fungiqueries running over 3 minutes with inner-joins several layers deep and majorly compound selects19:37
corvus#action corvus do something with the slow query log19:37
diablo_rojoShe could use that to help target what she can improve19:37
fungicorvus: other great news is that SotK thinks some of those may be solved with a change he has up for review19:37
clarkbfungi: corvus diablo_rojo probably make sense to throw that up on paste.o.o or an etherpad and get eyeballs on it?19:37
diablo_rojo(we have an outreachy intern and she has started now)19:37
diablo_rojoYes please :)19:37
corvusok i'll do that asap19:37
diablo_rojocorvus, merci :)19:38
*** bbowen has quit IRC19:38
fungi#link http://paste.openstack.org/show/752274 an egregious example from the slow-query log19:38
funginote that's a single entry for a single query19:38
clarkband time is in seconds?19:38
clarkbsounds like progress at least :)19:39
corvusi wonder if we can pair that up with a request log to figure out what code path triggered it19:39
fungigranted, that was the longest query execution i could find in the first 24 hours of the log, so it's a doozy19:39
corvusone could probably identify the code location from the query alone, but it might be difficult.19:40
*** altlogbot_0 has joined #openstack-meeting19:40
*** enriquetaso has quit IRC19:40
fungii think SotK knew where it likely was off the top of his head19:40
corvusoh good19:40
corvusthen maybe let's just grab the queries, and if we can't figure it out from that, go to the request logs as needed.19:40
clarkbsounds like a plan19:41
fungiyeah, i think that's a pragmatic approach19:41
SotKYeah I can probably guess at most things from the queries :)19:41
clarkbAnything else storyboard related before we move on?19:41
fungijust feed them to the sotk api ;)19:41
fungiclarkb: nothing from me19:42
*** jamesmcarthur has joined #openstack-meeting19:42
clarkb#topic General Topics19:42
*** openstack changes topic to "General Topics (Meeting topic: infra)"19:42
clarkb#link https://etherpad.openstack.org/p/201808-infra-server-upgrades-and-cleanup19:42
clarkbmordred: any luck with status?19:42
clarkband fungi same question for you re wiki19:42
diablo_rojoNothing from me19:43
fungii have another puppet module change for more packages which needed replacing19:43
*** slaweq has quit IRC19:43
* fungi finds19:43
fungi#link https://review.opendev.org/661848 More transitional Xenial package name replacements19:43
fungiit's rather like whack-a-mole19:43
clarkblooks like it only needs one more +2 (I already got that one)19:44
fungithe entire kitchen sink was explicitly specified in the manifest19:44
clarkband now we have to do dishes19:44
fungiin a variety of places too19:44
fungiyeah, next time we need to do dishes i think it may be easier to burn down the kitchen and build a new one19:44
mordredclarkb: oh - crappit - THAT was the thing I had on my plate before gitea project creation19:44
*** slaweq has joined #openstack-meeting19:44
*** _hemna has joined #openstack-meeting19:45
clarkbmordred: yay for reminders then :)19:45
corvusmordred: too late, we just put the plate in the dishwasher19:45
corvusi think19:45
clarkbagain let me know if I can assist on these19:45
corvusit's hard to follow the metaphors here19:45
*** jamesmcarthur has quit IRC19:45
mordredI thought the plate got put in the mole19:45
clarkbNext up was following up on the mirror rebuilds19:45
*** jamesmcarthur has joined #openstack-meeting19:45
clarkbianw do you think we are ready to do that in all regions now? We did have a slight hiccup with mirror.dfw.rax.opendev.org after the apache disk ran out of space btu it had to do with afs ubuntu volume cache19:46
clarkbI did a flushvolume and all is well again19:46
ianwi think so19:46
fungiyeah, still a mystery what happened there19:47
ianw#link https://review.opendev.org/#/c/662357/19:47
ianwone possible very minor thing is to get the names a bit clearer on the logs ^19:47
fungilooks like we were returning truncated results from the afs cache starting around the time the apache proxy ran out of disk space in the other volume19:47
clarkbgetting that rolled out then adding https to the docker proxy will simplify image building things too so worthwhile to keep pushing on that19:47
ianwfungi: i'm not sure i want to think too hard on why they're related19:48
corvusalso it looks like it may be required in order to use buildkit to build images19:48
ianwbut yeah, the misconfiguration of disk sizes was my mistake19:48
clarkbNext item on the list is cleaning up our arm64 cloud configuration from nodepool. This is a bit intertwined with mordred's work to have nodepool builders build control plane images19:49
clarkbmordred: ^ if you want to start approving those I'm able to help keep an eye on stuff this afternoon19:49
clarkbwe had to cleanup nb03's disk and reboot it (restarting services) to get it building images again. But it is now doing that successfully so I think the functional problems are alrgely addressed19:50
clarkbnow just a matter of cleaning out the clouds that no longer exist19:50
mordredclarkb: sweet19:50
clarkbAnd then if we can sneak it in before our hour is up I wanted to bring up setting a flag day(s) for bindep fallback removal and z-c shim removal from our base jobs19:51
fungirelated to mirroring and arm64, hrw has a change up since yesterdayish to add arm64 mirrors of centos to our rsync config19:51
mordredclarkb: ++ to both of those19:51
clarkbThis week is an openstack milestone week but we are clear on openstack's shcedule for like the next 6 weeks19:51
fungi#link https://review.opendev.org/662727 centos-mirror-update: handle non-x86 architectures19:51
fungi(for reference)19:51
fungippc64 as well i guess19:52
clarkbmaybe do the switch on the 24th (that should be plenty of time for warning people and testing)19:52
ianwfungi: yeah, i can babysit that today19:52
clarkbfungi: ^ do you think the 24th would be too soon to do the bindep and z-c removals?19:52
fungiclarkb: of june? yeah that's probably sufficient notice19:53
fungiif we get to communicating broadly right away19:53
clarkbfungi: I think step 0 is get the chagne up for z-c removal so we can link to it in communications19:53
clarkbthen ya step 1 is send email todayish19:53
fungisure, i'll push it right after the meeting, was mostly looking for consensus on the direction first19:53
clarkband I figure one day with one big bandaid pull is better than multiple days19:53
fungii think my jaunt through time reminded me that it was something we'd been meaning to do early in the v3 transition and then we lost track of it19:54
clarkbI think we realized that people were making new legacy jobs19:54
clarkband so the jobs that depended on it were getting larger in number and not smaller19:54
corvusnext up after this, removing legacy-base :)19:55
clarkb#topic Open Discussion19:56
*** openstack changes topic to "Open Discussion (Meeting topic: infra)"19:56
clarkbWe have a few minutes for any last items19:56
fungior more sandwiches19:57
corvusi'm all out of both19:57
clarkbme too. My burger is gone19:57
clarkboh wait I have a small thing. Larissa has jury duty tomorrow so I'm doing dadops19:57
clarkbI'll be less useful tomorrow than today19:57
clarkbtrying to explain jury duty to a 4 year old is difficult if anyone is wondering19:58
mordredclarkb: does that make it doubly difficult to explain to 2 4 year olds?19:59
corvusit's been FOUR years?19:59
fungione reason the week of the 24th is good for ripping off the legacy job bandages is that i'll be out of town the following week so immune to the flamewar which ensues19:59
clarkbcorvus: as of two days ago it has been four years19:59
clarkbfungi: ha19:59
corvushappy birthdays!19:59
fungilook on the bright side... you only need to throw one birthday party ;)20:00
clarkbfungi: its great actually :)20:00
clarkband we are at time20:00
clarkbthank you everyone20:00
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"20:00
openstackMeeting ended Tue Jun  4 20:00:15 2019 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)20:00
openstackMinutes:        http://eavesdrop.openstack.org/meetings/infra/2019/infra.2019-06-04-19.01.html20:00
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/infra/2019/infra.2019-06-04-19.01.txt20:00
openstackLog:            http://eavesdrop.openstack.org/meetings/infra/2019/infra.2019-06-04-19.01.log.html20:00
fungithanks clarkb!20:00
clarkbmordred: I've only attempted to explain it to one of them so far20:01
clarkbthe other was napping or something20:01
*** Shrews has left #openstack-meeting20:02
*** radeks_ has joined #openstack-meeting20:06
*** radeks has quit IRC20:09
*** _hemna has quit IRC20:19
*** e0ne has joined #openstack-meeting20:23
*** Lucas_Gray has joined #openstack-meeting20:25
*** Lucas_Gray has quit IRC20:27
*** Lucas_Gray has joined #openstack-meeting20:29
*** enriquetaso has joined #openstack-meeting20:30
*** enriquetaso has quit IRC20:37
*** eharney has quit IRC20:41
*** wwriverrat has quit IRC20:41
*** e0ne has quit IRC20:41
*** e0ne has joined #openstack-meeting20:42
*** e0ne has quit IRC20:43
*** _hemna has joined #openstack-meeting20:43
*** dklyle has quit IRC20:49
*** dklyle has joined #openstack-meeting20:49
*** bbowen has joined #openstack-meeting20:52
*** whoami-rajat has quit IRC20:54
*** mattw4 has quit IRC20:55
*** mattw4 has joined #openstack-meeting20:58
*** jamesmcarthur has quit IRC21:00
*** jamesmcarthur has joined #openstack-meeting21:00
*** jamesmcarthur has quit IRC21:04
*** pcaruana has quit IRC21:07
*** Qiming has quit IRC21:11
*** Qiming has joined #openstack-meeting21:14
*** jamesmcarthur has joined #openstack-meeting21:14
*** jamesmcarthur has quit IRC21:18
*** jamesmcarthur has joined #openstack-meeting21:20
*** mriedem is now known as mriedem_away21:22
*** _hemna has quit IRC21:23
*** ekcs has quit IRC21:26
*** jamesmcarthur has quit IRC21:28
*** raildo has quit IRC21:29
*** munimeha1 has quit IRC21:39
*** jamesmcarthur has joined #openstack-meeting21:49
*** yamamoto has joined #openstack-meeting21:50
*** jamesmcarthur has quit IRC21:51
*** yamamoto has quit IRC21:55
*** jamesmcarthur has joined #openstack-meeting22:04
*** slaweq has quit IRC22:07
*** mattw4 has quit IRC22:08
*** slaweq has joined #openstack-meeting22:11
*** enriquetaso has joined #openstack-meeting22:13
*** jamesmcarthur has quit IRC22:16
*** enriquetaso has quit IRC22:18
*** enriquetaso has joined #openstack-meeting22:19
*** slaweq has quit IRC22:24
*** mattw4 has joined #openstack-meeting22:24
*** slaweq has joined #openstack-meeting22:34
*** enriquetaso has quit IRC22:44
*** slaweq has quit IRC22:47
*** artom has quit IRC22:50
*** radeks_ has quit IRC22:58
*** slaweq has joined #openstack-meeting23:04
*** slaweq has quit IRC23:17
*** Lucas_Gray has quit IRC23:17
*** ykatabam has joined #openstack-meeting23:17
*** rcernin has joined #openstack-meeting23:20
*** lseki has quit IRC23:22
*** ykatabam has quit IRC23:29
*** gyee has joined #openstack-meeting23:29
*** ykatabam has joined #openstack-meeting23:30
*** yamamoto has joined #openstack-meeting23:41
*** enriquetaso has joined #openstack-meeting23:43
*** slaweq has joined #openstack-meeting23:50
*** lseki has joined #openstack-meeting23:53

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!