| *** yamamoto has joined #openstack-meeting | 00:02 | |
| *** markvoelker has quit IRC | 00:24 | |
| *** Liang__ has joined #openstack-meeting | 00:27 | |
| *** _erlon_ has quit IRC | 00:29 | |
| *** Liang__ is now known as LiangFang | 00:30 | |
| *** ekcs has quit IRC | 00:38 | |
| *** Adri2000 has quit IRC | 00:40 | |
| *** mriedem has quit IRC | 00:44 | |
| *** Adri2000 has joined #openstack-meeting | 00:45 | |
| *** bobh has joined #openstack-meeting | 00:58 | |
| *** ttsiouts has joined #openstack-meeting | 01:00 | |
| *** yamamoto has quit IRC | 01:03 | |
| *** ttsiouts has quit IRC | 01:06 | |
| *** lseki has quit IRC | 01:10 | |
| *** gyee has quit IRC | 01:19 | |
| *** jamesmcarthur has quit IRC | 01:22 | |
| *** njohnston_ has quit IRC | 01:27 | |
| *** jamesmcarthur has joined #openstack-meeting | 01:31 | |
| *** jamesmcarthur has quit IRC | 01:43 | |
| *** jamesmcarthur has joined #openstack-meeting | 01:44 | |
| *** _hemna has joined #openstack-meeting | 01:53 | |
| *** bobh has quit IRC | 01:54 | |
| *** jamesmcarthur has quit IRC | 02:00 | |
| *** jamesmcarthur has joined #openstack-meeting | 02:00 | |
| *** hyunsikyang has quit IRC | 02:16 | |
| *** hyunsikyang has joined #openstack-meeting | 02:17 | |
| *** hongbin has joined #openstack-meeting | 02:17 | |
| *** _hemna has quit IRC | 02:17 | |
| *** yamahata has quit IRC | 02:24 | |
| *** jamesmcarthur has quit IRC | 02:39 | |
| *** jamesmcarthur has joined #openstack-meeting | 02:40 | |
| *** jamesmcarthur has quit IRC | 02:43 | |
| *** jamesmcarthur has joined #openstack-meeting | 02:43 | |
| *** yamamoto has joined #openstack-meeting | 02:51 | |
| *** _hemna has joined #openstack-meeting | 02:52 | |
| *** whoami-rajat has joined #openstack-meeting | 03:14 | |
| *** _hemna has quit IRC | 03:25 | |
| *** hyunsikyang has quit IRC | 03:37 | |
| *** igordc has quit IRC | 03:39 | |
| *** jamesmcarthur has quit IRC | 03:45 | |
| *** jamesmcarthur has joined #openstack-meeting | 03:46 | |
| *** jamesmcarthur has quit IRC | 03:49 | |
| *** jamesmcarthur_ has joined #openstack-meeting | 03:49 | |
| *** tashiromt has joined #openstack-meeting | 03:55 | |
| *** LiangFang has quit IRC | 03:59 | |
| samP | #startmeeting masakari | 04:00 |
|---|---|---|
| openstack | Meeting started Tue Jun 4 04:00:05 2019 UTC and is due to finish in 60 minutes. The chair is samP. Information about MeetBot at http://wiki.debian.org/MeetBot. | 04:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 04:00 |
| *** openstack changes topic to " (Meeting topic: masakari)" | 04:00 | |
| openstack | The meeting name has been set to 'masakari' | 04:00 |
| samP | Hi all for Masakari | 04:00 |
| *** Liang__ has joined #openstack-meeting | 04:01 | |
| samP | #topic Critical bugs and patches | 04:01 |
| *** openstack changes topic to "Critical bugs and patches (Meeting topic: masakari)" | 04:01 | |
| *** imsurit has joined #openstack-meeting | 04:01 | |
| samP | Please share any items need to discuss here. | 04:02 |
| *** yamahata has joined #openstack-meeting | 04:02 | |
| tashiromt | Hi | 04:02 |
| samP | tashiromt: Hi | 04:02 |
| samP | Seems like we do not have any item to discuss today. | 04:03 |
| *** tpatil has joined #openstack-meeting | 04:04 | |
| tpatil | Sorry to join late | 04:04 |
| samP | tpatil: no problem | 04:04 |
| *** shilpasd has joined #openstack-meeting | 04:04 | |
| samP | Now in the topic of Critical bugs and patches | 04:05 |
| samP | Any issues to share? | 04:05 |
| tpatil | samP: No critical bugs to be discussed from my side | 04:05 |
| samP | tpatil: thanks | 04:05 |
| samP | #topic Train work items | 04:06 |
| *** openstack changes topic to "Train work items (Meeting topic: masakari)" | 04:06 | |
| samP | Please share if you have any update on Train work items | 04:06 |
| *** igordc has joined #openstack-meeting | 04:06 | |
| tpatil | 1) Add devstack support to install host-monitors | 04:07 |
| tpatil | We are working on understanding how to configure STONITH resource | 04:07 |
| tpatil | I think in host-monitor, it's possible to use STONITH_TYPE ssh and external/ipmi | 04:07 |
| tpatil | Any idea, how does ssh works? | 04:08 |
| tpatil | samP: Also, in our dev env. we don't have support for IPMI, so we couldn't make any progress. I would like to discuss with you about the dev. env offline. | 04:09 |
| samP | tpatil: Sure, I will make time to do that. And I think we have IPMI in for bare metal nodes. I will share those details later | 04:11 |
| tpatil | #link : https://github.com/openstack/masakari-monitors/blob/master/masakarimonitors/hostmonitor/hostmonitor.sh#L707 | 04:11 |
| tpatil | above link for ssh STONITH_TYPE | 04:11 |
| tpatil | I have one question about STONITH_TYPE: ssh, if one compute node is down, how is it possible to interact with that node using SSH from other compute node? | 04:13 |
| samP | tpatil: Thanks for the link. Before use that STONITH method, we need to configure pacemaker cluster with ssh settings | 04:13 |
| samP | tpatil: you can't. STONITH is use to kill a node. | 04:13 |
| tpatil | samP: Kill a node proactively or after pacemaker detects an issue in that node? | 04:15 |
| samP | tpatil: As an Ex: if operator (or Pacemaker) find something wrong in node, then he (it) try to kill that node and isolate it from the other nodes | 04:16 |
| samP | tpatil: correct. | 04:16 |
| *** hongbin has quit IRC | 04:17 | |
| tpatil | samP: The job of stonith resource device is to kill that node completely, correct? | 04:18 |
| samP | tpatil: correct | 04:19 |
| tpatil | samP: in that sense, how does STONITH_TYPE ssh works is my main question? becoz it's possible that failed node may fail to response to ssh | 04:19 |
| tpatil | samP: We can discuss about supported STONITH_TYPE in host-monitors and how it works separately as I'm completely new in this area | 04:21 |
| samP | tpatil: yes, there is a possibility. Normally we configure several type of STONITH_TYPS such as IPMI, ssh .. etc. | 04:21 |
| samP | if ssh failed, then it will try to use the IPMI or any other methods | 04:22 |
| tpatil | samP: Ok | 04:22 |
| samP | different stonith method has its own advantages and disadvantages | 04:22 |
| *** haleyb has quit IRC | 04:22 | |
| *** armax has quit IRC | 04:22 | |
| samP | in ssh, you execute commands such as take the logs, wait for core dump. | 04:23 |
| samP | on the other hand it will take time. | 04:23 |
| samP | in IPMI, you can kill a node very quick. However, you may loose core dump in servers with large RAM | 04:24 |
| samP | tpatil: I will make detail doc for this and make time to explain about this. | 04:25 |
| tpatil | samP: That will help, Thank you. | 04:25 |
| samP | I will contact you offline for this. | 04:25 |
| tpatil | samP: Ok, Thank you | 04:26 |
| tpatil | 2) Real time rendering of recovery workflow in masakari-dashboard | 04:26 |
| tpatil | as explained in last weekly meeting, fastener commit will be reverted : https://github.com/harlowja/fasteners/issues/36 | 04:26 |
| tpatil | So it's a blocker to implement real time rendering as we need to add more progress details during the evacuation of instances during execution of workflow | 04:28 |
| tpatil | We haven't yet tried the workaround solution to monkey patch method from fastener to make this thing work in masakari | 04:28 |
| tpatil | 3) Improve documentation | 04:31 |
| samP | tpatil: Thanks for update. | 04:31 |
| tpatil | Unfortunately, not yet started. Will take up this task sometime in this week | 04:31 |
| samP | tpatil: no problem | 04:31 |
| tpatil | That's all update about Train items from my end | 04:32 |
| samP | tpatil: thanks | 04:32 |
| samP | I have no update from my side. | 04:33 |
| samP | I will make time to discuss about pacemaker and stonith settings. I will let you know the details | 04:33 |
| samP | #topic AOB | 04:33 |
| *** openstack changes topic to "AOB (Meeting topic: masakari)" | 04:33 | |
| samP | Any other issues need to discuss | 04:34 |
| tpatil | sam: OK | 04:34 |
| tpatil | samP: OK | 04:34 |
| tpatil | samP: No other topics for discussion from my side | 04:35 |
| samP | tpatil: thanks | 04:35 |
| *** zaneb has quit IRC | 04:35 | |
| samP | OK then, let's finish today's meeting here | 04:35 |
| *** baojg has quit IRC | 04:35 | |
| samP | Thank you all! | 04:35 |
| tpatil | samP: Ok, Thanks | 04:36 |
| samP | please use ML or #openstack-masakari IRC@freenode for further discussions. | 04:36 |
| samP | Thank you all again.. | 04:36 |
| samP | #endmeeting | 04:36 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 04:36 | |
| openstack | Meeting ended Tue Jun 4 04:36:37 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 04:36 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/masakari/2019/masakari.2019-06-04-04.00.html | 04:36 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/masakari/2019/masakari.2019-06-04-04.00.txt | 04:36 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/masakari/2019/masakari.2019-06-04-04.00.log.html | 04:36 |
| *** baojg has joined #openstack-meeting | 04:41 | |
| *** zaneb has joined #openstack-meeting | 04:47 | |
| *** pwu has joined #openstack-meeting | 04:57 | |
| *** tpatil has quit IRC | 04:58 | |
| *** lbragstad has quit IRC | 05:09 | |
| *** pcaruana has joined #openstack-meeting | 05:12 | |
| *** radeks has joined #openstack-meeting | 05:18 | |
| *** _hemna has joined #openstack-meeting | 05:22 | |
| *** pcaruana has quit IRC | 05:23 | |
| *** pwu_ has joined #openstack-meeting | 05:24 | |
| *** pwu has quit IRC | 05:24 | |
| *** pcaruana has joined #openstack-meeting | 05:28 | |
| *** jamesmcarthur_ has quit IRC | 05:29 | |
| *** e0ne has joined #openstack-meeting | 05:43 | |
| *** belmoreira has joined #openstack-meeting | 05:49 | |
| *** igordc has quit IRC | 05:51 | |
| *** Liang__ has quit IRC | 05:53 | |
| *** Liang__ has joined #openstack-meeting | 05:54 | |
| *** _hemna has quit IRC | 05:55 | |
| *** kopecmartin|off is now known as kopecmartin | 06:07 | |
| *** lpetrut has joined #openstack-meeting | 06:10 | |
| *** bbowen has joined #openstack-meeting | 06:12 | |
| *** slaweq has joined #openstack-meeting | 06:13 | |
| *** bbowen_ has quit IRC | 06:14 | |
| *** zaneb has quit IRC | 06:16 | |
| *** wwriverrat has joined #openstack-meeting | 06:17 | |
| *** zaneb has joined #openstack-meeting | 06:23 | |
| *** _hemna has joined #openstack-meeting | 06:25 | |
| *** hyunsikyang has joined #openstack-meeting | 06:34 | |
| *** dkushwaha has joined #openstack-meeting | 06:36 | |
| *** jbadiapa has joined #openstack-meeting | 06:41 | |
| *** yamamoto has quit IRC | 06:41 | |
| *** belmoreira has quit IRC | 06:43 | |
| *** ianychoi has quit IRC | 06:57 | |
| *** ianychoi has joined #openstack-meeting | 06:58 | |
| *** pwu_ has quit IRC | 06:58 | |
| *** _hemna has quit IRC | 06:58 | |
| *** yamamoto has joined #openstack-meeting | 06:59 | |
| *** ttsiouts has joined #openstack-meeting | 07:09 | |
| *** e0ne has quit IRC | 07:09 | |
| *** _hemna has joined #openstack-meeting | 07:11 | |
| *** ykatabam has quit IRC | 07:15 | |
| *** _pewp_ has quit IRC | 07:16 | |
| *** _hemna has quit IRC | 07:16 | |
| *** _pewp_ has joined #openstack-meeting | 07:16 | |
| *** slaweq has quit IRC | 07:16 | |
| *** rossella_s has joined #openstack-meeting | 07:17 | |
| *** yamamoto has quit IRC | 07:20 | |
| *** rubasov has joined #openstack-meeting | 07:21 | |
| *** yamamoto has joined #openstack-meeting | 07:21 | |
| *** yamamoto has quit IRC | 07:30 | |
| *** zbr has joined #openstack-meeting | 07:31 | |
| *** markvoelker has joined #openstack-meeting | 07:32 | |
| *** yamamoto has joined #openstack-meeting | 07:33 | |
| *** ttsiouts has quit IRC | 07:34 | |
| *** ttsiouts has joined #openstack-meeting | 07:35 | |
| *** yamamoto has quit IRC | 07:39 | |
| *** ttsiouts has quit IRC | 07:39 | |
| *** slaweq has joined #openstack-meeting | 07:40 | |
| *** yamamoto has joined #openstack-meeting | 07:42 | |
| *** jaewook_lee has joined #openstack-meeting | 07:55 | |
| *** boxiang has joined #openstack-meeting | 07:56 | |
| *** pwu has joined #openstack-meeting | 07:56 | |
| *** takahashi-tsc has joined #openstack-meeting | 08:01 | |
| *** ttsiouts has joined #openstack-meeting | 08:02 | |
| *** markvoelker has quit IRC | 08:05 | |
| *** tetsuro has joined #openstack-meeting | 08:07 | |
| *** takahashi-tsc has quit IRC | 08:10 | |
| *** bhagyashris has joined #openstack-meeting | 08:10 | |
| *** tpatil has joined #openstack-meeting | 08:11 | |
| *** imsurit has quit IRC | 08:13 | |
| *** jaewook_lee has quit IRC | 08:13 | |
| *** boxiang has quit IRC | 08:14 | |
| *** boxiang has joined #openstack-meeting | 08:14 | |
| *** imsurit has joined #openstack-meeting | 08:14 | |
| *** priteau has joined #openstack-meeting | 08:23 | |
| *** tpatil has quit IRC | 08:28 | |
| *** e0ne has joined #openstack-meeting | 08:32 | |
| *** janki has joined #openstack-meeting | 08:39 | |
| *** tssurya has joined #openstack-meeting | 08:40 | |
| *** tashiromt has quit IRC | 08:50 | |
| *** lennyb has quit IRC | 08:53 | |
| *** ayoung has quit IRC | 08:54 | |
| *** ykatabam has joined #openstack-meeting | 08:55 | |
| *** tesseract has joined #openstack-meeting | 08:56 | |
| *** ttsiouts has quit IRC | 09:01 | |
| *** ttsiouts has joined #openstack-meeting | 09:02 | |
| *** Liang__ has quit IRC | 09:03 | |
| *** Liang__ has joined #openstack-meeting | 09:05 | |
| *** ttsiouts has quit IRC | 09:06 | |
| *** rcernin has quit IRC | 09:07 | |
| *** imsurit has quit IRC | 09:08 | |
| *** _hemna has joined #openstack-meeting | 09:12 | |
| *** lennyb has joined #openstack-meeting | 09:12 | |
| *** panda is now known as panda|ruck | 09:38 | |
| *** tesseract has quit IRC | 09:40 | |
| *** bhagyashris has quit IRC | 09:41 | |
| *** _hemna has quit IRC | 09:46 | |
| *** mugsie_ is now known as mugsie | 09:52 | |
| *** ociuhandu has joined #openstack-meeting | 09:54 | |
| *** markvoelker has joined #openstack-meeting | 10:02 | |
| *** Liang__ has quit IRC | 10:03 | |
| *** ykatabam has quit IRC | 10:17 | |
| *** slaweq has quit IRC | 10:30 | |
| *** baojg has quit IRC | 10:33 | |
| *** markvoelker has quit IRC | 10:36 | |
| *** yamamoto has quit IRC | 10:36 | |
| *** yamamoto has joined #openstack-meeting | 10:37 | |
| *** yamamoto has quit IRC | 10:38 | |
| *** yoctozepto_ is now known as yoctozepto | 10:42 | |
| *** bbowen has quit IRC | 10:49 | |
| *** baojg has joined #openstack-meeting | 10:59 | |
| *** slaweq has joined #openstack-meeting | 11:01 | |
| *** beagles has quit IRC | 11:03 | |
| *** dkushwaha has quit IRC | 11:06 | |
| *** carloss has joined #openstack-meeting | 11:09 | |
| *** pwu has quit IRC | 11:11 | |
| *** yamamoto has joined #openstack-meeting | 11:21 | |
| *** ttsiouts has joined #openstack-meeting | 11:22 | |
| *** raildo has joined #openstack-meeting | 11:26 | |
| *** janki has quit IRC | 11:30 | |
| *** ttsiouts has quit IRC | 11:33 | |
| *** markvoelker has joined #openstack-meeting | 11:33 | |
| *** ttsiouts has joined #openstack-meeting | 11:34 | |
| *** ttsiouts_ has joined #openstack-meeting | 11:37 | |
| *** ttsiouts has quit IRC | 11:38 | |
| *** tetsuro has quit IRC | 11:39 | |
| *** _hemna has joined #openstack-meeting | 11:42 | |
| *** panda|ruck is now known as panda|ruck|eat | 11:45 | |
| *** panda|ruck|eat is now known as panda|ruck | 11:51 | |
| *** bbowen has joined #openstack-meeting | 11:52 | |
| *** yamahata has quit IRC | 11:56 | |
| *** yamamoto has quit IRC | 11:58 | |
| *** yamamoto has joined #openstack-meeting | 12:00 | |
| *** b3nt_pin has joined #openstack-meeting | 12:02 | |
| *** dmacpher has joined #openstack-meeting | 12:02 | |
| *** baojg has quit IRC | 12:02 | |
| *** baojg has joined #openstack-meeting | 12:04 | |
| *** markvoelker has quit IRC | 12:05 | |
| *** yamamoto has quit IRC | 12:13 | |
| *** _hemna has quit IRC | 12:17 | |
| *** mugsie is now known as mugsie_ | 12:18 | |
| *** mugsie_ is now known as mugsie | 12:18 | |
| *** njohnston has joined #openstack-meeting | 12:24 | |
| *** priteau has quit IRC | 12:34 | |
| *** panda|ruck is now known as panda|ruck|eat | 12:37 | |
| *** davidsha has joined #openstack-meeting | 12:48 | |
| *** yamamoto has joined #openstack-meeting | 12:51 | |
| *** panda|ruck|eat is now known as panda|ruck | 12:51 | |
| *** plutoinmii has joined #openstack-meeting | 12:53 | |
| *** priteau has joined #openstack-meeting | 12:58 | |
| *** yamamoto has quit IRC | 12:59 | |
| *** yamamoto has joined #openstack-meeting | 12:59 | |
| *** jamesmcarthur has joined #openstack-meeting | 13:00 | |
| *** Luzi has joined #openstack-meeting | 13:00 | |
| *** priteau has quit IRC | 13:04 | |
| *** baojg has quit IRC | 13:09 | |
| *** baojg has joined #openstack-meeting | 13:10 | |
| *** ttsiouts_ has quit IRC | 13:10 | |
| *** ttsiouts has joined #openstack-meeting | 13:11 | |
| *** rubasov has quit IRC | 13:12 | |
| *** Luzi has quit IRC | 13:13 | |
| *** baojg has quit IRC | 13:14 | |
| *** ttsiouts has quit IRC | 13:15 | |
| *** mriedem has joined #openstack-meeting | 13:17 | |
| *** diablo_rojo has joined #openstack-meeting | 13:18 | |
| *** b3nt_pin is now known as beagles | 13:19 | |
| *** jamesmcarthur has quit IRC | 13:20 | |
| *** jamesmcarthur has joined #openstack-meeting | 13:20 | |
| *** diablo_rojo has quit IRC | 13:22 | |
| *** Liang__ has joined #openstack-meeting | 13:23 | |
| *** diablo_rojo has joined #openstack-meeting | 13:23 | |
| *** jamesmcarthur has quit IRC | 13:25 | |
| *** ttsiouts has joined #openstack-meeting | 13:26 | |
| *** enriquetaso has joined #openstack-meeting | 13:28 | |
| *** lbragstad has joined #openstack-meeting | 13:32 | |
| *** rubasov has joined #openstack-meeting | 13:38 | |
| *** lpetrut has quit IRC | 13:43 | |
| *** boxiang has quit IRC | 13:49 | |
| *** boxiang has joined #openstack-meeting | 13:52 | |
| *** liuyulong has joined #openstack-meeting | 13:53 | |
| *** mlavalle has joined #openstack-meeting | 13:54 | |
| *** boxiang has quit IRC | 13:55 | |
| *** boxiang has joined #openstack-meeting | 13:55 | |
| *** lajoskatona has joined #openstack-meeting | 13:56 | |
| *** jrbalderrama has joined #openstack-meeting | 13:58 | |
| *** eharney has joined #openstack-meeting | 14:00 | |
| slaweq | #startmeeting networking | 14:01 |
| openstack | Meeting started Tue Jun 4 14:01:07 2019 UTC and is due to finish in 60 minutes. The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot. | 14:01 |
| slaweq | hi | 14:01 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 14:01 |
| *** openstack changes topic to " (Meeting topic: networking)" | 14:01 | |
| openstack | The meeting name has been set to 'networking' | 14:01 |
| lajoskatona | Hi | 14:01 |
| mlavalle | o/ | 14:01 |
| tidwellr | hi | 14:01 |
| *** lseki has joined #openstack-meeting | 14:01 | |
| rubasov | o/ | 14:02 |
| bcafarel | hello | 14:02 |
| slaweq | #topic Announcements | 14:02 |
| *** openstack changes topic to "Announcements (Meeting topic: networking)" | 14:02 | |
| njohnston | o/ | 14:02 |
| slaweq | Milestone Train-1 is this week | 14:02 |
| slaweq | #link https://releases.openstack.org/train/schedule.html | 14:02 |
| slaweq | mlavalle: do we need to release new versions this week then? | 14:03 |
| davidsha | o/ | 14:03 |
| mlavalle | yes, that's the agreement we did last ccyle | 14:03 |
| *** haleyb has joined #openstack-meeting | 14:04 | |
| mlavalle | release stable branches | 14:04 |
| *** munimeha1 has joined #openstack-meeting | 14:04 | |
| slaweq | that's what I thought | 14:04 |
| slaweq | so amotoki and mlavalle You will do this, right? | 14:04 |
| mlavalle | haleyb did that last time | 14:04 |
| * haleyb arrives after being in nickserv detention | 14:04 | |
| haleyb | i actually started stable patches, but think there are one or two more things to merge in each branch | 14:05 |
| mlavalle | haleyb: will you create the patches for stable branches this time around? | 14:05 |
| mlavalle | cool, thanks! | 14:05 |
| *** shintaro has joined #openstack-meeting | 14:05 | |
| haleyb | so yes, i can do it, i'll ping people about getting reviews done on stragglers | 14:05 |
| slaweq | haleyb: thx | 14:05 |
| mlavalle | :-) | 14:05 |
| slaweq | next one then | 14:06 |
| slaweq | Just a reminder: Open Infrastructure Summit CFP Open - Deadline: July 2 | 14:06 |
| slaweq | link: http://lists.openstack.org/pipermail/openstack-discuss/2019-May/006262.html | 14:06 |
| slaweq | so if You want to have propose some talk for next Summit, it's time to prepare proposal :) | 14:07 |
| slaweq | another reminder, is someone missed it on previous meeting: Photos from recent Denver PTG are available at: | 14:07 |
| slaweq | #link https://www.dropbox.com/sh/fydqjehy9h5y728/AAC1gIc5bJwwNd5JkcQ6Pqtra/Neutron?dl=0&subfolder_nav_tracking=1 | 14:07 |
| slaweq | and last reminder | 14:08 |
| slaweq | Train PTG summary: | 14:08 |
| slaweq | #link http://lists.openstack.org/pipermail/openstack-discuss/2019-May/006408.html | 14:08 |
| slaweq | any other announcements? | 14:08 |
| mlavalle | none from me | 14:09 |
| slaweq | ok, lets move on then | 14:09 |
| slaweq | #topic Blueprints | 14:10 |
| *** openstack changes topic to "Blueprints (Meeting topic: networking)" | 14:10 | |
| slaweq | This is what we have planned for T-1: | 14:10 |
| slaweq | #link https://launchpad.net/neutron/+milestone/train-1 | 14:10 |
| mlavalle | on this topic I have to say that last week we completed https://blueprints.launchpad.net/neutron/+spec/smart-nic-ovs | 14:11 |
| slaweq | I just wanted to ask if we need anything else for that one | 14:11 |
| slaweq | thx mlavalle | 14:11 |
| mlavalle | this is the patch that completed the feature: https://review.opendev.org/#/c/586252/ | 14:11 |
| mlavalle | it merged last week | 14:12 |
| mlavalle | on https://blueprints.launchpad.net/neutron/+spec/enginefacade-switch | 14:12 |
| *** _hemna has joined #openstack-meeting | 14:13 | |
| mlavalle | we are making good progress | 14:13 |
| *** shintaro has quit IRC | 14:13 | |
| mlavalle | we are merging patches | 14:13 |
| munimeha1 | can we add this one to train-1 also https://bugs.launchpad.net/neutron/+bug/1458890 | 14:13 |
| openstack | Launchpad bug 1458890 in neutron "[RFE] Add segment support to Neutron" [Wishlist,Triaged] - Assigned to Carl Baldwin (carl-baldwin) | 14:13 |
| mlavalle | and there are several more in the pipeline | 14:13 |
| munimeha1 | based on changes for routed networks to support multiple segments | 14:14 |
| *** mriedem is now known as mriedem_away | 14:14 | |
| munimeha1 | we can also levereage same to enhance sriov | 14:14 |
| mlavalle | thanks to ralonsoh and njohnston for their hard work on this | 14:14 |
| slaweq | mlavalle: yes, I saw couple of patches merged/close to merge related to this one | 14:15 |
| slaweq | and it looks that it's going pretty easy so far :) | 14:15 |
| slaweq | (too easy :P) | 14:15 |
| njohnston | we're plucking the low hanging fruit first :-) | 14:15 |
| mlavalle | well, I think we are starting to hit the difficult ones | 14:15 |
| mlavalle | here's an example for last night: https://review.opendev.org/#/c/662869/ | 14:16 |
| mlavalle | so don't let your guard down and stay ready for a long fight | 14:16 |
| mlavalle | ;-) | 14:16 |
| slaweq | :) | 14:16 |
| njohnston | amen brother | 14:16 |
| slaweq | thx You all for working on this | 14:17 |
| mlavalle | munimeha1: regarding multiple segments in routed networks, we already have that as a target: https://blueprints.launchpad.net/neutron/+spec/multiple-segment-per-network-per-host | 14:17 |
| slaweq | munimeha1: according to Your question, I think that we can only add Your rfe to T-2 as T-1 is this week :) | 14:17 |
| slaweq | mlavalle: what do You think? | 14:17 |
| munimeha1 | ok T-2 is fine | 14:18 |
| munimeha1 | thanks | 14:18 |
| mlavalle | munimeha1: but look at the blueprint that I just mentioned ^^^^ | 14:18 |
| munimeha1 | sure | 14:18 |
| mlavalle | that is what you are talking about, right? | 14:18 |
| munimeha1 | yes | 14:18 |
| mlavalle | so it is already targeted for T-1. it is going to slip to T-2 | 14:19 |
| mlavalle | bt you can help revieiwing the spec: https://review.opendev.org/#/c/657170 | 14:19 |
| munimeha1 | sure | 14:19 |
| mlavalle | which I ask others to do as well | 14:19 |
| slaweq | speaking about https://blueprints.launchpad.net/neutron/+spec/multiple-segment-per-network-per-host there is patch https://review.opendev.org/#/c/657170 waiting for review also | 14:19 |
| mlavalle | it is my goal for this week | 14:19 |
| mlavalle | slaweq: right. thanks for bringing it up | 14:19 |
| slaweq | You were faster mlavalle :) | 14:20 |
| munimeha1 | Did you notice this one is related https://review.opendev.org/#/c/656885/ | 14:20 |
| mlavalle | munimeha1: yes, that one is a debt that Nova has owed to us for a long time | 14:21 |
| lajoskatona | and don't forget the bug for routed networks: https://bugs.launchpad.net/neutron/+bug/1828543 | 14:21 |
| openstack | Launchpad bug 1828543 in neutron "Routed provider networks: placement API handling errors" [Medium,New] - Assigned to Lajos Katona (lajos-katona) | 14:21 |
| *** zbr has quit IRC | 14:21 | |
| mlavalle | lajoskatona: thanks. I owe you a response to an email. Haven't forgotten | 14:21 |
| * mlavalle was feeling bad this morning while walking the dog | 14:22 | |
| lajoskatona | mlavalle: thanks, now in keystoneauth there is a fix for exception handling | 14:22 |
| slaweq | so I will update this BP https://blueprints.launchpad.net/neutron/+spec/multiple-segment-per-network-per-host to include links to all those things there, ok for You? | 14:22 |
| *** zbr has joined #openstack-meeting | 14:22 | |
| lajoskatona | mlavalle: so I continue to see what is missing to make this working again and add tests | 14:22 |
| mlavalle | cool, thanks | 14:23 |
| lajoskatona | slaweq: ok for me, thanks | 14:23 |
| slaweq | ok, I will do it after the meeting | 14:23 |
| slaweq | lets move on then | 14:23 |
| slaweq | we have also BP https://blueprints.launchpad.net/neutron/+spec/event-notifier-ironic | 14:23 |
| slaweq | and we have already merged https://review.opendev.org/#/c/658787/ | 14:23 |
| slaweq | which looks like the only needed thing on our side | 14:23 |
| slaweq | do You know if we can consieder it as done maybe? | 14:24 |
| slaweq | or what else is missing on our side for it | 14:24 |
| mlavalle | hang on... | 14:24 |
| mlavalle | I think that's it | 14:25 |
| mlavalle | we finished that one | 14:25 |
| slaweq | great, so mlavalle can You update BP and set proper status for it? | 14:25 |
| mlavalle | yes I will | 14:25 |
| slaweq | thx a lot mlavalle | 14:26 |
| slaweq | ok, lets move on | 14:26 |
| mlavalle | and I will also go over the PTG summary this week | 14:26 |
| mlavalle | there are other BPs in there that we need to add to the dashboard | 14:26 |
| slaweq | thx mlavalle | 14:26 |
| slaweq | #topic Bugs | 14:27 |
| *** openstack changes topic to "Bugs (Meeting topic: networking)" | 14:27 | |
| slaweq | last week amotoki was our bug deputy | 14:27 |
| slaweq | he sent summary http://lists.openstack.org/pipermail/openstack-discuss/2019-June/006850.html | 14:27 |
| slaweq | I would especially want to ask db experts (looking at You njohnston :)) to look at https://bugs.launchpad.net/neutron/+bug/1831009 | 14:28 |
| openstack | Launchpad bug 1831009 in neutron "Improper close connection to database leading to mysql/mariadb block connection." [Undecided,New] | 14:28 |
| mlavalle | is this reported against master branch? | 14:29 |
| mlavalle | oh, I see he is reporting the neutron version | 14:29 |
| slaweq | no, it's Neutron-server: openstack-neutron-13.0.2-1.el7.noarch | 14:29 |
| mlavalle | that's rocky | 14:30 |
| slaweq | yes, I think so :) | 14:30 |
| slaweq | we also have 2 bugs related to api performance: https://bugs.launchpad.net/neutron/+bug/1830679 and https://bugs.launchpad.net/neutron/+bug/1830630 | 14:31 |
| openstack | Launchpad bug 1830679 in neutron "Security groups RBAC cause a major performance degradation" [High,New] | 14:31 |
| openstack | Launchpad bug 1830630 in neutron "Get external networks too slowly because it would join subnet and rbac" [Medium,New] | 14:31 |
| slaweq | both are not assigned so would be good if someone would have time to take a look | 14:32 |
| slaweq | any other bugs You want to talk about today? | 14:32 |
| *** boden has joined #openstack-meeting | 14:33 | |
| slaweq | ok, I get it as no :) | 14:34 |
| slaweq | this week our bug deputy is haleyb | 14:34 |
| * mlavalle will try to take a look at some of the bugs mentioned above ^^^^ | 14:34 | |
| slaweq | and next week it will be manjeets | 14:34 |
| haleyb | slaweq: thanks for the reminder | 14:34 |
| slaweq | mlavalle: will You contact manjeets or should I ask him if he will be able to do it? | 14:35 |
| slaweq | haleyb: yw :) | 14:35 |
| *** armax has joined #openstack-meeting | 14:35 | |
| mlavalle | slaweq: go ahead and ping him, please | 14:35 |
| slaweq | sure, I will do | 14:35 |
| slaweq | ok, next topic then | 14:35 |
| slaweq | #topic neutron-lib | 14:35 |
| *** openstack changes topic to "neutron-lib (Meeting topic: networking)" | 14:35 | |
| slaweq | boden: hi :) | 14:36 |
| boden | hi | 14:36 |
| boden | there's a neutron-lib release patch out for review https://review.opendev.org/#/c/661839/ as far as I'm concerned it's good to go | 14:36 |
| * mlavalle just +1ed it | 14:37 | |
| boden | would be nice to get that out as I think some folks are waiting for it | 14:37 |
| boden | ok thanks! | 14:37 |
| mlavalle | yeah prometheanfire was very vocal last week | 14:37 |
| boden | the other topic is in regards to reviews for non-neutron networking projects (sfc, l2gw, etc.) | 14:37 |
| boden | wondering if we can look to expand the +2 powers in some of these | 14:37 |
| boden | perhaps the neutron core team | 14:37 |
| slaweq | boden: that was added to "on demand" section :) | 14:38 |
| slaweq | but we can talk about it now | 14:38 |
| slaweq | thx for bringing this up | 14:38 |
| lajoskatona | boden, mlavalle, slaweq: that was discussed during ptg as well, and with mlavalle we planned to have a meeting with l2gw ptl | 14:39 |
| mlavalle | is this for neutron-lib consumption patches? | 14:39 |
| liuyulong | I have a question, we have decided to use neutron-lib master branch for neutron CI, or not? | 14:39 |
| boden | mlavalle yes and no... in general its not easy to land patches in some gates due to lack of reviews.. these could be neutron-lib related or not | 14:39 |
| slaweq | yes, I agree with boden here | 14:40 |
| mlavalle | do me a favor. Send me an email with the list of projects | 14:41 |
| mlavalle | and I'll make sure all the core team members have +2 rights in them | 14:41 |
| slaweq | sometimes we have some "trivial" changes e.g. related to jobs definitions or some "mechanical" patches in stadium projects and there is no many people who can +2 those patches | 14:41 |
| boden | mlavalle cool thanks | 14:42 |
| boden | liuyulong we never got anywhere on that discussion | 14:42 |
| slaweq | mlavalle: IMHO all stadium projects should be in this list | 14:42 |
| bcafarel | sounds good, recent example is the rehoming of tempest plugins | 14:42 |
| mlavalle | I will also start a personal routines to go over the pending patches in those projects | 14:42 |
| mlavalle | yeah, but it's more than stadium | 14:42 |
| mlavalle | lajoskatona: yes, I will ping ricardo today | 14:43 |
| lajoskatona | mlavalle: thanks, | 14:43 |
| lajoskatona | mlavalle: otherwise the idea to add +2 right for cores is good. | 14:44 |
| lajoskatona | I mean for stadium projects | 14:44 |
| boden | we can start with stadium I guess and go from there... if other projects crop up we can discuss expanding +2 rights I suppose | 14:44 |
| njohnston | I think you can just include the neutron-cores group in the list of cores for each of the stadium projects... IIRC that is how the stable cores work for stadium projects (at least fwaas) | 14:45 |
| amotoki | in my understanding, neutron-release team members have an appripriate right for stadium projects, but it is not true for third-party prijects. | 14:45 |
| mlavalle | sounds good boden | 14:45 |
| boden | nothing else from me... just business as usual | 14:45 |
| slaweq | ok, thx boden for updates | 14:45 |
| -amotoki- just got home and is scrolling back the logs | 14:46 | |
| slaweq | anything else regarding neutron-lib? | 14:46 |
| slaweq | or we can move on? | 14:46 |
| boden | nothing from me | 14:47 |
| slaweq | ok, lets move on then | 14:47 |
| slaweq | last topic | 14:47 |
| slaweq | #topic On demand agenda | 14:47 |
| *** openstack changes topic to "On demand agenda (Meeting topic: networking)" | 14:47 | |
| slaweq | we already discussed one of things from the list | 14:47 |
| slaweq | there is also one from njohnston | 14:47 |
| slaweq | "Cleaning old OVO compatibility code." | 14:47 |
| *** yamamoto has quit IRC | 14:48 | |
| slaweq | go on njohnston | 14:48 |
| njohnston | oh, I think we talked about that last week; my apologies for not cleaning it off the agenda | 14:48 |
| slaweq | njohnston: no problem :) | 14:48 |
| slaweq | so I will clean it today to not forget next time | 14:48 |
| mlavalle | thanks | 14:48 |
| slaweq | anything else You want to talk about today? | 14:48 |
| slaweq | if not, I will give You couple of minutes back :) | 14:49 |
| mlavalle | I don't have anything else | 14:49 |
| liuyulong | Hold on | 14:49 |
| amotoki | I have nothing special. I still thinks njohnston's topic is worth checked by everyone https://review.opendev.org/#/c/661995/ | 14:49 |
| liuyulong | One minute about that neutron-lib dev | 14:50 |
| tidwellr | njohnston: I owe you an update on https://review.opendev.org/#/c/661995/ | 14:50 |
| njohnston | thanks amotoki and tidwellr for great feedback on that | 14:50 |
| njohnston | the change is to define a policy as regards OVO object compatibility code in such a way that we respect the upgrade windows for distributions; please take a look and let us know what you thing | 14:51 |
| njohnston | *think | 14:51 |
| tidwellr | njohnston: I can add this to the review, but as long as we suport N->N+2 or greater I think that will be acceptable | 14:51 |
| amotoki | njohnston: you're welcome. thanks for leading this coordination :) | 14:52 |
| liuyulong | Everytime if neutron touch a constant or a utils function, we will try to move it to neutron-lib, and then neutron dev was blocked by such procedure. | 14:52 |
| liuyulong | So it neutron CI does not run on master neutron-lib, this will happen again. | 14:52 |
| slaweq | liuyulong: we aren't blocked but a bit slowered down I would say :) | 14:52 |
| amotoki | liuyulong: it is true if it affects other stadium projects other than neturon. is my understanding right? | 14:53 |
| *** yamamoto has joined #openstack-meeting | 14:53 | |
| *** yamamoto has quit IRC | 14:53 | |
| liuyulong | amotoki, not exactly, event a constant is only used by neutron itself. | 14:53 |
| boden | at one point we were discussing something like that topic here https://review.opendev.org/#/c/602748/. but no one really had any interest | 14:54 |
| *** yamamoto has joined #openstack-meeting | 14:54 | |
| boden | there are pros and cons to using neutron-lib master vs. released... if someone wants to kick the tires on this topic again go ahead, but I probably don't have the bandwidth | 14:55 |
| mlavalle | liuyulong: if a constant or function is used only by neutron, it doesn't have to be sent to the lib | 14:55 |
| liuyulong | So, I'd like to say, if we should ensure neutron happy first, then do that move. | 14:55 |
| *** yamamoto has quit IRC | 14:55 | |
| amotoki | liuyulong: if the neutron repo is an only consumer of some constant, there is no need for neutron-lib in pricinpal. | 14:55 |
| *** yamamoto has joined #openstack-meeting | 14:55 | |
| boden | ^ true | 14:55 |
| mlavalle | amotoki is right | 14:55 |
| *** yamamoto has quit IRC | 14:55 | |
| mlavalle | in fact, if that util function or constant is only used by neutron, it shouldn't go to the lib | 14:56 |
| liuyulong | Yes, agress | 14:56 |
| boden | yep, that was the idea | 14:56 |
| liuyulong | agree | 14:56 |
| *** lpetrut has joined #openstack-meeting | 14:57 | |
| amotoki | API definitions are exceptions as we would like to define the neutron API as a whole of the neutron stadium. All others are to share something in stadium projects. | 14:57 |
| *** jrbalderrama has quit IRC | 14:57 | |
| *** Liang__ has quit IRC | 14:58 | |
| liuyulong | However, if a function is only used by neutron alone temporarily, IMO, it should implement in neutron first. | 14:58 |
| *** shilpasd has quit IRC | 14:58 | |
| amotoki | liuyulong: you'ree right | 14:59 |
| slaweq | ok, we are running out of time here | 14:59 |
| slaweq | thx for attending | 14:59 |
| njohnston | o/ | 14:59 |
| bcafarel | o/ | 14:59 |
| davidsha | o/ | 14:59 |
| slaweq | and have a great week! | 14:59 |
| slaweq | o/ | 14:59 |
| slaweq | #endmeeting | 14:59 |
| tidwellr | bye | 14:59 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 14:59 | |
| openstack | Meeting ended Tue Jun 4 14:59:38 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 14:59 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/networking/2019/networking.2019-06-04-14.01.html | 14:59 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/networking/2019/networking.2019-06-04-14.01.txt | 14:59 |
| mlavalle | o/ | 14:59 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/networking/2019/networking.2019-06-04-14.01.log.html | 14:59 |
| lajoskatona | bye | 14:59 |
| *** boden has left #openstack-meeting | 14:59 | |
| amotoki | o/ | 14:59 |
| *** cheng1__ has joined #openstack-meeting | 15:00 | |
| *** mlavalle has left #openstack-meeting | 15:01 | |
| *** ttsiouts has quit IRC | 15:04 | |
| *** lajoskatona has left #openstack-meeting | 15:04 | |
| *** ttsiouts has joined #openstack-meeting | 15:05 | |
| *** ttsiouts has quit IRC | 15:09 | |
| *** jamesmcarthur has joined #openstack-meeting | 15:11 | |
| *** yamamoto has joined #openstack-meeting | 15:15 | |
| *** yamamoto has quit IRC | 15:15 | |
| *** yamamoto has joined #openstack-meeting | 15:15 | |
| *** _hemna has quit IRC | 15:18 | |
| *** yamamoto has quit IRC | 15:20 | |
| *** lpetrut has quit IRC | 15:29 | |
| *** e0ne has quit IRC | 15:32 | |
| *** armax has quit IRC | 15:34 | |
| *** gyee has joined #openstack-meeting | 15:35 | |
| *** yamahata has joined #openstack-meeting | 15:39 | |
| *** bnemec has quit IRC | 15:44 | |
| *** bnemec has joined #openstack-meeting | 15:46 | |
| *** iyamahat has joined #openstack-meeting | 15:46 | |
| *** cheng1__ has quit IRC | 15:55 | |
| slaweq | #startmeeting neutron_ci | 16:01 |
| openstack | Meeting started Tue Jun 4 16:01:05 2019 UTC and is due to finish in 60 minutes. The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot. | 16:01 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 16:01 |
| slaweq | hi | 16:01 |
| *** openstack changes topic to " (Meeting topic: neutron_ci)" | 16:01 | |
| openstack | The meeting name has been set to 'neutron_ci' | 16:01 |
| njohnston | o/ | 16:01 |
| bcafarel | hi again | 16:01 |
| slaweq | I know that mlavalle will not be able to join this meeting today | 16:02 |
| slaweq | so I think we can start now | 16:02 |
| slaweq | Grafana dashboard: http://grafana.openstack.org/dashboard/db/neutron-failure-rate | 16:02 |
| *** davidsha has quit IRC | 16:02 | |
| slaweq | #topic Actions from previous meetings | 16:02 |
| *** openstack changes topic to "Actions from previous meetings (Meeting topic: neutron_ci)" | 16:02 | |
| slaweq | slaweq to continue work on fetch-journal-log zuul role | 16:03 |
| slaweq | I did patch https://review.opendev.org/#/c/643733/ and in https://review.opendev.org/#/c/661915/ it's tested that it works | 16:03 |
| slaweq | but I'm not sure if this will be needed in fact | 16:03 |
| slaweq | I noticed recently that full journal log is already dumped in devstack.journal.xz.gz file in job's logs | 16:03 |
| slaweq | but it's in binary journal format so You need to download it and use journalctl to examine it | 16:04 |
| njohnston | good to know! | 16:04 |
| slaweq | yeah, I had no idea about that before | 16:04 |
| bcafarel | maybe we could use a role to translate that file in job definition? | 16:05 |
| slaweq | bcafarel: You mean to text file? | 16:05 |
| bcafarel | yes, to skip the download step | 16:06 |
| bcafarel | for lazy folks like me :) | 16:06 |
| slaweq | it's basically what role proposed by me in https://review.opendev.org/#/c/643733/ is doing | 16:06 |
| slaweq | we can try to convice infra-root people that it may be useful :) | 16:07 |
| clarkb | slaweq: I'm still of the opinion that we should capture the serialized version in its entirety and compress it well as it is a very large file | 16:08 |
| fungi | it ends up taking up a lot of additional disk space in some jobs to keep two copies of the log, and the native version is useful for more flexible local filtering | 16:08 |
| clarkb | slaweq: it gives you way more flexibility that way, with a small amount of setup | 16:08 |
| clarkb | if we know we need specific service log files we can pull those out along with the openstack service logs | 16:09 |
| clarkb | but for the global capture I think what we've got now with the export format works really well | 16:09 |
| slaweq | clarkb: yes, but e.g. for neutron it is many different services, like keepalived, haproxy, dnsmasq which are spawned per router or network | 16:09 |
| clarkb | yup which is why the current setup is great. You can do journalctl -u keepalived -u haproxy -u dnsmasq -u devstack@q-agt and get only those logs interleaved | 16:10 |
| clarkb | you can also set date ranges to narrow down what you are looking at | 16:11 |
| clarkb | you cannot easily do that with the chnage you have proposed | 16:11 |
| slaweq | clarkb: so as I said, I can just abandon this my patch as now I know that there is this log available and how to get it | 16:11 |
| slaweq | I think that this will be the best approach, what do You think? | 16:11 |
| clarkb | oh I was going to suggest you update the role so that you don't have to use devstack to get that functionality | 16:11 |
| clarkb | basically do what devstack does but in a reconsumable role in zuul-jobs | 16:12 |
| clarkb | then we can apply it to non devstack jobs | 16:12 |
| slaweq | clarkb: tbh I need it to devstack based jobs so it's enough for us what is now :) | 16:12 |
| njohnston | I bet we could easily create a grabjournal script that could fetch the journal for a specific job of a specific change and run that journalctl command on it, if our issue is just that we want to make it more accessible for developers | 16:12 |
| clarkb | slaweq: ok | 16:13 |
| slaweq | njohnston: initially I wasn't aware of that this log exists already in job's logs, so I proposed patch | 16:13 |
| slaweq | but now IMO the only "issue" is this accessibility of log and I think that this is not something which we should spent a lot of time on :) | 16:14 |
| njohnston | +1 sounds good | 16:14 |
| bcafarel | yes, I can survive a download+parse :) | 16:15 |
| slaweq | so my proposal is: lets for now use what is already there | 16:15 |
| *** Shrews has joined #openstack-meeting | 16:15 | |
| slaweq | and we will see if this will have to be improved somehow :) | 16:15 |
| slaweq | ok, so lets move on to the next action | 16:16 |
| slaweq | slaweq to remove neutron-tempest-plugin-bgpvpn-bagpipe from "neutron-tempest-plugin-jobs" template | 16:16 |
| slaweq | Done: https://review.opendev.org/#/c/661899/ | 16:16 |
| *** Shrews has left #openstack-meeting | 16:16 | |
| slaweq | and the last one was; | 16:16 |
| slaweq | mlavalle to debug neutron-tempest-plugin-dvr-multinode-scenario failures (bug 1830763) | 16:16 |
| openstack | bug 1830763 in neutron "Debug neutron-tempest-plugin-dvr-multinode-scenario failures" [High,Confirmed] https://launchpad.net/bugs/1830763 - Assigned to Miguel Lavalle (minsel) | 16:16 |
| slaweq | but as mlavalle is not here now, I think we can assign it to him for next week to not forget about it | 16:17 |
| slaweq | #action mlavalle to debug neutron-tempest-plugin-dvr-multinode-scenario failures (bug 1830763) | 16:17 |
| slaweq | do You have anything else to add regarding actions from last week? | 16:17 |
| bcafarel | all good here | 16:18 |
| slaweq | ok, lets move on | 16:18 |
| slaweq | #topic Stadium projects | 16:18 |
| *** openstack changes topic to "Stadium projects (Meeting topic: neutron_ci)" | 16:18 | |
| slaweq | Python 3 migration | 16:18 |
| slaweq | Stadium projects etherpad: https://etherpad.openstack.org/p/neutron_stadium_python3_status | 16:18 |
| slaweq | any updates on it? | 16:18 |
| bcafarel | not from me sorry no progress here | 16:19 |
| njohnston | me neither | 16:19 |
| slaweq | same from my side | 16:19 |
| slaweq | so next thing | 16:19 |
| slaweq | tempest-plugins migration | 16:19 |
| *** wwriverrat has quit IRC | 16:19 | |
| slaweq | Etherpad: https://etherpad.openstack.org/p/neutron_stadium_move_to_tempest_plugin_repo | 16:19 |
| slaweq | for networking-bgpvpn both main patches are merged | 16:20 |
| *** wwriverrat has joined #openstack-meeting | 16:20 | |
| slaweq | but there was need to do some follow up cleanup which I forgot to do | 16:20 |
| slaweq | so there is also https://review.opendev.org/#/c/662231/ waiting for review | 16:20 |
| njohnston | oh, what needed to be cleaned up? | 16:20 |
| *** mriedem_away is now known as mriedem | 16:20 | |
| njohnston | ah I see | 16:20 |
| slaweq | and there was https://review.opendev.org/#/c/662142/ from masayukig but this one is already merged | 16:20 |
| slaweq | so please also remember about that in Your patches :) | 16:21 |
| njohnston | +1 | 16:21 |
| slaweq | and that's all from my side about this | 16:21 |
| slaweq | any updates on Your side? | 16:21 |
| bcafarel | for sfc https://review.opendev.org/#/c/653747 is close to merge (second patch), pending on some gate fixes | 16:21 |
| bcafarel | so mostly gerrit work left to do :) | 16:22 |
| slaweq | bcafarel: great :) | 16:22 |
| njohnston | I've been letting fwaas sit, but I will probably be able to dedicate some time to it about a week from now | 16:22 |
| slaweq | njohnston: great, if You would need any help, ping me :) | 16:23 |
| slaweq | ok, so lets move on to the next topic then | 16:24 |
| slaweq | #topic Grafana | 16:24 |
| *** openstack changes topic to "Grafana (Meeting topic: neutron_ci)" | 16:24 | |
| slaweq | #link http://grafana.openstack.org/dashboard/db/neutron-failure-rate | 16:24 |
| slaweq | still IMO things looks relatively good | 16:25 |
| slaweq | the biggest problem which we have are some failures in functional/fullstack tests | 16:25 |
| njohnston | yep | 16:25 |
| slaweq | and those ssh failure in tempest jobs | 16:25 |
| slaweq | but I think it happens less often recently :) | 16:25 |
| njohnston | I did have one general grafana question; in the "Number of Functional/Fullstack job runs (Gate queue)" graph I see that the lines diverge. Does that mean we lost data somewhere? I can't imagine a scenario where neutron-functional runs but not neutron-functional-python27 | 16:26 |
| njohnston | that's just one example; it happens from time to time elsewhere as well | 16:27 |
| *** lpetrut has joined #openstack-meeting | 16:27 | |
| slaweq | good question njohnston but I don't know an answer | 16:29 |
| haleyb | that is interesting, if it's only +/- 1 i could just see it being one had finished at that point in time but the other hadn't | 16:29 |
| slaweq | IMHO it may be lack of some data collected by infra - we have some gaps from time to time on graphs so maybe it's also something like that | 16:29 |
| njohnston | that's why I picked that one: neutron-functional = 10; neutron-functional-python27 = 7; neutron-fullstack = 6 | 16:29 |
| njohnston | yeah, it's just weird, something to be aware of | 16:30 |
| slaweq | I agree, thx njohnston for pointing this | 16:31 |
| slaweq | anything else regarding grafana in general? | 16:32 |
| haleyb | speaking of failure rates, i did update the neutron-lib dashboard last week, not that we talk about it much here, just an FYI | 16:32 |
| slaweq | haleyb: thx | 16:32 |
| haleyb | it looks similar to this one and ovn now | 16:32 |
| slaweq | I should take a look at it from time to time too | 16:32 |
| *** mattw4 has joined #openstack-meeting | 16:33 | |
| haleyb | http://grafana.openstack.org/d/Um5INcSmz/neutron-lib-failure-rate?orgId=1 | 16:33 |
| slaweq | not too much data there yet :) | 16:33 |
| haleyb | nope, not many failures or patches | 16:33 |
| slaweq | some periodic jobs are failing constantly | 16:35 |
| slaweq | which is maybe worth to check | 16:35 |
| haleyb | i think they are known failures, but will look! | 16:36 |
| slaweq | haleyb: thx a lot | 16:36 |
| slaweq | ok, lets move on then | 16:36 |
| slaweq | #topic fullstack/functional | 16:36 |
| *** openstack changes topic to "fullstack/functional (Meeting topic: neutron_ci)" | 16:36 | |
| slaweq | I was looking at results of some failed jobs from last couple of days | 16:37 |
| slaweq | and I found 2 new failed tests in functional job | 16:37 |
| slaweq | http://logs.openstack.org/78/653378/7/check/neutron-functional/c5ac6a3/testr_results.html.gz and | 16:37 |
| slaweq | http://logs.openstack.org/82/659882/3/check/neutron-functional-python27/5e30908/testr_results.html.gz | 16:37 |
| *** mattw4 has quit IRC | 16:37 | |
| slaweq | but each of those I saw only once | 16:38 |
| slaweq | did You maybe see something like that before? | 16:38 |
| haleyb | i haven't seen it, but failed running sysctl in the first? | 16:39 |
| slaweq | here are logs from this first failed test: http://logs.openstack.org/78/653378/7/check/neutron-functional/c5ac6a3/controller/logs/dsvm-functional-logs/neutron.tests.functional.agent.l3.test_dvr_router.TestDvrRouter.test_dvr_ha_router_unbound_from_agents.txt.gz#_2019-06-02_14_15_09_388 | 16:39 |
| haleyb | oh, rtnetlink error, that could be a bug? | 16:40 |
| haleyb | i.e. delete_gateway should deal with it | 16:41 |
| slaweq | yes, it's possible | 16:42 |
| slaweq | I will try to look deeper into this log this week | 16:42 |
| slaweq | maybe I will find something obvious to change :) | 16:42 |
| njohnston | for the second one nothing jumps out at me but this one error: http://logs.openstack.org/82/659882/3/check/neutron-functional-python27/5e30908/controller/logs/dsvm-functional-logs/neutron.tests.functional.agent.test_l2_ovs_agent.TestOVSAgent.test_assert_br_phys_patch_port_ofports_dont_change.txt.gz#_2019-05-30_08_52_08_873 | 16:42 |
| slaweq | #action slaweq to check logs with failed test_dvr_ha_router_unbound_from_agents functional test | 16:42 |
| haleyb | slaweq: i know rodolfo had a patchset up to change this area to use privsep, so might want to start there | 16:43 |
| slaweq | haleyb: good to know, thx | 16:43 |
| slaweq | I will ask him when he will back | 16:43 |
| haleyb | https://review.opendev.org/#/c/661981/ | 16:44 |
| haleyb | errors will change after that, into some pyroute2 one perhaps? | 16:44 |
| slaweq | ok, I will keep it in mind, thx :) | 16:45 |
| slaweq | njohnston: for the second one, this error can be maybe related | 16:46 |
| slaweq | or maybe not :) | 16:47 |
| njohnston | :) | 16:47 |
| slaweq | njohnston: this test is stoping agent: https://github.com/openstack/neutron/blob/86139658efdc739c6cc330304bdf4455613df78d/neutron/tests/functional/agent/test_l2_ovs_agent.py#L261 | 16:48 |
| slaweq | and IMO this "error" in log is related to this agent's stop | 16:48 |
| slaweq | so for me it don't look like possible problem at first glance | 16:48 |
| njohnston | I agree | 16:49 |
| slaweq | so lets keep this issue in mind, if it will happen more often we will investigate it :) | 16:50 |
| slaweq | do You agree? | 16:50 |
| slaweq | ok, I guess it means yes :) | 16:51 |
| slaweq | so that is all what I had for today | 16:51 |
| slaweq | do You have anything else You want to talk about today? | 16:52 |
| haleyb | -1 from me | 16:52 |
| bcafarel | :) | 16:52 |
| bcafarel | nothing from me either | 16:52 |
| haleyb | i'm hungry | 16:52 |
| slaweq | haleyb: I'm tired | 16:52 |
| slaweq | so let's finish it bit earlier today | 16:53 |
| slaweq | thx for attending :) | 16:53 |
| slaweq | o/ | 16:53 |
| bcafarel | o/ | 16:53 |
| slaweq | #endmeeting | 16:53 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 16:53 | |
| openstack | Meeting ended Tue Jun 4 16:53:18 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:53 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/neutron_ci/2019/neutron_ci.2019-06-04-16.01.html | 16:53 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/neutron_ci/2019/neutron_ci.2019-06-04-16.01.txt | 16:53 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/neutron_ci/2019/neutron_ci.2019-06-04-16.01.log.html | 16:53 |
| slaweq | bonne appetite haleyb :) | 16:53 |
| *** lpetrut has quit IRC | 16:55 | |
| *** mattw4 has joined #openstack-meeting | 16:58 | |
| *** _hemna has joined #openstack-meeting | 17:14 | |
| *** kopecmartin is now known as kopecmartin|off | 17:18 | |
| *** whoami-rajat has quit IRC | 17:24 | |
| *** yamahata has quit IRC | 17:27 | |
| *** iyamahat has quit IRC | 17:27 | |
| *** ociuhandu has quit IRC | 17:28 | |
| *** ekcs has joined #openstack-meeting | 17:35 | |
| *** tssurya has quit IRC | 17:38 | |
| *** ricolin has joined #openstack-meeting | 17:38 | |
| *** iyamahat has joined #openstack-meeting | 17:38 | |
| *** gyee has quit IRC | 17:41 | |
| *** iyamahat_ has joined #openstack-meeting | 17:45 | |
| *** jamesmcarthur has quit IRC | 17:46 | |
| *** yamamoto has joined #openstack-meeting | 17:47 | |
| *** _hemna has quit IRC | 17:48 | |
| *** iyamahat has quit IRC | 17:49 | |
| *** yamamoto has quit IRC | 17:52 | |
| *** yamahata has joined #openstack-meeting | 17:58 | |
| *** Shrews has joined #openstack-meeting | 18:01 | |
| *** brault has joined #openstack-meeting | 18:05 | |
| *** brault has quit IRC | 18:09 | |
| *** diablo_rojo has quit IRC | 18:12 | |
| *** whoami-rajat has joined #openstack-meeting | 18:25 | |
| *** diablo_rojo has joined #openstack-meeting | 18:31 | |
| *** ociuhandu has joined #openstack-meeting | 18:34 | |
| *** ociuhandu has quit IRC | 18:43 | |
| *** e0ne has joined #openstack-meeting | 18:54 | |
| *** e0ne has quit IRC | 18:55 | |
| clarkb | anyone else here for the infra meeting | 18:59 |
| clarkb | we'll get started in a minute or two | 18:59 |
| corvus | mmmmrph | 18:59 |
| clarkb | must be a good sanwich | 18:59 |
| fungi | sounds like it | 19:00 |
| ianw | o/ | 19:01 |
| clarkb | #startmeeting infra | 19:01 |
| openstack | Meeting started Tue Jun 4 19:01:16 2019 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. | 19:01 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 19:01 |
| *** openstack changes topic to " (Meeting topic: infra)" | 19:01 | |
| openstack | The meeting name has been set to 'infra' | 19:01 |
| clarkb | #link http://lists.openstack.org/pipermail/openstack-infra/2019-June/006396.html | 19:01 |
| clarkb | #topic Announcements | 19:01 |
| *** openstack changes topic to "Announcements (Meeting topic: infra)" | 19:01 | |
| clarkb | These didn't make it to the wiki agenda but I should mention them anyway. I'll be digging up rocks with family June 17-20 in the desert. Don't expect I'll have much of an internet connection | 19:02 |
| clarkb | as a result of that I plan to do ssl cert refreshes late next week | 19:02 |
| clarkb | (they expire on like the 20th or 21st) | 19:02 |
| fungi | i hear that desert has too many rocks anyway, they won't miss a few | 19:02 |
| Shrews | i'm sure it won't be hot at all | 19:03 |
| fungi | he's going before summer! | 19:03 |
| fungi | (a day before summer anyway) | 19:03 |
| clarkb | high of 83F today. Only 59F friday | 19:03 |
| clarkb | so could be scorching or cool | 19:03 |
| clarkb | #topic Actions from last meeting | 19:04 |
| *** openstack changes topic to "Actions from last meeting (Meeting topic: infra)" | 19:04 | |
| clarkb | #link http://eavesdrop.openstack.org/meetings/infra/2019/infra.2019-05-28-19.01.txt minutes from last meeting | 19:04 |
| clarkb | ianw volunteered to look into what is necessary to ansible our backup server (instead of updating puppet 3 to 4 on it) | 19:04 |
| clarkb | ianw: ^ anything to report on that yet? | 19:04 |
| ianw | i've started on https://review.opendev.org/#/c/662657 although it's in progress, but i think it will work | 19:05 |
| ianw | i think it should allow us to stop with all the manual user adds etc too (something i've had on my todo list for a long time!) ... so next week maybe something to review | 19:05 |
| clarkb | #link https://review.opendev.org/#/c/662657 backup server and backup ansible stuff | 19:05 |
| clarkb | thank you for kicking that off | 19:06 |
| ianw | also slightly related is https://review.opendev.org/#/c/662662 if people could look to use ansible 2.8 on bridge ... it has some features helpful for deploying ssh keys | 19:06 |
| clarkb | that will tie into the gitea mysql backups work I plan to start today | 19:06 |
| clarkb | #topic Specs approval | 19:07 |
| *** openstack changes topic to "Specs approval (Meeting topic: infra)" | 19:07 | |
| clarkb | No specs up for approval but a call out for reviews. https://review.opendev.org/#/q/project:opendev/infra-specs+status:open is worth a look if you haev a moment (its on my todo list somewhere after followups on gitea things) | 19:07 |
| clarkb | corvus: ^ you may be interested in the irc spec (involves email and irc) | 19:08 |
| corvus | ack, thx | 19:08 |
| clarkb | #topic Priority Efforts | 19:08 |
| *** openstack changes topic to "Priority Efforts (Meeting topic: infra)" | 19:08 | |
| clarkb | #topic Update Config Management | 19:09 |
| *** openstack changes topic to "Update Config Management (Meeting topic: infra)" | 19:09 | |
| clarkb | Any news on this topic other than ianw's ansible backup work and my plan to do mysql backups on gitea servers? | 19:09 |
| clarkb | Seems we're starting to transition from concrete targets to organic changes on this item | 19:09 |
| corvus | i think we're also waiting on the gitea-project-creation-in-python work | 19:09 |
| fungi | none i'm aware of | 19:09 |
| clarkb | corvus: oh right | 19:10 |
| corvus | that's a blocker for finishing up the independent zuul playbook | 19:10 |
| clarkb | right so the concrete target we have is the CD'ing end goal | 19:10 |
| corvus | mordred: now that you're back, do you think you might be able to get to that soon? if not, i should be able to do it | 19:10 |
| * mordred waves | 19:10 | |
| *** ricolin has quit IRC | 19:10 | |
| clarkb | which deps on a few things like independent zuul playbook and gitea project management in python | 19:10 |
| mordred | I can work on that soon | 19:10 |
| corvus | cool, thx | 19:11 |
| mordred | I believe I have no outstanding higher-priority tasks to clear first | 19:11 |
| corvus | great, then once that's in place, we can run the zuul service playbook in test, and then in cd | 19:12 |
| mordred | luckily the logic is in ansible already, so that's awesome | 19:12 |
| mordred | ++ | 19:12 |
| clarkb | we may want to consider this item as "done" (enough) and drop it from priority topics once we 've got CDing happening. Then we can transition to more organic updates like the mirror update and backup server update. Then decide if we want to have a concerted effort for replacing puppet | 19:13 |
| clarkb | but should probably consider that once we get to the CDing state | 19:14 |
| mordred | ++ | 19:14 |
| clarkb | alright lets move on to the next topic | 19:15 |
| clarkb | #topic OpenDev | 19:15 |
| *** openstack changes topic to "OpenDev (Meeting topic: infra)" | 19:15 | |
| fungi | that's a mighty fine topic you have there | 19:15 |
| clarkb | Last friday we did the project renaming thing. We learned we had a few more gaps in the renaming playbooks but I think we fixed them all (gerrit start timeout, lack of var loading in play, missing grab of existing orgs to check if we need to create new orgs) | 19:16 |
| clarkb | we've also got groups other than ara using zuul based git replication to github | 19:16 |
| clarkb | and openstack is talking about how to manage their github orgs now too | 19:17 |
| clarkb | all that to say I think we are in really good shape for managing new projects and renames (with the caveat that we still need gerrit downtime) | 19:17 |
| clarkb | thank you to everyone who helped with that | 19:17 |
| corvus | we did miss one rename we intended to do | 19:18 |
| corvus | i forget which | 19:18 |
| clarkb | yup, it wasn't in the gerrit projects.yaml so didn't make it into my repos.yaml input for the playbook | 19:18 |
| clarkb | it was ansible-role-cloud-launcher | 19:18 |
| clarkb | ajaeger has pushed changes up to get it on the list for next time | 19:18 |
| clarkb | I believe the spec for the migration is still open, but at this point I'm comfortable marking it complete. Any objections? | 19:20 |
| clarkb | opendev-gerrit specifically | 19:21 |
| corvus | i think we've done all the worke items | 19:21 |
| corvus | so yeah, i think it's done :) | 19:21 |
| corvus | #link opendev-gerrit spec https://docs.opendev.org/opendev/infra-specs/latest/specs/opendev-gerrit.html | 19:21 |
| fungi | we also had a last-minute rename request... did we end up folding it into the batch? i don't recall now | 19:21 |
| clarkb | fungi: we did | 19:22 |
| clarkb | we did not fold mnasers last minute request in due to people awnting more discussion on whether or not those belong in openstack/ | 19:22 |
| fungi | okay, so it's just x/ansible-role-cloud-launcher which we're currently missing | 19:22 |
| clarkb | yup | 19:22 |
| fungi | oh, right, i was confusing two last-minute requests | 19:22 |
| fungi | seems likely the ones mnaser brought up may move into the openstack org, so would still be renamed | 19:23 |
| clarkb | ya but to a different target | 19:23 |
| fungi | right | 19:23 |
| mnaser | yes, sorry, I will -w that for now | 19:23 |
| fungi | but i'm expecting them to be in the list for the next maintenance, whenever that winds up getting scheduled | 19:23 |
| fungi | as the logistics will likely be worked out by then | 19:23 |
| clarkb | ++ | 19:24 |
| clarkb | Anything else opendev related worth bringin up? | 19:24 |
| clarkb | corvus: ^ did you want to talk about tarballs.opendev.org/ | 19:24 |
| corvus | oh, yeah... | 19:25 |
| corvus | zuul needs a place to put branch-tip tarballs before it can move out of the openstack zuul tenant | 19:25 |
| corvus | (presumably, other opendev projects will want the same) | 19:26 |
| corvus | some opendev admins and zuul maint folks had a quick chat in #zuul about it, and we figured making tarballs.opendev.org, similar to tarballs.openstack.org, is the best way forward for now | 19:26 |
| corvus | i don't think that's controversial, so i plan on just doing it | 19:27 |
| corvus | changes are in progress and nearly complete | 19:27 |
| clarkb | there are two big differences to the old oepnstack tarballs ost which is it will be using afs instead of "local" disk and projects are expected to be fully namespaced (not just using shortnames) | 19:27 |
| corvus | main differences between this and existing tarballs site are: 1) new site will be in afs; 2) it will be scoped by full project name rather than the short name | 19:27 |
| clarkb | yay I didn't say inaccurate things :) | 19:28 |
| corvus | #link opendev tarball publish jobs https://review.opendev.org/662877 | 19:28 |
| corvus | that's the current tip of development (follow depends-on for the rabbit hole | 19:28 |
| corvus | i'm currently testing its dependencies | 19:28 |
| clarkb | thanks for the update on that. | 19:28 |
| corvus | np. eol. | 19:29 |
| clarkb | any other opendev items before we move on to storyboard? | 19:29 |
| *** e0ne has joined #openstack-meeting | 19:29 | |
| fungi | also we have the option to migrate tarballs.openstack.org into that afs volume if we update the publishing jobs and adjust teh vhost config for it, i think? | 19:30 |
| fungi | (well, amd move the vhost to somewhere like files.o.o or add an afs client on static.o.o) | 19:30 |
| clarkb | fungi: should be possible. basically have new runs of jobs publish to afs then rsync the old stuff over | 19:30 |
| corvus | yeah. we should schedule that since we're changing publication locations | 19:30 |
| *** e0ne has quit IRC | 19:30 | |
| fungi | yeah, would like an eventual result where we're not indefinitely maintaining two different solutions for it | 19:30 |
| clarkb | ++ | 19:31 |
| corvus | but it's branch-tip tarballs -- i don't think it's going to be a big deal. | 19:31 |
| corvus | well, it's also our release tarballs | 19:31 |
| ianw | one thought popped up ... we've had issues before with parallel vos release's ... i haven't looked but is good locking around this explicitly handled? | 19:31 |
| corvus | ianw: i plan on adding it to the periodic vos release cron | 19:31 |
| corvus | so publication to the new site will be delayed up to 5m. | 19:31 |
| fungi | i think that's fine if we set that expectation | 19:32 |
| corvus | (and that's only one actor) | 19:32 |
| clarkb | that cron runs on afsdb01 iirc | 19:32 |
| clarkb | so ya only one actor for that | 19:32 |
| fungi | if jobs need a tighter timeline to reuse those artifacts, then they can use paused jobs or something | 19:32 |
| fungi | like the image jobs do it with their own ephemeral repository | 19:33 |
| clarkb | fungi: can also read from the RW afs mount | 19:33 |
| corvus | could the pypi publication jobs be affected? | 19:33 |
| fungi | i suppose, but probably better we just warn against reconsuming those artifacts in dependent jobs where you may race a vos release | 19:33 |
| corvus | off the top of my head, i don't remember the sequence for that | 19:34 |
| fungi | the pypi publication jobs don't reconsume what's pushed onto the tarballs site any longer | 19:34 |
| clarkb | that change happened with the zuulv3 transition | 19:34 |
| fungi | they used to, but the zuul v3 refactor elimniated teh push/pull/push model | 19:34 |
| fungi | yeah, what clarkb said | 19:34 |
| corvus | ok we're probably good then | 19:34 |
| fungi | we only did it back then to be able to have signing keys on a different (static) host for added protection | 19:34 |
| fungi | zuul v3 solves that problem more cleanly for us | 19:35 |
| fungi | well, and also pypi creds | 19:35 |
| clarkb | alright we have ~25 minutes left in our meeting so lets continue on | 19:35 |
| clarkb | #topic Storyboard | 19:35 |
| *** openstack changes topic to "Storyboard (Meeting topic: infra)" | 19:35 | |
| clarkb | fungi: diablo_rojo how are storyboard things? | 19:36 |
| corvus | oh, i set up the slow query log but have not pulled in the net to see what it's caught yet | 19:36 |
| corvus | i need to do that | 19:36 |
| diablo_rojo | No negative feedback fromt he swift team yet | 19:36 |
| diablo_rojo | post test migration | 19:36 |
| diablo_rojo | so thats cool | 19:36 |
| fungi | corvus: i skimmed it the day after you turned it on and there are some egregious examples | 19:36 |
| corvus | fungi: great news! | 19:36 |
| diablo_rojo | We should definitely pull that. | 19:37 |
| diablo_rojo | Now that our outreachy intern is getting started | 19:37 |
| fungi | queries running over 3 minutes with inner-joins several layers deep and majorly compound selects | 19:37 |
| corvus | #action corvus do something with the slow query log | 19:37 |
| diablo_rojo | She could use that to help target what she can improve | 19:37 |
| fungi | corvus: other great news is that SotK thinks some of those may be solved with a change he has up for review | 19:37 |
| clarkb | fungi: corvus diablo_rojo probably make sense to throw that up on paste.o.o or an etherpad and get eyeballs on it? | 19:37 |
| diablo_rojo | (we have an outreachy intern and she has started now) | 19:37 |
| diablo_rojo | Yes please :) | 19:37 |
| corvus | ok i'll do that asap | 19:37 |
| diablo_rojo | corvus, merci :) | 19:38 |
| *** bbowen has quit IRC | 19:38 | |
| fungi | #link http://paste.openstack.org/show/752274 an egregious example from the slow-query log | 19:38 |
| fungi | note that's a single entry for a single query | 19:38 |
| clarkb | and time is in seconds? | 19:38 |
| fungi | yup | 19:38 |
| mordred | wow. | 19:39 |
| diablo_rojo | Ouch. | 19:39 |
| clarkb | sounds like progress at least :) | 19:39 |
| corvus | i wonder if we can pair that up with a request log to figure out what code path triggered it | 19:39 |
| fungi | granted, that was the longest query execution i could find in the first 24 hours of the log, so it's a doozy | 19:39 |
| corvus | one could probably identify the code location from the query alone, but it might be difficult. | 19:40 |
| *** altlogbot_0 has joined #openstack-meeting | 19:40 | |
| *** enriquetaso has quit IRC | 19:40 | |
| fungi | i think SotK knew where it likely was off the top of his head | 19:40 |
| corvus | oh good | 19:40 |
| corvus | then maybe let's just grab the queries, and if we can't figure it out from that, go to the request logs as needed. | 19:40 |
| clarkb | sounds like a plan | 19:41 |
| fungi | yeah, i think that's a pragmatic approach | 19:41 |
| SotK | Yeah I can probably guess at most things from the queries :) | 19:41 |
| clarkb | Anything else storyboard related before we move on? | 19:41 |
| fungi | just feed them to the sotk api ;) | 19:41 |
| fungi | clarkb: nothing from me | 19:42 |
| *** jamesmcarthur has joined #openstack-meeting | 19:42 | |
| clarkb | #topic General Topics | 19:42 |
| *** openstack changes topic to "General Topics (Meeting topic: infra)" | 19:42 | |
| clarkb | #link https://etherpad.openstack.org/p/201808-infra-server-upgrades-and-cleanup | 19:42 |
| clarkb | mordred: any luck with status? | 19:42 |
| clarkb | and fungi same question for you re wiki | 19:42 |
| diablo_rojo | Nothing from me | 19:43 |
| fungi | i have another puppet module change for more packages which needed replacing | 19:43 |
| *** slaweq has quit IRC | 19:43 | |
| * fungi finds | 19:43 | |
| fungi | #link https://review.opendev.org/661848 More transitional Xenial package name replacements | 19:43 |
| fungi | it's rather like whack-a-mole | 19:43 |
| clarkb | looks like it only needs one more +2 (I already got that one) | 19:44 |
| fungi | the entire kitchen sink was explicitly specified in the manifest | 19:44 |
| clarkb | and now we have to do dishes | 19:44 |
| fungi | in a variety of places too | 19:44 |
| fungi | yeah, next time we need to do dishes i think it may be easier to burn down the kitchen and build a new one | 19:44 |
| mordred | clarkb: oh - crappit - THAT was the thing I had on my plate before gitea project creation | 19:44 |
| *** slaweq has joined #openstack-meeting | 19:44 | |
| *** _hemna has joined #openstack-meeting | 19:45 | |
| clarkb | mordred: yay for reminders then :) | 19:45 |
| mordred | yay! | 19:45 |
| corvus | mordred: too late, we just put the plate in the dishwasher | 19:45 |
| corvus | i think | 19:45 |
| clarkb | again let me know if I can assist on these | 19:45 |
| corvus | it's hard to follow the metaphors here | 19:45 |
| *** jamesmcarthur has quit IRC | 19:45 | |
| mordred | I thought the plate got put in the mole | 19:45 |
| clarkb | Next up was following up on the mirror rebuilds | 19:45 |
| *** jamesmcarthur has joined #openstack-meeting | 19:45 | |
| clarkb | ianw do you think we are ready to do that in all regions now? We did have a slight hiccup with mirror.dfw.rax.opendev.org after the apache disk ran out of space btu it had to do with afs ubuntu volume cache | 19:46 |
| clarkb | I did a flushvolume and all is well again | 19:46 |
| ianw | i think so | 19:46 |
| fungi | yeah, still a mystery what happened there | 19:47 |
| ianw | #link https://review.opendev.org/#/c/662357/ | 19:47 |
| ianw | one possible very minor thing is to get the names a bit clearer on the logs ^ | 19:47 |
| fungi | looks like we were returning truncated results from the afs cache starting around the time the apache proxy ran out of disk space in the other volume | 19:47 |
| clarkb | getting that rolled out then adding https to the docker proxy will simplify image building things too so worthwhile to keep pushing on that | 19:47 |
| ianw | fungi: i'm not sure i want to think too hard on why they're related | 19:48 |
| corvus | also it looks like it may be required in order to use buildkit to build images | 19:48 |
| ianw | but yeah, the misconfiguration of disk sizes was my mistake | 19:48 |
| clarkb | Next item on the list is cleaning up our arm64 cloud configuration from nodepool. This is a bit intertwined with mordred's work to have nodepool builders build control plane images | 19:49 |
| clarkb | mordred: ^ if you want to start approving those I'm able to help keep an eye on stuff this afternoon | 19:49 |
| clarkb | we had to cleanup nb03's disk and reboot it (restarting services) to get it building images again. But it is now doing that successfully so I think the functional problems are alrgely addressed | 19:50 |
| clarkb | now just a matter of cleaning out the clouds that no longer exist | 19:50 |
| mordred | clarkb: sweet | 19:50 |
| clarkb | And then if we can sneak it in before our hour is up I wanted to bring up setting a flag day(s) for bindep fallback removal and z-c shim removal from our base jobs | 19:51 |
| fungi | related to mirroring and arm64, hrw has a change up since yesterdayish to add arm64 mirrors of centos to our rsync config | 19:51 |
| mordred | clarkb: ++ to both of those | 19:51 |
| clarkb | This week is an openstack milestone week but we are clear on openstack's shcedule for like the next 6 weeks | 19:51 |
| fungi | #link https://review.opendev.org/662727 centos-mirror-update: handle non-x86 architectures | 19:51 |
| fungi | (for reference) | 19:51 |
| fungi | ppc64 as well i guess | 19:52 |
| clarkb | maybe do the switch on the 24th (that should be plenty of time for warning people and testing) | 19:52 |
| ianw | fungi: yeah, i can babysit that today | 19:52 |
| clarkb | fungi: ^ do you think the 24th would be too soon to do the bindep and z-c removals? | 19:52 |
| fungi | clarkb: of june? yeah that's probably sufficient notice | 19:53 |
| clarkb | ya | 19:53 |
| fungi | if we get to communicating broadly right away | 19:53 |
| clarkb | fungi: I think step 0 is get the chagne up for z-c removal so we can link to it in communications | 19:53 |
| clarkb | then ya step 1 is send email todayish | 19:53 |
| fungi | sure, i'll push it right after the meeting, was mostly looking for consensus on the direction first | 19:53 |
| clarkb | and I figure one day with one big bandaid pull is better than multiple days | 19:53 |
| fungi | i think my jaunt through time reminded me that it was something we'd been meaning to do early in the v3 transition and then we lost track of it | 19:54 |
| clarkb | I think we realized that people were making new legacy jobs | 19:54 |
| clarkb | and so the jobs that depended on it were getting larger in number and not smaller | 19:54 |
| corvus | next up after this, removing legacy-base :) | 19:55 |
| clarkb | #topic Open Discussion | 19:56 |
| *** openstack changes topic to "Open Discussion (Meeting topic: infra)" | 19:56 | |
| clarkb | We have a few minutes for any last items | 19:56 |
| fungi | or more sandwiches | 19:57 |
| corvus | i'm all out of both | 19:57 |
| clarkb | me too. My burger is gone | 19:57 |
| clarkb | oh wait I have a small thing. Larissa has jury duty tomorrow so I'm doing dadops | 19:57 |
| clarkb | I'll be less useful tomorrow than today | 19:57 |
| clarkb | trying to explain jury duty to a 4 year old is difficult if anyone is wondering | 19:58 |
| mordred | clarkb: does that make it doubly difficult to explain to 2 4 year olds? | 19:59 |
| corvus | it's been FOUR years? | 19:59 |
| fungi | one reason the week of the 24th is good for ripping off the legacy job bandages is that i'll be out of town the following week so immune to the flamewar which ensues | 19:59 |
| clarkb | corvus: as of two days ago it has been four years | 19:59 |
| clarkb | fungi: ha | 19:59 |
| corvus | happy birthdays! | 19:59 |
| fungi | look on the bright side... you only need to throw one birthday party ;) | 20:00 |
| clarkb | fungi: its great actually :) | 20:00 |
| clarkb | and we are at time | 20:00 |
| clarkb | thank you everyone | 20:00 |
| clarkb | #endmeeting | 20:00 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 20:00 | |
| openstack | Meeting ended Tue Jun 4 20:00:15 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 20:00 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/infra/2019/infra.2019-06-04-19.01.html | 20:00 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/infra/2019/infra.2019-06-04-19.01.txt | 20:00 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/infra/2019/infra.2019-06-04-19.01.log.html | 20:00 |
| fungi | thanks clarkb! | 20:00 |
| clarkb | mordred: I've only attempted to explain it to one of them so far | 20:01 |
| clarkb | the other was napping or something | 20:01 |
| *** Shrews has left #openstack-meeting | 20:02 | |
| *** radeks_ has joined #openstack-meeting | 20:06 | |
| *** radeks has quit IRC | 20:09 | |
| *** _hemna has quit IRC | 20:19 | |
| *** e0ne has joined #openstack-meeting | 20:23 | |
| *** Lucas_Gray has joined #openstack-meeting | 20:25 | |
| *** Lucas_Gray has quit IRC | 20:27 | |
| *** Lucas_Gray has joined #openstack-meeting | 20:29 | |
| *** enriquetaso has joined #openstack-meeting | 20:30 | |
| *** enriquetaso has quit IRC | 20:37 | |
| *** eharney has quit IRC | 20:41 | |
| *** wwriverrat has quit IRC | 20:41 | |
| *** e0ne has quit IRC | 20:41 | |
| *** e0ne has joined #openstack-meeting | 20:42 | |
| *** e0ne has quit IRC | 20:43 | |
| *** _hemna has joined #openstack-meeting | 20:43 | |
| *** dklyle has quit IRC | 20:49 | |
| *** dklyle has joined #openstack-meeting | 20:49 | |
| *** bbowen has joined #openstack-meeting | 20:52 | |
| *** whoami-rajat has quit IRC | 20:54 | |
| *** mattw4 has quit IRC | 20:55 | |
| *** mattw4 has joined #openstack-meeting | 20:58 | |
| *** jamesmcarthur has quit IRC | 21:00 | |
| *** jamesmcarthur has joined #openstack-meeting | 21:00 | |
| *** jamesmcarthur has quit IRC | 21:04 | |
| *** pcaruana has quit IRC | 21:07 | |
| *** Qiming has quit IRC | 21:11 | |
| *** Qiming has joined #openstack-meeting | 21:14 | |
| *** jamesmcarthur has joined #openstack-meeting | 21:14 | |
| *** jamesmcarthur has quit IRC | 21:18 | |
| *** jamesmcarthur has joined #openstack-meeting | 21:20 | |
| *** mriedem is now known as mriedem_away | 21:22 | |
| *** _hemna has quit IRC | 21:23 | |
| *** ekcs has quit IRC | 21:26 | |
| *** jamesmcarthur has quit IRC | 21:28 | |
| *** raildo has quit IRC | 21:29 | |
| *** munimeha1 has quit IRC | 21:39 | |
| *** jamesmcarthur has joined #openstack-meeting | 21:49 | |
| *** yamamoto has joined #openstack-meeting | 21:50 | |
| *** jamesmcarthur has quit IRC | 21:51 | |
| *** yamamoto has quit IRC | 21:55 | |
| *** jamesmcarthur has joined #openstack-meeting | 22:04 | |
| *** slaweq has quit IRC | 22:07 | |
| *** mattw4 has quit IRC | 22:08 | |
| *** slaweq has joined #openstack-meeting | 22:11 | |
| *** enriquetaso has joined #openstack-meeting | 22:13 | |
| *** jamesmcarthur has quit IRC | 22:16 | |
| *** enriquetaso has quit IRC | 22:18 | |
| *** enriquetaso has joined #openstack-meeting | 22:19 | |
| *** slaweq has quit IRC | 22:24 | |
| *** mattw4 has joined #openstack-meeting | 22:24 | |
| *** slaweq has joined #openstack-meeting | 22:34 | |
| *** enriquetaso has quit IRC | 22:44 | |
| *** slaweq has quit IRC | 22:47 | |
| *** artom has quit IRC | 22:50 | |
| *** radeks_ has quit IRC | 22:58 | |
| *** slaweq has joined #openstack-meeting | 23:04 | |
| *** slaweq has quit IRC | 23:17 | |
| *** Lucas_Gray has quit IRC | 23:17 | |
| *** ykatabam has joined #openstack-meeting | 23:17 | |
| *** rcernin has joined #openstack-meeting | 23:20 | |
| *** lseki has quit IRC | 23:22 | |
| *** ykatabam has quit IRC | 23:29 | |
| *** gyee has joined #openstack-meeting | 23:29 | |
| *** ykatabam has joined #openstack-meeting | 23:30 | |
| *** yamamoto has joined #openstack-meeting | 23:41 | |
| *** enriquetaso has joined #openstack-meeting | 23:43 | |
| *** slaweq has joined #openstack-meeting | 23:50 | |
| *** lseki has joined #openstack-meeting | 23:53 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!