| *** jamesmcarthur has quit IRC | 00:22 | |
| *** sdake has joined #openstack-meeting | 00:26 | |
| *** tetsuro has joined #openstack-meeting | 00:34 | |
| *** ijw has quit IRC | 00:35 | |
| *** ijw has joined #openstack-meeting | 00:35 | |
| *** jamesmcarthur has joined #openstack-meeting | 00:35 | |
| *** lbragstad has quit IRC | 00:39 | |
| *** jamesmcarthur has quit IRC | 00:40 | |
| *** slaweq has quit IRC | 00:41 | |
| *** erlon has joined #openstack-meeting | 00:45 | |
| *** ijw has quit IRC | 00:52 | |
| *** sdake has quit IRC | 00:58 | |
| *** markvoelker has joined #openstack-meeting | 01:10 | |
| *** sdake has joined #openstack-meeting | 01:16 | |
| *** sdake has quit IRC | 01:16 | |
| *** sdake has joined #openstack-meeting | 01:22 | |
| *** whoami-rajat has joined #openstack-meeting | 01:22 | |
| *** ijw has joined #openstack-meeting | 01:29 | |
| *** lbragstad has joined #openstack-meeting | 01:32 | |
| *** hongbin has joined #openstack-meeting | 01:33 | |
| *** erlon has quit IRC | 01:38 | |
| *** ijw has quit IRC | 01:41 | |
| *** ijw has joined #openstack-meeting | 01:41 | |
| *** sdake has quit IRC | 01:42 | |
| *** markvoelker has quit IRC | 01:44 | |
| *** sdake has joined #openstack-meeting | 01:45 | |
| *** jamesmcarthur has joined #openstack-meeting | 01:46 | |
| *** zaneb has joined #openstack-meeting | 01:50 | |
| *** sdake has quit IRC | 01:54 | |
| *** jamesmcarthur has quit IRC | 01:56 | |
| *** jamesmcarthur has joined #openstack-meeting | 01:58 | |
| *** sdake has joined #openstack-meeting | 02:00 | |
| *** armstrong has joined #openstack-meeting | 02:11 | |
| *** sdake has quit IRC | 02:22 | |
| *** _alastor_ has joined #openstack-meeting | 02:33 | |
| *** _alastor_ has quit IRC | 02:38 | |
| *** markvoelker has joined #openstack-meeting | 02:41 | |
| *** jamesmcarthur has quit IRC | 02:41 | |
| *** armstrong has quit IRC | 02:44 | |
| *** psachin has joined #openstack-meeting | 02:59 | |
| *** markvoelker has quit IRC | 03:13 | |
| *** apetrich has quit IRC | 03:14 | |
| *** ya has joined #openstack-meeting | 03:15 | |
| *** ya has quit IRC | 03:20 | |
| *** yaawang has joined #openstack-meeting | 03:22 | |
| *** dklyle has joined #openstack-meeting | 03:23 | |
| *** rfolco has quit IRC | 03:36 | |
| *** tpatil has joined #openstack-meeting | 03:43 | |
| *** armstrong has joined #openstack-meeting | 03:45 | |
| *** psachin has quit IRC | 03:50 | |
| *** armstrong has quit IRC | 03:51 | |
| *** psachin has joined #openstack-meeting | 04:00 | |
| *** samP has joined #openstack-meeting | 04:01 | |
| samP | Hi all for masakari | 04:01 |
|---|---|---|
| tpatil | Hi | 04:01 |
| samP | #startmeeting masakari | 04:02 |
| openstack | Meeting started Tue Feb 19 04:02:11 2019 UTC and is due to finish in 60 minutes. The chair is samP. Information about MeetBot at http://wiki.debian.org/MeetBot. | 04:02 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 04:02 |
| *** openstack changes topic to " (Meeting topic: masakari)" | 04:02 | |
| openstack | The meeting name has been set to 'masakari' | 04:02 |
| *** tashiromt has joined #openstack-meeting | 04:02 | |
| samP | tpatil: hi | 04:02 |
| samP | sorry bit late | 04:02 |
| samP | #topic bugs and tickets | 04:02 |
| *** openstack changes topic to "bugs and tickets (Meeting topic: masakari)" | 04:02 | |
| *** sw3 has joined #openstack-meeting | 04:02 | |
| samP | Any critical bugs or patches to discuss? | 04:03 |
| tpatil | # link : https://launchpad.net/bugs/1815657 | 04:03 |
| openstack | Launchpad bug 1815657 in masakari "python 3 failures - /usr/bin/masakari-wsgi and unit tests" [Undecided,In progress] - Assigned to Corey Bryant (corey.bryant) | 04:03 |
| tpatil | Patch is up for review: https://review.openstack.org/#/c/636749/ | 04:03 |
| tpatil | I will reproduce this issue in 18.04 | 04:03 |
| tpatil | and then review the patch | 04:03 |
| samP | tpatil: thanks. | 04:04 |
| tpatil | In patch https://review.openstack.org/#/c/637486/ I have updated hacking version > 1.1.0 | 04:06 |
| tpatil | please review this patch | 04:06 |
| samP | I will review this to. But I don't have 18.04 to reproduce this, so I will wait for your results | 04:06 |
| samP | tpatil: sure I will | 04:06 |
| tpatil | samP: OK | 04:06 |
| tpatil | Needs review : https://review.openstack.org/635710 | 04:07 |
| tpatil | I have voted +2 on this patch | 04:08 |
| *** andreaf has quit IRC | 04:08 | |
| samP | tpatil: Thanks. lgtm, I will merge this. | 04:09 |
| *** andreaf has joined #openstack-meeting | 04:09 | |
| tpatil | samP: Ok, Thanks | 04:09 |
| *** belmoreira has quit IRC | 04:10 | |
| tpatil | #link : https://review.openstack.org/#/c/636553/2 | 04:10 |
| tpatil | This patch addresses LP bug : https://bugs.launchpad.net/masakari/+bug/1815718 | 04:10 |
| openstack | Launchpad bug 1815718 in masakari "masakari should ignore notification when host status is "UNKNOWN"" [Undecided,In progress] - Assigned to Lucas hua (lucas.hua) | 04:10 |
| tpatil | if host status is UNKNOWN, then the notification should be ignored | 04:11 |
| tpatil | for notification_event STOPPED | 04:12 |
| tpatil | If host_status is "UNKNOWN" that means we don't know the power state of host, evacuate nova compute will cause VM split brain. Just ignore notification | 04:12 |
| samP | before host-monitor sends the notifitcaion, pacemaker (in general case) have already kill the node, isn't it? | 04:13 |
| tpatil | from host-monitor point of view, maybe it's not reproducible. but if some service calls notification api , this issue could happen | 04:14 |
| *** janki has joined #openstack-meeting | 04:14 | |
| *** imsurit_ofc has joined #openstack-meeting | 04:15 | |
| samP | That's true. but the main point is, masakari does not responsible for fencing a node. monitor or service which send the notification should responsible for fence the node before send the notificaion | 04:16 |
| samP | I am not against to ignore this notification, but I need to check the nova code for put host to an unknown state. | 04:19 |
| tpatil | samP: Ok, Got it | 04:20 |
| samP | tpatil: I will put my comments on the patch | 04:20 |
| tpatil | samP: OK | 04:21 |
| samP | any other bugs or patches? | 04:21 |
| tpatil | No | 04:21 |
| samP | thanks | 04:22 |
| samP | #topic Stein work items | 04:22 |
| *** openstack changes topic to "Stein work items (Meeting topic: masakari)" | 04:22 | |
| samP | please share any updates on Stein work items | 04:22 |
| tpatil | Event Notification : Specs https://review.openstack.org/#/c/473057/ | 04:24 |
| tpatil | needs review | 04:24 |
| tpatil | Implementation is ready : https://review.openstack.org/#/q/topic:bp/notifications-in-masakari+(status:open+OR+status:merged) | 04:24 |
| samP | tpatil: Thanks, I will review those. sorry for the delay | 04:25 |
| tpatil | samP : NP | 04:25 |
| tpatil | Add progress details for recovery workflow: Specs https://review.openstack.org/#/c/632079/ | 04:26 |
| tpatil | Needs review | 04:26 |
| *** jamesmcarthur has joined #openstack-meeting | 04:26 | |
| tpatil | Implementation is in progress | 04:26 |
| tpatil | Presently taskflow doesn't support to fetch progress details in the order of workflow execution | 04:27 |
| *** zaneb has quit IRC | 04:27 | |
| tpatil | We have reported this issue in taskflow | 04:27 |
| samP | tpatil: thanks | 04:27 |
| ShilpaSD | here is the link https://bugs.launchpad.net/taskflow/+bug/1815738 | 04:27 |
| openstack | Launchpad bug 1815738 in taskflow "Taskflow doesn't support to retrieve tasks progress details in the order of workflow task execution" [Undecided,New] | 04:27 |
| tpatil | until this issue not fixed, we will write extra logic to sorted out the progress details as per the required order | 04:27 |
| samP | ShilpaSD: thanks for the link | 04:28 |
| tpatil | Add functional CI job and tests for segments: In progress | 04:29 |
| tpatil | #link : https://review.openstack.org/#/c/633663/ | 04:29 |
| tpatil | Waiting for | 04:30 |
| tpatil | waiting for author to fix the review comments | 04:30 |
| samP | tpatil: thanks | 04:30 |
| tpatil | Devstack support to install and run masakari-monitors: Good progress | 04:31 |
| *** jamesmcarthur has quit IRC | 04:31 | |
| tpatil | Patch will be uploaded in this week, except host-monitors | 04:32 |
| tpatil | As it has dependency on pacemaker | 04:32 |
| tpatil | the host-monitor fails to start if pacemaker is not installed and running | 04:32 |
| samP | tpatil: got it. | 04:32 |
| *** radez has quit IRC | 04:33 | |
| samP | I will try to review all the pending patches asap | 04:34 |
| tpatil | samP: Ok, Thanks | 04:34 |
| samP | thanks for the updates. | 04:35 |
| samP | #topic AOB | 04:36 |
| *** openstack changes topic to "AOB (Meeting topic: masakari)" | 04:36 | |
| samP | Any other items to discuss? | 04:36 |
| tpatil | Nothing from my side | 04:37 |
| samP | OK then, but early but let finish the meeting here. | 04:37 |
| tpatil | samP: Ok | 04:38 |
| samP | Please use IRC #openstack-masakari or ML for further discussion. | 04:38 |
| samP | Thank you all. | 04:38 |
| samP | #endmeeting | 04:38 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 04:38 | |
| openstack | Meeting ended Tue Feb 19 04:38:26 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 04:38 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/masakari/2019/masakari.2019-02-19-04.02.html | 04:38 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/masakari/2019/masakari.2019-02-19-04.02.txt | 04:38 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/masakari/2019/masakari.2019-02-19-04.02.log.html | 04:38 |
| *** zaneb has joined #openstack-meeting | 04:42 | |
| *** vishakha has joined #openstack-meeting | 04:45 | |
| *** tpatil has quit IRC | 04:47 | |
| *** irclogbot_1 has quit IRC | 04:57 | |
| *** jamesmcarthur has joined #openstack-meeting | 04:58 | |
| *** jamesmcarthur has quit IRC | 05:03 | |
| *** bdperkin has quit IRC | 05:08 | |
| *** bdperkin has joined #openstack-meeting | 05:25 | |
| *** ijw has quit IRC | 05:27 | |
| *** ijw has joined #openstack-meeting | 05:29 | |
| *** ijw has quit IRC | 05:34 | |
| *** jamesmcarthur has joined #openstack-meeting | 05:38 | |
| *** hongbin has quit IRC | 05:41 | |
| *** markvoelker has joined #openstack-meeting | 05:41 | |
| *** jamesmcarthur has quit IRC | 05:43 | |
| *** ralonsoh has joined #openstack-meeting | 05:48 | |
| *** lbragstad has quit IRC | 05:53 | |
| *** sridharg has joined #openstack-meeting | 05:53 | |
| *** jamesmcarthur has joined #openstack-meeting | 06:10 | |
| *** markvoelker has quit IRC | 06:14 | |
| *** jamesmcarthur has quit IRC | 06:15 | |
| *** e0ne has joined #openstack-meeting | 06:25 | |
| *** e0ne has quit IRC | 06:26 | |
| *** apetrich has joined #openstack-meeting | 06:38 | |
| *** Luzi has joined #openstack-meeting | 06:48 | |
| *** zhubx has joined #openstack-meeting | 06:49 | |
| *** zhubx has quit IRC | 06:51 | |
| *** tashiromt has quit IRC | 06:56 | |
| *** kopecmartin|off is now known as kopecmartin | 07:02 | |
| *** slaweq has joined #openstack-meeting | 07:10 | |
| *** markvoelker has joined #openstack-meeting | 07:11 | |
| *** jamesmcarthur has joined #openstack-meeting | 07:11 | |
| *** jamesmcarthur has quit IRC | 07:15 | |
| *** belmoreira has joined #openstack-meeting | 07:16 | |
| *** yaawang has quit IRC | 07:19 | |
| *** e0ne has joined #openstack-meeting | 07:23 | |
| *** rcernin has quit IRC | 07:25 | |
| *** yaawang has joined #openstack-meeting | 07:26 | |
| *** aojea has joined #openstack-meeting | 07:29 | |
| *** takamatsu has joined #openstack-meeting | 07:40 | |
| *** ociuhandu has joined #openstack-meeting | 07:40 | |
| *** cloudrancher has quit IRC | 07:43 | |
| *** cloudrancher has joined #openstack-meeting | 07:43 | |
| *** markvoelker has quit IRC | 07:43 | |
| *** ociuhandu has quit IRC | 07:47 | |
| *** cloudrancher has quit IRC | 07:47 | |
| *** jamesmcarthur has joined #openstack-meeting | 07:48 | |
| *** hadai has joined #openstack-meeting | 07:50 | |
| *** jamesmcarthur has quit IRC | 07:53 | |
| *** nnsingh has joined #openstack-meeting | 07:56 | |
| *** shubham_potale_ has joined #openstack-meeting | 07:56 | |
| *** pcaruana has joined #openstack-meeting | 08:11 | |
| *** e0ne has quit IRC | 08:13 | |
| *** sdake has joined #openstack-meeting | 08:27 | |
| *** cloudrancher has joined #openstack-meeting | 08:30 | |
| *** tssurya has joined #openstack-meeting | 08:31 | |
| *** liuyulong_ has joined #openstack-meeting | 08:33 | |
| *** _alastor_ has joined #openstack-meeting | 08:36 | |
| *** sdake has quit IRC | 08:37 | |
| *** electrofelix has joined #openstack-meeting | 08:38 | |
| *** _alastor_ has quit IRC | 08:40 | |
| *** markvoelker has joined #openstack-meeting | 08:40 | |
| *** sdake has joined #openstack-meeting | 08:45 | |
| *** jamesmcarthur has joined #openstack-meeting | 08:50 | |
| *** liuyulong_ has quit IRC | 08:54 | |
| *** jamesmcarthur has quit IRC | 08:55 | |
| *** ttsiouts has joined #openstack-meeting | 08:57 | |
| *** priteau has joined #openstack-meeting | 08:59 | |
| *** ociuhandu has joined #openstack-meeting | 09:02 | |
| *** hadai has quit IRC | 09:04 | |
| *** ttsiouts has quit IRC | 09:08 | |
| *** imsurit_ofc has quit IRC | 09:08 | |
| *** ttsiouts has joined #openstack-meeting | 09:08 | |
| *** imsurit_ofc has joined #openstack-meeting | 09:09 | |
| *** ttsiouts has quit IRC | 09:13 | |
| *** markvoelker has quit IRC | 09:14 | |
| *** ociuhandu has quit IRC | 09:23 | |
| *** yamamoto has joined #openstack-meeting | 09:23 | |
| *** sdake has quit IRC | 09:25 | |
| *** ttsiouts has joined #openstack-meeting | 09:25 | |
| *** sdake has joined #openstack-meeting | 09:28 | |
| *** ociuhandu has joined #openstack-meeting | 09:31 | |
| *** slaweq has quit IRC | 09:31 | |
| *** slaweq has joined #openstack-meeting | 09:32 | |
| *** ociuhandu has quit IRC | 09:35 | |
| *** ttsiouts has quit IRC | 09:38 | |
| *** ttsiouts has joined #openstack-meeting | 09:38 | |
| *** tetsuro has quit IRC | 09:41 | |
| *** sdake has quit IRC | 09:49 | |
| *** jamesmcarthur has joined #openstack-meeting | 09:51 | |
| *** e0ne has joined #openstack-meeting | 09:53 | |
| *** jamesmcarthur has quit IRC | 09:55 | |
| *** ociuhandu has joined #openstack-meeting | 09:56 | |
| *** ttsiouts has quit IRC | 09:56 | |
| *** ociuhandu has quit IRC | 09:57 | |
| *** ociuhandu has joined #openstack-meeting | 09:58 | |
| *** ttsiouts has joined #openstack-meeting | 09:58 | |
| *** markvoelker has joined #openstack-meeting | 10:11 | |
| *** jamesmcarthur has joined #openstack-meeting | 10:27 | |
| *** jamesmcarthur has quit IRC | 10:32 | |
| *** ttsiouts has quit IRC | 10:40 | |
| *** priteau has quit IRC | 10:41 | |
| *** ttsiouts has joined #openstack-meeting | 10:41 | |
| *** ttsiouts_ has joined #openstack-meeting | 10:42 | |
| *** cloudrancher has quit IRC | 10:43 | |
| *** markvoelker has quit IRC | 10:43 | |
| *** cloudrancher has joined #openstack-meeting | 10:43 | |
| *** cloudrancher has quit IRC | 10:44 | |
| *** ttsiouts has quit IRC | 10:45 | |
| *** cloudrancher has joined #openstack-meeting | 10:46 | |
| *** lpetrut has joined #openstack-meeting | 10:46 | |
| *** carlos_silva has joined #openstack-meeting | 10:50 | |
| *** cloudrancher has quit IRC | 10:54 | |
| *** yamamoto has quit IRC | 11:00 | |
| *** yamamoto has joined #openstack-meeting | 11:06 | |
| *** yamamoto has quit IRC | 11:11 | |
| *** yamamoto has joined #openstack-meeting | 11:11 | |
| *** JangwonLee_ has joined #openstack-meeting | 11:19 | |
| *** JangwonLee__ has joined #openstack-meeting | 11:20 | |
| *** cloudrancher has joined #openstack-meeting | 11:21 | |
| *** JangwonLee has quit IRC | 11:23 | |
| *** JangwonLee_ has quit IRC | 11:24 | |
| *** jawad_axd has joined #openstack-meeting | 11:28 | |
| *** jamesmcarthur has joined #openstack-meeting | 11:29 | |
| *** cloudrancher has quit IRC | 11:32 | |
| *** cloudrancher has joined #openstack-meeting | 11:33 | |
| *** jamesmcarthur has quit IRC | 11:33 | |
| *** markvoelker has joined #openstack-meeting | 11:40 | |
| *** thgcorrea has joined #openstack-meeting | 11:41 | |
| *** imsurit_ofc has quit IRC | 11:46 | |
| *** raildo has joined #openstack-meeting | 11:49 | |
| *** jawad_axd has quit IRC | 11:50 | |
| *** jawad_axd has joined #openstack-meeting | 11:55 | |
| *** ttsiouts_ has quit IRC | 11:57 | |
| *** jawad_axd has quit IRC | 12:03 | |
| *** enriquetaso has joined #openstack-meeting | 12:07 | |
| *** erlon has joined #openstack-meeting | 12:14 | |
| *** markvoelker has quit IRC | 12:14 | |
| *** erlon has quit IRC | 12:14 | |
| *** erlon has joined #openstack-meeting | 12:15 | |
| *** erlon has quit IRC | 12:16 | |
| *** erlon has joined #openstack-meeting | 12:16 | |
| *** cloudrancher has quit IRC | 12:22 | |
| *** rfolco has joined #openstack-meeting | 12:33 | |
| *** cloudrancher has joined #openstack-meeting | 12:39 | |
| *** janki has quit IRC | 12:41 | |
| *** janki has joined #openstack-meeting | 12:44 | |
| *** janki has quit IRC | 12:52 | |
| *** janki has joined #openstack-meeting | 12:53 | |
| *** janki has quit IRC | 12:54 | |
| *** janki has joined #openstack-meeting | 12:54 | |
| *** JangwonLee_ has joined #openstack-meeting | 12:58 | |
| *** JangwonLee__ has quit IRC | 13:01 | |
| *** jamesmcarthur has joined #openstack-meeting | 13:06 | |
| *** markvoelker has joined #openstack-meeting | 13:11 | |
| *** jamesmcarthur has quit IRC | 13:11 | |
| *** ttsiouts has joined #openstack-meeting | 13:11 | |
| *** cloudrancher has quit IRC | 13:11 | |
| *** jamesmcarthur has joined #openstack-meeting | 13:12 | |
| *** ttsiouts has quit IRC | 13:18 | |
| *** ttsiouts has joined #openstack-meeting | 13:19 | |
| *** sdake has joined #openstack-meeting | 13:23 | |
| *** mjturek has joined #openstack-meeting | 13:31 | |
| *** vishakha has quit IRC | 13:31 | |
| *** jamesmcarthur has quit IRC | 13:32 | |
| *** mjturek has quit IRC | 13:32 | |
| *** mjturek has joined #openstack-meeting | 13:37 | |
| *** markvoelker has quit IRC | 13:43 | |
| *** jamesmcarthur has joined #openstack-meeting | 13:49 | |
| *** janki has quit IRC | 13:54 | |
| *** eharney has joined #openstack-meeting | 13:55 | |
| *** jamesmcarthur has quit IRC | 13:56 | |
| *** ttsiouts has quit IRC | 14:04 | |
| *** ttsiouts has joined #openstack-meeting | 14:04 | |
| *** sdake has quit IRC | 14:05 | |
| *** rfolco has quit IRC | 14:07 | |
| *** rfolco has joined #openstack-meeting | 14:08 | |
| *** liuyulong has quit IRC | 14:08 | |
| *** ttsiouts has quit IRC | 14:09 | |
| *** ttsiouts has joined #openstack-meeting | 14:09 | |
| *** yamamoto has quit IRC | 14:10 | |
| *** sdake has joined #openstack-meeting | 14:14 | |
| *** sdake has quit IRC | 14:19 | |
| *** yamamoto has joined #openstack-meeting | 14:24 | |
| *** mriedem has joined #openstack-meeting | 14:25 | |
| *** yamamoto has quit IRC | 14:26 | |
| *** yamamoto has joined #openstack-meeting | 14:27 | |
| *** JangwonLee_ has quit IRC | 14:30 | |
| *** lbragstad has joined #openstack-meeting | 14:34 | |
| *** radez has joined #openstack-meeting | 14:35 | |
| *** aagate has joined #openstack-meeting | 14:37 | |
| *** sdake has joined #openstack-meeting | 14:38 | |
| *** _alastor_ has joined #openstack-meeting | 14:38 | |
| *** mriedem has left #openstack-meeting | 14:40 | |
| *** mriedem has joined #openstack-meeting | 14:40 | |
| *** markvoelker has joined #openstack-meeting | 14:41 | |
| *** _alastor_ has quit IRC | 14:43 | |
| *** munimeha1 has joined #openstack-meeting | 14:45 | |
| *** sdake has quit IRC | 14:53 | |
| *** sdake has joined #openstack-meeting | 15:00 | |
| *** whoami-rajat has quit IRC | 15:01 | |
| *** awaugama has joined #openstack-meeting | 15:03 | |
| *** vishakha has joined #openstack-meeting | 15:04 | |
| *** markvoelker has quit IRC | 15:14 | |
| *** whoami-rajat has joined #openstack-meeting | 15:18 | |
| *** Luzi has quit IRC | 15:19 | |
| *** sdake has quit IRC | 15:28 | |
| *** cloudrancher has joined #openstack-meeting | 15:33 | |
| *** sdake has joined #openstack-meeting | 15:35 | |
| *** KurtB has joined #openstack-meeting | 15:45 | |
| *** lpetrut has quit IRC | 15:45 | |
| *** njohnston_ has joined #openstack-meeting | 15:51 | |
| *** sdake has quit IRC | 15:52 | |
| *** sdake has joined #openstack-meeting | 15:59 | |
| *** e0ne has quit IRC | 16:00 | |
| *** mlavalle has joined #openstack-meeting | 16:00 | |
| *** _alastor_ has joined #openstack-meeting | 16:00 | |
| slaweq | #startmeeting neutron_ci | 16:00 |
| openstack | Meeting started Tue Feb 19 16:00:31 2019 UTC and is due to finish in 60 minutes. The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot. | 16:00 |
| slaweq | hi | 16:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 16:00 |
| *** openstack changes topic to " (Meeting topic: neutron_ci)" | 16:00 | |
| openstack | The meeting name has been set to 'neutron_ci' | 16:00 |
| mlavalle | o/ | 16:00 |
| njohnston_ | o/ | 16:00 |
| *** ianychoi has quit IRC | 16:00 | |
| slaweq | lets wait few minutes for others | 16:01 |
| *** jamesmcarthur has joined #openstack-meeting | 16:01 | |
| slaweq | maybe bcafarel and hongbin will join too | 16:01 |
| slaweq | I know that haleyb is on PTO today | 16:01 |
| bcafarel | o/ | 16:01 |
| bcafarel | thanks for the ping slaweq, I was writing some doc (easy to forget the clock then) | 16:02 |
| slaweq | bcafarel: :) | 16:02 |
| slaweq | ok, let's start then | 16:03 |
| slaweq | #topic Actions from previous meetings | 16:03 |
| *** sdake has quit IRC | 16:03 | |
| *** openstack changes topic to "Actions from previous meetings (Meeting topic: neutron_ci)" | 16:03 | |
| *** e0ne has joined #openstack-meeting | 16:03 | |
| slaweq | mlavalle to check if adding KillFilter for neutron-keepalived-state-change will solve issues with L3 agent in dvr jobs | 16:03 |
| mlavalle | I porposed patch https://review.openstack.org/#/c/636710/ | 16:03 |
| *** hongbin has joined #openstack-meeting | 16:03 | |
| hongbin | o/ | 16:03 |
| slaweq | hi hongbin :) | 16:03 |
| mlavalle | and actually dougwig made an ineteresting comment | 16:03 |
| mlavalle | which I intend to explore | 16:04 |
| njohnston_ | yeah, that is interesting | 16:04 |
| slaweq | but patch basically looks like is helping to solve some of failing tests in multinode job, right? | 16:05 |
| mlavalle | I agree in that we should keep the door as narrowly open as possible | 16:05 |
| bcafarel | +1 | 16:05 |
| slaweq | ++ | 16:05 |
| mlavalle | slaweq: I haven't had time to chack on that since yesterday? Did you look? | 16:06 |
| slaweq | I looked at test's results right now: http://logs.openstack.org/10/636710/2/check/neutron-tempest-plugin-dvr-multinode-scenario/f44d655/testr_results.html.gz | 16:06 |
| slaweq | it looks that "only" 2 tests failed which is much better than it was | 16:07 |
| mlavalle | oh yeah, in fact the trunk lifecycle test is passing | 16:07 |
| mlavalle | which I haven't seen in a long time | 16:07 |
| mlavalle | so it looks we are moving in the right direction | 16:08 |
| slaweq | also from quick look at l3-agent's log, it looks much better and IMO agent was working all the time properly | 16:08 |
| bcafarel | so there really was a rootwrap filter missing all along? | 16:08 |
| mlavalle | it seems so | 16:08 |
| slaweq | bcafarel: for some time at least | 16:08 |
| mlavalle | it was only for some months | 16:08 |
| slaweq | bcafarel: I removed them when I removed old metadata proxy code | 16:08 |
| bcafarel | slaweq: ok, missing for some months makes more sense :) | 16:09 |
| slaweq | :) | 16:09 |
| mlavalle | bcafarel: when I started suspecting the filters were the cause, I had the exact same question in my mind | 16:09 |
| mlavalle | how come did this work before? | 16:09 |
| mlavalle | but we really introduced the bug recently | 16:10 |
| mlavalle | ok, so I'll play with some more rechecks of the patch | 16:10 |
| slaweq | I think that this bug was exposed by switch to python3 | 16:10 |
| slaweq | on python2.7 it is "working" somehow | 16:11 |
| mlavalle | and I'll explore dougwig's suggestion, which is very sensible | 16:11 |
| slaweq | at least agent is not dying when this issue occurs | 16:11 |
| mlavalle | yeah, I think that in python2.7 we still miss the filter | 16:11 |
| *** cloudrancher has quit IRC | 16:11 | |
| *** markvoelker has joined #openstack-meeting | 16:11 | |
| mlavalle | but for some reason we don't kill the agent | 16:11 |
| *** ianychoi has joined #openstack-meeting | 16:12 | |
| mlavalle | anbd therefore we don't have a chain reaction in the tests | 16:12 |
| slaweq | yep | 16:12 |
| slaweq | ok, moving on to the next one then | 16:13 |
| slaweq | bcafarel to continue work on grafana jobs switch to python 3 | 16:13 |
| bcafarel | s/grafana/grenade/ | 16:13 |
| slaweq | bcafarel: right :) | 16:13 |
| bcafarel | I commented on #openstack-neutron yesterday on it, I think it will be better to have grenade job definition in grenade repo once it is full zuul v3 (which will allow devstack_localrc vars to work) | 16:14 |
| bcafarel | so I updated https://review.openstack.org/#/c/636356/ to simply switch to py3 in the meantime | 16:14 |
| bcafarel | just got zuul +1 :) | 16:15 |
| slaweq | and now it looks that it is running on python 3 indeed :) | 16:15 |
| slaweq | for example here: http://logs.openstack.org/56/636356/7/check/neutron-grenade-multinode/0ea452a/logs/screen-q-l3.txt.gz#_Feb_19_14_14_16_092790 | 16:15 |
| bcafarel | yes, wrong variable in my first try sorry, now it looks good | 16:16 |
| slaweq | +2 it already | 16:16 |
| slaweq | IMO it will be good if we just have single node grenade on python 2.7 | 16:16 |
| slaweq | and others on python3 | 16:17 |
| njohnston_ | and the nature of the grenade job is that it upgrades from an old version running py27 to a new version running py3 | 16:17 |
| njohnston_ | so regardless of what code is running the grenade harness we get some testing on both versions of neutron code | 16:18 |
| slaweq | njohnston_: I think that in those jobs it runs on python3 for both old and new | 16:18 |
| slaweq | see e.g. here: http://logs.openstack.org/56/636356/7/check/neutron-grenade-multinode/0ea452a/logs/old/local_conf.txt.gz | 16:18 |
| slaweq | if it would be possible to do upgrade from py27 to py3 that would be great IMO | 16:19 |
| slaweq | but maybe that should be discussed with QA team? | 16:19 |
| njohnston_ | I checked the grenade output, for example | 16:20 |
| slaweq | njohnston_: in grenade-py3 job (defined in grenade repo) it is also like that, all on py3: http://logs.openstack.org/56/636356/7/check/grenade-py3/b075864/logs/old/local_conf.txt.gz | 16:20 |
| *** cloudrancher has joined #openstack-meeting | 16:20 | |
| njohnston_ | old: http://logs.openstack.org/56/636356/7/check/grenade-py3/b075864/logs/grenade.sh.txt.gz#_2019-02-19_13_45_48_138 "New python executable in /opt/stack/old/requirements/.venv/bin/python2" | 16:20 |
| njohnston_ | new: http://logs.openstack.org/56/636356/7/check/grenade-py3/b075864/logs/grenade.sh.txt.gz#_2019-02-19_14_19_45_871 "New python executable in /opt/stack/new/requirements/.venv/bin/python3.5" | 16:20 |
| hongbin | if the upgrade from py3 to py3 success, is there any scenario that upgrade from py2 to py3 break? | 16:21 |
| slaweq | njohnston_: but is devstack deployed in venv in grenade job? I really don't think so | 16:22 |
| slaweq | but I'm not grenade expert so may be wrong here :) | 16:22 |
| slaweq | hongbin: we don't have any other jobs which tests upgrade | 16:22 |
| njohnston_ | slaweq: I can ask the QA team to be sure | 16:23 |
| slaweq | njohnston_: ok, thx | 16:24 |
| hongbin | ok, from me, i think testing from py3 to py3 upgrade is enough | 16:24 |
| slaweq | njohnston_: but still, patch from bcafarel is "consistent" with this single node grenade job so we can IMO go forward with it :) | 16:24 |
| njohnston_ | slaweq: Absolutely, no disagreement there | 16:24 |
| slaweq | njohnston_: great :) | 16:25 |
| slaweq | ok, can we move on? | 16:25 |
| njohnston_ | yep | 16:27 |
| slaweq | ok, next one then | 16:27 |
| slaweq | slaweq to propose patch with new decorator skip_if_timeout in functional tests | 16:27 |
| slaweq | I did patch https://review.openstack.org/#/c/636892/ | 16:27 |
| slaweq | please review it if You can | 16:27 |
| njohnston_ | looks like functional tests failed on the last run | 16:28 |
| njohnston_ | I guess that would be the run still running | 16:29 |
| slaweq | njohnston_: where? | 16:29 |
| bcafarel | http://logs.openstack.org/92/636892/1/gate/neutron-functional/85da30c/logs/testr_results.html.gz ? | 16:29 |
| slaweq | ahh, in the gate | 16:30 |
| slaweq | ok, so maybe I will need to investigate it more :) | 16:30 |
| slaweq | thx | 16:30 |
| njohnston_ | http://logs.openstack.org/92/636892/1/gate/neutron-functional/85da30c/job-output.txt.gz#_2019-02-19_15_32_02_801108 | 16:30 |
| *** yamamoto has quit IRC | 16:31 | |
| slaweq | so it looks that it maybe don't work properly, I will investigate that tomorrow morning then | 16:31 |
| slaweq | #action slaweq to fix patch with skip_if_timeout decorator | 16:31 |
| bcafarel | you did get your test failure in the end at least :) | 16:31 |
| slaweq | LOL, indeed | 16:31 |
| slaweq | ok, that's all for actions from last week | 16:32 |
| slaweq | next topic is: | 16:32 |
| slaweq | #topic Python 3 | 16:32 |
| *** openstack changes topic to "Python 3 (Meeting topic: neutron_ci)" | 16:32 | |
| slaweq | as we already talked today, we have (now in the gate even) patch for grenade multinode jobs to switch to py3 | 16:33 |
| slaweq | thx bcafarel :) | 16:33 |
| slaweq | there is also patch https://review.openstack.org/633979 for neutron-tempest-dvr-ha-multinode-full job | 16:33 |
| *** yamamoto has joined #openstack-meeting | 16:33 | |
| *** yamamoto has quit IRC | 16:33 | |
| slaweq | but this one is still failing with some tests | 16:33 |
| *** yamamoto has joined #openstack-meeting | 16:33 | |
| *** yamamoto has quit IRC | 16:33 | |
| slaweq | I'm not sure if that is issue with tests or maybe similar problem like we have in neutron-tempes-plugin-scenario-dvr-multinode job | 16:34 |
| slaweq | so I will probably split it into 2 pieces: migration to zuulv3 and then second patch with switch to py3 | 16:34 |
| slaweq | do You agree? | 16:35 |
| mlavalle | yes | 16:35 |
| bcafarel | sounds good yes | 16:35 |
| slaweq | thanks :) | 16:35 |
| njohnston_ | +1 | 16:35 |
| slaweq | #action slaweq to split patch https://review.openstack.org/633979 into two: zuulv3 and py3 parts | 16:35 |
| slaweq | and I think that this will be all for switch to py3 | 16:36 |
| slaweq | we will still have some experimental jobs to switch but that can be done slowly later I think | 16:36 |
| *** jaypipes has quit IRC | 16:36 | |
| *** jaypipes has joined #openstack-meeting | 16:36 | |
| bcafarel | yeah it would be a bad sign if important tests we needed for proper python3 support are hidden in experimental jobs | 16:37 |
| bcafarel | so we can do these "leisurely" | 16:37 |
| mlavalle | I like leisurely | 16:37 |
| slaweq | LOL | 16:38 |
| slaweq | me too | 16:38 |
| slaweq | ok, any other questions/something to add about python3? | 16:38 |
| slaweq | or can we move on? | 16:38 |
| mlavalle | not from me | 16:38 |
| njohnston_ | I need to send an email to to openstack-discuss to ask the stadium projects about their py3 status | 16:39 |
| bcafarel | all good here | 16:39 |
| * mlavalle will have to drop off at 45 minutes after the hour | 16:39 | |
| slaweq | njohnston_++ | 16:39 |
| slaweq | #action njohnston to ask stadium projects about python3 status | 16:39 |
| njohnston_ | I can help with some, but I wouldn't want to but in to projects like midonet that I know little about | 16:39 |
| slaweq | njohnston_: yes, same for me | 16:40 |
| njohnston_ | I believe I sent an email a while back and got no requests for help, but let's see if anyone is so motivated now | 16:40 |
| slaweq | thx njohnston_ | 16:40 |
| slaweq | ok, lets move on quickly | 16:40 |
| slaweq | #topic Grafana | 16:40 |
| *** openstack changes topic to "Grafana (Meeting topic: neutron_ci)" | 16:40 | |
| slaweq | #link http://grafana.openstack.org/dashboard/db/neutron-failure-rate | 16:40 |
| slaweq | I was looking at grafana today and TBH I don't see anything very bad on it in last few days | 16:42 |
| slaweq | but maybe You noticed something what You want to discuss about | 16:42 |
| slaweq | even all tempest jobs are in quite good shape this week | 16:43 |
| mlavalle | looks pretty good to me | 16:43 |
| *** markvoelker has quit IRC | 16:44 | |
| mlavalle | the only thing I see if functional py27 in gate | 16:44 |
| slaweq | mlavalle: yes, functional tests are not very good currently | 16:44 |
| slaweq | and I have cuplrits of it | 16:44 |
| slaweq | #topic fullstack/functional | 16:44 |
| *** openstack changes topic to "fullstack/functional (Meeting topic: neutron_ci)" | 16:44 | |
| *** cloudrancher has quit IRC | 16:44 | |
| slaweq | we recently noticed at least 3 bugs in functional tests: | 16:45 |
| slaweq | - https://bugs.launchpad.net/neutron/+bug/1816239 - patch proposed https://review.openstack.org/#/c/637544/ | 16:45 |
| openstack | Launchpad bug 1816239 in neutron "Functional test test_router_processing_pool_size failing" [High,In progress] - Assigned to Brian Haley (brian-haley) | 16:45 |
| slaweq | - https://bugs.launchpad.net/neutron/+bug/1815585 - if there will be no anyone to look at this, I will try to debug it | 16:45 |
| openstack | Launchpad bug 1815585 in neutron "Floating IP status failed to transition to DOWN in neutron-tempest-plugin-scenario-linuxbridge" [High,Confirmed] | 16:45 |
| slaweq | - https://bugs.launchpad.net/neutron/+bug/1816489 - same here, we need volunteer for this one | 16:45 |
| openstack | Launchpad bug 1816489 in neutron "Functional test neutron.tests.functional.agent.l3.test_ha_router.LinuxBridgeL3HATestCase. test_ha_router_lifecycle failing" [High,Confirmed] | 16:45 |
| slaweq | first one is hitting as the most often I think and it has already fix proposed | 16:45 |
| slaweq | so we should be better after this will be merged | 16:46 |
| slaweq | other 2 needs volunteers to debug :) | 16:46 |
| mlavalle | oh, oh, do we depend on haleyb for this? we are in trouble :-) | 16:46 |
| slaweq | I can take a look at https://bugs.launchpad.net/neutron/+bug/1815585 but someone else could look at the last one maybe :) | 16:46 |
| openstack | Launchpad bug 1815585 in neutron "Floating IP status failed to transition to DOWN in neutron-tempest-plugin-scenario-linuxbridge" [High,Confirmed] | 16:46 |
| mlavalle | ok, I'll try to look at the last one | 16:47 |
| slaweq | mlavalle: no, fortunatelly fix was done by liuyulong :) | 16:47 |
| slaweq | thx mlavalle | 16:47 |
| mlavalle | LOL | 16:47 |
| slaweq | #action mlavalle to check bug https://bugs.launchpad.net/neutron/+bug/1816489 | 16:47 |
| openstack | Launchpad bug 1816489 in neutron "Functional test neutron.tests.functional.agent.l3.test_ha_router.LinuxBridgeL3HATestCase. test_ha_router_lifecycle failing" [High,Confirmed] | 16:47 |
| *** macza has joined #openstack-meeting | 16:47 | |
| slaweq | #action slaweq to check bug https://bugs.launchpad.net/neutron/+bug/1815585 | 16:47 |
| mlavalle | ok guyas I got to leave | 16:47 |
| slaweq | ok, thx mlavalle | 16:47 |
| slaweq | see You later | 16:47 |
| mlavalle | o/ | 16:47 |
| slaweq | basically that is all what I have for today | 16:48 |
| bcafarel | o/ mlavalle | 16:48 |
| slaweq | other jobs are in pretty good shape | 16:48 |
| slaweq | so do You have something else You want to talk about today? | 16:48 |
| bcafarel | nothing from me | 16:49 |
| slaweq | ok, so lets have 10 minutes back today :) | 16:49 |
| slaweq | thx for attending and see You all around | 16:49 |
| slaweq | o/ | 16:49 |
| bcafarel | \o/ | 16:49 |
| slaweq | #endmeeting | 16:49 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 16:49 | |
| openstack | Meeting ended Tue Feb 19 16:49:45 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:49 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/neutron_ci/2019/neutron_ci.2019-02-19-16.00.html | 16:49 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/neutron_ci/2019/neutron_ci.2019-02-19-16.00.txt | 16:49 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/neutron_ci/2019/neutron_ci.2019-02-19-16.00.log.html | 16:49 |
| mlavalle | slaweq: ahh, didn't miss much | 16:50 |
| slaweq | mlavalle: LOL | 16:50 |
| slaweq | mlavalle: indeed | 16:50 |
| bcafarel | mlavalle: just to make you feel bad, we waited until you leave | 16:50 |
| slaweq | :) | 16:50 |
| mlavalle | slaweq: yeah, you made me feel guilty... LOL | 16:51 |
| slaweq | mlavalle: I didn't want that, really ;P | 16:51 |
| *** hongbin has quit IRC | 16:54 | |
| *** ttsiouts has quit IRC | 16:56 | |
| *** ttsiouts has joined #openstack-meeting | 16:56 | |
| *** mattw4 has joined #openstack-meeting | 16:57 | |
| *** pcaruana has quit IRC | 16:57 | |
| *** mattw4 has quit IRC | 16:59 | |
| *** tssurya has quit IRC | 17:01 | |
| *** ttsiouts has quit IRC | 17:01 | |
| *** raildo_ has joined #openstack-meeting | 17:02 | |
| *** raildo has quit IRC | 17:03 | |
| *** raildo has joined #openstack-meeting | 17:06 | |
| *** raildo_ has quit IRC | 17:07 | |
| *** raildo has quit IRC | 17:08 | |
| *** raildo has joined #openstack-meeting | 17:09 | |
| *** yamamoto has joined #openstack-meeting | 17:10 | |
| *** raildo has quit IRC | 17:14 | |
| *** bbowen has joined #openstack-meeting | 17:15 | |
| *** aojea has quit IRC | 17:16 | |
| *** yamamoto has quit IRC | 17:16 | |
| *** artom has quit IRC | 17:24 | |
| *** raildo has joined #openstack-meeting | 17:27 | |
| *** lpetrut has joined #openstack-meeting | 17:27 | |
| *** Vadmacs has joined #openstack-meeting | 17:32 | |
| *** raildo has quit IRC | 17:33 | |
| *** ociuhandu_ has joined #openstack-meeting | 17:33 | |
| *** e0ne has quit IRC | 17:35 | |
| *** ociuhandu has quit IRC | 17:36 | |
| *** eharney has quit IRC | 17:37 | |
| *** ociuhandu_ has quit IRC | 17:38 | |
| *** psachin has quit IRC | 17:39 | |
| *** lpetrut has quit IRC | 17:40 | |
| *** markvoelker has joined #openstack-meeting | 17:41 | |
| *** raildo has joined #openstack-meeting | 17:48 | |
| *** ociuhandu has joined #openstack-meeting | 17:48 | |
| *** ociuhandu has quit IRC | 17:52 | |
| *** bbowen has quit IRC | 17:53 | |
| *** kopecmartin is now known as kopecmartin|off | 17:53 | |
| *** raildo_ has joined #openstack-meeting | 18:02 | |
| *** ociuhandu has joined #openstack-meeting | 18:04 | |
| *** raildo has quit IRC | 18:05 | |
| *** ociuhandu has quit IRC | 18:08 | |
| *** ijw has joined #openstack-meeting | 18:08 | |
| *** raildo has joined #openstack-meeting | 18:11 | |
| *** raildo_ has quit IRC | 18:12 | |
| *** raildo has quit IRC | 18:13 | |
| *** raildo_ has joined #openstack-meeting | 18:14 | |
| *** markvoelker has quit IRC | 18:14 | |
| *** jamesmcarthur has quit IRC | 18:17 | |
| *** jamesmcarthur has joined #openstack-meeting | 18:18 | |
| *** raildo has joined #openstack-meeting | 18:20 | |
| *** raildo_ has quit IRC | 18:21 | |
| *** tssurya has joined #openstack-meeting | 18:28 | |
| *** mattw4 has joined #openstack-meeting | 18:29 | |
| *** raildo_ has joined #openstack-meeting | 18:30 | |
| *** raildo has quit IRC | 18:30 | |
| *** ociuhandu has joined #openstack-meeting | 18:30 | |
| *** artom has joined #openstack-meeting | 18:32 | |
| *** artom has quit IRC | 18:33 | |
| *** artom has joined #openstack-meeting | 18:33 | |
| *** hongbin has joined #openstack-meeting | 18:34 | |
| *** hongbin has quit IRC | 18:35 | |
| *** hongbin has joined #openstack-meeting | 18:35 | |
| *** lpetrut has joined #openstack-meeting | 18:39 | |
| *** e0ne has joined #openstack-meeting | 18:41 | |
| *** e0ne has quit IRC | 18:42 | |
| *** jamesmcarthur has quit IRC | 18:53 | |
| *** e0ne has joined #openstack-meeting | 18:54 | |
| fungi | aloha, infrateers | 19:00 |
| clarkb | hello there | 19:00 |
| clarkb | we'll get started momentarily | 19:00 |
| clarkb | #startmeeting infra | 19:01 |
| openstack | Meeting started Tue Feb 19 19:01:08 2019 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. | 19:01 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 19:01 |
| *** openstack changes topic to " (Meeting topic: infra)" | 19:01 | |
| openstack | The meeting name has been set to 'infra' | 19:01 |
| clarkb | #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting | 19:01 |
| gary_perkins | o/ Hi! | 19:01 |
| ianw | o/ | 19:01 |
| *** Shrews has joined #openstack-meeting | 19:01 | |
| clarkb | I seem to remember scribbling a note to link to the mailing list threads when I send those out instead /me scribbles note bigger to do that | 19:02 |
| clarkb | #topic Announcements | 19:02 |
| *** openstack changes topic to "Announcements (Meeting topic: infra)" | 19:02 | |
| *** prometheanfire has joined #openstack-meeting | 19:02 | |
| clarkb | I have nothing to announce. Any announcements others would like to call out? | 19:02 |
| fungi | nothing springs to mind | 19:03 |
| *** electrofelix has quit IRC | 19:03 | |
| clarkb | I guess I should note the openstack TC election nomination period is now and ends shortly | 19:04 |
| clarkb | so get your self nomination pushed if that is something you would like to do | 19:04 |
| fungi | ~4.75 hours remaining | 19:04 |
| clarkb | Then be on the lookout for ballots in the near future | 19:04 |
| clarkb | #topic Actions from last meeting | 19:05 |
| *** openstack changes topic to "Actions from last meeting (Meeting topic: infra)" | 19:05 | |
| clarkb | #link http://eavesdrop.openstack.org/meetings/infra/2019/infra.2019-02-12-19.01.txt minutes from last meeting | 19:05 |
| clarkb | We didn't call this out as an action in the meeting notes but ianw took the action of updating the letsencrypt spec to consider dns based acme validation. This has been done and is the subject of our next topic | 19:05 |
| clarkb | #topic Specs approval | 19:06 |
| *** openstack changes topic to "Specs approval (Meeting topic: infra)" | 19:06 | |
| clarkb | #link https://review.openstack.org/#/c/587283/ LetsEncrypt Spec now updated with DNS ACME | 19:06 |
| clarkb | I don't think this is ready for approval based on the good conversations this has started | 19:06 |
| clarkb | It does seem we are making good progress through the conversation though. So please take a look if this interests you at all | 19:06 |
| clarkb | and thank you ianw for getting that put together with some poc ansible work to show how it might actually run in production | 19:07 |
| fungi | it seems to be zeroing in on a more concrete design | 19:07 |
| ianw | yes i've updated from comments yesterday, a small changelog is in the notes for ps14 | 19:07 |
| ianw | i also updated the POC to provide self-signed certs in a testing environment | 19:08 |
| ianw | (one thought for future development is that we could run our own LE, containerised) | 19:08 |
| ianw | future use in CI I mean | 19:09 |
| clarkb | ianw: as far as next steps go it seems like we are nearing where we need to start making some decisions and moving forward? do we think we might be able to put it up for approval next week? if we can get through another round of reviews and post a new ps with some decisions? | 19:09 |
| ianw | and responded to corvus' thoughts on breaking up base.yaml in https://review.openstack.org/637645 | 19:09 |
| clarkb | corvus: fungi ^ you've reviewed it recently too so your thoughts on how quickly we might be able to come to consensus is appreciated | 19:10 |
| ianw | clarkb: yes, i think so; i'll commit to responding to any review comments quickly so we can get things going | 19:10 |
| corvus | i think i'm generally in favor of what's written. | 19:10 |
| *** markvoelker has joined #openstack-meeting | 19:11 | |
| corvus | i would love one more update that said "we aren't going to run anything that touches proprietary rax dns" | 19:11 |
| fungi | i haven't looked at the diff from the most recent patchset yet | 19:11 |
| fungi | skimming now | 19:11 |
| clarkb | as the purchasers of openstack.org certs for our services I'm ok with corvus' request and doing one last bulk purchase round | 19:12 |
| clarkb | if that helps swing any opinions | 19:12 |
| corvus | but i'm at least happy enough that we're all on the same page that it's at least not something we'll use for any new services and will only shrink, that i can go along with what's written | 19:12 |
| clarkb | alright anything else on this before we move on? (thanks again to everyone that helped push this forward, ianw in particular) | 19:13 |
| ianw | my feeling is that it really is quite a small change, and we can have certificates for anything under openstack.org which is something that we've talked about for a long time | 19:13 |
| fungi | having previously obtained the certs for openstack.org for a number of years too, i agree that generating them all at once for a renewal is less painful than adding new ones for services we add over time (and new services should be going into opendev.org anyway) | 19:13 |
| fungi | also, moving our existing http-only services to opendev.org before switching them to https shouldn't pose significant burden | 19:14 |
| clarkb | seems like we should be able to sort that out in review. Please do review the spec. But I think we need to move to our next topic(s) | 19:15 |
| *** ijw has quit IRC | 19:15 | |
| ianw | ++ | 19:15 |
| clarkb | #topic Priority Efforts | 19:15 |
| *** openstack changes topic to "Priority Efforts (Meeting topic: infra)" | 19:15 | |
| clarkb | #topic Storyboard | 19:15 |
| *** openstack changes topic to "Storyboard (Meeting topic: infra)" | 19:16 | |
| clarkb | fungi you have good news for us on this topic iirc | 19:16 |
| clarkb | Storyboard prod and dev servers are now both running on xenial with local mysql DBs right? | 19:16 |
| clarkb | last remaining cleanups are to configure backups? | 19:16 |
| fungi | oh, well we replaced the trusty server with a xenial deployment in production (and upped the flavor help with performance), plus moved the database out of trove and onto the server itself | 19:17 |
| mordred | yay xenial! | 19:17 |
| fungi | #link https://review.openstack.org/637388 Clean up after StoryBoard replacement maintenance | 19:18 |
| SotK | \o/ | 19:18 |
| prometheanfire | good for another 2 years iirc (EOL)? | 19:18 |
| fungi | that's the cleanup change | 19:18 |
| fungi | and yeah, i still need to init the remote backups | 19:18 |
| clarkb | prometheanfire: yup | 19:18 |
| *** lpetrut has quit IRC | 19:18 | |
| fungi | also clarkb spotted that karbor has asked to move to storyboard | 19:19 |
| fungi | #link https://review.openstack.org/636574 Update karbor to use storyboard | 19:19 |
| fungi | and cinder is adding a new repo to sb to test it out too | 19:19 |
| fungi | #link https://review.openstack.org/637613 Add cinderlib project | 19:20 |
| clarkb | cinderlib a library to consume cinder drivers without an api service | 19:20 |
| fungi | yeah, i get the impression that'll be handy for some container orchestration system use cases | 19:20 |
| smcginnis | oVirt is also adopting it. | 19:21 |
| fungi | neat! | 19:21 |
| clarkb | anything else to be aware of or help with in storyboard land? | 19:21 |
| SotK | If anyone has spare time to look at the attachments patches it'd be much appreciated :) | 19:22 |
| *** vishakha has quit IRC | 19:22 | |
| clarkb | #link https://review.openstack.org/#/q/status:open+topic:story-attachments storyboard file attachment changes | 19:22 |
| fungi | nothing else on my side | 19:23 |
| diablo_rojo_phon | Mine either | 19:23 |
| clarkb | #topic Configuration Management Updates | 19:23 |
| *** openstack changes topic to "Configuration Management Updates (Meeting topic: infra)" | 19:23 | |
| *** jamesmcarthur has joined #openstack-meeting | 19:24 | |
| clarkb | Now that I've got pbx off my plate I intend to pick up the puppet 4 upgrades and futureparser changes again. | 19:24 |
| clarkb | Also worht noting (beacuse some had missed it) that we are now running a service with ansible + docker and no puppet. This service is the insecure image registry that will be used for zuul jobs that build docker images | 19:24 |
| fungi | i had forgotten after reviewing many (most?) of those changes even | 19:25 |
| clarkb | I think work here has slowed beacuse its starting to become our new normal. Though still more work to be done with puppet upgrades and other service flips to docker. | 19:26 |
| clarkb | Are there any other items we need to call out on this topic? | 19:26 |
| corvus | i'm planning on having zuul-preview deployed that way too | 19:27 |
| corvus | i think it's easy and fast :) | 19:27 |
| clarkb | ya once the underlying tooling is in place it seems to be pretty straightforard | 19:27 |
| fungi | and sounds like the plan for now is to also do gitea that way? | 19:28 |
| clarkb | the big bootstrapping need was the image building wheel and we have that now | 19:28 |
| clarkb | fungi: thats our next topic :) | 19:28 |
| * fungi returns to the present in that case | 19:28 | |
| clarkb | So maybe time to move on to that | 19:28 |
| clarkb | #topic OpenDev | 19:28 |
| *** openstack changes topic to "OpenDev (Meeting topic: infra)" | 19:28 | |
| clarkb | Last we met at this venue corvus expected we'd have the gitea cluster in a production like state ready for us to really start interacting with it | 19:29 |
| corvus | reality happened | 19:29 |
| clarkb | Then we did a couple gitea upgrades and discovered that gitea had never intended for shared everything gitea clusters to "work" and they fixed it | 19:29 |
| *** sridharg has quit IRC | 19:29 | |
| clarkb | long story short you kind of need to run a single gitea process today the way it is built | 19:29 |
| corvus | #link gitea bug for shared indexes https://github.com/go-gitea/gitea/issues/5798 | 19:30 |
| corvus | to be fair it's the search engine that has this problem | 19:30 |
| corvus | but the search engine is a major selling point | 19:30 |
| clarkb | so after a discussion in #opendev earlier today I think we are leaning towards running multiple shared nothing gitea instances similar to how we run multiple shared nothing cgit instances today | 19:30 |
| fungi | which does mean independently replicating to them for now | 19:31 |
| mordred | ++ | 19:31 |
| clarkb | And longer term it sounds like they'll add support for elasticsearch backed search indexes for the code which will allow us to go back to the shared everything cluster model in k8s | 19:31 |
| fungi | also, i think we didn't touch on this, but we'll likely need some manner of client affinity in the load balancing layer to deal with replication being out of sync between cluster members | 19:32 |
| fungi | (if memory serves we do this in the cgit haproxy instance today?) | 19:32 |
| corvus | i'd like to keep the k8s cluster up and running to continue work in parallel on deployment and testing tooling | 19:32 |
| corvus | fungi: yeah, i expect us to keep exactly the haproxy config we have now with docker-compose gitea. | 19:33 |
| clarkb | fungi: we switched to least connections backend with haproxy but can go back to client affinity based on ip | 19:33 |
| clarkb | iirc | 19:33 |
| mordred | corvus: basically just switching the backend ips really, right? | 19:33 |
| corvus | mordred: yep | 19:33 |
| mordred | \o/ | 19:33 |
| *** eharney has joined #openstack-meeting | 19:34 | |
| fungi | clarkb: oh, did we not actually end up with any git fetch errors from dropping the client affinity? | 19:34 |
| *** sdake has joined #openstack-meeting | 19:34 | |
| fungi | in that case it's probably safe to do the same with gitea | 19:34 |
| corvus | though, maybe it's more complex because we're also changing hostnames... so... i dunno, maybe it's a duplicate haproxy. | 19:34 |
| corvus | and actually... | 19:34 |
| corvus | do we want to host the docker gitea instances in vexxhost? | 19:34 |
| corvus | (that's where we put the k8s cluster) | 19:34 |
| fungi | is there a reason not to? | 19:34 |
| corvus | only reason i can think of is ipv6, which should be fixed before it matters to us. | 19:35 |
| corvus | but if we do, then we definitely want a new lb, since that should be in the same dc as the backends :) | 19:35 |
| fungi | have we gotten working reverse dns for v6 addresses in vexxhost yet? | 19:36 |
| corvus | fungi: we don't have v6 in vexxhost sjc1 yet. but it should be there RSN. | 19:36 |
| fungi | ahh | 19:36 |
| clarkb | corvus: what was the reason for using sjc1 over montreal? | 19:36 |
| corvus | well, originally, because magnum was newer there :) | 19:36 |
| clarkb | there is working ipv6 in the other region today | 19:37 |
| corvus | we could ask mnaser where he'd like us to put them :) | 19:37 |
| clarkb | ++ | 19:37 |
| corvus | either way, it'll be a new load balancer. | 19:37 |
| clarkb | Other news in opendev land is we've started booting replacement servers with opendev dns names and CNAMEing openstack records to them | 19:38 |
| clarkb | I updated the launch docs with some short notes on how this changes our server laucnh processes | 19:38 |
| clarkb | please do propose updates to that little README if there are inconsistencies with reality | 19:38 |
| mordred | sweet | 19:39 |
| *** e0ne has quit IRC | 19:39 | |
| fungi | yeah, the storyboard servers are this way now | 19:39 |
| clarkb | as is the new pbx | 19:39 |
| *** e0ne has joined #openstack-meeting | 19:39 | |
| clarkb | Anything else on opendev? | 19:40 |
| clarkb | #topic General Topics | 19:41 |
| *** openstack changes topic to "General Topics (Meeting topic: infra)" | 19:41 | |
| clarkb | First up frickler points out that I have been bad with emacs and gpg agent and the secrets file | 19:41 |
| prometheanfire | do we know why image sizes jumped? https://nb01.openstack.org/images/ | 19:41 |
| *** mjturek has quit IRC | 19:42 | |
| *** diablo_rojo has joined #openstack-meeting | 19:42 | |
| corvus | me too | 19:42 |
| clarkb | prometheanfire: we can follow up after the agena'd topics | 19:42 |
| frickler | clarkb: I was also wondering whether we somehow automating this to be safer. like having a cronjob that checks for running gpg-agent | 19:42 |
| corvus | we should write a /usr/local/bin script | 19:43 |
| clarkb | frickler: or maybe create a small script to use to run emacs for that file | 19:43 |
| corvus | editpw or something | 19:43 |
| prometheanfire | clarkb: k | 19:43 |
| clarkb | corvus: jinx | 19:43 |
| mordred | I have also been bad at emacs | 19:43 |
| mordred | clarkb, corvus: ++ | 19:43 |
| clarkb | corvus: I think that solution will be easy for me to remember once i use it a couple times | 19:43 |
| clarkb | frickler: ^ what do you think? | 19:43 |
| frickler | sounds good, yes | 19:43 |
| corvus | yeah. there's no way i'll remember "gpg-agent --daemon emacs" | 19:43 |
| ianw | was this an email? i've missed this | 19:43 |
| corvus | no one think about that too much. | 19:44 |
| ianw | re: gpg agent | 19:44 |
| fungi | can we configure gpg-agent to run on-demand and terminate immediately by default? | 19:44 |
| clarkb | ianw: no frickler pointed it out in irc then added it to the meeting agenda | 19:44 |
| *** markvoelker has quit IRC | 19:44 | |
| clarkb | ianw: the tldr is running emacs to open the gpg encrypted file starts a gpg agent that never goes away | 19:44 |
| fungi | i want to say i've seen instructions for how to do that when building and signing debian packages in a chroot | 19:44 |
| frickler | ianw: and it caches the passphrase for an hour or so | 19:45 |
| clarkb | fungi: the gpg-agent --daemon emacs command is the command version of that | 19:45 |
| clarkb | fungi: so it is possible | 19:45 |
| ianw | oh right, yeah from time to time i've killed gpg-agents after emacs went bananas and totally messed up the terminal too | 19:45 |
| *** e0ne has quit IRC | 19:45 | |
| mordred | fungi: we were unable to find any way to cause the correct behavior when we were looking at this in berlin | 19:46 |
| clarkb | I think the editpw script solves this in a pretty straightforward way | 19:46 |
| clarkb | so I'm good with that option | 19:46 |
| mordred | fungi: other than the gpg-agent --daemon emacs ... but if there is a way to just configure the system to not be terrible, that would bae awesome | 19:46 |
| mordred | clarkb: ++ | 19:47 |
| clarkb | frickler: is that something you would like to push up? | 19:47 |
| fungi | yeah, i'm hunting for a config option there | 19:48 |
| clarkb | ok we've only got ~12 minutes so lets get to the next item | 19:48 |
| frickler | clarkb: I can give it a try, but if someone else wants to do it, I'm not too sad either | 19:48 |
| clarkb | frickler: ok | 19:48 |
| clarkb | Next up Trusty server upgrades | 19:48 |
| clarkb | #link https://etherpad.openstack.org/p/201808-infra-server-upgrades-and-cleanup | 19:48 |
| clarkb | fungi and I have made recent progress on this | 19:48 |
| clarkb | there are still a few outstanding and potentially getting more difficult servers in that list. | 19:49 |
| fungi | yeah, frickler just now approved the inventory addition for the openstackid.org xenial replacement so we're probably close to swapping it out | 19:49 |
| clarkb | If you can grab one or two it will go quickly. Another thought was maybe a sprint sometime after openstack release candidate processes quiet down? ( think that is mid march? | 19:49 |
| fungi | we probably need to plan another in-place ubuntu release upgrade for lists.o.o | 19:50 |
| clarkb | I can send out an email to try and organize a sprint if we think we'd sit down for a few days to focus on this. Maybe let me know in the infra channel today if you think you could help with a sprint | 19:50 |
| clarkb | prometheanfire: ok image size questions. The easiest way to start sorting that out would likely be to grab the qcow2's from before and after the size job, and either nbd mount them or boot them and run du? | 19:51 |
| clarkb | prometheanfire: I do not know why there is a size jump | 19:51 |
| clarkb | most of that image size cost is git repos iirc. So it is possible we've got a git repo growing quickly | 19:51 |
| ianw | i will go back and see why we turned of the automatic size reporting in the dib logs | 19:51 |
| corvus | i'd like to let folks know that i'm continuing work on using the intermediate docker registry. still hitting bugs, but hopefully soon we'll be able to have speculative docker images, which means we can test changes to images in our deployment tooling before deploying them. | 19:51 |
| ianw | it was intended for exactly this :) | 19:52 |
| clarkb | corvus: that is exciting | 19:52 |
| clarkb | corvus: also yay for ansible bugs/behavior | 19:52 |
| mordred | corvus: \o/ | 19:52 |
| clarkb | #topic Open Discussion | 19:52 |
| *** openstack changes topic to "Open Discussion (Meeting topic: infra)" | 19:52 | |
| prometheanfire | clarkb: ya, will probably do that | 19:52 |
| clarkb | we've started discussing open things anyway :) | 19:52 |
| *** sdake has quit IRC | 19:52 | |
| prometheanfire | ptg schedule? | 19:52 |
| clarkb | prometheanfire: I brought it up last week and have an etherpad to start putting ideas on | 19:53 |
| * clarkb finds link | 19:53 | |
| clarkb | #link https://etherpad.openstack.org/2019-denver-ptg-infra-planning | 19:53 |
| clarkb | I think we're moving at a relatively quick pace on a lot of items so it is hard to know exactly what we'll want to work on at the PTG | 19:54 |
| clarkb | but there should be plenty of topics when we get there | 19:54 |
| prometheanfire | k | 19:54 |
| corvus | i think we should do some containerization, but i don't think we'll know what until much closer. | 19:55 |
| *** sdake has joined #openstack-meeting | 19:55 | |
| *** bbowen has joined #openstack-meeting | 19:56 | |
| clarkb | One thing I've struggeld with (and maybe this means I need to try hard to use gertty again) is keeping track of outstanding reviews that need to be done for these various efforts. I think we have been reasonably good about posting stacks/topics that need eyeballs but maybe I should look at using storyboard worklists again | 19:57 |
| ianw | this is where i got sidetracked with graphite; it's probably better to just do the easy upgrade to xenial, then tackle containerisation separately? | 19:57 |
| clarkb | ianw: I think that is where we've ended up on a couple other upgrades. | 19:58 |
| corvus | clarkb: i try to make sure the topic is set on opendev-gerrit changes | 19:58 |
| *** sdake has quit IRC | 19:58 | |
| corvus | it's not perfect | 19:58 |
| * corvus changes some topics | 19:58 | |
| clarkb | corvus: ya maybe what I really need is a gerrit dashboard to collect a number of those topics for me | 19:58 |
| clarkb | (I'm mostly just brainstorming out loud about this) | 19:58 |
| ianw | clarkb: gerrit-dash-creator might help too, certainly a shared one that highlights current important topics | 19:58 |
| corvus | well, we're supposed to have that in the specs page | 19:59 |
| corvus | http://specs.openstack.org/openstack-infra/infra-specs/ | 19:59 |
| clarkb | corvus: ya, you can fine tune it a bit more with a dashboard though | 19:59 |
| corvus | that url seems broken | 19:59 |
| corvus | https://review.openstack.org/#/q/status:open+AND+(topic:storyboard-migration+OR+topic:opendev-gerrit) | 19:59 |
| *** thgcorrea has quit IRC | 19:59 | |
| clarkb | like show all the changes with one +2 and no -1's to flush them quickly etc | 19:59 |
| fungi | sphinx's url highlighting may simply be broken | 19:59 |
| ianw | oh right, yeah we could do a little fancier one that filters out the different votes | 20:00 |
| fungi | looks like it doesn't include the trailing ) in the hyperlink and instead shows it as text | 20:00 |
| corvus | sure, but we don't need that for priority efforts | 20:00 |
| clarkb | corvus: that is true | 20:00 |
| corvus | there are never more than a handful of those | 20:00 |
| clarkb | and we are at time | 20:00 |
| fungi | thanks clarkb! | 20:00 |
| clarkb | Thank you everyone. I'll send you back to your regularly scheduled day | 20:00 |
| corvus | so what i'd love is if we kept up making sure the topics are set right | 20:00 |
| clarkb | #endmeeting | 20:00 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 20:00 | |
| openstack | Meeting ended Tue Feb 19 20:00:40 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 20:00 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/infra/2019/infra.2019-02-19-19.01.html | 20:00 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/infra/2019/infra.2019-02-19-19.01.txt | 20:00 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/infra/2019/infra.2019-02-19-19.01.log.html | 20:00 |
| corvus | then we can use that link and simple dashboards/queries etc | 20:00 |
| clarkb | corvus: ++ I can clean that up today since i brought it up | 20:00 |
| *** prometheanfire has left #openstack-meeting | 20:01 | |
| *** Shrews has left #openstack-meeting | 20:01 | |
| *** bbowen has quit IRC | 20:02 | |
| *** b1airo has joined #openstack-meeting | 20:22 | |
| *** efried has quit IRC | 20:29 | |
| *** efried has joined #openstack-meeting | 20:29 | |
| *** tssurya has quit IRC | 20:32 | |
| *** ttsiouts has joined #openstack-meeting | 20:36 | |
| *** awaugama has quit IRC | 20:36 | |
| *** e0ne has joined #openstack-meeting | 20:37 | |
| *** Vadmacs has quit IRC | 20:38 | |
| *** markvoelker has joined #openstack-meeting | 20:41 | |
| *** erlon has quit IRC | 20:55 | |
| *** igordc has joined #openstack-meeting | 20:58 | |
| *** martial has joined #openstack-meeting | 21:00 | |
| martial | Hello | 21:00 |
| martial | (On my phone for now) | 21:00 |
| *** ttsiouts has quit IRC | 21:00 | |
| *** ttsiouts has joined #openstack-meeting | 21:01 | |
| *** janders has joined #openstack-meeting | 21:01 | |
| b1airo | morning martial | 21:01 |
| b1airo | #startmeeting Scientific-SIG | 21:02 |
| openstack | Meeting started Tue Feb 19 21:02:11 2019 UTC and is due to finish in 60 minutes. The chair is b1airo. Information about MeetBot at http://wiki.debian.org/MeetBot. | 21:02 |
| janders | good morning, good evening All | 21:02 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 21:02 |
| *** openstack changes topic to " (Meeting topic: Scientific-SIG)" | 21:02 | |
| openstack | The meeting name has been set to 'scientific_sig' | 21:02 |
| b1airo | #chair martial | 21:02 |
| openstack | Current chairs: b1airo martial | 21:02 |
| b1airo | hi janders | 21:02 |
| janders | I just had a really interesting chat about baremetal cloud & networking | 21:03 |
| janders | would you be interested in spendning some time on this? | 21:03 |
| b1airo | oh yeah! who with? | 21:03 |
| janders | RHAT, Mellanox and Bright Computing | 21:03 |
| *** eharney has quit IRC | 21:03 | |
| b1airo | do tell | 21:03 |
| janders | have you guys played much with pxe booting baremetals from neutron tenant networks (as opposed to the nominated provisioning network)? | 21:04 |
| b1airo | no, not at all, but it's clearly one of the key issues in multi-tenant bare-metal... | 21:05 |
| janders | with Bright, we're trying to make Bright Cluster Manager a Cloud-native app | 21:05 |
| *** ttsiouts has quit IRC | 21:05 | |
| janders | it might seem out of the blue but I do see a ton of benefits of this approach and few disadvantages (other than the initial work to make it work) | 21:05 |
| b1airo | right, that's a very sensible approach for them | 21:05 |
| janders | now - they're all about pxe booting (as you probably know - you're a Bright user right?) | 21:06 |
| b1airo | presumably they'd make a new agent or something that sat inside each tenant | 21:06 |
| *** gt1437 has joined #openstack-meeting | 21:06 | |
| janders | so you need to spin up frontend and backend networks. You put BCM-cloud on both. Then you pxeboot the compute nodes off BCM's backend | 21:06 |
| b1airo | NeSI has Bright (BCM and BOS) so I'm picking some stuff up, but I'm not hands on in the ops stuff here | 21:07 |
| janders | same as me - big bright user but we have bright gurus in the team so I know what it does but log in maybe once a month | 21:07 |
| *** raildo_ has quit IRC | 21:07 | |
| *** njohnston_ has quit IRC | 21:08 | |
| janders | on Ethernet - it's all easy and just works. Networking-ansible is plumbing VLANs, there's no NIC side config | 21:08 |
| janders | on IB though provisioning_network pkey needs to be pre-set in the HCA FW | 21:08 |
| *** njohnston_ has joined #openstack-meeting | 21:08 | |
| janders | after that's done it's rock-solid but one negative side effect is you can't pxeboot off tenant networks. We tried some clever hacks with custom compiled ipxe but nah. The silicon will drop any packets that do not match the pre-set pkey | 21:09 |
| janders | looks like dead end | 21:09 |
| janders | so - back to the drawing board | 21:09 |
| janders | the kit I've got does have ethernet ports, they are just shut off cause the switching has no SDN | 21:09 |
| janders | it does have ssh though hence... networking-ansible! | 21:10 |
| b1airo | hmm, until MLNX creates a new firmware with some pre-shared trusted key or some such | 21:10 |
| janders | I was told it can be made work in parallel to Mellanox SDN, both can be configured at the same time just for different physnets | 21:10 |
| janders | I think this can be made work (pending networking-mellanox testing on these switches) | 21:10 |
| b1airo | which kit do you have? | 21:11 |
| janders | M1000es from Dell | 21:11 |
| janders | (I have many different platforms but that's the one I got in large numbers for OpenStack to start with) | 21:11 |
| janders | here I wanted to ask you what you think about all this and what is your experience with networking-ansible | 21:11 |
| janders | I'll VPN into work to get the exact switch details (RHAT are asking too) bear with me if I drop out | 21:12 |
| janders | ok.. still here? | 21:13 |
| *** yamamoto has joined #openstack-meeting | 21:13 | |
| *** markvoelker has quit IRC | 21:13 | |
| b1airo | haven't used networking-ansible myself yet, though it was on the radar when looking at trying to insert an Ironic Undercloud into things at Monash, where our networks are all 100G Mellanox Spectrum running Cumulus | 21:14 |
| janders | Force10 MXL Blade | 21:16 |
| janders | that's my eth platform | 21:16 |
| janders | on that - what are your thoughts on NEO+Mellanox ethernet vs networking-ansible+Mellanox ethernet? | 21:17 |
| *** yamamoto has quit IRC | 21:18 | |
| janders | I bet there won't be feature parity yet but conceptually it's quite interesting | 21:18 |
| *** ralonsoh has quit IRC | 21:19 | |
| janders | very different approach | 21:19 |
| janders | I suppose the same question applies to Juniper SDNs etc | 21:19 |
| b1airo | we first looked at NEO over 2 years ago, it wasn't very polished or nice from a UI perspective yet, so was difficult to actually realise the potential value add. i had been promising the MLNX folks we'd try again for at least 6 months before I left Monash - no doubt way back in the pile for them now | 21:19 |
| b1airo | we spent quite a lot of effort moving to Cumulus + Ansible for basic network management and I was/am very keen to see that progress to full automation of the network as a service | 21:21 |
| *** whoami-rajat has quit IRC | 21:21 | |
| gt1437 | We will get there | 21:21 |
| janders | yeah this tenant-pxeserver on baremetal/IB challenge might actually lead to interesting infrastructure changes | 21:22 |
| janders | on my side | 21:23 |
| janders | running SDN and networking-ansible side-by-side does more complexity but I think more capability going forward | 21:23 |
| b1airo | aiming for bare-metal multi-tenant networking with Mellanox RoCE capable gear - it seemed like we would need to try Cumulus + Neutron native (networking-ansible would be a possible alternative there) and also NEO + Neutron. interesting thing there is that value of Cumulus suddenly becomes questionable if you're managing the BAU network operations via Neutron, given e.g. NEO can do all the day0 switch provisioning | 21:24 |
| b1airo | stuff | 21:24 |
| janders | OpenStack internal traffic (other than storage) couldn't care less about IB - maybe it's better if these are keep separate so there is even more IB bandwidth for the workloads (and less jitter) | 21:24 |
| janders | with Cumulus, do they have the same SDN-controller-centric architecture as NEO? | 21:25 |
| janders | Do you configure Cumulus URL in Neutron SDN section, and the flow is Neutron>Cumulus>Switching gear? | 21:25 |
| b1airo | yeah our basic cloud hpc networking architecture used dualport eth with active-backup bonding for host stuff (openstack, management, etc) and sriov into tenant machines on top of the default backup interface | 21:26 |
| janders | https://docs.cumulusnetworks.com/display/CL35/OpenStack+Neutron+ML2+and+Cumulus+Linux | 21:27 |
| janders | I see Cumulus has HTTP service listening | 21:27 |
| janders | However, in Neutron, individual switches are configured | 21:27 |
| b1airo | no central controller that i'm aware of for Cumulus, the driver has to talk to all the devices | 21:27 |
| janders | ok.. so what's the HTTP with REST API service for? Do switches talk to it? | 21:27 |
| b1airo | but yes, all restful apis etc, no ssh over parallel python ;-) | 21:28 |
| *** e0ne has quit IRC | 21:28 | |
| janders | or is the REST API running on each switch? | 21:30 |
| b1airo | the rest api is on the switches themselves | 21:30 |
| b1airo | ya | 21:30 |
| janders | ok! | 21:30 |
| janders | right... | 21:30 |
| janders | nice - no two sources of truth (Neutron DB and then SDN DB) | 21:30 |
| janders | however troubleshooting will be fun (log aggregation will help to a degree) | 21:31 |
| b1airo | that's true | 21:31 |
| janders | with centralised SDN my experience is - they work 99% - but when they stop, life sucks | 21:31 |
| b1airo | exactly, one of the things we were trying to wrangle with - monitoring, visibility, etc etc | 21:31 |
| janders | manual syncing of the DBs would be a nightmare and a high risk operation | 21:32 |
| b1airo | ok, i'm at a conference and talk i'm in (Barbara Chapman talking about OpenMP for Exascale) just finished | 21:32 |
| b1airo | so i'm going to have to scoot pretty soon | 21:33 |
| martial | what are the other topics to cover? | 21:33 |
| b1airo | better touch on the ISC19 BOF in case anyone is lurking | 21:33 |
| janders | what conference? :) | 21:33 |
| b1airo | ... | 21:33 |
| b1airo | eResearchNZ :-) | 21:33 |
| janders | do you guys know if we're still on track for speaker notifications for Denver? | 21:34 |
| janders | it would be great to book travel by the end of this week if possible | 21:34 |
| b1airo | had a plenary before that on the Integrated Data Infrastructure at Stats NZ - enviable resource coming from the Aus context, clearly digital health in australia didn't talk to them before launching my health record... | 21:35 |
| janders | haha :) | 21:35 |
| martial | janders: I heard they are runing a day late | 21:35 |
| b1airo | ah yes, the track chairing process got extended till Sunday just gone as there was a slight booboo that locked everyone out of the tool | 21:35 |
| janders | right.. so no notifications this week? | 21:36 |
| martial | I heard the 20th instead of the 19th | 21:36 |
| b1airo | but we've done our job now, so i guess foundation staff will be doing the final checks and balances before locking in | 21:36 |
| janders | ah ok | 21:37 |
| janders | I will keep this in mind while making plans | 21:37 |
| janders | thanks guys - this is very helpful | 21:37 |
| b1airo | sounds possible martial , let me check my email | 21:37 |
| b1airo | janders: just quietly, you can confidently make plans | 21:37 |
| *** _pewp_ has quit IRC | 21:37 | |
| *** _pewp_ has joined #openstack-meeting | 21:38 | |
| janders | :) :) :) thank you | 21:39 |
| janders | b1airo: not sure if this is relevant to you but Qantas double status credits are on | 21:41 |
| janders | or are you with AirNZ these days? | 21:41 |
| b1airo | yeah kiwi-air all the way ;-) | 21:42 |
| janders | Nice. Good airline and nice aircraft. | 21:43 |
| b1airo | ok i'd better run. martial , shall we finish up now or do you want to kick on and close things out? | 21:43 |
| janders | I don't have anything else. | 21:43 |
| martial | I can close | 21:43 |
| b1airo | agreed janders , they can be a bit too progressive/cringe-worthy with their safety videos at times though | 21:43 |
| martial | although not sure if we have much to add | 21:43 |
| martial | #topic ISC BoF | 21:44 |
| *** openstack changes topic to "ISC BoF (Meeting topic: Scientific-SIG)" | 21:44 | |
| b1airo | ok, i'm off to the coffee cart. o/ | 21:44 |
| janders | safe travels mate! | 21:44 |
| martial | like @b1airo mentioned, we are proposing a OpenStack HPC panel for ISC 19 (Frankfurt, June) | 21:44 |
| janders | I haven't managed to get anything out of my colleagues who might be going to ISC so unfortunately can't contribute | 21:44 |
| gt1437 | I put in a talk in vHPC, so I might be there for ISC | 21:45 |
| martial | gt1437: cool :) | 21:46 |
| janders | nice! | 21:46 |
| gt1437 | and just realised the one after Denver is Shanghai, that's cool | 21:47 |
| *** artom has quit IRC | 21:47 | |
| janders | oh wow | 21:47 |
| janders | thanks, I didn't know (Shanghai was in my top 3 guesses though) | 21:48 |
| gt1437 | yeah thought it was going to be beijing, anyway, still good | 21:49 |
| *** hongbin has quit IRC | 21:49 | |
| janders | any goss about the exact dates? | 21:49 |
| janders | (the website states just "nov 2019") | 21:50 |
| gt1437 | no idea | 21:50 |
| martial | no dates yet | 21:51 |
| *** mkarray has joined #openstack-meeting | 21:52 | |
| gt1437 | I'm off to meetings..ciao | 21:54 |
| *** gt1437 has quit IRC | 21:55 | |
| janders | have a good day | 21:55 |
| martial | have a good day all, | 21:55 |
| martial | anything else? | 21:55 |
| martial | otherwise just a reminder that we will have the Lighting talk at the summit :) | 21:55 |
| janders | I think we're good | 21:56 |
| janders | thank you Martial | 21:56 |
| martial | #endmeeting | 21:57 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 21:57 | |
| openstack | Meeting ended Tue Feb 19 21:57:10 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 21:57 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/scientific_sig/2019/scientific_sig.2019-02-19-21.02.html | 21:57 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/scientific_sig/2019/scientific_sig.2019-02-19-21.02.txt | 21:57 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/scientific_sig/2019/scientific_sig.2019-02-19-21.02.log.html | 21:57 |
| *** janders has quit IRC | 21:57 | |
| *** cloudrancher has joined #openstack-meeting | 21:57 | |
| *** sdake has joined #openstack-meeting | 21:58 | |
| *** enriquetaso has quit IRC | 22:01 | |
| *** ttsiouts has joined #openstack-meeting | 22:07 | |
| *** carlos_silva has quit IRC | 22:08 | |
| *** sdake has quit IRC | 22:09 | |
| *** markvoelker has joined #openstack-meeting | 22:10 | |
| *** ttsiouts has quit IRC | 22:12 | |
| *** rcernin has joined #openstack-meeting | 22:26 | |
| *** aagate has quit IRC | 22:27 | |
| *** cloudrancher has quit IRC | 22:39 | |
| *** markvoelker has quit IRC | 22:44 | |
| *** cloudrancher has joined #openstack-meeting | 22:47 | |
| *** cloudrancher has quit IRC | 22:51 | |
| *** munimeha1 has quit IRC | 23:02 | |
| *** eharney has joined #openstack-meeting | 23:26 | |
| *** mattw4 has quit IRC | 23:29 | |
| *** ociuhandu has quit IRC | 23:30 | |
| *** ociuhandu_ has joined #openstack-meeting | 23:30 | |
| *** imacdonn has joined #openstack-meeting | 23:32 | |
| *** imacdonn_ has joined #openstack-meeting | 23:32 | |
| *** imacdonn_ has quit IRC | 23:32 | |
| *** liuyulong has joined #openstack-meeting | 23:40 | |
| *** markvoelker has joined #openstack-meeting | 23:41 | |
| *** mriedem has quit IRC | 23:41 | |
| *** yaawang has quit IRC | 23:53 | |
| *** mattw4 has joined #openstack-meeting | 23:53 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!