15:00:11 #startmeeting kolla 15:00:11 Meeting started Wed Oct 27 15:00:11 2021 UTC and is due to finish in 60 minutes. The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:00:11 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:00:11 The meeting name has been set to 'kolla' 15:00:16 mnasiadka: where do you keep that ping list? I'd like to add myself 15:00:18 #topic rollcall 15:00:21 o/ 15:00:23 \o 15:00:30 frickler: it's currently at https://wiki.openstack.org/wiki/Meetings/Kolla 15:00:41 o/ 15:00:42 mnasiadka: thx 15:00:53 [°][_] 15:02:53 o/ 15:03:05 #topic agenda 15:03:06 My management network (provider based vswitch), which is used to also tunnel the (non dvr) ovn network, has an mtu of 1400. When adding the provider network, i used a mtu of 1400 - but that seems not to be enough. Do i need to set something with kolla in general? I have found https://docs.openstack.org/neutron/queens/admin/config-mtu.html but not sure that applies here 15:03:18 mnasiadka: need to move it to the whiteboard (the official ping list) 15:03:28 em_: we have a weekly meeting now - please wait until it ends (around 1hr) 15:03:31 yoctozepto: yup 15:03:40 (oh sorry, was not aware about the irc based meeting, will shut up. Excuse me) 15:03:41 * Roll-call 15:03:41 * Agenda 15:03:41 * Announcements 15:03:41 * Review action items from the last meeting 15:03:41 * CI status 15:03:43 * Release tasks 15:03:43 * Yoga cycle planning 15:03:45 * Open discussion 15:04:07 #topic Announcements 15:04:24 I have none - anyone anything? 15:04:38 RC!? 15:04:40 RC1? 15:04:52 PTG? 15:04:55 RC!!!!!1111oneoneoneeleven 15:04:55 wu.chunyang proposed openstack/kolla-ansible master: Fix wrong opts in cyborg.conf https://review.opendev.org/c/openstack/kolla-ansible/+/815672 15:05:10 Ah, right - RC1 for Kolla, Kolla-Ansible and Kayobe has been cut. 15:05:19 #info 15:05:26 #info RC1 for Kolla, Kolla-Ansible and Kayobe has been cut. 15:06:19 Ok then, let's move on I guess - unless somebody else wants to announce anything? 15:07:03 #topic Review action items from the last meeting 15:07:06 Seems there were none. 15:07:22 #topic CI Status 15:07:25 Are we green? 15:08:32 Seems we are - based on the whiteboard. 15:09:16 #topic Release tasks 15:09:36 So, do we have a list of blockers for doing RC2? 15:09:48 I think all MariaDB related patches have been merged? 15:10:43 yoctozepto: ? 15:11:19 yeah, I think so 15:11:30 any release tasks still to do? 15:11:45 centos-openstack-release done? 15:12:39 the gerrit dashboards seem a bit broken 15:12:44 no project filtering 15:12:58 mayhaps we need also https://review.opendev.org/c/openstack/kolla-ansible/+/814276 15:12:59 for mariadb 15:13:57 mgoddard: for dashboards you need to remove the /#/ from the path 15:14:01 Uwe Grawert proposed openstack/kolla-ansible master: [Grafana] Add unified alerting and smtp options https://review.opendev.org/c/openstack/kolla-ansible/+/815694 15:14:03 or reload 15:14:39 see https://gerrit-review.googlesource.com/c/gerrit/+/321535 15:14:39 frickler: thanks 15:14:47 yoctozepto: c-r-o-xena exists 15:15:01 and we use it 15:15:23 ok 15:15:30 so only that mariadb patch 15:15:37 but I'm not sure what the impact is 15:15:44 perhaps it garbles the config 15:15:47 around gerrit dashboards - I see master branch in stable branch backports section of Kolla dashboard, so maybe we need to revisit those and check what's going on 15:16:38 About CI status, we are amber on wallaby for Kayobe (just updated the booard) 15:16:47 It's caused by frequent disk full issues 15:17:02 The wallaby images must be a bit bigger than other releases 15:17:25 We have a proposed workaround which is to disable heat from CI upgrade jobs 15:18:22 Ok, I think one of the changes to make it better for Wallaby is merging/merged today 15:18:41 yoctozepto: I don't see a bug report in that patch, so can't really tell we should wait to get it reviewed and merged. 15:18:58 So, should we post RC2 for Kolla/Kolla-Ansible/Kayobe? 15:20:24 https://review.opendev.org/c/openstack/kolla-ansible/+/814942 15:20:35 mnasiadka: it's in the reno 15:20:58 mgoddard: good catch 15:21:04 yoctozepto: what about closes-bug? 15:21:19 mnasiadka: yeah, you can comment that on it 15:21:31 but the bug report is not satisfactory tbh 15:21:48 it's like writing "it doesn't work." 15:22:47 mnasiadka: let https://review.opendev.org/c/openstack/kolla/+/815440 merge and then RC2? 15:23:22 ok, so two changes are +w and we need to wait for them to merge 15:23:29 if we're going to merge this revert then let's do it before release https://review.opendev.org/c/openstack/kolla-ansible/+/814949 15:23:52 (discuss) 15:24:50 Merged openstack/kayobe stable/wallaby: Set proxy option in early dnf invocation https://review.opendev.org/c/openstack/kayobe/+/814658 15:24:50 mgoddard: I think you owe us some more description and reason ;-) 15:25:28 I would say the same about the original patch :) 15:25:30 I added enable_host_ntp and cinder-volume fix as RC2 blockers in the whiteboard (L297) 15:25:55 Well, the original patch stated everything works, and CI didn't explode - so I'd like to know what does it break ;-) 15:26:46 mgoddard: like mnasiadka said - it was well described :-) 15:27:09 If we are issuing RC2 for Kayobe we may want to merge https://review.opendev.org/c/openstack/kayobe/+/812687 in xena 15:27:34 I don't mind reverting if it really breaks something; but then again we should probably rewrite it to work differently as it does not make sense to use it with ovs native firewall and ovn 15:28:17 priteau: especially that the comment says "since Xena"... ;-) 15:28:35 I don't see where neutron is loading that module 15:29:18 I do see neutron will print a warning if it is not loaded 15:30:52 I think br_netfilter can be loaded by docker 15:31:00 if it uses iptables 15:31:09 we default to disable iptables now I think 15:31:14 I can try to justify, but not in real time 15:31:31 mgoddard: did it break somewhere IRL? 15:31:43 not yet 15:32:03 but that means we could have non-working SGs? 15:32:05 then try to make it break :-) 15:32:09 I tried and it works 15:32:18 hence did not bother to improve, just removed 15:32:31 it jumped out as one of those patches that could bite us 15:33:04 well, at least we know how to fix it quickly 15:33:09 and my gut has often been right on those in the past but I let them slide then suffer latet 15:33:24 and we fix when it bites 15:33:25 So why change a tradition? 15:33:32 anyway 15:33:35 guys , is this visible in CI ? 15:33:35 2021-10-27 15:11:40.879 25 ERROR octavia.api.drivers.driver_factory [-] Unable to load provider driver ovn due to: No module named 'ovn_octavia_provider.common': ModuleNotFoundError: No module named 'ovn_octavia_provider.common' 15:33:36 oh well, that's it for the scientific method :D 15:33:47 kevko: guy, we are in a meeting 15:33:49 :-) 15:33:57 oh, sorry :D 15:34:22 I just have a freshly deployed kolla using xena branches, br_netfilter is loaded on compute hosts 15:34:40 I can try and see what enabled it 15:34:42 do you know how/when it got loaded? 15:35:28 I don't know yet 15:35:37 [Wed Oct 27 14:11:49 2021] Bridge firewalling registered 15:36:08 now we know when 15:36:18 your mileage may vary though 15:36:43 But in neutron-openvswitch-agent logs: 15:36:54 2021-10-27 14:08:13.127 7 WARNING neutron.agent.linux.iptables_firewall [req-60c2a81f-edaa-4c34-a2ae-37017aeac72f - - - - -] Kernel module br_netfilter is not loaded. 15:36:54 2021-10-27 14:08:13.128 7 WARNING neutron.agent.linux.iptables_firewall [req-60c2a81f-edaa-4c34-a2ae-37017aeac72f - - - - -] Please ensure that netfilter options for bridge are enabled to provide working security groups. 15:37:26 well, so after the warning something did it ;-) 15:37:49 this should be done by something on host before containers start 15:38:12 otherwise we would need to have hostos == containeros 15:38:16 Systems that don't override default settings for those knobs would work 15:38:19 fine except for this exception in the log file and agent resync. This is 15:38:21 because the first attempt to add a iptables rule using 'physdev' module 15:38:23 (-m physdev) will trigger the kernel module loading. In theory, we could 15:38:25 silently swallow missing knobs, and still operate correctly. But on 15:38:27 second thought, it's quite fragile to rely on that implicit module 15:38:29 loading. In the case where we can't detect whether firewall is enabled, 15:38:31 it's better to fail than hope for the best. 15:38:37 neutron commit e83a44b96a8e3cd81b7cc684ac90486b283a3507 15:39:38 which I linked to in the br_netfilter patch 2 weeks ago 15:39:42 It was loaded when I launched a VM 15:40:06 created | 2021-10-27T14:11:45Z 15:40:18 Loaded 4 seconds late 15:40:20 later 15:40:49 we should move on 15:41:10 Yup, what's the plan? Leave it as is since it seems to work? 15:42:56 It works but it produces WARNING messages in logs, that's not nice 15:43:18 What do we gain from not loading it? 15:44:01 priteau: this is for ovs native firewall and ovn to not have this oddity 15:44:12 but we can revert, it does not hurt 15:44:21 I can make a better version of it 15:44:30 well, we can tweak it to at least not enable when neutron_plugin_agent=ovn 15:44:31 at some point ;p 15:45:26 ok, let's revert for now and post a tweak 15:45:34 yeah, that makes sense 15:46:09 #agreed to revert https://review.opendev.org/c/openstack/kolla-ansible/+/814949 and post a better version (to skip loading when not required e.g. ovn) 15:46:21 #topic Yoga cycle planning 15:46:30 sorry, laptop died 15:47:00 I have a draft summary mail I'm going to send to openstack-discuss after the PTG and then will populate the Whiteboard with priorities/tasks 15:47:57 I'm also planning to use Kolla Klub mailing list to get feedback around our single-distro plans for Kolla 15:47:59 cool 15:48:15 Anything else that needs to be done? 15:48:44 Radosław Piliszek proposed openstack/kolla-ansible master: Revert "Do not load br_netfilter" https://review.opendev.org/c/openstack/kolla-ansible/+/814949 15:48:44 should we deprecate CentOS now in Xena to be able to drop it with Yoga or we deprecate in Yoga to drop in Zeus? 15:49:04 I think we should wait until yoga 15:49:16 mgoddard, mnasiadka: I improved the error message there 15:49:19 still a lot of uncertainty 15:49:21 sorry, the revert reason 15:49:26 thanks yoctozepto 15:49:27 * yoctozepto tired lol 15:49:31 mgoddard: yw 15:49:46 Well, we need to make sure we're not going cs9 15:49:48 yoctozepto: tired like old lamb? 15:49:54 hrw: yup 15:50:17 Does that mean we need to drop centos-binary in Yoga? 15:50:26 mnasiadka: is RDO goes cs9 only for yoga then no binary in yoga for us 15:50:43 deprecate & drop in one cycle 15:51:19 Ok, so we need to carry out the plan to deprecate all binary, and then if RDO goes cs9 (which they most probably will) in Yoga - we will drop that with an appropriate message? 15:51:49 looks like 15:51:58 Ok, at least that's clear :) 15:52:16 ok, makes sense 15:52:43 I'll phrase it like this in the summary mail and in the whiteboard items, so it's clear for everybody. 15:53:15 And then we also deprecate CentOS as whole in Yoga and drop it in Zeus, right? 15:53:34 But still we need to rework centos-source to not pull in anything from RDO packages, so centos-source on cs8 works... 15:54:12 I thought we said A for dropping? 15:54:34 mgoddard: I hoped for A to be Debian/source only iirc 15:55:25 as "Y drops binary, Z drops distros" but that can be one release too short 15:55:37 I remember we wanted a clean slate from A, but maybe we're just rushing it. 15:56:30 I think we need a period where we're deploying the chosen containers on all host distros by default 15:56:49 ideally have that released and in real world use for a while before dropping 15:57:17 of course this is going to explode the test matrix 15:57:41 only debian on all 15:57:46 others dedicated 15:57:53 the question was about cs9 15:57:59 ok, double the test matrix 15:58:01 as we can't rely on rdo then ;d 15:58:07 yeah, double 15:58:50 on the libvirt question, I found a bit more context on the issue I mentioned 15:58:51 I guess in practice we don't need to verify anything other than qemu/kvm 15:58:57 infra from xena times would probably fine 15:59:00 mgoddard: which one? 15:59:28 we had problems with centos 8.2 libvirt containers on an 8.1 host 15:59:44 qemu-kvm: error: failed to set MSR 0x48e to 0xfff9fffe04006172 15:59:46 qemu-kvm: /builddir/build/BUILD/qemu-4.2.0/target/i386/kvm.c:2695: kvm_buf_set_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs\' failed. 16:00:21 Ok, I think we need to continue the discussion next week (or after the meeting). 16:00:32 Thanks for attending. 16:00:34 #endmeeting