13:00:31 <mnasiadka> #startmeeting kolla
13:00:31 <opendevmeet> Meeting started Wed Aug  9 13:00:31 2023 UTC and is due to finish in 60 minutes.  The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot.
13:00:31 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
13:00:31 <opendevmeet> The meeting name has been set to 'kolla'
13:00:34 <mnasiadka> #topic rollcall
13:00:36 <mnasiadka> o/
13:00:40 <mmalchuk> \o
13:00:44 <jangutter> o/
13:00:56 <mattcrees> o/
13:01:01 <ihalomi> \o
13:01:03 <opendevreview> Michal Arbet proposed openstack/kolla-ansible master: Add support for LetsEncrypt-managed certs  https://review.opendev.org/c/openstack/kolla-ansible/+/741340
13:01:26 <SvenKieske> o/
13:01:47 <SvenKieske> fantastic timing, meeting ended early, right into the next one
13:02:07 <opendevreview> Michal Nasiadka proposed openstack/kolla master: Move to Debian 12 'bookworm'  https://review.opendev.org/c/openstack/kolla/+/886088
13:03:05 <mnasiadka> #topic agenda
13:03:05 <mnasiadka> * CI status
13:03:05 <mnasiadka> * Release tasks
13:03:05 <mnasiadka> * Regular stable releases (first meeting in a month)
13:03:05 <mnasiadka> * Current cycle planning
13:03:06 <mnasiadka> * Additional agenda (from whiteboard)
13:03:06 <mnasiadka> * Open discussion
13:03:09 <mnasiadka> #topic CI status
13:03:30 <mnasiadka> So, kolla and kolla-ansible sound green-ish, magnum jobs are failing due to some designate issue
13:03:42 <mnasiadka> kayobe upgrade jobs still red - fixing in https://review.opendev.org/c/openstack/kolla-ansible/+/890198/46
13:03:53 <mnasiadka> #topic Release tasks
13:04:17 <mnasiadka> It's R-8 week
13:04:43 <mnasiadka> we have https://docs.openstack.org/kolla/latest/contributor/release-management.html#r-8-switch-images-to-current-release
13:05:14 <mnasiadka> Anybody wants to handle switching RDO and UCA to current in-development release?
13:05:54 <opendevreview> Michal Nasiadka proposed openstack/kolla-ansible stable/yoga: Add documentation for migrating from CS8 to RL9  https://review.opendev.org/c/openstack/kolla-ansible/+/884858
13:05:55 <frickler> I wonder whether we still need to do that at all
13:06:37 <mnasiadka> we use some packages from RDO for sure
13:06:46 <mnasiadka> I can handle RDO
13:06:49 <frickler> 2023.2 should work fine with ceph+libvirt from 2023.1
13:07:29 <mnasiadka> that true, but OVN users would be happy to get newer OVN version
13:08:05 <opendevreview> Michal Arbet proposed openstack/kolla-ansible master: Add support for LetsEncrypt-managed certs  https://review.opendev.org/c/openstack/kolla-ansible/+/741340
13:08:06 <frickler> is OVN in UCA? I don't know about RDO
13:08:07 <SvenKieske> I guess I could maybe take a stab at the UCA part, never did it, but the linked change looks easy enough
13:08:12 <mnasiadka> frickler: yes it is
13:08:23 <mnasiadka> ok
13:08:26 <SvenKieske> don't know anything about RDO "delorean"
13:08:46 <mnasiadka> #action mnasiadka handle RDO switch to bobcat
13:08:54 <mnasiadka> #action SvenKieske handle UCA switch to bobcat
13:09:29 <mnasiadka> #topic Regular stable releases (first meeting in a month)
13:09:40 <mnasiadka> We agreed last time to do it after the systemd fix gets backported to 2023.1
13:09:56 <mnasiadka> it's still not merged in master, so let's try again next week
13:10:02 <mnasiadka> #topic Current cycle planning
13:10:07 <mnasiadka> I see good progress on Let's Encrypt
13:10:19 <mnasiadka> ihalomi: sorry, I didn't have time to have a look in Rocky failures on podman
13:10:33 <mnasiadka> ihalomi: any other issues on podman patches? Is there something we could review?
13:10:49 <mnasiadka> (as in merge before we fix the rocky problem)
13:11:25 <ihalomi> mnasiadka: no other issues, this is basically this is the last obstacle to getting +1
13:11:48 <SvenKieske> last time I looked patches looked pretty good, afaik I did a full review, will have another look
13:12:01 <opendevreview> Michal Nasiadka proposed openstack/kolla stable/yoga: ovsdpdk: add libdpdk-dev  https://review.opendev.org/c/openstack/kolla/+/880317
13:12:19 <mnasiadka> ok, I'll have a look too
13:12:34 <mnasiadka> #topic Additional agenda (from whiteboard)
13:13:05 <mnasiadka> octavia jobboard patch seems to get better, I'll have a look at it later
13:13:27 <mnasiadka> debian bookworm support - I'll fix the conflicts in the kolla patch and it seems it's good to merge
13:13:27 <frickler> I added the precheck verification as discussed
13:13:43 <mnasiadka> and then we could have a crack at the kolla-ansible side
13:13:49 <frickler> we need to wait for the haproxy option fixes for bookworm
13:14:03 <mnasiadka> ok, there's a series of patches
13:14:09 <mnasiadka> I've seen a proposal to squash them together
13:14:13 <frickler> I reviewed one patch but didn't track responses
13:14:18 <frickler> yes, that was me
13:14:27 <mnasiadka> but maybe it's just easier to merge them as is when they pass properly?
13:15:04 <mnasiadka> but yes, we need to have a look at pushing those forward, I think the author is not going to work on them promptly enough ;-)
13:15:15 <frickler> I can have a look again, but it seemed too complicated to me
13:15:24 <mnasiadka> mattcrees - rabbitmq enable ha queues by default is ready for review: https://review.opendev.org/c/openstack/kolla-ansible/+/882825
13:16:20 <mattcrees> Basically I've added support in the CI upgrade jobs to migrate the queue types, and have proposed a KA command to reset rabbit state as this is a key part of the process
13:17:10 <mnasiadka> ok then, another thing to review queue ;)
13:17:16 <frickler> ack
13:18:13 <mnasiadka> ok then
13:18:16 <mnasiadka> #topic Open discussion
13:18:19 <SvenKieske> could people with +2 powers take a look at the backports of: https://review.opendev.org/c/openstack/kolla-ansible/+/889189 ? thank you
13:18:25 <mnasiadka> Do we need to discuss the server-status thing again?
13:18:28 <SvenKieske> should be straight forward
13:18:56 <SvenKieske> I _guess_ the current server-status patch now looks fine? :)
13:19:29 <mmalchuk> mnasiadka I can give an update
13:20:05 <mmalchuk> this can't be fixed by 'require ip 127.0.0.1' because of https://opendev.org/openstack/kolla-ansible/src/branch/master/ansible/roles/horizon/templates/horizon.conf.j2#L16
13:20:25 <mmalchuk> there is already 'require local' - https://httpd.apache.org/docs/2.4/mod/mod_authz_host.html
13:20:52 <mmalchuk> so accept - the client address matches 127.0.0.0/8, the client address is ::1, both the client and the server address of the connection are the same
13:20:59 <mmalchuk> last case - is haproxy
13:21:18 <mmalchuk> I've updated change: https://review.opendev.org/c/openstack/kolla-ansible/+/888943/7..8
13:21:28 <mmalchuk> add more info and update release note
13:21:45 <mnasiadka> Well, I'm still not convinced - Rocky/CentOS have mod_status enabled, but not configured - and it does not show up there.
13:22:05 <mmalchuk> I'm ready to add mod_status to Rocky/Centos
13:22:06 <frickler> with no config, the handler isn't used
13:22:06 <mnasiadka> So if we would remove the default configuration for mod_status in Debian/Ubuntu - we would get to the same state?
13:22:10 <mmalchuk> will propose Kolla fix
13:22:50 <mnasiadka> And then on top of this, we could add a feature to configure that properly in kolla-ansible?
13:23:07 <mnasiadka> Or is my thinking wrong?
13:23:21 <mmalchuk> frickler wrong
13:23:24 <SvenKieske> imho your thinking is only "wrong" that it breaks existing users
13:23:33 <mmalchuk> it can't be configured because of haproxy
13:23:56 <mmalchuk> add 'require 127.0.0.1' will completely disable status
13:24:14 <mmalchuk> because we use internal vip for apache listen
13:24:19 <frickler> mmalchuk: I was talking about why the issue isn't seen in rocky
13:24:21 <mnasiadka> SvenKieske: it's not a feature, it's a bug for now :)
13:24:28 <SvenKieske> i.e. removing the default config and adding a role to properly configuring this would require us to backport a completely new role, no? and it must match the old behaviour at least for internal usage
13:24:35 <mnasiadka> an unplanned feature is a bug from my perspective
13:24:37 <mmalchuk> frickler did you see last change? https://review.opendev.org/c/openstack/kolla-ansible/+/888943/7..8
13:24:49 <mmalchuk> and Rocky is not affeced
13:25:02 <mmalchuk> this is in release note
13:25:38 <frickler> yes, I was replying to mnasiadka
13:25:41 <SvenKieske> mnasiadka: that's really not helpful humor here: it's a security bug because it's reachable from potential untrusted networks on deb/ubuntu; it's a feature there because you can monitor apache stuff from internal networks, I don't want to break those users, do you?
13:26:08 <mnasiadka> SvenKieske: it's a feature only on Debian/Ubuntu, and it's a Debian/Ubuntu package and default config feature, not a kolla-ansible feature :)
13:26:39 <SvenKieske> or we backport completely new roles to old releases to fix this, could also work, but imho backports should be minimal, which mmalchuks change really is, there's not much to argue there from the maintenance perspective imho.
13:26:45 <mmalchuk> mnasiadka I will add the 'feature' to the Rocky/Centos but this is not related to bug
13:27:09 <mnasiadka> It's not a feature.
13:27:36 <mmalchuk> we will got the same behavior
13:27:45 <mmalchuk> on all installations
13:28:16 <SvenKieske> mnasiadka: do you really want to argue that we only promise to not break users if we built a feature in k-a ourselves? because than I'm out of this discussion, this is ridiculous
13:28:20 <mnasiadka> It's like with this ansible-core breakage, because we used handlers with import_tasks - bug was rejected because nobody ever planned this usage.
13:28:20 <mmalchuk> Centos/Redhat users can use monitoring localay, but will not be affected by bug we close now
13:28:55 <SvenKieske> server-status is a clear apache feature, it's not a bug per se, you just need to care how you enable that
13:29:03 <mnasiadka> We don't enable tht.
13:29:06 <mnasiadka> *that
13:29:18 <mnasiadka> It's enabled by default
13:29:25 <mnasiadka> only on 50% of the distributions we support
13:29:27 <mmalchuk> we enable that in horizon)
13:29:29 <SvenKieske> yeah sure, but don't break the default then
13:29:38 <mnasiadka> From my perspective it's a Debian/Ubuntu packaging bug
13:29:39 <mmalchuk> https://opendev.org/openstack/kolla-ansible/src/branch/master/ansible/roles/horizon/templates/horizon.conf.j2#L30
13:29:39 <frickler> yes, and it is safe by default, adding haproxy in front is what breaks things
13:29:46 <SvenKieske> "only on 50%"..I'm out
13:29:55 <frickler> and adding haproxy is what kolla does
13:30:02 <SvenKieske> your not having a honests helpful discussion imho
13:30:05 <mmalchuk> without this server status not available even without haproxy
13:30:29 <mnasiadka> do we really need to override Location / in horizon.conf?
13:30:48 <frickler> mnasiadka: that's actually a second issue
13:30:58 <mnasiadka> frickler: which could fix the first issue
13:31:08 <mmalchuk> mnasiadka Ubuntu/Debian enable more modules than Centos/Rocky not only mod_status
13:31:14 <frickler> mnasiadka: no
13:31:14 <mnasiadka> second thing - I'm all in deny'ing /server-status in haproxy on ALL frontends
13:31:28 <mnasiadka> but the patch denies it only on some of the services
13:31:42 <mnasiadka> and doesn't include a CI task that will check if we don't encounter the same bug in future
13:31:56 <mnasiadka> are we going to iterate on this bug every time it shows up?
13:32:17 <SvenKieske> I honestly don't care _how_ we do this. I care about two things: a) fix the security bug (I don't care whoever introduced it, upstream, we, little green men from mars) b) preserve legitimate use cases like internal monitoring without breaking them
13:32:34 <mmalchuk> on some - because of apache2 only services need this
13:32:35 <frickler> ok, but then we agree on doing the filtering as proposed in haproxy?
13:32:43 <SvenKieske> I'm baffled that needs to be argued, like, at all.
13:33:07 <mmalchuk> we didnt use external vip on CI - this is another issue
13:33:18 <mnasiadka> Seems we have no other option, that to do the filtering as proposed in haproxy - question what about users that use external load balancers, should we update the docs for them?
13:33:56 <SvenKieske> "question what about users that use external load balancers" <- can you elaborate what you mean by that? I don't understand
13:33:59 <frickler> if the loadbalancer isn't on the same host as apache, only the fix in the horizon location statement is needed
13:34:05 <mmalchuk> imho there are no such users
13:34:21 <mnasiadka> mmalchuk: of course there are
13:34:24 <frickler> there are, but not affected
13:34:40 <mnasiadka> thanks for clarification instead of denying the problem ;-)
13:34:44 <frickler> those won't match "Require local"
13:34:54 <mnasiadka> right
13:34:56 <mmalchuk> server-status on cinder api ? for example? kidding
13:35:13 <mnasiadka> but now, with the current shape of the patch - we enable mod_status on CentOS and Rocky for Horizon, right?
13:35:18 <SvenKieske> you mean people who don't use haproxy but use a different LB? well they need to do the filtering themselves I guess? a doc patch would be nice, but not strictly necessary imho
13:35:30 <mmalchuk> right
13:35:36 <mmalchuk> i will propose patch
13:36:02 <SvenKieske> they need to provide all balancing config themselves either way, so I hope they know what they are doing I guess.
13:36:05 <mmalchuk> we just add handler to config in kolla
13:36:34 <mmalchuk> should we backport this 'feature' ?
13:36:39 <SvenKieske> it's really funny that we are back to the original solution again  (are we? I have lost track..)
13:37:11 <mnasiadka> mmalchuk: I'm asking, because https://review.opendev.org/c/openstack/kolla-ansible/+/888943/8/ansible/roles/horizon/templates/horizon.conf.j2#34 will change behaviour on CentOS/Rocky
13:37:17 <SvenKieske> well if it breaks existing users without the backport I'd argue we need a backport
13:37:25 <mnasiadka> if it's planned - then include that in the reno, but for backporting - we shouldn't probably doing that
13:37:37 <mmalchuk> frickler as said 'not affected'
13:37:55 <mmalchuk> there is 'require local' already in apache config from package
13:38:01 <mnasiadka> not in CentOS/Rocky
13:38:08 <mnasiadka> and here you add it for ALL distributions
13:38:12 <frickler> mnasiadka: this will not enable server-status
13:38:17 <mnasiadka> are you sure?
13:38:19 <mmalchuk> not. but there is no handler there!
13:38:20 <frickler> yes
13:38:23 <mmalchuk> sorry)
13:38:28 <frickler> exactly, no handler
13:38:45 <mnasiadka> ah ok, SetHandler is missing
13:38:49 <mmalchuk> so centos/redhat not affected at all
13:38:52 <mmalchuk> even with fix
13:39:07 <SvenKieske> yeah this only requires a local ip to get to the route, but if apache has no route to that path everything is fine, I didn't check rocky myself
13:39:33 <mmalchuk> but wen I propose change to Kolla container - there will be work, so we need merge first fix
13:39:38 <mmalchuk> when
13:39:54 <SvenKieske> I wish the "enable this on rocky" would be a separate discussion, I have no eggs in the basket regarding rocky :)
13:40:22 <frickler> I would skip that part unless rocky users ask for it
13:41:18 <mmalchuk> I will propose, lets people decide
13:41:35 <mmalchuk> we will have same behaviour in all installations
13:42:00 <mnasiadka> that's another discussion
13:42:05 <mmalchuk> yep
13:42:10 <SvenKieske> yeah from a strict technical standpoint we should try to provide similar functionality for all distros
13:42:21 <mmalchuk> indeed
13:42:39 <SvenKieske> afaik that's why we use upstream packages, maybe we should package apache httpd ourselves? /troll
13:42:57 <mmalchuk> it can be described in the docs. about 'curl... server-status' to check
13:43:29 <mmalchuk> huge overhead
13:43:48 <jangutter> do we have 2 minutes for a quick report about our favourite etcd version? (timeboxed)
13:43:50 <mnasiadka> SvenKieske: better tell me why Debian/Ubuntu exposes all users to that ,,issue''
13:44:33 <mmalchuk> the answer 'because'))
13:44:38 <SvenKieske> just debian things I guess, not a fan of the tech side - I was always a fedora guy personally - I stumbled upon this error in a different context some years ago.
13:44:51 <mmalchuk> because they want
13:44:58 <frickler> jangutter: imo later is better, but we can ony bump minor version by one per cycle?
13:45:11 <jangutter> TL;DR etcd 3.4 drops deprecated API interfaces. Nearly every kolla-ansible service uses the old interface.
13:45:13 <frickler> also please stop these distro wars
13:45:27 <mmalchuk> they enable huge list of modules and expose more info
13:45:42 <SvenKieske> debian has the philosophy of enabling everything to nanny the user (autostarting services), except where it's useful (providing auditd with remote support)..
13:45:44 <mnasiadka> frickler: not a distro war, but it should be semi-secure by default ;)
13:45:58 <frickler> jangutter: oh, so 3.4 is too new for us?
13:46:10 <mnasiadka> jangutter: so all services need a change in their config
13:46:12 <mnasiadka> jangutter: ?
13:46:17 <SvenKieske> I can work with every distro, and there are objectively bad and good things in all of them
13:46:27 <jangutter> and Zun is particularly badly affected - they depend on a particularly painful combination of docker and old etcd. I only see WIP things for it, but it requires some hard rework for them.
13:46:34 <frickler> jangutter: afaict devstack is also hardcoded to 3.3, guess we need to bump there first
13:46:51 <jangutter> Cinder (and hopefully all tooz affected things) are configurable and "works" on 3.4.
13:46:52 <mnasiadka> jangutter: we can drop zun support, it's been painful already ;)
13:47:26 <SvenKieske> uh that's a bummer, should there be a blueprint or ML post to coordinate upgrades for etcd then?
13:48:06 <jangutter> Yah, noting that 3.3 is no longer maintained, and 3.4 is likely to be soon, sooner migration is better.
13:48:45 <frickler> I'll propose a bump in devstack and see what happens there
13:49:00 <frickler> and then we can check in kolla again
13:49:12 <jangutter> There's a lot of "v3alpha" hardcoded everywhere in clients, especially python ones.
13:49:42 <mnasiadka> ok, so that seems like a longer topic - maybe worth a thread on ML?
13:49:50 <jangutter> (I got it to work in k-a, trying to fix the unrelated noise in the job)
13:49:59 <jangutter> I'll send out a post!
13:50:01 <opendevreview> Maksim Malchuk proposed openstack/kolla-ansible master: Deny access to public /server-status in http Openstack services  https://review.opendev.org/c/openstack/kolla-ansible/+/888943
13:50:22 <frickler> I have two more small patches: https://review.opendev.org/c/openstack/kolla-ansible/+/871054 and https://review.opendev.org/c/openstack/kolla-ansible/+/876650
13:50:48 <jangutter> Zun might be a casualty unfortunately, they hit a deprecated "feature" in docker _and_ etcd.
13:51:32 <mnasiadka> I don't think it's overly active, so no hard feelings
13:51:36 <mnasiadka> don't know if there's a lot of users
13:51:51 <jangutter> Thanks! end of timebox, feel free to ping me if you're interested or have Zun contacts
13:52:45 <mnasiadka> ok
13:52:51 <mnasiadka> anybody anything else?
13:53:41 <mmalchuk> kayobe reviews?
13:53:57 <mnasiadka> once kayobe CI is fixed we can handle that
13:54:03 <mnasiadka> currently it's broken due to systemd issue
13:54:10 <mmalchuk> https://review.opendev.org/c/openstack/kayobe/+/879554
13:54:14 <mmalchuk> https://review.opendev.org/c/openstack/kayobe/+/861397
13:54:22 <mmalchuk> please these two
13:54:33 <mnasiadka> they won't pass the CI, nor the gate, upgrade jobs are broken
13:55:06 <mnasiadka> ok, I think we've had the longest meeting this year
13:55:08 <mmalchuk> already passed
13:55:09 <mnasiadka> thanks for coming
13:55:15 <mmalchuk> thank you
13:55:15 <mnasiadka> mmalchuk: but they won't pass now.
13:55:20 <mnasiadka> #endmeeting