15:00:00 <mnasiadka> #startmeeting kolla
15:00:00 <opendevmeet> Meeting started Wed Mar  9 15:00:00 2022 UTC and is due to finish in 60 minutes.  The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:00 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:00 <opendevmeet> The meeting name has been set to 'kolla'
15:00:04 <mnasiadka> #topic rollcall
15:00:12 <mnasiadka> o/
15:00:52 <kevko_> #help
15:01:14 <halomiva> \o
15:01:18 <hinermar> o/
15:01:20 <kevko> \o/
15:01:38 <yoctozepto> o/
15:01:49 <ohorecny2> o/
15:01:58 <yarovkon> o/
15:03:04 <mnasiadka> #topic agenda
15:03:04 <mnasiadka> * Review action items from the last meeting
15:03:04 <mnasiadka> * CI status
15:03:04 <mnasiadka> * Release tasks
15:03:04 <mnasiadka> * Current cycle planning
15:03:05 <mnasiadka> * Additional agenda (from whiteboard)
15:03:05 <mnasiadka> * Open discussion
15:03:11 <mnasiadka> #topic Review action items from the last meeting
15:03:52 <mnasiadka> mnasiadka to triage security bugs and update them with resolution plan (if needed)
15:03:52 <mnasiadka> Enable osbpo in Debian APT sources, abandon extrepo command use then?
15:03:52 <mnasiadka> mnasiadka to update kolla review dashboard with kolla collection
15:04:07 <mnasiadka> I triaged security bugs, but mgoddard created a new one :)
15:04:11 <mnasiadka> Dashboard not updated.
15:04:18 <mnasiadka> the osbpo task was on hrw
15:04:24 <mnasiadka> does anybody know if he has done it?
15:04:40 <yoctozepto> I've seen something
15:04:58 <yoctozepto> abandoned
15:05:11 <yoctozepto> https://review.opendev.org/c/openstack/kolla/+/831548
15:05:16 <yoctozepto> shift of approach
15:05:44 <mnasiadka> so extrepo will have offline config, ok
15:05:50 <mnasiadka> so then it's still sort of in progress
15:06:00 <yoctozepto> yup
15:06:34 <mnasiadka> #action mnasiadka to triage security bugs and update them with resolution plan (if needed)
15:06:41 <mnasiadka> #action mnasiadka to update kolla review dashboard with kolla collection
15:06:52 <mnasiadka> #action hrw Enable osbpo in Debian APT sources, abandon extrepo command use then?
15:06:57 <mnasiadka> #topic CI Status
15:07:00 <mnasiadka> How's our CI yoctozepto ?
15:07:40 <yoctozepto> mnasiadka: happy
15:07:48 <mnasiadka> goodie
15:07:49 <yoctozepto> no idea about kayobe though
15:08:07 <mnasiadka> I've seen an occasional OOM on TLS job, but nothing very concerning.
15:08:16 <mnasiadka> #topic Release tasks
15:08:21 <mnasiadka> None this week.
15:08:29 <mnasiadka> #topic Current cycle planning
15:09:28 <mnasiadka> Most of the leftovers have been postponed to Z, but let's get anything we can in Y.
15:09:58 <mnasiadka> yoctozepto: some more changes from mgoddard on baremetal role split
15:10:17 <yoctozepto> mnasiadka: where?
15:10:38 <mnasiadka> whiteboard L428
15:12:02 <yoctozepto> oh my
15:12:52 <yoctozepto> does not seem look like a priority to me, to be honest, add me as a reviewer and I will get to it
15:12:57 <mnasiadka> ok
15:13:29 <mnasiadka> I think it's important for Kayobe to reach the last patch on the list to be merged
15:13:46 <mnasiadka> #topic Additional agenda (from whiteboard)
15:13:58 <mnasiadka> (yoctozepto) Having both PXE and iPXE working at the same time https://review.opendev.org/c/openstack/kolla-ansible/+/832159/
15:14:58 <yoctozepto> yes, I think that's something sensible to do
15:15:10 <yoctozepto> especially for people having existing pxe deployments
15:15:17 <yoctozepto> and wanting to progressively move to ipxe
15:15:39 <yoctozepto> the downside is we need to touch the volumes
15:15:54 <mnasiadka> as long as it works and doesn't break anything
15:15:58 <yoctozepto> yeah
15:16:11 <yoctozepto> the question is whether there is some state that we should preserve
15:16:19 <yoctozepto> in tftpboot and httpboot
15:16:30 <yoctozepto> to me it looks easy/not-expensive to regenerate
15:16:33 <mnasiadka> I'm not an Ironic expert
15:16:36 <yoctozepto> but I would like to know your opinions
15:16:38 <yoctozepto> yeah
15:16:45 <yoctozepto> sadly mgoddard is not around
15:16:52 <mnasiadka> so ideally we would need mgoddard to look into this
15:17:01 <yoctozepto> that was the point
15:17:33 <mnasiadka> ok then, I guess he'll try, given the notifications he already did get ;-)
15:18:32 <yoctozepto> mnasiadka: yeah ;-)
15:18:44 <mnasiadka> (yarovkon) Migration path for kolla-ansible from docker to podman (https://etherpad.opendev.org/p/Kolla-ansible_migration_plan_docker-podman)
15:18:50 <mnasiadka> that's next
15:19:27 <yarovkon> Wanted to ask if that path looks fine for you
15:20:26 <mnasiadka> I think we need to automate those first two steps, you don't expect people to do it manually on let's say - 300 servers?
15:20:51 <ohorecny2> of course that it will be automated in ansible
15:21:06 <ohorecny2> we even have it already (not in gerrit yet)
15:21:29 <kevko> ohorecny2: post to gerrit ..i'm curious
15:22:11 <mnasiadka> yes, post it to gerrit and let's review
15:22:18 <mnasiadka> but I doubt we will get to podman in Y
15:23:31 <yoctozepto> uninstalling docker cannot be the first step
15:23:34 <yoctozepto> as it will down containers
15:23:43 <yoctozepto> we should be able to migrate container by container
15:23:43 <kevko> it doesn't matter i think
15:24:02 <yoctozepto> kevko: how so?
15:24:13 <mnasiadka> yoctozepto: installing both podman and docker on the same host (at least in CentOS world) is problematic
15:24:17 <kevko> it was reply to mnasiadka that it will be not in Y
15:24:23 <yoctozepto> kevko: ack
15:24:25 <ohorecny2> yes, it doesn't matter we tested it with active VMs (instances) and they were still alive, during whole migration process
15:24:40 <yoctozepto> mnasiadka: hmm, that's sad - what's the issue?
15:24:53 <mnasiadka> yoctozepto: containerd mumbo jumbo
15:25:03 <kevko> haha, kolla will be tested directly from kolla itself :D ...so we will see how HA works :D :D :D
15:25:27 <mnasiadka> here is some writeup, it's a bit problematic: https://faun.pub/how-to-install-simultaneously-docker-and-podman-on-rhel-8-centos-8-cb67412f321e
15:25:32 <yoctozepto> ohorecny2: well, vms yes but not all other processes ;-)
15:25:38 <yoctozepto> mnasiadka: ack, sad
15:25:44 <yoctozepto> kevko: where?
15:25:47 <kevko> well, problem will be with rabbitmq, mysql and haproxy
15:26:02 <kevko> and definitively has to be run with limit to one host
15:26:27 <ohorecny2> yes, but small outage during migration process is ok, I think
15:27:00 <mnasiadka> as long as everything works afterwards, with rabbitmq it's not granted ;-)
15:27:10 <kevko> i'm just trying to say that on the end of process there should be tasks which will check everything is working well
15:27:36 <kevko> and when i am thinking about it ..same tests should run before these actions are going happen
15:27:50 <mnasiadka> we need to start with something, post the patch to gerrit and let's work on it collectively
15:28:15 <kevko> mnasiadka: for example I know that kolla-ansible changed autoheal politics (sorry i don't know the exact keyword in config) to something different
15:28:17 <ohorecny2> yes, it is ok to do some healthchecks before and after migration
15:28:21 <kevko> (for rabbitmq)
15:28:36 <mnasiadka> autoheal? it's not Kubernetes
15:28:42 <mnasiadka> ah, for rmq
15:28:51 <kevko> so on deployments with three rabbitmqs (small ones) rabbit will stop working immediately
15:29:05 <kevko> for rabbitmq
15:30:02 <kevko> mnasiadka: rabbitmq_cluster_partition_handling: "pause_minority" << this
15:30:08 <opendevreview> Merged openstack/kolla stable/train: CentOS Stream 8: drop CentOS RabbitMQ 3.8 repo  https://review.opendev.org/c/openstack/kolla/+/832650
15:30:48 <kevko> in moment where rabbitmq will not see a node ..rabbitmq will be unresponsive ..
15:30:56 <mnasiadka> kevko: pause_minority is partition handling, there's no partition if you stop one node.
15:31:37 <mnasiadka> it should only cause havoc on network issues
15:31:50 <mnasiadka> but we've diverted the topic
15:32:30 <ohorecny2> yes, there can be some errors, because it will be for some time down, but after migration it is going up again
15:32:59 <mnasiadka> as long as we have some CI that is testing the move - we can work out the quirks I guess
15:33:21 <mnasiadka> so, post a patch - and let's discuss in the review (and probably on next meetings)
15:33:54 <kevko> hmm ok
15:34:04 <mnasiadka> There's another podman-ish topic
15:34:05 <mnasiadka> (halomiva,hinermar) structure of podman testing and moving baremetal role to openstack collection
15:34:51 <mnasiadka> what about this one?
15:34:54 <halomiva> i think we should talk about systemd first
15:35:01 <halomiva> because podman depends on it
15:35:22 <mnasiadka> yes, systemd is something we should be able to land in Y
15:36:22 <mnasiadka> so what's the status of systemd implementation?
15:36:39 <halomiva> tests are passing but reviews are not coming so we would really appreciate some reviews
15:36:59 <halomiva> because we implemented everything you asked for
15:37:21 <mnasiadka> https://review.opendev.org/c/openstack/kolla-ansible/+/816724 - this one?
15:37:37 <halomiva> yes
15:38:23 <mnasiadka> yoctozepto: have cycles to look there again?
15:40:15 <yoctozepto> mnasiadka: currently not
15:40:33 <opendevreview> Merged openstack/kolla-ansible master: Explicitly unset net.ipv4.ip_forward sysctl  https://review.opendev.org/c/openstack/kolla-ansible/+/832087
15:40:57 <mnasiadka> ok, I'll do some testing and review that until end of the week.
15:41:01 <mnasiadka> and ask mgoddard to do the same
15:41:09 <hinermar> Thank you
15:41:34 <hinermar> And regarding the testing of podman patch, since the single file for kolla_docker would be too large I have split up the specific test parts and put them in directory /tests/kolla_docker_tests/
15:41:51 <hinermar> I just want check with you if that's alright
15:42:52 <mnasiadka> sure, who likes big files
15:43:25 <hinermar> Great, thank you
15:43:53 <halomiva> since baremetal role was move to collection and is under refactor, does it make sense to try push our patch to it for installing podman?
15:44:32 <halomiva> or should we keep baremetal role in kolla-ansible for testing purposes?
15:45:00 <mnasiadka> baremetal role is already out, removed
15:45:22 <mnasiadka> so you need to add podman installation to kolla Ansible collection
15:45:38 <mnasiadka> and without that, you won't be able to test the podman patch ;-)
15:45:56 <halomiva> yea but there are some changes coming to that role no?
15:46:42 <halomiva> so should I wait for the changes to come or push it now
15:46:42 <mnasiadka> yes, look if there are any patches conflicting with your work, and adapt to that direction?
15:47:04 <mnasiadka> I don't think mgoddard is planning a lot more work with the collection this cycle
15:47:36 <mnasiadka> https://review.opendev.org/q/project:openstack/ansible-collection-kolla
15:50:27 <mnasiadka> halomiva: probably best would be to create new role(s) for podman in the collection
15:50:47 <mnasiadka> ok then
15:51:01 <mnasiadka> #topic Open discussion
15:51:08 <mnasiadka> Anybody? Anything?
15:51:17 <ohorecny2> yes, just ask for review
15:51:30 <ohorecny2> beside these podman changes to kolla-ansible
15:51:36 <ohorecny2> we have additional to kolla
15:51:48 <ohorecny2> for image building by podman
15:52:13 <ohorecny2> from our point of view they are ready, is there possibility to merge them to Y release?
15:52:21 <mnasiadka> url?
15:52:33 <ohorecny2> I think it is quite independent thing
15:52:55 <ohorecny2> mnasiadka: https://review.opendev.org/q/owner:konstantin.yarovoy%2540tietoevry.com
15:54:15 <mnasiadka> ok, I see them - will try to review before next meeting
15:54:33 <ohorecny2> thank you very much
15:55:55 <mnasiadka> yoctozepto: I'm nearly finished with the fqdn based single external frontend on HAProxy (https://review.opendev.org/c/openstack/kolla-ansible/+/823395) - would be nice if you could look into that
15:56:17 <mnasiadka> I'll add a CI job as well, guess without that it's going to be hard to merge ;)
15:56:41 <mnasiadka> path based should be easy to add, probably I'll do that when adding CI job
15:57:48 <mnasiadka> ok, let's finish for today
15:57:50 <mnasiadka> thanks for coming!
15:57:53 <mnasiadka> #endmeeting