13:00:01 <mnasiadka> #startmeeting kolla
13:00:01 <opendevmeet> Meeting started Wed Aug 16 13:00:01 2023 UTC and is due to finish in 60 minutes.  The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot.
13:00:01 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
13:00:01 <opendevmeet> The meeting name has been set to 'kolla'
13:00:04 <mnasiadka> #topic rollcall
13:00:06 <mnasiadka> o/
13:00:11 <jangutter> o\
13:00:17 <bbezak> o/
13:01:00 <mmalchuk> \o
13:01:19 <mattcrees> o/
13:01:30 <frickler> \o
13:02:26 <mnasiadka> #topic agenda
13:02:26 <mnasiadka> * Announcements
13:02:26 <mnasiadka> * CI status
13:02:26 <mnasiadka> * Release tasks
13:02:26 <mnasiadka> * Current cycle planning
13:02:28 <mnasiadka> * Additional agenda (from whiteboard)
13:02:28 <mnasiadka> * Open discussion
13:02:30 <mnasiadka> #topic Announcements
13:03:08 <mnasiadka> the PTL and TC election has begun for 2024.1 - I've put my name for Kolla PTL for another cycle
13:03:37 <frickler> yay
13:03:41 <mmalchuk> vote +1
13:04:17 <mnasiadka> I thought so you're not going to oppose ;-)
13:04:22 <mnasiadka> #topic CI Status
13:04:25 <mgoddard> \o
13:04:31 <mnasiadka> Kayobe CI (upgrade) is still red
13:04:52 <mnasiadka> there's a patch ongoing review, hopefully we can merge it this week
13:05:17 <mmalchuk> give a link please
13:05:17 <mnasiadka> the magnum CI job is red as well (everywhere) - anybody with designate knowledge wants to have a look?
13:05:42 <mnasiadka> #link https://review.opendev.org/c/openstack/kolla-ansible/+/890198
13:06:20 <mnasiadka> if nobody wants to have a look - should we disable the designate part in Magnum CI jobs?
13:06:32 <frickler> hmm, I'll have a look
13:06:45 <mmalchuk> me too
13:06:45 <frickler> have some interest in designate being tested
13:06:50 <mnasiadka> thanks
13:06:56 <mmalchuk> there several patches for manbum today
13:07:04 <mmalchuk> magnum*
13:07:24 <mnasiadka> #topic Release tasks
13:07:42 <mnasiadka> We're close to standard projects releasing
13:08:13 <mnasiadka> It's time to switch to UCA and RDO Bobcat
13:08:24 <mnasiadka> UCA patch: https://review.opendev.org/c/openstack/kolla/+/891018
13:09:02 <mnasiadka> RDO repo is still empty for Bobcat
13:09:24 <mnasiadka> UCA patch needs second core to merge
13:09:32 <mnasiadka> #topic Current cycle planning
13:10:02 <mnasiadka> So, kevko has put some work into Let's Encrypt - hrw has been so kind to do a review of a lot of bash scripting
13:10:22 <mnasiadka> #link https://review.opendev.org/c/openstack/kolla/+/887347
13:10:33 <mnasiadka> and here is the kolla-ansible part
13:10:35 <mnasiadka> #link https://review.opendev.org/c/openstack/kolla-ansible/+/741340
13:10:58 <mnasiadka> It would be nice if some other cores (or non-cores) would do a thorough review - we should aim for getting this merged in the upcoming weeks
13:11:51 <mmalchuk> a lot of comments did SvenKieske and me too)
13:12:00 <hrw> UCA/Bobcat +2+w
13:12:04 <mnasiadka> Ok, thanks - let's keep it rolling
13:12:13 <mnasiadka> Another thing, there's a container called openvswitch-netcontrold
13:12:26 <mnasiadka> For some reason we've been installing the package via pip outside of a venv
13:12:35 <mnasiadka> The container is only used by tripleo
13:12:53 <mnasiadka> Is there anybody that is using ovs-dpdk?
13:13:19 <mnasiadka> netcontrold is a tool for optimizing PMD threads in OVS
13:13:27 <mnasiadka> From my perspective - it would be easier to drop it
13:13:40 <mmalchuk> +1
13:13:42 <hrw> +2
13:14:17 <mnasiadka> frickler: any dpdk usage on your side and you'd like to use netcontrold? ;-)
13:15:03 <frickler> nope
13:15:26 <jangutter> https://github.com/netcontrold/netcontrold-py <--- last update july 2020 ....
13:15:30 <mnasiadka> Ok, I'll send a mail to ML and drop this image
13:15:49 <mnasiadka> #action mnasiadka to send a mail to ML and drop openvswitch-netcontrold
13:16:21 <mnasiadka> #topic Additional agenda (from whiteboard)
13:17:09 <mnasiadka> so, regarding jobboard we have second patch to review - https://review.opendev.org/c/openstack/kolla-ansible/+/888588
13:17:17 <mnasiadka> it has RP+12
13:17:18 <mnasiadka> it has RP+1
13:18:37 <mnasiadka> ok, second thing is debian bookworm
13:18:45 <mnasiadka> #link https://review.opendev.org/c/openstack/kolla/+/886088
13:18:59 <mnasiadka> that should pass now and from my perspective we can merge that - and work on the kolla-ansible side of things
13:19:45 <mmalchuk> it can be merged afaik
13:20:01 <frickler> we need haproxy fix first
13:20:09 <frickler> I guess I should rework https://review.opendev.org/c/openstack/kolla-ansible/+/883559/5
13:20:25 <mmalchuk> why? whats wrong with haproxy?
13:20:28 <frickler> if we kind of agree to do it all in one step
13:20:38 <frickler> the option we use is no longer there in bookworm
13:21:17 <mmalchuk> another version there?
13:21:31 <frickler> newer version, yes, removed deprecated option
13:21:33 <mnasiadka> newer versions of haproxy does not support that option
13:21:49 <mmalchuk> lets stack changes
13:22:08 <mnasiadka> so basically we could fine tune those three patches and merge them, unless frickler wants to work on making out of those three patches (I'm lazy and I wouldn't) ;-)
13:22:10 <frickler> can't stack k-a on k
13:22:27 <frickler> they need rebasing anyway afaict
13:22:40 <mnasiadka> and there's also the thing about python version in wsgi paths
13:23:01 <mnasiadka> I did have a stupid approach on linking /var/lib/kolla/venv/lib/python3.X to /var/lib/kolla/venv/lib/python
13:23:21 <mnasiadka> to stop this madness of doing a lot of changes just to pass the variable properly in CI
13:23:50 <mmalchuk> lets merge stupid approach first)
13:24:34 <frickler> which patch is that?
13:24:58 <kevko> is  /var/lib/kolla/venv/lib/python good name ?
13:25:03 <mnasiadka> so, in the past we did: https://review.opendev.org/c/openstack/kolla/+/788304/2/.zuul.d/debian.yaml#9
13:25:36 <mnasiadka> and now I was thinking of https://review.opendev.org/c/openstack/kolla/+/874989 and https://review.opendev.org/c/openstack/kolla-ansible/+/874990
13:25:59 <kevko> wouldn't be better to include some custom path into path ?
13:26:36 <mmalchuk> you have another path to venv?
13:26:59 <frickler> I'm lacking context as to why this is/was all needed
13:27:02 <kevko> it's not my case ..but what if you are just not using venv ?
13:27:08 <frickler> but I can read up later
13:27:11 <mnasiadka> mainly because of this: https://review.opendev.org/c/openstack/kolla-ansible/+/874990/3/ansible/roles/horizon/templates/horizon.conf.j2
13:27:16 <mnasiadka> (and other wsgi services)
13:27:23 <mnasiadka> we need to pass the whole path to mod_wsgi
13:27:37 <mmalchuk> kevko pip requieres venev now
13:28:14 <mnasiadka> other option is just a play in kolla-ansible that find proper python version
13:28:23 <mnasiadka> and set distro_python_version to that
13:28:31 <mnasiadka> (the one in container images)
13:28:39 <mmalchuk> bad idea
13:28:55 <mmalchuk> what if there are two suitable versions?
13:29:42 <mnasiadka> two versions in a venv?
13:30:17 <mmalchuk> in system. but search in venv for other path? lol
13:30:23 <frickler> it seems wrong to me to run the wsgi from site-packages, why is this not in xxx/bin?
13:31:06 <mmalchuk> according the venv the path always lib/...
13:31:29 <kevko> frickler +1
13:31:32 <mmalchuk> there can't be several versions so the path is constant
13:32:05 <mmalchuk> frickler why?
13:32:29 <mmalchuk> it run within venv
13:33:17 <mmalchuk> if there is no bin/... file - this is an issue of the package itself
13:33:19 <frickler> because of the mess we are seeing now. seems like a bug in horizon to me. but I need to take a closer look, might be I'm missing something
13:33:32 <frickler> mmalchuk: exactly
13:33:44 <mmalchuk> lets fix python lib
13:34:11 <frickler> but that's not from python lib, it is horizon packaging
13:34:21 <mmalchuk> show ma your bug later... will take a look
13:34:25 <mnasiadka> frickler: horizon freezer and skyline seem to be using it
13:34:48 <frickler> one bad precedent copied two times, doesn't make it more valid IMHO
13:35:58 <frickler> but we don't need to decide on this now, do we?
13:36:25 <mmalchuk> lets dig deeper... later
13:36:45 <mnasiadka> yup
13:36:52 <mnasiadka> ok then
13:37:15 <kevko> btw, can we also drop kolla_distro ? :)
13:37:23 <mnasiadka> we can drop kolla if you like
13:37:24 <mnasiadka> :)
13:37:27 <mmalchuk> lol
13:38:03 <mmalchuk> an move to the k8s)))
13:38:05 <kevko> sometimes i am trying something with different image locally to just be sure that some implementation is working on both distros ...
13:38:46 <mnasiadka> that's how it is and have been for long time ;)
13:38:50 <mnasiadka> anyway, let's continue
13:38:57 <mnasiadka> mattcrees would like us to review https://review.opendev.org/c/openstack/kolla-ansible/+/882825
13:39:23 <mattcrees> Yes please
13:39:36 <mnasiadka> and https://review.opendev.org/c/openstack/kolla-ansible/+/890391/4
13:39:45 <mattcrees> Would also be good to get a non-shpc core to review this: https://review.opendev.org/c/openstack/kolla-ansible/+/881392
13:39:46 <kevko> i would like to see some test with rabbitmq nodes stop/start to see everything is working ok
13:40:03 <kevko> queues recreated , clients reconnected ...etc
13:40:14 <kevko> can we add something like this into ci ?
13:40:57 <frickler> kevko wants to be our chaos monkey ;)
13:41:11 <mmalchuk> mattcrees I'm not shpc and would like to be core) as you wish
13:41:28 <frickler> I'm not sure if it is possible to do in a stable way in CI
13:41:41 <mmalchuk> anyway reviewed
13:41:52 <mnasiadka> let's first get the CI in a stable form :)
13:42:12 <kevko> well, today there was some guy asking for these types of error ...
13:42:25 <kevko> *problems
13:42:50 <mnasiadka> true, but we all know that not all errors are bugs in our configuration
13:42:57 <mnasiadka> there's also oslo.messaging bugs and RMQ code bugs
13:43:00 <mattcrees> Does sound like it would cause CI problems. The recent rabbit changes improve stability, but don't guarantee it ;)
13:43:45 <mattcrees> Perhaps a not-voting job would be useful, to at least test regularly even if it falls over sometimes?
13:44:25 <frickler> but who would look at that regularly?
13:44:48 <mnasiadka> that's the problem, now there's not a lot of people looking at periodic publish jobs, not counting anything else :)
13:45:43 <mnasiadka> I think this can be a topic for the 2024.1 PTG, let's focus on existing patches to review now :)
13:45:47 <mnasiadka> #topic Open Discussion
13:45:52 <mmalchuk> this can be linked to whiteboard, and we check this atleast each week
13:45:56 <mnasiadka> ok then, anybody has any additional topic?
13:46:11 <mmalchuk> yep
13:46:23 <jangutter> (me after that?)
13:46:23 <mmalchuk> https://review.opendev.org/c/openstack/kolla/+/891098
13:46:54 <mmalchuk> handler to RH distros to make server-status everywhere identical
13:47:11 <mmalchuk> the same as in Ubuntu/Debian
13:47:19 <mmalchuk> same way
13:47:43 <mmalchuk> and also, folks please review https://review.opendev.org/q/I3ec1af6353c3ecc64589599abe375b0ae9b14d5c
13:47:49 <mmalchuk> X and Y
13:48:04 <mnasiadka> maybe let's merge the kolla-ansible patch first, so we don't introduce a security bug :)
13:48:18 <mmalchuk> we don't
13:48:37 <mmalchuk> and we're talking in master first
13:48:47 <mmalchuk> the fix is already there
13:49:13 <mnasiadka> I don't know if doing that in httpd_setup is the best way to do it, we could just do it in openstack-base by COPYing the file - but I have no strong opinion, let's have some other people voice in that patch
13:49:51 <mmalchuk> in Ubuntu/Debian this did in the kolla image so this is the same way
13:50:22 <mmalchuk> it configured and enabled right after apt-get install apache2
13:50:46 <mmalchuk> so for RH distros the httpd_setup the only place
13:51:54 <mnasiadka> not really - you can do it in here as well: https://github.com/openstack/kolla/blob/7f12d216dc4de2c8d32291c3d6223185ecf2b510/docker/openstack-base/Dockerfile.j2#L59 (we install apache here)
13:52:00 <mnasiadka> but let's discuss in the patch
13:52:03 <mnasiadka> 9 minutes left
13:52:05 <mnasiadka> jangutter: ?
13:52:30 <jangutter> etcd  3.3 -> 3.4 ... looks like fixing it in tooz will take a while.
13:52:31 <mmalchuk> apache not installed in base
13:52:43 <mnasiadka> mmalchuk: openstack-base
13:52:58 <mnasiadka> jangutter: and before it is fixed we can't do the upgrade?
13:53:09 <mmalchuk> mnasiadka RH only
13:53:26 <jangutter> So, it will require a change in both kolla and kolla-ansible if we can't wait for tooz. It passes all CI so far (flaky job just rarely)
13:53:36 <mmalchuk> mnasiadka it still kolla not kolla-ansible
13:54:00 <jangutter> If tooz can be updated, then kolla-ansible doesn't need changes.
13:54:07 <mnasiadka> mmalchuk: your patch is in kolla ;)
13:54:29 <frickler> tooz won't be updated in time for bobcat very likely
13:54:29 <mnasiadka> jangutter: what kind of changes? a lot?
13:54:33 <mmalchuk> mnasiadka but first you propose fix this in k-a
13:55:00 <jangutter> config file changes, minimal.... but until the endpoint is updated, cinder won't work, for example.
13:55:05 <mnasiadka> mmalchuk: that was long time ago, not fix, but allow users to configure it
13:55:28 <frickler> does cinder need etcd? or is it just an alternative to redis?
13:55:42 <mnasiadka> alternative - tooz supports both
13:55:43 <kevko> i think it's alternative, isn't it ?
13:55:57 <jangutter> It can use anything that tooz supports as a backend, but etcd seems to be the default in the CI.
13:56:00 <mnasiadka> some people like etcd, have no clue why :)
13:56:27 <frickler> but other services use redis, maybe we should make that consistent as a default?
13:56:39 <kevko> mnasiadka for similar reasons as some people like rh clones :)
13:56:51 <kevko> don't know why :D
13:56:57 <mnasiadka> kevko: I don't like them, they pay my rent :)
13:57:33 <mnasiadka> frickler: with tooz we support both, but I thought we have a recommendation for redis, not etcd
13:58:14 <frickler> so at least CI should be consistent? will need to look into that, too
13:58:18 <mnasiadka> basically we use what is enabled
13:58:19 <mnasiadka> #cinder_coordination_backend: "{{ 'redis' if enable_redis|bool else 'etcd' if enable_etcd|bool else '' }}"
13:58:34 <jangutter> I think it doesn't really matter what coordination backend is in use ... but at the moment everything in the CI (masakari and something else) points to etcd...
13:58:45 <jangutter> Aha
13:59:26 <mnasiadka> masakari? I thought only zun
13:59:37 <mnasiadka> but zun for a different reason (not tooz, docker overlay something)
13:59:55 * frickler needs to run, will check back later
14:00:04 <mnasiadka> anyway, time is up
14:00:14 <jangutter> --- masakari, cinder, designate, ironic (ironically!)
14:00:17 <mnasiadka> I think I have a patch for enabling redis in the ceph jobs
14:00:29 <mnasiadka> let's talk about this next week or somewhere outside of the meeting
14:00:33 <mnasiadka> need to run for a different meeting
14:00:36 <mnasiadka> thanks for coming!
14:00:36 <jangutter> thanks!
14:00:37 <mnasiadka> #endmeeting