15:01:24 <yoctozepto> #startmeeting kolla
15:01:25 <openstack> Meeting started Wed May  5 15:01:24 2021 UTC and is due to finish in 60 minutes.  The chair is yoctozepto. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:27 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:01:29 <openstack> The meeting name has been set to 'kolla'
15:01:35 <yoctozepto> #topic Roll-call
15:01:39 <yoctozepto> \o/
15:01:46 <yoctozepto> who else is here?
15:01:56 <headphoneJames> o/
15:02:26 <mgoddard> o/
15:02:37 <mgoddard> sorry, was recharging drink
15:02:44 <yoctozepto> #chair mgoddard
15:02:45 <openstack> Current chairs: mgoddard yoctozepto
15:02:48 <yoctozepto> glad you here
15:02:51 <yoctozepto> please continue
15:02:57 <hrw> ]o[
15:04:08 <priteau> \o
15:04:18 <mgoddard> #topic agenda
15:04:20 <mgoddard> * Roll-call
15:04:22 <mgoddard> * Agenda
15:04:24 <mgoddard> * Announcements
15:04:26 <mgoddard> * Review action items from the last meeting
15:04:28 <mgoddard> * CI status
15:04:30 <mgoddard> * Wallaby release planning
15:04:32 <mgoddard> * Xena cycle planning
15:04:34 <mgoddard> ** Ansible (prepackaged collections) vs ansible-base/core
15:04:36 <mgoddard> ** Ansible 2.10 and auto transformation of invalid group names (https://github.com/ansible/ansible/issues/56930)
15:04:38 <mgoddard> ** master branch life cycle
15:04:40 <mgoddard> * Open discussion
15:04:42 <mgoddard> #topic announcements
15:04:57 <mgoddard> #info Kayobe RC1 & stable/wallaby branch created
15:05:02 <mgoddard> Any others?
15:05:10 <yoctozepto> none here
15:05:41 <mgoddard> #topic Review action items from the last meeting
15:05:57 <mgoddard> mgoddard to write up Xena PTG summary for openstack-discuss
15:05:58 <mgoddard> mgoddard email openstack-discuss about quay.io credentials
15:06:00 <mgoddard> yoctozepto create a bug for alertmanager issue
15:06:02 <mgoddard> mgoddard fix one line nfv job
15:06:04 <mgoddard> yoctozepto deprecate rally & tempest
15:06:06 <mgoddard> 1: done
15:06:09 <mgoddard> 2: no!
15:06:14 <mgoddard> 3: done
15:06:25 <mgoddard> 4: done
15:06:28 <mgoddard> 5: done
15:06:48 <mgoddard> #action mgoddard email openstack-discuss about quay.io credentials
15:06:53 <mgoddard> unsure why this one is so hard
15:07:12 <mgoddard> #topic CI status
15:07:51 <mgoddard> https://etherpad.opendev.org/p/KollaWhiteBoard
15:07:59 <mgoddard> L40
15:08:04 <hrw> Debian in master/wallaby is broken - no new instance can be created. have to add to whiteboard
15:09:01 <hrw> added
15:09:37 <mgoddard> I saw that was on the bullseye nodepool image job. Does it work on a buster image
15:10:28 <mgoddard> ?
15:10:35 <hrw> I suspect newer libvirt
15:10:48 <hrw> which we need anyway for wallaby
15:11:06 <yoctozepto> but but
15:11:12 <hrw> yoctozepto: yes?
15:11:12 <yoctozepto> newer images work fine
15:11:17 <yoctozepto> it's just the host that fails, no?
15:11:23 <yoctozepto> due to missing cgroup support
15:11:59 <hrw> what you mean?
15:12:19 <hrw> I just deployed whole Debian all-in-one. it fails to boot VM
15:12:45 <mgoddard> https://zuul.openstack.org/builds?job_name=kolla-ansible-debian-source&branch=master
15:12:52 <yoctozepto> yeah, but have not we switched the images already?
15:12:58 <yoctozepto> and it worked with buster on host?
15:13:05 <mgoddard> jobs seem to pass on a buster host
15:13:08 <hrw> ah. that
15:13:12 <mgoddard> at least sometimes
15:13:28 <mgoddard> e.g. https://zuul.openstack.org/build/381d9350f36a458680d7a4777df45375
15:14:03 <hrw> and that's buster
15:14:10 <mgoddard> yes
15:14:25 <mgoddard> has bullseye dropped cgroups v1?
15:15:03 <yoctozepto> perhaps needs some tweak in the config then?
15:15:06 <mgoddard> https://www.debian.org/releases/testing/amd64/release-notes/ch-information.en.html
15:15:20 <mgoddard> OpenStack Victoria (released in bullseye) requires cgroup v1 for block device QoS. Since bullseye also changes to using cgroupv2 by default (see Section 2.2.4, “Control groups v2”), the sysfs tree in /sys/fs/cgroup will not include cgroup v1 features such as /sys/fs/cgroup/blkio, and as a result cgcreate -g blkio:foo will fail. For OpenStack nodes running nova-compute or cinder-volume, it is
15:15:22 <mgoddard> strongly advised to add the parameters systemd.unified_cgroup_hierarchy=false and systemd.legacy_systemd_cgroup_controller=false to the kernel command line in order to override the default and restore the old cgroup hierarchy.
15:15:55 <hrw> 3>
15:16:14 <hrw> (nova-libvirt)[root@jagular machine]# mount|grep cgro
15:16:15 <hrw> cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)
15:16:35 <yoctozepto> oh my
15:16:38 <yoctozepto> that sad
15:16:44 <mgoddard> looks like something to go on
15:16:47 <yoctozepto> deprecate debian as host?
15:16:59 <mgoddard> shall we move on?
15:17:16 <hrw> move on
15:17:40 <yoctozepto> move on from debian
15:18:32 <mgoddard> RabbitMQ
15:18:46 <mgoddard> I think we're good apart from Train, correct?
15:19:14 <hrw> should be
15:19:30 <yoctozepto> yes
15:19:41 <yoctozepto> I suggest we focus on getting wallaby out
15:20:31 <mgoddard> when is bintray EOL?
15:20:43 <hrw> 4 days ago?
15:22:09 <mgoddard> so kolla stable/train is borked?
15:22:54 <mgoddard> to me that seems more urgent than wallaby
15:23:05 <yoctozepto> oh well
15:23:35 <hrw> let me check
15:24:34 <mgoddard> I can try to pick it up over the next few days if you're out of time
15:24:54 <openstackgerrit> Merged openstack/kayobe stable/victoria: CI: switch to quay.io for container images  https://review.opendev.org/c/openstack/kayobe/+/789779
15:24:58 <openstackgerrit> Merged openstack/kayobe stable/wallaby: Ubuntu: install qemu-img on seed-hypervisor  https://review.opendev.org/c/openstack/kayobe/+/789799
15:25:18 <yoctozepto> train has a different situation I guess
15:25:20 <hrw> by 'let me check' I meant 'let me run build'
15:25:39 <yoctozepto> whiteboard is not relevant, I progressed further actually
15:25:45 <priteau> https://bintray.com/rabbitmq/ 404
15:25:51 <yoctozepto> btw, I updated https://ethercalc.openstack.org/kolla-infra-service-matrix
15:26:02 <yoctozepto> so
15:26:11 <yoctozepto> in Train we have erlang not from rmq upstream
15:26:24 <yoctozepto> and we use rmq upstream only on centoses
15:26:41 <yoctozepto> so need a different approach than that backport
15:26:47 <mgoddard> nice, thanks
15:26:57 <openstackgerrit> Merged openstack/kayobe stable/victoria: Stop using platform-python  https://review.opendev.org/c/openstack/kayobe/+/789753
15:27:23 <hrw> INFO:kolla.common.utils.rabbitmq-3.7.24:https://dl.bintray.com/rabbitmq-erlang/rpm/erlang/22/el/7/repodata/repomd.xml: [Errno 14] HTTPS Error 403 - Forbidde
15:27:36 <yoctozepto> I think I may get back to it
15:27:38 <yoctozepto> this week
15:27:43 <hrw> INFO:kolla.common.utils.rabbitmq:Successfully tagged train/centos-source-rabbitmq:12.1.0
15:28:31 <mgoddard> tis a shame we don't have that infra service matrix in docs
15:29:14 <mgoddard> I think that's enough for CI issues
15:29:25 <mgoddard> #topic Wallaby release planning
15:29:27 <yoctozepto> yeah, that's what it is for
15:29:38 <mgoddard> https://etherpad.opendev.org/p/KollaWhiteBoard
15:29:44 <mgoddard> L177
15:30:04 <yoctozepto> so, debian?
15:30:07 <mgoddard> I trimmed some completed stuff
15:30:13 <hrw> cool
15:30:47 <mgoddard> kevko: have time to update mariadb shards docs patch?
15:31:06 <mgoddard> I added an item to stop the chrony container on upgrade
15:31:36 <mgoddard> Otherwise, seems like UCA & bullseye are the big blockers
15:31:44 <hrw> I added pointer to cgroup
15:32:08 <hrw> need to deploy on debian/x86-64 and compare issues
15:32:46 <hrw> on aarch64 it is now nova complaining at neutron so I suspect rmq
15:34:02 <mgoddard> what about this one: https://review.opendev.org/c/openstack/kolla/+/787513
15:34:39 <mgoddard> I guess it removes that testing/unstable label
15:34:54 <mgoddard> although doesn't change that the release is still unstable :)
15:35:13 <hrw> I am all for 787513 ;D
15:35:17 <mgoddard> initial reaction is, can we rely on the pretty name to be stable?
15:35:22 <hrw> mgoddard: yes
15:36:03 <mgoddard> hrw: how do you know?
15:36:16 <hrw> or it will switch to 'Debian GNU/Linux (11) bullseye'
15:36:31 <hrw> in both cases we need to adapt as name we use for check will change
15:36:41 <hrw> or it will switch to 'Debian GNU/Linux 11 (bullseye)'
15:36:45 <hrw> etc
15:37:13 <yoctozepto> oh
15:37:22 <yoctozepto> right, it still says buster now
15:37:54 <yoctozepto> argh
15:38:00 <hrw> mgoddard: we either check for 'lsb_release -r -s' or PRETTY_NAME - both will change on bullseye release
15:38:05 <mgoddard> I hadn't even noticed that..
15:38:15 <hrw> 787513 moves that check to be 1st thing on build
15:38:21 <hrw> and unifies all distros
15:38:35 <mgoddard> but I was meaning more generally - can we rely on these strings to be immutable
15:38:49 <hrw> yes, they will not change during lifecycle
15:38:56 <hrw> once released
15:39:00 <openstackgerrit> Merged openstack/kolla-ansible stable/wallaby: Deprecate tempest and rally  https://review.opendev.org/c/openstack/kolla-ansible/+/789802
15:39:08 <hrw> we are not the only ones using them
15:40:01 <mgoddard> ok
15:40:54 <mgoddard> how about UCA
15:41:00 <mgoddard> mnasiadka: any progress in debugging?
15:41:49 <mnasiadka> Not really, we know that something is wrong with external net - router is not pingable
15:42:07 <hrw> mnasiadka: can you reproduce it locally?
15:42:25 <mnasiadka> hrw: didn’t try yet - will do tomorrow
15:42:33 <yoctozepto> so
15:42:40 <yoctozepto> we switch to wallaby uca
15:42:42 <imtiazc> Why are Rally and Tempest getting deprecated? Rally is still quite useful for doing performance and even functional tests.
15:42:49 <yoctozepto> installs work
15:42:59 <yoctozepto> and upgrades brek
15:43:01 <mgoddard> imtiazc: we're in a meeting currently
15:43:07 <yoctozepto> yet it does not happen without a switch
15:43:15 <openstackgerrit> Merged openstack/kayobe stable/wallaby: Add os_release variable, build CentOS stream images  https://review.opendev.org/c/openstack/kayobe/+/789800
15:45:07 <mgoddard> ok, wait for mnasiadka to debug more
15:45:43 <mgoddard> On the kayobe side, I think we have all patches we need at least proposed to master
15:46:02 <mgoddard> just need to get them merged and backported
15:46:19 <hrw> speaking of UCA... I just built 190 UCA/xena images ;D
15:46:32 <mgoddard> #topic Xena: Ansible (prepackaged collections) vs ansible-base/core
15:46:50 <mgoddard> mnasiadka: I think this was your item?
15:47:34 <mnasiadka> Yes, question which way we want to go
15:48:36 <mgoddard> So, the ansible package is the batteries included package with a set of specific collections
15:48:54 <mnasiadka> Yes, usually a bit older than latest versions
15:48:57 <mgoddard> and ansible-core (was ansible-base) is just the core ansible code without collections
15:49:06 <yoctozepto> we want it
15:49:13 <mgoddard> which?
15:49:25 <hrw> iirc kolla-toolbox has 363MB of ansible collections
15:49:46 <mgoddard> that's a good point - we may not have the same answer for toolbox and k-a
15:49:50 <mnasiadka> well, in toolbox I made a change to install later version of openvswitch collection
15:50:03 <mnasiadka> so we might use ansible-core + collections in toolbox
15:50:12 <mnasiadka> and stick to packaged for the user installation
15:50:12 <mgoddard> +`
15:50:16 <mgoddard> +1
15:50:53 <hrw> we have ansible only in kolla-toolbox, right?
15:50:54 <mgoddard> I suppose the benefit of ansible-core is that we get more control over which collections are in use?
15:51:29 <mnasiadka> yes, as mentioned - community.openvswitch collection was older in packaged full blown Ansible, and I had to update to get a bug-less version
15:51:52 <mnasiadka> and it seems Ansible is updating those only on new major release
15:52:15 <yoctozepto> mgoddard: core
15:52:19 <yoctozepto> we move faster
15:52:22 <yoctozepto> and are lighter
15:52:32 <yoctozepto> what not to love
15:52:50 <mnasiadka> so let's agree to first do it in toolbox, and see how it goes
15:53:06 <mgoddard> OTOH, it would involve an extra step to install collections
15:53:20 <mgoddard> we already do this in kayobe
15:53:41 <yoctozepto> mnasiadka: ++
15:53:44 <mnasiadka> anyway I think we need to move to using those FQCNs (long module names)
15:53:59 <yoctozepto> yeah, we also need to fix some inventory names
15:54:02 <yoctozepto> :-)
15:54:13 <mnasiadka> or ignore them
15:54:15 <yoctozepto> ...which is the next topic actually
15:54:19 <mnasiadka> that's I think the second topic in the agenda
15:54:47 <mnasiadka> I assume we would like to bump up Ansible to 2.10?
15:55:15 <hrw> 2.10 or 3.0?
15:55:24 <hrw> I lost count which is latest now
15:55:47 <mnasiadka> 3.0 is packaged, that uses 2.10 ansible-core :D
15:56:00 <mgoddard> we are on 2.10 max
15:56:26 <mnasiadka> and 2.9 min
15:56:32 <mgoddard> yes
15:56:51 <mnasiadka> So I guess it would make sense to bump min to 2.10, and think about bumping max to 2.11
15:57:07 <mnasiadka> but bumping max will bring problems like the next agenda item
15:58:11 <yoctozepto> ok
15:58:36 <yoctozepto> it seems it was done only for users unaware that - is not a part of python var name
15:58:37 <yoctozepto> oh well
15:58:56 <mgoddard> #topic Ansible 2.10 and auto transformation of invalid group names (https://github.com/ansible/ansible/issues/56930)
15:59:06 <yoctozepto> then we tell users what the config should be :-)
15:59:30 <mgoddard> I don't think Ansible will stop working with 'invalid' group names?
15:59:47 <yoctozepto> it seems it would make too many ppl unhappy
16:00:16 <mgoddard> so do we need to do it?
16:01:03 <mgoddard> We're out of time
16:01:33 <mgoddard> Hopefully we'll get to stable branch lifecycle next time
16:01:50 <mgoddard> Since we're basically at the point of needing to choose
16:01:52 <yoctozepto> thank you mgoddard
16:01:56 <mnasiadka> I think we can delay it, but it will bite us sooner or later :)
16:02:12 <mgoddard> I added some notes about the ansible changes to the priority list on the whiteboard
16:02:17 <mgoddard> Thanks all
16:02:20 <mgoddard> #endmeeting