15:00:10 <mgoddard> #startmeeting kolla
15:00:10 <opendevmeet> Meeting started Wed Jun 23 15:00:10 2021 UTC and is due to finish in 60 minutes.  The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:10 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:10 <opendevmeet> The meeting name has been set to 'kolla'
15:00:16 <mgoddard> #topic rollcall
15:00:31 <yoctozepto> \o/
15:00:38 <mgoddard> \o/
15:00:49 <headphoneJames> o/
15:02:31 <ohorecny2> Hi all, I am first time on this meeting so in case that I am doing something wrong please correct me.
15:02:52 <yoctozepto> welcome ohorecny2
15:02:59 <headphoneJames> welcome
15:03:44 <mgoddard> welcome ohorecny2, thanks for joining
15:04:00 <mgoddard> #topic agenda
15:04:10 <mgoddard> * Roll-call
15:04:12 <mgoddard> * Agenda
15:04:14 <mgoddard> * Announcements
15:04:16 <mgoddard> * Review action items from the last meeting
15:04:18 <mgoddard> * CI status
15:04:20 <mgoddard> * Wallaby release planning
15:04:22 <mgoddard> ** libvirt/OVMF bug
15:04:24 <mgoddard> * Xena cycle planning
15:04:26 <mgoddard> ** podman (ohorecny)
15:04:28 <mgoddard> * Open discussion
15:04:44 <mgoddard> #topic Announcements
15:04:52 <yoctozepto> that agenda needed updating
15:05:04 <mgoddard> #info Kolla Wallaby should be released this week
15:05:08 <yoctozepto> \o/
15:05:39 <mgoddard> closer to the deadline than I'd hoped, but alas it always ends up this way
15:05:42 <yoctozepto> it's actually just going through the release team
15:05:48 <yoctozepto> yeah
15:05:58 <yoctozepto> let's try better next time
15:06:49 <mgoddard> the main blockers have been out of our control, as usual
15:07:10 <yoctozepto> yeah, sadly
15:07:29 <mgoddard> one problem is that we hold new releases to a higher standard than existing ones
15:07:38 <mgoddard> anyway
15:07:56 <mgoddard> #topic Review action items from the last meeting
15:08:00 <yoctozepto> that is a quality of ours
15:08:12 <mgoddard> There were none
15:08:19 <mgoddard> #topic CI status
15:09:02 <yoctozepto> kolla and kolla ansible fully green
15:09:11 <mgoddard> Ussuri fails sporadically on Ubuntu due to Neutron migrations failing
15:09:32 <yoctozepto> yeah, it somehow got more common recently; and only affects ubuntu
15:09:48 <mgoddard> weird
15:09:51 <yoctozepto> no idea what changed; some lib? kernel?
15:09:55 <mgoddard> some difference in mariadb?
15:09:56 <yoctozepto> i checked mariadb and neutron
15:10:03 <yoctozepto> and no version coincidence
15:10:06 <yoctozepto> no
15:10:17 <yoctozepto> we even have the same mariadb in centos8
15:10:32 <yoctozepto> it just started popping
15:10:38 <yoctozepto> looks like a race condition
15:10:56 <mnasiadka> Late again, eh
15:11:07 <yoctozepto> I updated the description
15:11:14 <mgoddard> k
15:11:15 <yoctozepto> it should be "upgrades to Ussuri fail..."
15:11:24 <yoctozepto> clean deploys do not
15:11:54 <mgoddard> is it multinode only?
15:12:00 <yoctozepto> no, singlenode too
15:12:06 <yoctozepto> that's why we got so many gate rechecks
15:12:22 <mgoddard> kk
15:12:39 <mgoddard> #topic Wallaby release planning
15:13:00 <mgoddard> libvirt/OVMF bug now fixed
15:13:12 <yoctozepto> yeah, and libvirt 7.4.0 in stream
15:13:20 <yoctozepto> 7.0 in non-stream
15:13:21 <mgoddard> wallaby on its way out of the door
15:13:35 <yoctozepto> therefore we got 7.0 in stable branches too
15:13:39 <yoctozepto> mind that
15:14:01 <mgoddard> does that break them?
15:14:12 <yoctozepto> no, they work
15:14:17 <mgoddard> goood
15:14:17 <yoctozepto> just watch out in production
15:14:59 <mgoddard> so we can just leave wallaby alone until we need to add support for CentOS stream 9 :D
15:15:13 <yoctozepto> I actually had a thought about it
15:15:22 <mgoddard> please share
15:15:27 <yoctozepto> and I think we should not follow rdo steps since we containerise
15:15:34 <yoctozepto> just apply the trick we do with debuntu
15:15:37 <yoctozepto> check if it works
15:15:40 <yoctozepto> and live happily
15:15:49 <mgoddard> I had a similar thought
15:15:59 <yoctozepto> then it's sealed
15:16:02 <mgoddard> well
15:16:16 <yoctozepto> it depends on whether it works
15:16:24 <yoctozepto> I just want to avoid the backporting circus
15:16:28 <mgoddard> actually mine was about CentOS Linux to CentOS stream, where there is an in-place upgrade
15:16:42 <mgoddard> there won't necessarily be one for CS8 to CS9
15:16:46 <yoctozepto> ahm
15:17:02 <mgoddard> in which case we're back to a rolling reinstall
15:17:05 <yoctozepto> I meant more like allowing Xena to deploy on CentOS Stream 8 still
15:17:09 <yoctozepto> and have 9 in containers
15:17:45 <yoctozepto> that's what we allow on debuntu, except for a slight version reversal
15:17:48 <mgoddard> could do that
15:17:54 <yoctozepto> but then you could tell
15:18:08 <yoctozepto> cs8 + wallaby -> cs8 + xena -> cs9 + xena
15:18:14 <yoctozepto> roll as you wish
15:18:19 <mgoddard> although I think ideally containers should be ahead of hosts
15:18:32 <yoctozepto> yeah, they will be
15:18:39 <mgoddard> sorry, other way around
15:18:50 <yoctozepto> I think we are fine
15:19:02 <mgoddard> https://access.redhat.com/support/policy/rhel-container-compatibility
15:19:03 <yoctozepto> people been running focal on bionic for a year now
15:19:16 <mgoddard> maybe
15:19:19 <mgoddard> we can speculate
15:19:25 <yoctozepto> tier 3 is fine for what we don't have anyway
15:19:26 <mgoddard> but it needs to be tested
15:19:40 <yoctozepto> yes, that's the goal
15:19:51 <yoctozepto> test - if it works, go with it
15:20:04 <yoctozepto> if not - cry and adapt
15:20:06 <mgoddard> perhaps centos will have imploded by then
15:20:16 <yoctozepto> that could work too
15:20:24 <mnasiadka> Wonder if stream Victoria gives us 7.4 as well
15:20:41 <yoctozepto> mnasiadka: yes, it does imho
15:20:53 <yoctozepto> because it's just adv virt repo for stream
15:21:05 <yoctozepto> but yeah, check it please
15:21:05 <mgoddard> let us move on
15:21:15 <mgoddard> #topic podman (ohorecny)
15:21:27 <yoctozepto> ohorecny2, the floor is yours
15:21:27 <ohorecny2> hi all again
15:21:33 <ohorecny2> ok, thanks
15:21:35 <yoctozepto> hi ohorecny2 again
15:21:50 <ohorecny2> mayve I can quickly introduce myself
15:21:57 <yoctozepto> please go ahead
15:22:28 <ohorecny2> my name is Oliver and I am working in TietoEvry. In our company we are interested in support of Podman in kolla-ansible.
15:22:57 <ohorecny2> The main reason is that docker isn't supported by RedHat since Rhel8.
15:23:28 <ohorecny2> In our company we decided to invest some time for implementation of this support.
15:24:13 <ohorecny2> I am leader of this project and for now we are in stage that we are able to deploy basic all-in-one deployment based on Podman containers.
15:24:55 <ohorecny2> These containers are running as services and it seems that OpenStack is functional (we are able to spawn new VM, etc.)
15:25:12 <ohorecny2> For now we are testing only on CentOS 8
15:26:03 <ohorecny2> Regarding code we are using master og kolla-ansible, where we added new option to globals.yml fro user to choose contaner engine (Docker / Podman).
15:26:26 <ohorecny2> So each service has its own tasks for Docker and also for Podman.
15:26:43 <ohorecny2> This means that whole change is quite big.
15:27:25 <ohorecny2> The biggest challenge was replacement of kolla-docker module, which is mostly replaced by existing ansible modules for Podman.
15:28:25 <ohorecny2> Unfortunately it was not possible to replace everything and we needed to use also podman-py library for API calls (for getting some container facts)
15:28:48 <ohorecny2> I guess that this can be replaced somehow in future.
15:29:28 <ohorecny2> We would like to propose this change for review as soon as possible. We need to firstly do some internal review and do squash of commits.
15:30:06 <ohorecny2> I just want to know what do you think about it and few other questions.
15:30:08 <mgoddard> Thank you for the overview ohorecny2
15:30:43 <mgoddard> I think people will want to see the code to get a feel for the approach
15:30:54 <yoctozepto> ++
15:30:56 <mgoddard> are you using systemd to run the containers?
15:31:57 <ohorecny2> sure, as I mentioned we would like to propose this change in next few days. But it is quite big so reviewing will be so complicated. Also CI checks will be needed to adapt for this.
15:32:03 <yoctozepto> btw, debian now supports podman with systemd natively as well, ubuntu will in 22.04
15:32:31 <ohorecny2> mgoddard: yes, each container has its own service file
15:32:52 <mgoddard> ohorecny2: did you see this PoC? https://github.com/stackhpc/kolla-ansible/commit/e44d4b028e3aa24955dd12271783287ae43a5603
15:32:53 <yoctozepto> I think we might be able to save some complexity by applying some refactoring we have in proposals
15:34:27 <ohorecny2> yoctozepto: yes that is right, we did not consider Ubuntu or Debian yet
15:34:37 <yoctozepto> no problem
15:34:59 <ohorecny2> mgoddard: yes, I saw it some time ago
15:35:02 <mgoddard> ok
15:35:03 <yoctozepto> just mentioning we could test portability with debian
15:35:54 <mgoddard> ohorecny2: it is possible to push a patch chain to gerrit. If the commits are already cleanly separated then no need to squash them
15:36:25 <ohorecny2> mgoddard: by that way how it is possible to start / restart containers during some actions, but not possible to create new containers.
15:36:50 <ohorecny2> so, ansible modules for podman are used for container creatin
15:36:55 <ohorecny2> *creation
15:37:17 <mgoddard> L20 of the unit file does a docker/podman run
15:37:24 <mgoddard> which creates a container
15:38:18 <mgoddard> I did not test it very much
15:38:29 <mgoddard> I just wanted to get the concept into a commit
15:39:09 <mnasiadka> Ceph uses the same approach for a long time, so I guess not a lot of testing needed
15:39:34 <mgoddard> mnasiadka: LOL
15:39:38 <mgoddard> just chuck it in
15:40:04 <mnasiadka> mgoddard: I mean just running docker/podman run --rm from a systemd unit, not the whole functionality :D
15:40:05 <yoctozepto> lol
15:40:12 <yoctozepto> meh
15:40:42 <ohorecny2> mgoddard: right, but what about pulling images and etc?
15:40:45 <mnasiadka> Can't wait to see the proposal, and then we can discuss :)
15:41:06 <mgoddard> ohorecny2: yeah, there will be cases where systemd is not enough
15:41:26 <mgoddard> I would like to see a short spec on this
15:41:36 <ohorecny2> mgoddard: yes, there are several cases, that it is not possible to do by that way I think
15:42:06 <ohorecny2> mnasiadka: sure, as I mentioned, it is on the way :)
15:42:07 <mgoddard> it is a large enough change that we should do some up front thinking/design
15:42:34 <mgoddard> so I would suggest this as a rough plan
15:42:34 <opendevreview> Michal Nasiadka proposed openstack/kolla-ansible stable/train: baremetal: Don't start Docker after install on Debian/Ubuntu  https://review.opendev.org/c/openstack/kolla-ansible/+/791582
15:42:48 <mnasiadka> mgoddard: agree, we don't normally do specs - but that's core functionality
15:43:06 <mgoddard> 1. ohorecny2 to share current state of code with community via gerrit
15:43:27 <mgoddard> 2. community reads code and does some thinking
15:43:46 <ohorecny2> mgoddard: yes, exactly, this change is huge I think. For few basic services it was more than 255 files changed and around 9000 insertions
15:43:50 <mgoddard> 3. community discusses the approach and agrees a rough direction
15:44:12 <mgoddard> 4. ohorecny2 & colleagues write a spec describing the agreed direction
15:44:35 <mgoddard> 5.adapt code to spec & iterate
15:45:17 <yoctozepto> ++
15:45:52 <mgoddard> what I would like to know is, do ohorecny2 & team have the capacity to drive this through to completion?
15:46:17 <ohorecny2> regarding #5 yes, when this will be in review we will definitely need to adapt it, because we have there some workarounds which needs to be changed
15:46:24 <mgoddard> we are very limited on review resources, and I would hate to see us put in a *lot* of review time for this patch then not see it completed
15:47:00 <opendevreview> Michal Nasiadka proposed openstack/kolla-ansible stable/victoria: CI: Fix nfv job with kolla dependency  https://review.opendev.org/c/openstack/kolla-ansible/+/797698
15:47:57 <ohorecny2> mgoddard: yes, I discussed this with management and we would like to finish this till end. It is agreed at least till end of this year.
15:48:07 <mgoddard> great!
15:48:32 <ohorecny2> I hope that it is possible to finalize.
15:48:51 <ohorecny2> we would like to even adapt kolla to image build base on podman
15:49:07 <mgoddard> it will be important to find out answers to important questions early
15:49:23 <ohorecny2> yes that is right
15:49:32 <ohorecny2> that is the reason why I am here :D
15:49:39 <mgoddard> in particular, do all distros supported by kolla have support for podman
15:50:03 <ohorecny2> maybe also, which version of ansible is planned for next release
15:50:29 <mgoddard> probably min 2.10, max 2.11
15:50:35 <mnasiadka> around podman - remember each distro basically has a different version
15:51:15 <mnasiadka> mgoddard: there is no 2.11, unless you're speaking about ansible-base :)
15:51:29 <mgoddard> yes
15:51:40 <yoctozepto> I think we want to go ansible-2.11 and choose the installed modules
15:51:43 <yoctozepto> need to discuss that
15:51:48 <ohorecny2> so far we are testing with ansible 2.9
15:52:03 <mnasiadka> I think we discussed that already on the PTG, and agreed we start with that approach with kolla-toolbox, and see how it goes.
15:52:12 <mnasiadka> yoctozepto: ^^
15:52:13 <yoctozepto> something like that indeed
15:52:19 <yoctozepto> yes, I am here
15:52:38 <mnasiadka> sorry, too many calls today discussing network packet processing ;)
15:53:06 <yoctozepto> mnasiadka: did you mention "encapsulation"?
15:53:18 <mnasiadka> yoctozepto: I even started that, but then we also need to move to FQCNs most probably
15:53:54 <mgoddard> does anyone have any more questions for ohorecny2 ?
15:53:56 <yoctozepto> mnasiadka: we should; I think there is a tool to migrate that
15:54:19 <yoctozepto> mgoddard: I need to see the code; and test on debian
15:54:31 <yoctozepto> I can help with CI
15:55:51 <mgoddard> ok, then let's move on
15:55:58 <mnasiadka> nevermind the podman versions, seems now it's 3.0 everywhere
15:56:00 <mnasiadka> so we should be ok
15:56:01 <mgoddard> Thanks ohorecny2 that was a helpful discussion
15:56:05 <yoctozepto> ++
15:56:22 <mgoddard> If you'd like to discuss again, feel free to ping one of us or add an item to the agenda on the wiki
15:56:24 <ohorecny2> thank you too
15:56:36 <mgoddard> #topic Open discussion
15:56:43 <mgoddard> Does anyone have anything today?
15:56:43 <yoctozepto> I believe podman is able to be less problematic than docker
15:56:49 <yoctozepto> but we will see
15:56:56 <headphoneJames> yes - I have a basic question
15:57:06 <yoctozepto> I have a lot of local stuff atm
15:57:36 <headphoneJames> It seems like DevStack and tripleo use cloud.yaml to define the initial openstack users with scope.
15:57:55 <opendevreview> Michal Nasiadka proposed openstack/kolla-ansible stable/victoria: [CI] Fix the NFV scenario  https://review.opendev.org/c/openstack/kolla-ansible/+/797702
15:58:12 <opendevreview> Michal Nasiadka proposed openstack/kolla-ansible stable/victoria: CI: Fix nfv job with kolla dependency  https://review.opendev.org/c/openstack/kolla-ansible/+/797698
15:58:14 <headphoneJames> KA creates users using OpenStack Ansible modules like "os_user" - those modules do not accept scope as an argument.
15:58:22 <mgoddard> headphoneJames: users for what purpose?
15:58:45 <headphoneJames> initial roles + users
15:58:52 <headphoneJames> to admin the cloud
15:59:04 <mnasiadka> headphoneJames: have you checked if ansible collection does? https://github.com/openstack/ansible-collections-openstack/
15:59:45 <headphoneJames> mnasiadka: not yet - so I can start there
16:00:07 <mgoddard> headphoneJames: do you mean for tasks using the kolla_toolbox module to register endpoints, users, etc?
16:00:38 <headphoneJames> yes
16:00:51 <mgoddard> the downside of that is that you have to put some full admin creds on disk
16:01:25 <yoctozepto> we can use a ramdisk
16:01:28 <mgoddard> whereas we specify them on demand
16:01:47 <opendevreview> Mark Goddard proposed openstack/kayobe master: Ubuntu: add upgrade jobs in CI  https://review.opendev.org/c/openstack/kayobe/+/797626
16:02:16 <mgoddard> we could use a ramdisk, but it's still accessible always
16:02:18 <headphoneJames> however, we may not be able to specify the scope of a user/role using kolla_toolbox approach?
16:02:40 <mgoddard> ah, openstack modules don't provide the option?
16:02:49 <headphoneJames> or really I'm just not sure how we do that
16:03:08 <mnasiadka> they don't provide the option, even the latest ones in the collection
16:03:15 <mnasiadka> so first it would need to be added there
16:03:29 <headphoneJames> it didn't look that way from my initial digging, but I haven't looked at the code in https://github.com/openstack/ansible-collections-openstack/
16:04:03 <headphoneJames> so the first step is to update the ansible modules for openstack?
16:04:30 <mnasiadka> we already update the Ansible modules for OVS, so you can follow the approach
16:04:36 <mgoddard> it might be worth looking into - now the modules are in a collection we could potentially pull in a newer version in the kolla-toolbox image
16:05:20 <mnasiadka> headphoneJames: https://review.opendev.org/c/openstack/kolla/+/782906
16:05:28 <mgoddard> anyways, we should wrap up
16:05:30 <mgoddard> thanks all
16:05:41 <mnasiadka> thanks mgoddard
16:05:44 <mgoddard> #endmeeting