15:00:12 <mgoddard> #startmeeting kolla
15:00:13 <openstack> Meeting started Wed Feb 24 15:00:12 2021 UTC and is due to finish in 60 minutes.  The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:14 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:17 <openstack> The meeting name has been set to 'kolla'
15:00:19 <mgoddard> #topic rollcall
15:00:23 <mgoddard> \o
15:00:33 <headphoneJames> o/
15:00:47 <wuchunyang> \o
15:03:02 <yoctozepto> \o/
15:03:17 <hrw> ]o[
15:03:48 <mgoddard> #topic agenda
15:03:59 <kevko_> hi :P
15:04:03 <mgoddard> * Roll-call
15:04:04 <mgoddard> * Announcements
15:04:06 <mgoddard> * Review action items from the last meeting
15:04:08 <mgoddard> * CI status
15:04:10 <mgoddard> * Review requests
15:04:12 <mgoddard> * yoctozepto: Stop caring about NTP/chrony
15:04:14 <mgoddard> * Wallaby release planning
15:04:16 <mgoddard> #topic announcements
15:04:19 <mgoddard> None from me
15:04:22 <mgoddard> Anyone else?
15:04:57 <wuchunyang> no
15:05:11 <mgoddard> #topic Review action items from the last meeting
15:05:15 <mgoddard> There were none
15:05:23 <mgoddard> #topic CI status
15:07:14 <mgoddard> Kolla Train still broken
15:07:25 <mgoddard> just rechecked https://review.opendev.org/c/openstack/kolla/+/774602
15:08:19 <hrw> uf. it was just pull limit
15:09:14 <mgoddard> yeah
15:09:25 <mnasiadka> I've seen multiple occurences of jobs where one of the hosts fail on "Failed to connect to the host via ssh", but that's independent from us...
15:10:01 <mgoddard> I started this to track mounting kayobe CI failures: https://etherpad.opendev.org/p/kayobe-ci-failures
15:10:03 <mnasiadka> do we have ssh retries configured in Ansible?
15:10:36 <mnasiadka> Seems not really, I'll submit a patch
15:11:17 <mgoddard> mnasiadka: is it zuul's ansible or ours?
15:12:24 <ohorecny2> Hi all, I would like to ask if there are some plans to support Podman instead of Docker in case of kolla and kolla-ansible. Do community want to support it in future?
15:14:33 <mgoddard> ohorecny2: hi, we are currently in a meeting
15:14:49 <mnasiadka> mgoddard: it's ours, during deploy
15:15:07 <mgoddard> ohorecny2: there is interest in the community for podman, but no one has started work on it yet. You might want to talk to Fl1nt about it
15:15:28 <mgoddard> ohorecny2: I also made a small PoC: https://github.com/stackhpc/kolla-ansible/commit/e44d4b028e3aa24955dd12271783287ae43a5603
15:15:45 <mgoddard> mnasiadka: we could try adding retries
15:15:59 <mnasiadka> mgoddard: that's what I said - will raise a patch :)
15:16:08 <mgoddard> mnasiadka: I was agreeing
15:16:43 <mnasiadka> mgoddard: In darkest corners of my brain I couldn't see a situation somebody would disagree :)
15:16:55 <mnasiadka> But thank you for agreeing :)
15:17:28 <mgoddard> it will only help if the network issues are temporary
15:19:02 <mgoddard> #topic Review requests
15:19:11 <mgoddard> Does anyone have a patch they would like to be reviewer
15:19:15 <mgoddard> *reviewed
15:19:32 <headphoneJames> https://review.opendev.org/c/openstack/kolla-ansible/+/741340
15:19:47 <yoctozepto> all WIP
15:20:03 <headphoneJames> The Lets Encrypt patch is ready, still putting the test together
15:20:45 <openstackgerrit> Michal Nasiadka proposed openstack/kolla-ansible master: CI: Add ssh retries and disable group names validation  https://review.opendev.org/c/openstack/kolla-ansible/+/777402
15:21:14 <wuchunyang> what 's the plan  of the octavia management network patch ? i have raised it to email . but no help for me.
15:22:07 <ohorecny2> @mgoddard: thank you for reply, I will contact @Fl1nt
15:22:11 <mgoddard> wuchunyang: bbezak has been investigating
15:22:46 <mgoddard> wuchunyang: will see if he's available to discuss
15:22:54 <mgoddard> headphoneJames: I'll add RP+1
15:23:41 <mgoddard> seems like bbezak is busy
15:23:59 <kevko_> mgoddard: yes, i want to review , but you know :) ..just joking
15:24:03 <wuchunyang> ok, BTW, who is bbezak . sorry  i don't know him.
15:24:06 <mgoddard> wuchunyang: bbezak found that qemu-kvm package is the cause.
15:24:09 <mnasiadka> I've been investigating with him
15:24:18 <mgoddard> ok, mnasiadka can update
15:24:33 <wuchunyang> nice. that's enough to me.. haha
15:24:34 <mgoddard> wuchunyang: bbezak is my colleague, Bartosz Bezak
15:24:46 <wuchunyang> thanks very much.
15:24:59 <mnasiadka> qemu-kvm 5.1 brings the issue, but it seems it's only problematic if the VM is Ubuntu and the flavor is like 1vcpu 2-4G RAM
15:25:08 <mnasiadka> on octavia CentOS 8 test image it works properly
15:25:30 <mnasiadka> on bigger flavors it also works with Ubuntu
15:25:47 <mnasiadka> and the issue is crashing the VM while adding second port (for vrrp)
15:26:23 <mnasiadka> bbezak will raise a bug in Nova/qemu, and do a followup on the mailing list
15:26:37 <mnasiadka> in the meantime, we could pin qemu-kvm to 4.2 in CentOS images
15:26:57 <mnasiadka> wuchunyang/mgoddard: do we have the same problem on Ubuntu hypervisors?
15:27:06 <wuchunyang> no
15:27:09 <wuchunyang> just centos 8
15:27:31 <bbezak> Sorry I'afk. Will update this change and reply to openstack discuss group
15:27:43 <bbezak> I'm afk
15:27:52 <mnasiadka> bbezak: that's what I wanted to write, that you'll also update the change :)
15:28:38 <wuchunyang> you said if i use centos image. it should work around this problem. right ?
15:28:46 <mgoddard> yes
15:29:19 <wuchunyang> nice. Let's work around for CI first.
15:30:00 <mgoddard> ok
15:30:17 <wuchunyang> thanks for mgoddard mgoddard mgoddard
15:30:38 <mgoddard> mgoddard^3
15:30:58 <mgoddard> #topic yoctozepto: Stop caring about NTP/chrony
15:31:07 <mgoddard> yoctozepto stopped caring about time
15:31:29 <yoctozepto> yup
15:31:35 <yoctozepto> that's the plan
15:31:38 <yoctozepto> 8-)
15:31:54 <mnasiadka> you mean finally deprecate chrony container? :)
15:31:59 <yoctozepto> joking aside, this is what we have planned
15:32:02 <yoctozepto> yes, mnasiadka :P
15:32:12 <yoctozepto> we are moving this cycle-cycle
15:32:21 <yoctozepto> time to enact :D
15:32:22 <mnasiadka> thank to gods of total time chaos
15:33:20 <yoctozepto> it is time to drop time
15:33:58 <mgoddard> seriously though, what is the plan?
15:38:01 <mgoddard> yoctozepto?
15:38:48 <mgoddard> anyone?
15:39:48 <wuchunyang> ?
15:40:10 <mgoddard> seems like we have lost the leader of this discussion. Let's come back to it later
15:40:28 <mgoddard> #topic Wallaby release planning
15:40:48 <mgoddard> Friendly reminder: Kolla feature freeze: Mar 29 - Apr 02
15:41:09 <yoctozepto> mgoddard: sorry, yes, funky connection
15:41:33 <yoctozepto> you can't see me gone because of my bouncer
15:41:46 <yoctozepto> anyhow
15:41:49 <mgoddard> ok
15:41:51 <mgoddard> #undo
15:41:52 <openstack> Removing item from minutes: #topic Wallaby release planning
15:42:02 <yoctozepto> thanks
15:42:06 <yoctozepto> so
15:42:21 <yoctozepto> afair, kayobe handles ntp on the host side
15:42:41 <yoctozepto> and any default install configures ntp nowadays
15:42:45 <yoctozepto> for a myriad of reasons
15:43:19 <yoctozepto> poor man's option is to recommend an ansible module to manage ntp
15:43:36 <yoctozepto> (I guess kayobe uses one already, can go with it)
15:43:54 <yoctozepto> luxurious option would be to integrate it with kolla-ansible
15:43:58 <yoctozepto> but it is blurry
15:44:08 <yoctozepto> and I think it belongs to the level of tools like kayobe
15:45:15 <mgoddard> kayobe actually removed support for NTP config since CentOS 8
15:45:16 <yoctozepto> we have timesync prechecks in already
15:45:26 <mgoddard> and relies on Kolla Ansible :)
15:45:39 <yoctozepto> mgoddard: oh gosh
15:45:44 <mgoddard> however, we'll probably change change approach
15:46:14 <mgoddard> and configure chrony on CentOS, possibly systemd-resolved or chrony on Ubuntu
15:46:31 <openstackgerrit> Marcin Juszkiewicz proposed openstack/kolla master: switch to CentOS 8 Stream  https://review.opendev.org/c/openstack/kolla/+/772841
15:46:41 <hrw> why not systemd-resolved on all distros?
15:46:45 <mgoddard> so I don't think it affects this discussion
15:46:56 <mgoddard> hrw: chrony is standard out of the box on centos
15:47:14 <mgoddard> hold on
15:47:26 <mgoddard> sorry, systemd-timesyncd
15:47:36 <Fl1nt> I'm in sorry ^^
15:47:45 <mgoddard> too many systemd-*d in ubuntu :)
15:47:49 <yoctozepto> yup
15:48:00 <yoctozepto> all right then
15:48:20 <yoctozepto> so we have the timesync prechecks since victoria
15:48:22 <yoctozepto> https://docs.openstack.org/releasenotes/kolla-ansible/victoria.html
15:48:24 <yoctozepto> as I mentioned
15:48:38 <yoctozepto> so the sanity checks are there in case users want to go rogue 8-)
15:48:50 <mgoddard> I suppose my only concern is for non-kayobe users who want more than the out of the box NTP config
15:48:52 <yoctozepto> we can adapt docs
15:48:54 <yoctozepto> slap renos
15:48:55 <yoctozepto> go ml
15:49:10 <yoctozepto> yeah, we would point them to ansible roles for that
15:49:30 <yoctozepto> galaxy should have something nice
15:49:39 <yoctozepto> fwiw, I have this part in puppet
15:49:43 <yoctozepto> <no judging>
15:50:20 <Fl1nt> systemd-*d are not too many as it is a way to address concerns to people that can't stop complaining that systemd is a monolithic monster that isn't respecting unix principles :p
15:50:21 <yoctozepto> we have *not* officially deprecated chrony
15:50:28 <yoctozepto> so we should do a deprec this cycle
15:50:56 <Fl1nt> which alternative would you like to use?
15:51:14 <yoctozepto> deprec on both kolla and k-a
15:51:18 <yoctozepto> and with recommendations
15:51:37 <yoctozepto> Fl1nt: give recommendations and leave it to the users
15:52:17 <yoctozepto> ntp is nothing extraordinary nowadays
15:52:26 <Fl1nt> you mean, we let it free for user as we're now relying on hosts right?
15:53:40 <Fl1nt> mgoddard, I synced with ohorecny2 about podman btw
15:54:10 <mgoddard> Fl1nt: ok, we can discuss in a min if you want
15:54:30 <mgoddard> does anyone have opinions on the proposed approach to NTP?
15:54:39 <Fl1nt> TBN: as we talk about wallaby, in order for me to work on SAML2 keystone SSO, I need the OIDC patch to be merged on master if everything is fine as I'll base my patch on it.
15:54:39 <yoctozepto> Fl1nt: what hosts?
15:54:48 <yoctozepto> Fl1nt: we are talking time synchronization
15:54:51 <Fl1nt> yes
15:54:52 <mgoddard> Fl1nt: it's merged
15:55:04 <Fl1nt> mgoddard, oooh sweet, perfect!
15:55:22 <yoctozepto> mgoddard: do you have any opinions?
15:55:47 <Fl1nt> yoctozepto, HW host that provide time to containers services?
15:56:08 <mgoddard> yoctozepto: I gave one
15:56:35 <mgoddard> yoctozepto: I think we should solicit feedback from the community
15:56:52 <Fl1nt> personally I can use chrony or systemd-timesyncd no issue as long as it works ^^
15:56:59 <yoctozepto> mgoddard: ack
15:57:10 <yoctozepto> Fl1nt: leaving ntp to users
15:57:16 <Fl1nt> yes
15:57:56 <mgoddard> an NTP client isn't difficult to setup, but it might be a bit annoying for users currently relying on k-a for it
15:58:21 <kevko_> can I just ask why you want to remove chrony/ntp from kolla ?
15:58:32 <mgoddard> but if there are no strong objections from the community then I'm fine with it
15:58:37 <mgoddard> good question kevko_ :)
15:58:43 <kevko_> :D
15:59:37 <kevko_> so, what is the answer ? :)
15:59:56 <Fl1nt> honestly, I don't have strong opinion on this one, leave it or remove it I don't really care as long as services are able to get time, personnally all my hosts are ansible bootstraped using chrony for now but don't really care.
16:00:20 <Fl1nt> to switch to something else
16:00:46 <yoctozepto> kevko_: mostly because we are forced to handle whatever the host throws at us
16:00:53 <yoctozepto> and it's extra code to maintain
16:00:58 <yoctozepto> for no real benefit
16:01:08 <yoctozepto> it's not like kolla-ansible brings new ntp features to the table :-)
16:01:30 <Fl1nt> good reason, if it doesn't add any feature or advantage, just remove it.
16:01:43 <mgoddard> well hold on
16:01:59 <mgoddard> kolla-ansible is about automation, not augmenting network time protocols
16:02:10 <Fl1nt> yes
16:02:26 <kevko_> well, I don't thing that especially some small chrony image (and k-a part) is difficult to maintain
16:02:46 <rockey> not to throw a log on the fire, but doesn't k-a still need to do routine checks on ntpd's on system before relying on the hosts either way? k-a is dependant on synced time, laying this out on the enduser could backfire, right?
16:02:51 <Fl1nt> but the point is, time is already automated pretty much everywhere in enterprise by either the provisioning service or another bootstrap/base ansible internal playbook or puppet or whatever
16:02:52 <mgoddard> so if installing and configuring an NTP client makes users lives easier then that is arguably beneficial
16:03:33 <mgoddard> rockey: we did add a check that time is synced
16:03:43 <Fl1nt> are you talking about the chrony container or the host bootstrap configuration task ?
16:03:48 <mgoddard> ansible/roles/prechecks/tasks/timesync_checks.yml
16:04:05 <mgoddard> Fl1nt: probably both
16:04:16 <yoctozepto> yeah, the point is that most hosts come with some ntp daemon running
16:04:30 <mgoddard> I think for various reasons, we can probably agree that running chrony in a container isn't a great idea
16:04:40 <yoctozepto> and we spend time fixing bugs on detecting those or adapting security profiles
16:04:42 <mgoddard> but mainly what because yoctozepto just said
16:04:51 <yoctozepto> and it's a very basic service nowadays
16:04:56 <yoctozepto> yeah, mgoddard
16:05:09 <kevko_> agree, I also added some patch ..
16:05:13 <mgoddard> so I'm definitely +1 to removing the container
16:05:17 <kevko_> so, remember
16:05:18 <Fl1nt> container is probably too much, automate the ntp configuration at task level can still be let as is using chrony or up to the user as long as the appropriate binding is made within services containers.
16:05:32 <mgoddard> the question for me is whether it is useful to manage a service on the host
16:05:42 <mgoddard> the main use case being to use a custom NTP server
16:05:57 <yoctozepto> mgoddard: yup, that's what the recommendations would be for
16:06:09 <yoctozepto> do note k-a never configured a server
16:06:15 <yoctozepto> so users had to do it elsewhere
16:06:32 <mgoddard> I mean, configure the client to use a custom NTP server
16:07:08 <mgoddard> that was possible with the chrony container
16:07:19 <Fl1nt> that only make sense as you pointed out, if kolla/k-a is providing with a ntp server for the cluster, but it's highly unlikely that anyone on enterprise side of things will base their cluster on it as most of ITs or new installation do it on auxilliary/commander/bastion/whatever its named server.
16:07:42 <yoctozepto> yes, my point is that such users are already knowledgeable enough to do it with any ntp role :-)
16:08:01 <rockey> mgoddard: well then, as a happy user, i'd +1 the removal too, mostly due to "that most hosts come with some ntp daemon running" statement from yoctozepto
16:08:14 <yoctozepto> ayeee, we are past time
16:08:17 <mgoddard> indeed
16:08:18 <yoctozepto> action this on me
16:08:24 <yoctozepto> and let's close
16:08:29 <yoctozepto> sorry for taking so much time
16:08:37 <mgoddard> #action yoctozepto to ask openstack-discuss about NTP
16:08:40 <mgoddard> Thanks all
16:08:45 <mgoddard> #endmeeting