15:01:50 <mgoddard> #startmeeting kolla
15:01:51 <openstack> Meeting started Wed Jan 13 15:01:50 2021 UTC and is due to finish in 60 minutes.  The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:52 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:01:55 <openstack> The meeting name has been set to 'kolla'
15:01:59 <mgoddard> #topic rollcall
15:02:03 <mgoddard> \o
15:02:09 <openstackgerrit> Michal Arbet proposed openstack/kolla-ansible master: Rename role haproxy to loadbalancer  https://review.opendev.org/c/openstack/kolla-ansible/+/770618
15:02:11 <openstackgerrit> Michal Arbet proposed openstack/kolla-ansible master: Add proxysql support for database  https://review.opendev.org/c/openstack/kolla-ansible/+/770215
15:02:14 <openstackgerrit> Michal Arbet proposed openstack/kolla-ansible master: Edit services roles to support database sharding  https://review.opendev.org/c/openstack/kolla-ansible/+/770216
15:02:18 <openstackgerrit> Michal Arbet proposed openstack/kolla-ansible master: [CI] Enable proxysql test  https://review.opendev.org/c/openstack/kolla-ansible/+/770621
15:02:33 <yoctozepto> \o/
15:02:47 <priteau> o/
15:03:09 <yoctozepto> meeting now mnasiadka hrw egonzalez rafaelweingartne cosmicsound osmanlicilegi bbezak
15:03:18 <mnasiadka> o/
15:03:29 <bbezak> o/
15:03:41 <kevko> yoctozepto: better now , isn't it ?
15:04:38 <yoctozepto> kevko: probably but we are in a meeting now
15:04:46 <kevko> ah, ok
15:04:49 <yoctozepto> ;-)
15:04:50 <kevko> maybe i connect also
15:04:55 <kevko> *join
15:04:58 <yoctozepto> please do
15:05:05 <yoctozepto> *this* is the meeting
15:05:25 <mgoddard> #topic agenda
15:05:35 <mgoddard> * Roll-call
15:05:37 <mgoddard> * Announcements
15:05:39 <mgoddard> ** Victoria released!
15:05:41 <mgoddard> * Review action items from the last meeting
15:05:43 <mgoddard> * CI status
15:05:45 <mgoddard> * Hot issues
15:05:47 <mgoddard> ** Dockerhub pull rate limits https://etherpad.opendev.org/p/docker-pull-limits
15:05:49 <mgoddard> ** Cinder active/active https://bugs.launchpad.net/kolla-ansible/+bug/1904062
15:05:51 <openstack> Launchpad bug 1904062 in kolla-ansible wallaby "external ceph cinder volume config breaks volumes on ussuri upgrade" [High,In progress] - Assigned to Michal Nasiadka (mnasiadka)
15:05:51 <mgoddard> ** CentOS 8.3 & stream https://lists.centos.org/pipermail/centos-devel/2020-December/075451.html
15:05:53 <mgoddard> * Wallaby release planning
15:05:55 <mgoddard> ** PTG actions
15:05:57 <mgoddard> ** Guidance on non-uniform features: https://review.opendev.org/c/openstack/kolla/+/759855
15:05:59 <mgoddard> * Kayobe operator feedback from Tu Wien
15:06:01 <mgoddard> #topic announcements
15:06:06 <mgoddard> #info Victoria released!
15:06:33 <mgoddard> #link http://lists.openstack.org/pipermail/openstack-discuss/2021-January/019774.html
15:06:37 <mgoddard> Any others?
15:08:07 <mgoddard> #topic Review action items from the last meeting
15:08:27 <mgoddard> mgoddard send email to openstack-discuss about meeting cancellations
15:08:29 <mgoddard> done
15:08:35 <mgoddard> #topic CI status
15:09:14 <mgoddard> generally seems reasonably happy for now
15:09:32 <yoctozepto> indeed, today infra has mirror issues
15:09:54 <yoctozepto> but this is quite mild
15:09:57 <mnasiadka> todays publish job although vanished without any logs - allegedly success
15:10:04 <mnasiadka> but images were not published to docker hub
15:10:13 <mgoddard> you read the logs daily?
15:10:16 <yoctozepto> that bad
15:10:29 <mnasiadka> mgoddard: not daily, tried to find out why images are 3 days old
15:10:40 <yoctozepto> mgoddard: you don't? :D
15:10:47 <mgoddard> ah
15:11:00 <yoctozepto> joking aside, I thought this meeting is supposed to be time for us to check for things like that :P
15:11:07 <yoctozepto> so mnasiadka ++
15:11:35 <mnasiadka> if it happens again tomorrow, I'm going to bug infra :)
15:12:07 <yoctozepto> but yesterday's look fine
15:12:09 <openstackgerrit> Michal Arbet proposed openstack/kolla-ansible master: Add proxysql support for database  https://review.opendev.org/c/openstack/kolla-ansible/+/770215
15:12:11 <openstackgerrit> Michal Arbet proposed openstack/kolla-ansible master: Edit services roles to support database sharding  https://review.opendev.org/c/openstack/kolla-ansible/+/770216
15:12:12 <openstackgerrit> Michal Arbet proposed openstack/kolla-ansible master: [CI] Enable proxysql test  https://review.opendev.org/c/openstack/kolla-ansible/+/770621
15:12:15 <yoctozepto> the images should be from yesterday
15:12:45 <mgoddard> ok, keep an eye on the publishing jobs then
15:12:48 <mgoddard> #topic Dockerhub pull rate limits https://etherpad.opendev.org/p/docker-pull-limits
15:13:16 <mgoddard> Limits still exist, still no mitigation
15:14:03 <mnasiadka> did we change to use pull through cache on Kolla jobs?
15:14:17 <yoctozepto> I did not so I assume not?
15:14:18 <yoctozepto> :D
15:14:46 <mgoddard> no, and it actually hit our victoria tagged release publish job :)
15:15:10 <mnasiadka> Ok, I can look into this tomorrow, if somebody gives me pointers :)
15:15:40 <yoctozepto> let's indeed split the workload on this and decide on steps
15:16:22 <yoctozepto> I have sincere progress in the cinder issue so am open to giving this one to mnasiadka
15:16:35 <yoctozepto> let's find the latest notes
15:16:46 <yoctozepto> #link https://etherpad.opendev.org/p/docker-pull-limits
15:19:14 <yoctozepto> uhm, anyone here with me?
15:19:37 <mgoddard> yes
15:20:02 <mnasiadka> I thought we're done with this topic to be honest :)
15:20:14 <mgoddard> we can easily do the registry-mirrors config for kolla
15:20:27 <mgoddard> that plugs most of the holes
15:20:54 <yoctozepto> so action for mnasiadka and "we'll see"? :D
15:20:55 <mgoddard> at which point our options are to switch registries or use the infra registry
15:21:21 <yoctozepto> ok, then let's start lightweight
15:21:47 <mgoddard> ok
15:22:07 <mgoddard> #action mnasiadka configure registry mirror for kolla CI
15:22:18 <mgoddard> #topic Cinder active/active https://bugs.launchpad.net/kolla-ansible/+bug/1904062
15:22:21 <openstack> Launchpad bug 1904062 in kolla-ansible wallaby "external ceph cinder volume config breaks volumes on ussuri upgrade" [High,In progress] - Assigned to Michal Nasiadka (mnasiadka)
15:22:27 <mgoddard> yoctozepto: you say you have some progress?
15:22:37 <yoctozepto> yes, in gerrit already
15:22:42 <yoctozepto> and some local testing
15:22:55 <yoctozepto> redis as coordinator seems quite sane
15:23:20 <yoctozepto> so regarding coordination the minimum is to improve docs in that regard, post a reno and to mailing list
15:23:36 <yoctozepto> for the cinder `cluster` - it works without hassle for ceph
15:23:51 <yoctozepto> the volumes get migrated to the "clusterised" version
15:24:02 <yoctozepto> now unto open issues
15:24:06 <mnasiadka> what about volumes in "-ing" state, like creating?
15:24:35 <yoctozepto> mnasiadka: that's a very good question - I have had all them stable when changing the config
15:24:45 <mnasiadka> unless we just cover that with a reno, or add a check
15:24:57 <yoctozepto> and the open issues are:
15:25:13 <yoctozepto> 1) backend_host is not that easy to undo
15:25:24 <yoctozepto> but we should undo it
15:25:32 <yoctozepto> (per the ml discussion)
15:25:43 <mnasiadka> undo as in remove the agent from the db?
15:26:02 <yoctozepto> yes, have all volumes be "served" from "real" hosts
15:26:15 <mnasiadka> ah right, cinder-manage to reassign
15:26:29 <yoctozepto> yup, and check whether it works 100% happily or not
15:26:33 <yoctozepto> but that's the next step
15:26:36 <yoctozepto> the other issue is
15:26:53 <yoctozepto> 2) what about drivers that do not support active-active?
15:26:57 <yoctozepto> what do we know about them?
15:27:08 <yoctozepto> are they happy when we just slap coordination and "cluster" all around
15:27:20 <yoctozepto> do note "cluster" is per cinder-volume daemon, not backend/driver
15:27:28 <yoctozepto> (same coordination)
15:27:42 <mgoddard> generally, AFAIU, most drivers do not support active/active
15:27:54 <yoctozepto> do note also we were always doing it likely wrong due to no proper active-passice
15:27:57 <yoctozepto> passive*
15:27:59 <yoctozepto> mgoddard: yes
15:28:22 <yoctozepto> so the other issue actually boils down we don't have proper active-passive
15:28:27 <yoctozepto> though users never complained :-)
15:28:41 <eliaswimmer> https://bugs.launchpad.net/kolla-ansible/+bug/1905542 same here for manila
15:28:42 <openstack> Launchpad bug 1905542 in kolla-ansible "Manila ceph configuration won't work in HA mode" [Undecided,New]
15:28:43 <yoctozepto> or cinder is good enough 99% of the time
15:30:06 <yoctozepto> eliaswimmer: good point; I have confirmed our redis coordination should be fine so it is a similar matter for manila
15:31:17 <mnasiadka> yoctozepto: we never had proper active/passive, so we need a word in the docs we only support active/active
15:32:20 <yoctozepto> mnasiadka: sure, that's precisely my point for now
15:34:28 <mnasiadka> good work :)
15:34:34 <mgoddard> +1
15:34:37 <mgoddard> so, plan?
15:34:43 <mgoddard> start with setting cluster
15:35:04 <mgoddard> document coordination
15:35:20 <yoctozepto> thanks :-)
15:35:29 <mgoddard> then fiddle with backend_host?
15:35:41 <yoctozepto> yes, let me write that down somewhere
15:35:46 <yoctozepto> in the whiteboard
15:35:47 <mgoddard> ok
15:36:06 <mgoddard> will you be working on it yoctozepto?
15:36:07 <mnasiadka> about coordination, I remember we test only redis as backend in the CI?
15:36:10 <yoctozepto> mgoddard: yes
15:36:14 <mgoddard> great
15:36:17 <yoctozepto> mnasiadka: we don't at all
15:36:25 <yoctozepto> mnasiadka: I am starting this
15:36:34 <yoctozepto> well, started, as WIP
15:36:42 <mnasiadka> ok, but redis should be less flakey, etcd will surely give us some headaches on slow disk setups
15:36:54 <yoctozepto> etcd is not HA
15:37:01 <mgoddard> we should fix that
15:37:05 <yoctozepto> needs some serious love
15:37:16 <mnasiadka> ok, I'll add it to my list
15:37:17 <yoctozepto> etcd3gw also needs love
15:37:23 <yoctozepto> I am a "core" in there as well
15:37:42 <mgoddard> ok, let's keep going
15:37:50 <yoctozepto> ok, I'll write it down for you to see
15:37:53 <mgoddard> thanks
15:37:56 <mgoddard> #topic CentOS 8.3 & stream https://lists.centos.org/pipermail/centos-devel/2020-December/075451.html
15:38:02 <mgoddard> Do we have much more to say about this?
15:38:10 <yoctozepto> kill it with fireeeeeeeeeeeeee
15:38:20 <yoctozepto> nothing for now
15:38:23 <mnasiadka> deprecate centos :)
15:38:27 <mgoddard> ok, will remove from agenda
15:38:33 <yoctozepto> I have not checked the "free rhel" news
15:38:39 <mgoddard> #topic PTG actions
15:39:00 <mnasiadka> yoctozepto had a change for testing bootstrap/deploy with Stream, and we can't really do anything more now.
15:39:04 <mgoddard> #link https://etherpad.opendev.org/p/kolla-wallaby-ptg
15:39:05 <yoctozepto> [this is regarding Wallaby]
15:39:09 <mgoddard> L1006
15:39:35 <yoctozepto> does not smell like me
15:40:10 <mnasiadka> I would rather ask why we need to justify not using k8s?
15:40:17 <mgoddard> I don't think any more TODOs have been done
15:40:29 <mnasiadka> if somebody wants to use k8s, let him go to openstack-helm
15:40:46 <yoctozepto> mnasiadka: but we should just summarise why it
15:40:49 <yoctozepto> is docker
15:41:03 <mnasiadka> well, let's support podman, and then we don't have to justify
15:41:09 <yoctozepto> perhaps a simple faq
15:41:19 <yoctozepto> duh, then you still justify but two
15:41:25 <mgoddard> #topic Guidance on non-uniform features: https://review.opendev.org/c/openstack/kolla/+/759855
15:41:45 <yoctozepto> mgoddard, our hero, ended this pointless discussion
15:42:14 <yoctozepto> I have added this particular point
15:42:18 <yoctozepto> so it's kevko's proposal
15:42:22 <mgoddard> arguing over TODOs that will never get done :D
15:42:27 <yoctozepto> I added it before it turned out to be problematic
15:42:32 <yoctozepto> mgoddard: precisely! :D
15:42:47 <yoctozepto> problematic for other reasons
15:42:54 <yoctozepto> (fluent lagging behind u-c deps)
15:43:06 <yoctozepto> but anyhow, what's our stance on such features
15:43:19 <mgoddard> I'm going to go with 'it depends'
15:43:27 <yoctozepto> thank you, that explains everything
15:43:43 <yoctozepto> i.e. as soon as one secures +2 before any -2
15:43:49 <yoctozepto> 2 +2 *
15:44:13 <mgoddard> for small features, or non-core projects, non-uniformity is ok
15:44:45 <mgoddard> for core parts of kolla (which arguably this would be), I would strongly prefer to have broad support
15:44:55 <yoctozepto> well, we have non-uniformity at very high level with entire projects from binary
15:45:06 <yoctozepto> hmm, but still, this is something optional
15:45:13 <yoctozepto> but nice to have
15:45:25 <yoctozepto> otoh, this particular does not look well-loved upstream nor downstream
15:47:05 <mgoddard> it also depends on the reason
15:47:17 <mgoddard> if it is a fundamental limitation, that is easier to accept
15:47:29 <yoctozepto> mgoddard: current parsing is fugly
15:47:39 <yoctozepto> this is the "right way"
15:47:48 <yoctozepto> but seemingly not popular enough still
15:47:56 <mgoddard> I mean the reason for non-uniformity
15:48:10 <yoctozepto> ah
15:48:21 <mgoddard> if it's the right way, shouldn't we support it everywhere if possible?
15:48:22 <mnasiadka> well, for now there's nothing to talk about really - the ,,tool'' is using too old msgpack
15:48:24 <yoctozepto> well, this here is low popularity and lack of packaging
15:48:29 <mnasiadka> to be included anywhere
15:48:36 <yoctozepto> mnasiadka: indeed
15:48:43 <mnasiadka> once this is fixed, it needs to be packaged properly
15:48:47 <yoctozepto> kevko is checking whether this is only superficial
15:48:52 <mnasiadka> and then we can merge it
15:49:02 <yoctozepto> will be hard for ubuntu
15:49:10 <yoctozepto> or perhaps it is packaged but by uca?
15:49:18 <yoctozepto> I have not checked
15:49:23 <yoctozepto> though I doubt it
15:49:44 <mnasiadka> yoctozepto: that's the reason for pinning - https://github.com/fluent/fluent-logger-python/issues/157
15:49:44 <mgoddard> sounds like we have a few unknowns to work through
15:49:48 <mnasiadka> so it's not trivial
15:49:53 <mgoddard> let's move on and come back to it
15:49:59 <yoctozepto> it's not
15:50:35 <yoctozepto> mnasiadka: "short term solution"
15:50:45 <mgoddard> #topic Kayobe operator feedback from Tu Wien
15:50:52 <mgoddard> eliaswimmer: hi
15:51:21 <eliaswimmer> hi
15:51:29 <eliaswimmer> there is not so much to say
15:51:33 <mgoddard> Thanks for agreeing to share your feedback
15:51:45 <eliaswimmer> one thing is the manila active active setup I am using
15:52:01 <eliaswimmer> the other thing is centos :)
15:52:17 <eliaswimmer> but for now we will wait I guess
15:52:50 <eliaswimmer> some things I would like to see is networkmanager support, but maybe this is just a kayobe issue
15:54:06 <eliaswimmer> and disabling firewalld is also quite annoying, as I have a system that is partly exposed to the public
15:55:03 <mgoddard> ok, let's work through it
15:55:16 <eliaswimmer> Ah and also for kayobe, I would like to see kolla on kolla :)
15:55:18 <mgoddard> manila active/active issue is https://bugs.launchpad.net/kolla-ansible/+bug/1905542 ?
15:55:19 <openstack> Launchpad bug 1905542 in kolla-ansible "Manila ceph configuration won't work in HA mode" [Undecided,New]
15:55:45 <yoctozepto> redis coordination should work in here as well, just need to try
15:55:52 <yoctozepto> no idea about other manila bolts for it
15:56:10 <mgoddard> have you tried either of the suggestions in the bug report - configure coordination and use separate Ceph keys?
15:56:14 <openstackgerrit> Merged openstack/kolla-ansible stable/victoria: Fix dpdk deploy failed  https://review.opendev.org/c/openstack/kolla-ansible/+/770506
15:56:17 <yoctozepto> I just know that when I repeat "manila" a few times here, then manila angels fall from heavens and pick up questions
15:56:57 <mnasiadka> tbarron is the Manila angel usually ;)
15:57:12 <eliaswimmer> mgoddard: not yet
15:57:43 <mgoddard> ok. Would be interesting to hear if it works, especially just coordination which doesn't require code changes
15:58:06 <eliaswimmer> It was just before xmas when I stumbled upon it
15:58:28 <eliaswimmer> ok, I will try it and report my results
15:58:36 <tbarron> yoctozepto: beelzebub
15:58:39 <mgoddard> talking to the manila folks would be a good idea
15:58:58 <mgoddard> as if by magic
15:59:10 <mgoddard> tbarron: we are discussing https://bugs.launchpad.net/kolla-ansible/+bug/1905542
15:59:11 <openstack> Launchpad bug 1905542 in kolla-ansible "Manila ceph configuration won't work in HA mode" [Undecided,New]
15:59:40 <yoctozepto> tbarron: sorry, there is already an anime with that title: https://en.wikipedia.org/wiki/Beelzebub_(manga) try again
15:59:46 <yoctozepto> :D
16:01:05 <mgoddard> ok, out of time for the meeting. We can continue to discuss kayobe feedback if you have time eliaswimmer
16:01:19 <mgoddard> Thanks all!
16:01:22 <mgoddard> #endmeeting