15:00:23 <mgoddard> #startmeeting kolla
15:00:24 <openstack> Meeting started Wed Mar 18 15:00:23 2020 UTC and is due to finish in 60 minutes.  The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:25 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:28 <openstack> The meeting name has been set to 'kolla'
15:01:18 <mgoddard> #topic rollcall
15:01:22 <mgoddard> \o
15:01:29 <osmanlicilegi> o/
15:01:50 <stackedsax> \o
15:02:11 <hrw> o/
15:02:13 <yoctozepto> o/
15:02:30 <yoctozepto> sorry for breaking the chain
15:02:31 <yoctozepto> :D
15:02:38 <generalfuzz> o/
15:02:51 <mgoddard> #topic agenda
15:02:57 <mgoddard> * Roll-call
15:02:59 <mgoddard> * Announcements
15:03:01 <mgoddard> * Review action items from last meeting
15:03:03 <mgoddard> * CI status
15:03:05 <mgoddard> * Kolla --SIG-- (aka Kolla Klub?) https://etherpad.openstack.org/p/kolla-sig
15:03:07 <mgoddard> * Ussuri release planning (kayobe)
15:03:09 <mgoddard> * Ussuri release planning (kolla & kolla ansible)
15:03:11 <mgoddard> * Stein/Train backports
15:03:13 <mgoddard> - https://bugs.launchpad.net/kolla/+bug/1867365 - on aarch64 we lack mariabackup in mariadb image. Train has it as we dropped Percona. Backporting to Stein == dropping Percona which makes a change on x86-64 or adding 'if !x86 then install mariadb-backup' block.
15:03:14 <openstack> Launchpad bug 1867365 in kolla train "nova-libvirt needs fixed libvirt on AArch64 servers with ThunderX cpu" [High,Fix committed] - Assigned to Marcin Juszkiewicz (hrw)
15:03:15 <mgoddard> #topic announcements
15:03:28 <hrw> mariadb part handled by https://review.opendev.org/713647 basically
15:03:28 <patchbot> patch 713647 - kolla (stable/stein) - mariadb: install mariabackup on Debian - 1 patch set
15:03:37 <mgoddard> #info Kolla CentOS 8 images now tagged as master on master
15:03:47 <yoctozepto> except horizon
15:03:52 <mgoddard> true
15:04:02 <yoctozepto> but it looks worky
15:04:03 <mgoddard> Any other announcements?
15:04:32 <mgoddard> #topic Review action items from last meeting
15:04:43 <mgoddard> mgoddard to raise a bug on Ceph job fails
15:04:45 <mgoddard> mnasiadka to look into Ceph job fails
15:04:54 <mgoddard> I did not do mine
15:05:02 <yoctozepto> but mnasiadka did his
15:05:07 <mnasiadka> Fixed
15:05:12 <yoctozepto> at least master got nice
15:05:22 <yoctozepto> worse for older branches still
15:05:25 <yoctozepto> so not fully
15:05:35 <mgoddard> yeah, I think this was about kolla ceph not ceph-ansible?
15:05:37 <mnasiadka> Yeah, need some more time on this
15:05:37 <yoctozepto> something went wrong regarding our ceph deploys
15:05:44 <yoctozepto> yeah, mgoddard ++
15:05:51 <mnasiadka> mgoddard: but c-a also broke :)
15:05:55 <mgoddard> indeed
15:05:56 <yoctozepto> indeed :D
15:06:13 <mgoddard> #action mgoddard to raise a bug on Ceph job fails
15:06:18 <mgoddard> #action mnasiadka to look into Ceph job fails
15:06:25 <mgoddard> #topic CI status
15:07:10 <mgoddard> We had a horizon issue, has the fix merged yet?
15:07:33 <hrw> yes
15:07:38 <hrw> master done, train in queue
15:07:46 <yoctozepto> train not broken
15:07:55 <yoctozepto> as we pin releases
15:07:59 <yoctozepto> so merge any time
15:08:09 <hrw> horizon merges pyscss2 into train...
15:08:21 <yoctozepto> hrw: but we have pinned releases
15:08:26 <hrw> so once we update we break
15:08:37 <yoctozepto> hrw: we will be blocked from updating :-)
15:08:46 <mgoddard> Otherwise we're good
15:08:52 <yoctozepto> seems so
15:08:56 <mgoddard> Kayobe seems green
15:09:00 <openstackgerrit> Grzegorz Sikorski proposed openstack/kolla-ansible stable/rocky: rabbitmq: update cluster version on reconfigure  https://review.opendev.org/713628
15:09:01 <hrw> yoctozepto: right
15:09:05 <yoctozepto> rocky nearing em
15:09:11 <yoctozepto> but folks sending patches ;D
15:09:12 <mgoddard> #topic Kolla --SIG-- (aka Kolla Klub?) https://etherpad.openstack.org/p/kolla-sig
15:09:27 <yoctozepto> let's kickstart this soon
15:09:30 <mgoddard> +1
15:09:40 <mgoddard> I think we have around 20 people signed up now
15:10:28 <mgoddard> #action mgoddard to spam people about kolla SIG
15:10:39 <mgoddard> Will try to get a few more signed up
15:10:54 <mgoddard> Then doodle poll for a slot for first meeting
15:11:03 <Fl1nt> can I join the SIG part of the meeting?
15:11:14 <hrw> Fl1nt: anyone can
15:11:17 <mgoddard> Fl1nt: anyone can join in the meeting
15:11:25 <Fl1nt> I'm in for this part so.
15:11:42 <mgoddard> great to hear
15:12:00 <cosmicsound> o/
15:12:24 <yoctozepto> Fl1nt: any part of the meeting :-) as long as on topic
15:12:35 <Fl1nt> Ok, noticed that.
15:13:13 <mgoddard> I don't think we have too much more to say on it at the moment - just need to get it started
15:13:43 <mgoddard> #topic Ussuri release planning (kayobe)
15:15:06 <mgoddard> priteau, dougsz: \o
15:15:17 <priteau> o/
15:15:27 <dougsz> \o/
15:15:54 <mgoddard> we are about 5-6 weeks from feature freeze
15:16:00 <openstackgerrit> Merged openstack/kolla-ansible stable/stein: Combined fluentd fixes  https://review.opendev.org/713519
15:16:02 <openstackgerrit> Merged openstack/kolla-ansible stable/rocky: Wait for MariaDB to be accessible via HAProxy  https://review.opendev.org/713500
15:16:18 <mgoddard> so we need to be finishing things off fairly soon
15:16:32 <mgoddard> we have quite a few patches in flight needing reviews
15:16:33 <dougsz> sorry, are we talking kayobe or kolla here?
15:16:50 <mgoddard> topic: Ussuri release planning (kayobe)
15:17:24 <dougsz> will look at review list later
15:17:33 <priteau> I saw you flagged some patches as Review-Priority, I will try to review ASAP
15:17:35 <mgoddard> thanks
15:17:50 <mgoddard> right, just trying to provide some guidance on priority
15:18:08 <mgoddard> do either of you have features you would like to be included in ussuri?
15:18:21 <mgoddard> will there be more cells work in kayobe?
15:18:26 <priteau> The nclu patch might be nice to have
15:18:37 <mgoddard> +1
15:19:38 <dougsz> not sure yet about cells, focus is on kolla-ansible patches
15:19:39 <yoctozepto> ( dougsz: yeah, I switched order to let you have more time in case we get too chatty )
15:20:03 <mgoddard> good thinking
15:21:05 <mgoddard> I think jovial was going to look at customising kayobe commands, but he's not here
15:21:25 <mgoddard> wow, summoned by name
15:21:42 <mgoddard> jovial[m]: hi, welcome back
15:22:02 <mgoddard> are you still planning to look at customising kayobe commands?
15:22:40 <jovial[m]> hi, apologies for missing the start, got carried away writing some docs!
15:23:01 <jovial[m]> yes, I am still keen on looking at that :)
15:23:27 <mgoddard> ok. As I mentioned earlier we are about 5-6 weeks from feature freeze
15:23:36 <jovial[m]> no progress to report yet though
15:23:37 <mgoddard> which will no doubt go quite quickly
15:23:51 <mgoddard> no problem, I know you've been busy
15:24:01 <mgoddard> Thanks, let's move onto kolla.
15:24:11 <mgoddard> #topic Ussuri release planning (kolla & kolla ansible)
15:24:32 <mgoddard> I'll kick off with CentOS 8
15:24:49 <mgoddard> I added some acceptance criteria to https://etherpad.openstack.org/p/kolla-centos8
15:25:10 <mgoddard> So we'll have a better idea of when we're finished
15:25:30 <mgoddard> Please read through and amend or add as necessary
15:25:55 <mgoddard> I think kolla is quite close now - just a few last images to tidy up
15:26:30 <mgoddard> kolla ansible also not too far off
15:26:57 <mgoddard> Documentation and release notes still required
15:27:03 <mgoddard> And some train backprots
15:27:55 <mgoddard> That's probably all I have to say on it
15:28:02 <mgoddard> Any thoughts?
15:28:10 <mgoddard> Anything missing from acceptance criteria?
15:29:11 * yoctozepto thinking
15:32:16 <jovial[m]> looks reasonable to me at first glance
15:32:47 <yoctozepto> ++
15:33:01 <yoctozepto> no more ideas, I reworded some parts and switched order
15:33:30 <yoctozepto> we are looking good
15:33:34 <openstackgerrit> Marcin Juszkiewicz proposed openstack/kolla master: Revert "openstack-base: pin setuptools < 46 to get horizon working"  https://review.opendev.org/713681
15:34:02 <mgoddard> cool
15:34:37 <mgoddard> Anyone interested in picking up any remaining CentOS 8 work?
15:35:53 <mgoddard> I thought that would make it go quiet
15:35:57 <mgoddard> ok, let's move on
15:36:07 <mgoddard> Any other Ussuri features we need to discuss?
15:36:31 <generalfuzz> I would like to have a review of https://review.opendev.org/#/c/712005 - first pass at backend TLS implemented for keystone using uwsgi web server
15:36:31 <patchbot> patch 712005 - kolla-ansible - Add support for encrypting backend HAProxy traffic - 9 patch sets
15:37:25 <mgoddard> that's a reasonable request
15:38:00 <mgoddard> what has been your experience of uwsgi so far generalfuzz ?
15:38:51 <generalfuzz> mostly fine and straight forward. There are a number of peer reset exceptions always show in the log, but functionality seems to work
15:39:27 <generalfuzz> the "peer reset" seems to be a common issue with uwsgi and tls
15:40:06 <generalfuzz> we will need to make uwsgi available in tripleo builds
15:40:49 <mgoddard> tripleo can provide config to remove the uwsgi package if they wish
15:41:01 <mgoddard> what is the cause of the peer reset error?
15:41:11 <yoctozepto> peer reset sounds bad
15:41:33 <generalfuzz> it does, but seems like many people hit this with no functionality issues
15:41:56 <generalfuzz> still looking into it
15:42:29 <generalfuzz> it doesn't happen when using http - only https
15:43:27 <yoctozepto> hmm
15:43:39 <generalfuzz> for example - https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_122/712005/9/check/kolla-ansible-ubuntu-source/12291c8/primary/logs/kolla/keystone/keystone-admin.txt
15:43:41 <mgoddard> I was thinking we should send something to the ML about experience with uwsgi etc
15:43:41 <generalfuzz> [uwsgi-http key:  client_addr: 192.0.2.1 client_port: 42150] hr_ssl_read(): Connection reset by peer [plugins/http/https.c line 393]
15:44:53 <yoctozepto> weird if only tls triggers it
15:44:59 <yoctozepto> uwsgi bug?
15:45:27 <generalfuzz> or possible configuration issue, but not an obvious one
15:45:48 <mgoddard> do we have a suitable package for uwsgi?
15:45:51 <yoctozepto> well, it blames peer
15:46:03 <yoctozepto> so could be clients doing something wrong with tls
15:46:54 <yoctozepto> https://extras.getpagespeed.com/release-el8-latest.rpm <- this looks bad bad bad
15:46:58 <Fl1nt> or the server doing something wrong and triggering the client to reset (something like improper handle and downgrade).
15:47:07 <stackedsax> would the tripleo build issue prevent generalfuzz' work from getting accepted?
15:47:13 <yoctozepto> Fl1nt: indeed
15:47:37 <generalfuzz> yoctozepto: that is for tripleo
15:48:02 <mgoddard> where does the package come from for non-tripleo?
15:48:08 <yoctozepto> generalfuzz: for binary in general you mean
15:48:19 <yoctozepto> mgoddard: for source from pypi
15:48:24 <yoctozepto> https://review.opendev.org/#/c/710879/17/docker/keystone/keystone-base/Dockerfile.j2
15:48:25 <patchbot> patch 710879 - kolla - Install uwsgi for Keystone - 17 patch sets
15:48:27 <yoctozepto> all look here
15:49:20 <generalfuzz> yoctozepto: rhel 8 doesn't have uwsgi in epel, so this is the workaround
15:49:56 <mgoddard> we don't need to worry too much about that
15:50:11 <generalfuzz> I will remove it then
15:50:25 <mgoddard> we don't really test RHEL on kolla-ansible
15:50:38 <yoctozepto> mgoddard: I think generalfuzz also meant centos8
15:50:49 <mgoddard> if base_distro == 'rhel'
15:51:05 <Fl1nt> @mgoddard, working with a restricted infra, such package is going to be a nightmare for companies with high security measures and containments. it won't pass any COTS.
15:51:22 <yoctozepto> mgoddard: ah, missed
15:51:27 <mgoddard> Fl1nt: because it is from EPEL?
15:52:26 <mgoddard> or do you mean https://extras.getpagespeed.com/release-el8-latest.rpm ?
15:52:35 <mgoddard> we can drop that
15:52:36 <Fl1nt> because the epel repo installer is from getpagespeed.com
15:53:50 <yoctozepto> 😂
15:54:06 <Fl1nt> it can be override, but still, having that kind of urls triggers a lot of alerts when screening the dockerfile template in here ^^ then I need to introspect them and either validate them with the SOC team etc.
15:54:21 <mgoddard> I mean we will drop that from the patch
15:54:36 <Fl1nt> oh ok cool ^^
15:54:39 <mgoddard> it's only for RHEL, and we don't really test RHEL for kolla ansible
15:55:05 <mgoddard> tripleo use httpd/mod_wsgi
15:55:18 <mgoddard> which of course means we would need both in our images :(
15:55:28 <mgoddard> any idea how big uwsgi and its dependencies are?
15:55:42 <yoctozepto> I'm a bit confused still
15:55:42 <Fl1nt> I personally don't use tripleO but RHEL+kolla with some customers requiring professional agreements etc.
15:55:47 <yoctozepto> I thought we disabled epel by default
15:56:19 <mgoddard> true, we do
15:56:30 <yoctozepto> then how come uwsgi gets installed 😂
15:57:14 <mgoddard> comes from delorean-master-testing
15:57:14 <yoctozepto> INFO:kolla.common.utils.keystone-base: uwsgi                    x86_642.0.18-4.el8                            delorean-master-testing     385 k
15:57:19 <mgoddard> https://3f2e3a89d291731c913a-0da77774ba1e870a646f19d85e727a87.ssl.cf2.rackcdn.com/710879/17/check/kolla-build-centos8-binary/7520d7e/kolla/build/keystone-base.log
15:57:27 <yoctozepto> mgoddard: yeah, I pasted it for all
15:57:43 <yoctozepto> so they do care about uwsgi
15:57:49 <mgoddard> seems so
15:58:03 <yoctozepto> just rhel not having it *yet*
15:58:03 <mgoddard> wonder why it is in RDO but not RHOSP
15:58:17 <yoctozepto> rhosp always behind
15:58:27 <yoctozepto> still, it's delorean for ussuri
15:58:39 <hrw> pyscss author responded ;D horizon developer and I may became comaintainers
15:59:21 <yoctozepto> dlrn-master-rhel8-deps                          1.4 MB/s | 2.4 MB     00:01
15:59:44 <yoctozepto> hrw: ++ but now that it moved to 2, are you going to move it back to no 2?
15:59:51 <hrw> yoctozepto: yes
16:00:03 <yoctozepto> looks like rhel8 also uses delorean so should be aligned... hmm
16:00:20 <mgoddard> rhel8 builds are broken generally atm
16:00:28 <mgoddard> Anyway, we're out of time
16:00:34 <mgoddard> Thanks all
16:00:47 <mgoddard> #action mgoddard to ask ML about uwsgi etc
16:00:50 <mgoddard> #endmeeting