15:00:23 #startmeeting kolla 15:00:24 Meeting started Wed Mar 18 15:00:23 2020 UTC and is due to finish in 60 minutes. The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:00:25 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:00:28 The meeting name has been set to 'kolla' 15:01:18 #topic rollcall 15:01:22 \o 15:01:29 o/ 15:01:50 \o 15:02:11 o/ 15:02:13 o/ 15:02:30 sorry for breaking the chain 15:02:31 :D 15:02:38 o/ 15:02:51 #topic agenda 15:02:57 * Roll-call 15:02:59 * Announcements 15:03:01 * Review action items from last meeting 15:03:03 * CI status 15:03:05 * Kolla --SIG-- (aka Kolla Klub?) https://etherpad.openstack.org/p/kolla-sig 15:03:07 * Ussuri release planning (kayobe) 15:03:09 * Ussuri release planning (kolla & kolla ansible) 15:03:11 * Stein/Train backports 15:03:13 - https://bugs.launchpad.net/kolla/+bug/1867365 - on aarch64 we lack mariabackup in mariadb image. Train has it as we dropped Percona. Backporting to Stein == dropping Percona which makes a change on x86-64 or adding 'if !x86 then install mariadb-backup' block. 15:03:14 Launchpad bug 1867365 in kolla train "nova-libvirt needs fixed libvirt on AArch64 servers with ThunderX cpu" [High,Fix committed] - Assigned to Marcin Juszkiewicz (hrw) 15:03:15 #topic announcements 15:03:28 mariadb part handled by https://review.opendev.org/713647 basically 15:03:28 patch 713647 - kolla (stable/stein) - mariadb: install mariabackup on Debian - 1 patch set 15:03:37 #info Kolla CentOS 8 images now tagged as master on master 15:03:47 except horizon 15:03:52 true 15:04:02 but it looks worky 15:04:03 Any other announcements? 15:04:32 #topic Review action items from last meeting 15:04:43 mgoddard to raise a bug on Ceph job fails 15:04:45 mnasiadka to look into Ceph job fails 15:04:54 I did not do mine 15:05:02 but mnasiadka did his 15:05:07 Fixed 15:05:12 at least master got nice 15:05:22 worse for older branches still 15:05:25 so not fully 15:05:35 yeah, I think this was about kolla ceph not ceph-ansible? 15:05:37 Yeah, need some more time on this 15:05:37 something went wrong regarding our ceph deploys 15:05:44 yeah, mgoddard ++ 15:05:51 mgoddard: but c-a also broke :) 15:05:55 indeed 15:05:56 indeed :D 15:06:13 #action mgoddard to raise a bug on Ceph job fails 15:06:18 #action mnasiadka to look into Ceph job fails 15:06:25 #topic CI status 15:07:10 We had a horizon issue, has the fix merged yet? 15:07:33 yes 15:07:38 master done, train in queue 15:07:46 train not broken 15:07:55 as we pin releases 15:07:59 so merge any time 15:08:09 horizon merges pyscss2 into train... 15:08:21 hrw: but we have pinned releases 15:08:26 so once we update we break 15:08:37 hrw: we will be blocked from updating :-) 15:08:46 Otherwise we're good 15:08:52 seems so 15:08:56 Kayobe seems green 15:09:00 Grzegorz Sikorski proposed openstack/kolla-ansible stable/rocky: rabbitmq: update cluster version on reconfigure https://review.opendev.org/713628 15:09:01 yoctozepto: right 15:09:05 rocky nearing em 15:09:11 but folks sending patches ;D 15:09:12 #topic Kolla --SIG-- (aka Kolla Klub?) https://etherpad.openstack.org/p/kolla-sig 15:09:27 let's kickstart this soon 15:09:30 +1 15:09:40 I think we have around 20 people signed up now 15:10:28 #action mgoddard to spam people about kolla SIG 15:10:39 Will try to get a few more signed up 15:10:54 Then doodle poll for a slot for first meeting 15:11:03 can I join the SIG part of the meeting? 15:11:14 Fl1nt: anyone can 15:11:17 Fl1nt: anyone can join in the meeting 15:11:25 I'm in for this part so. 15:11:42 great to hear 15:12:00 o/ 15:12:24 Fl1nt: any part of the meeting :-) as long as on topic 15:12:35 Ok, noticed that. 15:13:13 I don't think we have too much more to say on it at the moment - just need to get it started 15:13:43 #topic Ussuri release planning (kayobe) 15:15:06 priteau, dougsz: \o 15:15:17 o/ 15:15:27 \o/ 15:15:54 we are about 5-6 weeks from feature freeze 15:16:00 Merged openstack/kolla-ansible stable/stein: Combined fluentd fixes https://review.opendev.org/713519 15:16:02 Merged openstack/kolla-ansible stable/rocky: Wait for MariaDB to be accessible via HAProxy https://review.opendev.org/713500 15:16:18 so we need to be finishing things off fairly soon 15:16:32 we have quite a few patches in flight needing reviews 15:16:33 sorry, are we talking kayobe or kolla here? 15:16:50 topic: Ussuri release planning (kayobe) 15:17:24 will look at review list later 15:17:33 I saw you flagged some patches as Review-Priority, I will try to review ASAP 15:17:35 thanks 15:17:50 right, just trying to provide some guidance on priority 15:18:08 do either of you have features you would like to be included in ussuri? 15:18:21 will there be more cells work in kayobe? 15:18:26 The nclu patch might be nice to have 15:18:37 +1 15:19:38 not sure yet about cells, focus is on kolla-ansible patches 15:19:39 ( dougsz: yeah, I switched order to let you have more time in case we get too chatty ) 15:20:03 good thinking 15:21:05 I think jovial was going to look at customising kayobe commands, but he's not here 15:21:25 wow, summoned by name 15:21:42 jovial[m]: hi, welcome back 15:22:02 are you still planning to look at customising kayobe commands? 15:22:40 hi, apologies for missing the start, got carried away writing some docs! 15:23:01 yes, I am still keen on looking at that :) 15:23:27 ok. As I mentioned earlier we are about 5-6 weeks from feature freeze 15:23:36 no progress to report yet though 15:23:37 which will no doubt go quite quickly 15:23:51 no problem, I know you've been busy 15:24:01 Thanks, let's move onto kolla. 15:24:11 #topic Ussuri release planning (kolla & kolla ansible) 15:24:32 I'll kick off with CentOS 8 15:24:49 I added some acceptance criteria to https://etherpad.openstack.org/p/kolla-centos8 15:25:10 So we'll have a better idea of when we're finished 15:25:30 Please read through and amend or add as necessary 15:25:55 I think kolla is quite close now - just a few last images to tidy up 15:26:30 kolla ansible also not too far off 15:26:57 Documentation and release notes still required 15:27:03 And some train backprots 15:27:55 That's probably all I have to say on it 15:28:02 Any thoughts? 15:28:10 Anything missing from acceptance criteria? 15:29:11 * yoctozepto thinking 15:32:16 looks reasonable to me at first glance 15:32:47 ++ 15:33:01 no more ideas, I reworded some parts and switched order 15:33:30 we are looking good 15:33:34 Marcin Juszkiewicz proposed openstack/kolla master: Revert "openstack-base: pin setuptools < 46 to get horizon working" https://review.opendev.org/713681 15:34:02 cool 15:34:37 Anyone interested in picking up any remaining CentOS 8 work? 15:35:53 I thought that would make it go quiet 15:35:57 ok, let's move on 15:36:07 Any other Ussuri features we need to discuss? 15:36:31 I would like to have a review of https://review.opendev.org/#/c/712005 - first pass at backend TLS implemented for keystone using uwsgi web server 15:36:31 patch 712005 - kolla-ansible - Add support for encrypting backend HAProxy traffic - 9 patch sets 15:37:25 that's a reasonable request 15:38:00 what has been your experience of uwsgi so far generalfuzz ? 15:38:51 mostly fine and straight forward. There are a number of peer reset exceptions always show in the log, but functionality seems to work 15:39:27 the "peer reset" seems to be a common issue with uwsgi and tls 15:40:06 we will need to make uwsgi available in tripleo builds 15:40:49 tripleo can provide config to remove the uwsgi package if they wish 15:41:01 what is the cause of the peer reset error? 15:41:11 peer reset sounds bad 15:41:33 it does, but seems like many people hit this with no functionality issues 15:41:56 still looking into it 15:42:29 it doesn't happen when using http - only https 15:43:27 hmm 15:43:39 for example - https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_122/712005/9/check/kolla-ansible-ubuntu-source/12291c8/primary/logs/kolla/keystone/keystone-admin.txt 15:43:41 I was thinking we should send something to the ML about experience with uwsgi etc 15:43:41 [uwsgi-http key: client_addr: 192.0.2.1 client_port: 42150] hr_ssl_read(): Connection reset by peer [plugins/http/https.c line 393] 15:44:53 weird if only tls triggers it 15:44:59 uwsgi bug? 15:45:27 or possible configuration issue, but not an obvious one 15:45:48 do we have a suitable package for uwsgi? 15:45:51 well, it blames peer 15:46:03 so could be clients doing something wrong with tls 15:46:54 https://extras.getpagespeed.com/release-el8-latest.rpm <- this looks bad bad bad 15:46:58 or the server doing something wrong and triggering the client to reset (something like improper handle and downgrade). 15:47:07 would the tripleo build issue prevent generalfuzz' work from getting accepted? 15:47:13 Fl1nt: indeed 15:47:37 yoctozepto: that is for tripleo 15:48:02 where does the package come from for non-tripleo? 15:48:08 generalfuzz: for binary in general you mean 15:48:19 mgoddard: for source from pypi 15:48:24 https://review.opendev.org/#/c/710879/17/docker/keystone/keystone-base/Dockerfile.j2 15:48:25 patch 710879 - kolla - Install uwsgi for Keystone - 17 patch sets 15:48:27 all look here 15:49:20 yoctozepto: rhel 8 doesn't have uwsgi in epel, so this is the workaround 15:49:56 we don't need to worry too much about that 15:50:11 I will remove it then 15:50:25 we don't really test RHEL on kolla-ansible 15:50:38 mgoddard: I think generalfuzz also meant centos8 15:50:49 if base_distro == 'rhel' 15:51:05 @mgoddard, working with a restricted infra, such package is going to be a nightmare for companies with high security measures and containments. it won't pass any COTS. 15:51:22 mgoddard: ah, missed 15:51:27 Fl1nt: because it is from EPEL? 15:52:26 or do you mean https://extras.getpagespeed.com/release-el8-latest.rpm ? 15:52:35 we can drop that 15:52:36 because the epel repo installer is from getpagespeed.com 15:53:50 😂 15:54:06 it can be override, but still, having that kind of urls triggers a lot of alerts when screening the dockerfile template in here ^^ then I need to introspect them and either validate them with the SOC team etc. 15:54:21 I mean we will drop that from the patch 15:54:36 oh ok cool ^^ 15:54:39 it's only for RHEL, and we don't really test RHEL for kolla ansible 15:55:05 tripleo use httpd/mod_wsgi 15:55:18 which of course means we would need both in our images :( 15:55:28 any idea how big uwsgi and its dependencies are? 15:55:42 I'm a bit confused still 15:55:42 I personally don't use tripleO but RHEL+kolla with some customers requiring professional agreements etc. 15:55:47 I thought we disabled epel by default 15:56:19 true, we do 15:56:30 then how come uwsgi gets installed 😂 15:57:14 comes from delorean-master-testing 15:57:14 INFO:kolla.common.utils.keystone-base: uwsgi x86_642.0.18-4.el8 delorean-master-testing 385 k 15:57:19 https://3f2e3a89d291731c913a-0da77774ba1e870a646f19d85e727a87.ssl.cf2.rackcdn.com/710879/17/check/kolla-build-centos8-binary/7520d7e/kolla/build/keystone-base.log 15:57:27 mgoddard: yeah, I pasted it for all 15:57:43 so they do care about uwsgi 15:57:49 seems so 15:58:03 just rhel not having it *yet* 15:58:03 wonder why it is in RDO but not RHOSP 15:58:17 rhosp always behind 15:58:27 still, it's delorean for ussuri 15:58:39 pyscss author responded ;D horizon developer and I may became comaintainers 15:59:21 dlrn-master-rhel8-deps 1.4 MB/s | 2.4 MB 00:01 15:59:44 hrw: ++ but now that it moved to 2, are you going to move it back to no 2? 15:59:51 yoctozepto: yes 16:00:03 looks like rhel8 also uses delorean so should be aligned... hmm 16:00:20 rhel8 builds are broken generally atm 16:00:28 Anyway, we're out of time 16:00:34 Thanks all 16:00:47 #action mgoddard to ask ML about uwsgi etc 16:00:50 #endmeeting