15:01:04 #startmeeting kolla 15:01:05 Meeting started Wed Feb 26 15:01:04 2020 UTC and is due to finish in 60 minutes. The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:01:06 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:01:08 #topic rollcall 15:01:09 The meeting name has been set to 'kolla' 15:01:12 \o 15:01:19 o/ 15:01:21 o/ 15:01:21 Michal Nasiadka proposed openstack/kolla-ansible master: Add /run/netns bindmount to Neutron containers https://review.opendev.org/710051 15:01:23 /o/ 15:01:29 o/ 15:01:50 \o\ /°\ 15:02:30 \o/\o/\o/ 15:03:20 #topic agenda 15:03:40 * Roll-call 15:03:42 * Announcements 15:03:44 * Review action items from last meeting 15:03:46 * CI status 15:03:48 * Ussuri release planning (kolla & kolla ansible) 15:03:50 * Ussuri release planning (kayobe) 15:03:52 * Kolla SIG (aka Kolla Klub?) https://etherpad.openstack.org/p/kolla-sig 15:03:54 #topic announcements 15:04:25 #info Rocky will move to extended maintenance (EM) soon 15:04:43 24.02 was a date iirc 15:04:47 Waiting for final rocky releases then we can bump versions and release our own final 15:05:08 Any other announcements? 15:05:11 we finally tested to-instance network connectivity in CI 15:05:27 though that did not catch netns failure ;D 15:06:28 yeah that's nice 15:06:38 #topic Review action items from last meeting 15:06:47 yoctozepto: if ci catches everything, we couldn't have adventure :] 15:06:55 mnasiadka request neutron 14.1.0 in stein UCA 15:06:57 yoctozepto to remove kayobe ceph block device labelling support https://storyboard.openstack.org/#!/story/2007295 15:06:59 jovial[m] to work on custom extension points 15:07:01 dougsz to write bug report about nova SSH nproc issue 15:07:03 mnasiadka: done? 15:07:24 mgoddard: complained, but they said they have their own testing regime and will take some time 15:07:32 regime 15:07:41 totalitarian I presume 15:07:41 whatever that means 15:07:50 ok 15:08:05 drop ubuntu 15:08:08 RIP Stein CI 15:08:23 yoctozepto: done? 15:08:25 yes 15:08:27 thanks 15:08:36 yw 15:08:46 jovial[m] is away, probably more of a long term thing 15:09:03 Michal Nasiadka proposed openstack/kolla-ansible master: OVN Support https://review.opendev.org/696841 15:09:49 dougsz: you wrote the bug report didn't you? 15:10:37 ah sorry, still on my todo list, will do it today 15:10:53 #action dougsz to write bug report about nova SSH nproc issue 15:10:59 #topic CI status 15:11:29 Looks green apart from stein ubuntu-binary 15:11:49 indeed 15:11:58 I saw some weirdness with ubuntu+keystone on master earlier, hopefully it's my patch 15:12:06 although I doubt it 15:12:16 #topic Ussuri release planning (kolla & kolla ansible) 15:12:39 I've been working on the centos8 train backport 15:12:43 mgoddard: all the more reason to depreacate ubuntu 15:13:05 #link https://review.opendev.org/#/q/topic:bp/centos-rhel-8+status:open+branch:stable/train 15:13:14 thanks for all reviews so far 15:13:41 lots of approvals 15:13:57 last few: 15:13:59 https://review.opendev.org/709757 15:13:59 patch 709757 - kolla (stable/train) - Switch to python3 in bindep.txt - 3 patch sets 15:14:04 https://review.opendev.org/709204 15:14:04 patch 709204 - kolla (stable/train) - CentOS 8: Use upstream Ceph/master - 4 patch sets 15:14:11 https://review.opendev.org/709203 15:14:11 patch 709203 - kolla (stable/train) - CentOS 8: Remove shellinabox from ironic-conductor - 4 patch sets 15:14:49 https://review.opendev.org/709718 15:14:50 patch 709718 - kolla-ansible (stable/train) - CI: Use python 3 for local kolla-ansible execution - 2 patch sets 15:14:54 https://review.opendev.org/709717 15:14:54 patch 709717 - kolla-ansible (stable/train) - CI: Move ansible installation & configuration to A... - 1 patch set 15:15:15 those last two need to merge before deploy jobs will pass - I was too lazy to add depends-on 15:16:20 and this one will fix ironic jobs: https://review.opendev.org/709751 15:16:20 patch 709751 - kolla-ansible (stable/train) - CI: Use upper constraints when installing clients - 2 patch sets 15:16:32 * mgoddard stops begging for reviews 15:16:55 lol 15:16:57 What other nice ussuri work should we discuss today? 15:17:23 I spoke with CentOS Storage SIG, it seems in Ussuri we will use Nautilus 15:17:47 Which is in a bit of contrary to what Sage said about CentOS 8 and Ceph release support 15:18:08 Will investigate that topic, since upstream Ceph repo doesn't have Nautilus on el8 15:18:13 not sage adivce 15:18:19 *advice 15:18:26 and CentOS Storage SIG seems to have it working 15:18:37 quite a nice desync in one company :) 15:18:41 Octopus was not released yet iirc 15:18:58 mnasiadka: which company you mean? 15:19:03 hrw: Red Hat 15:19:17 mnasiadka: Ceph is not RH product 15:19:27 they probably realised someone might want to run ceph on centos 8 before the middle of this year 15:20:05 probably 15:20:22 mgoddard: yeah, but still those packages are from CentOS Storage SIG, not Ceph upstream - but those were always built with different deps 15:20:26 we have a patch to switch to nautilus, seems to work 15:21:09 Now that we don't have ceph-kolla, life should be easier - whatever the release of Ceph we are using 15:21:42 mnasiadka: a bit 15:21:50 still need to cater for right client libs 15:22:56 mgoddard: reviewed 15:22:58 ok, end of Ceph topic - I just want to make sure no weird bugs will happen and Ceph bug scrubbing team will tell "we don't support this" 15:22:59 ;-) 15:23:44 well we can't release with master 15:23:59 we can bump to octopus when available if we choose 15:24:28 well, Ussuri release is 13th May, Octopus release is 31st March 15:24:52 generalfuzz: you around? 15:25:04 yes 15:25:48 how is the backend tls work going? 15:26:54 I believe it is progressing nicely. I would like to get another set of reviews on the current patch - https://review.opendev.org/#/c/664516 15:26:54 patch 664516 - kolla-ansible - Add support for encrypting backend HAProxy traffic - 20 patch sets 15:27:12 Michal Nasiadka proposed openstack/kolla-ansible master: Fix fernet bootstrap and key distribution - follow up https://review.opendev.org/707080 15:27:31 are you and yongjun bai communicating? 15:28:13 some 15:28:17 I don't know if you've seen the patches coming in from them 15:28:26 just want to make sure everyone's on the same page 15:29:08 we've talked about trying to create common roles for some of this stuff to reduce duplication 15:29:11 I will send a note today. My goal is to have an agreed upon implementation in https://review.opendev.org/#/c/664516. Then we can split out the services 15:29:12 patch 664516 - kolla-ansible - Add support for encrypting backend HAProxy traffic - 20 patch sets 15:29:35 makes sense 15:29:48 I'm going to look into a wsgi template generation task next 15:30:15 ok 15:30:35 quick poll: is a generic wsgi/apache config template worth doing? 15:30:54 +1 15:30:56 I thought about the same today 15:31:03 are we doing deprecated mod_wsgi now? 15:31:14 mod_wsgi is deprecated? 15:31:32 mnasiadka: some os services marked it not recommended 15:31:42 yoctozepto: and what is recommended? 15:31:46 mnasiadka: uwsgi is the way forward 15:31:46 uwsgi I think 15:31:58 it actually works in devstack 15:32:00 ;p 15:32:01 I was unable to get uwsgi to work with certs 15:32:10 hmm 15:32:10 yoctozepto: devstack is a buzzword, me don't believe 15:32:10 and this wasn't mentioned because... 15:32:33 mnasiadka: trust me, I'm core ;p 15:33:03 yoctozepto: maybe I'm old fashioned, but can OpenStack make a statement on what is the direction? because I feel in next release they will say uwsgi is bad, and we should go to this shiny new tool 15:33:05 should we be using uwsgi instead then? 15:33:26 generally yes, except for glance which wants to stay with its old eventlet 15:34:37 ok, we have two separate things here 15:34:42 yoctozepto: and we have a change for glance to use mod_wsgi 15:34:44 I got exceptions when I configured services with uwsgi and defined the certs. I can revisit that today to pinpoint the exceptions 15:35:08 mnasiadka: block it 15:35:12 1. backend tls - general pattern, usage, etc 15:35:33 2. backend web server 15:35:52 if 2 is contentious we can continue with 1 15:36:22 apache can do mod_proxy 15:36:25 but let's not go adding mod_wsgi everywhere if its genuinely deprecated 15:36:28 generalfuzz: what broke with tls in uwsgi? 15:36:30 https://governance.openstack.org/tc/goals/selected/pike/deploy-api-in-wsgi.html#uwsgi-vs-mod-wsgi 15:36:45 (it's pike - but states devstack done move to uwsgi) 15:36:45 I'm fairly sure OSA supports uwsgi 15:36:55 "with the intent that the mod_wsgi support is deleted from devstack in Queens." 15:36:57 Merged openstack/kolla stable/train: CentOS 8: base and openstack-base images https://review.opendev.org/709537 15:37:02 #link https://docs.openstack.org/ansible-role-uwsgi/latest/ 15:37:03 yeah, osa is uwsgi 15:37:26 or at least to some degree 15:37:27 not in devstack doesn't mean deprecated 15:38:21 yoctozepto: there were python openssl exceptions. I will revisit today 15:39:58 generalfuzz: would be interesting to see what you changed to get uwsgi going 15:40:38 mgoddard: right but if osa and devstack push towards uwsgi, then mod_wsgi becomes obsolete and can break 15:40:49 what about tripleo? 15:41:13 also see: https://bugs.launchpad.net/neutron/+bug/1864418 15:41:14 Launchpad bug 1864418 in neutron "has wrong with use apache to start neutron api in docker container" [Undecided,New] 15:41:33 this might be neutron behind mod_wsgi issue 15:41:52 mgoddard: good question 15:42:12 Merged openstack/kolla stable/train: CentOS 8: Update packages in images https://review.opendev.org/709202 15:42:35 looks like quite a lot of wsgi in tripleo 15:42:48 which means kolla images probably need to keep apache packages 15:42:57 (unless we get them to override) 15:43:07 mgoddard: uwsgi is supported out of the box for nova. I'll need to look at Placement + keystone 15:43:33 ok, sounds like more research required on wsgi vs. uwsgi 15:43:36 generalfuzz: keystone is uwsgi-only in osa 15:43:51 mgoddard: well, in theory we could support both 15:43:54 mhm, tripleo looks mod_wsgi-only 15:44:00 we may need a hybrid approach 15:44:02 mnasiadka: no thanks :) 15:44:10 mgoddard: mnasiadka is right 15:44:20 stuffing both required parts in kolla is nobrainer 15:44:27 mgoddard: I think it will end up like this unfortunately 15:44:31 why? 15:44:31 and we might want a phasing out approach 15:44:36 ^ 15:44:42 as it may break any day 15:45:21 for now it works I guess, so it's not critical ;) 15:45:27 indeed 15:45:34 agreed 15:45:44 more convenient would be to have some common template or role that unifies mod_wsgi configs 15:45:47 generalfuzz: I would suggest not adding any more mod_wsgi configs for now :) 15:46:07 I will re-look at uwsgi with TLS. 15:46:09 well, those two kinda contradict each other 15:46:15 well no point in a common mod_wsgi role if we move to uwsgi 15:46:48 screw wsgi, I need a whiskey 15:47:24 Is adding ability to execute the container with the "root" user an acceptable solution? 15:47:24 let's try to get backend tls polished and merged with just keystone support 15:47:26 wsgiey 15:47:56 mgoddard: I will remove the nova + placement for now 15:47:56 it seriously should not be necessary 15:48:51 around uwsgi - I just hope uwsgi version between distro is at least a bit consistent, looking at mod_wsgi versions that we have now (and have to use medieval configs due to CentOS) 15:48:52 yoctozepto: how can I have the container run the apache script with sudo from k-a code? 15:48:53 on the root user - normally we change it in the container image 15:49:07 USER root 15:49:12 ^ 15:49:35 question is whether this presents a transition problem for tripleo or k-a 15:50:01 so I should go into docker scripts in kolla as a related change? 15:50:04 looks like tripleo might already use wsgi for now 15:50:08 yes 15:50:42 or eventlet 15:50:45 hard to catch that 15:50:49 I'd be interested to see what files the uwsgi config requires for TLS 15:50:52 as it's just running py script 15:50:57 Merged openstack/kolla-ansible stable/train: CentOS 8: Support variable image tag suffix https://review.opendev.org/709534 15:51:34 this looks like the easiest thing on earth: https://uwsgi-docs.readthedocs.io/en/latest/HTTPS.html 15:51:37 i.e. if we merge backend tls support for wsgi, could we switch to uwsgi without any change in the user interface (config files) 15:51:39 So I'm pretty sure it doesn't work 15:52:49 --https 0.0.0.0:8443,foobar.crt,foobar.key 15:52:59 well, that pretty much explains what tls really is 15:53:13 mgoddard: config files would change, since we would declare cert/key in .conf file 15:53:19 mgoddard: on my server I use nginx to wrap uwsgi with tls 15:53:53 generalfuzz: that's fine - we control those 15:54:11 looks like it's just a key and cert, same as wsgi 15:55:02 we always needs this at min 15:55:22 and then any shim to convert/glue to the required form would be sufficient 15:55:30 like that ugly haproxy habing key with cert 15:55:39 who came up with that 15:56:12 does devstack use mod_uwsgi or uwsgi binary? 15:56:23 mgoddard, kayobe can auto resize lvm thinpool on seed if the disk is increase ? 15:56:31 ktibi: meeting time 15:57:20 ok, seems this has taken most of the meeting 15:57:38 #topic Ussuri release planning (kayobe) 15:57:46 mgoddard: uwsgi 15:57:58 kayobe as wsgi app? 15:57:59 mgoddard: external binary 15:58:18 (just laughing to continue wsgi topic) 15:58:18 mnasiadka: yes, wsgify kayobe 15:58:26 I don't have much to say other than I have more CentOS 8 patches for kayobe - reviews please dougsz & priteau :) 15:58:46 uwsgi yoctozepto mnasiadka 15:58:46 :) 15:58:51 will do 15:58:54 thanks 15:59:01 Marcin Juszkiewicz proposed openstack/kolla stable/train: Bump train versions https://review.opendev.org/710067 15:59:15 * yoctozepto cannot be deployed as wsgi app under mnasiadka 15:59:15 I put together a testing checklist for kayobe & centos 8 15:59:39 https://etherpad.openstack.org/p/kolla-centos8 15:59:46 please add to it if you think of anything else 16:00:01 and if you have any time for testing please let me know 16:00:10 (no doubt testing & fixing) 16:00:27 we outta time 16:00:33 thanks all 16:00:40 #endmeeting