15:01:04 <mgoddard> #startmeeting kolla
15:01:05 <openstack> Meeting started Wed Feb 26 15:01:04 2020 UTC and is due to finish in 60 minutes.  The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:06 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:01:08 <mgoddard> #topic rollcall
15:01:09 <openstack> The meeting name has been set to 'kolla'
15:01:12 <mgoddard> \o
15:01:19 <osmanlicilegi> o/
15:01:21 <yoctozepto> o/
15:01:21 <openstackgerrit> Michal Nasiadka proposed openstack/kolla-ansible master: Add /run/netns bindmount to Neutron containers  https://review.opendev.org/710051
15:01:23 <hrw> /o/
15:01:29 <mnasiadka> o/
15:01:50 <hrw> \o\ /°\
15:02:30 <osmanlicilegi> \o/\o/\o/
15:03:20 <mgoddard> #topic agenda
15:03:40 <mgoddard> * Roll-call
15:03:42 <mgoddard> * Announcements
15:03:44 <mgoddard> * Review action items from last meeting
15:03:46 <mgoddard> * CI status
15:03:48 <mgoddard> * Ussuri release planning (kolla & kolla ansible)
15:03:50 <mgoddard> * Ussuri release planning (kayobe)
15:03:52 <mgoddard> * Kolla SIG (aka Kolla Klub?) https://etherpad.openstack.org/p/kolla-sig
15:03:54 <mgoddard> #topic announcements
15:04:25 <mgoddard> #info Rocky will move to extended maintenance (EM) soon
15:04:43 <hrw> 24.02 was a date iirc
15:04:47 <mgoddard> Waiting for final rocky releases then we can bump versions and release our own final
15:05:08 <mgoddard> Any other announcements?
15:05:11 <yoctozepto> we finally tested to-instance network connectivity in CI
15:05:27 <yoctozepto> though that did not catch netns failure ;D
15:06:28 <mgoddard> yeah that's nice
15:06:38 <mgoddard> #topic Review action items from last meeting
15:06:47 <osmanlicilegi> yoctozepto: if ci catches everything, we couldn't have adventure :]
15:06:55 <mgoddard> mnasiadka request neutron 14.1.0 in stein UCA
15:06:57 <mgoddard> yoctozepto to remove kayobe ceph block device labelling support https://storyboard.openstack.org/#!/story/2007295
15:06:59 <mgoddard> jovial[m] to work on custom extension points
15:07:01 <mgoddard> dougsz to write bug report about nova SSH nproc issue
15:07:03 <mgoddard> mnasiadka: done?
15:07:24 <mnasiadka> mgoddard: complained, but they said they have their own testing regime and will take some time
15:07:32 <yoctozepto> regime
15:07:41 <yoctozepto> totalitarian I presume
15:07:41 <mnasiadka> whatever that means
15:07:50 <mgoddard> ok
15:08:05 <mgoddard> drop ubuntu
15:08:08 <yoctozepto> RIP Stein CI
15:08:23 <mgoddard> yoctozepto: done?
15:08:25 <mgoddard> yes
15:08:27 <mgoddard> thanks
15:08:36 <yoctozepto> yw
15:08:46 <mgoddard> jovial[m] is away, probably more of a long term thing
15:09:03 <openstackgerrit> Michal Nasiadka proposed openstack/kolla-ansible master: OVN Support  https://review.opendev.org/696841
15:09:49 <mgoddard> dougsz: you wrote the bug report didn't you?
15:10:37 <dougsz> ah sorry, still on my todo list, will do it today
15:10:53 <mgoddard> #action dougsz to write bug report about nova SSH nproc issue
15:10:59 <mgoddard> #topic CI status
15:11:29 <mgoddard> Looks green apart from stein ubuntu-binary
15:11:49 <yoctozepto> indeed
15:11:58 <mgoddard> I saw some weirdness with ubuntu+keystone on master earlier, hopefully it's my patch
15:12:06 <mgoddard> although I doubt it
15:12:16 <mgoddard> #topic Ussuri release planning (kolla & kolla ansible)
15:12:39 <mgoddard> I've been working on the centos8 train backport
15:12:43 <yoctozepto> mgoddard: all the more reason to depreacate ubuntu
15:13:05 <mgoddard> #link https://review.opendev.org/#/q/topic:bp/centos-rhel-8+status:open+branch:stable/train
15:13:14 <mgoddard> thanks for all reviews so far
15:13:41 <mgoddard> lots of approvals
15:13:57 <mgoddard> last few:
15:13:59 <mgoddard> https://review.opendev.org/709757
15:13:59 <patchbot> patch 709757 - kolla (stable/train) - Switch to python3 in bindep.txt - 3 patch sets
15:14:04 <mgoddard> https://review.opendev.org/709204
15:14:04 <patchbot> patch 709204 - kolla (stable/train) - CentOS 8: Use upstream Ceph/master - 4 patch sets
15:14:11 <mgoddard> https://review.opendev.org/709203
15:14:11 <patchbot> patch 709203 - kolla (stable/train) - CentOS 8: Remove shellinabox from ironic-conductor - 4 patch sets
15:14:49 <mgoddard> https://review.opendev.org/709718
15:14:50 <patchbot> patch 709718 - kolla-ansible (stable/train) - CI: Use python 3 for local kolla-ansible execution - 2 patch sets
15:14:54 <mgoddard> https://review.opendev.org/709717
15:14:54 <patchbot> patch 709717 - kolla-ansible (stable/train) - CI: Move ansible installation & configuration to A... - 1 patch set
15:15:15 <mgoddard> those last two need to merge before deploy jobs will pass - I was too lazy to add depends-on
15:16:20 <mgoddard> and this one will fix ironic jobs: https://review.opendev.org/709751
15:16:20 <patchbot> patch 709751 - kolla-ansible (stable/train) - CI: Use upper constraints when installing clients - 2 patch sets
15:16:32 * mgoddard stops begging for reviews
15:16:55 <mnasiadka> lol
15:16:57 <mgoddard> What other nice ussuri work should we discuss today?
15:17:23 <mnasiadka> I spoke with CentOS Storage SIG, it seems in Ussuri we will use Nautilus
15:17:47 <mnasiadka> Which is in a bit of contrary to what Sage said about CentOS 8 and Ceph release support
15:18:08 <mnasiadka> Will investigate that topic, since upstream Ceph repo doesn't have Nautilus on el8
15:18:13 <mgoddard> not sage adivce
15:18:19 <mgoddard> *advice
15:18:26 <mnasiadka> and CentOS Storage SIG seems to have it working
15:18:37 <mnasiadka> quite a nice desync in one company :)
15:18:41 <hrw> Octopus was not released yet iirc
15:18:58 <hrw> mnasiadka: which company you mean?
15:19:03 <mnasiadka> hrw: Red Hat
15:19:17 <hrw> mnasiadka: Ceph is not RH product
15:19:27 <mgoddard> they probably realised someone might want to run ceph on centos 8 before the middle of this year
15:20:05 <yoctozepto> probably
15:20:22 <mnasiadka> mgoddard: yeah, but still those packages are from CentOS Storage SIG, not Ceph upstream - but those were always built with different deps
15:20:26 <mgoddard> we have a patch to switch to nautilus, seems to work
15:21:09 <mnasiadka> Now that we don't have ceph-kolla, life should be easier - whatever the release of Ceph we are using
15:21:42 <yoctozepto> mnasiadka: a bit
15:21:50 <yoctozepto> still need to cater for right client libs
15:22:56 <yoctozepto> mgoddard: reviewed
15:22:58 <mnasiadka> ok, end of Ceph topic - I just want to make sure no weird bugs will happen and Ceph bug scrubbing team will tell "we don't support this"
15:22:59 <mnasiadka> ;-)
15:23:44 <mgoddard> well we can't release with master
15:23:59 <mgoddard> we can bump to octopus when available if we choose
15:24:28 <mnasiadka> well, Ussuri release is 13th May, Octopus release is 31st March
15:24:52 <mgoddard> generalfuzz: you around?
15:25:04 <generalfuzz> yes
15:25:48 <mgoddard> how is the backend tls work going?
15:26:54 <generalfuzz> I believe it is progressing nicely. I would like to get another set of reviews on the current patch - https://review.opendev.org/#/c/664516
15:26:54 <patchbot> patch 664516 - kolla-ansible - Add support for encrypting backend HAProxy traffic - 20 patch sets
15:27:12 <openstackgerrit> Michal Nasiadka proposed openstack/kolla-ansible master: Fix fernet bootstrap and key distribution - follow up  https://review.opendev.org/707080
15:27:31 <mgoddard> are you and yongjun bai communicating?
15:28:13 <generalfuzz> some
15:28:17 <mgoddard> I don't know if you've seen the patches coming in from them
15:28:26 <mgoddard> just want to make sure everyone's on the same page
15:29:08 <mgoddard> we've talked about trying to create common roles for some of this stuff to reduce duplication
15:29:11 <generalfuzz> I will send a note today. My goal is to have an agreed upon implementation in https://review.opendev.org/#/c/664516. Then we can split out the services
15:29:12 <patchbot> patch 664516 - kolla-ansible - Add support for encrypting backend HAProxy traffic - 20 patch sets
15:29:35 <mgoddard> makes sense
15:29:48 <generalfuzz> I'm going to look into a wsgi template generation task next
15:30:15 <mgoddard> ok
15:30:35 <mgoddard> quick poll: is a generic wsgi/apache config template worth doing?
15:30:54 <osmanlicilegi> +1
15:30:56 <mnasiadka> I thought about the same today
15:31:03 <yoctozepto> are we doing deprecated mod_wsgi now?
15:31:14 <mnasiadka> mod_wsgi is deprecated?
15:31:32 <yoctozepto> mnasiadka: some os services marked it not recommended
15:31:42 <mnasiadka> yoctozepto: and what is recommended?
15:31:46 <yoctozepto> mnasiadka: uwsgi is the way forward
15:31:46 <osmanlicilegi> uwsgi I think
15:31:58 <yoctozepto> it actually works in devstack
15:32:00 <yoctozepto> ;p
15:32:01 <generalfuzz> I was unable to get uwsgi to work with certs
15:32:10 <yoctozepto> hmm
15:32:10 <mnasiadka> yoctozepto: devstack is a buzzword, me don't believe
15:32:10 <mgoddard> and this wasn't mentioned because...
15:32:33 <yoctozepto> mnasiadka: trust me, I'm core ;p
15:33:03 <mnasiadka> yoctozepto: maybe I'm old fashioned, but can OpenStack make a statement on what is the direction? because I feel in next release they will say uwsgi is bad, and we should go to this shiny new tool
15:33:05 <mgoddard> should we be using uwsgi instead then?
15:33:26 <yoctozepto> generally yes, except for glance which wants to stay with its old eventlet
15:34:37 <mgoddard> ok, we have two separate things here
15:34:42 <mnasiadka> yoctozepto: and we have a change for glance to use mod_wsgi
15:34:44 <generalfuzz> I got exceptions when I configured services with uwsgi and defined the certs. I can revisit that today to pinpoint the exceptions
15:35:08 <yoctozepto> mnasiadka: block it
15:35:12 <mgoddard> 1. backend tls - general pattern, usage, etc
15:35:33 <mgoddard> 2. backend web server
15:35:52 <mgoddard> if 2 is contentious we can continue with 1
15:36:22 <yoctozepto> apache can do mod_proxy
15:36:25 <mgoddard> but let's not go adding mod_wsgi everywhere if its genuinely deprecated
15:36:28 <yoctozepto> generalfuzz: what broke with tls in uwsgi?
15:36:30 <mnasiadka> https://governance.openstack.org/tc/goals/selected/pike/deploy-api-in-wsgi.html#uwsgi-vs-mod-wsgi
15:36:45 <mnasiadka> (it's pike - but states devstack done move to uwsgi)
15:36:45 <mgoddard> I'm fairly sure OSA supports uwsgi
15:36:55 <yoctozepto> "with the intent that the mod_wsgi support is deleted from devstack in Queens."
15:36:57 <openstackgerrit> Merged openstack/kolla stable/train: CentOS 8: base and openstack-base images  https://review.opendev.org/709537
15:37:02 <mgoddard> #link https://docs.openstack.org/ansible-role-uwsgi/latest/
15:37:03 <yoctozepto> yeah, osa is uwsgi
15:37:26 <yoctozepto> or at least to some degree
15:37:27 <mgoddard> not in devstack doesn't mean deprecated
15:38:21 <generalfuzz> yoctozepto: there were python openssl exceptions. I will revisit today
15:39:58 <mgoddard> generalfuzz: would be interesting to see what you changed to get uwsgi going
15:40:38 <yoctozepto> mgoddard: right but if osa and devstack push towards uwsgi, then mod_wsgi becomes obsolete and can break
15:40:49 <mgoddard> what about tripleo?
15:41:13 <yoctozepto> also see: https://bugs.launchpad.net/neutron/+bug/1864418
15:41:14 <openstack> Launchpad bug 1864418 in neutron "has wrong with use apache to start neutron api in docker container" [Undecided,New]
15:41:33 <yoctozepto> this might be neutron behind mod_wsgi issue
15:41:52 <yoctozepto> mgoddard: good question
15:42:12 <openstackgerrit> Merged openstack/kolla stable/train: CentOS 8: Update packages in images  https://review.opendev.org/709202
15:42:35 <mgoddard> looks like quite a lot of wsgi in tripleo
15:42:48 <mgoddard> which means kolla images probably need to keep apache packages
15:42:57 <mgoddard> (unless we get them to override)
15:43:07 <generalfuzz> mgoddard: uwsgi is supported out of the box for nova. I'll need to look at Placement + keystone
15:43:33 <mgoddard> ok, sounds like more research required on wsgi vs. uwsgi
15:43:36 <yoctozepto> generalfuzz: keystone is uwsgi-only in osa
15:43:51 <mnasiadka> mgoddard: well, in theory we could support both
15:43:54 <yoctozepto> mhm, tripleo looks mod_wsgi-only
15:44:00 <generalfuzz> we may need a hybrid approach
15:44:02 <mgoddard> mnasiadka: no thanks :)
15:44:10 <yoctozepto> mgoddard: mnasiadka is right
15:44:20 <yoctozepto> stuffing both required parts in kolla is nobrainer
15:44:27 <mnasiadka> mgoddard: I think it will end up like this unfortunately
15:44:31 <mgoddard> why?
15:44:31 <yoctozepto> and we might want a phasing out approach
15:44:36 <yoctozepto> ^
15:44:42 <yoctozepto> as it may break any day
15:45:21 <mnasiadka> for now it works I guess, so it's not critical ;)
15:45:27 <mgoddard> indeed
15:45:34 <yoctozepto> agreed
15:45:44 <mnasiadka> more convenient would be to have some common template or role that unifies mod_wsgi configs
15:45:47 <mgoddard> generalfuzz: I would suggest not adding any more mod_wsgi configs for now :)
15:46:07 <generalfuzz> I will re-look at uwsgi with TLS.
15:46:09 <yoctozepto> well, those two kinda contradict each other
15:46:15 <mgoddard> well no point in a common mod_wsgi role if we move to uwsgi
15:46:48 <mgoddard> screw wsgi, I need a whiskey
15:47:24 <generalfuzz> Is adding ability to execute the container with the "root" user an acceptable solution?
15:47:24 <mgoddard> let's try to get backend tls polished and merged with just keystone support
15:47:26 <yoctozepto> wsgiey
15:47:56 <generalfuzz> mgoddard: I will remove the nova + placement for now
15:47:56 <yoctozepto> it seriously should not be necessary
15:48:51 <mnasiadka> around uwsgi - I just hope uwsgi version between distro is at least a bit consistent, looking at mod_wsgi versions that we have now (and have to use medieval configs due to CentOS)
15:48:52 <generalfuzz> yoctozepto: how can I have the container run the apache script with sudo from k-a code?
15:48:53 <mgoddard> on the root user - normally we change it in the container image
15:49:07 <mgoddard> USER root
15:49:12 <yoctozepto> ^
15:49:35 <mgoddard> question is whether this presents a transition problem for tripleo or k-a
15:50:01 <generalfuzz> so I should go into docker scripts in kolla as a related change?
15:50:04 <mgoddard> looks like tripleo might already use wsgi for now
15:50:08 <mgoddard> yes
15:50:42 <yoctozepto> or eventlet
15:50:45 <yoctozepto> hard to catch that
15:50:49 <mgoddard> I'd be interested to see what files the uwsgi config requires for TLS
15:50:52 <yoctozepto> as it's just running py script
15:50:57 <openstackgerrit> Merged openstack/kolla-ansible stable/train: CentOS 8: Support variable image tag suffix  https://review.opendev.org/709534
15:51:34 <mnasiadka> this looks like the easiest thing on earth: https://uwsgi-docs.readthedocs.io/en/latest/HTTPS.html
15:51:37 <mgoddard> i.e. if we merge backend tls support for wsgi, could we switch to uwsgi without any change in the user interface (config files)
15:51:39 <mnasiadka> So I'm pretty sure it doesn't work
15:52:49 <yoctozepto> --https 0.0.0.0:8443,foobar.crt,foobar.key
15:52:59 <yoctozepto> well, that pretty much explains what tls really is
15:53:13 <generalfuzz> mgoddard: config files would change, since we would declare cert/key in .conf file
15:53:19 <hrw> mgoddard: on my server I use nginx to wrap uwsgi with tls
15:53:53 <mgoddard> generalfuzz: that's fine - we control those
15:54:11 <mgoddard> looks like it's just a key and cert, same as wsgi
15:55:02 <yoctozepto> we always needs this at min
15:55:22 <yoctozepto> and then any shim to convert/glue to the required form would be sufficient
15:55:30 <yoctozepto> like that ugly haproxy habing key with cert
15:55:39 <yoctozepto> who came up with that
15:56:12 <mgoddard> does devstack use mod_uwsgi or uwsgi binary?
15:56:23 <ktibi> mgoddard, kayobe can auto resize lvm thinpool on seed if the disk is increase ?
15:56:31 <mgoddard> ktibi: meeting time
15:57:20 <mgoddard> ok, seems this has taken most of the meeting
15:57:38 <mgoddard> #topic Ussuri release planning (kayobe)
15:57:46 <yoctozepto> mgoddard: uwsgi
15:57:58 <mnasiadka> kayobe as wsgi app?
15:57:59 <yoctozepto> mgoddard: external binary
15:58:18 <mnasiadka> (just laughing to continue wsgi topic)
15:58:18 <yoctozepto> mnasiadka: yes, wsgify kayobe
15:58:26 <mgoddard> I don't have much to say other than I have more CentOS 8 patches for kayobe - reviews please dougsz & priteau :)
15:58:46 <mgoddard> uwsgi yoctozepto mnasiadka
15:58:46 <dougsz> :)
15:58:51 <priteau> will do
15:58:54 <mgoddard> thanks
15:59:01 <openstackgerrit> Marcin Juszkiewicz proposed openstack/kolla stable/train: Bump train versions  https://review.opendev.org/710067
15:59:15 * yoctozepto cannot be deployed as wsgi app under mnasiadka
15:59:15 <mgoddard> I put together a testing checklist for kayobe & centos 8
15:59:39 <mgoddard> https://etherpad.openstack.org/p/kolla-centos8
15:59:46 <mgoddard> please add to it if you think of anything else
16:00:01 <mgoddard> and if you have any time for testing please let me know
16:00:10 <mgoddard> (no doubt testing & fixing)
16:00:27 <mgoddard> we outta time
16:00:33 <mgoddard> thanks all
16:00:40 <mgoddard> #endmeeting