15:00:19 <mgoddard> #startmeeting kolla
15:00:20 <openstack> Meeting started Wed Aug 14 15:00:19 2019 UTC and is due to finish in 60 minutes.  The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:22 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:25 <mgoddard> #topic rollcall
15:00:25 <openstack> The meeting name has been set to 'kolla'
15:00:34 <yoctozepto> o/
15:00:43 <mgoddard> \o
15:00:44 <yoctozepto> hrw, mnasiadka
15:00:45 <scottsol> o/
15:01:12 <chason> o/
15:01:16 <generalfuzz> o/
15:01:35 <priteau> \o
15:01:45 <dougsz> o/ (sort of - in another meeting)
15:03:18 <mgoddard> #topic agenda
15:03:28 <mgoddard> * Roll-call
15:03:31 <mgoddard> * Announcements
15:03:33 <mgoddard> * Review action items from last meeting
15:03:35 <mgoddard> * Kolla whiteboard https://etherpad.openstack.org/p/KollaWhiteBoard
15:03:37 <mgoddard> * Kayobe Stein release status
15:03:39 <mgoddard> * Train release planning
15:03:41 <mgoddard> * Ceph ansible migration
15:03:43 <mgoddard> * Kolla Ansible TLS Internal API
15:03:45 <mgoddard> #topic announcements
15:03:53 <mgoddard> None from me. Anyone else?
15:04:34 <yoctozepto> no major
15:04:37 <mgoddard> #topic Review action items from last meeting
15:04:44 <mnasiadka> o/ (sorry for being late)
15:04:47 <mgoddard> mgoddard to ask infra about restarting gerrit
15:04:50 <mgoddard> mgoddard or someone else to check stable backports
15:05:09 <mgoddard> this week seemed to go very quickly, didn't do either
15:05:16 <mgoddard> #action mgoddard to ask infra about restarting gerrit
15:05:23 <mgoddard> #action mgoddard or someone else to check stable backports
15:05:32 <mgoddard> #topic Kolla whiteboard https://etherpad.openstack.org/p/KollaWhiteBoard
15:06:36 <mgoddard> How is CI looking
15:07:03 <yoctozepto> mgoddard: never been greener when it comes to master/stein
15:07:07 <yoctozepto> did not check others
15:07:40 <mgoddard> that's what I like to hear
15:08:31 <mgoddard> Is everyone keeping their feature status up to date in the whiteboard?
15:08:37 <mgoddard> (in the priority list)
15:09:06 <yoctozepto> mine yeah, no progress since last meeting
15:10:15 <mgoddard> #topic Train release planning
15:10:40 <mgoddard> We covered this in detail last time, no need to go over it again IMO
15:10:58 <mgoddard> Please all just keep in mind the train release schedule
15:11:07 <mgoddard> #link https://releases.openstack.org/train/schedule.html
15:11:19 <mgoddard> (we will lag by 2+ weeks)
15:11:31 <mgoddard> #topic Kayobe Stein release status
15:11:50 <mgoddard> priteau, dougsz: how are we looking?
15:12:11 <mgoddard> I'd like to cut a stable branch soon
15:12:49 <priteau> There are many patches that we could land soon
15:12:51 <mgoddard> quite a few patches with 1x +2 we could land before
15:13:23 <priteau> https://review.opendev.org/#/q/project:x/kayobe+is:open+branch:master+label:Code-Review%252B2
15:14:21 <mgoddard> ok, let's get those patches in this week
15:14:37 <openstackgerrit> Scott Solkhon proposed openstack/kolla-ansible master: Decrypt keyring files using ansible-vault  https://review.opendev.org/676441
15:14:49 <priteau> And also should we proceed with the rename before branching, so there is not too much to change in both master and stable/stein?
15:14:53 <mgoddard> if there are others without +2 we want in, please add review priority
15:15:09 <dougsz> sounds good, will try and spend some time on them later today
15:15:25 <mgoddard> priteau: https://review.opendev.org/#/c/674505/ ?
15:16:06 <mgoddard> I think we need to wait for infra to do the rename
15:16:18 <mgoddard> should only need to cherry pick that patch to stable/stein
15:17:34 <priteau> OK
15:18:06 <mgoddard> thanks dougsz
15:18:17 <mgoddard> #topic Ceph ansible migration
15:18:24 <mgoddard> mnasiadka: you're up
15:18:51 <mnasiadka> Well, I have started CI work and some docs work
15:19:11 <mnasiadka> Will continue next week (I’m off tomorrow and on Fri)
15:19:25 <mnasiadka> So I’ll have more to say/show on Wed
15:20:18 <mgoddard> Did you want to discuss the overall approach?
15:20:27 <mnasiadka> Yeah
15:20:35 <mnasiadka> We have two options:
15:20:56 <mnasiadka> 1) role with importong ceph-ansible roles like OSA does
15:21:17 <mnasiadka> 2) totally separate flow of ceph deployment
15:21:24 <goldyfruit> o/
15:21:56 <mnasiadka> mgoddard: currently I’m leaning towards 1), but would like to hear comments
15:22:01 <kplant> OSA's approach is better imo, it's nice to have the familiar interface. vars being the same as ceph-ansible, etc.
15:22:19 <kplant> importing what already works and people may be familiar with
15:22:22 <yoctozepto> (2) seems more like us though
15:22:40 <yoctozepto> "outsource what we can"
15:23:14 <yoctozepto> maybe we could get something inbetween
15:23:24 <mnasiadka> yoctozepto: it’s still outsourcing, just we need to run preparation tasks (which we need to do in 2) as well)
15:23:28 <yoctozepto> like orchestrating the basic set of services
15:23:39 <mgoddard> how much of ceph-ansible would we need to reimplement in order to just use the roles?
15:23:42 <mnasiadka> And then importing role by role from ceph-ansible
15:23:54 <yoctozepto> ok, this way...
15:24:22 <mnasiadka> mgoddard: generate vars, reuse our inventory and import roles in correct order
15:24:32 <yoctozepto> seems reasonable
15:24:58 <yoctozepto> you have my blessing with (1)
15:25:34 <mgoddard> would we execute these roles as part of the normal deploy?
15:25:35 <yoctozepto> unless it gets super clumsy, I will +2 it once done
15:26:01 <mgoddard> one change in the workflow I like about 2 is that it cleanly separates ceph and openstack
15:26:12 <mnasiadka> mgoddard: we can, or we make it separate command
15:26:18 <mnasiadka> Or we can do both...
15:26:22 <yoctozepto> mgoddard: yeah, I originally thought this way
15:26:27 <mgoddard> you can't accidentally modify your ceph cluster during an openstack upgrade
15:26:36 <yoctozepto> exactly ^
15:27:09 <yoctozepto> the very reason I have my ceph cluster deployed externally - more control, less worries
15:27:18 <mgoddard> I'm open to 1), but would like to see a PoC before committing
15:27:21 <yoctozepto> though people MAY like the simplicity
15:27:39 <openstackgerrit> Merged openstack/kolla-ansible master: Add missing Octavia policy file to Horizon  https://review.opendev.org/676176
15:27:45 <mgoddard> mnasiadka: do you have code you could throw at gerrit?
15:28:03 <yoctozepto> mnasiadka: ^ even not working
15:28:44 <priteau> How complex is each approach to maintain as Ceph-Ansible evolves?
15:28:53 <mgoddard> that's a good question
15:28:53 <openstackgerrit> Mark Goddard proposed openstack/kolla-ansible master: CI: Test accessing dashboard  https://review.opendev.org/676412
15:29:03 <mnasiadka> mgoddard: I have some, I can push but that’s halfway done
15:29:15 <mgoddard> 2 is very simple to maintain - we just point to ceph-ansible docs
15:31:02 <mgoddard> how will we access ceph-ansible code? git clone from within kolla-ansible?
15:31:07 <mnasiadka> And make a writeup on using generated keys, basically updating external ceph docs
15:31:32 <mnasiadka> mgoddard: in CI I’m cloning, because CentOS rpms are uninstallable :)
15:32:10 <mnasiadka> At least 4.0 for now
15:32:15 <yoctozepto> <mgoddard> 2 is very simple to maintain - we just point to ceph-ansible docs
15:32:17 <yoctozepto> lol
15:32:31 <yoctozepto> we need to make docs of our own anyway
15:32:54 <yoctozepto> we had some on-channel report about ceph-ansible being incompatible with our manual for external ceph
15:33:05 <yoctozepto> something cinder/nova key not being separate related
15:33:10 <mgoddard> I think there was a bug too
15:33:32 <mnasiadka> Not really incompatible, just default config is incompatible :)
15:33:48 <michaelbarkdoll> I clone ceph-ansible currently.  Interesting the 4.0 branch started to require a grafana-server section to be defined which resulted in docker version mismatch if a ceph node is also used as a os node.
15:34:26 <mnasiadka> michaelbarkdoll: true, but you can override grafana container name
15:34:35 <yoctozepto> modern ops ("infrastructure as code" blah) advocates cloning everything ;-)
15:34:40 <yoctozepto> as in "git everywhere"
15:35:46 <yoctozepto> mnasiadka: any idea on ceph-ansible dynamics?
15:35:55 <yoctozepto> re: supporting the (1) option ^
15:36:25 <mnasiadka> yoctozepto: I doubt they will change all role names next release
15:36:47 <mnasiadka> It’s rather small now, but what do I know
15:36:55 <yoctozepto> and the options we want/have to use is probably a small set?
15:36:56 <mnasiadka> RDO surprises us every release
15:37:04 <yoctozepto> heh
15:37:15 <mnasiadka> yoctozepto: yeah
15:37:43 <mnasiadka> So I’m still thinking about improving docs so it matches today state
15:37:46 <yoctozepto> since (2) is about adapting docs
15:37:55 <yoctozepto> we should do at least (2) and most probably (1) as well
15:37:57 <mnasiadka> And then we can think if we do (1)
15:38:29 <mgoddard> that's a fair point
15:38:53 <yoctozepto> ok, then all agreed
15:39:02 <yoctozepto> move on, mgoddard
15:39:26 <mgoddard> #topic Kolla Ansible TLS Internal API
15:39:41 <mgoddard> scottsol, stackedsax, kklimonda, generalfuzz: you're up
15:39:54 <generalfuzz> we met yesterday for a bit
15:40:16 <generalfuzz> we discussed the "local-proxy" approach, which we all felt like be a good first step, and then as a second step we would switch services to use native tls via wgsi one at a time.
15:41:39 <mgoddard> if we add an intermediate step, we would need to support it
15:41:46 <mgoddard> which could complicate things long term
15:42:06 <yoctozepto> generalfuzz: can you elaborate on "local-proxy"
15:42:11 <priteau> Does every single openstack service support tls?
15:42:37 <yoctozepto> priteau: all APIs because of HTTP webserver in front
15:42:37 <mgoddard> priteau: rephrase: does every single openstack service support WSGI?
15:42:57 <yoctozepto> rabbitmq for communication also
15:43:01 <yoctozepto> mariadb checked
15:43:18 <mgoddard> I think they're just looking at internal API for now
15:43:37 <yoctozepto> mgoddard: yeah but answering priteau's doubts :-)
15:43:43 <scottsol> no this is why the intermediate step is being looked at
15:43:59 <scottsol> as a "catch all"
15:44:13 <mgoddard> generalfuzz: does the implementation of TLS on the frontend for the internal API offer an improvement in your view?
15:44:20 <scottsol> it may take some time to convince all projects to support wsgi
15:44:21 <yoctozepto> scottsol: please give an example for those not blessed with the knowledge :-)
15:44:35 <mgoddard> i.e. only https://review.opendev.org/#/c/663555
15:45:20 <generalfuzz> the "local-proxy" apporach Is to have a proxy server on every node that terminates TLS communication before communicating with the service. Currently HAProxy does the termination, and then communicates using http interally to services
15:46:21 <generalfuzz> mgoddard - I don't thinkso
15:46:56 <yoctozepto> generalfuzz: understood, thanks
15:47:15 <yoctozepto> I like the approach, it seems enough for me
15:47:22 <yoctozepto> no need to encrypt loopback traffic anyway
15:48:35 <mgoddard> how do we prevent a port conflict between haproxy and the backend?
15:48:45 <mgoddard> backend listens on localhost?
15:49:21 <mgoddard> haproxy would need to listen both on the VIP and api_interface_address
15:49:22 <generalfuzz> ports would be defined as global variables
15:50:33 <generalfuzz> why the api_interface_address?
15:50:59 <mgoddard> if you need to access a particular haproxy instance, you can't use the VIP
15:52:50 <generalfuzz> I imagine the "local-proxy" to be a separate container from HAProxy, listening on a defined intermediary port to do SSL termination
15:53:08 <yoctozepto> ^ exactly what I understood too
15:53:31 <yoctozepto> you want to abuse haproxy for ssl termination
15:53:34 <mgoddard> it could be. that's not how the similar original design worked - it reused a singe haproxy
15:54:14 <mgoddard> what about using apache within the continer to do the same thing?
15:54:53 <mgoddard> then we could use wsgi where available, and just proxy otherwise
15:54:57 <generalfuzz> The advantage to having separate proxy containers is that It is simple to reroute haproxy Directly to service when they support  SSL termination natively
15:56:07 <mgoddard> using apache to do this gives us a few things
15:56:08 <generalfuzz> using apache instead of HAProxy - Does it make any difference?
15:56:31 <mgoddard> we don't need to deploy haproxy on every node
15:56:58 <mgoddard> WSGI becomes an implementation detail
15:57:06 <yoctozepto> <generalfuzz> using apache instead of HAProxy - Does it make any difference?
15:57:12 <yoctozepto> yeah, we have extra apaches :-)
15:57:18 <mgoddard> so the architecture doesn't change when you switch
15:57:40 <yoctozepto> I like mgoddard's idea more
15:57:41 <mgoddard> and we don't need to add an intermediate design which we then have to support or deprecate and remove
15:57:47 <yoctozepto> especially if most are wsgi anyway
15:58:05 <mnasiadka> If most are wsgi, then it’s a simple fix?
15:58:18 <mnasiadka> And then let’s worry about the rest?
15:58:33 <yoctozepto> is there some wsgi/non-wsgi listing anywhere?
15:58:56 <generalfuzz> I believe kklimonda did a first pass at that list
15:59:01 <mgoddard> if people think the local-haproxy architecture is better, we could do that path instead. I'm just not so keen on doing both :)
16:00:06 <mgoddard> I thought the TCP passthrough idea was interesting. I assume we could still do it with apache? Would there be any certificate weirdness?
16:00:13 <scottsol> mgoddard - has there been any work done towards putting apache inside the container where wsgi does not already exist?
16:00:32 <mgoddard> scottsol: nope
16:00:50 <scottsol> I think that could be a good solution
16:00:50 <mgoddard> but a proxy config should be easy enough
16:01:12 <mgoddard> just configure the service to listen on localhost, then you don't get a port conflict
16:01:33 <mgoddard> we are out of meeting time I'm afraid
16:01:49 <mnasiadka> I don’t like that path to be honest... sounds like obfuscation and double work :)
16:01:51 <mgoddard> let's carry on in this channel though if people are available - it's good discussion
16:02:04 <mgoddard> #endmeeting