15:00:53 <mnasiadka> #startmeeting Kolla
15:00:53 <opendevmeet> Meeting started Wed Feb 23 15:00:53 2022 UTC and is due to finish in 60 minutes.  The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:53 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:53 <opendevmeet> The meeting name has been set to 'kolla'
15:01:04 <mnasiadka> #topic rollcall
15:01:50 <yoctozepto> o/
15:01:58 <mnasiadka> o/
15:02:03 <ohorecny2> \o
15:02:30 <frickler> o/
15:04:37 <mnasiadka> #topic agenda
15:04:37 <mnasiadka> * Announcements
15:04:37 <mnasiadka> * Review action items from the last meeting
15:04:37 <mnasiadka> * CI status
15:04:37 <mnasiadka> * Release tasks
15:04:37 <yoctozepto> mnasiadka gogo
15:04:38 <mnasiadka> * Current cycle planning
15:04:38 <mnasiadka> * Additional agenda (from whiteboard)
15:04:40 <mnasiadka> * Open discussion
15:04:42 <yoctozepto> :-)
15:04:49 <mnasiadka> #topic Announcements
15:05:11 <mnasiadka> I booked the same PTG slots as last time - Mon-Wed (Wed for Kayobe) - 13-17UTC (13-15 UTC on Wed)
15:05:22 <mnasiadka> Created etherpad
15:05:23 <mnasiadka> https://etherpad.opendev.org/p/kolla-zed-ptg
15:05:29 <mnasiadka> #url https://etherpad.opendev.org/p/kolla-zed-ptg
15:05:59 <mnasiadka> Please put your topic proposals in there
15:06:01 <mgoddard> \o
15:06:03 <yoctozepto> (psst, it's #link)
15:06:10 <mnasiadka> ah
15:06:16 <mnasiadka> #link https://etherpad.opendev.org/p/kolla-zed-ptg
15:06:18 <mnasiadka> thanks yoctozepto
15:06:21 <yoctozepto> yw mnasiadka
15:06:29 <mnasiadka> #topic Review action items from the last meeting
15:06:39 <mnasiadka> mnasiadka post a patch for docs - standard topics that should be discussed over PTG and then revisited in mid-cycle
15:06:39 <mnasiadka> mnasiadka to triage security bugs and update them with resolution plan (if needed)
15:06:39 <mnasiadka> hrw to discuss with pynacl upstream to release binary wheel of 1.4.0 for aarch64
15:06:45 <mnasiadka> did first, a bit - patch posted
15:06:55 <mnasiadka> https://review.opendev.org/c/openstack/kolla/+/830613
15:07:00 <mnasiadka> second to be continued
15:07:07 <mnasiadka> #action mnasiadka to triage security bugs and update them with resolution plan (if needed)
15:07:13 <mnasiadka> #action hrw to discuss with pynacl upstream to release binary wheel of 1.4.0 for aarch64
15:07:16 <mnasiadka> since hrw is not here
15:07:24 <mnasiadka> #topic CI status
15:07:31 <mnasiadka> How is CI?
15:07:58 <mnasiadka> Whiteboard says Kayobe CI is RED due to ping issue?
15:08:01 <mnasiadka> (probably outdated)
15:08:11 <yoctozepto> k and k-a seem fine
15:08:36 <mgoddard> kob fixed
15:08:40 <mnasiadka> thanks mgoddard
15:09:05 <mnasiadka> #topic Release tasks
15:09:39 <mnasiadka> Release mgmt team has asked for Cycle highlights, I'll post up a patch and ask for reviews
15:09:55 <mnasiadka> #action mnasiadka to post patch for cycle highlights
15:10:49 <mnasiadka> #topic Current cycle planning
15:11:31 <mnasiadka> mgoddard: you wanted to discuss Let's Encrypt?
15:11:39 <mnasiadka> We can do that in the additional topics slot if you prefer
15:11:43 <mgoddard> yes
15:11:55 <mgoddard> either is fine
15:13:00 <mnasiadka> Ok - just a reminder: Kolla feature freeze: Mar 21 - Mar 25
15:13:19 <yoctozepto> it's going to be chilly in March!
15:13:31 <mnasiadka> So let's go with Let's Encrypt
15:13:50 <yoctozepto> let's go and let's encrypt indeed
15:13:53 <mgoddard> has anyone reviewed the patch recently?
15:13:55 <yoctozepto> I did not have time to read the patch
15:14:03 <yoctozepto> I would love a tl;dr
15:15:09 <mgoddard> I think we need a rethink.
15:15:11 <mgoddard> I don't think we can expose the HAProxy admin socket unauthenticated via TCP
15:15:13 <mgoddard> openstack-ansible suggests they use separate certs for each load balancer. That would avoid the sync, and greatly simplify the design. We could also use a unix admin socket. See https://docs.openstack.org/openstack-ansible/latest/user/security/ssl-certificates.html#certbot-certificates and https://opendev.org/openstack/openstack-ansible-haproxy_server
15:15:15 <mgoddard> we need to store the certs on disk, as well as dynamically updating HAProxy. This would be a lot easier if we only had to update the local HAproxy
15:15:17 <mgoddard> the bootstrapping process seems clumsy, and it concerns me that a reconfigure doesn't work. A colleague suggested using certbot standalone mode to bootstrap when we don't have certificates.That could be fiddly, but either way, I'd like to see a clean, documented way to bootstrap this (that ensures we don't overwrite the LE certs with our own self-signed ones). It might involve getting
15:15:19 <mgoddard> HAProxy running first to bootstrap LE, then running another deploy with everything else.
15:15:21 <mgoddard> the internal API support doesn't seem that useful to me, and if we're going to iterate the design then it might be easier to remove it
15:15:23 <mgoddard> Overall, I'd like to see a written plan for the approach, that a few people can agree on - we should have enough context at this point to agree on a design.
15:15:30 <mgoddard> a bit long for a tl;dr, but that was my summary comment
15:15:38 <yoctozepto> I was about to say that!
15:15:49 * yoctozepto reading
15:16:16 <mnasiadka> Ok, just to be clear - we're not going to support DNS-01? only HTTP-01 challenge?
15:16:52 <mgoddard> correct
15:17:04 <mnasiadka> I'm not utterly happy about that.
15:17:06 <mgoddard> at least for now
15:17:34 <mgoddard> I don't know what's involved in DNS-01
15:18:12 <mnasiadka> a DNS server that can be ,,orchestrated'' or manual TXT entries in the domain
15:18:24 <mnasiadka> I'm just saying it might be even easier - and that's required for wildcard certificates
15:18:35 <mnasiadka> We don't need to expose anything.
15:19:18 <mgoddard> that's about as much as I know about DNS-01
15:19:54 <yoctozepto> the problem with DNS-01 and k-a is that k-a does not care about the user's DNS server
15:19:58 <mgoddard> what I don't know is whether we could provide any form of general support for it
15:20:12 <yoctozepto> mgoddard paraphrased me
15:20:38 <mnasiadka> With certbot and it's semi-broken support for any normal forms of DNS-01, it might be complicated.
15:22:53 <mgoddard> it's proving difficult enough to implement HTTP-01. If you'd like to ask James to implement DNS-01 too he might not be wild about it
15:23:47 <yoctozepto> what is the admin socket on tcp for?
15:24:00 <mgoddard> to update the certs dynamically
15:24:25 <mnasiadka> So, my problem is currently, that with the merged patch to Kolla - we're limiting ourselves to certbot (which in most cases won't work for most DNS-01 providers). I'm fine with first doing HTTP-01 and then DNS-01 (if it's possible to add later).
15:25:16 <mgoddard> this patch has been around for some time, and this is the first time I'm hearing a request for DNS-01
15:26:03 <frickler> couldn't the cert updates be done by a service container similar to e.g. keystone-fernet? that would need the admin socket neither via tcp nor on the host I think
15:26:05 <mgoddard> does anyone know how many deployments would be likely to use HTTP-01 vs DNS-01?
15:26:40 <mgoddard> frickler: it was like that in a previous iteration
15:27:07 <mgoddard> it seems that openstack-ansible just uses a different cert for each host, and avoids syncing
15:27:16 <mgoddard> that seems like a great simplifier to me
15:27:41 <mgoddard> probably we should look at their implementation
15:27:55 <mgoddard> (we == headphoneJames)
15:28:27 <frickler> for HTTP-01 vs. DNS-01, my deployments all would use the latter, but I also consider that to be out of the scope of k-a. I just need a nice interface to rotate the certs I refreshed outside of kolla
15:28:59 <yoctozepto> yeah, cert rotation is probably one thing to tackle
15:29:46 <frickler> for a general survey, does it make sense to add that question to the openstack user survey? would be some time until we get results, though
15:30:10 <mgoddard> probably too long, although this patch has been around for some time
15:30:29 <mnasiadka> yes, but from what I understand (from headphoneJames' email) HAProxy 2.2 is rejecting multi certificate pem files in the ''hot reload'' feature?
15:31:06 <mnasiadka> maybe frickler is right - we just need to focus on means to dynamically update certificates - who cares if a user is using certbot or not.
15:32:29 <frickler> mnasiadka: do you have a link to that email?
15:33:30 <mnasiadka> frickler: no, that was shared private - I can forward
15:34:39 <frickler> ah, that explains why I didn't see it ;)
15:35:01 <mnasiadka> https://www.mail-archive.com/haproxy@formilux.org/msg40150.html
15:35:21 <mnasiadka> a bit related to single file with multiple certs ;-)
15:37:55 <mnasiadka> So - is there any rough plan for that feature?
15:38:59 <headphoneJames> Fyi, 2.2 did turn out to support dynamic reload
15:40:04 <mgoddard> sorry, had to run - poorly child
15:41:22 <yoctozepto> mgoddard: understandable! best wishes!
15:42:23 <mgoddard> what do we mean by dynamic reload without certbot here though? how would new certs get placed?
15:43:05 <mnasiadka> user-provided mechanism, for those that don't want to use certbot ;-)
15:43:27 <mnasiadka> just a kolla-ansible command to update the certs to newly uploaded ones?
15:43:33 <mgoddard> I suppose we could drop certs to /etc/kolla/haproxy/haproxy.pem, then provide a script to do the dynamic reload
15:45:35 <mnasiadka> sounds good to me, that gives us some functionality we could merge this cycle?
15:45:58 <mgoddard> potentially
15:46:29 <mgoddard> assuming headphoneJames is on board
15:46:54 <headphoneJames> Would we make cert bot available to kolla Ansible to generate certs?
15:47:33 <opendevreview> Radosław Piliszek proposed openstack/kolla-ansible master: [WIP] [CI] Use Tenks in Ironic job  https://review.opendev.org/c/openstack/kolla-ansible/+/644271
15:48:36 <mnasiadka> certbot container patch is merged already
15:48:57 <mgoddard> yes, but that's the easy part :)
15:50:10 <yoctozepto> yeah
15:50:13 <mnasiadka> as I said, I'm not a certbot user - I can understand it can fit some cases - but I'd like to also have the option of not using it - and having a separate mechanism delivering the certs to haproxy and just signalling that it should reload the cert ;-)
15:50:22 <headphoneJames> From what in reading, It sounded like the certificates would be generated during deployment instead of after container is deployed
15:51:00 <mnasiadka> if we can have reliable automation for the certbot part - I'm all in (but maybe these should be separate patches)
15:51:04 <mgoddard> if someone can write up how dynamic reload would work in a way that would be generally useful, that would be helpful
15:52:08 <mgoddard> is it still using certs on the deployment host and copying those across, or does it assume some process has put them into place on the haproxy hosts?
15:53:53 <mnasiadka> so, for dns-01 case, it would be nice if kolla-ansible would copy out the cert to nodes and update them in haproxy
15:54:00 <headphoneJames> I'm assuming the former based on this conversation
15:54:09 <mnasiadka> frickler: opinions?
15:54:30 <mgoddard> the former doesn't really work with HTTP-01
15:54:48 <frickler> I'm not sure how the dynamic update works
15:55:01 <mnasiadka> #link https://www.haproxy.com/blog/dynamic-ssl-certificate-storage-in-haproxy/
15:55:33 <mnasiadka> mgoddard: for http-01 we need to stand up a backend on each of the hosts that serve haproxy?
15:56:24 <mgoddard> typically, yes
15:56:40 <yoctozepto> afair, we were discussing that we need only https://www.haproxy.com/blog/hitless-reloads-with-haproxy-howto/
15:57:08 <frickler> I was just wondering whether hitless reload wouldn't be good enough in our case
15:57:09 <yoctozepto> (as the mnasiadka's linked post suggests to use if one does not have many many certs)
15:57:20 <yoctozepto> and that is what we discussed
15:57:31 <frickler> the dynamic update seems a bit overkill
15:57:31 <mgoddard> makes sense
15:57:41 <yoctozepto> the issue was we did not have the possibility to reload
15:57:44 <yoctozepto> and still do not have
15:57:50 <yoctozepto> I mean, in k-a
15:58:09 <yoctozepto> the reason was the file copying
15:58:20 <yoctozepto> as the certs have to be first copied into the running container
15:58:34 <mgoddard> true that
15:58:45 <yoctozepto> it seems the patch has grown much beyond the original plan
15:59:40 <mgoddard> well 1 minute to go
15:59:44 <frickler> can't we bindmount the certs in and update them on the host?
15:59:51 <mnasiadka> 1 minute to go, yes
15:59:57 <mnasiadka> should we have some dedicated meeting for this?
15:59:58 <yoctozepto> 5 sec
16:00:02 <yoctozepto> and go
16:00:13 <yoctozepto> dedicated meeting ++
16:00:21 <yoctozepto> the PTG
16:00:23 <yoctozepto> :D
16:00:35 <mnasiadka> probably we would like to have something merged this cycle :D
16:00:45 <mnasiadka> ok, let's discuss about the dedicated meeting after the official meeting :D
16:00:46 <yoctozepto> yeah, true that
16:00:50 <yoctozepto> ++
16:00:54 <mnasiadka> thanks for joining, sorry for not covering all topics...
16:00:58 <mnasiadka> #endmeeting