15:00:28 <noonedeadpunk> #startmeeting openstack_ansible_meeting 15:00:28 <opendevmeet> Meeting started Tue Nov 9 15:00:28 2021 UTC and is due to finish in 60 minutes. The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:00:28 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:00:28 <opendevmeet> The meeting name has been set to 'openstack_ansible_meeting' 15:00:39 <noonedeadpunk> #topic rollcall 15:00:41 <noonedeadpunk> o/ 15:00:49 <admin1> o/ 15:01:00 <jrosser_> o/ hello 15:01:20 <damiandabrowski[m]> hey! 15:01:48 <noonedeadpunk> I almost failed again with sumer time hehe 15:03:17 <mgariepy> hoo. early ! 15:04:19 <noonedeadpunk> #topic office hours 15:04:32 <noonedeadpunk> I don't think we have any recent bugs, so jumping directly here 15:04:51 <noonedeadpunk> I saw really great work regarding tls encryption for nova 15:05:05 <noonedeadpunk> and even for VNC encryption - that's awesome 15:05:08 <jrosser_> not related directly to OSA but we are seeing leaking fd in nova-compute after wallaby upgrade 15:05:28 <noonedeadpunk> I fully failed my part due to internal stuff that I couldn't put away... 15:05:38 <noonedeadpunk> oh 15:06:02 <noonedeadpunk> wow, you already upgraded! 15:06:09 <jrosser_> in case anyone else finds the same thing it looks like this https://bugs.launchpad.net/oslo.messaging/+bug/1949964 15:07:10 * noonedeadpunk subscribed 15:07:13 <noonedeadpunk> interesting 15:07:15 <jrosser_> and yes for nova TLS i think the patches are really close 15:07:26 <jrosser_> we have it deployed in multinode lab 15:07:45 <noonedeadpunk> may I dare to ask if you tested live migrations? 15:08:02 <jrosser_> it would be great to get more testing of this, in particular as it takes ansible hostname and nodename facts and uses them in the certificate 15:08:17 <jrosser_> and i think there are differences in the way people name their hosts, like fqdn or not 15:08:23 <noonedeadpunk> Yeah, we have ppl who can test this I believe 15:08:34 <jrosser_> and this may interact with DNS blah blah and cause the cert verification to fail 15:08:51 <noonedeadpunk> At least I got several requests to notify about having first beta of X for test 15:09:29 <jrosser_> we are currently testing internal VIP = https with cert from PKI role, and nova TLS 15:09:39 <noonedeadpunk> and, we have mixed scenario of naming hosts in our sandbox :D 15:09:54 <jrosser_> migration looks good i think, was just trying to get james here in IRC 15:09:55 <noonedeadpunk> so that might be interesting 15:10:26 <jrosser_> we were discussing what needs doing next here earlier today 15:10:32 <jrosser_> and there is cleanup of the nova SSH keys 15:10:54 <noonedeadpunk> oh, so you also implemented tls ssh auth there? 15:10:54 <jrosser_> and also that we decide (?) that TLS is the only supported nova migration from now? 15:11:13 <jrosser_> well, i think now that there is no need for those keys at all 15:11:24 <jrosser_> unless i misunderstand how its working 15:11:41 <noonedeadpunk> actually it might be that, yes 15:11:55 <noonedeadpunk> since we set client certs there anyway 15:11:59 * jrosser_ waves to JamesGibo 15:12:09 <jrosser_> james has done all this excellent work on nova TLS 15:12:34 <noonedeadpunk> answering on if it's the only supported migration - I think yes 15:12:50 <noonedeadpunk> At least I don't see other reall non-deprecated options 15:13:01 <opendevreview> Merged openstack/ansible-role-python_venv_build stable/ussuri: Set centos-7 jobs to non voting https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/816316 15:13:04 <noonedeadpunk> We might keep it for another release just in case though 15:13:34 <jrosser_> removing all the code for nova SSH keys would be great, but we can leave that for Y if we want to 15:14:12 <noonedeadpunk> I think we should default to TLS now, with easy option for fallback 15:14:31 <jrosser_> it does kind of have to be all one way or the other, i think 15:14:44 <jrosser_> you can't have a mixed config across the compute nodes 15:14:45 <noonedeadpunk> yeah, it can't be both 15:15:46 <jrosser_> is there anything else that we want to complete TLS/PKI wise for the X release? 15:16:10 <noonedeadpunk> Well we talked about memcached 15:16:20 <noonedeadpunk> But it's not a requirement for sure 15:16:37 <jrosser_> we did discuss a little about how to start transitioning the backend services to https, but that looks really quite "interesting" problem 15:16:57 <jrosser_> like how to do it without a huge downtime 15:17:57 <noonedeadpunk> Oh, yes, that's interesting... I guess you can't have mix of backends? 15:18:02 <noonedeadpunk> in terms of http/https 15:18:07 <mgariepy> if the services listen both http and https, can we configure backend with both scheme in haproxy and confirm it work on https ? 15:18:51 <mgariepy> or we need to move 1 backend at the time in haproxy over https.. 15:19:01 <jrosser_> it's not clear really 15:19:12 <jrosser_> as the haproxy play runs kind of first for all the services 15:19:27 <JamesGibo> Hi, just caught up with meeting via irclogs! 15:19:28 <mgariepy> why couldn't we have multiple time the backends in the haproxy backend list ? 15:19:50 <noonedeadpunk> same host/same port? 15:19:52 <mgariepy> it will be on a different port anyway. 15:20:04 <noonedeadpunk> but port for services is the same 15:20:12 <mgariepy> ho.. yep .. :/ 15:20:33 <noonedeadpunk> I _think_ we can manage backends during runtime 15:20:47 <noonedeadpunk> the same way we put them to maint? 15:21:30 <jrosser_> right so anyway - reason i bring this up is its quite a complex problem 15:21:37 <noonedeadpunk> nah, module supports only enable/disable/drain I guess 15:21:42 <jrosser_> and we need to start thinking about it even if theres no answer right now 15:21:48 <spatel> jrosser_ that rabbitMQ bug is interesting.. i haven't seen any behavior yet. 15:22:35 <mgariepy> can we have 2 pools of backends ? 15:23:31 <noonedeadpunk> Well, at least I'm think that you can pass to haproxy socket to drop a specific backend or add another one 15:23:52 <noonedeadpunk> So even if there's no ansible module ready for that - that is not impossible 15:24:39 <jrosser_> the awkward part is that when all this is completed you want the https backend to be on the same well known port numbers as the http one used to be 15:24:39 <noonedeadpunk> And we can write config after all services are reloaded. or actually trigger haproxy role for migration in each playbook, which would be pretty nasty I guess. 15:25:00 <jrosser_> so it feels like there are multiple phases involved 15:25:06 <noonedeadpunk> yeah 15:26:17 <jrosser_> ok well maybe most important is testing the nova TLS stuff 15:26:41 <noonedeadpunk> I suggest merge it to be able to test easily 15:26:53 <noonedeadpunk> we can patch if afterwards anytime 15:27:20 <jrosser_> JamesGibo: we've tested migration is working? :) 15:27:54 <JamesGibo> Yeah, it is working for us 15:29:07 <JamesGibo> Using the HAproxy API to manage the backends is an intressting idea, i will have a think about that 15:30:53 <jrosser_> noonedeadpunk: how much of the hashicorp vault stuff would you like to get done for X? 15:31:06 <jrosser_> maybe we should start an etherpad with todo/patch links 15:31:34 <noonedeadpunk> personally I don't care _that_ much:) 15:32:24 <noonedeadpunk> as I don't have time for it at all. But folks eager to push stuff 15:32:43 <noonedeadpunk> it's tough given their awareness of the project overall 15:33:11 <noonedeadpunk> I think we can merge vault role and hopefully agree on "concept" patch 15:34:03 <noonedeadpunk> and we can iterate on the vault to add internal storage and safe tokens storage then we have now 15:34:11 <opendevreview> Merged openstack/ansible-role-python_venv_build stable/ussuri: Workaround distro provided pip having old CA certs on centos-7 https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/816317 15:34:25 <noonedeadpunk> but at least not to get overwhelmed with depends-on there 15:35:32 <opendevreview> Dmitriy Rabotyagov proposed openstack/ansible-role-python_venv_build stable/ussuri: Revert "Set centos-7 jobs to non voting" https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/817250 15:35:53 <opendevreview> Dmitriy Rabotyagov proposed openstack/ansible-role-python_venv_build stable/ussuri: Revert "Set centos-7 jobs to non voting" https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/817250 15:36:25 <opendevreview> Dmitriy Rabotyagov proposed openstack/ansible-role-python_venv_build stable/ussuri: Revert "Set centos-7 jobs to non voting" https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/817250 15:36:35 <noonedeadpunk> I think more important is to fix zun 15:36:53 * jrosser_ looks for andrewbonney 15:37:11 <noonedeadpunk> as I haven't return to it. And the question is in libcyrur that's they don't build for deb 15:37:20 <andrewbonney> Didn't realise it was broken. I'll take a look tomorrow 15:37:29 <noonedeadpunk> so options were either snap (fewwww) or build from source 15:37:39 <jrosser_> this is no deb for focal, or just new version theres no deb at all? 15:37:49 <noonedeadpunk> no versions for deb at all 15:37:53 <noonedeadpunk> *new 15:38:38 <noonedeadpunk> well eventually we relied opensuse repo for deb which was a bit naive but worked 15:39:24 <noonedeadpunk> oh, wait, I guess it was not kuryr but kata... 15:40:36 <noonedeadpunk> yes, it was kata... 15:40:50 <noonedeadpunk> so whole 2.0 version is not available for deb 15:42:35 <noonedeadpunk> and to be specific - way was broken for debian only 15:43:00 <noonedeadpunk> so we might set it as nv now I guess 15:43:14 <andrewbonney> kata isn't required for zun to work, so in the worst case it could be disabled, but it would be nice to fix it 15:43:16 <noonedeadpunk> but we still need to find the way to move forward 15:45:30 <noonedeadpunk> Also, I asked damiandabrowski[m] to be another pair of eyes for https://etherpad.opendev.org/p/db_pool_calculations and help out with landing patches 15:46:07 <noonedeadpunk> Would be great if we can get new numbers soon 15:47:37 <noonedeadpunk> ok, awesome, thanks everyone for joining! 15:56:38 <johnsom> noonedeadpunk I didn't want to bug you during the meeting, but DNS records are really only tied to neutron ports/fips/etc. so neutron handles interacting with designate on behalf of nova during the port plugs. There is no longer a direct link from nova to designate. 15:58:15 <noonedeadpunk> aha, fair enough 15:59:29 <noonedeadpunk> I just had some recallings that nova-metadata was pushing for sink or smth like that 15:59:45 <noonedeadpunk> but yes, I agree that makes sense 15:59:49 <noonedeadpunk> #endmeeting