15:00:22 <noonedeadpunk> #startmeeting openstack_ansible_meeting
15:00:22 <opendevmeet> Meeting started Tue Nov 15 15:00:22 2022 UTC and is due to finish in 60 minutes.  The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:22 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:22 <opendevmeet> The meeting name has been set to 'openstack_ansible_meeting'
15:00:28 <noonedeadpunk> #topic rollcall
15:00:32 <noonedeadpunk> o/
15:00:46 <damiandabrowski> hi!
15:01:34 <opendevreview> Merged openstack/openstack-ansible stable/xena: Bump OpenStack-Ansible Xena  https://review.opendev.org/c/openstack/openstack-ansible/+/864431
15:03:15 <mgariepy> hey !
15:03:24 <noonedeadpunk> #topic office hours
15:03:41 <noonedeadpunk> so, repo for zookeeper has been create. There's one BUT though
15:04:02 <noonedeadpunk> there're redirect from openstack namespace to windmill on gitea
15:04:16 <noonedeadpunk> so whenever you clone from opendev - you will clone wrong one
15:04:42 <noonedeadpunk> Infra is aware and will discuss better solution for this situation
15:04:52 <noonedeadpunk> skyline repo is not created yet
15:05:52 <noonedeadpunk> I still going to push zookeeper role to gerrit to start reviewing/preapre tests
15:09:26 <noonedeadpunk> I also trying to figure out failure reasons for sahara. Octiavia is broken due to collections version we use - patch for bumping version is already proposed
15:10:13 <noonedeadpunk> Another failure is regarding swift/ironic
15:10:22 <jrosser> o/ in another meeting at the same time :(
15:10:29 <noonedeadpunk> Will try to take a look unless someone is already working with that
15:10:51 <jrosser> i am unlikely to have time this week
15:11:35 <noonedeadpunk> ok, then I will try to check on that
15:11:49 <noonedeadpunk> damiandabrowski: any progress on internal ssl?
15:13:29 <damiandabrowski> yeah, I've been extensively testing James Gibson's changes. Generally they look fine but I found some bugs, I will upload patches soon. My target is to finish internal SSL project before Zed release
15:13:33 <damiandabrowski> I have 2 things to discuss
15:14:43 <damiandabrowski> 1. Even this whole thing is called "internal" SSL, we probably want to provide a way to switch from http to https for external haproxy vip along with internal one, right? (from technical perspective I don't see any blockers)
15:14:56 <noonedeadpunk> Well, yeah, that would be sweet. But mention that before release deadline we also need to branch repos and do RC
15:15:50 <noonedeadpunk> well, I think switching ssl on/off for VIP should not cause any extra complexity from what we have now
15:16:40 <damiandabrowski> complexity in terms of our code or user actions?
15:16:53 <damiandabrowski> I'm talking about this change which for now covers only internal VIP: https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/829899
15:16:57 <cloudnull> has anyone had issues with the setting "send-proxy-v2" w/ galera? -- I just updated to the latest head of master and that setting was resulting in "ERROR 1130 (HY000): Proxy header is not accepted from ..."
15:17:03 <noonedeadpunk> well, both. It's smth we already have now and there shouldn't be changes to that?
15:17:03 <damiandabrowski> it's super easy to make it cover also external one
15:17:18 <cloudnull> it looks like galera is the only service that uses that backend option ?
15:17:44 <damiandabrowski> transition from unencrypted to encrypted traffic is not covered for now
15:18:16 <damiandabrowski> https://review.opendev.org/c/openstack/openstack-ansible-specs/+/822850/1..3/specs/yoga/internal-tls.rst#b43
15:18:24 <damiandabrowski> Jonathan and James explain it here ^
15:19:23 <noonedeadpunk> cloudnull: um, no, never sam that
15:20:54 <jrosser> damiandabrowski: doesnt james spec allow haproxy to have either http or https backends so it is possible to have a transition, or switch back and forth?
15:21:08 <damiandabrowski> yes, but only for internal VIP
15:21:20 <damiandabrowski> my proposal is to allow that transition also for external VIP
15:21:42 <damiandabrowski> it's just a minor change in a code
15:21:59 <damiandabrowski> (and i'm not talking about backends now)
15:22:21 <noonedeadpunk> but PR you're mentioning only about backends?
15:22:26 <noonedeadpunk> (kind of)
15:23:11 <damiandabrowski> nope, that's why it also implements `haproxy_tcp_upgrade_frontend` var
15:23:22 <noonedeadpunk> ah,ok,yes
15:23:53 <noonedeadpunk> so we "force" tls for internal VIP when we want to secure backends
15:24:29 <noonedeadpunk> but we might need also to force external vip in case it's not tls?
15:24:41 <noonedeadpunk> Sounds like valid thing to do
15:25:49 <damiandabrowski> ok
15:26:38 <noonedeadpunk> it was so long time ago I already forgot what were caveats
15:27:10 <damiandabrowski> i need to leave for a moment, sorry
15:28:08 <noonedeadpunk> you also had second thing to discuss ;)
15:28:21 <mgariepy> ovn ssl patch is ready for reviews, i probably need to add a note. and update the commit message. https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/862403
15:28:54 <mgariepy> do we want to support upgrading existing deployment to ssl or we skip that?
15:31:05 <noonedeadpunk> oh, wow, it's huge
15:31:27 <noonedeadpunk> and what upgrade would involve? As I'm not sure it's feasible at all?
15:31:57 <mgariepy> i have no idea how we could proceed.
15:32:21 <noonedeadpunk> Though I dunno if ovn is alike mysql, when client can talk both tls and non-tls
15:33:01 <mgariepy> i'm not sure it would cause big issue (the state won't change on connection loss i guess)
15:33:10 <noonedeadpunk> maybe we should set `neutron_ovn_ssl: false`  in defaults, but enable that for CI?
15:33:33 <noonedeadpunk> But indeed I have no idea on impact during such upgrade
15:33:51 <mgariepy> i'd like to have it on by default
15:34:02 <mgariepy> we can add release note for upgrade?
15:34:11 <noonedeadpunk> yes, we can do that
15:34:13 <mgariepy> jamesdenton, what do you think ?
15:34:30 <mgariepy> we only have a couple deployers we know that uses it.
15:36:22 <mgariepy> the deployment order is kinda weird also.
15:36:36 <jrosser> i would be surprised if it can talk tls and non-tls at the same time
15:36:52 <jrosser> the point of the tls is to secure the comms so doing also non-tls would be odd
15:38:01 <jrosser> feels like an upgrade would be hard becasue as soon as the control plane parts get configured for tls then the compute/gateway nodes will be unable to connect to them
15:38:13 <noonedeadpunk> well, mysql does that as well as libvirt and rabbit for example
15:39:12 <mgariepy> the order is kinda weird there https://github.com/openstack/openstack-ansible/blob/master/playbooks/os-neutron-install.yml
15:39:15 <noonedeadpunk> while mysql does support that on the same port, and it's up to client either establish tls or not (unless explicitly restricted on server side), rabbit/libvirt just listen on different ports
15:39:22 <mgariepy> northd is not deployed before the computes.
15:40:07 <mgariepy> but if the connection is cut the ports in and security rules won't be updated until it's back online ?
15:40:11 <mgariepy> i guess..
15:41:30 <noonedeadpunk> well when connection in ovs is cut to rabbit - huge shitshtorm with everything blicking comes when it's re-established
15:41:45 <noonedeadpunk> I've heard that ovsdb is not a strongest part either...
15:42:39 <mgariepy> it's kinda hard to test that in the CI.
15:42:49 <noonedeadpunk> yeah, totally
15:43:05 <NeilHanlon> o/ sorry I'm (very) late... timezones and Dst. I'm in Texas for SuperComputing22
15:43:21 <noonedeadpunk> but I think if it's only management that will be cut and flows will remain - it might be good enough
15:43:46 <noonedeadpunk> Oh, lucky you NeilHanlon!
15:43:50 <mgariepy> just like anything else when you don't have load on the server it works just like on the dev computer :D
15:44:04 <jamesdenton> mgariepy until now we have OVN in experimental state, so i'm not sure upgrades are in scope if we want to default only to SSL
15:44:18 <jamesdenton> seems like solving for the... 1 deployment out there
15:44:31 <mgariepy> all that work for you jamesdenton ;p
15:44:40 <noonedeadpunk> so indeed if it's just mgmt - let's cover with release note and set ssl enabled by default
15:44:40 <jamesdenton> and maybe spatel lol
15:45:00 <noonedeadpunk> also anskiy I guess?
15:45:06 <mgariepy> yay ok will do that.
15:45:18 <jamesdenton> perhaps. early adopters get to work out the kinks
15:45:28 <jamesdenton> cobra kai. no mercy.
15:46:35 <mgariepy> when you upgrade jamesdenton tell us about your experience :D
15:46:36 <mgariepy> haha
15:48:35 <jamesdenton> i most certainly will
15:48:52 <noonedeadpunk> I will try to check PR then today/tomorrow morning
15:50:13 <mgariepy> cool jrosser i'd like you opinion as well on the pki role usage :D
15:50:20 <jrosser> ok :)
15:55:12 <jrosser> unrelated to all this i have tried and failed to use the ironic openstack ansible modules
15:55:39 <jrosser> certainly for Y the ironic API appears to want a system scoped token when the modules talk to it
15:55:52 <jrosser> but in my utility container the CLI is fine
15:56:42 <jrosser> i've not had time to investigate further but ironic api seems to have different behaviour to the others
15:57:36 <noonedeadpunk> system scoped tokens should not be enforced in Y at least...
15:57:49 <jrosser> i did note that our openrc in the utility container is making a token scoped to the admin project and not a system scoped one
15:57:56 <noonedeadpunk> I can imagine bugs in collection itself - it's really quite huge mess as of today
15:57:59 <jrosser> though i was sure we had a patch related to that some time ago
15:58:18 <noonedeadpunk> I think, it;s matter of variable now
15:58:47 <noonedeadpunk> https://opendev.org/openstack/openstack-ansible-openstack_openrc/commit/fdc640ddcbc13de17609005f4ca34cc4067cd5f8
15:58:49 <spatel> spatel i am up for any kind of testing :)
15:58:54 <noonedeadpunk> openrc_system_scope
15:59:29 <noonedeadpunk> So we basically have multiple openrc files when it;s enabled
15:59:51 <noonedeadpunk> some logic for clouds.yaml present though
16:00:02 <jrosser> ahha ok
16:00:07 <noonedeadpunk> but tbh I'm really not sure if that's ironic or sdk/collection
16:00:36 <noonedeadpunk> As with X sdk and Y collections even keystone failed the same way
16:00:56 <noonedeadpunk> So might be API expects some param to be provided, but collection does not provide it
16:01:02 <noonedeadpunk> as it was missed/not updated
16:02:05 <noonedeadpunk> Evenrually sad thing about collections, that 2.0 is going to be release in January
16:02:07 <damiandabrowski> i'm back sorry. So regarding internal TLS there is one more problem with uWSGI.
16:02:08 <damiandabrowski> In order to use https/https-socket options, the OpenSSL development headers are required(e.g. libssl-dev on Debian).
16:02:11 <damiandabrowski> It's not installed by default and after dev headers are installed, uWSGI python package needs to be reinstalled.
16:02:14 <damiandabrowski> But the old package without https support can be still stored in pip cache(/root/.cache/) which will prevent the correct uWSGI package to be installed.
16:02:17 <damiandabrowski> It can be fixed by disabling pip cache when rebuilding venv: `openstack-ansible  /opt/openstack-ansible/playbooks/os-glance-install.yml -e venv_rebuild=true -e uwsgi_pip_install_args='--no-cache-dir'`
16:02:23 <noonedeadpunk> So we will have to release with SHA in ansible-collection-requirements
16:02:25 <damiandabrowski> I guess we want to add an extra logic for that, right? So when libssl-dev is not installed but TLS is enabled, uwsgi playbook will re-install uwsgi.
16:02:28 <damiandabrowski> Or maybe you have some better ideas?
16:03:30 <noonedeadpunk> I don't think it's an issue?
16:04:05 <noonedeadpunk> As we can always build uwsgi with ssl support?
16:04:30 <damiandabrowski> it's the issue on existing environments which doesn't have SSL enabled
16:04:40 <damiandabrowski> i guess we need to provide a smooth upgrade path
16:04:40 <noonedeadpunk> and since we introduce this with new release - uwsgi version will be different, thus cache won't have any effect?
16:05:17 <jrosser> i think there is a recent new release of uwsgi
16:05:20 <noonedeadpunk> because cache might be an issue only when we re-build same uwsgi version but with different flags, while we don't?
16:05:33 <jrosser> so we can probably catch this with enable ssl for it and bump version all at the same time
16:06:09 <noonedeadpunk> oh... where we bump uwsgi :D
16:06:22 <jrosser> global pins?
16:06:23 <noonedeadpunk> ah https://opendev.org/openstack/openstack-ansible/src/branch/master/global-requirement-pins.txt#L11
16:06:38 <noonedeadpunk> yeah, was surprised not to find it in u-c
16:06:42 <jrosser> yeah so i think we have a neat way to do that hopefully
16:06:46 <noonedeadpunk> yup
16:07:08 <jrosser> it just needs libssl-dev in the repo server?
16:07:16 <damiandabrowski> ah yes i think you are right, if we assume that uwsgi will be upgraded it's just a matter of adding libssl-dev to the  `uwsgi_package_list`
16:07:30 <noonedeadpunk> 2.0.21 was released ~month ago
16:08:30 <noonedeadpunk> it's weird kind of... does pip just - oh, I have libssl - I will build SSL support for uwsgi?
16:08:46 <noonedeadpunk> and if not - well, ignore that?
16:08:51 <damiandabrowski> yeah
16:09:01 <damiandabrowski> it's mentioned here: https://uwsgi-docs.readthedocs.io/en/latest/HTTPS.html
16:09:04 <noonedeadpunk> well, actually I think it works same for journald I guess....
16:09:06 <damiandabrowski> "In order to use https option be sure that you have OpenSSL development headers installed (e.g. libssl-dev on Debian). Install them and rebuild uWSGI so the build system will automatically detect it."
16:09:19 <jrosser> uwsgi has a totally custom build system
16:09:23 <jrosser> it's pretty wild
16:09:48 <noonedeadpunk> ok then....
16:09:56 <damiandabrowski> okok, thanks for the input guys
16:10:02 <noonedeadpunk> So yes - let's bump uwsgi version and call it a day
16:10:15 <noonedeadpunk> oh. we're overtime:)
16:10:16 <jrosser> iirc there was some discussion in the infra channel this last week about conflucting versions of pyOpenssl (?) and that breaking uwsgi buids
16:10:18 <noonedeadpunk> #endmeeting