15:03:52 <noonedeadpunk> #startmeeting openstack_ansible_meeting
15:03:52 <opendevmeet> Meeting started Tue Apr 18 15:03:52 2023 UTC and is due to finish in 60 minutes.  The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:03:52 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:03:52 <opendevmeet> The meeting name has been set to 'openstack_ansible_meeting'
15:03:57 <noonedeadpunk> #topic rollcall
15:03:59 <NeilHanlon> o/
15:04:01 <noonedeadpunk> o/
15:04:06 <noonedeadpunk> hey everyone
15:04:14 <damiandabrowski> hi!
15:04:19 <jrosser> o/ hello
15:06:02 <noonedeadpunk> #topic office hours
15:06:26 <noonedeadpunk> I've jsut found out, that we somehow missed trove for https://review.opendev.org/q/topic:osa/pki
15:06:55 <noonedeadpunk> I'm going to prepare a fix for that and was kinda wondering if we wanna backport it
15:07:57 <jrosser> what does it use it for?
15:08:26 <noonedeadpunk> rabbitmq?
15:09:06 <jrosser> oh you mean kind of like this one https://review.opendev.org/c/openstack/openstack-ansible-os_murano/+/791726
15:09:17 <noonedeadpunk> yup, exctly
15:09:45 <noonedeadpunk> another thing that is broken at the moment is zun.
15:10:22 <noonedeadpunk> It's been a while since we've bumped version of kata, and now kata is gone from suse repos (obviously)
15:10:23 <damiandabrowski> i was gonna talk about it
15:10:47 <noonedeadpunk> I've attempted to try install kata from github sources but did not spent much time to be frank
15:11:20 <noonedeadpunk> also I'm not quite sure if still it should be integrated with docker or just podman should be good enough with modern zun
15:11:28 <noonedeadpunk> go on damiandabrowski :)
15:12:30 <damiandabrowski> firstly i thought that disabling katacontainers as default for debian/ubuntu is the best option(it's an optional component anyway and IMO it should be somehow "fixed" on zun side because they mention invalid repo in their docs: https://docs.openstack.org/zun/latest/install/compute-install.html#enable-kata-containers-optional)
15:12:49 <damiandabrowski> but disabling kata on master is not enough because obviously upgrade jobs do not pass CI
15:13:13 <damiandabrowski> so it can be solved by cherry-picking this change to stable branches which doesn't sound good :|
15:13:16 <damiandabrowski> https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/880683
15:14:55 <noonedeadpunk> I think issue is also that main jobs times out
15:15:02 <noonedeadpunk> so it actuially does not work as well
15:15:19 <noonedeadpunk> I kinda have same "result" with https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/880288?tab=change-view-tab-header-zuul-results-summary
15:15:27 <noonedeadpunk> But the thing is that container is never ready
15:15:58 <noonedeadpunk> so regardless of kata - it needs a closer look
15:16:28 <noonedeadpunk> good thing is that octavia seems to be sorted out now
15:16:46 <damiandabrowski> ah, i thought timeout issue is just the matter of recheck but probably you're right "|
15:17:37 <jrosser> i will look through our notes here
15:17:56 <jrosser> we did some stuff with zun but didnt ever deploy it for real, but the kata thing is a mess
15:20:21 <noonedeadpunk> I think it was a while back as well...
15:20:25 <jrosser> yeah
15:20:37 <noonedeadpunk> I'd imagine that docker/podman can be a mess as well
15:21:00 <noonedeadpunk> as kata now suggests to just go with podman
15:21:25 <noonedeadpunk> and no idea if zun has support for that, as it wasn't really developent lately
15:22:57 <noonedeadpunk> we're also super close to merging https://review.opendev.org/q/topic:osa/systemd_restart_on_unit_change+status:open
15:23:33 <noonedeadpunk> So Zun, Adjutant and Magnum
15:23:50 <noonedeadpunk> For Adjutant and Magnum we need to fix upgrade jobs by backporting stuff.
15:24:12 <noonedeadpunk> Once we land this I will go through patches and backport then as we've agreed on PTG
15:25:34 <opendevreview> Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_trove master: Add variables for rabbitmq ssl configuration  https://review.opendev.org/c/openstack/openstack-ansible-os_trove/+/880760
15:27:33 <noonedeadpunk> Another thing. There was a ML during weekends calling for volunteers to maintain OVS->OVN migration in neutron code. Migration is done for TripleO but I decided to pick this challange and adopt/refactor for OSA as well
15:28:02 <damiandabrowski> great!
15:28:24 <mgariepy> i will have a ovs to migrate to ovn at some point.
15:28:41 <mgariepy> but i don't have any cycle right now
15:29:23 <opendevreview> Merged openstack/openstack-ansible-os_keystone master: Use chain cert file for apache  https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/879914
15:30:06 <noonedeadpunk> Yeah, me neither, but it's smth I'd love to have :)
15:30:24 <noonedeadpunk> jamesdenton: btw, I can recall you saying smth about LXB->OVN? Do you have any draft?
15:30:42 <noonedeadpunk> As maybe it's smth I could take a look during this work as well
15:31:37 <opendevreview> Dmitriy Rabotyagov proposed openstack/openstack-ansible-openstack_hosts master: Update release name to Antelope  https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/880761
15:33:05 <noonedeadpunk> Also. RDO folks were looking into adding OSA aio deployment to their CI to spot issues early. They were barely aware that we have this feature, so with death of tripleo this path can get some attention. At very least awareness is raised a bit
15:37:26 <noonedeadpunk> I guess that's why we've seen these great doc patches for aio :)
15:37:54 <jrosser> they were nice patches
15:40:16 <noonedeadpunk> Btw, I've also changed naming of the project on this page https://docs.openstack.org/zed/deploy/index.html
15:40:34 <noonedeadpunk> So it was more clear to what project does comparing to others
15:40:54 <noonedeadpunk> Damn. Jast spotted that "Guide" is used twice....
15:44:56 <damiandabrowski> another thing I wanted to raise is blazar haproxy service
15:45:06 <noonedeadpunk> mhm
15:45:28 <damiandabrowski> https://review.opendev.org/c/openstack/openstack-ansible/+/880564
15:45:39 <damiandabrowski> seems like blazar doesn't have '/healthcheck' implemented and it's required to authenticate for all API requests
15:45:46 <damiandabrowski> it makes hard for haproxy to monitor backends(haproxy always receives 401 http code):
15:45:49 <damiandabrowski> {"error": {"code": 401, "title": "Unauthorized", "message": "The request you have made requires authentication."}}
15:45:58 <damiandabrowski> Do you think it's ok to fix it by applying below change? (at least it works on my aio)
15:46:03 <damiandabrowski> haproxy_backend_httpcheck_options:
15:46:07 <damiandabrowski> - 'expect rstatus (200|401)'
15:46:28 <noonedeadpunk> So it's requiring also for `/`?
15:46:32 <damiandabrowski> yeah
15:46:39 <noonedeadpunk> that sucks
15:47:22 <noonedeadpunk> yeah, it doesn't seem to have api-paste...
15:47:24 <damiandabrowski> we have something similar for murano(but without regex)
15:47:25 <damiandabrowski> https://opendev.org/openstack/openstack-ansible/src/commit/3f9c8300d8d09832607d2670cb3425a59bb26ac1/inventory/group_vars/haproxy/haproxy.yml#L392
15:48:12 <noonedeadpunk> Btw. I kind wonder if for murano we could jsut drop /v1 instead
15:48:42 <noonedeadpunk> damiandabrowski: we have smth simmilar for rgw btw https://opendev.org/openstack/openstack-ansible/src/commit/3f9c8300d8d09832607d2670cb3425a59bb26ac1/inventory/group_vars/haproxy/haproxy.yml#L168
15:49:20 <noonedeadpunk> but yes, I think that fix would be fine
15:50:47 <jamesdenton> hi noonedeadpunk
15:50:59 <jamesdenton> https://www.jimmdenton.com/migrating-lxb-to-ovn/
15:51:22 <noonedeadpunk> aha, great, thanks!
15:51:44 <noonedeadpunk> have you tried btw running vxlans with ovn?
15:52:00 <jamesdenton> I think it was written before we went to OVN in Zed, so the skel manipulation may not be required anymore
15:52:39 <jamesdenton> I don't think i have tried vxlan, as i recall this: Also, according to the OVN manpage, VXLAN networks are only supported for gateway nodes and not traffic between hypervisors:
15:52:45 <jamesdenton> https://www.ovn.org/support/dist-docs/ovn-controller.8.html
15:53:00 <jamesdenton> " Supported  tunnel  types  for  connecting hypervisors are
15:53:00 <jamesdenton> geneve and stt. Gateways may use geneve, vxlan, or stt."
15:53:08 <jamesdenton> /shrug
15:53:27 <noonedeadpunk> aha
15:54:39 <noonedeadpunk> ok, yes, I see. As I heard rumors that it's doable...
15:55:13 <jamesdenton> it might be, let me see if i get any sort of error trying. If all nodes are gateway nodes, then maybe?
15:56:49 <noonedeadpunk> huh, might be.. But then all communication between VMs will be possible only through public networks I assume?
15:56:58 <noonedeadpunk> or well, through gateways
15:58:52 <damiandabrowski> ah, there's one more thing. As jrosser pointed out, we should somehow test tls backend in CI.
15:58:55 <damiandabrowski> Do you have any ideas how should we do this?
15:59:01 <damiandabrowski> enable tls backend on some of already existing jobs or create new ones?
15:59:06 <damiandabrowski> (I'd appreciate some help here as I'm not really experienced with zuul :|)
16:00:59 <noonedeadpunk> I think we need to add new job at least for 1 distro (like jammy) that would differ from default
16:01:18 <noonedeadpunk> but we should then discuss what job do we want
16:01:27 <jamesdenton> not public, just that every node can be an egress point
16:01:49 <jamesdenton> probably better to get vxlan->geneve
16:02:02 <noonedeadpunk> yeah, I think it's better indeed...
16:02:21 <noonedeadpunk> damiandabrowski: meaning - no tls at all, or no tls between haproxy and api, or no tls for internal at all
16:03:02 <noonedeadpunk> maybe no tls for internal endpoint and no between haproxy->uwsgi would make most sense for me
16:03:41 <noonedeadpunk> I think this is good example https://opendev.org/openstack/openstack-ansible/commit/b59b392813c060139860afb74682ce664d895562
16:03:58 <noonedeadpunk> #endmeeting