15:00:18 <noonedeadpunk> #startmeeting openstack_ansible_meeting
15:00:18 <opendevmeet> Meeting started Tue Nov  1 15:00:18 2022 UTC and is due to finish in 60 minutes.  The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:18 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:18 <opendevmeet> The meeting name has been set to 'openstack_ansible_meeting'
15:00:23 <noonedeadpunk> #topic rollcall
15:00:36 <noonedeadpunk> o/
15:00:44 <jrosser> hello
15:00:49 <mgariepy> hello
15:01:39 <noonedeadpunk> yay, gerrit finally back -jsut in time :D
15:02:15 <noonedeadpunk> with regards time change I guess it makes sense to raise question if current meeting time is ok for you?
15:02:30 <jrosser> this is ok for me
15:02:31 <noonedeadpunk> Or should I make a poll and pick up new one ?
15:04:29 <noonedeadpunk> ok, if everyone is fine with 15UTC - let's leave it as is
15:04:32 <mgariepy> works for me
15:04:49 <noonedeadpunk> If not - let me know and we will arrange a poll
15:05:02 <noonedeadpunk> #topic office hours
15:05:12 <noonedeadpunk> So I have good progress on zookeeper role
15:05:48 <noonedeadpunk> I still feel quite confused about how to have 2 repos for same purpose under opendev umbrella...
15:06:22 <noonedeadpunk> I tired to fit in what's already there and it's quite far from beeing usable by us I'd say
15:06:44 <jrosser> there is more than one deployment project so maybe we don't worry too much
15:07:10 <noonedeadpunk> yeah....
15:07:28 <noonedeadpunk> I planned to push patches for repo creation today after the meeting
15:07:48 <noonedeadpunk> And also create skyline repo with that
15:08:51 <noonedeadpunk> We have quite a few patches for review
15:10:54 * noonedeadpunk checking PTG etherpad
15:11:37 <mgariepy> i'm still working on ovn ssl stuff, i'm getting closer (i think)
15:12:10 <noonedeadpunk> I think we have quite good progress for things we want for Z
15:13:12 <jrosser> i am doing another ironic deployment so hopefully we find any last bits there too
15:13:20 <noonedeadpunk> damiandabrowski is not around today and I'm not sure about conclusions what we should do with glance. As we've landed (or about to land) changes that will disable show_multiple_locations by default
15:14:02 <noonedeadpunk> and if we still need to run 2 api servers just for show_directl_url as it was considered as lower risk from what I got
15:16:32 <jrosser> yes i think that was the result
15:16:38 <jrosser> just need a tidy way to do it
15:19:51 <noonedeadpunk> Yeah, will try to check that once done with zookeeper
15:21:07 <noonedeadpunk> oh, btw, one frustrating thing has happened during PTG. Regarding u-c and our way of filtering
15:22:10 <noonedeadpunk> So basically I was told that u-c as of today should be never used for stable branches as basically we deploy outdated software - requirements team does not manage security issues in packages that are in u-c
15:22:31 <jrosser> thats always been the case i think
15:23:10 <jamesdenton> hey, sorry i'm latye
15:23:25 <noonedeadpunk> And when I asked how then openstack should be installed, as it will hardly pass without u-c they said that system packages is the only way to do that
15:23:46 <noonedeadpunk> because we all trust distributions to maintain python bindings...
15:24:15 <noonedeadpunk> On top - filtering projects (like neutron or ceilometer) should be considered as a bug and fixed from our side
15:24:29 <jrosser> but - you cant?
15:24:38 <jrosser> like pip blows up?
15:24:39 <noonedeadpunk> While having these project in u-c is okey.
15:25:11 <noonedeadpunk> So what they proposed - install just using u-c, and then update package on top of installed one
15:25:25 <jrosser> i thought neutron is only there becasue work on neutron-lib is not complete
15:25:28 <jrosser> thats the actual bug
15:25:37 <noonedeadpunk> ie - do installation 2 times separately, first install all requirements with constraints and then install whatever needed
15:25:59 <noonedeadpunk> There was quite harsh argue for this topic
15:26:44 <noonedeadpunk> And because they didn't want to change anything it all ended up in - u-c for CI only, never use on prod, use system packaged
15:27:01 <noonedeadpunk> which got me very frustrated and confused
15:27:13 <jrosser> i can imagine
15:27:19 <jrosser> anyone from kolla perspective there?
15:27:37 <noonedeadpunk> by that time I guess not. But infra folks were on releases team side
15:27:54 <jrosser> no comment
15:29:08 <noonedeadpunk> Well, when things a bit calmed down I got suggestion to install requirements+u-c and then package from source independently
15:30:01 <noonedeadpunk> while this can work - there's one tricky thing (at least) - if package is older then from u-c and we're building wheels - it can still be troublesome
15:31:40 * noonedeadpunk got frustrated again after raising this topic....
15:32:54 <jrosser> perhaps it's worth talking to mgoddard or someone from kolla as the problem i guess is identical for them?
15:33:23 <noonedeadpunk> They install things jsut from pip though
15:34:28 <noonedeadpunk> We quite recently got issue when running cinder-api deployed at beginning on Xena, but we added a bunch of cinder-volume from top of stable/xena (due to some bugfixes in code) and they were ignoring detach commands - nova was detaching, but cinder-volumes just ignored that. Until we had to upgrade cinder-api to same version
15:35:12 <noonedeadpunk> So I wonder if I will be suggested to enable unattended-upgrades as well to cover that issue with system packages...
15:35:19 <noonedeadpunk> anyway
15:36:07 <noonedeadpunk> well, I've checked devstack and it's also filtering the same way we do. But argument was - devstack is CI only, so we can do nasty things there while osa/kolla should not do that
15:37:13 <jamesdenton> wouldn't you want prod to mirror what you're doing in testing? <insert kermit meme>
15:37:45 <noonedeadpunk> yeah, sure, you're right.
15:39:06 <noonedeadpunk> btw, we can drop that filtering for tempest, as tempest in not in u-c for a while now
15:39:27 <noonedeadpunk> eventually, neutron also is not for Zed as of today. But not sure if it's intended or not
15:40:24 <noonedeadpunk> so if it's only ceilometer that left.... ugh
15:40:43 <noonedeadpunk> fix bug by dropping telemetry roles ? :D
15:40:59 <noonedeadpunk> anyway
15:41:11 <jamesdenton> :D
15:41:18 <noonedeadpunk> I don't have anything else on agenda
15:41:32 <jamesdenton> i hope to revisit the default ml2 plugin drama today or tomorrow
15:42:06 <noonedeadpunk> aha, yes, good
15:42:10 <mgariepy> i hope being able to fix the ovn ssl stuff today :/
15:42:21 <noonedeadpunk> My personal opinion is that we should provide some default....
15:42:37 <noonedeadpunk> (and we still do in neutron role)
15:42:53 <jamesdenton> the haproxy templates need to be adjusted to account for neutron_plugin_type not being global, which in one case is for calico but i thought about adding an 'is defined' check
15:43:16 <jamesdenton> yes, a default is fine. just trying to avoid stepping on the toes of lxb default deployments
15:43:28 <noonedeadpunk> yeah, let's add this check and will clean up later with calico itself
15:43:46 <jamesdenton> that works
15:43:47 <noonedeadpunk> I tried to take a look on calico yestarday actually and get done quite fast
15:44:11 <noonedeadpunk> I think current issue at very least is that etcdgw driver can't work with modern etcd properly
15:44:34 <noonedeadpunk> as it tries to check URLs for health that are not valid with latest stable etcd
15:45:05 <jamesdenton> I found this, not sure how relevant it is today (from 2020): https://github.com/projectcalico/calico/issues/3015#issuecomment-573094997
15:45:08 <noonedeadpunk> And I didn't want to dig much deeper there
15:45:14 <jamesdenton> " In particular, we used to advise core_plugin = ml2 and configuring Calico as an ML2 mechanism driver, but now we prefer core_plugin = calico"
15:46:22 <noonedeadpunk> So `calico-dhcp-agent.service` fails now with not able to connect to etcd
15:46:37 <noonedeadpunk> Not sure if it's related to ml2 overall...
15:48:24 <noonedeadpunk> And we don't install neutron-metadata-agent... But yes, we still have core_plugin=ml2 for this scenario
15:49:24 <noonedeadpunk> But I'm not sure how it will solve that calico services rely on https://opendev.org/openstack/etcd3gw/commits/branch/master/etcd3gw/client.py that seems not to work with modern etcd
15:50:42 <noonedeadpunk> They even don't override DEFAULT_API_PATH.. Also I kind of hardly understand how it's supposed to be overriden when its set as constant...
15:50:53 <noonedeadpunk> anyway
15:50:59 <noonedeadpunk> I gave up
15:51:10 <jamesdenton> well, to be fair, even the upstream docs stop at 18.04
15:51:23 <jamesdenton> so who knows how well it's been maintained
15:52:12 <jamesdenton> i don't know how much it will be missed if it were deprecated and removed
15:53:45 <noonedeadpunk> yeah, no idea. I know logan- was using it, but I haven't seen him for a while now...
15:53:48 <noonedeadpunk> or heard
15:54:19 <noonedeadpunk> so no idea
15:57:37 <jrosser> i think that we should remove it
15:58:23 <jrosser> there is a big mess with calico metadata service and internal SSL as well that there is no good solution to
15:58:58 <jrosser> it uses an iptables rule to forward traffic to the metadata service, rather than haproxy like neutron would normally do
15:59:21 <jamesdenton> the iptables method was the old way, even for neutron
15:59:22 <jrosser> so there is no opportunity to resolve (metadata http request) -> (internal endpoint being https)
16:02:00 <noonedeadpunk> #endmeeting