13:00:40 <mnasiadka> #startmeeting kolla 13:00:40 <opendevmeet> Meeting started Wed Sep 13 13:00:40 2023 UTC and is due to finish in 60 minutes. The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot. 13:00:40 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 13:00:40 <opendevmeet> The meeting name has been set to 'kolla' 13:00:42 <mnasiadka> #topic rollcall 13:00:43 <mnasiadka> o/ 13:00:46 <bbezak> o/ 13:00:47 <frickler> \o 13:00:52 <mattcrees> o/ 13:00:56 <jangutter> \o 13:01:22 <mmalchuk> o/ 13:01:41 <SvenKieske> o/ 13:02:15 <jsuazo> o/ 13:02:21 <ihalomi> \o 13:02:39 <mnasiadka> #topic agenda 13:02:39 <mnasiadka> * CI status 13:02:39 <mnasiadka> * Release tasks 13:02:39 <mnasiadka> * Regular stable releases (first meeting in a month) 13:02:39 <mnasiadka> * Current cycle planning 13:02:40 <mnasiadka> * Additional agenda (from whiteboard) 13:02:40 <mnasiadka> * Open discussion 13:02:42 <mnasiadka> #topic CI status 13:03:33 <mnasiadka> Overall is green I guess 13:03:48 <mnasiadka> But I think I don't like that multinode (ceph) jobs are really working nice in master 13:03:55 <mnasiadka> but they are failing in stable branches 13:04:06 <mnasiadka> the same way as they were breaking before we reconfigured RMQ to HA 13:04:28 <mnasiadka> Would we want to enable RMQ HA in tests/templates/globals-defaults for stable branches? 13:04:51 <mmalchuk> no? 13:04:53 <bbezak> in CI only? 13:05:04 <mnasiadka> in CI only 13:05:08 <mnasiadka> mmalchuk: why not? 13:05:16 <mmalchuk> oh on CI is ok 13:05:27 <mnasiadka> frickler: any thoughts? 13:05:29 <SvenKieske> seems ok 13:05:46 <mattcrees> Seems reasonable to me 13:05:48 <frickler> my other idea whether doing single node rmq only would be better 13:06:08 <frickler> but likely HA is fine, too, gives it some testing 13:06:12 <mmalchuk> frickler we need HA 13:06:28 <mmalchuk> on CI 13:06:33 <mnasiadka> let's try if switching those jobs to HA solve the issue of randomness 13:06:48 <mnasiadka> anybody willing to change it and do a couple of rechecks? 13:07:25 <bbezak> can take a look probably soonish 13:07:42 <mnasiadka> ok, bbezak wins ;) 13:07:48 <mnasiadka> #topic Release tasks 13:08:06 <mnasiadka> I didn't raise a patch for cycle highlights, will do after this meeting 13:08:12 <mnasiadka> no other release tasks seem to be waiting 13:08:22 <mnasiadka> #topic Current cycle planning 13:08:51 <mnasiadka> I don't think anybody spend time reviewing podman patches 13:09:06 <mnasiadka> and I think kevko did not update the LE Kolla patch according to reviews 13:09:14 <mnasiadka> so those two things need focus :) 13:09:50 <mnasiadka> I also proposed setting OVN jobs as voting, because it's getting more and more deployments, so I guess it would make sense to have it as voting 13:10:03 <mnasiadka> #link https://review.opendev.org/c/openstack/kolla-ansible/+/894914 13:10:33 <mnasiadka> SvenKieske: did you mention that td-agent needs bumping up? 13:10:57 <SvenKieske> yeah, currently figuring that one out, might not be soo trivial 13:11:12 <mmalchuk> do we switch to OVN from OVS later? 13:11:15 <mnasiadka> do you have that amount of time, or do you need help? 13:11:34 <mnasiadka> mmalchuk: that's on my backlog for working on that, we should have something beginning of next cycle 13:11:46 <mmalchuk> cool thanks 13:13:24 <SvenKieske> I'll should figure it out today or cry for help :D 13:13:29 <mnasiadka> ok :) 13:13:42 <mnasiadka> #topic Additional agenda (from whiteboard) 13:13:55 <mnasiadka> to zun or not to zun (jangutter). TL;DR, zun needs fundamental fixes to work with docker>20, etcd>3.3. Docker is a _host_ level dependency so vendoring options in the container won't work. 13:14:24 <jangutter> Does anyone have contacts with folks on the Zun team? 13:14:44 <mnasiadka> jangutter: I think it does work, but only for Ubuntu and Rocky 9 (since they still have 20.x available) 13:15:06 <mnasiadka> jangutter: there is one person I think, and he already responded on the ticket I created in Zun 13:15:06 <jangutter> yep, unless etcd is updated, of course. 13:15:11 <kevko> mnasiadka: didn't have a time 13:15:13 <kevko> :( 13:15:29 <frickler> there's only hongbin left in zun afaict, who I think has been around doing kolla patches, too 13:15:31 <mnasiadka> That's why I proposed we would copy the current etcd as etcd-zun, and upgrade etcd (that is used for everything else) 13:16:03 <mnasiadka> jangutter: if they don't shape up in C - we deprecate Zun and drop it in D (or earlier if we have to) 13:16:11 <jangutter> ah, right... running on different ports? 13:16:15 <frickler> mnasiadka: do you have a link to that ticket? 13:16:38 <mnasiadka> #link https://bugs.launchpad.net/zun/+bug/2007142 13:16:40 <frickler> maybe deprecate in B still? and undeprecate if things improve? 13:16:48 <mnasiadka> the goal is to fix it in C 13:16:59 <mnasiadka> I'm fine with deprecating in B, gives clear mandate to drop if they don't shape up 13:17:39 <jangutter> i.e. not remove, just don't run CI jobs on it? 13:17:45 <mnasiadka> jangutter: well, then the role would need to support it, right 13:17:57 <mnasiadka> deprecate means we're planning to drop, don't use it ;) 13:19:21 <jangutter> ok, I'll see what I can do with etcd-zun, can't promise too many hours. 13:20:59 <jangutter> will report back if it's feasible (migration path might be "fun") 13:22:31 <mnasiadka> well, seems like too much work 13:23:20 <jangutter> oof, yeah, bootstrap involvement: docker on the host needs access to it. 13:23:55 <mnasiadka> oh boy 13:24:18 <mnasiadka> so what's the alternative? do not update etcd or drop zun? 13:24:26 <mnasiadka> Anybody here that cares for Zun? 13:24:53 <jangutter> or pin your etcd container to a previous version of kolla? 13:25:07 <frickler> I'd favor updating etdc. maybe one can pin etcd for zun deployments? 13:25:20 <frickler> just add a reno for that, right 13:25:28 <SvenKieske> well not upgrading is not really an alternative imho 13:25:56 <SvenKieske> if we could pin for zun, that'd be nice, but I assume it's not that easy? would need a separate deployment somehow? 13:26:02 <SvenKieske> also zun relies on old docker 13:26:40 <mnasiadka> well, so maybe still a separate etcd container image for zun, and if you use zun - it's the one you get for deployment? 13:26:55 <mnasiadka> but then for such people - the upgrade will happen later 13:27:01 <mnasiadka> and they can't skip versions 13:27:02 <frickler> so ... drop zun now, restore when it gets fixed? 13:27:04 <mnasiadka> oh holy crap 13:27:09 <jangutter> that sound like a plan. etcd and etcd-3.3 13:27:11 <mnasiadka> yes, that's what I vote for 13:27:16 <mnasiadka> drop zun 13:27:23 <frickler> and people really needing zun will need to stay on 2023.1 until then 13:27:23 <mmalchuk> +1 13:27:53 <jangutter> any workaround is just going to postpone the inevitable, if not make it worse :-( 13:27:55 <mnasiadka> yeah, drop now, if they backport support for new docker into 2023.1 - we can revert 13:28:22 <SvenKieske> should we maybe announce that separately via ML? 13:28:43 <SvenKieske> maybe someone who cares is encourage to fix zun earlier then ;) 13:28:50 <mnasiadka> jangutter: can you please send a mail to openstack-discuss that we are going to drop zun support for that reason, and the plan is to revert the drop once Zun supports new Docker version - because we can't be running this etcd version forever? :) 13:29:13 <mnasiadka> let's wait a week if any massacre happens and start dropping 13:29:17 <SvenKieske> +1 13:29:31 <jangutter> will do, adding that if zun gains support it will be considered for the stable 2023.1 branch? 13:30:22 <mnasiadka> yes, something like that - we should probably drop it in kolla-ansible in a way that the Ansible inventory groups stay there, so it's easy to revert (without breaking anything) 13:30:41 <mnasiadka> and if they don't shape up - we drop the remnants in C 13:30:43 <jangutter> :thumbsup: 13:31:34 <mnasiadka> ok, let's hope case solved for now 13:31:53 <mnasiadka> Tap as a Service installation and configuration (jsuazo) 13:32:01 <mnasiadka> Did we get to any conclusion last time? :) 13:32:57 <jsuazo> I have a clearer picture as to why we are touching the upper constraints, which I addressed in the kolla proposal 13:33:27 <mnasiadka> Yeah, I see that 13:33:42 <mnasiadka> Question is if we should be installing it from a tarball, or honor the u-c 13:34:03 <jsuazo> TL;DR the pinned version is not working on X and above (tested versions), we tested de latest release, 11.0.0, which seems to work fine 13:34:28 <mmalchuk> afaik we need bump X version in u-c 13:34:53 <mnasiadka> release 11.0.0 is in master u-c 13:35:03 <jsuazo> We managed to add the plugin as a neutron addition from git, the branch of the source could be changed to the correct one, currently we are overriding the source to master on deployments 13:35:09 <mnasiadka> we are not considering to add that to any of the stable branches (no backports) 13:35:19 <mnasiadka> (it's a feature) 13:35:26 <opendevreview> Merged openstack/kolla-ansible stable/zed: CI: add q35 hardware machine type to tests https://review.opendev.org/c/openstack/kolla-ansible/+/894582 13:35:28 <opendevreview> Merged openstack/kolla-ansible stable/2023.1: CI: add q35 hardware machine type to tests https://review.opendev.org/c/openstack/kolla-ansible/+/894581 13:35:33 <opendevreview> Merged openstack/kolla-ansible stable/yoga: CI: add q35 hardware machine type to tests https://review.opendev.org/c/openstack/kolla-ansible/+/894583 13:36:11 <mmalchuk> so honor u-c should be fine 13:37:26 <mnasiadka> frickler: any controversial opinions? ;-) (apart the ones, that who needs TaaS or is that project alive) 13:37:46 <mnasiadka> up until now removing packages from u-c was rather exception, not a way of working 13:38:34 <jsuazo> mnasiadka: Should i propose a change to the u-c in X then ? (and remove the u-c editing from the proposal) 13:38:45 <mnasiadka> in Xena? 13:39:00 <mmalchuk> jsuazo no since it is feature 13:39:04 <mnasiadka> (changing u-c in stable branches is not something is going to pass from my perspective) 13:39:08 <mmalchuk> will not backport to xena 13:39:36 <jsuazo> ok, got it, will update the proposal then 13:39:55 <mnasiadka> you can backport the taas feature in your downstream branches and override the u-c, but I have a feeling taas is there for some reason 13:40:29 <mnasiadka> unless requirements team is happy with removing tap-as-a-service from u-c, then we can rework the build to use tarballs 13:40:49 <SvenKieske> td-agent is meeh, the repo installation changed to a good old "curl $internet | bash" -> voila you got a new repo under /etc/apt/sources.d/ :( 13:42:01 <jsuazo> We actually had a tarball installation initially, but my team didn´t like it that much :( 13:42:23 <mnasiadka> SvenKieske: oh boy, I'll try to have a look in mythical free time 13:42:25 <SvenKieske> and everything is now called "fluentd*" instead of "td-agent*" at least that can be mass changed I guess 13:42:40 <SvenKieske> https://docs.fluentd.org/installation/install-by-deb 13:42:50 <SvenKieske> https://www.fluentd.org/blog/upgrade-td-agent-v4-to-v5 13:42:59 <mnasiadka> mhm 13:43:07 <mnasiadka> let's discuss that next week when I'll know more about it :) 13:43:22 <mnasiadka> S3 backend support for Glance and Cinder-Buckup (jsuazo) 13:43:24 <SvenKieske> I guess it's doable, will try to extract URLs for gpg keys et all from that shell script :) 13:43:41 <mnasiadka> jsuazo: I think I commented on this patch today, do you need any help? 13:44:01 <jsuazo> i would appreciate if you gave the kolla-ansible TaaS proposal a look 13:44:02 <mmalchuk> me too as always add some comments 13:44:24 <mnasiadka> jsuazo: I'll have a look on the k-a side as well this week 13:44:38 <SvenKieske> have a link? I know I saw it somewhere but it got lost in the torrent of other tabs :D 13:44:42 <jsuazo> mnasiadka No help needed, Im keeping an eye and comments and trying to address them as I see them 13:45:15 <mnasiadka> kolla-ansible: https://review.opendev.org/c/openstack/kolla-ansible/+/885417 13:45:17 <mnasiadka> SvenKieske: ^^ 13:45:22 <mnasiadka> ok then 13:45:35 <mnasiadka> #topic Open discussion 13:45:39 <mnasiadka> anybody anything? 13:45:52 <mmalchuk> orphans-backports 13:45:57 <mmalchuk> https://review.opendev.org/q/I3ca4e10508c26b752412789502ceb917ecb4dbeb 13:46:01 <mmalchuk> https://review.opendev.org/q/I0401bfc03cd31d72c5a2ae0a111889d5c29a8aa2 13:46:05 <mmalchuk> https://review.opendev.org/q/I169e51c0f5c691518ada1837990b5bdd2a3d1481 13:46:10 <mmalchuk> https://review.opendev.org/q/Ief8dca4e27197c9576e215cbd960da75f6fdc20c 13:46:16 <mmalchuk> some merged, some one 13:46:42 <mmalchuk> and kayobe reviews 13:46:45 <mmalchuk> https://review.opendev.org/c/openstack/kayobe/+/861397 13:46:49 <mmalchuk> https://review.opendev.org/c/openstack/kayobe/+/879554 13:46:59 <mmalchuk> thats all from my side 13:47:42 <kevko> guys 13:47:42 <kevko> 1. ovn/ovs redeployed ..not working :( ... 13:47:42 <kevko> 2. Can anyone check podman kolla patch ? locally working ..zuul is failing while building SOME images 13:48:01 <mnasiadka> I should have some more time next week 13:48:08 <mnasiadka> I can have a look around podman kolla patch 13:48:34 <SvenKieske> that would be great, I'm also out of ideas there, and I looked at the logs.. (maybe the wrong way) 13:49:01 <kevko> mnasiadka: there is no reason to not work ..but it's not in zuul ...locally working like a charm 13:50:27 <frickler> if all other debugging fails, we can hold a node and check what is going on there 13:50:37 <mnasiadka> kevko: well, basically if it fails on upgrade jobs, so it's checkin out previous release code - but what do I know :) 13:51:04 <kevko> but it is failing on kolla build 13:51:12 <kevko> so it is not related to upgrade job 13:51:31 <kevko> because upgrade job is just deploy -> build images -> upgrade 13:51:39 <kevko> and it is failing on build images 13:51:46 <kevko> everything is built OK ..but not glance,horizon,eutron 13:52:38 <mnasiadka> I saw pip is failing or something similar 13:52:51 <mnasiadka> but it's working ok on all other jobs, just not upgrade :) 13:53:34 <mnasiadka> anyway - I unfortunately need to run - feel free to resolve that issue while I'm gone :) 13:53:38 <mnasiadka> Thank you all for coming 13:53:40 <mnasiadka> #endmeet 13:53:43 <mnasiadka> #endmeeting