18:00:52 <gouthamr> #startmeeting tc 18:00:52 <opendevmeet> Meeting started Tue Mar 25 18:00:52 2025 UTC and is due to finish in 60 minutes. The chair is gouthamr. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:00:52 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 18:00:52 <opendevmeet> The meeting name has been set to 'tc' 18:01:03 <gouthamr> Welcome to the weekly meeting of the OpenStack Technical Committee. A reminder that this meeting is held under the OpenInfra Code of Conduct available at https://openinfra.dev/legal/code-of-conduct. 18:01:09 <gouthamr> Today's meeting agenda can be found at https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee 18:01:11 <gouthamr> #topic Roll Call 18:01:18 <bauzas> \o 18:01:19 <spotz[m]> o/ 18:01:35 <gtema> o/ 18:01:56 <frickler> \o 18:01:58 <noonedeadpunk> o/ 18:02:18 <mnasiadka> o/ 18:03:35 <gouthamr> courtesy-ping gmann cardoe 18:03:44 <gmann> o/ 18:05:49 <gouthamr> alright, its that magical time of 18:05 UTC, lets get started.. 18:05:57 <gouthamr> #topic Last Week's AIs 18:07:05 <gouthamr> we had a couple of things: 18:07:05 <gouthamr> 1) take operator engagement concerns to the PTG 18:07:20 <gouthamr> we added a topic to the etherpad: 18:07:23 <gouthamr> #link https://etherpad.opendev.org/p/apr2025-ptg-os-operators 18:07:49 <bauzas> +1 18:07:51 * gouthamr hopes that's the etherpad we'd use :) either case, save the date if you can attend: 18:07:51 <gouthamr> 17 UTC on Friday April 11 18:08:18 <gouthamr> 2) TC meeting time poll 18:08:52 <gouthamr> #link https://framadate.org/os-tc-2025-2 18:09:23 <gouthamr> cardoe and spotz[m] haven't weighed in here 18:09:45 <spotz[m]> Dang though I had hang on 18:10:08 <gouthamr> okay, we can visit this topic at the end of this meeting 18:10:21 <gouthamr> that's all the AIs i was tracking, was anyone else working on anything? 18:12:14 <spotz[m]> Ok looking at my current meeting schedule it's bad:( But might be different after Kubecon 18:12:43 <gouthamr> #topic PTG Planning 18:12:43 <gouthamr> ^ a reminder to add topics to the etherpad here: 18:12:54 <gouthamr> #link https://etherpad.opendev.org/p/apr2025-ptg-os-tc (OpenStack Technical Committee vPTG etherpad) 18:13:48 <cardoe> sorry got stuck on a call. 18:14:57 <gouthamr> i'll slot these into specific times next week.. i'm hoping to get a split of topics that are good for the community to attend/participate in, and regular business where we'd take whatever participation we can get 18:15:31 <gouthamr> ack cardoe, please do fill out https://framadate.org/os-tc-2025-2 18:15:40 <cardoe> doing it now 18:15:41 <spotz[m]> I'll be PTO that week 18:15:57 <gouthamr> ++ i've noted that about you and mnasiadka 18:16:36 <gouthamr> please feel free to add topics nevertheless if you think one of us can seed the discussion 18:16:49 <mnasiadka> Yeah, I'll be in NZST timezone that week 18:20:00 <gouthamr> #topic A check on gate health 18:20:16 <gouthamr> any CI updates to share this week? 18:20:33 <frickler> ubuntu kernel bug breaking jobs in neutron and kolla 18:20:41 <clarkb> the same issue that hit jammy in december 18:20:57 <frickler> we reverted to an old noble image and stopped rebuilds, so we are fine for now 18:21:23 <clarkb> and setuptools 78 rolled out breaking changes that broke many people though impact to openstack seemed minimal. They rolled back the change and now there is much discussion in python land about how to move forward 18:21:39 <clarkb> it is a good reminder that our packages should convert -'s in metadata names to _'s though 18:21:56 <gouthamr> ah, ty for both these updates.. 18:21:57 <mnasiadka> We managed to do it in Kolla before they did a revert, so maybe they did succeed ;-) 18:23:05 <noonedeadpunk> just seen a couple of timeouts last week which were not there for quite some time 18:24:00 <fungi> the centos 9 mirror was broken (sync'd an inconsistent state from an upstream mirror) for a few hours yesterday too 18:25:45 <gmann> one thing to update, devstack/grenade/tempest setup for new stable/2025.1 and current master is almost done. main setting are merged but a few more things are in gate 18:25:49 <gmann> #link https://review.opendev.org/q/topic:%22qa-2025-1-release%22 18:27:38 <gouthamr> clarkb: my very quick search on codesearch.o.o shows me that all openstack setup.cfg files are fixed up, there is some boilerplate/tests/examples that need to be addressed.. i see lots of fixes possible for the non openstack/ though: like, https://opendev.org/zuul/zuul-jobs/src/branch/master/setup.cfg 18:28:00 <clarkb> gouthamr: ya an in theory pbr is doing the conversion for us but then setuptools did its own validation again and exploded 18:28:14 <clarkb> which is like we tried to do the right thing the easiest way possible and they broke us anyweay 18:28:56 <clarkb> gouthamr: within openstack I guess the problems were all in dependencies 18:29:40 <gouthamr> ++ 18:30:28 <fungi> well, and the aforementioned kolla patch 18:30:34 <gouthamr> ty for all the updates, the grenade one is important, and nice to knock it off as soon as the cycle begins 18:31:07 <gouthamr> #topic TC Tracker 18:31:13 <gouthamr> #link https://etherpad.opendev.org/p/tc-2025.1-tracker (Technical Committee activity tracker - 2025.1) 18:32:19 <gouthamr> frickler++ on the war footing merges last week :D 18:33:05 <frickler> yes, sadly even more zuul config errors now, need to do some follow ups 18:33:10 <gouthamr> #link https://review.opendev.org/c/openstack/releases/+/942218 (Yoga EOL) 18:33:10 <gouthamr> #link https://review.opendev.org/c/openstack/releases/+/942201 (Xena EOL) 18:33:10 <gouthamr> #link https://review.opendev.org/c/openstack/releases/+/941458 (Wallaby EOL) 18:33:30 <gouthamr> ah.. yes, we couldn't know if we didn't start cleaning up 18:34:02 <frickler> also no progress afaict on cleaning up issues for the things we did not eol 18:34:43 <gouthamr> ack 18:35:18 <gouthamr> these are repos that have (un) maintainers.. i suppose we can narrow things down at the PTG 18:36:10 <gouthamr> i'll go down the list and seek updates, because we'll close this etherpad and create a new tracker at the PTG 18:36:18 <gouthamr> https://etherpad.opendev.org/p/tc-2025.1-tracker 18:36:35 <gouthamr> please share any updates if you'd like on items that you've been tagged with 18:37:01 <gouthamr> anything else on the tracker? 18:37:37 <gouthamr> #topic Open Discussion and Reviews 18:37:37 <gouthamr> 18:37:48 * gouthamr copy-pastes from teh agenda 18:37:50 <gouthamr> Non-auditable process of skyline releases, ie: https://opendev.org/openstack/openstack-ansible-os_skyline/src/branch/master/tasks/skyline_install_yarn.yml#L126-L127 That is a result of building static files with yarn, but potentially it should be completely offloaded to Zuul to prevent malicious code injection during such manual patches. 18:39:39 <fungi> yeah, i recall we discussed it in #openstack-infra recently at length 18:40:35 <fungi> pep 770 will in time provide a mechanism for recording sboms as static data files shipped in sdists/wheels 18:40:53 <mnasiadka> FWIW I don't think we're building static files with yarn in kolla - but I haven't used skyline really. frickler do you have any... experience? 18:40:58 <fungi> there's a yarn plugin apparently to auto-generate cyclonedx sboms 18:41:13 <noonedeadpunk> well we do in osa 18:41:29 <fungi> but also, a short term stop-gap would be to amend the manifest to include the yarn.lock file used at build time 18:41:38 <noonedeadpunk> but the biggest problem is that they do a human made patch for the realease of the amount that is non-verifiable 18:41:40 <frickler> I never did that 18:42:14 <bauzas> I have no context either so far 18:42:27 <clarkb> noonedeadpunk: patch of what? Sorry I don't understand what is being patched 18:43:02 <noonedeadpunk> and that is actully somehow reminds me of xz being compromised in an alike way 18:43:14 <noonedeadpunk> #link https://review.opendev.org/c/openstack/skyline-console/+/945065 18:43:49 <fungi> bauzas: the larger problem is that we have openstack projects (horizon does it too) shipping embedded copies of random libraries developed outside openstack, and these are not easily inspected or tracked for updates, often falling well out of date and including known vulnerabilities, which our users of those files are not notified about in any way 18:43:50 <noonedeadpunk> so they do prepare it for releasing skyline-console so that it was containing the rightfully built content 18:43:53 <clarkb> oh they are committing the build artifacts into the repo. they shouldn't do that either way 18:44:10 <noonedeadpunk> yup... 18:44:16 <clarkb> butthen I agree that is the same sort of vector used by xz. Use opaque gzip data as the transport layetr 18:44:50 <fungi> yeah, the more narrow problem in skyline is that they're committing compiled versions of those libs into git, not even doing it automated at build time 18:45:08 <gmann> If i am recalling correctly but isn't that one of the things to check when skyline project status changed from emerging to active projects ? 18:45:29 <bauzas> I see, a security attack vector indeed 18:45:39 <noonedeadpunk> So sorry if I mislead by original description 18:45:59 <noonedeadpunk> no I think we totally missed the process 18:47:04 <gouthamr> gmann: not the same issue: https://review.opendev.org/c/openstack/governance/+/924109/comments/510391ea_9cf4bc38 18:47:55 <gmann> gouthamr: I mean we missed to check this in that change. I think that was one of the thing we discussed to take it as emerging project and not active 18:48:21 <gmann> and one of the few things they should solve before becoming the Active project 18:48:42 <bauzas> looks important indeed 18:48:53 <frickler> well we made it active, didn't we? 18:50:20 <gmann> yes, we made it active 18:50:21 <noonedeadpunk> I think we did 18:51:05 <gmann> this is good email thread I found where fungi mentioned all points for skyline team to solve 18:51:07 <gmann> #link https://lists.openstack.org/pipermail/openstack-discuss/2021-December/026254.html 18:51:54 <spotz[m]> cardoe: I know you all are using Skyline, is this something you all could possible help with? 18:52:15 <spotz[m]> My thought maybe they just need help and guidance to resolve this 18:52:19 <cardoe> I've really wanted our folks to get involved. 18:55:02 <fungi> it's wholly possible i missed things though, i had limited available time to audit the state of their projects 18:55:08 <noonedeadpunk> so I guess it's a question now on how we should proceed with this, given that project was made active 18:55:30 <noonedeadpunk> as apparently this is a case for TC to step in a way 18:55:31 <clarkb> step 0 might be trying to reproduce what was built 18:55:34 <bauzas> Should we signal it ? 18:55:38 <fungi> looks like i didn't bring up any of the javascript content at all 18:55:46 <clarkb> if that chceks out then the risk is probably low and they can work to fix in the next cycle 18:55:54 <clarkb> if that doesn't check out then you have bigger questions 18:56:00 <bauzas> like a disclosure 18:57:54 <frickler> pretty likely yarn builds are not reproducible bit-by-bit like when deps got updated, what then? 18:58:14 <clarkb> frickler: they should have a lockfile and the diff should probably be minimal if using the same version of the lock? 18:58:24 <clarkb> I mean its effort and I'm not signing up myself for this. But I think it is one path forward 18:58:44 <mnasiadka> Looking at the brief list of items from this Dec 2021 thread - shouldn't there be a resolution that this is the framework that all projects need to comply with? (briefly the list that fungi mentioned there and probably some more) 18:59:31 <fungi> we have a list: 18:59:37 <fungi> #link https://governance.openstack.org/tc/reference/new-projects-requirements.html Requirements for new OpenStack Project applications 18:59:44 <fungi> but it could certainly stand to be improved 19:01:04 <gouthamr> we're at the hour, but we can close out with this topic 19:01:19 <bauzas> please yeah 19:01:34 <gouthamr> can someone take a stab at bringing this issue to the ML? 19:02:40 <gouthamr> we've struggled to get conversations going with skyline contributors on the ML/IRC, but, i can't think of a better way to have a public discussion on something that's not a code change 19:03:21 <gouthamr> we'd do this also to bring attention to deployers/distros and operators apart from the contributors 19:03:31 <fungi> i think project leaders have reached out to them through wechat in the past 19:03:41 <noonedeadpunk> well, I can try to reproduce the process, sure, as we do perform yarn build in osa 19:03:44 <fungi> might at least be able to give them a heads up that it's being discussed 19:04:04 <fungi> (and where, in case they want to participate in the discussion) 19:04:23 <gouthamr> we can alert them wherever to come respond to the ML :D 19:05:12 <gmann> I think language is also one of the challenge for them to be less active on ML 19:05:27 <gmann> at least to read a lengthy emails or so 19:06:45 <gouthamr> yes, we need this to be broken down into problem and suggestion to be helpful.. i think we've identified problems with them in the past, and they don't know what they'd do to fix it? or they may not understand why they should care.. 19:07:04 <bauzas> language is a barrier for many of us :-) 19:08:03 <gouthamr> alright, 7 minutes over, don't mean to keep us on this.. let me end the meeting so we can chat async about this 19:08:10 <gouthamr> thank you all for attending 19:08:14 <gouthamr> #endmeeting