Tuesday, 2024-11-19

*** elodilles_pto is now known as elodilles08:03
tkajinamI'm always confused by the policy about EM/EOL, but who should decide whether an unmaintained branch should be retired, in the project without unmaintained liaison assigned. Can PTL make the decision ?08:26
fricklertkajinam: yes, in my understanding of the policy, the final decision is always with the PTL09:44
opendevreviewMerged openstack/project-team-guide master: Update instructions for adding new requirements  https://review.opendev.org/c/openstack/project-team-guide/+/93527009:56
cardoeI need to unfortunately miss TC meeting. It's right at the same time I have to take my son to a doctor's appointment.13:56
gouthamr cardoe ack; no problem!15:11
slaweqgouthamr hi, I have some errands to do today and I may be late on the TC meeting. I will do my best to be there but please don't wait for me to start15:47
gouthamrack slaweq thanks for letting me know15:47
opendevreviewDmitriy Rabotyagov proposed openstack/governance master: Retire Murano/Senlin/Sahara OpenStack-Ansible roles  https://review.opendev.org/c/openstack/governance/+/93567716:35
gouthamrtc-members: a gentle reminder that we're meeting here in ~59 minutes17:01
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-manuals master: Retire Senlin/Sahara/Murano roles for OSA  https://review.opendev.org/c/openstack/openstack-manuals/+/93568417:02
opendevreviewDmitriy Rabotyagov proposed openstack/governance master: Add ansible-role-httpd repo to OSA-owned projects  https://review.opendev.org/c/openstack/governance/+/93569417:57
gouthamr#startmeeting tc18:01
opendevmeetMeeting started Tue Nov 19 18:01:09 2024 UTC and is due to finish in 60 minutes.  The chair is gouthamr. Information about MeetBot at http://wiki.debian.org/MeetBot.18:01
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.18:01
opendevmeetThe meeting name has been set to 'tc'18:01
gouthamrWelcome to the weekly meeting of the OpenStack Technical Committee. A reminder that this meeting is held under the OpenInfra Code of Conduct available at https://openinfra.dev/legal/code-of-conduct.18:01
gouthamrToday's meeting agenda can be found at https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee18:01
gouthamr#topic Roll Call18:01
noonedeadpunko/18:01
gmanno/18:01
gtemao/18:01
bauzas\o18:02
gouthamrnoted absence: c a r d o e, s l a w e q18:02
fricklero/18:02
gouthamrcourtesy-ping: spotz [m]18:03
gouthamrnot in the channel, likely somewhere in the conference circuit :) 18:03
gouthamrlets get started.. 18:04
fricklermay be an issue with the matrix bridge hiding ppl18:04
gouthamrack18:04
gouthamr#topic Last Week's AIs18:04
gouthamrwe took an AI to see if we can revive the erstwhile Third-party CI SIG through the help of project teams that have a requirement for third party CI18:06
spotz[m]o/18:06
gouthamrwe had a pretty long discussion following our meeting, and jbernard brought it up at the cinder weekly meeting.. i don't suppose we have updates already.. we did suggest that project teams must evaluate the requirement if we're unable to support contributors putting together their CI 18:07
gouthamrre-evaluate*18:07
gmannI think this is settle now? I saw it was discussed in cinder meeting and some operator having 3rd party CI mentioned to help/share doc etc to cinder team18:07
jbernardi can give a quick update from cinder if there is interest18:07
gouthamrplease do, jbernard 18:08
jbernardsure,18:08
gouthamr#link https://meetings.opendev.org/meetings/cinder/2024/cinder.2024-11-13-14.01.log.html#l-23 (cinder meeting discussion of third party CI)18:08
jbernardi reached out to Inori directly to understand the cinder-specific issues, i think progress as been made on that front, my plan is to incorporate that feedback into the doc we create18:08
jbernardas a team, we agreed that we need to do better in documenting our ci requirements and setup proceedure18:09
jbernarda few volunteered to help with a doc18:09
jbernardim in the process of collecting all that we have currently, with the plan of consolidating it into a single workign document for our ci reference, both for newcomers and existing vendors for reference18:10
gouthamr++ that's great18:10
clarkbwould also be good to share with other projects that require/use third party ci so they can see if it can be adapted to their needs too18:10
gmannjbernard: ++, thanks. really appreciate that.18:11
clarkbI don't think cinder necessarily has to solve any manila or ironic or nova problems, but at least make them aware that the documentation is there and can be reconsumed18:11
jbernardclarkb: certainly, once we have something that's usable, ill send a mail to the list18:11
clarkb++ thanks18:11
gouthamrthanks jbernard 18:11
gmannclarkb: ++18:11
gmannmaybe we can add something in p-t-g even it is a cinder example but will be good ref18:11
gouthamrthat's all the AIs we took last week.. was anyone tracking anything else?18:11
jbernardideally we could have a working poc project to use as a starting point, but im not yet sure how feasible or how much time that will take18:11
jbernardbut if it looks like it coule be in reach, we'll try to go for it18:12
jbernardthat's all on ci that I have18:12
gouthamryeah, if the docs are frustrating to follow, someone will step in and automate it :) 18:12
gouthamrmoving on.. 18:13
gouthamr#topic Status of migration to Ubuntu Noble (gmann)18:13
gouthamrgmann: floor is your18:13
gouthamrgmann: floor is yours*18:13
gmannI sent the status mail yesterday night18:14
gmann#link https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/thread/JOMDY26TCW7OX3NXRGOYQCIDXNNJ4E25/18:14
gmannor you can see the current status in ertherpad which will be more live18:14
gmann#link https://etherpad.opendev.org/p/migrate-to-noble#L3718:14
bauzasthanks gmann for the hard work18:14
gmann14 projects are all green on Noble (for many fixes/changes are up  to merge once base devstack/tox jobs are ready to migrate18:15
gmannother projects failing but there are cases which might be non-noble related issue18:15
gtemafor it is honestly a pain - openstackdocstheme is broken due to pbr and to fix this one is also not trivial18:15
gmannI am going through those and trying to fix/filter out them for noble specific issue18:16
gouthamri'd throw in one more: manila.. we identified a couple of issues and i've pinned jammy to debug them18:16
clarkbgtema: fwiw I've seen that pbr change but pbr works fine with python3.12 in zuul. It still isn't clear to my why that change is necessary18:16
gtemaI also wonder about some statuses in the etherpad since i.e. first change to keystone today failed on docs job where now we need to drop sphinxcontrib-*diag18:16
gmanngtema: yes. doc job is not yet ready due to openstackdocstheme 18:16
bauzasgtema: because of ceph, heh ? :)18:16
gmann#link https://bugs.launchpad.net/pbr/+bug/208836018:16
bauzaswhoops18:16
bauzass/gtema/gouthamr18:16
clarkbbut also I think that change is close? its the logging changes that are holding it up now18:16
gtemaclarkb - locally it passes18:17
gmanngtema: clarkb:  can you add the link?18:17
gtemawith py36 I mean (no test for py27)18:17
gtema#link https://review.opendev.org/c/openstack/pbr/+/92421618:17
gouthamrbauzas: actually, i'm scratching my head about ceph jobs.. i've to test locally to reproduce issues you and rdhasman reported.. but, gmann's changes to move ceph jobs to noble pass in the CI.. 18:17
gtemait is referred in the launchpad issue18:17
bauzasgouthamr: :cross_fingers_emoji_ish:18:18
gmanngtema: ++18:18
gouthamrbauzas: here's a no-op against devstack-plugin-ceph: https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/935398 ,.. the failure is in the docs job18:18
clarkbgtema: if I had to guess setuptools in other platforms doesn't configure logging at that "low" (info) level or similar18:18
clarkband going through the old distutils mechanism did?18:19
gtemaand literally - without doing anything docs job in keystone are broken today (due to blockdiag) and it is also a pain to replace them - there are lot of those18:19
gmanngtema: I logged for sdk as openstackdoctheme use story and it is very hard to log and track in two places18:19
clarkbbut again it isn't clear to me why this is needed18:19
gmannbut thanks for checing and adding detail there18:19
clarkbpbr works under python3.12 if you install setuptools18:19
gtemaclarkb - openstackdoctheme does import from pbr which is not working18:19
clarkbgot it this is runtime use of pbr not install time18:20
clarkbthat is helpful18:20
gtemaclarkb: soo https://bugs.launchpad.net/pbr/+bug/2088360/comments/318:20
fungiis that with setuptools already installed?18:20
clarkbfungi: based on that traceback I'm thinking no18:20
gtemasome bits in pbr still import distutils18:20
fricklercan we maybe discuss the technical details after the meeting? this is getting a bit out of scope for me18:20
clarkbfrickler: sure18:20
gmannyeah, we can discuss after meeting18:20
fungiyeah, newer python dropped distutils, if you need distutils on newer python you can get it by installing setuptools, which now provides it18:21
gmannso my main purpose to add this topic here is that deadline for migrating the base jobs is Nov 29 and even there are many projects not green yet. I would like to know if we want to extent the deadline ?18:21
gtemagmann - looking to amount of issues I do not believe we would make it18:21
gouthamrawesome, thanks for sharing the status of the goal here and for all the hard work around this, gmann .. 18:21
gouthamrwhere does that deadline come from?18:22
gmannpersonally I do not want to extend and if any project stay in broken state then we have option to pin the nodeset as jammy and they can continue fixing it18:22
gtemaespecially that some changes were open since months and nobody reviewed them18:22
fricklerIMO nothings going to happen in december, either, so I don't think that that would help much18:22
gouthamr^ +118:22
clarkbfrickler: the inverse may be true though right? its a low impact time to make changes like that and let them more slowly get fixed?18:22
gmannwell we will be migrating base jobs early in cycle and moving deadline make situation wortst18:22
clarkbvs trying to do it closer to the release when everyone is cramming in features last second18:23
noonedeadpunkI aree that it's better to do in the beginning18:23
frickleryes, so keep the deadline, switch, put more pressure on projects to fix things18:23
* gouthamr isn't clear how we picked the deadline.. 18:23
gmanngtema: true but somewhere we have to break the things to fix and if any project is not activly checking their failure then they can pin the nodeset18:23
gtemahmm, that is so untrue - pbr is broken and it affects everybody18:24
fricklergouthamr: iiuc it is already delayed from "early in the cycle"18:24
gtemaand the change there is open since July18:24
gmannand if some complex issue is there and project need time we have workarond also instead of holding complete CI18:24
noonedeadpunkwait, so you're saying that pbr is broken with py3.12?18:24
gtemayes, pbr is not working with 312 as of now18:25
bauzaswhy exactly ?18:25
fungiinstalling pbr without setuptools is broken18:25
gtemait has imports of distutils which are dropped18:25
bauzasI got no issues with noba18:25
bauzasah that18:25
gmanndisutils is not there 18:25
fungidistutils moves from python stdlib into setuptools18:25
noonedeadpunkbut wasn't 3.12 in pti for 2024.2?18:25
clarkbthis is fud18:25
gmannnoonedeadpunk: it was18:25
clarkbpbr works fine it requires setuptools you must install setuptools18:25
gmannwhere there is no setuptools it is issue18:26
gtemanoonedeadpunk - it makes so much difference whether you do tox -e py312 or you use py312 as a base for all jobs18:26
clarkbit could be improved but it is not required to work with python3.1218:26
noonedeadpunkYeah, I think it's true what clarkb is saying18:26
noonedeadpunkgtema: well, I'm running 24.04 for almost half a year, and never saw tox failing due to pbr so far18:27
noonedeadpunkmaybe it's not always used....18:27
gtemaright, so now pls get i.e. keystone project and try to build docs with py31218:27
gmannlet's review it on gerrit but I do not think that should be blocker for migration as things work with setuptools18:27
noonedeadpunksec18:27
gouthamrokay; couple of you have expressed that this is ETOOMUCHDETAIL for this meeting; lets discuss implementation concerns right after we end.. 18:27
gmannwe still have 10 days and we can see how many get fixed. I am concentrating on this migration only and requested project team also18:27
gouthamrgmann: it looks like there's no need to delay this18:28
gmannso I think we are good to stay on deadline Nov 29 and not ok for extending it right?18:28
gmanngouthamr: yeah, holiday time is good to fix the things also18:28
fricklerswitching docs builds to noble/3.12 doesn't need to be coupled with that18:28
gmannfrickler: exactly 18:28
gtema-1 for blindly sticking18:28
fricklerso just switch unit tests + devstack18:28
gmannwe do not need to hold all things for a small portion/projects18:28
bauzashow big is that portion ?18:29
gmanngtema: doc job will not be migrated part of this 18:29
bauzasI tend to agree with you but I want to be sure that's a minority18:29
gmannthat is already separated out18:29
gouthamrdocs jobs, failing project jobs noted in the etherpad.. 18:29
gmann#link https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/935459/118:29
gtemaI hate this mix - it is too dirty and not helpful18:30
gmannwell, holding the integration testing for this is not good idea18:30
gouthamrgtema: i think we'd want to move incrementally.. 18:30
gmannand delaying the migration make it worst to do it during rlease time18:30
noonedeadpunkgtema: it failed but with completely different issue: https://paste.openstack.org/show/bKzQbfNEiaAoG7QQbrz4/18:31
gmannwe have faced that in past when I personally wanted to have EVERYTHING GREEN before any job migration18:31
gouthamrgtema: they're really not testing the same things, so we're batching things that are alike already.. no need to club all OpenStack CI Ubuntu jobs under one bucket, imo18:31
gtemanoonedeadpunk - that is what I mean, we just say we take different py version and all of a sudden there are plenty of issues not related to the project itself supporting this version or not18:32
noonedeadpunkI don\t think it's py verison, I would put on u-c recent update18:32
gmannit passed here18:32
gmann#link https://review.opendev.org/c/openstack/keystone/+/93293418:32
gtemaI am frustrated - for me now nearly everything is broken, for already a week I am just on fixing (workaround) things everywhere in and out of my reach18:33
gtemaI can't work on any features, only repairing what was working before18:33
noonedeadpunkI have exact same issue with python 3.11 fwiw18:33
fungigmann: 932934 ran openstack-tox-docs on jammy, fwiw18:34
gmannfungi: yeah just checked https://zuul.opendev.org/t/openstack/build/a3d99e308ad74ee0ad7919f7b07490fd/log/job-output.txt#4918:34
gmannit was after I separated out the doc job migration18:34
noonedeadpunkcould it be that? https://opendev.org/openstack/requirements/commit/376cf882ece239abee0e2c2ffb83f3bbefc1819c18:34
noonedeadpunkwell. plenty u-c were bumped today18:35
gmannanyways, I am keeping deadline same and we will see how things goes in those 10 days. doc job migration can be done later once we are ready18:36
noonedeadpunklet's try recheck your patch and see if it still works :)18:36
gtemanoonedeadpunk - everybody knows that those blockdiag are not really expected to work and majority of projects replaced them with graphviz, so it is not a question what broke it. We need to replace it. I am just tired of only fixing things to the left and right18:36
noonedeadpunkI know, I know...18:36
noonedeadpunkI'm not arguing that plenty of things are broken and bumping u-c results in failures like that quite often...18:37
gouthamrokay, lets wrap up this discussion, and move to the next topic 18:37
gouthamr#topic Watcher's launchpad tracker recovery18:37
gouthamr#link https://answers.launchpad.net/launchpad/+question/81933618:37
gmanngtema: feel free to add bugs you are facing in etherpad that will be helpful to know where we are18:38
bauzasno answers unfortunately from my side :(18:38
gouthamr^ so launchpad admins provided the projects (watcher, watcher-dashboard and python-watcherclient) to openstack-admins18:38
gmanngouthamr: this is still not openstack-admin 18:38
gouthamrcan you folks kick out the watcher-drivers here?18:38
gmann#linl https://launchpad.net/~watcher-drivers18:38
gmann#link https://launchpad.net/~watcher-drivers18:38
bauzasI invited him thru LinkedIn but no response so far and I can't InMail18:38
gmannor you are expecting to create a new driver team?18:38
gouthamryes18:38
gmannI was expecting to have ownership of watcher-driver changed18:39
gouthamri'm asking openstack-admins to take over now, and allow sean-k-mooney and folks to create a different team18:39
gouthamrgmann: me too18:39
spotz[m]Yeah I don't see any of the admins there even individually18:39
gouthamrbut, is this sufficient?18:39
gmannhumm, this is workarond we can do but did they deny changing ownewrship of driver?18:39
gouthamrbauzas: david's been reading my linkedin messages, but, like i mentioned earlier.. i don't think he can do anything, his launchpad account is associated with his former employer.. he may have no way to log in18:40
gmanngouthamr: maybe we can try that also, if you can ask them to do this change also?18:40
gouthamrgmann: because we want consistency?18:40
fungiopenstack-admins group members don't allow/prevent creating new groups in lp, just make sure to add openstack-admins as the owner when creating18:40
bauzasgouthamr: ack18:40
gmanngouthamr: yeah, otherwise we have to create another driver team with different name18:41
sean-k-mooneyi think normally there are 3 teams18:41
gouthamrgmann: yeah, my problem was the delay we've had to get LP folks to act on this18:41
gmanngouthamr: I mean that is doable but let's try if they can hand over the existing driver team18:41
sean-k-mooneydrvier, a open bug team and a coresec team18:41
gouthamri must send Billy Olsen a thank you message :D18:41
fricklerseems they added openstack admins as maintainers of the 3 projects, so we should be able to go on from that18:41
gouthamrsean-k-mooney: if that's what the team wants18:41
gmannas you asked "We'd like help to reassign the "watcher-drivers" team to "openstack-admins".. Or a way to change the ownership of the three projects to "openstack-admins"." 18:41
gouthamri know, but they ignored that :D 18:42
fungiprojects can get by without a bug supervisor group if they allow open bug assigning and targeting18:42
gmannand they did 2nd portion. I think they can do 1st also?18:42
sean-k-mooneygouthamr: well we dont nessiarly need 3 but i woudl like to have at least 2. but we can figure out the details later once openstack-admins is an owner of the current one18:42
gmanngouthamr: can you post it explicitly and if they do not do withtin this week then we can go with another driver team18:42
gouthamrgmann: okay can do.. 18:43
gmanngouthamr: thanks18:43
gmannas we are waiting for sean-k-mooney proposal for core team things also this week, we can wait for LP also for a week18:43
sean-k-mooneyyep18:43
gouthamrsean-k-mooney: ack; do let openstack-admins know what you'd like once that's done.. 18:43
gouthamrthis topic is not entirely about watcher18:44
sean-k-mooneywe are hoping ot restart the watcher irc meetings so we can add it as an adgenda item and report back18:44
gmannI can handle that from admin part side once it is added in driver or create new one. will work with sean-k-mooney on that18:44
gmannsean-k-mooney: ++18:44
gouthamrif there are any other launchpad trackers that are out of "openstack-admins" purview, we'd need to get them fixed.. please alert if you know any18:44
gouthamrlets move on.. 18:45
gouthamr#topic A check on gate health18:45
fricklerflaky I'd say18:45
gouthamrany gate health concerns, not pertaining to the noble effort?18:45
fricklerlots of gate failures I saw during the release and reqs rush today18:46
clarkbdocker hub may or may not be enforcing more restrictive rate limits18:46
gmannI have not checked on much this week but saw tempest jobs not so bad and things are merging in Tempest/nova18:46
fricklernothing standing out in particular, the usual volume stuff18:46
clarkbthat probably doesn't affect most openstack projects though. The deployment projects are affected though18:46
gouthamrthere were 10 requirements bumps merged in the past few hours :O18:46
gtemapartially: https://review.opendev.org/c/openstack/releases/+/934416 (os-api-ref) release is necessary since some api-ref jobs are broken without people knowing that18:46
gmanngouthamr: :) good timing :P18:47
gtemanova api-ref is broken and fix is struggling to be done since 2 weeks18:47
fricklergouthamr: and more pending18:47
gouthamr\o/18:47
bauzasgtema: yup, indeed 18:47
bauzasI can say that nova api-ref changes are broken now18:48
fricklerI'll ping the other release team members about that patch, missed it due to the e-1 rush18:49
gouthamrthanks gtema frickler 18:50
gouthamrso gate flakiness must be expected as we start rolling out with these UC bumps18:50
gouthamrwe'll check back next week and hopefully there isn't a need for any reverts...18:50
gouthamranything else about $topic 18:51
fricklerIMO it is unrelated, but we'll see, ack18:51
bauzasyet again a question of how we can ensure we don't trample projects everytime we bump u-c18:51
bauzas:)18:51
fungirunning a lot more test jobs is the only solution i'm aware of18:51
gouthamrwe have "cross" jobs.. but these are unit test jobs and maybe the coverage is insufficient - this is something project teams can assess and help with18:51
fricklerbauzas: have stable (non-flaky) integration jobs that we can afford to run on the reqs repo18:52
fricklerand make them faster than 2h if possible18:52
gmannwe hve many project testing on req but not all18:52
bauzasyeah, I'd appreciate if we could run tempest scenarios on a u-c change18:52
gmannwe do right?18:52
gmanntempest-full-py3 is there18:53
fricklerwe run tempest-full-py3 and one sdk integration job18:53
bauzasI dunno, b/c we recently hit hard problems to solve in a short timeframe18:53
bauzashmmm18:53
fricklerfeel free to add more jobs, but also be prepared to have them removed again if they turn out to be failing too often18:53
bauzasI then need to recollect *why* we missed that last time18:53
* gouthamr recalls the issue being with openstacksdk changes18:54
bauzasmy brain is fried tonight so I miss context18:54
fricklerand also make sure to react to issues coming up. we have ~ 10 bumps blocked because projects need to fix stuff18:54
gouthamrsry, openstackclient i think18:54
gmannyeah, if jobs are stable those can be added and those should not be taking more time than tempest-ful-py3 18:54
fricklerand I'm currently the only one that actually looks at those failures and pings projects, help could be really well spent there18:54
bauzasgouthamr: correct, it was a major OSC change that broke our world, so we needed to revert to old OSC version 18:55
gmannbauzas: ah, i remember now. 18:55
gmannwe talked about adding greande job there18:55
bauzasso I wonder why we didn't captured that in CI18:55
fricklerbauzas: because nova doesn't run an sdk/osc-tips job?18:55
gouthamr~~ time check; we've 5 mins ~~18:55
gtemabauzas - because like in the beginning of the meeting: project may run well with updated dep, but that doesn't mean that nobody else is using it in a totally different way18:56
bauzasanyway, I agree with the timecheck, we shouldn't solve that problem now18:56
bauzasI'll just take attention 18:56
gmannI was adding more cross service job on sdks but that is not green yet18:57
gouthamr+1 it's in the TC tracker 18:57
gouthamrbut, lets move the "TC Tracker" topic to next week; i do want to find owners for some PTG takeaways too.. we've three minutes to head into 18:57
gouthamr#topic Open Discussion18:57
gouthamra few new changes have showed up here:18:58
gmannfrickler: bauzas this one18:58
gmann#link https://review.opendev.org/c/openstack/python-openstackclient/+/93185818:58
gouthamr#link https://review.opendev.org/q/project:openstack/governance+status:open (Open Governance changes)18:58
gouthamrcan i have some eyes on https://review.opendev.org/c/openstack/governance/+/931254 18:58
gouthamrthis moves a lot of the eventlet removal content out of the goal doc and into a wiki because (a) it is background information (b) alternatives will continue to be flushed out, this wasn't meant to be prescriptive of specific directions 19:00
bauzasI wanted to review the eventlet goal but had no time yet :(19:00
gouthamrack; please do take a look soon .. 19:00
gouthamrthat said, we've reached the hour.. 19:00
gouthamrthank you all for attending19:00
gouthamrwe can continue discussing things here after i endmeeting...19:00
gouthamrbut see you here next week if you have to run.. 19:01
gouthamr#endmeeting19:01
opendevmeetMeeting ended Tue Nov 19 19:01:10 2024 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)19:01
opendevmeetMinutes:        https://meetings.opendev.org/meetings/tc/2024/tc.2024-11-19-18.01.html19:01
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/tc/2024/tc.2024-11-19-18.01.txt19:01
opendevmeetLog:            https://meetings.opendev.org/meetings/tc/2024/tc.2024-11-19-18.01.log.html19:01

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!