Monday, 2025-03-24

karolinku[m]Hello, I made glean code working on C10, can I ask for a review? ^07:20
*** mrunge_ is now known as mrunge07:30
*** ralonsoh_ is now known as ralonsoh07:55
fungikarolinku[m]: i recommend asking in #openstack-dib too13:12
clarkbkarolinku[m]: there are a couple of reviews on the change now. Mine is a quick one and I haven't gotten too in depth yet. But I think adding the unittest cases if we can is important15:19
clarkband now it is time for my monday morning reboot for updates15:19
corvusclarkb: zuul-launcher is happy with the new config now; if you want to throw a +2 on https://review.opendev.org/945056 i can merge that stack15:34
clarkbdone15:35
opendevreviewMerged opendev/zuul-providers master: Add rackspace classic to zuul-launcher  https://review.opendev.org/c/opendev/zuul-providers/+/94505515:43
opendevreviewMerged opendev/zuul-providers master: Add Openmetal provider  https://review.opendev.org/c/opendev/zuul-providers/+/94505615:43
opendevreviewMerged opendev/zuul-providers master: Add OVH provider  https://review.opendev.org/c/opendev/zuul-providers/+/94505715:43
clarkbthe next set of servers in my sights to move to noble are the nodepool launchers. I think appraoch I'll use is to add nl05, nl06, nl07, and nl08 config files in project-config that are direct copies of the nb01-4 files. Then boot new servers and add them to the inventory in some sequence. When an inventory addition change lands I'll shutdown the corresponding old nl01-4 launcher15:45
clarkbprocess and put its host in the emergency file to avoid conflicts15:45
clarkbdoes that approach make sense or is there an easier way? I thought about having both nl01 and nl05 running when 01 having positive max-servers values and nl05 having max-servers set to 0 then flip the values over to cut over. But I'm not sure how safe that is15:45
clarkb(I think it is probably safe but not compeltely confident whereas I'm pretty confident in the propsal I'm making)15:46
fungithat seems fine, with the state in zk any cleanup needed should be handled normally by the new replacements15:48
fungiwe "lose" a bit of git history for changes to the original filenames...15:49
fungianother option might be to put the old servers in emergency disable, then git rename the files in a change?15:49
funginot sure if that helps more than the additional complexity it requires though15:49
clarkbthe main downside to it is probably needing more changes since I would want to go them one by one in that setup15:53
clarkbwhereas  Ican add the new configs in one shot, flip over one launcher to check it is all happy then do the other three in one go maybe?15:53
fungiyeah, i'm cool with whatever's fastest and/or least effort required15:54
clarkbok Ill start on this shortly15:55
opendevreviewJeremy Stanley proposed opendev/engagement master: Rewrite maintainers.py functionality  https://review.opendev.org/c/opendev/engagement/+/94526215:56
opendevreviewJeremy Stanley proposed opendev/engagement master: Rewrite maintainers.py functionality  https://review.opendev.org/c/opendev/engagement/+/94526216:10
clarkbI'm booting nl05 now. Will proceed with 06-08 once I'm confident in the launch command I'm using (basically waiting for networking to be up)16:21
opendevreviewClark Boylan proposed openstack/project-config master: Add config files for nl05, nl06, nl07 and nl08  https://review.opendev.org/c/openstack/project-config/+/94535916:33
opendevreviewClark Boylan proposed opendev/zone-opendev.org master: Add nl05, nl06, nl07, and nl08 to DNS  https://review.opendev.org/c/opendev/zone-opendev.org/+/94536416:37
opendevreviewClark Boylan proposed opendev/system-config master: Add new nl06 Noble nodepool launcher  https://review.opendev.org/c/opendev/system-config/+/94536916:43
clarkbok I think thats the beginning of that process. All four nodes are up and that updates DNS to match. Then project-config to add config files for all four and finally the deployment of nl06 to replace nl0216:44
clarkbinfra-root friendly reminder that I drafted some content for the openinfra newsletter here: https://etherpad.opendev.org/p/opendev_newsletter if you have a moment to check it for accuracy and also weigh in on whether or not we want to put our hat in the ai crawlers are causing problems situation that would be great16:47
clarkb945369 passes testing so should be safe to run the launcher on noble17:27
clarkbI'm around all day today and can do the manual steps if we want to review those changes and approve them17:27
opendevreviewMerged opendev/zone-opendev.org master: Add nl05, nl06, nl07, and nl08 to DNS  https://review.opendev.org/c/opendev/zone-opendev.org/+/94536417:37
fungii guess it was decided that replacing nl02 first was going to be less disruptive than nl01?17:37
clarkbfungi: yes nl02 only serves openmetal so the impact is minimal if something goes wrong17:44
clarkbnl01 serves all of rax and rax flex and is the largest impact if something goes wrong17:44
opendevreviewMerged openstack/project-config master: Add config files for nl05, nl06, nl07 and nl08  https://review.opendev.org/c/openstack/project-config/+/94535918:03
clarkb945359 has "deployed"18:15
clarkbany objection to me approving https://review.opendev.org/c/opendev/system-config/+/945369 now?18:16
tonybclarkb: Sounds good to me.  I added a comment but go for it18:20
clarkbtonyb: 945359 does a deployment test of a nodepool launcher on a noble node within opendev's system-configuration. The change you linked to is about running nodepool unittesting on noble right?18:21
clarkbin the case of my change we're stilling running nodepool launcher itself on python3.11 within a bookworm container. Only the hosting system changes18:21
tonybooooo nm I understand.  In system-config we already know how to do the correct "{podman,docker} compose" install, so the parent change isn't needed.18:23
clarkbya18:23
clarkbI've approved it18:23
tonybbonza18:23
clarkband if that deployment looks good I'll do nl06 and nl08 next (so that we get all of our capacity but rax swithced over. Then do rax last as it is the most capacity so biggest impact18:24
clarkbsorry nl06 is now. nl07 and nl08 would be next18:24
mnaserany noise right now because of https://github.com/pypa/setuptools/pull/4870?18:33
mnaserone of our devstack plugins installs a dependency that is broken by this and i'm wondering if that's been hit by other projects :\18:33
clarkbopenstack packaging switched over to using the normalized form when the release jobs switched to using the build tool instead of setup.py commands18:34
clarkbbut no I haven't heard of anyone being broken by it. Sime people wondered why the names changed18:34
mnaserhttps://github.com/pypa/setuptools/pull/4911 seems like they are reverting it, even requests doesn't even build anymore :<18:35
clarkboh wait sorry this is setup.cfg stuff not the tarball names18:35
mnaseryeah, setup.cfg not acceepting - instead of _18:35
clarkbsimilar in that -'s become _'s18:35
clarkbbut no I hadn't heard anything yet18:35
clarkbpossibly because newer setuptools only works on very new python and older python is using old setuptools?18:35
clarkbfrickler: fungi: the openstack release team may want to double check that this won't explode the release though18:35
mnaserhmm, our devstack blows up during a pip install, but it's because of a specific package at least18:36
mnaserhttps://zuul.oss.vexxhost.dev/build/869d15ac6001460a87b143a49b14d2c118:36
mnaseri suspect/wonder if it's because there is no pre-existing packaged file for that python version18:37
clarkbI suspect you need the new neough setuptools in the first place. If using build's isolated envs it may update to latest there18:37
fungilooks like you need to be installing with setuptools 78.0.0 (or 78.0.1) for this to be in effect18:37
clarkbotherwise in our ci jobs we may be lagging and waiting for new image builds? I'm not sure18:37
mnaserwe're using plain devstack and this is written as a devstack plugin and afaik i can double check but it uses the devstack built in pip_install function18:37
clarkbya the fundamental issue with changes like that is setuptools can't stop accepting old stuff until you decide the old stuff isn't supported aynmore period18:38
mnaser+ inc/python:setup_devstack_virtualenv:44  :   pip_install -U pip 'setuptools[core]'18:38
clarkband that claerly isn't hte case but this is also something the pypa ecosystem has really struggled with understanding18:38
fungisetuptools 78.0.0 doesn't seem to have been uploaded to pypi (not even yanked), 78.0.1 first appeared 4 hours ago18:38
mnaseryeah that lines up with when things started going back18:38
mnaseras part of devstack: Successfully installed autocommand-2.2.2 backports.tarfile-1.2.0 jaraco.context-6.0.1 jaraco.functools-4.1.0 jaraco.text-4.0.0 more_itertools-10.6.0 packaging-24.2 pip-25.0.1 platformdirs-4.3.6 setuptools-78.0.1 tomli-2.2.1 wheel-0.45.118:39
clarkbbecause you should be able to install an old package with new setuptools... ugh18:39
fungi14:39:50 utc was the hweel upload time18:39
fungiyeah, setuptools choking on setup.cfg should only happen at installation if there's no wheel for something18:40
clarkbhttps://zuul.opendev.org/t/openstack/build/1a9dafde050448d0808c97e472e001f8/log/job-output.txt#27046-27100 this job seems to have upated setuptools but passed18:40
clarkbfungi: oh!18:40
mnaseryeah so in that case this is only because in our case we're installing something that needs to build a wheel (haproxyadmin)18:40
clarkbthat is probably why that job above passed tehn it was able to find wheels for everything18:40
fungiif there's a wheel, it just gets unpacked into the environment, setuptools doesn't have any involvement18:41
mnaserbut i would GUESS that maybe the wheel builder tool might be broken right now18:41
clarkbbut still this seems like a change that isn't thought through if you can't suddenly build a bunch of pacakges over an - vs _ and latest setuptools18:41
clarkbmnaser: ya that is probably the case18:41
clarkb(though we never set it up for python3.12 because the number of packages without wheels was tiny and we wanted to see if we could stop and so far seems to have been fine)18:42
mnaserblargh, fair enough, so i guess i'm not sure how i can workaround it for now, i'll see what i can do to figure out to get devstack to pin .. or wait out the revert18:42
fungithe python packaging ecosystem take on this is that everyone should be using pyproject.toml now, and can therefore decide for themselvs what version of setuptools should be used to build their package. the reality is that plenty of older setuptools-based packages aren't using pyproject.toml to specify some particular maximum version of setuptools as a build-backend18:43
clarkbI think your options are to wait for revert (if that happens), pin setuptools back, find another way to build a wheel for what you need, or use a different dependency18:43
clarkbfungi: right18:43
mnaserfungi: i think the openstack ecosystem is probably part of that group of people that should get that done =P18:43
fungimnaser: get what done, specifically?18:43
clarkbsounds like uv and requests are lagging too18:43
mnasersetup.cfg => pyproject.toml18:43
clarkbso its not like a small corner of the python world is the propblem its the major stuff18:43
fungimnaser: wouldn't help in this case unless they also start pinning maximum versions of setuptools18:44
fungipre-declaring that they won't support future versions18:44
mnaseryeah, and that brings in other kidns of fun anyways18:44
clarkbswitching to _'s is a straightforward fix but doesn't help for existing packages18:45
fungii mean, technically it could have sidestepped this problem if they moved settings out of setup.cfg into a project table in pyproject.toml, but only because this was a regression in setup.cfg file parsing18:45
clarkbI'm not sure I understand that18:46
clarkbare you saying that moving into pyproject.toml forces you to normalize the keys so would fix it in a round about way?18:46
fungii'm saying that not having content in setup.cfg sidesteps regressions with setup.cfg content parsing18:48
fungican't break what's not used18:49
clarkbright so to fix this within our own projcets we can update the setup.cfg keys or move into pyproject.toml. Both are options one being a bit simpler than the other. But doing so doesn't fix any of the existing sdists that are published which is where I think the real problem with this change is18:49
clarkbsetuptools shouldn't just decide that a large portion of sdists built valid yesterday should stop working today18:50
clarkbbut I know better than to waste my time arguing for them to change anything and instead our efforts are likely best placed in updating things on our side18:50
fungibut the bigger problem is that openstack has, in some cases, dependencies that haven't seen new releases since years before pyproject.toml was a twinkle in anyone's eye18:50
clarkbyup18:50
mnaserfwiw we've built a self-hosted instance of renovatebot using zuul since it has a native gerrit integration18:51
fungiso breaking the ability to install those essentially means finding replacements that ascribe to newer packaging paradigms, even if they're working fine otherwise18:51
mnasera periodic+post pipeline and ability to keep things up to date by autogenerated gerrit changes18:51
mnaserkinda helps with keeping you up to date on things..18:51
clarkbmnaser: ya we'ev got zuul jobs that do similar for things like requirements and translations18:52
fungii don't see the connection18:52
mnaserclarkb: it's inspired mostly by that18:52
clarkbfungi: I think fungi is suggesting that openstack could run renovatebot to fix these things and propose chagnes autoamtically18:52
clarkber mnaser is suggesting18:53
mnaserfungi: humans tend to forget to bump dependencies, but i think you were thinking more about if the libraries are abandoned vs consistently updating deps18:53
clarkbbut even then as you mention the old deps won't update18:53
fungiif a simple library hasn't needed a new release in 10 years, a bot that syncs you to the latest still-10-year-old version of that dependency doesn't really help18:53
fungiupdating to newer available versions of dependencies makes sense, yes, but is basically already a solved problem18:54
clarkbright the fundamental problem here isn't in openstack it is in setuptools/pypa18:54
clarkbthey just made a decision to mark hundreds/thousands/millions? of previously valid sdists as invalid18:55
fungithough also not really relevant to several-year-old stable branches with frozen depencdencies that we still need to test changes for18:55
fungiif it were easier to globally pin those to a contemporary setuptools version, that would be less of a concern18:55
clarkband they marked them invalid for a trivial reason18:56
clarkbbasically we can't install our dependencies anymore because someone decided a _ looks better than a - in metadata18:56
fungiwe can't update dependencies on a unmaintained/yoga branch of nova in order to switch to newer versions of packages that keep working with new setuptools18:56
opendevreviewMerged opendev/system-config master: Add new nl06 Noble nodepool launcher  https://review.opendev.org/c/opendev/system-config/+/94536918:57
fungithough, again, having wheels available mostly shields things from this problem18:57
mnaserif you built them before this :P18:57
fungiright, which for a frozen branch of a project you will have done18:58
fungiand for non-frozen branches at least updating dependencies is still possible18:58
fungion a related note, pbr got a mention in https://github.com/pypa/setuptools/issues/491319:02
fungii felt obliged to correct the historical record there though19:05
mordredalso, for the record, pbr has never been called oslo.packaging19:06
mordredwait19:06
mordredwhat? was it? crazy19:07
fungii've dissolved most of those brain cells with strong liquor too19:07
fungibut for example https://github.com/markmc/oslo.packaging still exists19:07
mordredwow. I don't remember that at all. I thought we started that as a standalone d2to1 fork19:08
mordredoh right - we used to copy things around 19:10
clarkbnl06 is getting close to being deployed so I have put nl02.opendev.org in the emergency file and shutdown the container on nl0219:19
fungithe setuptools 78 situation has spilled over to discuss.python.org as well: https://discuss.python.org/t/pinning-build-dependencies/836319:23
clarkbis build constraints supported by build? their docs don't say it is. I'm not even sure pip documents this either19:25
fricklerI was out today but I saw a discussion in the kolla channel which seems to be affected, too https://review.opendev.org/c/openstack/kolla-ansible/+/94534219:25
clarkbthe launcher is running on nl06 now19:26
clarkbhttps://grafana.opendev.org/d/75a5d1cf58/nodepool3a-openmetal?orgId=1&from=now-30m&to=now&timezone=utc&var-region=$__all initial signs lgtm but waiting for state changes on existing nodes and/or newly built nodes19:29
clarkband it is lunch time. So I'll catch up on that after eating something and if it looks good get the next round of rotations pushed up19:29
fungilooks like the revert merged: https://github.com/pypa/setuptools/pull/491119:29
clarkb2025-03-24 19:26:18,627 this is when the new launcher started19:31
clarkblooks like shortly after the launcher was able to switch ready nodes to in use nodes so that is an additional good sign19:32
clarkboh neat it just built and made ready then handed over some nodes19:33
clarkbso ya after lunch I'll push up a change to switch nl07 and nl08. Then we can do nl01 last as it is our greatest capacity launcher19:33
fungisetuptools seems to have tagged a 78.0.2 on github including the revert, so i expect it will find its way to pypi in short order (but isn't there quite yet)19:34
opendevreviewClark Boylan proposed opendev/system-config master: Add nl07 and nl08 to inventory  https://review.opendev.org/c/opendev/system-config/+/94539519:53
opendevreviewClark Boylan proposed opendev/system-config master: Add new nl05 to inventory  https://review.opendev.org/c/opendev/system-config/+/94539619:57
opendevreviewClark Boylan proposed opendev/system-config master: Cleanup nl01, nl02, nl03, nl04  https://review.opendev.org/c/opendev/system-config/+/94539719:57
opendevreviewClark Boylan proposed opendev/system-config master: Cleanup nl01, nl02, nl03, nl04  https://review.opendev.org/c/opendev/system-config/+/94539719:58
opendevreviewClark Boylan proposed openstack/project-config master: Cleanup configs for nl01, nl02, nl03, and nl04  https://review.opendev.org/c/openstack/project-config/+/94539819:59
mnaseraaand https://pypi.org/project/setuptools/78.0.2/20:00
opendevreviewClark Boylan proposed opendev/zone-opendev.org master: Cleanup nl01, nl02, nl03, and nl04 DNS records  https://review.opendev.org/c/opendev/zone-opendev.org/+/94539920:00
clarkbinfra-root if nl06 looks good to you then I think we are ready to proceedwith 945395. I can be around to put nl03 and nl04 in the emergency file and shutdown their containers when the new servers are about to deploy20:01
clarkbthe changes above should be a fairly compelte accounting of the step by step process if you want to review the whole thing I can approve them in order as I'm able to monitor20:02
opendevreviewJeremy Stanley proposed opendev/bindep master: Comment reminding to replace extras with depgroups  https://review.opendev.org/c/opendev/bindep/+/94540220:05
opendevreviewJeremy Stanley proposed opendev/bindep master: Replace extras with dependency groups  https://review.opendev.org/c/opendev/bindep/+/94540320:05
clarkbwhile I wait for reviews and ci results I'm going to take a short walk. I won't be long or far so is safe to approve 945395 if that looks good20:17
fungidone20:26
Clark[m]Thanks20:32
opendevreviewMerged opendev/system-config master: Add nl07 and nl08 to inventory  https://review.opendev.org/c/opendev/system-config/+/94539520:50
clarkbI'm back and will add hosts to emergency file and shtudown services when closer to reaching that point of the deployment20:53
clarkbthere is a new gitea release if anyone else wants to push up a change for that. I'll probably get to it later this evening20:58
clarkbwhen the LE jobs completes is when I'll shutdown services on nl03 and nl0421:03
fungihttps://zuul.opendev.org/t/opendev/build/1511c67a477741b592027fd53133a44a is giving me the impression that our centos 9 mirroring is in an inconsistent state21:07
clarkbya thats the indicator of that since they use hashes in the file names21:07
clarkbso some index is pointing at hashed files that don't exist yet21:07
clarkbusually the fault is upstream of us but may be worth double checking we haven't run out of disk or similar21:08
fungihttps://grafana.opendev.org/d/9871b26303/afs says it updated 3 hours ago21:08
fungiso it's likely not stuck21:08
clarkbLE is done I've shutdown services on nl03 and nl04 and they are in the emergencyfile 21:09
clarkbfungi: should I drop the bindep topic because the other changes merged or keep it and add these new changes?21:16
clarkbfrom the meeting agenda I mean21:16
fungithe new changes aren't particularly relevant yet21:17
fungione is just todo comments as a placeholder until the other can merge, which won't be possible until pip 25.1 is released21:17
clarkbok I can clean up the item then21:18
fungiagreed21:18
fungithough for the record, i've duplicated bindep's pyproject.toml usage in https://review.opendev.org/945151 for the engagement tooling, since i needed to update some bits there anyway before adding a new entrypoint21:19
fungiseems to be working fine21:19
clarkbok my first pass on agenda edits is in21:20
clarkbnodepool deployment reports success21:21
clarkbthe affected providers are ovh, vexxhost, and osuosl21:21
clarkbit looks good on initial inspection21:22
clarkbok ovh is looking good. osuosl and vexxhost are both idle enough that we're not seeing any movement there21:40
clarkbWe can probably proceed with nl01/nl05 now though since ovh and openmetal both seem ahppy21:40
fungiapproved21:43
fungialso, on the earlier discussion, https://github.com/pypa/setuptools/issues/4913 is worth keeping an eye on for anyone who cares about old pbr-originated setup.cfg files continuing to be usable with newer setuptools versions21:45
clarkbI kinda watn to be a bit more forceful about pointing out how they decdied to use setup.cfg in a non compatible manner21:46
clarkbwe used the system first (as did other tools as you pointed out) then they decided to change what it meant21:47
clarkbif they aer going to suddenly decide only they get to define what is in those files they should've picked another file name or something21:47
clarkbin a way its setuptools extending what we did not the other way around21:49
clarkband I think that is an important point to make when talking about backward compatibility and not breaking your ecosystem21:49
clarkbbut I also don't have the patience or sanity to go through it with pypa21:49
opendevreviewClark Boylan proposed opendev/system-config master: Update to Gitea 1.23.6  https://review.opendev.org/c/opendev/system-config/+/94541422:07
clarkbthere is the gitea update22:07
opendevreviewMerged opendev/system-config master: Add new nl05 to inventory  https://review.opendev.org/c/opendev/system-config/+/94539622:15
clarkbI'll do what I did previously and shutdown old server services when deployment gets to lE for ^22:16
clarkbI just stopped the launcher on nl0122:33
opendevreviewJeremy Stanley proposed opendev/bindep master: Switch from license classifier to SPDX expression  https://review.opendev.org/c/opendev/bindep/+/94541622:38
clarkbnl05 should be running rax provider requests now22:43
clarkbI see some movement on rax classic graphs but nothing on flex yet22:48
clarkbmaybe let that run for a little bit longer and if nothing pops up we can land https://review.opendev.org/c/opendev/system-config/+/945397. Then I can delete the servers tomorrow after landing the two related cleanup changes22:49
clarkbin the meantime last call on meeting agenda items22:49
clarkbraxflex dfw3 just did some things22:54
clarkbso ya unless there are any objections I'll approve the cleanup change momentarily22:54
clarkband approved22:58
corvusclarkb: https://review.opendev.org/944878 is a change to the docker image mirror role to get all arches using skopeo.  that makes sense to me and i'm happy with the testing.  i think it's worth a look from you if you have a min.22:59
clarkbcorvus: ack23:00
corvusi think that become important when we mirrored the docker builder image; i think up to that point we just didn't expect any non x86 images23:02
clarkbcorvus: I +2'd but left a comment suggestion about improving testing and why I think that may be useful so didn't approve23:09
clarkbI'm going to send the meeting aenda now23:09
corvusclarkb: thx; replied with a clarification, but i'm not in a rush if we want to add more testing.23:17
clarkbcorvus: ya I figured skopeo was smart here and I was over thinking it. I think you can approve it given our undersatnding seems to be in agreement23:18
clarkbI just wanted to make sure we double checked these assumptions before approving23:18
opendevreviewMerged opendev/system-config master: Cleanup nl01, nl02, nl03, nl04  https://review.opendev.org/c/opendev/system-config/+/94539723:28

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!