| opendevreview | OpenStack Proposal Bot proposed openstack/project-config master: Normalize projects.yaml https://review.opendev.org/c/openstack/project-config/+/952006 | 02:26 |
|---|---|---|
| opendevreview | tianyutong proposed openstack/project-config master: Allow branch creation for heterogeneous-distributed-training-framework https://review.opendev.org/c/openstack/project-config/+/952539 | 02:33 |
| *** elodilles_ooo is now known as elodilles | 06:08 | |
| opendevreview | tianyutong proposed openstack/project-config master: Enable release team to create branches and signed tags https://review.opendev.org/c/openstack/project-config/+/952547 | 06:18 |
| opendevreview | Michal Nasiadka proposed openstack/diskimage-builder master: Add Rocky Linux 10 support to rocky-container element https://review.opendev.org/c/openstack/diskimage-builder/+/952548 | 06:25 |
| opendevreview | Michal Nasiadka proposed openstack/diskimage-builder master: Add Rocky Linux 10 support to rocky-container element https://review.opendev.org/c/openstack/diskimage-builder/+/952548 | 06:34 |
| tonyb | I didn't make any real progress today on the DIB+devstack work. I found a bunch of stuff I missed pulling over from nodepool so I'm further behind than I thought :/ | 06:45 |
| opendevreview | tianyutong proposed openstack/project-config master: Enable release team to create branches https://review.opendev.org/c/openstack/project-config/+/952554 | 06:47 |
| opendevreview | tianyutong proposed openstack/project-config master: Allow branch creation for heterogeneous-distributed-training-framework https://review.opendev.org/c/openstack/project-config/+/952555 | 07:14 |
| ianw | tonyb: thanks a ton for working on it. if it were easy, it would have happened already, so i really appreciate you digging into it to keep things alive | 07:22 |
| tonyb | all good. I "blame" corvus for upholding the "only one general purpose devstack role" line. :-P | 07:43 |
| *** darmach8 is now known as darmach | 11:42 | |
| opendevreview | Merged openstack/project-config master: Allow branch creation for heterogeneous-distributed-training-framework https://review.opendev.org/c/openstack/project-config/+/952539 | 12:50 |
| corvus | lol "blame" gladly accepted :) | 13:40 |
| opendevreview | James E. Blair proposed opendev/zuul-providers master: DNM: noop test change https://review.opendev.org/c/opendev/zuul-providers/+/952576 | 13:43 |
| corvus | https://zuul.opendev.org/t/opendev/nodeset-requests | 13:44 |
| corvus | oops, looks like we're not deleting fulfilled noop requests | 13:44 |
| corvus | good news is the new web ui for deleting nodeset requests works | 13:45 |
| corvus | okay, wrote changes that should address that. i'm not going to manually cleanup the rest because one of the changes i wrote should automatically clean them up. | 14:45 |
| fungi | gonna go grab lunch, back in a while | 14:50 |
| clarkb | `The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.` | 15:05 |
| clarkb | I don't understand | 15:05 |
| clarkb | corvus: fyi ^ I saw that in nodepool build failures but I think the real issue is that pip is scanning all of these ibm packages to try and find the most appropriate version and its looking for requirements.txt files in those tarballs and one of them does not have that file? https://zuul.opendev.org/t/zuul/build/28e0c4720fae47918f49c42cba4f94bf/log/job-output.txt#1041 | 15:08 |
| clarkb | corvus: that same error has broken the nodepool image builds. I think the solution here is to add that dep with a lower version bound | 15:09 |
| clarkb | I'll work on a change momentarily | 15:09 |
| clarkb | but also removing pkg_resources is going to break many package on pypi | 15:09 |
| clarkb | packages that we and others don't have control over but that we all rely on | 15:09 |
| clarkb | possibly even pbr too... I guess the PBR compatibility cleanups are becoming more and more important. Its just frustrating that we can't assume older packages (for some values of not that old) will continue to build in the near future | 15:10 |
| clarkb | remote: https://review.opendev.org/c/zuul/nodepool/+/952587 Give IBM cloud deps recent lower bounds | 15:19 |
| clarkb | I think this will hopefully unblock the dib tests for comparison to the new dib testing framework | 15:19 |
| corvus | those ibm packages have been a source of a number of issues relating to how many of them get scanned and how slow it is... maybe that will help in general. | 15:26 |
| corvus | i think zuul hase the same deps, want to try bumping them there too? | 15:26 |
| clarkb | corvus: can do | 15:28 |
| clarkb | remote: https://review.opendev.org/c/zuul/zuul/+/952592 Set lower bounds on IBM cloud deps | 15:31 |
| frickler | oh, nice, likely we'll have to cap setuptools for most stable branches, then | 15:31 |
| clarkb | frickler: ya I'm not sure what we can do about pkg_resources going away. That seems like a very major change | 15:31 |
| clarkb | I assume they want everyone using the new library which doesn't do everything pkg_resources does. import metadata or whatever it is called. Probably can update in many places but some may be problematic | 15:32 |
| frickler | "Some useful APIs are also provided by packaging" from https://setuptools.pypa.io/en/latest/pkg_resources.html. but yeah, python in general seems like a pretty unstable environment these days more than ever | 15:42 |
| *** dhill is now known as Guest18100 | 15:44 | |
| clarkb | tonyb: fwiw the dib testing update did successfully boot and test a ubuntu node | 15:50 |
| clarkb | tonyb: I think the framework there is sound we've just got a few improvements to make (like qcow2 -> raw and the stuff ianw has suggested etc) and then sort out the weird platform issues. It might be worthwhile to avoid alma linux for now and focus on centos and rocky on the rpm distro side of things since we have more exposure and experience with them | 15:52 |
| clarkb | I wonder if the existing testing for alma linux is using glean. In any case getting nodepool fixed should allow us to recheck and get some data we can compare to | 15:53 |
| clarkb | corvus: I think the node request leak cleanups are failing in ci | 15:57 |
| corvus | boo. i only ran the affected tests locally | 15:58 |
| clarkb | looks like tests.unit.test_gerrit.TestGerritCherryPickWeb.test_gerrit_cherry_pick_web_with_update_queue is what failed in one of them. | 15:58 |
| clarkb | I wonder if that is just an order problem independent of the changes. Looks like a mismatch of job run history maybe | 15:59 |
| corvus | passes locally | 15:59 |
| corvus | ooh there's a missing git add | 15:59 |
| corvus | (which explains the other failures) | 16:00 |
| corvus | fixed | 16:00 |
| clarkb | corvus: that failed in 81 (the parent too) but you updated 82. Not sure if we needed the fix in the parent? | 16:01 |
| corvus | the cherry pick test? that's unrelated | 16:01 |
| clarkb | ya. Got it | 16:02 |
| clarkb | wasn't sure if you meant the missing git add affected that somehow. I understand now. The child was broken. The parent hit something in the random number generator of the test suite | 16:02 |
| corvus | exactly | 16:06 |
| fungi | frickler: pinning setuptools versions may not even be possible in relevant ways unless those branches are already using pyproject.toml to define their build-system.requires lists | 16:12 |
| clarkb | nodepool image builds now fail on gcc: internal compiler error: Segmentation fault signal terminated program cc1 during yappi compilation | 16:58 |
| clarkb | I'm going to look at that now I guess. Will probably need the fix for thta and the ibm deps fix in one change | 16:59 |
| clarkb | ok this is an arm only problem | 17:00 |
| clarkb | I don't see any upstream bug for this in yappi. Also this must be fairly recent possibly due to updates in debian? I want to say bookworm got a big update a month or two ago | 17:03 |
| clarkb | Debian 12.11 was released on May 17th, 2025 | 17:09 |
| clarkb | I suspect this is when it broke but don't know for sure | 17:09 |
| fungi | do we have any rough window for when the build broke? | 17:42 |
| clarkb | I haven't looked yet been more focused on just fixing it | 17:45 |
| clarkb | https://zuul.opendev.org/t/zuul/builds?job_name=nodepool-build-image&skip=0 | 17:47 |
| fungi | just wondering if i can match it up to debian updates. if we're upgrading debian packages from the updates channel rather than just sticking with the point release versions, then the point release itself would be a non-event since it's just copying packages from updates to main stable | 17:47 |
| clarkb | this seems to indicate that the problems are not consistent. Its possible that timing issues with running emulated arm triggers the gcc segfault inconsistently | 17:47 |
| clarkb | I think the ibm issue was not platform specific though and may be related to more recent package releases for that family of packages | 17:48 |
| fungi | yeah, looks like it passed once today | 17:48 |
| fungi | or was that the fix you're working on? | 17:49 |
| clarkb | fungi: yes the pass included the ibm fixup but not the yappi fix | 17:49 |
| clarkb | fungi: so yappi segfault is inconsistent. I think the ibm problem was consistent | 17:49 |
| fungi | got it | 17:49 |
| clarkb | ibm problem is very recent (last couple of days). yappi compile error may be older | 17:49 |
| fungi | looks like it was mostly passing yesterday and earlier in the week | 17:50 |
| clarkb | there were failures with dib builds too which I think is a different job we can corss check with | 17:50 |
| fungi | yeah, the sample size here is small with lots of big holes | 17:50 |
| fungi | hard to draw much of a conclusion | 17:51 |
| clarkb | https://zuul.opendev.org/t/openstack/builds?job_name=nodepool-build-image-siblings&skip=0 | 17:51 |
| clarkb | I suspect sometime yesterday it broke | 17:51 |
| clarkb | latest round of tests with the last patchset are looking good if anyone wants to review that change now (952587) | 18:28 |
| fungi | lgtm, thanks! | 18:34 |
| opendevreview | Clark Boylan proposed openstack/diskimage-builder master: Add new openstack/devstack based functional testing https://review.opendev.org/c/openstack/diskimage-builder/+/949942 | 18:51 |
| clarkb | I added a deepnds on to the nodepool fix on ^ so that we can get data to compare to | 18:51 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!