| stephenfin | Did something changes with images used for openstack-tox-py* jobs yesterday? We've started seeing the following error `No package matching 'libffi-dev' is available` in SDK jobs | 10:24 |
|---|---|---|
| stephenfin | I don't see any changes in openstack/openstack-zuul-jobs | 10:24 |
| *** agalica__ is now known as agalica | 11:36 | |
| fungi | stephenfin: i released a new dib yesterday, could it be images built with it are missing packages previously included? | 12:12 |
| stephenfin | Possibly? I'm not sure how to fix that. bookworm should have a libffi-dev package available so idk why it's giving me that error | 12:15 |
| fungi | stephenfin: if you can get me a link to an example build result i can try to dig into it in a few minutes | 12:43 |
| stephenfin | fungi: it's in the py311 job https://zuul.opendev.org/t/openstack/builds?job_name=openstack-tox-py311&project=openstack/openstacksdk however, I see a new passing result | 12:44 |
| stephenfin | so maybe it was a transient infra issue? | 12:45 |
| fungi | if it's hitting our package mirrors, it's unlikely to manifest that way unless ubuntu removed the package from their distribution temporarily, but i'll look to see if that was maybe a symptom of an earlier message in the log or something | 12:46 |
| fungi | stephenfin: was it just the tests for 942627 where you saw that error recurring? | 12:56 |
| fungi | looks like the failing builds ran on debian-bookworm nodes | 12:58 |
| stephenfin | afaict, yes. But I didn't change anything in e.g. bindep.txt in that change, and py313 passed? 😕 | 12:58 |
| fungi | in theory it should have found https://packages.debian.org/bookworm/libffi-dev | 12:59 |
| stephenfin | right? very strange | 13:00 |
| fungi | it's using ansible's "package" module which unfortunately is quite inscrutable | 13:01 |
| fungi | it ran bindep on src/opendev.org/openstack/openstacksdk/bindep.txt and libffi-dev was the only missing package on the system, so it told ansible's package module to install that | 13:02 |
| fungi | stephenfin: okay, there was this which happened earlier in the build i'm looking at: https://zuul.opendev.org/t/openstack/build/92f204d6b66e44df941d2b8aa9895b87/console#0/3/22/debian-bookworm | 13:04 |
| fungi | `W: Failed to fetch https://mirror.gra1.ovh.opendev.org/debian/dists/bookworm/InRelease Could not connect to mirror.gra1.ovh.opendev.org:443 (2001:41d0:302:1000::786). - connect (113: No route to host) Could not connect to mirror.gra1.ovh.opendev.org:443 (217.182.142.170), connection timed out` | 13:04 |
| stephenfin | aha, good spot. Dodgy node so | 13:05 |
| fungi | so seems like the job node had trouble reading from the mirror | 13:05 |
| * stephenfin has never used that view in Zuul before | 13:05 | |
| fungi | oh, i just find it easier than digging through a linear log file | 13:05 |
| fungi | i can zoom in on the specific ansible tasks i suspect could be involved | 13:06 |
| fungi | but anyway, this was the third in a sequence of three pre-run phase failures eventually ending in retry_limit, chances all three of them would get scheduled to the same region are low, so i'll check that first | 13:07 |
| fungi | yeah, all three happened to run in ovh's gra1 region | 13:08 |
| fungi | i'll check on that server | 13:08 |
| fungi | seems to be responding fine for me at the moment, uptime is nearly 2 months and no sign of any recent load average spikes, memory pressure, exhausted disk... | 13:10 |
| fungi | though looks like these ran a little before utc midnight yesterday so it's been a while | 13:11 |
| stephenfin | yeah, that view is super handy. I'll remember that for the future | 13:11 |
| fungi | in dmesg i see a few `afs: Waiting for busy volume 536871042 () in cell openstack.org` yesterday at 23:20:07 and again at 23:39:22 utc which could be related, since the packages in question are hosted in afs | 13:12 |
| fungi | also volume 536871042 | 13:12 |
| fungi | i'll have to check which names those map to | 13:12 |
| fungi | nope, the mirror.debian volume is 536870983 | 13:14 |
| fungi | er, i mis-pasted the same id twice there, the mirror.debian volume was actually one of the two it complained about | 13:16 |
| fungi | `[Tue Sep 8 23:39:22 2026] afs: Waiting for busy volume 536870983 () in cell openstack.org` | 13:16 |
| fungi | okay, so it seems like there may have been some sort of communication issue between the mirror in ovh gra1 and the openafs fileservers in rackspace | 13:17 |
| fungi | which directly impacted reads from the debian mirror volume at the time those builds were running | 13:18 |
| fungi | the other affected volume at that time (536871042) was mirror.ubuntu-ports which would only have affected arm64 builds which don't run in ovh anyway | 13:19 |
| fungi | stephenfin: so that's my best guess, looking at https://zuul.opendev.org/t/openstack/builds?result=RETRY_LIMIT&skip=0&limit=100 there's a cluster of retry_limit results between 2026-09-08 20:02:25 and 2026-09-09 03:54:50 | 13:21 |
| stephenfin | That's as good an explanation as I could have. In any case, it sounds like the issue has resolved itself, at least for now | 13:23 |
| stephenfin | I'll know to check for apt issues next time too. I saw missing package and just assumed we'd changed the base image in the job | 13:23 |
| fungi | ovh cloud doesn't indicate any incidents that could have caused this, but i'm checking their network incidents list which is separate | 13:24 |
| fungi | same for past network incidents in gra1 | 13:25 |
| fungi | yeah, looking through similar retry_limit results in other projects around that same time, they also seem to have happened in gra1 so at least at this point i only see evidence of impact in that one provider region | 13:27 |
| fungi | and whatever was going on probably recovered about 10 hours ago now | 13:27 |
| stephenfin | fungi: entirely unrelated, Uggla is +1 on https://review.opendev.org/c/openstack/project-config/+/988094/ now. Anything else needed to progress with that? | 15:17 |
| fungi | stephenfin: once it merges, i'll need to add someone as an initial member of the new nova-reviewers group so they can start configuring and populating it. normally that would be the tact sig liaison(s) on the team, or the ptl if there is no indicated liaison | 15:22 |
| fungi | s/merges/deploys/ (that'll take a few extra minutes to complete) | 15:23 |
| fungi | so i'll add uggla once zuul reports back on the change saying it's done | 15:25 |
| stephenfin | hmm, so Uggla is PTL but not currently in nova-core | 15:29 |
| stephenfin | and we've no tact sig liason(s) | 15:29 |
| stephenfin | could we add nova-core as a proxy? | 15:30 |
| fungi | but from a strict openstack governance perspective, it's uggla's responsibility to decide who is and isn't a nova core reviewer | 15:30 |
| fungi | i'll let uggla do that | 15:30 |
| fungi | alternatively, i can add the tc and they can choose who to add (and then remove themselves once done) | 15:31 |
| fungi | i don't want to do an end-run around governance | 15:32 |
| fungi | uggla can always remove themselves from the group once they decide who they want in it | 15:33 |
| stephenfin | understood | 15:40 |
| stephenfin | I think adding the TC would be better for now in that case | 15:40 |
| stephenfin | I suspect Uggla will be added to this group before long, but the proposal I've given strictly focuses on SMEs from other services. I don't want to give any reason for people to walk back support for said proposal | 15:41 |
| stephenfin | the proposal being https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/message/34R3AFYLUO6W4HN6HCNMYSENVCICXDAR/ | 15:42 |
| fungi | i mean, from a governance perspective uggla can just declare themselves a core reviewer for nova any time they like, presumably the only thing preventing that is a desire not to alienate the current nova core reviewers. the nova contributors elected (even if tacitly by producing no alternative candidate) uggla to lead the nova team and make or delegate those decisions | 15:44 |
| fungi | that's the entire point of the ptl election process, after all | 15:46 |
| stephenfin | yeah, I was wondering if we'd somehow done an end-run around governance with the current setup | 15:46 |
| JayF | I mean, there's soft power and hard power | 15:46 |
| JayF | The whole situation with Nova PTL being unwilling to self-core themselves is someone trading off the use of their hard power to avoid the loss of some of the soft power | 15:47 |
| stephenfin | king vs. parliament | 15:47 |
| stephenfin | *king/queen / sovereign | 15:47 |
| opendevreview | Merged openstack/project-config master: Add nova-reviewers to openstack/nova ACL https://review.opendev.org/c/openstack/project-config/+/988094 | 15:48 |
| JayF | the only thing that goes here that makes it a little muddy is when the PTL + existing cores share an employer, which makes you wonder if there's a hidden reason hard power isn't being used -- but we all have to assume good intentions and motives for OSS community to work, so I'm discarding that as am likely cause -- but it is something to have awareness of something that it does make perceptable | 15:48 |
| stephenfin | my understanding of the current PTL situation (from second hand knowledge: I wasn't present at the relevant PTGs) is that it was determined an ability to merge code wasn't a prerequisite for doing all the other functions that being a PTL entails | 15:52 |
| stephenfin | ...but I guess that doesn't account for the fact that a PTL is ultimately the "sovereign" for a project in some sense. I've never really given it much thought before now | 15:53 |
| JayF | stephenfin: from a hard power standpoint: the PTL is the sole arbiter of who is a core | 15:53 |
| fungi | right, officially speaking the ptl can choose to delegate that, but it's still ultimately their responsibility | 15:53 |
| JayF | so anytime a PTL chooses to not be a core, it is *their choice* even if there is soft power being used to influence that positino | 15:53 |
| JayF | I agree with you that given the structure of the community, we don't have many PTLs acting like BDF-cycle, but it does mean that they ultimately have the power to break logjams if they choose to use it. | 15:54 |
| stephenfin | I don't know if that's the case https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/message/ECTUP2U7X776UQHU7XVIM22V6SDQLP5M/ | 15:55 |
| fungi | and openstack's governance was designed that way for a reason: the contributors can elect a ptl who can replace the whole core review team if necessary, in order to change the direction of the project | 15:55 |
| stephenfin | (I refrained from participating in that conversation given how infrequently I contribute to Nova at the moment) | 15:55 |
| JayF | stephenfin: that is someone asking for consent to do something they have power to do without that consent, if they were sufficiently motivated. That happens all the time in normal life. | 15:55 |
| stephenfin | good point | 15:56 |
| JayF | in some ways, I have a lot of respect for the approach; but I don't think I would've invested as much time as they did in the spot they are in | 15:57 |
| JayF | it's a massive green flag for someone to have power and avoid using it | 15:57 |
| stephenfin | absolutely | 15:57 |
| stephenfin | assuming they do realise the fact, that is 😅 As I said, I'd personally never considered this before. I doubt I'm alone | 15:58 |
| JayF | I suspect someone has had that conversation with them... I hope? | 15:59 |
| * stephenfin ensures that is the case, just to be safe | 16:00 | |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!