| Clark[m] | test | 00:10 |
|---|---|---|
| clarkb | bridge isn't down yet | 00:10 |
| opendevreview | Michael Johnson proposed openstack/diskimage-builder master: Remove deprecated datetime.utcnow() https://review.opendev.org/c/openstack/diskimage-builder/+/946007 | 00:55 |
| opendevreview | Leonardo Mendes proposed openstack/project-config master: Add IPSec Policy Operator App to StarlingX https://review.opendev.org/c/openstack/project-config/+/946008 | 02:09 |
| opendevreview | Leonardo Mendes proposed openstack/project-config master: Add IPSec Policy Operator App to StarlingX https://review.opendev.org/c/openstack/project-config/+/946008 | 02:17 |
| opendevreview | Merged opendev/irc-meetings master: Move TC meeting to 1700 UTC on Tuesdays https://review.opendev.org/c/opendev/irc-meetings/+/946005 | 07:37 |
| opendevreview | Lukas Kranz proposed zuul/zuul-jobs master: Add limit-log-file-size role. https://review.opendev.org/c/zuul/zuul-jobs/+/945795 | 08:43 |
| opendevreview | Lukas Kranz proposed zuul/zuul-jobs master: Add limit-log-file-size role. https://review.opendev.org/c/zuul/zuul-jobs/+/945795 | 09:10 |
| opendevreview | Lukas Kranz proposed zuul/zuul-jobs master: Add limit-log-file-size role. https://review.opendev.org/c/zuul/zuul-jobs/+/945795 | 10:31 |
| opendevreview | Lukas Kranz proposed zuul/zuul-jobs master: Add limit-log-file-size role. https://review.opendev.org/c/zuul/zuul-jobs/+/945795 | 10:39 |
| sean-k-mooney | frickler: o/ so two mysteries | 11:30 |
| frickler | infra-root: looks like a normal comment triggered a recheck, 10:54 is when sean-k-mooney made their first comment on https://review.opendev.org/c/openstack/osc-placement/+/943759 and the timestamp matches https://zuul.opendev.org/t/openstack/buildset/fdbca01550bf4e10b774eadb977591df | 11:30 |
| sean-k-mooney | ya that the first | 11:31 |
| frickler | there is also the question why the bindep file isn't processed as expected, I'll hold a node for checking, but I may not have time today to check it myself | 11:32 |
| sean-k-mooney | i can proably create a DNM change to just call tree on the zuul work dir or something like that | 11:33 |
| sean-k-mooney | this is failing in the pre playbook of the current job so ill have to define a new job to do that but we basically need to confirm 3 thigns , one the work dir is set properly, 2 the file exists (and the owershipt/permissions) 3 the content of the file can be read | 11:34 |
| sean-k-mooney | frickler: was there a zuul restart or anythin like that in the last day | 11:36 |
| sean-k-mooney | frickler: im wonderign if zuul bacially "just became aware of the patch" after a restart | 11:36 |
| sean-k-mooney | the build set says "Triggered by GerritChange openstack/osc-placement 943759,1 | 11:37 |
| sean-k-mooney | 2025-04-01 10:54:30""" | 11:37 |
| sean-k-mooney | which i normally assocaite with a change was pushed but this is patset 1 so the ohter possiblity is maybe there was a change in gerrit behvior or how the gerrit driver works, i.e. how it treats comments | 11:38 |
| sean-k-mooney | that would only make sense if there was a zuul or gerrit update in the last 24 hours | 11:38 |
| sean-k-mooney | i guess it say the same when tirggered by a recheck actully https://zuul.opendev.org/t/openstack/buildset/ff90325de0c4472da97753ff5c7ca3d0 | 11:40 |
| sean-k-mooney | so maybe it does not render diffently for the comment tirgger and that just reflect it was tiggered by the gerrit connection rather then how | 11:40 |
| frickler | someone would need to check zuul logs to see if they have something more specific, but I don't have time for that right now. or maybe that was zuul's april's fools joke? you never know ... ;) seriously maybe we shouldn't worry too much unless it repeats | 11:53 |
| sean-k-mooney | sure. ill keep an eye out | 11:56 |
| sean-k-mooney | and y im no treally worried about the trigger right now the job failure is odd but ill see if i can do some digging seperatly | 11:57 |
| sean-k-mooney | honestly the lack of release note for osc-placement wont matter much, i dont think we actully changed anythign imporant last cycle | 11:57 |
| sean-k-mooney | there were no api changes to placement last cycle so i dont think we did any osc-placement changes as a result | 11:58 |
| frickler | sean-k-mooney: actually ... it looks like there is no bindep.txt in osc-placement, so likely you'd need to add one for Noble now. but that would surely explain why it doesn't currently get processed ;) | 12:05 |
| sean-k-mooney | oh... i just realised i was looking at the placement repo | 12:06 |
| sean-k-mooney | frickler++ | 12:06 |
| sean-k-mooney | ok easy fix, you were right. ill fix the patch. if i see a random trigger like that ill let you know but probble not a concern as you said | 12:07 |
| nileshthathagar | Hello Everyone, | 12:13 |
| nileshthathagar | We are working on third party ci job with zuul + nodepool. | 12:14 |
| nileshthathagar | Everything seems to be running fine, but we are exploring options to re-run only the failed test cases. | 12:14 |
| nileshthathagar | Does anyone have insights on how to achieve this? Any guidance would be greatly appreciated! | 12:14 |
| sean-k-mooney | zuul is desiged to run piplines of jobs as a build set and to not allow reruning specific jobs | 12:15 |
| sean-k-mooney | if you want to re run tests within a job you need to do that in yoru test runner | 12:15 |
| nileshthathagar | we are using tox + tempest | 12:16 |
| nileshthathagar | to run the test cases | 12:16 |
| sean-k-mooney | tox uses stestr to run the tempest tests underneat | 12:16 |
| nileshthathagar | yeah we have used stestr to re-run the test cases. | 12:17 |
| sean-k-mooney | you coudl add a recover task to do tox -e<your env> -- --failing --combine | 12:17 |
| sean-k-mooney | maybe | 12:17 |
| sean-k-mooney | tempest does nto currently have a way to externally state that some test should be rerun | 12:17 |
| sean-k-mooney | it does however have decorator to make thing as unsable | 12:18 |
| nileshthathagar | yeah we are doing the same but the thing is --combine option duplicate the subunit output for failed test cases. | 12:18 |
| sean-k-mooney | they woudl prefer not to add a flakey decorator because that will encurage peopel to not fix the test | 12:18 |
| sean-k-mooney | nileshthathagar: it may i have only seen it used to run non overlapping test | 12:19 |
| sean-k-mooney | i actully kind of think having it there multipel times woudl be a feature not a bug | 12:19 |
| sean-k-mooney | in that you still report that it needed ot be retried | 12:19 |
| opendevreview | Lukas Kranz proposed zuul/zuul-jobs master: mirror-workspace-git-repos: Allow deleting current branch https://review.opendev.org/c/zuul/zuul-jobs/+/946033 | 12:40 |
| opendevreview | Lukas Kranz proposed zuul/zuul-jobs master: mirror-workspace-git-repos: Allow deleting current branch https://review.opendev.org/c/zuul/zuul-jobs/+/946033 | 12:40 |
| opendevreview | Lukas Kranz proposed zuul/zuul-jobs master: emit-job-header: add instance and region information https://review.opendev.org/c/zuul/zuul-jobs/+/946035 | 12:42 |
| fungi | corvus: i approved 921879 just now, meant to do it many months ago | 12:52 |
| opendevreview | Leonardo Mendes proposed openstack/project-config master: Add IPSec Policy Operator App to StarlingX https://review.opendev.org/c/openstack/project-config/+/946008 | 13:02 |
| fungi | sean-k-mooney: frickler: the behavior you saw on https://review.opendev.org/c/openstack/osc-placement/+/943759 is expected due to the state the change was in at the time of the comment: cr+2,w+1,v-2 | 13:08 |
| sean-k-mooney | fungi: no its not | 13:10 |
| frickler | fungi: is it? why didn't my comment trigger it then? might be because it was actually sean-k-mooney who made the W+1, so zuul saw that in the gerrit event? | 13:11 |
| sean-k-mooney | the +2w was already present | 13:11 |
| sean-k-mooney | i didnt add them new in that comment | 13:11 |
| sean-k-mooney | its true that the comement tecnially still has them | 13:12 |
| fungi | sean-k-mooney: gerrit would have re-included your existing w+1 in the comment, matching https://opendev.org/openstack/project-config/src/branch/master/zuul.d/pipelines.yaml#L26-L32 | 13:12 |
| sean-k-mooney | btu zuul is not ment to retirgger as if it was the first time i added +w | 13:12 |
| sean-k-mooney | yep but that should not retrigger | 13:12 |
| frickler | zuul cannot know the previous state, it only sees the event | 13:13 |
| sean-k-mooney | that sound alike a behvior change to me | 13:13 |
| opendevreview | Merged opendev/git-review master: Stop setting a default topic on new changes https://review.opendev.org/c/opendev/git-review/+/921879 | 13:14 |
| frickler | what I'm more confused about is that I've never seen this happen before. or maybe I just don't remember, that's always an option. but the explanation does sound reasonable to me. I'll try to crosscheck when I run across a gate failure on a change that I did workflow myself | 13:14 |
| sean-k-mooney | im i have never seen this happen before either | 13:15 |
| fungi | 2025-04-01 10:54:30,851 DEBUG zuul.ConnectionEventQueue: [e: 32e202bfeb224ba9bc030491cf59f7d8] Submitting connection event to queue /zuul/events/connection/gerrit/events/queue: {'timestamp': 1743504870.8515441, 'zuul_event_id': '32e202bfeb224ba9bc030491cf59f7d8', 'span_context': {'trace_id': 4568303850427311688800157343880657024, 'span_id': 10834952290029550363, 'trace_flags': 1}, | 13:15 |
| fungi | 'payload': {'author': {'name': 'sean mooney', 'email': 'smooney@redhat.com', 'username': 'sean-k-mooney'}, 'approvals': [{'type': 'Verified', 'description': 'Verified', 'value': '0'}, {'type': 'Code-Review', 'description': 'Code-Review', 'value': '2'}, {'type': 'Workflow', 'description': 'Workflow', 'value': '1'}, {'type': 'Review-Priority', 'description': 'Review-Priority', 'value': | 13:15 |
| fungi | '0'}], 'comment': 'Patch Set 1:\n\nthis is related to pcre... | 13:15 |
| sean-k-mooney | and i comment on failed patches that have +2w all the time | 13:15 |
| fungi | it could be a semi-recent gerrit behavior change | 13:15 |
| frickler | likely normally in 99% I'd do a comment that starts with "recheck" in such a case. after checking job logs of course ;) | 13:16 |
| sean-k-mooney | well i was not rechckign intentionally | 13:16 |
| sean-k-mooney | because i was still debuging | 13:16 |
| sean-k-mooney | which is somethign i do a lot if i dont have a root cause | 13:16 |
| frickler | sean-k-mooney: if you want to be sure to not recheck you could explicitly set W=0 in that case. still I'd also have expected the V-2 to act as a blocker | 13:18 |
| sean-k-mooney | i wonder if gerrit started reinclduing +w in the event? | 13:18 |
| sean-k-mooney | so that require block is a littel confusing https://opendev.org/openstack/project-config/src/branch/master/zuul.d/pipelines.yaml#L29-L32 | 13:19 |
| fungi | 2025-04-01 10:54:51,557 DEBUG zuul.Pipeline.openstack.check: [e: 32e202bfeb224ba9bc030491cf59f7d8] Event <GerritTriggerEvent comment-added opendev.org/openstack/osc-placement master 943759,1 Verified:0, Code-Review:2, Workflow:1, Review-Priority:0> for change <Change 0x7f4c39781410 openstack/osc-placement 943759,1> matched <GerritEventFilter connection: gerrit types: comment-added | 13:19 |
| fungi | ignore_deletes: True event_approvals: Workflow:1 require: <GerritRefFilter connection_name: gerrit required-approvals: [{'Verified': [-1, -2], 'username': <zuul.lib.re2util.ZuulRegex object at 0x7f4beb9f70d0>}]>> in pipeline <IndependentPipelineManager check | 13:19 |
| fungi | there's the more relevant log line | 13:19 |
| fungi | so it was definitely matching the event block i linked from the check pipeline config | 13:20 |
| sean-k-mooney | what exactly is this expressing https://opendev.org/openstack/project-config/src/branch/master/zuul.d/pipelines.yaml#L26-L32 | 13:20 |
| sean-k-mooney | it it sayign re run if workflow is added when tehre is a -1 or -2 form zuul | 13:20 |
| sean-k-mooney | because if so that is not some thing i expect to happen in general | 13:21 |
| fungi | if a comment is added with a workflow+1 and other merge requirements are already met and there's a v-1 or v-2 existing from the zuul user, then test the change again | 13:21 |
| sean-k-mooney | its been there for a very long tiem but i cant help but feel the behvior has changed | 13:22 |
| fungi | thats intended to allow you to approve a change with a negative verified score and have it automatically rechecked | 13:22 |
| sean-k-mooney | right which is not somethign i think we ever knew you coudl do and its not what i expect | 13:23 |
| sean-k-mooney | i have alwasy added recheck in that specific case if approving something | 13:24 |
| opendevreview | Leonardo Mendes proposed openstack/project-config master: Add IPSec Policy Operator App to StarlingX https://review.opendev.org/c/openstack/project-config/+/946008 | 13:24 |
| fungi | in the case of a change that fails in the gate, it's not uncommon to remove and reapply your w+1 vote, people have done that for years | 13:25 |
| fungi | but yes, i believe gerrit re-including prior votes in comment-added event data may have started around the same time that it started including implicit +0 votes | 13:26 |
| sean-k-mooney | i was just lookign at gerrits gerrit to see if i could find naything that might be related | 13:27 |
| sean-k-mooney | do you know when that was? | 13:27 |
| fungi | i do not, i think it's been a while | 13:27 |
| sean-k-mooney | im conflicted about what to do here other then annouch a psa on teh list | 13:28 |
| sean-k-mooney | to me this a breakign change in behvior requireign a use to change howe we review. we could just drop that comment trigger | 13:28 |
| sean-k-mooney | but that would also break people that use it | 13:28 |
| sean-k-mooney | so we are kidn fo stuck | 13:28 |
| sean-k-mooney | removing and adding +w is fine | 13:29 |
| sean-k-mooney | as a workflow i mean | 13:29 |
| sean-k-mooney | but any comment form the person that appoved it is a regression | 13:29 |
| sean-k-mooney | but i agree that that is vaild form a zuul point of view based on our current config | 13:29 |
| sean-k-mooney | so the regressin is realy in gerrit | 13:30 |
| opendevreview | Leonardo Mendes proposed openstack/project-config master: Add IPSec Policy Operator App to StarlingX https://review.opendev.org/c/openstack/project-config/+/946008 | 13:45 |
| opendevreview | Jeremy Stanley proposed opendev/bindep master: Replace extras with dependency groups https://review.opendev.org/c/opendev/bindep/+/945403 | 13:46 |
| clarkb | sean-k-mooney: fungi: iirc there are two camps that want opposite gerrit behavior. Both sides believing the other is a bug. I think Gerrit has gone with what they feel is the most complete set of information | 14:56 |
| clarkb | but I would have to dig through logs and reread a bunch of stuff to catch up on the history | 14:56 |
| clarkb | it may be that the supply both sets of info and the way our pipeline config is written is to effectively recheck if workflow is already set regardless of any new explicit votes? Again I would have to dig in, but I don't feel that this behavior is surprising or unexecpted | 14:57 |
| frickler | hmm, but gerrit sending an event that contains Verified:0 when there actually is an V-2 vote seems weird | 14:58 |
| fungi | i want to say if you expand the label schema then gerrit queries will tell you a timestamp for each of the votes, so in theory they could expand the comment-added event details similarly | 14:58 |
| clarkb | frickler: sean-k-mooney's vote was Verified 0 | 14:58 |
| fungi | right, zuul was matching on the workflow 1 vote associated with sean-k-mooney's comment (even though it was added by an earlier comment) | 14:59 |
| fungi | gerrit re-includes the user's "current" votes in each comment-added event, not just what they changed in that comment | 15:00 |
| frickler | ah, that's the user vote, not the aggregate state, ok | 15:00 |
| clarkb | ya they split it out by author | 15:01 |
| clarkb | There is a new Gerrit release this morning. The release notes don't feel urgent (whcih is good we can wait for after the openstack release). I will get up a change to build new images shortly though | 15:02 |
| frickler | btw. zuul's logging made be made nicer, this doesn't seem useful to me: 'username': <zuul.lib.re2util.ZuulRegex object at ... | 15:02 |
| sean-k-mooney | to know if it was form the current comment or a prior one they woudl need to have a changed lables list or old/new | 15:06 |
| sean-k-mooney | since they dont provide that info i dont see anything we can do on the zuul side to preerve the old behvior of only triging when +w was newly added | 15:06 |
| fungi | frickler: yeah, probably just missing an attribute for that match object | 15:07 |
| sean-k-mooney | verifed: 0 was added by gerrits internal defaulting i cant actully even set that with my permissoins | 15:07 |
| fungi | wouldn't take much to fix up the log statement, probably adding something like .text | 15:08 |
| sean-k-mooney | when i left the comemnt i did not set any lables | 15:08 |
| clarkb | fungi: frickler: no that means the username is field is treated as a regex. Either because we're supplying a regex or it is always interpreted as regex. I tried to improve the __repr__ and __str__ interpretation of those fields recently but discovered adding those methods broke a bunch of tests so I think something somewhere is relying on the default representation | 15:08 |
| fungi | aha | 15:08 |
| clarkb | but ya explicitly logging a n interpretation where we know it is safe to do so using a .text() or whatever method would probably work | 15:09 |
| frickler | maybe the tests are broken then, having bad implicit assumptions? ;) | 15:09 |
| clarkb | possibly | 15:10 |
| clarkb | oh I remember it has to do with serialization | 15:10 |
| clarkb | so not the tests explicitly but I think it serialzies wrong/differently when you have those implicit representation methods | 15:10 |
| clarkb | anyway fungi's suggestion would work I suspect without impacting things like serialization if someone wanted to work on that | 15:10 |
| frickler | I'd also be willing to discuss whether removing this trigger to avoid inintended merging of changes would be the better option https://opendev.org/openstack/project-config/src/branch/master/zuul.d/pipelines.yaml#L26-L32 if the change doesn't merge, users will come and complain, but no harm is done. unintentionally merging a change that has failed CI (due to some unstable job), might be worse | 15:12 |
| clarkb | don't approvechanges you don't want to merge | 15:13 |
| sean-k-mooney | im less concerned with that | 15:13 |
| clarkb | I think that should be a rule regardless of gerrit event behavior | 15:13 |
| sean-k-mooney | my main concern was with jobs running when i did not ask them too | 15:13 |
| sean-k-mooney | i.e. when i was still debuggin gwhy it failed and i was leaving comments for others to review | 15:13 |
| frickler | well it was approved earlier, before the failing CI was noticed. it would be good for reviever to remove their W+1 then, but that is easily missed when just writing a quick note | 15:14 |
| fungi | yeah, if there's a procedural takeaway for this, it's that if you notice a change you approved has failed tests and you don't want it to be rechecked, drop your approval | 15:14 |
| sean-k-mooney | in this case tehre was nothing wrong with this change. we needed to make a seperat change to account for a job change so unappoving it is not nessisarly correct | 15:14 |
| sean-k-mooney | fungi: that is not correct. | 15:14 |
| sean-k-mooney | i didnt want to recheck it yet | 15:15 |
| sean-k-mooney | i plannd to becaus ei didnt thinkg it was a issue with the patch | 15:15 |
| frickler | the recheck isn't bad in itself, but if it is successful, it will automatically regate, too. the latter may be unexpected and unwanted | 15:15 |
| sean-k-mooney | if the jobs passed it was find to merge | 15:15 |
| fungi | but also, in that state any user could leave a recheck comment to get a similar behavior | 15:16 |
| clarkb | which we've long seen as a feature | 15:16 |
| clarkb | otherwise people would have to wait for core reviewers to explicitly recheck/reapprove | 15:16 |
| sean-k-mooney | i didnt want to block someone else form merging it via a recheck with needed a core to reappove | 15:16 |
| sean-k-mooney | that why i did not clear +w | 15:16 |
| fungi | yeah, this is an unfortunate corner case with the new-ish gerrit behavior change. you can't leave a change with your approval vote and continue to comment on it yourself if you don't want it to get rechecked automatically when it fails | 15:18 |
| sean-k-mooney | yep | 15:18 |
| sean-k-mooney | we can send a PSA mail ot alert peopel of that change in behvior | 15:19 |
| sean-k-mooney | what was concerning to me was not knowing why the bevhiro chanaged when the zuul config didnt | 15:19 |
| fungi | we could drop the responsible trigger event from the check pipeline, as long as users are okay giving up the ability to reenqueue through removing and reapplying their workflow vote like they've done in the past | 15:19 |
| corvus | the behavior is 8 years old | 15:19 |
| sean-k-mooney | corvus: not the retrgger on any comment | 15:20 |
| sean-k-mooney | but the retriver via remove and add workflow iss | 15:20 |
| sean-k-mooney | so that why i was relucnt to remove that tirgger | 15:20 |
| fungi | the gerrit release that started incorporating earlier vote values in later comment-added events? | 15:20 |
| corvus | yes | 15:20 |
| corvus | 2.13 | 15:20 |
| fungi | was that a side effect of the shift to notedb, i guess? | 15:21 |
| frickler | in general it seems nice to follow a principle of least surprise and this certainly surprised at least two people today who aren't really new to the setup, but whatever | 15:22 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update Gerrit container image to python3.12 https://review.opendev.org/c/opendev/system-config/+/944408 | 15:26 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update Gerrit images to 3.10.5 and 3.11.2 https://review.opendev.org/c/opendev/system-config/+/946050 | 15:26 |
| clarkb | I decided that we should update Gerrit to latest release before switching the underlying python image. That is why the existing change gets updated (it is rebased on the gerrit new release change) | 15:26 |
| opendevreview | Merged openstack/project-config master: Temporarily remove release docs semaphores https://review.opendev.org/c/openstack/project-config/+/945884 | 16:25 |
| opendevreview | Clark Boylan proposed opendev/engagement master: Add bulk processing and csv output https://review.opendev.org/c/opendev/engagement/+/946061 | 16:35 |
| opendevreview | Brian Haley proposed opendev/irc-meetings master: Move Neutron meeting to 1300 UTC on Tuesdays https://review.opendev.org/c/opendev/irc-meetings/+/946063 | 17:00 |
| opendevreview | Dmitriy Rabotyagov proposed opendev/system-config master: Add Epoxy UCA to mirrors https://review.opendev.org/c/opendev/system-config/+/946068 | 17:16 |
| opendevreview | Merged openstack/project-config master: Add IPSec Policy Operator App to StarlingX https://review.opendev.org/c/openstack/project-config/+/946008 | 17:49 |
| noonedeadpunk | does anybody know from top of their head if there are afs-related constraints on adding epoxy uca mirror? (talking about https://review.opendev.org/c/opendev/system-config/+/946068) | 17:54 |
| clarkb | noonedeadpunk: you can check https://grafana.opendev.org/d/9871b26303/afs?orgId=1&from=now-6h&to=now&timezone=utc but usually uca isn't a big deal (its small relative to other mirrors) | 17:56 |
| noonedeadpunk | yeah, it seems that uca is indeed not taking much | 17:57 |
| noonedeadpunk | it's 7gb or smth overall | 17:58 |
| frickler | I would maybe want to wai until the pkgs are rebuilt after the release to avoid resyncing them all, but space should be fine, yes | 17:59 |
| noonedeadpunk | so that pretty much blocks osa to switch the provisioned "codename" for the release, as it's related to the repo naming in a way, resulting in https://zuul.opendev.org/t/openstack/build/fb1885d3cfcb43bf8dcc6e0a70c42a7f | 18:01 |
| noonedeadpunk | ofc we can overcome that in CI by saying - not to use infra mirror for now | 18:01 |
| clarkb | remember its a mirror so it shouldn't ever block you. You just stop using the mirror | 18:01 |
| noonedeadpunk | but ofc we can wait couple of days for this | 18:01 |
| clarkb | but I don't see any issue adding that new release | 18:01 |
| fungi | yeah, you should already be able to use uca packages in your jobs | 18:02 |
| fungi | just won't be as fast as if we had them mirrored | 18:02 |
| noonedeadpunk | yeah, sure, I just can recall that it's preferable to use infra mirrors in CI as well:) | 18:02 |
| frickler | noonedeadpunk: do you use the actual openstack pkgs there or just other stuff? | 18:02 |
| noonedeadpunk | in this job - actual openstack packages | 18:02 |
| fungi | it's also a small enough subset of the ubuntu packages those jobs probably install that the performance shouldn't be that noticeable | 18:03 |
| frickler | hmm, ok, for me that's a good enough reason to approve the mirror creation right away, if you want to propose it | 18:03 |
| noonedeadpunk | fungi: hm, so we have some misconfigured apt? or? | 18:03 |
| fungi | i'd have to see the sources list from one of your jobs to say for sure | 18:04 |
| noonedeadpunk | fungi: https://zuul.opendev.org/t/openstack/build/fb1885d3cfcb43bf8dcc6e0a70c42a7f/log/logs/etc/host/apt/sources.list.d/uca.sources.txt | 18:04 |
| noonedeadpunk | frickler: I think this should be the patrch https://review.opendev.org/c/opendev/system-config/+/946068 | 18:05 |
| fungi | noonedeadpunk: yeah, that entry refers to one of our package mirrors (which doesn't exist) rather than an official ubuntu mirror | 18:05 |
| noonedeadpunk | yeah, as we explicitly set them to the local mirror when in CI to avoid external connections... | 18:07 |
| noonedeadpunk | which ofc we can override... | 18:08 |
| fungi | if your job instead used https://ubuntu-cloud.archive.canonical.com/ubuntu/dists/noble-updates/epoxy/ until the mirror is available, it would work is what i'm saying | 18:08 |
| noonedeadpunk | yeah, ok | 18:08 |
| fungi | and then you can drop your override. not that i'm opposed to hurrying up with adding a mirror for it, just pointing out that those things can happen in parallel | 18:08 |
| frickler | yes, but that would require special casing the CI setup, maybe we can avoid that | 18:09 |
| noonedeadpunk | frickler: we actually have special case CI setup | 18:09 |
| frickler | I'll approve the change tomorrow when I can watch the initial sync or maybe someone else wants to do it even before that | 18:10 |
| fungi | wow, that job is still relying on the legacy `source /etc/ci/mirror_info.sh` | 18:10 |
| fungi | rather than using zuul's ansible vars | 18:10 |
| noonedeadpunk | https://opendev.org/openstack/openstack-ansible/src/branch/master/tests/roles/bootstrap-host/tasks/prepare_aio_config.yml#L79-L84 | 18:11 |
| noonedeadpunk | it's just that we''d need not to forget to revert the change:) | 18:11 |
| noonedeadpunk | so if we're talking about days - I;'d prefer to wait | 18:11 |
| fungi | i try to always push the revert with the change when i do that, then at worst you forget to approve it ;) | 18:11 |
| noonedeadpunk | oh, yes... I think I indeed need to look at replacing mirror_info.sh | 18:12 |
| noonedeadpunk | I somehow though it was already done :( | 18:13 |
| clarkb | infra-root fungi and I had a meeting on meetpad a little bit ago and we tested that etherpad works, as well as video and audio. Everything looked good so I'm going to put meetpad02 and jvb02 in the emergency file now so that we don't auto update before the ptg and break something | 18:20 |
| clarkb | corvus: fwiw my gerrit on python3.12 image change also hit npm issues with 403 errors from yarnpkg.com. Manually fetching those resources now seems to work so I'll recheck | 18:23 |
| corvus | yeah, i'm seeing better results now | 18:31 |
| clarkb | sidenote ^ that message came over the matrix oftc bridge. | 18:34 |
| fungi | noted | 18:36 |
| fungi | april fools joke? | 18:36 |
| corvus | the announcement from feb was a little unclear... maybe it's more like, if they don't get the money by end of march, they will announce a shutdown plan and date around this time? | 18:38 |
| fungi | but also i never found any updates on their progress toward the stated funding goal after that eitehr | 18:40 |
| fungi | either | 18:40 |
| fungi | so it's possible they got enough to keep it going for a while? | 18:41 |
| corvus | no funding thermometer billboard | 18:41 |
| clarkb | their latest blogpost (from the 28th) didn't make any mention of it | 18:41 |
| clarkb | maybe no news is good news | 18:42 |
| fungi | since it looks like things are quiet, i think i'm going to knock off now-ish to grab dinner and have an early night, so i can be reasonably alert when openstack release tasks get underway in the morning. night all! | 20:04 |
| clarkb | sounds good! see you in the morning | 20:05 |
| tonyb | ++ | 20:05 |
| clarkb | fwiw I just checked the gerrit caches since I'm thinking about replacing the server and sizes there have remained pretty stable. Still largeer than I would expect but not massive like we discovered in december | 20:10 |
| Clark[m] | corvus: https://news.ycombinator.com/item?id=43548589 | 20:21 |
| Clark[m] | You'll enjoy the npm failure root cause | 20:21 |
| opendevreview | Merged opendev/engagement master: Bitergia time to first review is a days value not seconds https://review.opendev.org/c/opendev/engagement/+/946006 | 20:22 |
| opendevreview | Merged opendev/engagement master: Add bulk processing and csv output https://review.opendev.org/c/opendev/engagement/+/946061 | 20:22 |
| frickler | wow, I'm around longer than fungi, that may be a first ;) gn8 | 20:23 |
| corvus | Clark: wow; can confirm our jobs failed on a package with camel in the name | 20:25 |
| clarkb | I guess the security concern is great enough to make that opt out rather than opt in. But still a global waf rule that prevents the word camel in your urls is something | 20:37 |
| opendevreview | Matt Anson proposed openstack/diskimage-builder master: Make mellanox element depend on epel https://review.opendev.org/c/openstack/diskimage-builder/+/946088 | 21:18 |
| fungi | that's an oddly specific filtering rule... just sayin' | 21:20 |
| fungi | and no, i.m not really here, my computer is merely responding with astonishment on my behalf | 21:21 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!