| -@gerrit:opendev.org- Dmitry Tantsur proposed: [openstack/project-config] 992711: Declare the Metal3 namespace in the projects configuration https://review.opendev.org/c/openstack/project-config/+/992711 | 09:42 | |
| @clarkb:matrix.org | infra-root thoughts on proceeding with the gerrit image cleanups and addition of 3.14 images starting with https://review.opendev.org/c/opendev/system-config/+/991945 ? I'm out monday and tuesday and don't want to forget these changes if we're feeling confident with 3.13 | 17:35 |
|---|---|---|
| @clarkb:matrix.org | that said happy to followup with this when I'm back next week if ew would like to wait a bit longer | 17:35 |
| @fungicide:matrix.org | all three in that series lgtm and are straightforward replication of what we've been doing already with only slight variation, so i went ahead and approved them | 17:45 |
| @clarkb:matrix.org | cool thanks | 17:48 |
| @clarkb:matrix.org | one thing to keep in mind is that due to me changing the bazelisk version we will rebuild the 3.13 image when we build the 3.14 image | 17:51 |
| @clarkb:matrix.org | but now that we use tags I don't think we should be worried about that | 17:51 |
| @fungicide:matrix.org | but also the testinfra testing passed on those images in check, so should be fine if we did end up running a rebuilt 3.13 | 17:53 |
| @clarkb:matrix.org | yup | 17:54 |
| @clarkb:matrix.org | oh we switched the replication plugin away from tags which is the one gotcha | 17:54 |
| @fungicide:matrix.org | i'd be surprised if a new bazelisk version broke the build in ways that were so subtle our test exercises were unimpeded | 17:54 |
| @mordred:waterwanders.com | gerrit 3.13 has been running long enough that I think if there were _large_ issues we would have noticed by now | 17:55 |
| @clarkb:matrix.org | oh and plugin/replication doesn't have a stable-3.14 so my override checkout must then fallback to master? Which I think is correct and isn't in the image we're running in prod yet so I think its fine | 17:55 |
| @mordred:waterwanders.com | so rolling back to 3.12 feels really unlikely at this point | 17:55 |
| @clarkb:matrix.org | the last update to replication stable-3.13 was the update we switched to the branch from the tag for so that should also be fine | 17:56 |
| @mordred:waterwanders.com | do we use their replication plugin for anything? | 17:56 |
| @clarkb:matrix.org | mordred: yes replication to gitea is done with the replication plugin | 17:56 |
| @fungicide:matrix.org | replicating | 17:56 |
| @clarkb:matrix.org | fungi: so I guess if you want me to fix the stable-3.14 override checkout in the 3.14 image update change I can update that. But I think it is doing the correct thing by using master anyway? | 17:56 |
| @clarkb:matrix.org | and we don't deploy that image so its very low risk | 17:57 |
| @fungicide:matrix.org | we no longer use it for local replicas (we got rid of those) or github (those are opt-in with zuul jobs) | 17:57 |
| @mordred:waterwanders.com | ah, duh. | 17:57 |
| @fungicide:matrix.org | Clark: yeah, i don't think it's a concern | 17:57 |
| @fungicide:matrix.org | and once that branch exists it'll still dtrt | 17:58 |
| @fungicide:matrix.org | unless we want to switch back to tags | 17:58 |
| @clarkb:matrix.org | yup exactly | 17:59 |
| @clarkb:matrix.org | https://gerrit.googlesource.com/plugins/replication/+log/refs/heads/stable-3.13 this is the stable-3.13 branch fwiw. | 18:00 |
| @clarkb:matrix.org | And ya no updates until the merge up for the bugfix that we pulled in with the switch to the branch checkout rather than the tag | 18:00 |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/system-config] 991945: Drop gerrit 3.12 image builds and testing https://review.opendev.org/c/opendev/system-config/+/991945 | 18:00 | |
| @fungicide:matrix.org | `ERROR: error loading package 'plugins/its-base': Unable to find package for @@com_googlesource_gerrit_bazlets//:gerrit_plugin.bzl: The repository '@@com_googlesource_gerrit_bazlets' could not be resolved: Repository '@@com_googlesource_gerrit_bazlets' is not defined.` | 19:24 |
| @fungicide:matrix.org | ummm, i have **no** idea what that means | 19:24 |
| @fungicide:matrix.org | https://zuul.opendev.org/t/openstack/build/5ca64cfa175a4428aeff088fae59cf06 | 19:24 |
| @fungicide:matrix.org | i guess one of the build dependencies moved out from under it? | 19:25 |
| @clarkb:matrix.org | Ya I guess its-base doesn't have tags either so maybe it updated between builds? | 19:33 |
| @clarkb:matrix.org | I am eating lunch now then have end of school year activities this afternoon so may not look closer until tomorrow | 19:33 |
| @fungicide:matrix.org | no worries, i'm juggling half a dozen things already but spotted that and thought it might be something temporary, so i rechecked the change to get a second opinion | 19:34 |
| @clarkb:matrix.org | Ack sounds good | 19:34 |
| @fungicide:matrix.org | yeah, it seems to have failed again in check, so probably needs one or more plugins re-resolved now | 19:54 |
| @clarkb:matrix.org | https://gerrit.googlesource.com/plugins/its-base/+/6cb6ecbb7c6b1eece34f7affbfb0efa36ed348bc this is almost certainly the cause | 19:55 |
| @clarkb:matrix.org | I'll send a query to discord | 19:56 |
| @clarkb:matrix.org | https://gerrit.googlesource.com/bazlets/ this repo exists. I'm guessing that maybe we need to include that repo in the build somehow? But before we add more repos to our zuul config I'll see what they say on discord | 20:00 |
| @clarkb:matrix.org | haven't heard back from gerrit yet but I think I understand the issue. https://gerrit-review.googlesource.com/c/plugins/its-base/+/592601/2/BUILD this change landed to switch to using bazlets which is loaded using the @key reference. You need to define that reference in your WORKSPACE or MODULE.bazel file. This was done in its-base itself here: https://gerrit.googlesource.com/plugins/its-base/+/6cb6ecbb7c6b1eece34f7affbfb0efa36ed348bc/WORKSPACE#5 but we are building the plugins within the context of Gerrit's WORKSPACE not the its-base plugin's WORKSPACE/MODULE.bazel file. Gerrit master defines bazlets resources in its WORKSPACE but stable-3.13 and stable-3.14 do not. I suspect that what we need are new stable branches for its-base based on the commit prior to this update (there are branches only up to stable-3.9: https://gerrit.googlesource.com/plugins/its-base/+refs) I have asked upstream if this is correct | 20:21 |
| @clarkb:matrix.org | we can't override checkout on a specific sha1 in the zuul config, but we should be able to checkout the specific sha1 in the build jobs after zuul checks out the bad commit. I'll work on an update that attempts that so that we have an out if we need one | 20:22 |
| @fungicide:matrix.org | okay, that makes sense | 20:23 |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/system-config] 993032: Workaround Gerrit its-base plugin build issue https://review.opendev.org/c/opendev/system-config/+/993032 | 20:34 | |
| @clarkb:matrix.org | ok I stacked that at the end of the stack in the hopes that the chagne won't be needed as upstream will give us stable branches we can use instead. But if that does fix the problem and we want to use it then we can rebase and stick it at the beginning of the stack of changes instead | 20:35 |
| @clarkb:matrix.org | I need to pop out now for school stuff. Feel free to update that change if there are errors that need correcting | 20:35 |
| @clarkb:matrix.org | corvus: there are some interesting failures for 993032. It almost looks like the system-config-run jobs tried to use the images from the prior change and failed because they don't exist. But I've got pipeline level job dependencies that say we have a soft dependency on the enqueued image build jobs | 20:38 |
| @clarkb:matrix.org | corvus: is this maybe a corner case zuul bug where if you both have a dependency and an artifact requires the lack of an artifact requires short circuits things and we don't wait for the dependency? | 20:39 |
| @clarkb:matrix.org | I feel like this has maybe come up before when artifacts are looked up when there are failures in earlier patchsets. Maybe this is another variant of that where we have a corner case we need to handle | 20:39 |
| @clarkb:matrix.org | anyway really have to go now | 20:39 |
| @jim:acmegating.com | haha | 20:39 |
| @jim:acmegating.com | sorry i guess this will be very asynchronous | 20:40 |
| @jim:acmegating.com | i think you're talking about https://zuul.opendev.org/t/openstack/build/0dc71bf93f5446c19f0778dbdb4e0cb6 | 20:40 |
| @jim:acmegating.com | which says: Error: Job system-config-run-review-3.13 requires artifact(s) gerrit-3.13-container-image provided by build 87f9bdc2782f4620afe562f66e896535 (triggered by change 991946 on project opendev/system-config), but that build failed with result "FAILURE" | 20:40 |
| @clarkb:matrix.org | yup that | 20:40 |
| @jim:acmegating.com | i'm just trying to page in a whole bunch of context... i think you're saying that you want it to use the build from the same buildset instead of the prior one | 20:43 |
| @jim:acmegating.com | we use provides/requires to tell it we need artifacts from other buildsets, but we have a soft dependency saying to use artifacts from this buildset. | 20:44 |
| @clarkb:matrix.org | Yes and the dependency at the job level should prevent the system config run job from running until the same buildset image build completes | 20:45 |
| @jim:acmegating.com | but is it also the case that the stack can not possibly merge as-is -- as long as the failure that's tripping up the provides/requires is there, this later change (which i think if i'm following correctly, fixes the failure) can't merge because it's grandparent can't merge due to the error | 20:46 |
| @clarkb:matrix.org | Correct. | 20:47 |
| @jim:acmegating.com | so, we might be tempted to do a bunch of stuff to override provides/requires with job graph dependencies for this case, but it's not actually going to change the gating outcome | 20:47 |
| @clarkb:matrix.org | I guess from a correctness standpoint this is ok. It's just less good from an evaluation standpoint since the change I just pushed is a huge hack and I'd like to avoid it if possible while still proving it would work | 20:47 |
| @clarkb:matrix.org | But changes can be reordered before merging anything so not a huge issue there withwr | 20:48 |
| @clarkb:matrix.org | * But changes can be reordered before merging anything so not a huge issue there either | 20:48 |
| @jim:acmegating.com | yeah. i think all of the following may be true: | 20:49 |
| * zuul is not incorrect | ||
| * it would be more convenient for us if it behaved like you describe | ||
| * there may be reasons to make changes like you describe | ||
| * this may not be the best example use-case for that | ||
| @jim:acmegating.com | so maybe for immediate evaluation purposes, you could just squash it all into one, or make a second, re-ordered stack? | 20:49 |
| @clarkb:matrix.org | I would agree with those assertions and yes I'll just reorder if we decide we need to test this more robustly | 20:50 |
| @clarkb:matrix.org | ok the image build did work. I'll go ahead and stack my hack under the other two changes. Sounds like upstream agrees with my assessment and will discuss it at their meeting tomorrow | 22:41 |
| @clarkb:matrix.org | maybe it won't be necessary but for now I'll make it possible to build the images | 22:41 |
| -@gerrit:opendev.org- Clark Boylan proposed: | 22:44 | |
| - [opendev/system-config] 993032: Workaround Gerrit its-base plugin build issue https://review.opendev.org/c/opendev/system-config/+/993032 | ||
| - [opendev/system-config] 991946: Add Gerrit 3.14 image builds and testing https://review.opendev.org/c/opendev/system-config/+/991946 | ||
| - [opendev/system-config] 992342: Test Gerrit 3.13 to 3.14 upgrades https://review.opendev.org/c/opendev/system-config/+/992342 | ||
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!