| -@gerrit:opendev.org- Lukas Kranz proposed: [zuul/zuul-jobs] 910582: Make prepare-workspace-git fail faster. https://review.opendev.org/c/zuul/zuul-jobs/+/910582 | 06:35 | |
| -@gerrit:opendev.org- Simon Westphahl proposed: [zuul/zuul] 913875: wip: Correctly limit buildsets with multiple refs https://review.opendev.org/c/zuul/zuul/+/913875 | 09:11 | |
| -@gerrit:opendev.org- Simon Westphahl proposed: [zuul/zuul] 913875: wip: Correctly limit buildsets with multiple refs https://review.opendev.org/c/zuul/zuul/+/913875 | 09:12 | |
| -@gerrit:opendev.org- Simon Westphahl proposed: [zuul/zuul] 913875: wip: Correctly limit buildsets with multiple refs https://review.opendev.org/c/zuul/zuul/+/913875 | 09:20 | |
| @harbott.osism.tech:regio.chat | seems the ensure-docker role is broken with upstream release of docker 26.0.0. see https://zuul.opendev.org/t/openstack/build/cb7357c58fca44dca88d952691a2d8de but having similar results downstream, too | 11:21 |
|---|---|---|
| -@gerrit:opendev.org- Dr. Jens Harbott proposed: [zuul/zuul-jobs] 913897: DNM: testing ensure-docker role https://review.opendev.org/c/zuul/zuul-jobs/+/913897 | 11:24 | |
| @harbott.osism.tech:regio.chat | `Mar 21 11:28:14 np0037123857 dockerd[2606]: invalid DOCKER_MIN_API_VERSION: minimum supported API version is 1.24: 1.22` | 11:30 |
| @harbott.osism.tech:regio.chat | ah, nice, this even mentions that this will break https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/ensure-docker/tasks/docker-setup.yaml#L60-L75 | 11:32 |
| -@gerrit:opendev.org- Dr. Jens Harbott proposed: [zuul/zuul-jobs] 913808: Revert "Override DOCKER_MIN_API_VERSION for skopeo when installing docker" https://review.opendev.org/c/zuul/zuul-jobs/+/913808 | 11:45 | |
| -@gerrit:opendev.org- Dr. Jens Harbott proposed: [zuul/zuul-jobs] 913808: Revert "Override DOCKER_MIN_API_VERSION for skopeo when installing docker" https://review.opendev.org/c/zuul/zuul-jobs/+/913808 | 11:51 | |
| -@gerrit:opendev.org- Radosław Piliszek https://matrix.to/#/@yoctozepto:matrix.org proposed on behalf of Jens Harbott: [zuul/zuul-jobs] 913808: Revert "Override DOCKER_MIN_API_VERSION for skopeo when installing docker" https://review.opendev.org/c/zuul/zuul-jobs/+/913808 | 12:00 | |
| -@gerrit:opendev.org- Radosław Piliszek proposed on behalf of Jens Harbott: [zuul/zuul-jobs] 913808: Revert "Override DOCKER_MIN_API_VERSION for skopeo when installing docker" https://review.opendev.org/c/zuul/zuul-jobs/+/913808 | 12:28 | |
| -@gerrit:opendev.org- Radosław Piliszek proposed on behalf of Jens Harbott: [zuul/zuul-jobs] 913808: Revert "Override DOCKER_MIN_API_VERSION for skopeo when installing docker" https://review.opendev.org/c/zuul/zuul-jobs/+/913808 | 12:55 | |
| @fungicide:matrix.org | zuul-maint: bringing discussion from #opendev irc back around to here, it looks like merging 913808 will probably require us to bypass gating since the zuul-jobs-test-registry-buildset-registry job running for it inherits from a definition in opendev/base-jobs (a trusted config repo) which includes the currently broken ensure-docker role from its pre-run playbook and therefore isn't testing the fix. opinions? | 12:57 |
| @fungicide:matrix.org | also what's a good way to know that the change won't break skopeo? | 12:57 |
| @yoctozepto:matrix.org | fungi: I dropped testing that because it does not make sense to confuse committers that it is testing the right thing! | 12:58 |
| @yoctozepto:matrix.org | and I second the skopeo q | 12:58 |
| @yoctozepto:matrix.org | I have no idea what impact we have with this revert w.r.t. skopeo | 12:58 |
| @fungicide:matrix.org | my reading of the code comments and discussion on the prior change would suggest that as long as we require the recent skopeo version from last month it will be compatible, but some confirmation would be swell | 13:01 |
| @yoctozepto:matrix.org | my understanding to that is also that some platforms might be affected while others not | 13:02 |
| @yoctozepto:matrix.org | but I don't have the time to go further on this on my own | 13:03 |
| @fungicide:matrix.org | yoctozepto: as for removing zuul-jobs-test-registry-buildset-registry it looks like it was being run for use by other test-registry jobs, not merely to test that it worked | 13:03 |
| @yoctozepto:matrix.org | fungi: oh! I have not noticed, argh | 13:03 |
| @yoctozepto:matrix.org | which ones? | 13:04 |
| @yoctozepto:matrix.org | ah you mean the others I disable | 13:04 |
| @yoctozepto:matrix.org | hmm, hmm | 13:05 |
| @fungicide:matrix.org | the job description states "Run a buildset registry for the test-registry jobs" | 13:05 |
| @yoctozepto:matrix.org | yeah, it was used byt zuul-jobs-test-registry-buildset-registry-k8s-microk8s and crio | 13:05 |
| @yoctozepto:matrix.org | s/was/is - more precisely | 13:05 |
| @yoctozepto:matrix.org | :D | 13:05 |
| @fungicide:matrix.org | yeah, it might make sense to disable temporarily in order to get that fix merged and then restore the jobs after, in order to avoid having to bypass gating | 13:05 |
| @yoctozepto:matrix.org | ok, so an immediate followup is to reenable that (sadly it is interdependent!) and its mk8s variant (as crio is independently b0rken) | 13:06 |
| @fungicide:matrix.org | and then discuss whether there are better ways to test those components in zuul-jobs without consuming some of them via a job defined in a config project | 13:06 |
| @yoctozepto:matrix.org | yeah, I guess we could simplify the opendev-buildset-registry - it does not have to rely on ensure-docker... | 13:07 |
| @yoctozepto:matrix.org | I mean, it's really only meant to run the registry 😅 | 13:07 |
| @fungicide:matrix.org | but those are definitely not the only jobs run for zuul-jobs changes which inherit from config project jobs that in turn use roles from zuul-jobs (log collection and uploading for example) | 13:08 |
| @yoctozepto:matrix.org | we could even have a prebaked image for that | 13:08 |
| @yoctozepto:matrix.org | yeah, sadly true, although those are less variable I guess | 13:08 |
| @yoctozepto:matrix.org | like, copying files breaks lot less than docker install :D | 13:08 |
| @yoctozepto:matrix.org | or at least I hope it does or else we are in some deep rabbit hole | 13:09 |
| @fungicide:matrix.org | sure, i agree that it's possible to consider different approaches for each of them based on the degree of risk they pose and other factors | 13:09 |
| @yoctozepto:matrix.org | as for the current change - I would love it if it merge it soon as everything-docker on opendev is very unhappy atm (including our awesome nebulous) | 13:10 |
| @fungicide:matrix.org | when we knowingly make changes to things used by opendev/base-jobs we try to (as much as possible) set up shadow jobs with alternate inheritance to prove the proposals with before putting them into the critical path | 13:11 |
| @yoctozepto:matrix.org | yeah, I saw that, your (opendev) practices are really some of the highest-quality in the public part of the world | 13:12 |
| @yoctozepto:matrix.org | very commendable | 13:12 |
| @fungicide:matrix.org | well, it's just being cautious really | 13:12 |
| @fungicide:matrix.org | nobody wants to be the person who brings a gigantic developer collaboration engine to a grinding halt | 13:13 |
| @yoctozepto:matrix.org | hah, true that, but one has to take it serious to put the effort into life! | 13:16 |
| @yoctozepto:matrix.org | otherwise it is just a wish | 13:16 |
| @yoctozepto:matrix.org | ok, the change is marked green by zuul | 13:24 |
| -@gerrit:opendev.org- Radosław Piliszek proposed on behalf of Jens Harbott: [zuul/zuul-jobs] 913808: Revert "Override DOCKER_MIN_API_VERSION for skopeo when installing docker" https://review.opendev.org/c/zuul/zuul-jobs/+/913808 | 13:31 | |
| -@gerrit:opendev.org- Radosław Piliszek proposed: [zuul/zuul-jobs] 913902: Reenable buildset-registry jobs https://review.opendev.org/c/zuul/zuul-jobs/+/913902 | 13:34 | |
| @yoctozepto:matrix.org | let me know if now it's all right | 13:35 |
| @fungicide:matrix.org | so... Clark mentioned in #opendev irc that the "skopeo as of february" mentioned in the comments as being compatible isn't buildable on the current ubuntu lts as it needs a newer golang toolchain. this is getting increasingly double-plus ungood | 13:35 |
| @tristanc_:matrix.org | fungi: that's unfortunate, but if it's broken, then it seems like we should force push until it works | 13:36 |
| @tristanc_:matrix.org | is there something we could change too to prevent this situation in the future? | 13:37 |
| @yoctozepto:matrix.org | oh my dear | 13:37 |
| @clarkb:matrix.org | > <@harbott.osism.tech:regio.chat> ah, nice, this even mentions that this will break https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/ensure-docker/tasks/docker-setup.yaml#L60-L75 | 13:37 |
| The expectation wasn't that the daemon would explode if the env var was set though. That's insane. The expectation was that skopeo would stop working again. A completely different failure mode | ||
| @yoctozepto:matrix.org | well, they certainly follow the fail-fast methodology! | 13:38 |
| @yoctozepto:matrix.org | which is not bad per se, can be helpful | 13:38 |
| @yoctozepto:matrix.org | nonetheless, we not here to say bad things about docker (I know we can find plenty!) but fix the current situation | 13:39 |
| @yoctozepto:matrix.org | reposting Clark in here: | 13:39 |
| @yoctozepto:matrix.org | 14:35:22 <Clark[m]> Note those registry jobs will be broken because skopeo should be broken too. | 13:39 |
| Clark[m] | I think a recheck on the reenable change should expose that if we land the first change | 14:35 |
| @clarkb:matrix.org | > <@tristanc_:matrix.org> is there something we could change too to prevent this situation in the future? | 13:39 |
| Stop using docker and skopeo as neither seem to be maintaining their software in a manner that is friendly to actual users? | ||
| More generally no as long as we update packages like docker (or anything else) when external updates are available we have to deal with incompatibilities and breakages when they arrive | ||
| @yoctozepto:matrix.org | yeah, I had this in mind that this might fail later | 13:39 |
| @yoctozepto:matrix.org | but can't confirm due to the zuul behaviour | 13:39 |
| @yoctozepto:matrix.org | so let's merge | 13:39 |
| @yoctozepto:matrix.org | and check the followup | 13:39 |
| @yoctozepto:matrix.org | we will see how it fares | 13:39 |
| @yoctozepto:matrix.org | and proceed from that | 13:40 |
| @yoctozepto:matrix.org | any better ideas? | 13:40 |
| @clarkb:matrix.org | > <@yoctozepto:matrix.org> which is not bad per se, can be helpful | 13:41 |
| I strongly disagree particularly in this case. Postel's law applies here. | ||
| @tristanc_:matrix.org | Clark: I meant with regards to the opendev/base-jobs config preventing to use the gate | 13:41 |
| @clarkb:matrix.org | I'm not sure what you mean by that? Preventing to use the gate? | 13:43 |
| @yoctozepto:matrix.org | > <@clarkb:matrix.org> I strongly disagree particularly in this case. Postel's law applies here. | 13:43 |
| I mean, I get them in that I don't see how I would let operators know their option is ignored early other than via some hidden-somewhere warning (and then we bitch about it only after we find it breaking skopeo ;p) | ||
| @clarkb:matrix.org | > <@yoctozepto:matrix.org> I mean, I get them in that I don't see how I would let operators know their option is ignored early other than via some hidden-somewhere warning (and then we bitch about it only after we find it breaking skopeo ;p) | 13:43 |
| You would know if your clients fail. And if they don't fail no one cares and can move on with their lives. | ||
| @yoctozepto:matrix.org | Postel's law is good in request-reply scenarios | 13:43 |
| @clarkb:matrix.org | But to break everyone regardless is just spiteful | 13:44 |
| @yoctozepto:matrix.org | agreed | 13:44 |
| @yoctozepto:matrix.org | nonetheless, let's fix what we can | 13:44 |
| @fungicide:matrix.org | > <@tristanc_:matrix.org> Clark: I meant with regards to the opendev/base-jobs config preventing to use the gate | 13:45 |
| we could perhaps duplicate those job definitions in zuul-jobs instead of inheriting them, but i'm not sure if there might be a reason they have to be defined in a trusted config repo (which would explain why we didn't do that before now) | ||
| @yoctozepto:matrix.org | the change is green again | 13:45 |
| @yoctozepto:matrix.org | merge/amend? | 13:46 |
| @clarkb:matrix.org | tristanC: if you mean avoiding situations where trusted code can't be speculatively updated then you need to keep as much code out of trusted state as possible. It isn't always possible to do that though. I'm not sure if it would be here or not | 13:46 |
| @yoctozepto:matrix.org | in this particular case, we can minimise our interdependencies | 13:46 |
| @yoctozepto:matrix.org | but it does not generalise | 13:46 |
| @clarkb:matrix.org | But also I don't think this is a huge emergency. You can always set jobs to noop and effectively force merge a change to work around | 13:47 |
| @fungicide:matrix.org | for the roles that enable log collection and uploading, as a counterexample, we need them run from jobs with access to secrets so the duplicative solution wouldn't be an option | 13:47 |
| @yoctozepto:matrix.org | could we please focus on the matter at hand for now? | 13:48 |
| @clarkb:matrix.org | We are? | 13:48 |
| @clarkb:matrix.org | We have to decide what the best route around the testing problems is and that is what we are discussing? | 13:49 |
| @yoctozepto:matrix.org | not exactly, we try to devise solutions to more general issues that we discovered... | 13:49 |
| @yoctozepto:matrix.org | as I see it at least | 13:49 |
| @clarkb:matrix.org | We need to understand the problem before we can prescribe a solution | 13:50 |
| @yoctozepto:matrix.org | I say we follow the path of merging this and checking how buildset-registry jobs fare otherwise | 13:50 |
| @clarkb:matrix.org | And I'm not sure I actually understand the testing issue yet | 13:50 |
| @clarkb:matrix.org | In particular everyone is acting like this is an emergency for all jobs but this only affects jobs using the container builds rogh | 13:51 |
| @clarkb:matrix.org | * In particular everyone is acting like this is an emergency for all jobs but this only affects jobs using the container builds right? | 13:51 |
| @yoctozepto:matrix.org | yeah, it does affect only docker jobs so containers | 13:51 |
| @clarkb:matrix.org | Ok so the blast radius is reasonably well contained. | 13:51 |
| @yoctozepto:matrix.org | well, the blast radius is nebulous project halted to a grind | 13:52 |
| @yoctozepto:matrix.org | so it is not the best containment I could imagine! | 13:52 |
| @yoctozepto:matrix.org | I mean, if we merge this, we can actually proceed with debugging and devising solutions | 13:52 |
| @yoctozepto:matrix.org | we don't make it worse as the same jobs are b0rken now | 13:53 |
| @clarkb:matrix.org | Sort of we're just going to have broke skopeo | 13:53 |
| @yoctozepto:matrix.org | which is broken now anyway | 13:53 |
| @clarkb:matrix.org | But yes no worse than now | 13:53 |
| @yoctozepto:matrix.org | exactly my point | 13:53 |
| @clarkb:matrix.org | Why does the change delete and comment out jobs? | 13:54 |
| @clarkb:matrix.org | It should only stop running the jobs? | 13:54 |
| @yoctozepto:matrix.org | I already answered that in the comments | 13:56 |
| @yoctozepto:matrix.org | because it is what the automation wants from me | 13:56 |
| @yoctozepto:matrix.org | these are autogenerated | 13:56 |
| @yoctozepto:matrix.org | at the bottom I mean | 13:56 |
| @yoctozepto:matrix.org | btw, I have a brain meltdown | 13:56 |
| @yoctozepto:matrix.org | https://codesearch.opendev.org/?q=ensure-skopeo&i=nope&literal=nope&files=&excludeFiles=&repos= | 13:56 |
| @yoctozepto:matrix.org | ensure-skopeo is actually not used beyond testing? | 13:57 |
| @yoctozepto:matrix.org | :D | 13:57 |
| @yoctozepto:matrix.org | (I tried figuring out the scope of testing the followup) | 13:57 |
| @clarkb:matrix.org | The builders registry jobs are also autogenerated or just the crio ones? | 13:58 |
| @clarkb:matrix.org | * The buildset registry jobs are also autogenerated or just the crio ones? | 13:59 |
| @yoctozepto:matrix.org | seemingly all are just picked up from the definitions to their use | 13:59 |
| @yoctozepto:matrix.org | I dropped the ones I am restoring in a different order (due to crio being bad...) in the followup | 13:59 |
| @yoctozepto:matrix.org | (please see there, that could shed some light too) | 13:59 |
| @clarkb:matrix.org | I think the crio jobs are autogenerated from their abstract parent to add platform specific tests. But the buildset registry jobs may not be? | 14:01 |
| @clarkb:matrix.org | I can't actually test that via tox until a little later | 14:02 |
| @tristanc_:matrix.org | Clark: yes, fungi mentioned that we might need to bypass the gate for 913808, but that no longer seems to be the case. But if we had to, then we might use this opportunity to also prevent that situation for occurring in the future, though I don't know how. Perhaps the offending job could be changed to non-voting? | 14:04 |
| @yoctozepto:matrix.org | Clark: we actually mean to reenable them so why would we need to know that exactly? | 14:05 |
| @fungicide:matrix.org | tristanC: the options were to either bypass gating to merge the change, or temporarily take out the failing jobs and then restore them in a subsequent change, yes | 14:05 |
| @clarkb:matrix.org | Yes that was my point you can always elect to run the noop job or mark things non voting. You should never truly be stuck | 14:05 |
| @yoctozepto:matrix.org | as for the skopeo - it seems we would test "our usage" with the followup just like Clark suggested | 14:05 |
| @clarkb:matrix.org | yoctozepto: I have an allergy to unnecessarily large diffs | 14:05 |
| @clarkb:matrix.org | I probably run `git log -p` more often than most but discipline in constructing diffs goes a long way to making that useful | 14:06 |
| @yoctozepto:matrix.org | I also prefer them short and nice | 14:06 |
| @yoctozepto:matrix.org | I can drop instead of commenting out | 14:06 |
| @yoctozepto:matrix.org | but I guess it will be easier later on | 14:06 |
| @yoctozepto:matrix.org | as we keep crio for later | 14:06 |
| @yoctozepto:matrix.org | * crio reenablement | 14:07 |
| @clarkb:matrix.org | No you misunderstand me | 14:07 |
| @yoctozepto:matrix.org | I truly do then | 14:07 |
| @yoctozepto:matrix.org | :D | 14:07 |
| @tristanc_:matrix.org | fungi: I see, thanks for the explanation. I was worried that something in opendev/base-job was causing an issue. | 14:07 |
| @clarkb:matrix.org | The crio stuff is fine because it is autogenerated. The other jobs are not autogenerated I don't think. So why do we delete them? | 14:07 |
| @clarkb:matrix.org | Just remove them from the list of jobs to run and then add them back to the list of jobs to run | 14:07 |
| @yoctozepto:matrix.org | because the automation sticks them in nonetheless | 14:07 |
| @yoctozepto:matrix.org | or let me check again | 14:08 |
| @yoctozepto:matrix.org | yeah, it wants to put everything visible in the list | 14:08 |
| @yoctozepto:matrix.org | so we need to either drop or comment out the definitions | 14:08 |
| @yoctozepto:matrix.org | and I agree this is not elegant but it's not mine automation... | 14:08 |
| @yoctozepto:matrix.org | ;-) | 14:08 |
| @yoctozepto:matrix.org | anyhow, to me this is "good enough" (TM) | 14:09 |
| @clarkb:matrix.org | Ok so in addition to autogenerated platform jobs we put all jobs in the run list | 14:09 |
| @fungicide:matrix.org | tristanC: indirectly, yes. basically ensure-docker is currently broken by a new docker release. a job we're running to test some parts of zuul-jobs inherits from a job definition in opendev/base-jobs which relies on the ensure-docker role from zuul/zuul-jobs so that job when testing the change to fix the ensure-docker role ends up using the broken current ensure-docker instead (which is fine, if inconvenient, as the job's purpose isn't to test ensure-docker exactly) | 14:09 |
| @yoctozepto:matrix.org | fungi: tristanC another take at this would be to treat ensure-docker as a job that has very minimal logic that is much less likely to fail | 14:10 |
| @yoctozepto:matrix.org | that customised API version was just too much for today | 14:10 |
| @yoctozepto:matrix.org | :D | 14:10 |
| @yoctozepto:matrix.org | in other words: anything in base jobs needs to be very carefully handled | 14:11 |
| @yoctozepto:matrix.org | which you all know already anyways | 14:11 |
| @yoctozepto:matrix.org | it is just that it is not always that obvious where the dependencies are | 14:11 |
| @yoctozepto:matrix.org | in the meantime, I will create a followup that reenables that poor crio | 14:12 |
| @yoctozepto:matrix.org | as well | 14:12 |
| @jim:acmegating.com | anyone have a link to an example of a broken crio job? | 14:12 |
| @fungicide:matrix.org | sure, we could duplicate a strictly necessary subset of things from zuul/zuul-jobs in opendev/base-jobs rather than reusing them, though that also increases the maintenance burden | 14:12 |
| @clarkb:matrix.org | > <@yoctozepto:matrix.org> that customised API version was just too much for today | 14:13 |
| I'll happily cede all fixing of these issues to others | ||
| @yoctozepto:matrix.org | fungi: well, all IT boils down to tradeoffs at the end of the day :D | 14:14 |
| @jim:acmegating.com | the commit msg says "disables crio jobs because they have their repo seemingly broken" and that's a little thin to follow up on. i don't think we need to make a new patchset, but i'd at least like to have a comment in gerrit describing how it's broken so whatever fix happens can refer back to that. | 14:15 |
| @clarkb:matrix.org | corvus: https://zuul.opendev.org/t/zuul/build/5c4166d3ad044522b9b006dcecde290c looks like one | 14:15 |
| -@gerrit:opendev.org- Radosław Piliszek proposed: [zuul/zuul-jobs] 913905: Reenable crio jobs https://review.opendev.org/c/zuul/zuul-jobs/+/913905 | 14:15 | |
| @jim:acmegating.com | Clark: thanks | 14:15 |
| @yoctozepto:matrix.org | ^ that of mine will need including the fix for crio of course | 14:16 |
| @clarkb:matrix.org | looks like it may be trying to use xenial packages on jammy and the xenial packages have gone away so the apt update fails | 14:16 |
| @jim:acmegating.com | yeah, so that may be something we need to fix in the job | 14:16 |
| @jim:acmegating.com | not necessarily that the repo itself is broken | 14:16 |
| @yoctozepto:matrix.org | yeah, more as in "repo entry" in apt config | 14:17 |
| @fungicide:matrix.org | interesting that a run on jammy is trying to get a package index that claims to be for xenial | 14:17 |
| @yoctozepto:matrix.org | could we simply iterate on https://review.opendev.org/c/zuul/zuul-jobs/+/913905/1 | 14:17 |
| @yoctozepto:matrix.org | and merge the sad docker's fix | 14:17 |
| @jim:acmegating.com | yoctozepto: i'm helping you complete the work on the docker fix so that i can merge it | 14:18 |
| @yoctozepto:matrix.org | corvus: I'm optimising for the number of jobs retried (considering the vast amount on that base change vs the children) | 14:19 |
| @jim:acmegating.com | yoctozepto: i understand, which is why i added the information in a comment, and did not request an update to the commit message. i said as much above. | 14:19 |
| @yoctozepto:matrix.org | ah, sorry, i misunderstoo clearly, thank you! | 14:20 |
| @yoctozepto:matrix.org | ok, we wait for this to merge | 14:20 |
| @yoctozepto:matrix.org | then recheck the first child | 14:20 |
| @jim:acmegating.com | i approved 808, thanks frickler yoctozepto | 14:20 |
| @yoctozepto:matrix.org | expect skopeo breakage in mk8s job | 14:20 |
| @yoctozepto:matrix.org | 🎊 | 14:21 |
| @yoctozepto:matrix.org | now, back to skopeo, I saw ensure-skopeo is not actually used - does it mean we have skopeo prebaked in the images? | 14:26 |
| @clarkb:matrix.org | no I think we have things just not using ensure-skopeo | 14:27 |
| @jim:acmegating.com | skopeo is installed in the zuul-executor image, but i don't know if that's relevant | 14:27 |
| @jim:acmegating.com | where are we looking? | 14:27 |
| @yoctozepto:matrix.org | corvus: I mean I am preparing for its potential breakage as foreseen by Clark | 14:28 |
| @yoctozepto:matrix.org | so looking for the place where it gets in | 14:28 |
| @yoctozepto:matrix.org | so as to know where to start fixing | 14:28 |
| @yoctozepto:matrix.org | :D | 14:28 |
| @jim:acmegating.com | yoctozepto: you said: `I saw ensure-skopeo is not actually used` -- show me where you saw that | 14:28 |
| @yoctozepto:matrix.org | corvus: ah, meaning in opendev, no hits except for tests https://codesearch.opendev.org/?q=%5Cbensure-skopeo%5Cb&i=nope&literal=nope&files=&excludeFiles=&repos= | 14:29 |
| @yoctozepto:matrix.org | so skopeo must be getting in a different way | 14:29 |
| @yoctozepto:matrix.org | and now I know it's preinstalled in the image | 14:29 |
| @yoctozepto:matrix.org | so thanks for that | 14:29 |
| @clarkb:matrix.org | skopeo 1.14 bumped the golang requirement from golang 1.12 to golang 1.19. Ubuntu jammy has 1.18. Debian Bookworm has 1.19. | 14:29 |
| @clarkb:matrix.org | The fix for skopeo's docker api version negotiation is in 1.14.x or 1.15.x. I'm trying to find an exact version now | 14:30 |
| @clarkb:matrix.org | One workaround we may be able to live with temporarily is to simply move testing that relies on skopeo to debian instead of ubuntu and then compile a working skopeo | 14:30 |
| @yoctozepto:matrix.org | Clark: but since we build the images, we can use any any toolkit we want... it's not python that we need to have it installed for skopeo to work afterwards | 14:31 |
| @clarkb:matrix.org | https://github.com/containers/skopeo/issues/2202 says this was fixed in skopeo 1.14.2 | 14:31 |
| @fungicide:matrix.org | it may also be possible to run a skopeo binary on ubuntu jammy that was built on debian bookworm, i thought golang executables were generally statically linked and relocatable, but i don't really know that much about the runtime | 14:31 |
| @yoctozepto:matrix.org | a different story would be with ensure-skopeo because we would have to host the binaries... | 14:31 |
| @jim:acmegating.com | the latest zuul-executor image has skopeo 1.9.3 | 14:31 |
| @clarkb:matrix.org | fungi: yes another option is to build the binary elsewhere and copy it | 14:32 |
| @yoctozepto:matrix.org | fungi: they are | 14:32 |
| @yoctozepto:matrix.org | I ++ any flavor of that approach | 14:32 |
| @clarkb:matrix.org | I'm not a huge fan of that approach because it requires a lot of coordination | 14:32 |
| @yoctozepto:matrix.org | Clark: I am not following where you get that problem from | 14:33 |
| @clarkb:matrix.org | either build locally using a new enough golang or build locally using a container with new enough golang I think | 14:33 |
| @fungicide:matrix.org | yes, i agree building on one platform for use on another is messy and involves solving other problems like storing and redistributing | 14:33 |
| @clarkb:matrix.org | > <@yoctozepto:matrix.org> Clark: I am not following where you get that problem from | 14:33 |
| I'd like to avoid building and hosting skopeo binaries longer term | ||
| @fungicide:matrix.org | i mentioned it mostly out of completeness | 14:34 |
| @yoctozepto:matrix.org | Clark: yeah, but I understood we ship the images anyways, so I get that we don't need to have a temporary place to ship the binaries | 14:34 |
| @fungicide:matrix.org | * i mentioned it mostly for the sake of completeness | 14:34 |
| @jim:acmegating.com | seems like the proposal on the table is: ignore ensure-skopeo because opendev and zuul are not using it, and then build skopeo in the executor dockerfile using a builder image (i think mordred suggested that) does that sound right? | 14:34 |
| @yoctozepto:matrix.org | or, wait, I don't know your image build process well enough to understand how you would be impacted | 14:34 |
| @yoctozepto:matrix.org | I was thinking in terms of docker multi-stage builds | 14:34 |
| @yoctozepto:matrix.org | so sorry if I am pushing the thinking in the wrong direction! | 14:34 |
| @yoctozepto:matrix.org | corvus: very right, sir! | 14:35 |
| @jim:acmegating.com | yoctozepto: yes multi-stage builds is an option for updating the skopeo in the executor image | 14:35 |
| @yoctozepto:matrix.org | if you point me in the place to deliver, I am happy to | 14:35 |
| @clarkb:matrix.org | corvus: I think the executor docker file is debian bookworm so doesn't need a different builder platform. But ya a separate builder multistage step could be a good option. But I don't think that will address other places we found broken skopeo which was in those buildset registry jobs | 14:35 |
| @clarkb:matrix.org | in the buildset registry jobs maybe we convert the testing to debian and then build locally | 14:35 |
| @yoctozepto:matrix.org | Clark: but they are getting skopeo from that image | 14:35 |
| @yoctozepto:matrix.org | or are not? | 14:36 |
| @clarkb:matrix.org | no | 14:36 |
| @yoctozepto:matrix.org | because I don't see how else they do get it in the end | 14:36 |
| @jim:acmegating.com | okay point me to those :) | 14:36 |
| @clarkb:matrix.org | they install it from the distro and/or build skoeo | 14:36 |
| @mordred:waterwanders.com | Multi-stage builds ftw! | 14:36 |
| @yoctozepto:matrix.org | the point is I see only ensure-skope installing skopeo | 14:37 |
| @yoctozepto:matrix.org | * the point is I see only ensure-skopeo installing skopeo | 14:37 |
| @yoctozepto:matrix.org | and it's not used | 14:37 |
| @yoctozepto:matrix.org | so otherwise I am blind | 14:37 |
| @yoctozepto:matrix.org | as to where this happens | 14:37 |
| @clarkb:matrix.org | `zuul-jobs/test-playbooks/registry/test-registry-pre.yaml` | 14:37 |
| @clarkb:matrix.org | which appears to use ensure-skopeo | 14:37 |
| @yoctozepto:matrix.org | that's only for tests | 14:37 |
| @yoctozepto:matrix.org | just my point | 14:37 |
| @yoctozepto:matrix.org | the actual jobs do not use it | 14:37 |
| @jim:acmegating.com | test jobs are actual jobs | 14:37 |
| @yoctozepto:matrix.org | but nebulous does not use test jobs | 14:38 |
| @yoctozepto:matrix.org | and we somehow have skopeo in there :S | 14:38 |
| @yoctozepto:matrix.org | nor do we use ensure-skopeo | 14:38 |
| @jim:acmegating.com | okay if you only want talk about nebulous this is not the right place. | 14:38 |
| @yoctozepto:matrix.org | no, no | 14:38 |
| @yoctozepto:matrix.org | I want to talk general usage of these jobs | 14:39 |
| @jim:acmegating.com | great :) | 14:39 |
| @yoctozepto:matrix.org | beyond testing | 14:39 |
| @yoctozepto:matrix.org | ;-) | 14:39 |
| @jim:acmegating.com | back to the issue, i don't think ensure-skopeo is necessary for that test. so i think we can entertain alternate ways of getting skopeo there | 14:39 |
| @yoctozepto:matrix.org | in other words, they do have skopeo but the test jobs try to install it nonetheless | 14:39 |
| -@gerrit:opendev.org- Joseph Kostreva proposed: [zuul/zuul] 913595: Change allow_pre_fail phases if will_retry https://review.opendev.org/c/zuul/zuul/+/913595 | 14:39 | |
| @yoctozepto:matrix.org | so maybe we drop it | 14:39 |
| @yoctozepto:matrix.org | and are happy | 14:39 |
| @yoctozepto:matrix.org | :D | 14:39 |
| @clarkb:matrix.org | yoctozepto: you are misunderstanding what the executor image's skopeo is for | 14:40 |
| @yoctozepto:matrix.org | Clark: I clearly am! | 14:40 |
| @yoctozepto:matrix.org | please teach me | 14:40 |
| @jim:acmegating.com | the point of that stanza is to simulate an executor, so i'm open to any way of getting a skopeo, and honestly, the closer to the way we do it in the executor the better | 14:40 |
| @clarkb:matrix.org | that puts a skopeo binary on the zuul executor for shuffling container image bits around so that you can safely publish container images at the end of builds | 14:40 |
| @clarkb:matrix.org | that does not put a skopeo binary on the "workload" side of your jobs | 14:40 |
| @clarkb:matrix.org | in the case of the buildset registry jobs we have a workload that uses skopeo and that needs to be addressed as well. | 14:41 |
| @clarkb:matrix.org | to be clear I think both things need addressing not one or the other | 14:41 |
| @jim:acmegating.com | so in that job, on our simulated executor -- is that node running jammy? if we switch it to bookworm will the ensure-skopeo role work and produce the correct updated skopeo version? | 14:42 |
| @jim:acmegating.com | alternatively, we could probably just replace that stanza with "copy skopeo from the real executor to the fake one" | 14:42 |
| @clarkb:matrix.org | corvus: yes, except we need to also bump the default version of skopeo built by ensure-skopeo or override it in the job (same as bumping the version on the executor itself) | 14:42 |
| @jim:acmegating.com | since it's statically linked golang, right? | 14:42 |
| @jim:acmegating.com | Clark: ack | 14:42 |
| @clarkb:matrix.org | and we want at least 1.14.2 if we want skopeo to negotiate the docker api version instead of hardcoding it to something from 2014 | 14:43 |
| @yoctozepto:matrix.org | corvus: Clark thanks, I get it now | 14:44 |
| @yoctozepto:matrix.org | these jobs exercise the tasks that are otherwise executed on the executor on the workload side of things instead | 14:45 |
| @yoctozepto:matrix.org | for some reason | 14:45 |
| -@gerrit:opendev.org- Zuul merged on behalf of Jens Harbott: [zuul/zuul-jobs] 913808: Revert "Override DOCKER_MIN_API_VERSION for skopeo when installing docker" https://review.opendev.org/c/zuul/zuul-jobs/+/913808 | 14:45 | |
| @yoctozepto:matrix.org | what confuses me still, why we test this way and not using the executor still? | 14:45 |
| @yoctozepto:matrix.org | ok, recheck in | 14:46 |
| @jim:acmegating.com | okay, so 2 things need fixing: | 14:46 |
| 1) multi-stage build in zuul/Dockerfile to build new skopeo (let's still use multi-stage even though the platform supports it just to keep things clean) | ||
| 2) fix test-playbooks/registry/test-registry-pre.yaml to either: | ||
| a) run on bookworm and build later version of skopeo | ||
| b) copy skopeo from the real executor to the fake one | ||
| @clarkb:matrix.org | so I think there are two things there. The first is that you should be able to run skopeo in your workload regardless of what the executor does | 14:46 |
| @yoctozepto:matrix.org | so it should fail on the fact that it's installing the old skopeo | 14:46 |
| @clarkb:matrix.org | thats perfectly acceptable. We don't have a rule that says "the only skopeo that should work is the executor". | 14:47 |
| @yoctozepto:matrix.org | Clark: agreed | 14:47 |
| @clarkb:matrix.org | The other is that I think these jobs may have been written before we loosened the rules on what can run on the executor which meant we had not choice but to emulate on the workload side of things | 14:47 |
| @yoctozepto:matrix.org | could be | 14:47 |
| @jim:acmegating.com | yoctozepto: yes that's correct; the reason for the fake executor is that they likely predate the ability to run arbitrary code on the executor. it's possible they could be reworked to just use the real executor at this point, but that's not clear. they may still need it for other reasons though. | 14:47 |
| @yoctozepto:matrix.org | because now they are testing a different thing that the actual jobs do | 14:48 |
| @jim:acmegating.com | Clark: jinx | 14:48 |
| @clarkb:matrix.org | corvus: matrix actually interleaved our messages after the fact. Neat | 14:48 |
| @yoctozepto:matrix.org | corvus: so we need to amend the (2) from your idea list | 14:48 |
| @jim:acmegating.com | Clark: i agree that ensure-skopeo should ideally work anywhere, but there's a good possibility that no one is using it except for us in that role, so if we drop our only use of it, we can see if anyone is using it and wants to fix it. otherwise i don't feel like we need to maintain it. | 14:50 |
| @clarkb:matrix.org | thats fair | 14:50 |
| @yoctozepto:matrix.org | yeah, less maintenance burden is always a good thing (TM) | 14:50 |
| @jim:acmegating.com | yoctozepto: what's your suggested amendment to (2) ? | 14:51 |
| @yoctozepto:matrix.org | 2024-03-21 14:51:12.140342 | TASK [Push test image into fake buildset registry] | 14:52 |
| 2024-03-21 14:51:12.740850 | ubuntu-jammy | time="2024-03-21T14:51:12Z" level=fatal msg="initializing source docker-daemon:zuul/testimage:latest: loading image from docker engine: Error response from daemon: {\"message\":\"client version 1.22 is too old. Minimum supported API version is 1.24, please upgrade your client to a newer version\"}" | ||
| @yoctozepto:matrix.org | fails as expected, folks | 14:52 |
| @yoctozepto:matrix.org | corvus: following your discussion with Clark, (2) should be to make the tests use the executor as the non-test jobs do | 14:52 |
| @yoctozepto:matrix.org | without any other faking/simulating/whatever | 14:53 |
| @jim:acmegating.com | yoctozepto: i think that's too big of a bite to take now | 14:53 |
| @jim:acmegating.com | i think we should fix the job minimally first, then refactor it later if possible | 14:53 |
| @yoctozepto:matrix.org | corvus: hmm, you mean it's not as simple as changing some all->localhost? could be, have not checked much | 14:54 |
| @yoctozepto:matrix.org | so for now I guess let's copy the binary | 14:54 |
| @yoctozepto:matrix.org | and forget about ensure-skopeo | 14:54 |
| @jim:acmegating.com | getting an updated skopeo on the fake executor is a bite sized piece of work we can do now and get the jobs working again; anything bigger will likely open a can of worms, which is worth doing, but will be much easier once the jobs are actually working agin | 14:55 |
| @yoctozepto:matrix.org | I have to go make myself some food and then go help my mom with her doctor's appointment so I will be out for some larger period of time (possibly till tomorrow) | 14:55 |
| @yoctozepto:matrix.org | corvus: got it! | 14:55 |
| @clarkb:matrix.org | I would try updated the job to run on bookworm and bump the version of skopeo we compile | 14:55 |
| @jim:acmegating.com | yeah, so 2b -- copy the binary, is probably easiest, though it does mean we have to finish 1 and get it into production first; 2a could probably happen in parallel with 1. | 14:55 |
| @clarkb:matrix.org | that may just work | 14:55 |
| @jim:acmegating.com | Clark: cool, if you think that stands a shot, let's try 2a first; we'll have time to switch to 2b after we finish 1 anyway | 14:56 |
| @clarkb:matrix.org | ++ | 14:57 |
| @clarkb:matrix.org | should I write the change? | 14:57 |
| @jim:acmegating.com | Clark: that'd be great; i can try starting work on item #1 unless anyone else wants that | 14:57 |
| @clarkb:matrix.org | ok I'm doing 2a to be clear | 14:58 |
| -@gerrit:opendev.org- Simon Westphahl proposed: [zuul/zuul] 913914: Don't reset buildset when cycle dependency merged https://review.opendev.org/c/zuul/zuul/+/913914 | 15:01 | |
| -@gerrit:opendev.org- Clark Boylan proposed on behalf of Radosław Piliszek: | 15:07 | |
| - [zuul/zuul-jobs] 913902: Reenable buildset-registry jobs https://review.opendev.org/c/zuul/zuul-jobs/+/913902 | ||
| - [zuul/zuul-jobs] 913905: Reenable crio jobs https://review.opendev.org/c/zuul/zuul-jobs/+/913905 | ||
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 913917: Build a new skopeo for the zuul-executor container image https://review.opendev.org/c/zuul/zuul/+/913917 | 15:27 | |
| @jim:acmegating.com | Clark: i think https://opendev.org/zuul/zuul-jobs/src/branch/master/zuul-tests.d/container-roles-jobs.yaml#L242 may need updating? | 15:31 |
| @jim:acmegating.com | Clark: (i think that happens 3 times -- maybe that should be a yaml anchor | 15:31 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 913917: Build a new skopeo for the zuul-executor container image https://review.opendev.org/c/zuul/zuul/+/913917 | 15:34 | |
| @jim:acmegating.com | Clark: i'll update the patch | 15:38 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed on behalf of Radosław Piliszek: | 15:41 | |
| - [zuul/zuul-jobs] 913902: Reenable buildset-registry jobs https://review.opendev.org/c/zuul/zuul-jobs/+/913902 | ||
| - [zuul/zuul-jobs] 913905: Reenable crio jobs https://review.opendev.org/c/zuul/zuul-jobs/+/913905 | ||
| -@gerrit:opendev.org- Clark Boylan proposed on behalf of Radosław Piliszek: | 15:48 | |
| - [zuul/zuul-jobs] 913902: Reenable buildset-registry jobs https://review.opendev.org/c/zuul/zuul-jobs/+/913902 | ||
| - [zuul/zuul-jobs] 913905: Reenable crio jobs https://review.opendev.org/c/zuul/zuul-jobs/+/913905 | ||
| @clarkb:matrix.org | oh shoot did I just push over what corvus pushed? | 15:48 |
| -@gerrit:opendev.org- Clark Boylan proposed on behalf of Radosław Piliszek: | 15:56 | |
| - [zuul/zuul-jobs] 913902: Reenable buildset-registry jobs https://review.opendev.org/c/zuul/zuul-jobs/+/913902 | ||
| - [zuul/zuul-jobs] 913905: Reenable crio jobs https://review.opendev.org/c/zuul/zuul-jobs/+/913905 | ||
| -@gerrit:opendev.org- Clark Boylan proposed on behalf of Radosław Piliszek: | 16:18 | |
| - [zuul/zuul-jobs] 913902: Reenable buildset-registry jobs https://review.opendev.org/c/zuul/zuul-jobs/+/913902 | ||
| - [zuul/zuul-jobs] 913905: Reenable crio jobs https://review.opendev.org/c/zuul/zuul-jobs/+/913905 | ||
| @clarkb:matrix.org | skopeo 1.15 actually requires golang 1.20. Their docs are not up to date /me pins back to 1.14.2 | 16:32 |
| -@gerrit:opendev.org- Clark Boylan proposed on behalf of Radosław Piliszek: | 16:35 | |
| - [zuul/zuul-jobs] 913902: Reenable buildset-registry jobs https://review.opendev.org/c/zuul/zuul-jobs/+/913902 | ||
| - [zuul/zuul-jobs] 913905: Reenable crio jobs https://review.opendev.org/c/zuul/zuul-jobs/+/913905 | ||
| @clarkb:matrix.org | if 1.14.2 also requires golang 1.20 then this attempt at fixing it won't work | 16:36 |
| @clarkb:matrix.org | I do think that skopeo should strongly consider backporting the api version negotiation fix if anyone knows anyone at skopeo that they can suggest thsi to | 16:36 |
| @yoctozepto:matrix.org | btw, the current setup of things works just fine with skopeo on the executor (as tested in nebulous) | 16:37 |
| @clarkb:matrix.org | skopeo + docker basically does not work on any stable distro release any longer | 16:37 |
| @clarkb:matrix.org | yoctozepto: this would only be an issue if skopeo had to fetch images (possibly for publication) from docker in your job | 16:37 |
| @yoctozepto:matrix.org | I think it does pull and push images | 16:39 |
| @clarkb:matrix.org | hrm I half wonder then if older skopeo did negotiate versions then they had a regression that stopped that | 16:40 |
| @yoctozepto:matrix.org | I think it does this only registry-registry | 16:41 |
| @clarkb:matrix.org | bookworm has skopeo 1.9.3 | 16:41 |
| @clarkb:matrix.org | oh ya registry to registry would be fine | 16:41 |
| @clarkb:matrix.org | its only talking to the docker daemon that breaks | 16:41 |
| @yoctozepto:matrix.org | I mean in this job it only would be doing that, I think | 16:41 |
| @clarkb:matrix.org | so if your jobs use docker cli to push the image to the registry then skopeo talks to the registry all is well | 16:42 |
| @clarkb:matrix.org | but as soon as you try to have skopeo talk to docker directly it explodes | 16:42 |
| @yoctozepto:matrix.org | yeah, I figured | 16:42 |
| @yoctozepto:matrix.org | but then it seems this can be well avoided | 16:43 |
| @yoctozepto:matrix.org | as if only the test jobs were meant to break 🤣 | 16:43 |
| @clarkb:matrix.org | I'm not sure that is a good attitude to take. The registries could start enforcing the same api limits to follow dockers lead | 16:45 |
| @clarkb:matrix.org | docker maintains one of the most popular registries too | 16:45 |
| @yoctozepto:matrix.org | no, that is well standardised now | 16:48 |
| @yoctozepto:matrix.org | unlike docker api | 16:48 |
| @yoctozepto:matrix.org | I mean that many people using skopeo will be doing so in tandem with the other oci tools, like podman | 16:49 |
| @clarkb:matrix.org | isn't it the same API for pushing and pulling images? | 16:49 |
| @yoctozepto:matrix.org | no | 16:50 |
| @yoctozepto:matrix.org | the docker api is for its internal image cache/store | 16:50 |
| @clarkb:matrix.org | I see. odd that they would invent two apis that do the same thing though | 16:51 |
| @clarkb:matrix.org | Not that I am surprised | 16:51 |
| @yoctozepto:matrix.org | almost the same thing, it has more operations I think | 16:51 |
| @yoctozepto:matrix.org | the problem is it has this microversioning now and whatnot | 16:52 |
| @yoctozepto:matrix.org | anyhow, good that the real blast radius is very minimal | 16:52 |
| -@gerrit:opendev.org- Clark Boylan proposed on behalf of Radosław Piliszek: | 17:02 | |
| - [zuul/zuul-jobs] 913902: Reenable buildset-registry jobs https://review.opendev.org/c/zuul/zuul-jobs/+/913902 | ||
| - [zuul/zuul-jobs] 913905: Reenable crio jobs https://review.opendev.org/c/zuul/zuul-jobs/+/913905 | ||
| @clarkb:matrix.org | I think that may actually have a chance at succeeding | 17:02 |
| @clarkb:matrix.org | and if so we probably want to refactor a bit to pull out the ensure-skopeo support for debian effort into its own change | 17:02 |
| @clarkb:matrix.org | but I'll defer to reviewers on all that | 17:03 |
| -@gerrit:opendev.org- Simon Westphahl proposed: [zuul/zuul] 913914: Don't reset buildset when cycle dependency merged https://review.opendev.org/c/zuul/zuul/+/913914 | 17:08 | |
| @yoctozepto:matrix.org | Clark: two little comments from me | 17:24 |
| -@gerrit:opendev.org- Clark Boylan proposed on behalf of Radosław Piliszek: | 17:30 | |
| - [zuul/zuul-jobs] 913902: Reenable buildset-registry jobs https://review.opendev.org/c/zuul/zuul-jobs/+/913902 | ||
| - [zuul/zuul-jobs] 913905: Reenable crio jobs https://review.opendev.org/c/zuul/zuul-jobs/+/913905 | ||
| @yoctozepto:matrix.org | Another little comment on the last one. That is how much I see on the little screen of my phone. I will have a deeper look later on. | 17:36 |
| @yoctozepto:matrix.org | have you figured out what the crio repos need? | 19:00 |
| @clarkb:matrix.org | No haven't looked yet | 19:02 |
| @yoctozepto:matrix.org | ok, let me see then | 19:03 |
| @yoctozepto:matrix.org | https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/ensure-kubernetes/tasks/main.yaml | 19:04 |
| @yoctozepto:matrix.org | well, it explicitly asks for xenial huh | 19:04 |
| @yoctozepto:matrix.org | https://kubernetes.io/blog/2023/08/31/legacy-package-repository-deprecation/#can-i-continue-to-use-the-legacy-package-repositories | 19:06 |
| @yoctozepto:matrix.org | UPDATE: The legacy packages are expected to go away by the end of February 2024. | 19:06 |
| @yoctozepto:matrix.org | oh well! | 19:06 |
| @yoctozepto:matrix.org | hmm | 19:06 |
| @yoctozepto:matrix.org | I do wonder why it was adding those | 19:07 |
| @yoctozepto:matrix.org | as it tries to do minikube | 19:07 |
| @yoctozepto:matrix.org | probably for kubectl | 19:07 |
| @yoctozepto:matrix.org | hmm | 19:07 |
| @yoctozepto:matrix.org | I know just the fix | 19:08 |
| -@gerrit:opendev.org- Radosław Piliszek proposed: [zuul/zuul-jobs] 913905: Reenable crio jobs https://review.opendev.org/c/zuul/zuul-jobs/+/913905 | 19:16 | |
| @yoctozepto:matrix.org | let's see this | 19:17 |
| @jim:acmegating.com | Clark: https://review.opendev.org/913917 to upgrade the executor skopeo is ready; i reckon we can merge it today and let the opendev executors start running it this weekend? | 19:21 |
| @jim:acmegating.com | Clark: oh i see you changed to 1.14.2 in the zuul-jobs change; should i match that in the dockerfile? | 19:23 |
| @clarkb:matrix.org | corvus: 1.14.2 was necessary because 1.15.0 actually requires golang 1.20. I think you can use 1.15.0 if you have golang 1.20 but keeping them in sync may be nice. I also had to change the make commands to run a make install in order to install the default policy.json which is now required | 19:24 |
| @clarkb:matrix.org | I think that the docker change may work until we run skopeo then it will explode. Also need to copy /etc/containers/policy.json or whatever the file is to the other host | 19:25 |
| @clarkb:matrix.org | and maybe into the bwrap env? | 19:25 |
| @jim:acmegating.com | oh yeah that.... how do we determine that works? can we just do a skopeo copy from a registry? | 19:25 |
| @jim:acmegating.com | (the extent of manual testing i have done is "skopeo --version") | 19:26 |
| @jim:acmegating.com | i don't think we need to worry about bwrap because i think we bind mount all that in, but i'll double check | 19:26 |
| @clarkb:matrix.org | corvus: I think so it was the registry job that tripped on it | 19:26 |
| @jim:acmegating.com | okay, i'll test my current build to see if it breaks, then change to 1.14 copy missing files and test again | 19:27 |
| @clarkb:matrix.org | sounds good | 19:27 |
| @jim:acmegating.com | (also, back burner item: we might want to think about some validation tests for the image since it's getting more complex than is represented in the unit tests) | 19:28 |
| @yoctozepto:matrix.org | crio is very happy now | 19:28 |
| @jim:acmegating.com | btw i guess we can abandon https://review.opendev.org/907100 and https://review.opendev.org/906916 | 19:34 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 913917: Build a new skopeo for the zuul-executor container image https://review.opendev.org/c/zuul/zuul/+/913917 | 19:48 | |
| @jim:acmegating.com | i built that locally and ran: `zuul-bwrap /tmp skopeo copy docker://quay.io/zuul-ci/zuul:latest oci:test` and it worked | 19:49 |
| @jim:acmegating.com | (and i confirmed the previous version did not for several reasons all of which you can see in the inter-patchset diff) | 19:49 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 913931: Add container image sanity checks https://review.opendev.org/c/zuul/zuul/+/913931 | 20:01 | |
| @jim:acmegating.com | ^ is an attempt to automate that | 20:01 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed on behalf of Simon Westphahl: [zuul/zuul] 913914: Don't reset buildset when cycle dependency merged https://review.opendev.org/c/zuul/zuul/+/913914 | 20:55 | |
| @yoctozepto:matrix.org | ooh, I found an issue in the reenabler change | 21:03 |
| @yoctozepto:matrix.org | it is by chance that the microk8s job work - the minikube path is not prepared for non-ubuntu at all | 21:04 |
| @yoctozepto:matrix.org | so that override from ubuntu to debian should not have been made | 21:04 |
| @yoctozepto:matrix.org | I think let this gate but i will undo this in the last followup | 21:05 |
| @yoctozepto:matrix.org | for consistency | 21:05 |
| @jim:acmegating.com | yoctozepto: oh i just yanked my +w from 902; feel free to update the stack | 21:05 |
| @yoctozepto:matrix.org | oh | 21:06 |
| @yoctozepto:matrix.org | ok | 21:06 |
| @jim:acmegating.com | (i was already looking into the crio failure and noticing the lack of a crio-debian file there) | 21:06 |
| -@gerrit:opendev.org- Radosław Piliszek proposed: [zuul/zuul-jobs] 913902: Reenable buildset-registry jobs https://review.opendev.org/c/zuul/zuul-jobs/+/913902 | 21:08 | |
| @fungicide:matrix.org | whoops, thanks for catching that! | 21:08 |
| -@gerrit:opendev.org- Radosław Piliszek proposed: [zuul/zuul-jobs] 913902: Reenable buildset-registry jobs https://review.opendev.org/c/zuul/zuul-jobs/+/913902 | 21:09 | |
| -@gerrit:opendev.org- Radosław Piliszek proposed: [zuul/zuul-jobs] 913905: Reenable crio jobs https://review.opendev.org/c/zuul/zuul-jobs/+/913905 | 21:09 | |
| -@gerrit:opendev.org- Radosław Piliszek proposed: [zuul/zuul-jobs] 913905: Reenable crio jobs https://review.opendev.org/c/zuul/zuul-jobs/+/913905 | 21:09 | |
| @yoctozepto:matrix.org | it should be fine now | 21:10 |
| @yoctozepto:matrix.org | let's wait for tests | 21:10 |
| @yoctozepto:matrix.org | nonetheless, I call it a day now | 21:11 |
| @yoctozepto:matrix.org | have a good rest of the day and good night sleep | 21:11 |
| @clarkb:matrix.org | fwiw I think the ubnutu based job will fail | 21:12 |
| @clarkb:matrix.org | because skopeo on ubuntu is 1.13.3 | 21:13 |
| @clarkb:matrix.org | but we can wait and see | 21:13 |
| @clarkb:matrix.org | maybe we just drop the job in that case | 21:13 |
| @vlotorev:matrix.org | Hi, i'm running Zuul 9.2 and want to upgrade to 9.5. | 22:37 |
| Zuul 9.3 release note notifies that there is SQL db migration. | ||
| Should I upgrade 9.2 -> 9.3 -> 9.5, or 9.2 -> 9.5 is acceptable? | ||
| @vlotorev:matrix.org | * Hi, i'm running Zuul 9.2 and want to upgrade to 9.5. | 22:38 |
| Zuul 9.3 release note notifies that there is SQL db migration required. | ||
| Should I upgrade 9.2 -> 9.3 -> 9.5, or 9.2 -> 9.5 is acceptable? | ||
| @jim:acmegating.com | vlotorev: i'm not in a position to recommend an upgrade procedure for your circumstances, but if you're asking whether the sql migration in 9.3 will run when you start 9.5, the answer is yes. | 22:49 |
| @clarkb:matrix.org | I think the intention is that you should only have to stop at major release boundaries. However, stepping through can't hurt. The db migration should run either way though? Didn't one of those versions require a zk data clear too? | 22:49 |
| @clarkb:matrix.org | Or no that was 10 | 22:50 |
| @jim:acmegating.com | yes that's in the notes for 10 | 22:50 |
| @jim:acmegating.com | * vlotorev: i'm not in a position to recommend an upgrade procedure for your circumstances, but if you're asking whether the sql migration in 9.3 will run when version 9.5 is started, the answer is yes. | 22:50 |
| @jim:acmegating.com | fwiw, i typically don't recommend to acme gating clients to step through versions when upgrading. in general it should be fine to jump within or up one major version, if more than that, clear the zk state. upgrade notes control though, and can override that (as in the case of 10 which requires a state clear in all cases). | 22:56 |
| @vlotorev:matrix.org | Thanks. | 23:00 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: | 23:57 | |
| - [zuul/zuul] 913727: Record merger operations https://review.opendev.org/c/zuul/zuul/+/913727 | ||
| - [zuul/zuul] 913938: Store a repo state file in the log directory https://review.opendev.org/c/zuul/zuul/+/913938 | ||
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed on behalf of Tobias Henkel: [zuul/zuul] 785562: WIP: Save the repo state in the job log dir https://review.opendev.org/c/zuul/zuul/+/785562 | 23:58 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!