| @jim:acmegating.com | Clark: continuing a conversation from #_oftc_#opendev:matrix.org ^ that is a small behavior change, but i think the behavior is within the documented specifications of the role, and i don't think it should affect any expected use of the role. i think it's a good change because it makes the two code paths more consistent. but i acknowledge that someone could be doing something unexpected and it might break them. | 00:04 |
|---|---|---|
| @jim:acmegating.com | Clark: i see two options: 1) accept that change and if it breaks someone, tell them they should explicitly make any necessary changes after the workspace mirror finishes mirroring the workspace. or 2) send an announcement to zuul-announce and give people a 2 week notice that we're going to make a change to a role that should have no effect if they're using the role as documented. | 00:06 |
| @clarkb:matrix.org | yes I think in opendev's case we would only be impacted if the ansible git module here: https://opendev.org/opendev/base-jobs/src/branch/master/playbooks/infra-prod/setup-src.yaml#L27-L53 is unwilling to reset the origin value | 00:06 |
| @jim:acmegating.com | strictly speaking, i don't think it should have to set the origin... | 00:07 |
| @clarkb:matrix.org | I agree, but it appears to have done so just looking at the repos (since they have origins set and they aren't null) | 00:07 |
| @jim:acmegating.com | (could be a fetch && reset) | 00:07 |
| @clarkb:matrix.org | whcih also implies that it has previously reset those values and is fine | 00:07 |
| @clarkb:matrix.org | I guess to me I would lean toawrds not changing the behavior in this change because this change fixes an error | 00:08 |
| @clarkb:matrix.org | adding additional behavior changes just makes the number of variables higher | 00:08 |
| @jim:acmegating.com | okay; how do we improve it? | 00:08 |
| @clarkb:matrix.org | and we can change that in a followup (though maybt not necessarily 2 weeks from now) | 00:08 |
| @jim:acmegating.com | i'm happy to split the change; i just want to know how we decide to merge the second one. :) | 00:09 |
| @clarkb:matrix.org | my preference would be to try and fix the immediate issue as carefully as possible then we can land the other thing and pay attention to it in opendev | 00:11 |
| @clarkb:matrix.org | if nothing breaks for opendev it'll probably be fine to leave as is. If opendev breaks maybe we revert then do the 2 week notice | 00:11 |
| @jim:acmegating.com | so merge one today and one tomorrow? | 00:11 |
| @clarkb:matrix.org | ya I think that works | 00:11 |
| @jim:acmegating.com | cool, wfm. thanks | 00:12 |
| @jim:acmegating.com | easy to revert the second one too | 00:12 |
| @clarkb:matrix.org | altneratively we could put opendev into a soft shutdown state right now and do everything tomorrow. But then anyone else affected by this issue will have to wait longer | 00:12 |
| @jim:acmegating.com | barely-related: i still don't know why test-prepare-workspace-git is different than the normal one | 00:13 |
| @clarkb:matrix.org | ya I don't know either but I guess I can pull up the code and look at it more closely | 00:13 |
| @jim:acmegating.com | they are supposed to be identical, unless someone is making a change they want to test out in base-test | 00:13 |
| @jim:acmegating.com | maybe be72e626e88fd1e19d475d95c0fe2f562575a0b5 | 00:14 |
| @jim:acmegating.com | looks like that added stuff to prod but not test | 00:14 |
| @clarkb:matrix.org | ya it looks like the stuff in sync-project.yaml matches what is in test-prepare-workspace-git | 00:15 |
| @clarkb:matrix.org | its just not split out into a separate task file in test-prepare-workspace-git | 00:15 |
| @jim:acmegating.com | okay i'll resync it as part of change #1 | 00:16 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul-jobs] 924802: Fix prepare-workspace-git operating on existing repos https://review.opendev.org/c/zuul/zuul-jobs/+/924802 | 00:18 | |
| @clarkb:matrix.org | +2 from me. I've got to help sort out dinner soon but can keep half an eye on it this evening. Also I think if we prefer we can just disable the jobs in opendev and land that tomorrow | 00:21 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul-jobs] 924804: Make prepare-workspace-git behavior more consistent https://review.opendev.org/c/zuul/zuul-jobs/+/924804 | 00:21 | |
| @jim:acmegating.com | thanks, i'll go ahead and approve it, and check the opendev job after dinner | 00:22 |
| @clarkb:matrix.org | fungi: if you are still around fyi on that^ | 00:23 |
| @clarkb:matrix.org | corvus: opendev will have an opendev-prod-hourly run starting at roughly 01:00 UTC in ~36 minutes and then separately the daily periodic jobs that run at 02:00 in about an hour and a half. The hourly jobs are much smaller in scope but the daily jobs hit everything. | 00:25 |
| @clarkb:matrix.org | I guess it would be good to sanity check things at the 01:00 point and if things don't look good decide if opendev needs to disable things before the 02:00 runs | 00:25 |
| @clarkb:matrix.org | I can also push and approve another trivial update for nodepool config if we like? | 00:26 |
| @clarkb:matrix.org | https://review.opendev.org/c/openstack/project-config/+/924806 pushed that as an option | 00:28 |
| @clarkb:matrix.org | I'll go ahead and approve ^ after the fixup change lands | 00:28 |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul-jobs] 924802: Fix prepare-workspace-git operating on existing repos https://review.opendev.org/c/zuul/zuul-jobs/+/924802 | 00:32 | |
| @clarkb:matrix.org | jobs with single use nodes have used the updated role successfully. Still waiting for 924806 to land to check things in the repo reuse case | 00:41 |
| @fungicide:matrix.org | i'm in and out, but looks like it merged now | 00:44 |
| @clarkb:matrix.org | Yes the fix merged. Waiting on other things to happen to confirm or deny if things are happy again | 00:47 |
| @clarkb:matrix.org | a job has started and got past the point of previous failure. Checking the git log in that repo directly shows the right commit is HEAD. This is looking good so far | 00:58 |
| @clarkb:matrix.org | this is the non periodic case though so we'll want to check the 01:00 jobs after the run in a few minutes as well | 00:58 |
| @clarkb:matrix.org | corvus: I'm happy with the results on nl01 it updated teh config as expected. Which means if the periodic jobs are also happy then I think we're good for the immediate future | 01:05 |
| @jim:acmegating.com | periodic nodepool is running now | 01:12 |
| @clarkb:matrix.org | ya I am going to check the config file on nl01 doesn't change (it shouldn't) and then we're probably good | 01:12 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 924807: WIP: add configure-projects https://review.opendev.org/c/zuul/zuul/+/924807 | 01:14 | |
| @jim:acmegating.com | https://zuul.opendev.org/t/openstack/build/8cf1a6dccab047dda3aec5e82d7c2cb8 finished successfully | 01:15 |
| @clarkb:matrix.org | corvus: I'm assuming that is the nodepool job. I confirm that the config file didn't change and has the latest content so we're applying valid git repos in the periodic jobs too I think | 01:16 |
| @jim:acmegating.com | yep | 01:16 |
| @clarkb:matrix.org | anyway tldr is that this looks much better. Tomorrow we can land the remote update change and then reconfirm these jobs are happy still | 01:16 |
| @clarkb:matrix.org | if not then we probably should consider a revert and announcement for a planned change, but we can cross the bridge when we get there | 01:16 |
| @jim:acmegating.com | https://zuul.opendev.org/t/openstack/build/8cf1a6dccab047dda3aec5e82d7c2cb8/console#1/1/6/bridge01.opendev.org | 01:16 |
| @jim:acmegating.com | > <@jim:acmegating.com> https://zuul.opendev.org/t/openstack/build/8cf1a6dccab047dda3aec5e82d7c2cb8/console#1/1/6/bridge01.opendev.org | 01:17 |
| that's the updated task in that build | ||
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: | 02:09 | |
| - [zuul/zuul] 923329: Minor improvements to pipeline manager https://review.opendev.org/c/zuul/zuul/+/923329 | ||
| - [zuul/zuul] 923331: Add max-changes-per-pipeline tenant limit https://review.opendev.org/c/zuul/zuul/+/923331 | ||
| -@gerrit:opendev.org- yatin proposed: [zuul/zuul-jobs] 924815: Fix wheel_mirror for Debian https://review.opendev.org/c/zuul/zuul-jobs/+/924815 | 05:01 | |
| -@gerrit:opendev.org- Dong Zhang proposed on behalf of Benedikt Löffler: [zuul/nodepool] 922014: Relax image status check https://review.opendev.org/c/zuul/nodepool/+/922014 | 06:22 | |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 924189: Publish zuul-launcher image https://review.opendev.org/c/zuul/zuul/+/924189 | 06:45 | |
| -@gerrit:opendev.org- Benjamin Schanzel proposed on behalf of Simon Westphahl: [zuul/zuul-jobs] 922910: Add ensure-dib role https://review.opendev.org/c/zuul/zuul-jobs/+/922910 | 07:27 | |
| -@gerrit:opendev.org- Benjamin Schanzel proposed wip on behalf of Simon Westphahl: [zuul/zuul-jobs] 922911: wip: Add build-diskimage role https://review.opendev.org/c/zuul/zuul-jobs/+/922911 | 07:27 | |
| -@gerrit:opendev.org- Benjamin Schanzel proposed on behalf of Simon Westphahl: [zuul/zuul-jobs] 922910: Add ensure-dib role https://review.opendev.org/c/zuul/zuul-jobs/+/922910 | 07:48 | |
| -@gerrit:opendev.org- Benjamin Schanzel proposed wip on behalf of Simon Westphahl: [zuul/zuul-jobs] 922911: wip: Add build-diskimage role https://review.opendev.org/c/zuul/zuul-jobs/+/922911 | 07:48 | |
| -@gerrit:opendev.org- Benjamin Schanzel proposed on behalf of Simon Westphahl: [zuul/zuul-jobs] 922910: Add ensure-dib role https://review.opendev.org/c/zuul/zuul-jobs/+/922910 | 07:50 | |
| -@gerrit:opendev.org- Benjamin Schanzel proposed wip on behalf of Simon Westphahl: [zuul/zuul-jobs] 922911: wip: Add build-diskimage role https://review.opendev.org/c/zuul/zuul-jobs/+/922911 | 07:50 | |
| -@gerrit:opendev.org- Benjamin Schanzel proposed wip on behalf of Simon Westphahl: [zuul/zuul-jobs] 922911: wip: Add build-diskimage role https://review.opendev.org/c/zuul/zuul-jobs/+/922911 | 08:09 | |
| -@gerrit:opendev.org- Benjamin Schanzel proposed wip on behalf of Simon Westphahl: [zuul/zuul-jobs] 922911: wip: Add build-diskimage role https://review.opendev.org/c/zuul/zuul-jobs/+/922911 | 08:20 | |
| -@gerrit:opendev.org- Simon Westphahl proposed wip: [zuul/zuul-jobs] 922912: wip: Add example role for converting images https://review.opendev.org/c/zuul/zuul-jobs/+/922912 | 08:43 | |
| -@gerrit:opendev.org- Benjamin Schanzel proposed wip on behalf of Simon Westphahl: [zuul/zuul-jobs] 922912: wip: Add example role for converting images https://review.opendev.org/c/zuul/zuul-jobs/+/922912 | 09:29 | |
| -@gerrit:opendev.org- Simon Westphahl proposed: [zuul/zuul] 924695: Deprecate and cleanup use of status-url https://review.opendev.org/c/zuul/zuul/+/924695 | 10:03 | |
| -@gerrit:opendev.org- Simon Westphahl proposed: [zuul/zuul] 924695: Deprecate and cleanup use of status-url https://review.opendev.org/c/zuul/zuul/+/924695 | 10:05 | |
| -@gerrit:opendev.org- Simon Westphahl marked as active: | 10:13 | |
| - [zuul/zuul-jobs] 922912: wip: Add example role for converting images https://review.opendev.org/c/zuul/zuul-jobs/+/922912 | ||
| - [zuul/zuul-jobs] 922911: wip: Add build-diskimage role https://review.opendev.org/c/zuul/zuul-jobs/+/922911 | ||
| -@gerrit:opendev.org- Jan Gutter proposed: | 10:35 | |
| - [zuul/zuul-jobs] 924750: Update sources for cri-o for newer Ubuntu https://review.opendev.org/c/zuul/zuul-jobs/+/924750 | ||
| - [zuul/zuul-jobs] 924738: Move minikube out of /tmp https://review.opendev.org/c/zuul/zuul-jobs/+/924738 | ||
| -@gerrit:opendev.org- Benjamin Schanzel proposed on behalf of Simon Westphahl: | 10:36 | |
| - [zuul/zuul-jobs] 922911: Add build-diskimage role https://review.opendev.org/c/zuul/zuul-jobs/+/922911 | ||
| - [zuul/zuul-jobs] 922912: Add a role to convert diskimages between formats https://review.opendev.org/c/zuul/zuul-jobs/+/922912 | ||
| -@gerrit:opendev.org- Jan Gutter proposed: | 10:56 | |
| - [zuul/zuul-jobs] 924750: Update sources for cri-o for newer Ubuntu https://review.opendev.org/c/zuul/zuul-jobs/+/924750 | ||
| - [zuul/zuul-jobs] 924738: Move minikube out of /tmp https://review.opendev.org/c/zuul/zuul-jobs/+/924738 | ||
| -@gerrit:opendev.org- Zuul merged on behalf of yatin: [zuul/zuul-jobs] 924815: Fix wheel_mirror for Debian https://review.opendev.org/c/zuul/zuul-jobs/+/924815 | 12:16 | |
| @jangutter:matrix.org | Hi folks, from some of the above churn, you might have noticed I've been "kicking the tires" of `zuul-operator` and the `ensure-kubernetes` role in zuul-jobs. If I started sending up proposed fixes would anyone be amenable to review? | 15:27 |
| I got things working pretty reliably on CentOS Stream 9 (we need to host on it), and in trying to send up the fixes, found out that there's been a few regressions. | ||
| From what it looks like: | ||
| * ensure-kubernetes has regressed over the weekend, cri-o on Ubuntu moved to a new place, but you need Ubuntu Jammy + in order to use it. I can take a stab at it, but I would strongly recommend dropping pre-Jammy distros for k8s workloads... | ||
| * there's a couple of old references (docker.io needs to move to quay.io) | ||
| * the db operator and zookeeper need updates. | ||
| Only the first one I haven't figured out yet, but if folks are willing to review, I'm willing to take a stab at a fix. | ||
| -@gerrit:opendev.org- Jan Gutter proposed: [zuul/zuul-jobs] 924738: Move minikube out of /tmp https://review.opendev.org/c/zuul/zuul-jobs/+/924738 | 15:39 | |
| @jim:acmegating.com | jangutter: thanks... i believe all operator progress is blocked on the test infrastructure issues that are blocking https://review.opendev.org/881245 -- perhaps your fixes to the minikube/crio stuff will fix that; if so, feel free to update 881245 with a depends-on to your changes, and if the tests work, then we can get all the operator fixes/updates merged. | 15:49 |
| -@gerrit:opendev.org- Jan Gutter proposed: [zuul/zuul-operator] 924748: Minor fixes and dependency updates https://review.opendev.org/c/zuul/zuul-operator/+/924748 | 15:53 | |
| @jangutter:matrix.org | > <@jim:acmegating.com> jangutter: thanks... i believe all operator progress is blocked on the test infrastructure issues that are blocking https://review.opendev.org/881245 -- perhaps your fixes to the minikube/crio stuff will fix that; if so, feel free to update 881245 with a depends-on to your changes, and if the tests work, then we can get all the operator fixes/updates merged. | 15:54 |
| hah, just noticed that the opensuse cert got renewed, so testing a small fix, but a better plan is definitely on the cards. | ||
| @jangutter:matrix.org | And the CI is working once again... the change looks big because the crd's got updated: 924748: Minor fixes and dependency updates | https://review.opendev.org/c/zuul/zuul-operator/+/924748 | 16:39 |
| @jim:acmegating.com | jangutter: that appears to be using dockerhub; will your fixes correct the issue that is blocking 881245? | 16:40 |
| @jangutter:matrix.org | Yeah, it passes on dockerhub now I think... I tested on quay.io locally and it seems to work... | 16:42 |
| @jangutter:matrix.org | So I can put a change on top of that for the prefixes, I haven't looked at the publishing reviews though. | 16:43 |
| @jim:acmegating.com | jangutter: 1 sec | 16:44 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul-operator] 924870: Publish container images to quay.io https://review.opendev.org/c/zuul/zuul-operator/+/924870 | 16:53 | |
| @jim:acmegating.com | jangutter: ^ that's an alternate version of 881245 that i rebased on your change; maybe you can take it over and do whatever's necessary to make it work (assuming it doesn't just work)? | 16:54 |
| @jim:acmegating.com | the last person that was working on this had a depends-on to Depends-On: https://review.opendev.org/c/zuul/zuul-jobs/+/894755/ -- i have no idea whether or not that's necessary | 16:56 |
| @jim:acmegating.com | but you can see the history in 881245 and related changes | 16:56 |
| @jangutter:matrix.org | I just took a look at that yeah: looks like it combines a few separate fixes (typically the case when unblocking a gate).... | 16:59 |
| I can split off the changes to just set use quay.io, I'll see what I can update. | ||
| I think the dependency is not required now (the abandoned project is still working), but it means we can update and test that separately. | ||
| @jim:acmegating.com | sounds good; i am not at all particular about how many changes (1 vs N) it takes or the order :) | 17:00 |
| @jamesharrisg:matrix.org | Fun question when looking at 'add_host' and ansible. does the set of hosts get reset per playbook call? i.e. if II call add_host in a pre_run playbook would I need to call it again in a run playbook or post_run playbook? | 17:22 |
| Most of my playbooks use `host: all` and I do not want those playbooks to run on hosts added with `add_host` | ||
| @jangutter:matrix.org | As far as I know, add_host does not leave the playbook (you'd need inventory manipulation, and I don't recall Zuul doing that) | 17:22 |
| @jamesharrisg:matrix.org | YAY! that makes me very happy indeed. | 17:23 |
| @jamesharrisg:matrix.org | thanks for the quick response. | 17:24 |
| @jangutter:matrix.org | had that one in my cache. | 17:24 |
| @mordred:waterwanders.com | corvus: I think something went sideways with 924870 | 17:33 |
| @jangutter:matrix.org | Yeah, almost done with fixing up the references, I'm splitting out the docker.io -> quay.io into a single review. | 17:35 |
| @mordred:waterwanders.com | oh - hah - I should read the _entire_ conversation here I suppose :) | 17:41 |
| -@gerrit:opendev.org- Jan Gutter proposed: [zuul/zuul-operator] 924872: Change references from docker.io to quay.io https://review.opendev.org/c/zuul/zuul-operator/+/924872 | 17:42 | |
| @jangutter:matrix.org | I still need to update the zuul job for ^, publish job still goes to dockerhub. | 17:46 |
| -@gerrit:opendev.org- Jan Gutter proposed: [zuul/zuul-operator] 924872: Change references from docker.io to quay.io https://review.opendev.org/c/zuul/zuul-operator/+/924872 | 18:51 | |
| @jangutter:matrix.org | Oh man, every now and then I find some "magic" code that makes things so much easier. For example the intermediate registry code... | 18:52 |
| @jangutter:matrix.org | I'll take a look at it tomorrow, but I think I know why the docker.io->quay.io has problems. I don't think minikube pulls correctly from the intermediate registry (in this case). | 19:20 |
| @clarkb:matrix.org | corvus: trying to catch up on stuff now that gitea is done. have we merged the git remote behavior chagne for prepare-workspace-git yet? | 19:48 |
| @clarkb:matrix.org | if not I think we can probably do that whenever you are ready and then check opendev's next hourly periodic runs | 19:49 |
| @jim:acmegating.com | Clark: wfm, i'll +3 | 19:55 |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul-jobs] 924804: Make prepare-workspace-git behavior more consistent https://review.opendev.org/c/zuul/zuul-jobs/+/924804 | 20:04 | |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 924792: Unify provider validation and deserialization schemas https://review.opendev.org/c/zuul/zuul/+/924792 | 22:21 | |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 924792: Unify provider validation and deserialization schemas https://review.opendev.org/c/zuul/zuul/+/924792 | 23:18 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!