| -@gerrit:opendev.org- Dr. Jens Harbott proposed: [openstack/project-config] 994392: Drop x/neutron-interconnection from zuul https://review.opendev.org/c/openstack/project-config/+/994392 | 05:42 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Dr. Jens Harbott: [openstack/project-config] 994392: Drop x/neutron-interconnection from zuul https://review.opendev.org/c/openstack/project-config/+/994392 | 10:51 | |
| @harbott.osism.tech:regio.chat | infra-root: I remember some of you had mixed feelings about formatting tools, you may want to look at https://review.opendev.org/c/openstack/project-config/+/993629 | 13:32 |
|---|---|---|
| @harbott.osism.tech:regio.chat | https://review.opendev.org/c/openstack/project-config/+/903666 could also do with another review, cleanup of retired repos | 16:05 |
| -@gerrit:opendev.org- Zuul merged on behalf of James Page: [openstack/project-config] 903666: sunbeam: retire all single charm repositories https://review.opendev.org/c/openstack/project-config/+/903666 | 16:23 | |
| @mordred:waterwanders.com | corvus: if you get a sec, https://review.opendev.org/c/zuul/zuul-jobs/+/990826 and https://review.opendev.org/c/opendev/base-jobs/+/990828 are ready and are shown to work with https://review.opendev.org/c/drizzle/drizzle/+/990830 ... there's also a followup to the base-jobs one that I'd love to get your take on | 17:36 |
| @jim:acmegating.com | mordred: ack, i'm going to need a bit of headspace for that, but i'll look later today, thanks! | 17:42 |
| @mordred:waterwanders.com | corvus: thanks! no rush of course | 17:43 |
| @mnasiadka:matrix.org | Clark, corvus : I added some more tests as requested in https://review.opendev.org/c/opendev/system-config/+/980994 - but it seems iptables rules take private ipv4 address of prometheus server, is that something known and I just need to change some knob? ;-) | 17:44 |
| @clarkb:matrix.org | mnasiadka: yes it is known let me find a link to the code where the issue is | 17:49 |
| @clarkb:matrix.org | it is intentional for historic reasons | 17:49 |
| @clarkb:matrix.org | mnasiadka: https://review.opendev.org/c/opendev/system-config/+/733409/11/playbooks/zuul/run-base.yaml this is where we introduced that behavior. I more recently pushed https://review.opendev.org/c/opendev/system-config/+/989022/16/playbooks/zuul/run-base.yaml to set actual_public_v4 for when we want to differentiate | 17:51 |
| @clarkb:matrix.org | mnasiadka: where is the prometheus change having trouble with it? | 17:52 |
| @mnasiadka:matrix.org | Clark: https://review.opendev.org/c/opendev/system-config/+/980994, see https://zuul.opendev.org/t/openstack/build/724d762fd90d46539be6a5260e0952f6 | 17:52 |
| @clarkb:matrix.org | got it so a testinfra test case is trying to make a tcp connection from test node A to B to check the ip tables rules and service are functioning. But we're using B's private address in the firewall but testinfra tries to connect to the public address? | 17:54 |
| @clarkb:matrix.org | mnasiadka: and this is where we list out the target ips: https://review.opendev.org/c/opendev/system-config/+/980994/12/playbooks/roles/prometheus/templates/prometheus.yml.j2 | 17:56 |
| @clarkb:matrix.org | mnasiadka: I thought we set ansible_host to the same value | 17:57 |
| @clarkb:matrix.org | oh I think I see the issue maybe. Its that ansible_host is the actual public ipv4 value but the firewall rules must not be? | 17:58 |
| @clarkb:matrix.org | ya we collect the rules v4 and v6 files and they use the private source addr | 17:59 |
| @mnasiadka:matrix.org | there's 10.x ip addres in rules.v4 | 17:59 |
| @mnasiadka:matrix.org | that's how I deduced something is wrong :) | 17:59 |
| @clarkb:matrix.org | https://review.opendev.org/c/opendev/system-config/+/980994/12/inventory/service/group_vars/prometheus-node-exporter.yaml so this is what we're looking at and it uses the group rather than a host as the source so maybe something about the iptables role is not using ansible_host but the v4 address? | 18:00 |
| @clarkb:matrix.org | mnasiadka: one easy solution may be to update https://review.opendev.org/c/opendev/system-config/+/980994/12/playbooks/roles/prometheus/templates/prometheus.yml.j2 to not use ansible_host? | 18:01 |
| @clarkb:matrix.org | (this is one of those things that solving properly is probably worth doing but of unknown pain levels) | 18:02 |
| @clarkb:matrix.org | this is a floating IP cloud too | 18:03 |
| @clarkb:matrix.org | (as an extra variable vs public + private addresses in some clouds) | 18:03 |
| @clarkb:matrix.org | sorry I'm in my 5 hour meeting block for the day so trying to do my best to point in the right direction. This may be fallout from my change as that cloud does not have ipv6 at all so we're replacing the ansible_host value | 18:08 |
| @clarkb:matrix.org | I'll have to think about this a bit more. | 18:08 |
| @mnasiadka:matrix.org | Clark: no worries, just wanted to move that forward before my vacation next week :) | 18:09 |
| @jim:acmegating.com | Clark: mordred in general we try to make self-contained simple roles in zuul-jobs, so that jobs look like a list of JJB-style builders: | 18:12 |
| ``` | ||
| - ensure-docker | ||
| - use-buildset-registry | ||
| - pull-from-intermediate-registry | ||
| ``` | ||
| and it's really easy to see what's going on and re-arrange them, etc, as a job requires. | ||
| when we start having options to certain roles that then include other roles, it starts to get harder to follow. the above list is still correct, but now | ||
| ``` | ||
| - ensure-docker: | ||
| docker_use_buildset_registry: true | ||
| - pull-from-intermediate-registry | ||
| ``` | ||
| is equivalent to the above... moreover, someone could tweak a job variable for the original and get use-buildset-registry run twice. you can no longer look at a playbook and see the steps that the job is taking. | ||
| for that reason, i think we should try to avoid having roles include other roles in zuul-jobs. i wish i could add a review comment to https://review.opendev.org/713115 six years ago to find out why we couldn't just solve the problem in the job level rather than at the role level. | ||
| i do agree that the podman and docker roles should be kept in sync, and since we allowed this in the ensure-docker role, i think we can accept it for ensure-podman. but i'd really like us to consider going the other direction: can we remove the option from the docker role and add the role to jobs instead? | ||
| @clarkb:matrix.org | I believe that opendev does the split out thing with these roles already so removing it from the docker role should be safe for us? | 18:13 |
| @clarkb:matrix.org | mnasiadka: for data gathering/experimentation can we switch https://review.opendev.org/c/opendev/system-config/+/980994/12/playbooks/roles/prometheus/templates/prometheus.yml.j2 line 13 to use ansible_default_ipv4.address ? | 18:15 |
| @jim:acmegating.com | https://codesearch.opendev.org/?q=docker_use_buildset_registry&i=nope&literal=nope&files=&excludeFiles=&repos= yeah, i expect they are used outside of opendev, so we'd do a normal deprecation, etc for that. | 18:15 |
| @clarkb:matrix.org | mnasiadka: alternatively we may need to improve my heuristics in https://review.opendev.org/c/opendev/system-config/+/989022/16/playbooks/zuul/run-base.yaml | 18:16 |
| @clarkb:matrix.org | mnasiadka: actually `public_v4` is what the iptables group rules use in the template. So that is probably the safest one to use there as it will align | 18:18 |
| @mordred:waterwanders.com | [@jim:acmegating.com](https://matrix.to/#/@jim:acmegating.com) agree, I think it would be cleaner and much more understandable to remove from docker instead. Want me to rework it in that direction? | 18:19 |
| @clarkb:matrix.org | mnasiadka: that isn't ideal if we end up with an ipv6 only host we want to have in prometheus but for now that is probably sufficient? Maybe a public_v4 | default(public_v6) or something to handle that case | 18:21 |
| @mordred:waterwanders.com | The stack on top of that one isn't strictly dependent, so I think it'll be easy enough to restack | 18:21 |
| @jim:acmegating.com | mordred: sounds great; hopefully between your change and the opendev jobs, we'll have good examples of how to construct playbooks that way. we can go ahead and ping mnaser to ask if he's still using that for k8s jobs, and if tweaking the playbooks to include use-buildset-registry is not too onerous. | 18:22 |
| @jim:acmegating.com | i'll copypasta my comment above to gerrit for posterity | 18:22 |
| -@gerrit:opendev.org- melanie witt proposed: [zuul/zuul-jobs] 994521: Add optional IPv6 address support to multi-node-bridge https://review.opendev.org/c/zuul/zuul-jobs/+/994521 | 18:33 | |
| @jim:acmegating.com | mordred: rest of that stack lgtm except i left a note on https://review.opendev.org/990850 | 18:34 |
| @jim:acmegating.com | mordred: https://review.opendev.org/990828 lgtm; that's a base-jobs change -- i'm guessing we don't have a test procedure for it; do you think we need any more testing, or should we yolo, recheck, and fast-revert? | 18:39 |
| @mordred:waterwanders.com | corvus: yeah - I mean - it worked for the drizzle change, but otherwise, I kinda feel like yolo/fast-revert is about as good as we get? the biggest risk I can think of is someone somewhere somehow setting container_command to podman but actually using docker and now they dont have docker. but I _think_ I did a codesearch search for that | 18:41 |
| @jim:acmegating.com | or somehow we misunderstand the iptables conditional | 18:45 |
| @jim:acmegating.com | Clark: do you have a suggestion for an opendev image build job we can recheck after that merges? | 18:46 |
| @jim:acmegating.com | maybe https://review.opendev.org/994326 ? | 18:46 |
| @clarkb:matrix.org | corvus: I think the etherpad 3.3.2 or gitea 1.26.4 changes are good candidates yes | 18:46 |
| @jim:acmegating.com | okay yoloing | 18:47 |
| -@gerrit:opendev.org- Monty Taylor https://matrix.to/#/@mordred:inaugust.com proposed: | 18:49 | |
| - [zuul/zuul-jobs] 994148: Remove sphinx version pin https://review.opendev.org/c/zuul/zuul-jobs/+/994148 | ||
| - [zuul/zuul-jobs] 994154: Make installing skopeo from source the default https://review.opendev.org/c/zuul/zuul-jobs/+/994154 | ||
| - [zuul/zuul-jobs] 990850: Add resolute and trixie to zuul-jobs test matrix https://review.opendev.org/c/zuul/zuul-jobs/+/990850 | ||
| @mordred:waterwanders.com | corvus: I pulled the trixie/resolute stack off of the podman change and updated it to use 8G (which really aligns with existing nodes anyway - figuring out different sizes for different jobs does feel like a followup) | 18:50 |
| @mordred:waterwanders.com | (I'll rework the podman into a "remove this from docker" instead) | 18:50 |
| @mordred:waterwanders.com | corvus: while you have things paged into your brain - the child of the base-jobs yolo job https://review.opendev.org/c/opendev/base-jobs/+/994146/ is worth a chat - and I also discovered https://review.opendev.org/c/zuul/zuul-jobs/+/994297 you need nftables for podman-compose :) | 18:52 |
| -@gerrit:opendev.org- Zuul merged on behalf of Monty Taylor https://matrix.to/#/@mordred:inaugust.com: [opendev/base-jobs] 990828: Honor container_command in buildset-registry playbooks https://review.opendev.org/c/opendev/base-jobs/+/990828 | 19:06 | |
| @mordred:waterwanders.com | time for recheck? | 19:08 |
| @jim:acmegating.com | mordred: agree https://review.opendev.org/994146 is worth a chat, left chatty comments | 19:08 |
| @mordred:waterwanders.com | corvus: yay! | 19:09 |
| @mordred:waterwanders.com | I like chats | 19:09 |
| @jim:acmegating.com | rechecked 326 | 19:09 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [opendev/base-jobs] 994538: Revert "Honor container_command in buildset-registry playbooks" https://review.opendev.org/c/opendev/base-jobs/+/994538 | 19:55 | |
| @jim:acmegating.com | I'm self-approving the yolo fast revert ^ | 19:55 |
| @mordred:waterwanders.com | corvus: oh for the love of crap | 19:57 |
| @mordred:waterwanders.com | the error on that is so sadpanda | 19:57 |
| @mordred:waterwanders.com | there had to be _something_ that the previous test path wouldn't catch | 19:57 |
| @mordred:waterwanders.com | corvus: for a followup fix patch - do you want me to split the iptables change into its own change? | 20:01 |
| @jim:acmegating.com | nah, that just sounds lie more work :) | 20:03 |
| @jim:acmegating.com | * nah, that just sounds like more work :) | 20:03 |
| @mordred:waterwanders.com | (the failure was a } intead of a ) in the "set a variable to select docker vs podman" which didn't get triggered when container_command was actually set) | 20:03 |
| -@gerrit:opendev.org- Monty Taylor https://matrix.to/#/@mordred:inaugust.com proposed: [opendev/base-jobs] 994539: Reapply "Honor container_command in buildset-registry playbooks" https://review.opendev.org/c/opendev/base-jobs/+/994539 | 20:04 | |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [opendev/base-jobs] 994538: Revert "Honor container_command in buildset-registry playbooks" https://review.opendev.org/c/opendev/base-jobs/+/994538 | 20:04 | |
| @mordred:waterwanders.com | ```_local_container_command: "{{ container_command | default('docker'} }}"``` vs ```_local_container_command: "{{ container_command | default('docker') }}"``` fwiw | 20:04 |
| -@gerrit:opendev.org- melanie witt proposed wip: [zuul/zuul-jobs] 994521: Add optional IPv6 address support to multi-node-bridge https://review.opendev.org/c/zuul/zuul-jobs/+/994521 | 21:05 | |
| -@gerrit:opendev.org- Clark Boylan proposed: [zuul/zuul-jobs] 994564: Support public ipv4 addresses in /etc/hosts https://review.opendev.org/c/zuul/zuul-jobs/+/994564 | 21:28 | |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/system-config] 994565: Make public ip address public not private https://review.opendev.org/c/opendev/system-config/+/994565 | 21:32 | |
| @clarkb:matrix.org | those two changes are a bit of an experiment. I think that should run most (all?) system-config-run jobs which should give us good coverage. mnasiadka fyi as this is related to the node exporter and prometheus issue | 21:33 |
| -@gerrit:opendev.org- Monty Taylor https://matrix.to/#/@mordred:inaugust.com proposed: [opendev/system-config] 994569: DNM Test updated buildset registry https://review.opendev.org/c/opendev/system-config/+/994569 | 22:41 | |
| @mordred:waterwanders.com | Clark, corvus I feel a little silly - should have just done that ^^ in the docker direction before landing the buildset-registry change. | 22:42 |
| @mordred:waterwanders.com | I've pushed up an updated drizzle change to to exercise the podman path | 22:47 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!