| opendevreview | Merged openstack/project-config master: openstack-afs.yaml : correct indentation https://review.opendev.org/c/openstack/project-config/+/869772 | 00:17 |
|---|---|---|
| opendevreview | Ian Wienand proposed opendev/system-config master: [wip] add variable to block UA's for mailman https://review.opendev.org/c/opendev/system-config/+/869779 | 00:37 |
| opendevreview | Ian Wienand proposed opendev/system-config master: [wip] add variable to block UA's for mailman https://review.opendev.org/c/opendev/system-config/+/869779 | 02:00 |
| opendevreview | Merged openstack/project-config master: Add nb04 config https://review.opendev.org/c/openstack/project-config/+/869769 | 02:41 |
| opendevreview | Ian Wienand proposed opendev/system-config master: [wip] add variable to block UA's for mailman https://review.opendev.org/c/opendev/system-config/+/869779 | 03:15 |
| opendevreview | Ian Wienand proposed opendev/system-config master: [wip] add variable to block UA's for mailman https://review.opendev.org/c/opendev/system-config/+/869779 | 04:12 |
| opendevreview | Ian Wienand proposed opendev/system-config master: mailman: add variable for matching UAs in Apache https://review.opendev.org/c/opendev/system-config/+/869779 | 04:48 |
| *** ysandeep is now known as ysandeep|ruck | 05:12 | |
| *** marios is now known as marios|rover | 06:01 | |
| *** bhagyashris|brb is now known as bhagyashris | 06:39 | |
| *** bhagyashris is now known as bhagyashris|afk | 06:39 | |
| *** ysandeep|ruck is now known as ysandeep|ruck|brb | 06:42 | |
| jexsie | Hello everyone, am new here.Anything for me?? | 06:52 |
| opendevreview | Michael Kelly proposed zuul/zuul-jobs master: prepare-workspace-git: Skip LFS checkout when mirroring repos https://review.opendev.org/c/zuul/zuul-jobs/+/869787 | 07:02 |
| *** ysandeep|ruck|brb is now known as ysandeep|ruck | 07:18 | |
| *** ysandeep|ruck is now known as ysandeep|lunch | 07:22 | |
| *** soniya29 is now known as soniya29|lunch | 08:01 | |
| opendevreview | Michael Kelly proposed zuul/zuul-jobs master: prepare-workspace-git: Skip LFS checkout when mirroring repos https://review.opendev.org/c/zuul/zuul-jobs/+/869787 | 08:15 |
| opendevreview | Michael Kelly proposed zuul/zuul-jobs master: prepare-workspace-git: Skip LFS checkout when mirroring repos https://review.opendev.org/c/zuul/zuul-jobs/+/869787 | 08:20 |
| opendevreview | Michael Kelly proposed zuul/zuul-jobs master: prepare-workspace-git: Skip LFS checkout when mirroring repos https://review.opendev.org/c/zuul/zuul-jobs/+/869787 | 08:24 |
| *** jpena|off is now known as jpena | 08:29 | |
| *** ysandeep|lunch is now known as ysandeep|ruck | 08:29 | |
| *** ysandeep|ruck is now known as ysandeep|ruck|afk | 08:40 | |
| *** soniya29|lunch is now known as soniya29 | 09:18 | |
| *** ysandeep|ruck is now known as ysandeep|ruck|afk | 09:44 | |
| *** ysandeep__ is now known as ysandeep|ruck | 11:07 | |
| *** rlandy|out is now known as rlandy | 11:15 | |
| *** bhagyashris|afk is now known as bhagyashris | 11:32 | |
| *** artom_ is now known as artom | 11:33 | |
| opendevreview | Cedric Jeanneret proposed opendev/system-config master: Correct (again) how ansible-galaxy proxy is configured https://review.opendev.org/c/opendev/system-config/+/869819 | 12:43 |
| amorin | hello clarkb ianw fungi and others, I confirmed we had an issue this night on our object storage: https://public-cloud.status-ovhcloud.com/incidents/sr0y0x7tr88b | 12:47 |
| amorin | the issue is now over, so it's safe to reopen the swift if you want | 12:47 |
| fungi | thanks for confirming it's back up amorin! | 12:49 |
| *** soniya29 is now known as soniya29|afk | 13:04 | |
| opendevreview | Dr. Jens Harbott proposed opendev/base-jobs master: Revert "Disable OVH BHS1 and GRA1 log uploads" https://review.opendev.org/c/opendev/base-jobs/+/869649 | 13:18 |
| fungi | i know we talked about testing that with base-test, but with the updated incident information it's probably unwarranted | 13:22 |
| opendevreview | Merged opendev/base-jobs master: Revert "Disable OVH BHS1 and GRA1 log uploads" https://review.opendev.org/c/opendev/base-jobs/+/869649 | 13:27 |
| *** ysandeep|ruck is now known as ysandeep|ruck|afk | 14:01 | |
| *** ysandeep|ruck|afk is now known as ysandeep|ruck | 14:30 | |
| *** ysandeep|ruck is now known as ysandeep|ruck|afk | 15:08 | |
| *** ysandeep|ruck|afk is now known as ysandeep|out | 16:31 | |
| opendevreview | Cedric Jeanneret proposed opendev/system-config master: Correct (again) how ansible-galaxy proxy is configured https://review.opendev.org/c/opendev/system-config/+/869819 | 16:34 |
| *** marios|rover is now known as marios|out | 16:36 | |
| *** jpena is now known as jpena|off | 17:21 | |
| *** gthiemon1e is now known as gthiemonge | 19:43 | |
| JayF | Heads up: I'm executing on Ironic changes to retire bugfix branches. | 20:54 |
| fungi | thanks for the warning! | 20:57 |
| JayF | completed ironic, moving to ironic-python-agent | 20:57 |
| JayF | IPA complete, moving to ironic-inspector | 21:00 |
| JayF | hmm. I appear to be missing perms for ironic-inspecotr | 21:01 |
| JayF | ! [remote rejected] bugfix/10.2-eol -> bugfix/10.2-eol (prohibited by Gerrit: not permitted: create signed tag) | 21:01 |
| fungi | i'm available to expedite an acl patch, give me a heads up when you push it | 21:02 |
| JayF | https://review.opendev.org/c/openstack/project-config/+/866937/2/gerrit/acls/openstack/ironic-inspector.config | 21:02 |
| JayF | I'm not sure I understand why it's needed | 21:03 |
| JayF | fungi: https://github.com/openstack/project-config/blob/master/gerrit/acls/openstack/ironic-inspector.config#L52 I don't see why I'm disallowed | 21:04 |
| JayF | Am I missing something? I've double checked two or three times | 21:08 |
| JayF | I wonder if I can see the effective ACLs in gerrit ui... | 21:09 |
| fungi | maybe there's a typo in the acl | 21:09 |
| JayF | yeah I checked again, if there's a typo it's evading me, even looking at ironic.config stanzas and ironic-inspector.config stanzas side by side | 21:10 |
| fungi | do you see refs/tags/* when you visit https://review.opendev.org/admin/repos/openstack/ironic-inspector,access | 21:11 |
| JayF | yes | 21:12 |
| JayF | no | 21:12 |
| JayF | I have Reference: refs/heads/bugfix/* | 21:12 |
| JayF | but not tags | 21:12 |
| JayF | even though tags appear to be in the config file | 21:12 |
| fungi | it should show up if you have access... | 21:12 |
| fungi | unfortunately gerrit filters that view to the things your account is granted access to, which makes it hard to use that to check the loaded acl | 21:12 |
| JayF | from ironic.config: | 21:13 |
| JayF | [access "refs/tags/*"] | 21:13 |
| JayF | createSignedTag = group ironic-release | 21:13 |
| JayF | from inspector.config | 21:13 |
| JayF | [access "refs/tags/*"] | 21:13 |
| JayF | createSignedTag = group ironic-release | 21:13 |
| JayF | confirmed in a different font/venue to be identical | 21:13 |
| JayF | WTF | 21:13 |
| fungi | yeah, and we heavily keyword check those acls with a zuul job, so the leeway for typos is pretty narrow anyway | 21:14 |
| JayF | can we confirm via gerrit logs or similar that the acl was read in? | 21:15 |
| fungi | so... your https://review.opendev.org/866937 change theoretically added that permission when it merged on 2022-12-09 | 21:15 |
| JayF | and I just exercised the permissions added for ironic, because I was able ot perform the work I wanted to | 21:15 |
| fungi | and yeah, the installed configuration gets committed to a git repository in gerrit which admins can clone, so i can check it there | 21:15 |
| JayF | I've literally done ironic + ironic-python-agent, nothing broke until ironic-inspector | 21:15 |
| JayF | and AFAICT there is no difference | 21:15 |
| Clark[m] | fungi: it gets committed to the ironic-inspector repo | 21:16 |
| Clark[m] | But under a ref you need your admin account to fetch | 21:16 |
| fungi | ahh, yeah, it's not in All-Projects | 21:17 |
| JayF | I do not follow | 21:17 |
| fungi | refs/meta/config is what contains it | 21:18 |
| JayF | I (still) don't follow (?) (I'm not sure if this is for me to comprehend anyway lol) | 21:21 |
| fungi | confirmed, the last update to the project.config file in the meta/config ref for openstack/ironic-inspector lacks that update. it was last committed Fri Apr 30 15:21:59 2021 +0000 | 21:21 |
| fungi | JayF: you'd need a gerrit admin account to be able to fetch that ref, so no don't worry too much about that part | 21:22 |
| JayF | aha | 21:22 |
| Clark[m] | Did that land around when we upgraded Gerrit? | 21:22 |
| JayF | 2022-12-09 | 21:22 |
| Clark[m] | That may explain it if so | 21:22 |
| Clark[m] | Ya I can't remember the exact day we upgraded but it was early December | 21:22 |
| JayF | but some of the changes in that patch ( https://review.opendev.org/866937 ) did apply; the ironic.config changes | 21:22 |
| fungi | i would expect to see errors in the manage-projects log if this wasn't getting successfully updated | 21:22 |
| Clark[m] | It may have cached that it performed the update and is just skipping it now | 21:23 |
| Clark[m] | (when it hadn't for some reason) | 21:23 |
| fungi | i'm being called away to cook dinner, but can probably resume digging into this in an hour | 21:23 |
| ianw | (we upgraded gerrit on 2022-12-13 .au date) | 21:23 |
| JayF | I'd prefer be able to apply the changes atomically, since they were announced atomically -- if there's any way to get it fixed before my EOD; I'd greatly appreciate it | 21:24 |
| ianw | i have a checkout of the meta/config of ironic-inspector and confirm the same, no commits since april | 21:24 |
| JayF | alternatvely; the commands I need to run are documented and someone with access could run them to allow more troubleshooting time without impacting repo state | 21:24 |
| ianw | just looking at infra-prod-manage-projects to see what might have happened | 21:25 |
| ianw | 22-12-09 we had a number of changes i guess -> https://zuul.opendev.org/t/openstack/builds?job_name=infra-prod-manage-projects&skip=0 | 21:26 |
| ianw | https://zuul.opendev.org/t/openstack/build/03287232227e40eca6f0b1fb5682f8c4 was the job for 866937 | 21:27 |
| ianw | ... unfortunately no logs | 21:27 |
| Clark[m] | There may be logs still on bridge | 21:28 |
| ianw | ... might be on bridge | 21:28 |
| ianw | heh, jinx | 21:28 |
| ianw | it ran at 13:53, must be manage-projects.yaml.log.2022-12-09T13:54:45 | 21:30 |
| ianw | # ls -lh manage-projects.yaml.log.2022-12-09T13:54:45 | 21:30 |
| ianw | -rw-r--r-- 1 root root 0 Dec 9 13:54 manage-projects.yaml.log.2022-12-09T13:54:45 | 21:30 |
| ianw | ... weird ... | 21:30 |
| ianw | oh jeez, they're all 0 sized | 21:31 |
| Clark[m] | Oh ya this one actually logs to zuul because we cleared it as safe iirc | 21:31 |
| Clark[m] | So we may intentionally not be recording bridge logs :( | 21:31 |
| ianw | we should at least tee it or something. anyway, i guess that's a problem for another time :/ | 21:32 |
| Clark[m] | I think we need to inspect the state on review. It stores a json cache file iirc and it should give us a clue for why it didn't update | 21:33 |
| ianw | 2023-01-10 14:57:03,645: manage_projects - INFO - Processing project: openstack/ironic-inspector | 21:34 |
| ianw | 2023-01-10 14:57:03,645: manage_projects - INFO - openstack/ironic-inspector has matching sha, skipping ACLs | 21:34 |
| ianw | the latest log says that about ironic-inspector | 21:34 |
| Clark[m] | Maybe we aren't comparing the correct thing? I didn't expect that | 21:34 |
| clarkb | ianw: reading jeepyb I think it is comparing the sha it generates for the files from project-config/gerrit/acls/openstack/foo.conf against what is in the project cache it maintains locally | 21:36 |
| clarkb | so ya it mustve updated the local project cache locally thinking it had updated things and has since skipped over it despite them actually differing in gerrit | 21:37 |
| ianw | yeah, is that cache in the container? | 21:38 |
| ianw | no, /opt on review | 21:39 |
| ianw | "acl-sha": "ae1bb09c706a3a99d71caf7805b4b58a3a61cd5a4fb1ee74ba6354f6af719326" | 21:40 |
| ianw | ironic-inspector | 21:40 |
| clarkb | the manage-projects command is the one that mounts the jeepyb stuff regular gerrit doesn't | 21:40 |
| clarkb | and it mounts /opt/lib/jeepyb and project.cache is there which has that value in it which matches the project config value | 21:41 |
| clarkb | so ya it thought it had updated things but apparently not | 21:41 |
| ianw | ae1bb09c706a3a99d71caf7805b4b58a3a61cd5a4fb1ee74ba6354f6af719326 ./ironic-inspector.config | 21:41 |
| ianw | well, that confirms what we know -- that manage-projects thinks it has applied ae1bb09... at least | 21:41 |
| JayF | in this direction; the failure scenario is OK: we didn't add credentials; isn't this potentially a dangerous situation in the other direction? | 21:41 |
| clarkb | the easiest thing is to make a noop edit to the config file and have automation try to apply it | 21:42 |
| clarkb | JayF: I mean if gerrit says it succeeded but didn't there isn't much we can do about that | 21:42 |
| ianw | there is some possibility, especially given the zero sized files, that we are somehow missing the exit code and showing success on jobs that maybe aren't | 21:43 |
| opendevreview | Merged opendev/system-config master: Add nb04.opendev.org https://review.opendev.org/c/opendev/system-config/+/869622 | 21:43 |
| clarkb | ianw: I mean within manage-projects I think the only way we record that sha permanently is if gerrit repsonds with success to the push | 21:43 |
| JayF | literally the config update is a git push to a special ref? I guess that makes sense | 21:44 |
| clarkb | JayF: yes | 21:44 |
| clarkb | almost everything in gerrit is git now for better or worse | 21:44 |
| ianw | clarkb: true -- unless something similar in that we're missing the failure of the push due to ... something. new git or something? clutching at straws :) | 21:45 |
| ianw | i'd agree a no-op push we can monitor and see the logs of will be most helpful | 21:45 |
| ianw | annoyingly the nb04 addition i made is probably running every infra-prod job now | 21:45 |
| clarkb | ianw: looks like while we catch an exception from the push the push might also return false to indicate failure which we don't seem tohandle arg | 21:45 |
| clarkb | so the fix here might be to check the return and raise an exception if its false to fall through the existing exception handling which should prevent us from recording the cached sha value | 21:46 |
| clarkb | I'm going to write that patch and we can think it over with something more concrete | 21:47 |
| JayF | clarkb: do you want me specifically to push a noop patch to that file in gerrit? | 21:47 |
| JayF | I was unsure if you wanted a noop change done locally or via review | 21:47 |
| clarkb | JayF: via review | 21:49 |
| clarkb | so that we exercise the whole thing | 21:49 |
| JayF | ack; incoming | 21:49 |
| clarkb | so I think that it may have been done this way to allow for subsets of projects to update and avoid short circuiting | 21:50 |
| opendevreview | Jay Faulkner proposed openstack/project-config master: Noop change to ironic-inspector.config https://review.opendev.org/c/openstack/project-config/+/869872 | 21:51 |
| clarkb | but I think short circuiting is probbaly preferable here | 21:51 |
| opendevreview | Clark Boylan proposed opendev/jeepyb master: Raise and error if acl pushes fail https://review.opendev.org/c/opendev/jeepyb/+/869873 | 21:53 |
| clarkb | that deserves careful review | 21:53 |
| fungi | okay, dinner has been made, consumed, and cleaned up | 22:05 |
| JayF | making a noop change in a repo | 22:07 |
| JayF | that enforces lint on whitespace | 22:07 |
| JayF | any suggestions on what to do, since adding a newline broke lint? | 22:07 |
| clarkb | JayF: reorder the entries or similar | 22:08 |
| JayF | that's not possible, right? they enforce abc order | 22:08 |
| clarkb | I thought it did that on the backend via a normalization pass and not via linting | 22:08 |
| clarkb | I could be wrong about that | 22:08 |
| JayF | linting will not pass if items are not in abc order | 22:08 |
| JayF | I know this from experience | 22:08 |
| fungi | we enforce normalization in the check job | 22:08 |
| * JayF looks to see if project.config has a comment character | 22:09 | |
| fungi | manage-projects allows us to manually specify a repository and force an update i think, if we want to go that route | 22:09 |
| JayF | okay; apparently # should be a comment, i'll try that | 22:09 |
| clarkb | fungi: it allows us to specify a repository but I don't think it has a force option | 22:10 |
| clarkb | we couod manually edit the json file to change the sha | 22:10 |
| fungi | i thought that was implicit when running with specific repositories | 22:10 |
| opendevreview | Jay Faulkner proposed openstack/project-config master: Noop change to ironic-inspector.config https://review.opendev.org/c/openstack/project-config/+/869872 | 22:10 |
| clarkb | but there is something nice about an end to end exercise if we can come up with something to do that | 22:10 |
| fungi | agreed | 22:10 |
| clarkb | fungi: no if you read the code there is no escape hatch for the sha check | 22:10 |
| fungi | ahh | 22:10 |
| ianw | we could delete the cache entry and run it manually too? | 22:11 |
| clarkb | ianw: yes or just change the sha value | 22:11 |
| fungi | yep | 22:11 |
| ianw | should i try that? | 22:12 |
| clarkb | lets do the noop first? | 22:12 |
| ianw | also, the deploy queue doesn't seem to have run for nb04 ... something must have failed | 22:12 |
| clarkb | JayF: I think you can drop the edithashtags entries from the more specific paths | 22:12 |
| clarkb | JayF: as a non noop cleanup option | 22:12 |
| ianw | infra-prod-base https://zuul.opendev.org/t/openstack/build/cb705ecb46b04a36a3fe3a54a85c65ad : FAILURE in 13m 43s | 22:12 |
| clarkb | they are redundant | 22:12 |
| JayF | I would strongly prefer not making actual-changes while troubleshooting is also occurring | 22:13 |
| JayF | I'm pretty sure that right now these configs are relatively similar between ironic projects; so if I was going to clean up inspector I'd want to do it across ironic projects | 22:13 |
| clarkb | sure the comment should be fine too and I've +2'd it | 22:13 |
| clarkb | I'm just pointing out redundancies that can go away if we need something more forceful | 22:13 |
| ianw | ok, just to add to the problems here | 22:14 |
| ianw | The error was: ansible.errors.AnsibleUndefinedVariable: 'ansible.vars.hostvars.HostVarsVars object' has no attribute 'public_v6' | 22:14 |
| clarkb | ianw: did you add a server without ipv6? | 22:14 |
| ianw | graphite02.opendev.org : ok=63 changed=3 unreachable=0 failed=1 skipped=9 rescued=0 ignored=0 | 22:14 |
| ianw | tracing01.opendev.org : ok=63 changed=3 unreachable=0 failed=1 skipped=9 rescued=0 ignored=0 | 22:14 |
| ianw | zk04.opendev.org : ok=63 changed=3 unreachable=0 failed=1 skipped=9 rescued=0 ignored=0 | 22:14 |
| ianw | zk05.opendev.org : ok=63 changed=3 unreachable=0 failed=1 skipped=9 rescued=0 ignored=0 | 22:14 |
| ianw | zk06.opendev.org : ok=63 changed=3 unreachable=0 failed=1 skipped=9 rescued=0 ignored=0 | 22:14 |
| ianw | this is an odd set of servers to fail | 22:15 |
| ianw | clarkb: yes, the new linaro cloud doesn't have ipv6 | 22:15 |
| clarkb | ya so the issue is we use public_v6 to genreate ip tables rules | 22:16 |
| ianw | ahhh | 22:16 |
| ianw | ok, then that does make sense | 22:16 |
| clarkb | and the new nb04 server is going to talk to all of those I think so they want their iptables updated with its ip addr | 22:16 |
| ianw | that's the servers nb04 would talk to | 22:16 |
| clarkb | yes | 22:16 |
| clarkb | we might be able to work through this just with group membership. But maybe better is just ignore public_v6 if it isn't set since there won't be an ipv6 addr to set an iptables rule for | 22:17 |
| clarkb | of course this gets scary because its going to apply to all the things | 22:17 |
| ianw | yeah, but we can't really stub out the ipv6 (ipv6: '') as that will make wrong rules | 22:18 |
| clarkb | {% for addr in groups.get(group.group) | map('extract', hostvars, 'public_v6') -%} <- is the line in question | 22:18 |
| clarkb | can we | somethingtoskipifnotset? | 22:18 |
| clarkb | if we give it an ipv6 addr for another host in the same group that would approximately work (until the hosts change) | 22:19 |
| clarkb | (I hate this idea for the record just talking out loud) | 22:19 |
| ianw | hosts: "{{ (zk_hosts['hosts']|default([])) + [{'port': '2281', 'host': hostvars[item]['public_v6'] | default(hostvars[item]['ansible_host']) }] }}" | 22:20 |
| ianw | this might be setting the zk hosts to the ipv6 addresses in the zk config too | 22:20 |
| fungi | JayF: clarkb: comments in acls are disallowed by our normalizing linter too: https://zuul.opendev.org/t/openstack/build/ca1fc445b3e94ce1bfbd8d212ac48b09 | 22:21 |
| clarkb | ianw: ah yup it is | 22:22 |
| fungi | the original goal was to be able to use file checksums to identify identical acl content for deduplication | 22:22 |
| clarkb | I feel like the linting is being more of a hindrance than a help right now... But removing the redundant hashtagedit lines should work | 22:23 |
| JayF | so literally if we were able to push a noop change; it'd be a bug in your linter | 22:23 |
| clarkb | JayF: no I've called out a noop change | 22:23 |
| clarkb | one that I'm 99% sure I called out on the original changes too | 22:23 |
| JayF | Your change is removing overlapping rules which will have no impact; not exactly the same as a specifically noop change | 22:23 |
| clarkb | but I didn't care neough to -1 for them | 22:24 |
| JayF | especially from the perspective of someone who barely understands these ACLs :) | 22:24 |
| JayF | I don't like pushing changes that I myself don't personally understand why it's OK | 22:24 |
| clarkb | JayF: do you have a link to the original change? | 22:24 |
| JayF | https://review.opendev.org/866937 | 22:24 |
| clarkb | ok this isn't the one I'm thinking of which change was it | 22:25 |
| fungi | which original change? the one adding the git tag permissions or the one adding hashtag permissions? | 22:25 |
| clarkb | fungi: the one adding the hashtag edits | 22:26 |
| fungi | https://review.opendev.org/c/openstack/project-config/+/772427 | 22:26 |
| clarkb | I remember a number of people all going out and doing this together and I commented on a number of changes to not do that redundantly and no one listened ut I wasn't going to argue it | 22:26 |
| ianw | i have to afk for ~ 1 hour ... i don't think infra-prod-base not working will affect manage-projects, if it is we can revert the nb04 addition. otherwise i'll work on the no ipv6 thing | 22:26 |
| JayF | clarkb: I'll point out my name isn't on that change as an author, committer, or reviewer :/ I still would prefer not push a change to one that wasn't done everywhere | 22:27 |
| JayF | and doing the bigger change means I run out of sunlight before completing my bugfix branch retirement changes | 22:27 |
| clarkb | ok, Ive made a suggestion that we believe will get us over the hump. I'm open to other suggestions | 22:28 |
| clarkb | I will not manually retire those branches | 22:28 |
| JayF | so I'll start building that, if that's hte only way out is for me to cleanup those hashtag things; I get to go read up on how acl inheritance works | 22:28 |
| clarkb | other ideas: update the normalization to allow comments and go with the comment change | 22:29 |
| ianw | for the immediate purposes of JayF I guess we could manually push a change to the refs to sync it | 22:30 |
| clarkb | ianw: no I don't think we should do that either | 22:30 |
| JayF | fungi indicated that blocking comments was intentional; so that the sha1 of the file always indicated the same effective contents | 22:30 |
| clarkb | because there may be something speficially broken with ironic-inspector meta config with jeepyb | 22:30 |
| ianw | the debugging of manage-projects will still happen right -- since that looks at the sha of the cache+project config, not what's committed? | 22:30 |
| ianw | (i'm not saying don't debug why manage-proejcts didn't apply) | 22:30 |
| clarkb | ianw: a git push that is a noop is different than a git push update | 22:30 |
| clarkb | ianw: we would end up doing a git push noop | 22:31 |
| clarkb | which might expose issues but it might also not | 22:31 |
| ianw | well we could push our change sufficiently with whitespace etc such that the new push does change it? | 22:31 |
| ianw | or you think if the actual parsed rules don't change, it would still be no-op-ish? | 22:31 |
| clarkb | but that won't change anything to cause a new manage projects push to happen | 22:31 |
| fungi | https://review.opendev.org/Documentation/access-control.html#_ref_permissions "For allowing access, all ALLOW/DENY rules that might apply to a ref are tested until one granting access is found, or until either an "exclusive" rule ends the search, or all rules have been tested." | 22:31 |
| fungi | that's the relevant inheritance information about ref sections, fwiw | 22:32 |
| JayF | thanks; that's what I was looking for. I do not push changes on the word of another person since it's my responsibility if they go kaboom | 22:32 |
| fungi | also just after that, "The rules are ordered from specific ref patterns to general patterns..." | 22:32 |
| fungi | so a more general ref with an allow is hit before the specific one, and basically obviates the latter | 22:33 |
| fungi | er, rather, a more general ref will apply even in the absence of a specific one | 22:34 |
| fungi | for an allow permission | 22:34 |
| fungi | anyway, that section of the docs is pretty thorough | 22:34 |
| ianw | clarkb: yeah ... i agree on getting manage-projects to still run somehow. just trying to think of a way to also get the branch retirement bits JayF wanted done | 22:35 |
| opendevreview | Jay Faulkner proposed openstack/project-config master: Remove redundant editHashtag lines https://review.opendev.org/c/openstack/project-config/+/869878 | 22:35 |
| JayF | I think that should be what clarkb wanted, ty for the docs fungi | 22:36 |
| fungi | +2 | 22:37 |
| fungi | let's get this exercised so we can collect an additional data point. probably it will get the acl up to current, or it will at least give us a better idea of what's going wrong since we should have fresh logs | 22:37 |
| JayF | in a best case scenario; is it likely for that to be landed and applied in less than an hour? | 22:38 |
| ianw | ... maybe; if it doesn't depend on the base job i've broken with the nb04 addition | 22:39 |
| ianw | i don't *think* so | 22:39 |
| clarkb | https://zuul.opendev.org/t/openstack/builds?project=openstack%2Fproject-config&pipeline=deploy&skip=0 looking at that I think ianw is correct | 22:41 |
| clarkb | manage project seems to trigger without the base job firing | 22:41 |
| clarkb | https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_filters.html#selecting-values-from-arrays-or-hashtables the ansible map extract incantation we're using doesn't document a "skip missing entries" behavior at least :( | 22:50 |
| *** rlandy is now known as rlandy|out | 22:58 | |
| opendevreview | Merged openstack/project-config master: Remove redundant editHashtag lines https://review.opendev.org/c/openstack/project-config/+/869878 | 22:58 |
| JayF | Change merged; I still do not have access | 23:05 |
| JayF | ! [remote rejected] bugfix/10.2-eol -> bugfix/10.2-eol (prohibited by Gerrit: not permitted: create signed tag) | 23:05 |
| clarkb | how long ago did you test it? it does take some time to apply | 23:06 |
| clarkb | (though the job is done running now) | 23:06 |
| JayF | that paste was sent about 10 seconds after I tried | 23:06 |
| JayF | I can retry :D | 23:06 |
| JayF | failed again | 23:06 |
| clarkb | https://zuul.opendev.org/t/openstack/build/a47bb7156dd9495b863ea941c1a5bc3c/log/manage-projects.yaml.log#6034-6042 | 23:08 |
| clarkb | toggle wip state config is invalid | 23:08 |
| clarkb | if you compare the ironci and ironic-inspector config files the reason is more clear | 23:08 |
| clarkb | missing group prefix value on the specification | 23:09 |
| clarkb | ironic-inspector-specs and python-ironic-inspector-client have the same problem | 23:10 |
| JayF | I knew somehow this would end up being my fault :| | 23:10 |
| opendevreview | Jay Faulkner proposed openstack/project-config master: Correct syntax on toggleWipState https://review.opendev.org/c/openstack/project-config/+/869882 | 23:12 |
| JayF | clarkb: I believe those other repos you list are getting their config from the same file; so the one change should be ssufficient | 23:12 |
| clarkb | yes they refer to that one file | 23:14 |
| clarkb | fungi: ianw: can we do ::FFFF:w.x.y.z safely and have that be the ip addr? | 23:14 |
| clarkb | and then switch the zookeeper connections to ipv4? | 23:15 |
| clarkb | I'm not sure how safe that is | 23:16 |
| *** dasm is now known as dasm|off | 23:20 | |
| fungi | clarkb: probably, but how well tools respect the v4-in-v6 notation tends to vary | 23:39 |
| fungi | i'm short on context for your question though... why do we need to update zk ip addresses? | 23:40 |
| fungi | the nb04 replacement i guess? | 23:41 |
| clarkb | fungi: this is related to our base job being broken. nb04.opendev.org has no public_v6 address in our inventory. This is breaking the iptables management that expects all nodes in the nodepool group to have a public_v6 address to update iptables for | 23:42 |
| clarkb | fungi: additionally nodepool connects to the ipv6 addresses of our zookeeper servers. We would need to flip that to ipv4 for nb04 (and should just do all of them that way) | 23:42 |
| fungi | oh. i guess we didn't have an available fip in osuosl? | 23:42 |
| clarkb | I think osuosl doesn't do ipv6 at ll | 23:42 |
| fungi | oh, no v6 | 23:42 |
| clarkb | so the issue is how do we convince the firewall to accept a lack of an ipv6 address (it would be fine for it to just skip the host because it doesn't do ipv6 but I can't figure out how to do that succinctly in jinja) | 23:43 |
| clarkb | we can add ipv6 somehow if that is doable. We could ignore ipv6 by giving it good enough data (my ::FFFF: prefix idea) | 23:44 |
| clarkb | or we can rewrite the templating to skip missing data somehow (I just haven't figured that out) | 23:44 |
| fungi | or we could upgrade to nftables | 23:45 |
| fungi | (partly joking, that's a bigger effort) | 23:45 |
| clarkb | well the error is in ansible so whatever implentation we use would need to handle the error in ansible first | 23:46 |
| fungi | the up side to nft is that we could have one ruleset which is address family agnostic and accepts v4 and v6 literals interchangeably | 23:46 |
| fungi | but not right now | 23:47 |
| ianw | ... so quickly back to manage-projects -- that passed despite not being able to apply the rule, and presumably updated the sha1 hash to the invalid config? | 23:47 |
| ianw | that seems wrong | 23:47 |
| clarkb | right but that doesn't solve the problem of "we assume you have ipv6 and it is an error in our ansible if ou don't" | 23:47 |
| clarkb | ianw: I pushed a change that should address that https://review.opendev.org/c/opendev/jeepyb/+/869873 | 23:47 |
| fungi | yeah, it would be more like we supply a list of addresses for the template and don't restrict them to or assume specific address families | 23:48 |
| clarkb | ianw: currentl manage-projects is definitely trying to apply as many configs as possible allowing us to have broken config like the ironic-inspector config. THis is a choice and one we appear to have lived with for some time | 23:48 |
| clarkb | ianw: that change should force it to short circuit and hopefully make it a lot more apparent something went wrong. I'm not convinced it is a correct change yet though. | 23:48 |
| fungi | i think bailing on an incorrect configuration is probably fine these days with closer attention paid to post-merge job results | 23:49 |
| ianw | yeah i see the logic of current failure, but given the monitoring situation these days probably the job bailing is better? | 23:50 |
| fungi | back when we first added manage-projects, it was definitely a throw-spaghetti-at-the-wall-and-see-what-sticks model | 23:50 |
| ianw | haha it's jinx day, i think fungi and i said the same thing | 23:50 |
| opendevreview | Merged openstack/project-config master: Correct syntax on toggleWipState https://review.opendev.org/c/openstack/project-config/+/869882 | 23:50 |
| clarkb | ya I think the main downside to bailing is we'll automatically bail until someone fixes it | 23:50 |
| clarkb | and someone else may want to apply their correct acls in the interim | 23:50 |
| fungi | we take that approach with most stuff these days, so it's more consistent with our current approach | 23:51 |
| clarkb | I think that is the balance here. Do we force user X to fix user Y's problem to enact their change | 23:51 |
| ianw | i wonder though how many incorrect ones we have now ...? | 23:51 |
| ianw | maybe we should delete the .json cache file for a run and work through it? | 23:51 |
| clarkb | ianw: its all in the lgo file I linked. The only three I Found are the ironic-inspector ones | 23:51 |
| clarkb | oh I see what you mean | 23:51 |
| clarkb | ya we could be ignoring some failures | 23:51 |
| fungi | we *could* be ignoring a very many failures, in fact | 23:52 |
| clarkb | if we do delete the cache file we should be prepared for it to take some time | 23:52 |
| ianw | it might be worth trying that out before trying https://review.opendev.org/c/openstack/project-config/+/867931 | 23:52 |
| JayF | I don't see any post jobs on zuul.opendev.org (under openstack), and https://review.opendev.org/admin/repos/openstack/ironic-inspector,access shows I do not have access even still :| | 23:52 |
| clarkb | JayF: they are deploy jobs | 23:53 |
| fungi | yeah, deploy pipeline not post | 23:53 |
| clarkb | the job is stillrunning | 23:53 |
| JayF | Can you teach me how to fish? Where I'd find this job from the original review ID, if possible? | 23:53 |
| clarkb | JayF: https://zuul.opendev.org/t/openstack/status looks for the deploy pipeline and any changes/jobs enqueued there | 23:54 |
| JayF | thanks; I see it now (I had a search string still applying on that page; so it wasn't showing) | 23:55 |
| ianw | clarkb: https://opendev.org/opendev/system-config/src/branch/master/zuul.d/infra-prod.yaml#L90 the manage job has a 4800 second timeout. do you think that would be sufficient if we mv'd the cache out of the way? | 23:57 |
| clarkb | ianw: its a good question. We'd still have all the git repos cached which is probably actually the bulk of the time we'd spend if we weren't cached. But we would be pushing a couple thousand refs/meta/config serially which isn't fast | 23:58 |
| JayF | post job completed; I still have no access per https://review.opendev.org/admin/repos/openstack/ironic-inspector,access | 23:58 |
| clarkb | ianw: maybe check how long it took ironic to push its refs meta config and then multiply that by the number of repos in projects.yaml? | 23:58 |
| clarkb | JayF: the job log says it pushed this time without error at least | 23:59 |
| clarkb | it might be caching at the web layer/ | 23:59 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!