Wednesday, 2023-01-11

opendevreviewMerged openstack/project-config master: openstack-afs.yaml : correct indentation
opendevreviewIan Wienand proposed opendev/system-config master: [wip] add variable to block UA's for mailman
opendevreviewIan Wienand proposed opendev/system-config master: [wip] add variable to block UA's for mailman
opendevreviewMerged openstack/project-config master: Add nb04 config
opendevreviewIan Wienand proposed opendev/system-config master: [wip] add variable to block UA's for mailman
opendevreviewIan Wienand proposed opendev/system-config master: [wip] add variable to block UA's for mailman
opendevreviewIan Wienand proposed opendev/system-config master: mailman: add variable for matching UAs in Apache
*** ysandeep is now known as ysandeep|ruck05:12
*** marios is now known as marios|rover06:01
*** bhagyashris|brb is now known as bhagyashris06:39
*** bhagyashris is now known as bhagyashris|afk06:39
*** ysandeep|ruck is now known as ysandeep|ruck|brb06:42
jexsieHello everyone, am new here.Anything for me??06:52
opendevreviewMichael Kelly proposed zuul/zuul-jobs master: prepare-workspace-git: Skip LFS checkout when mirroring repos
*** ysandeep|ruck|brb is now known as ysandeep|ruck07:18
*** ysandeep|ruck is now known as ysandeep|lunch07:22
*** soniya29 is now known as soniya29|lunch08:01
opendevreviewMichael Kelly proposed zuul/zuul-jobs master: prepare-workspace-git: Skip LFS checkout when mirroring repos
opendevreviewMichael Kelly proposed zuul/zuul-jobs master: prepare-workspace-git: Skip LFS checkout when mirroring repos
opendevreviewMichael Kelly proposed zuul/zuul-jobs master: prepare-workspace-git: Skip LFS checkout when mirroring repos
*** jpena|off is now known as jpena08:29
*** ysandeep|lunch is now known as ysandeep|ruck08:29
*** ysandeep|ruck is now known as ysandeep|ruck|afk08:40
*** soniya29|lunch is now known as soniya2909:18
*** ysandeep|ruck is now known as ysandeep|ruck|afk09:44
*** ysandeep__ is now known as ysandeep|ruck11:07
*** rlandy|out is now known as rlandy11:15
*** bhagyashris|afk is now known as bhagyashris11:32
*** artom_ is now known as artom11:33
opendevreviewCedric Jeanneret proposed opendev/system-config master: Correct (again) how ansible-galaxy proxy is configured
amorinhello clarkb ianw fungi and others, I confirmed we had an issue this night on our object storage:
amorinthe issue is now over, so it's safe to reopen the swift if you want12:47
fungithanks for confirming it's back up amorin!12:49
*** soniya29 is now known as soniya29|afk13:04
opendevreviewDr. Jens Harbott proposed opendev/base-jobs master: Revert "Disable OVH BHS1 and GRA1 log uploads"
fungii know we talked about testing that with base-test, but with the updated incident information it's probably unwarranted13:22
opendevreviewMerged opendev/base-jobs master: Revert "Disable OVH BHS1 and GRA1 log uploads"
*** ysandeep|ruck is now known as ysandeep|ruck|afk14:01
*** ysandeep|ruck|afk is now known as ysandeep|ruck14:30
*** ysandeep|ruck is now known as ysandeep|ruck|afk15:08
*** ysandeep|ruck|afk is now known as ysandeep|out16:31
opendevreviewCedric Jeanneret proposed opendev/system-config master: Correct (again) how ansible-galaxy proxy is configured
*** marios|rover is now known as marios|out16:36
*** jpena is now known as jpena|off17:21
*** gthiemon1e is now known as gthiemonge19:43
JayFHeads up: I'm executing on Ironic changes to retire bugfix branches. 20:54
fungithanks for the warning!20:57
JayFcompleted ironic, moving to ironic-python-agent20:57
JayFIPA complete, moving to ironic-inspector21:00
JayFhmm. I appear to be missing perms for ironic-inspecotr21:01
JayF ! [remote rejected] bugfix/10.2-eol -> bugfix/10.2-eol (prohibited by Gerrit: not permitted: create signed tag)21:01
fungii'm available to expedite an acl patch, give me a heads up when you push it21:02
JayFI'm not sure I understand why it's needed21:03
JayFfungi: I don't see why I'm disallowed21:04
JayFAm I missing something? I've double checked two or three times21:08
JayFI wonder if I can see the effective ACLs in gerrit ui...21:09
fungimaybe there's a typo in the acl21:09
JayFyeah I checked again, if there's a typo it's evading me, even looking at ironic.config stanzas and ironic-inspector.config stanzas side by side21:10
fungido you see refs/tags/* when you visit,access21:11
JayF no21:12
JayFI have Reference: refs/heads/bugfix/*21:12
JayFbut not tags21:12
JayFeven though tags appear to be in the config file21:12
fungiit should show up if you have access...21:12
fungiunfortunately gerrit filters that view to the things your account is granted access to, which makes it hard to use that to check the loaded acl21:12
JayFfrom ironic.config:21:13
JayF[access "refs/tags/*"]21:13
JayFcreateSignedTag = group ironic-release21:13
JayFfrom inspector.config21:13
JayF[access "refs/tags/*"]21:13
JayFcreateSignedTag = group ironic-release21:13
JayFconfirmed in a different font/venue to be identical21:13
fungiyeah, and we heavily keyword check those acls with a zuul job, so the leeway for typos is pretty narrow anyway21:14
JayFcan we confirm via gerrit logs or similar that the acl was read in?21:15
fungiso... your change theoretically added that permission when it merged on 2022-12-0921:15
JayFand I just exercised the permissions added for ironic, because I was able ot perform the work I wanted to21:15
fungiand yeah, the installed configuration gets committed to a git repository in gerrit which admins can clone, so i can check it there21:15
JayFI've literally done ironic + ironic-python-agent, nothing broke until ironic-inspector21:15
JayFand AFAICT there is no difference21:15
Clark[m]fungi: it gets committed to the ironic-inspector repo 21:16
Clark[m]But under a ref you need your admin account to fetch21:16
fungiahh, yeah, it's not in All-Projects21:17
JayFI do not follow21:17
fungirefs/meta/config is what contains it21:18
JayFI (still) don't follow (?) (I'm not sure if this is for me to comprehend anyway lol)21:21
fungiconfirmed, the last update to the project.config file in the meta/config ref for openstack/ironic-inspector lacks that update. it was last committed Fri Apr 30 15:21:59 2021 +000021:21
fungiJayF: you'd need a gerrit admin account to be able to fetch that ref, so no don't worry too much about that part21:22
Clark[m]Did that land around when we upgraded Gerrit? 21:22
Clark[m]That may explain it if so21:22
Clark[m]Ya I can't remember the exact day we upgraded but it was early December 21:22
JayFbut some of the changes in that patch ( ) did apply; the ironic.config changes21:22
fungii would expect to see errors in the manage-projects log if this wasn't getting successfully updated21:22
Clark[m]It may have cached that it performed the update and is just skipping it now21:23
Clark[m](when it hadn't for some reason)21:23
fungii'm being called away to cook dinner, but can probably resume digging into this in an hour21:23
ianw(we upgraded gerrit on 2022-12-13 .au date)21:23
JayFI'd prefer be able to apply the changes atomically, since they were announced atomically -- if there's any way to get it fixed before my EOD; I'd greatly appreciate it21:24
ianwi have a checkout of the meta/config of ironic-inspector and confirm the same, no commits since april21:24
JayFalternatvely; the commands I need to run are documented and someone with access could run them to allow more troubleshooting time without impacting repo state21:24
ianwjust looking at infra-prod-manage-projects to see what might have happened21:25
ianw22-12-09 we had a number of changes i guess ->
ianw was the job for 86693721:27
ianw... unfortunately no logs21:27
Clark[m]There may be logs still on bridge21:28
ianw... might be on bridge21:28
ianwheh, jinx21:28
ianwit ran at 13:53, must be manage-projects.yaml.log.2022-12-09T13:54:4521:30
ianw# ls -lh manage-projects.yaml.log.2022-12-09T13:54:4521:30
ianw-rw-r--r-- 1 root root 0 Dec  9 13:54 manage-projects.yaml.log.2022-12-09T13:54:4521:30
ianw... weird ...21:30
ianwoh jeez, they're all 0 sized21:31
Clark[m]Oh ya this one actually logs to zuul because we cleared it as safe iirc21:31
Clark[m]So we may intentionally not be recording bridge logs :(21:31
ianwwe should at least tee it or something.  anyway, i guess that's a problem for another time :/21:32
Clark[m]I think we need to inspect the state on review. It stores a json cache file iirc and it should give us a clue for why it didn't update 21:33
ianw2023-01-10 14:57:03,645: manage_projects - INFO - Processing project: openstack/ironic-inspector21:34
ianw2023-01-10 14:57:03,645: manage_projects - INFO - openstack/ironic-inspector has matching sha, skipping ACLs21:34
ianwthe latest log says that about ironic-inspector21:34
Clark[m]Maybe we aren't comparing the correct thing? I didn't expect that21:34
clarkbianw: reading jeepyb I think it is comparing the sha it generates for the files from project-config/gerrit/acls/openstack/foo.conf against what is in the project cache it maintains locally21:36
clarkbso ya it mustve updated the local project cache locally thinking it had updated things and has since skipped over it despite them actually differing in gerrit21:37
ianwyeah, is that cache in the container?21:38
ianwno, /opt on review21:39
ianw"acl-sha": "ae1bb09c706a3a99d71caf7805b4b58a3a61cd5a4fb1ee74ba6354f6af719326"21:40
clarkbthe manage-projects command is the one that mounts the jeepyb stuff regular gerrit doesn't21:40
clarkband it mounts /opt/lib/jeepyb and project.cache is there which has that value in it which matches the project config value21:41
clarkbso ya it thought it had updated things but apparently not21:41
ianwae1bb09c706a3a99d71caf7805b4b58a3a61cd5a4fb1ee74ba6354f6af719326  ./ironic-inspector.config21:41
ianwwell, that confirms what we know -- that manage-projects thinks it has applied ae1bb09... at least21:41
JayFin this direction; the failure scenario is OK: we didn't add credentials; isn't this potentially a dangerous situation in the other direction?21:41
clarkbthe easiest thing is to make a noop edit to the config file and have automation try to apply it21:42
clarkbJayF: I mean if gerrit says it succeeded but didn't there isn't much we can do about that21:42
ianwthere is some possibility, especially given the zero sized files, that we are somehow missing the exit code and showing success on jobs that maybe aren't21:43
opendevreviewMerged opendev/system-config master: Add
clarkbianw: I mean within manage-projects I think the only way we record that sha permanently is if gerrit repsonds with success to the push21:43
JayFliterally the config update is a git push to a special ref? I guess that makes sense21:44
clarkbJayF: yes21:44
clarkbalmost everything in gerrit is git now for better or worse21:44
ianwclarkb: true -- unless something similar in that we're missing the failure of the push due to ... something.  new git or something?  clutching at straws :)21:45
ianwi'd agree a no-op push we can monitor and see the logs of will be most helpful21:45
ianwannoyingly the nb04 addition i made is probably running every infra-prod job now21:45
clarkbianw: looks like while we catch an exception from the push the push might also return false to indicate failure which we don't seem tohandle arg21:45
clarkbso the fix here might be to check the return and raise an exception if its false to fall through the existing exception handling which should prevent us from recording the cached sha value21:46
clarkbI'm going to write that patch and we can think it over with something more concrete21:47
JayFclarkb: do you want me specifically to push a noop patch to that file in gerrit? 21:47
JayFI was unsure if you wanted a noop change done locally or via review21:47
clarkbJayF: via review21:49
clarkbso that we exercise the whole thing21:49
JayFack; incoming21:49
clarkbso I think that it may have been done this way to allow for subsets of projects to update and avoid short circuiting21:50
opendevreviewJay Faulkner proposed openstack/project-config master: Noop change to ironic-inspector.config
clarkbbut I think short circuiting is probbaly preferable here21:51
opendevreviewClark Boylan proposed opendev/jeepyb master: Raise and error if acl pushes fail
clarkbthat deserves careful review21:53
fungiokay, dinner has been made, consumed, and cleaned up22:05
JayFmaking a noop change in a repo22:07
JayFthat enforces lint on whitespace 22:07
JayFany suggestions on what to do, since adding a newline broke lint?22:07
clarkbJayF: reorder the entries or similar22:08
JayFthat's not possible, right? they enforce abc order22:08
clarkbI thought it did that on the backend via a normalization pass and not via linting22:08
clarkbI could be wrong about that22:08
JayFlinting will not pass if items are not in abc order22:08
JayFI know this from experience22:08
fungiwe enforce normalization in the check job22:08
* JayF looks to see if project.config has a comment character22:09
fungimanage-projects allows us to manually specify a repository and force an update i think, if we want to go that route22:09
JayFokay; apparently # should be a comment, i'll try that22:09
clarkbfungi: it allows us to specify a repository but I don't think it has a force option22:10
clarkbwe couod manually edit the json file to change the sha22:10
fungii thought that was implicit when running with specific repositories22:10
opendevreviewJay Faulkner proposed openstack/project-config master: Noop change to ironic-inspector.config
clarkbbut there is something nice about an end to end exercise if we can come up with something to do that22:10
clarkbfungi: no if you read the code there is no escape hatch for the sha check22:10
ianwwe could delete the cache entry and run it manually too?22:11
clarkbianw: yes or just change the sha value22:11
ianwshould i try that?22:12
clarkblets do the noop first?22:12
ianwalso, the deploy queue doesn't seem to have run for nb04 ... something must have failed22:12
clarkbJayF: I think you can drop the edithashtags entries from the more specific paths22:12
clarkbJayF: as a non noop cleanup option22:12
ianwinfra-prod-base : FAILURE in 13m 43s22:12
clarkbthey are redundant22:12
JayFI would strongly prefer not making actual-changes while troubleshooting is also occurring 22:13
JayFI'm pretty sure that right now these configs are relatively similar between ironic projects; so if I was going to clean up inspector I'd want to do it across ironic projects22:13
clarkbsure the comment should be fine too and I've +2'd it22:13
clarkbI'm just pointing out redundancies that can go away if we need something more forceful22:13
ianwok, just to add to the problems here22:14
ianwThe error was: ansible.errors.AnsibleUndefinedVariable: 'ansible.vars.hostvars.HostVarsVars object' has no attribute 'public_v6'22:14
clarkbianw: did you add a server without ipv6?22:14     : ok=63   changed=3    unreachable=0    failed=1    skipped=9    rescued=0    ignored=0   22:14      : ok=63   changed=3    unreachable=0    failed=1    skipped=9    rescued=0    ignored=0   22:14           : ok=63   changed=3    unreachable=0    failed=1    skipped=9    rescued=0    ignored=0   22:14           : ok=63   changed=3    unreachable=0    failed=1    skipped=9    rescued=0    ignored=0   22:14           : ok=63   changed=3    unreachable=0    failed=1    skipped=9    rescued=0    ignored=0   22:14
ianwthis is an odd set of servers to fail22:15
ianwclarkb: yes, the new linaro cloud doesn't have ipv622:15
clarkbya so the issue is we use public_v6 to genreate ip tables rules22:16
ianwok, then that does make sense22:16
clarkband the new nb04 server is going to talk to all of those I think so they want their iptables updated with its ip addr22:16
ianwthat's the servers nb04 would talk to22:16
clarkbwe might be able to work through this just with group membership. But maybe better is just ignore public_v6 if it isn't set since there won't be an ipv6 addr to set an iptables rule for22:17
clarkbof course this gets scary because its going to apply to all the things22:17
ianwyeah, but we can't really stub out the ipv6 (ipv6: '') as that will make wrong rules22:18
clarkb{% for addr in groups.get( | map('extract', hostvars, 'public_v6') -%} <- is the line in question22:18
clarkbcan we | somethingtoskipifnotset?22:18
clarkbif we give it an ipv6 addr for another host in the same group that would approximately work (until the hosts change)22:19
clarkb(I hate this idea for the record just talking out loud)22:19
ianwhosts: "{{ (zk_hosts['hosts']|default([])) + [{'port': '2281', 'host': hostvars[item]['public_v6'] | default(hostvars[item]['ansible_host']) }] }}"22:20
ianwthis might be setting the zk hosts to the ipv6 addresses in the zk config too22:20
fungiJayF: clarkb: comments in acls are disallowed by our normalizing linter too:
clarkbianw: ah yup it is22:22
fungithe original goal was to be able to use file checksums to identify identical acl content for deduplication22:22
clarkbI feel like the linting is being more of a hindrance than a help right now... But removing the redundant hashtagedit lines should work22:23
JayFso literally if we were able to push a noop change; it'd be a bug in your linter22:23
clarkbJayF: no I've called out a noop change22:23
clarkbone that I'm 99% sure I called out on the original changes too22:23
JayFYour change is removing overlapping rules which will have no impact; not exactly the same as a specifically noop change22:23
clarkbbut I didn't care neough to -1 for them22:24
JayFespecially from the perspective of someone who barely understands these ACLs :)22:24
JayFI don't like pushing changes that I myself don't personally understand why it's OK 22:24
clarkbJayF: do you have a link to the original change?22:24
clarkbok this isn't the one I'm thinking of which change was it22:25
fungiwhich original change? the one adding the git tag permissions or the one adding hashtag permissions?22:25
clarkbfungi: the one adding the hashtag edits22:26
clarkbI remember a number of people all going out and doing this together and I commented on a number of changes to not do that redundantly and no one listened ut I wasn't going to argue it22:26
ianwi have to afk for ~ 1 hour ... i don't think infra-prod-base not working will affect manage-projects, if it is we can revert the nb04 addition.  otherwise i'll work on the no ipv6 thing22:26
JayFclarkb: I'll point out my name isn't on that change as an author, committer, or reviewer :/ I still would prefer not push a change to one that wasn't done everywhere22:27
JayFand doing the bigger change means I run out of sunlight before completing my bugfix branch retirement changes22:27
clarkbok, Ive made a suggestion that we believe will get us over the hump. I'm open to other suggestions22:28
clarkbI will not manually retire those branches22:28
JayFso I'll start building that, if that's hte only way out is for me to cleanup those hashtag things; I get to go read up on how acl inheritance works22:28
clarkbother ideas: update the normalization to allow comments and go with the comment change22:29
ianwfor the immediate purposes of JayF I guess we could manually push a change to the refs to sync it22:30
clarkbianw: no I don't think we should do that either22:30
JayFfungi indicated that blocking comments was intentional; so that the sha1 of the file always indicated the same effective contents22:30
clarkbbecause there may be something speficially broken with ironic-inspector meta config with jeepyb22:30
ianwthe debugging of manage-projects will still happen right -- since that looks at the sha of the cache+project config, not what's committed?22:30
ianw(i'm not saying don't debug why manage-proejcts didn't apply)22:30
clarkbianw: a git push that is a noop is different than a git push update22:30
clarkbianw: we would end up doing a git push noop22:31
clarkbwhich might expose issues but it might also not22:31
ianwwell we could push our change sufficiently with whitespace etc such that the new push does change it?22:31
ianwor you think if the actual parsed rules don't change, it would still be no-op-ish?22:31
clarkbbut that won't change anything to cause a new manage projects push to happen22:31
fungi "For allowing access, all ALLOW/DENY rules that might apply to a ref are tested until one granting access is found, or until either an "exclusive" rule ends the search, or all rules have been tested."22:31
fungithat's the relevant inheritance information about ref sections, fwiw22:32
JayFthanks; that's what I was looking for. I do not push changes on the word of another person since it's my responsibility if they go kaboom22:32
fungialso just after that, "The rules are ordered from specific ref patterns to general patterns..."22:32
fungiso a more general ref with an allow is hit before the specific one, and basically obviates the latter22:33
fungier, rather, a more general ref will apply even in the absence of a specific one22:34
fungifor an allow permission22:34
fungianyway, that section of the docs is pretty thorough22:34
ianwclarkb: yeah ... i agree on getting manage-projects to still run somehow.  just trying to think of a way to also get the branch retirement bits JayF wanted done22:35
opendevreviewJay Faulkner proposed openstack/project-config master: Remove redundant editHashtag lines
JayFI think that should be what clarkb wanted, ty for the docs fungi 22:36
fungilet's get this exercised so we can collect an additional data point. probably it will get the acl up to current, or it will at least give us a better idea of what's going wrong since we should have fresh logs22:37
JayFin a best case scenario; is it likely for that to be landed and applied in less than an hour?22:38
ianw... maybe; if it doesn't depend on the base job i've broken with the nb04 addition22:39
ianwi don't *think* so22:39
clarkb looking at that I think ianw is correct22:41
clarkbmanage project seems to trigger without the base job firing22:41
clarkb the ansible map extract incantation we're using doesn't document a "skip missing entries" behavior at least :(22:50
*** rlandy is now known as rlandy|out22:58
opendevreviewMerged openstack/project-config master: Remove redundant editHashtag lines
JayFChange merged; I still do not have access23:05
JayF ! [remote rejected] bugfix/10.2-eol -> bugfix/10.2-eol (prohibited by Gerrit: not permitted: create signed tag)23:05
clarkbhow long ago did you test it? it does take some time to apply23:06
clarkb(though the job is done running now)23:06
JayFthat paste was sent about 10 seconds after I tried23:06
JayFI can retry :D 23:06
JayFfailed again23:06
clarkbtoggle wip state config is invalid23:08
clarkbif you compare the ironci and ironic-inspector config files the reason is more clear23:08
clarkbmissing group prefix value on the specification23:09
clarkbironic-inspector-specs and python-ironic-inspector-client have the same problem23:10
JayFI knew somehow this would end up being my fault :| 23:10
opendevreviewJay Faulkner proposed openstack/project-config master: Correct syntax on toggleWipState
JayFclarkb: I believe those other repos you list are getting their config from the same file; so the one change should be ssufficient23:12
clarkbyes they refer to that one file23:14
clarkbfungi: ianw: can we do ::FFFF:w.x.y.z safely and have that be the ip addr?23:14
clarkband then switch the zookeeper connections to ipv4?23:15
clarkbI'm not sure how safe that is23:16
*** dasm is now known as dasm|off23:20
fungiclarkb: probably, but how well tools respect the v4-in-v6 notation tends to vary23:39
fungii'm short on context for your question though... why do we need to update zk ip addresses?23:40
fungithe nb04 replacement i guess?23:41
clarkbfungi: this is related to our base job being broken. has no public_v6 address in our inventory. This is breaking the iptables management that expects all nodes in the nodepool group to have a public_v6 address to update iptables for23:42
clarkbfungi: additionally nodepool connects to the ipv6 addresses of our zookeeper servers. We would need to flip that to ipv4 for nb04 (and should just do all of them that way)23:42
fungioh. i guess we didn't have an available fip in osuosl?23:42
clarkbI think osuosl doesn't do ipv6 at ll23:42
fungioh, no v623:42
clarkbso the issue is how do we convince the firewall to accept a lack of an ipv6 address (it would be fine for it to just skip the host because it doesn't do ipv6 but I can't figure out how to do that succinctly in jinja)23:43
clarkbwe can add ipv6 somehow if that is doable. We could ignore ipv6 by giving it good enough data (my ::FFFF: prefix idea)23:44
clarkbor we can rewrite the templating to skip missing data somehow (I just haven't figured that out)23:44
fungior we could upgrade to nftables23:45
fungi(partly joking, that's a bigger effort)23:45
clarkbwell the error is in ansible so whatever implentation we use would need to handle the error in ansible first23:46
fungithe up side to nft is that we could have one ruleset which is address family agnostic and accepts v4 and v6 literals interchangeably23:46
fungibut not right now23:47
ianw... so quickly back to manage-projects -- that passed despite not being able to apply the rule, and presumably updated the sha1 hash to the invalid config?23:47
ianwthat seems wrong23:47
clarkbright but that doesn't solve the problem of "we assume you have ipv6 and it is an error in our ansible if ou don't"23:47
clarkbianw: I pushed a change that should address that
fungiyeah, it would be more like we supply a list of addresses for the template and don't restrict them to or assume specific address families23:48
clarkbianw: currentl manage-projects is definitely trying to apply as many configs as possible allowing us to have broken config like the ironic-inspector config. THis is a choice and one we appear to have lived with for some time23:48
clarkbianw: that change should force it to short circuit and hopefully make it a lot more apparent something went wrong. I'm not convinced it is a correct change yet though.23:48
fungii think bailing on an incorrect configuration is probably fine these days with closer attention paid to post-merge job results23:49
ianwyeah i see the logic of current failure, but given the monitoring situation these days probably the job bailing is better?23:50
fungiback when we first added manage-projects, it was definitely a throw-spaghetti-at-the-wall-and-see-what-sticks model23:50
ianwhaha it's jinx day, i think fungi and i said the same thing23:50
opendevreviewMerged openstack/project-config master: Correct syntax on toggleWipState
clarkbya I think the main downside to bailing is we'll automatically bail until someone fixes it23:50
clarkband someone else may want to apply their correct acls in the interim23:50
fungiwe take that approach with most stuff these days, so it's more consistent with our current approach23:51
clarkbI think that is the balance here. Do we force user X to fix user Y's problem to enact their change23:51
ianwi wonder though how many incorrect ones we have now ...?23:51
ianwmaybe we should delete the .json cache file for a run and work through it?23:51
clarkbianw: its all in the lgo file I linked. The only three I Found are the ironic-inspector ones23:51
clarkboh I see what you mean23:51
clarkbya we could be ignoring some failures23:51
fungiwe *could* be ignoring a very many failures, in fact23:52
clarkbif we do delete the cache file we should be prepared for it to take some time23:52
ianwit might be worth trying that out before trying
JayFI don't see any post jobs on (under openstack), and,access shows I do not have access even still :| 23:52
clarkbJayF: they are deploy jobs23:53
fungiyeah, deploy pipeline not post23:53
clarkbthe job is stillrunning23:53
JayFCan you teach me how to fish? Where I'd find this job from the original review ID, if possible?23:53
clarkbJayF: looks for the deploy pipeline and any changes/jobs enqueued there23:54
JayFthanks; I see it now (I had a search string still applying on that page; so it wasn't showing)23:55
ianwclarkb: the manage job has a 4800 second timeout.  do you think that would be sufficient if we mv'd the cache out of the way?23:57
clarkbianw: its a good question. We'd still have all the git repos cached which is probably actually the bulk of the time we'd spend if we weren't cached. But we would be pushing a couple thousand refs/meta/config serially which isn't fast23:58
JayFpost job completed; I still have no access per,access 23:58
clarkbianw: maybe check how long it took ironic to push its refs meta config and then multiply that by the number of repos in projects.yaml?23:58
clarkbJayF: the job log says it pushed this time without error at least23:59
clarkbit might be caching at the web layer/23:59

Generated by 2.17.3 by Marius Gedminas - find it at!