Thursday, 2023-03-09

clarkbthe two new giteas have their empty giteas deployed now. Its late enough in the day I think I'll save brain transplants and setting up replication for tomorrow00:18
fungisounds good00:18
clarkbOne thing I'll probably do is transplate from gitea09 rather than gitea0100:19
clarkbjust to prove out the transitive continuity of this process00:20
clarkbalso /me makes a note to look into why its ok to land multiple project rename changes after we do the owrk on gerrit00:26
opendevreviewIan Wienand proposed opendev/system-config master: dns: remove old nameservers from iptables list
ianwok the copy conditions change rolled out to gerrit with
opendevreviewIan Wienand proposed opendev/system-config master: remove adns1 host_vars file
opendevreviewIan Wienand proposed opendev/system-config master: system-config-run-dns : update nodes to jammy
opendevreviewIan Wienand proposed opendev/system-config master: Remove unused adns1/ns* host_vars files
opendevreviewIan Wienand proposed opendev/system-config master: system-config-run-dns : update nodes to jammy
opendevreviewIan Wienand proposed opendev/system-config master: dns variables : move to canonical locations
opendevreviewIan Wienand proposed opendev/system-config master: Refactor adns variables
opendevreviewIan Wienand proposed opendev/system-config master: system-config-run-dns : update nodes to jammy
opendevreviewIan Wienand proposed opendev/system-config master: Refactor adns variables
opendevreviewIan Wienand proposed opendev/system-config master: bind9 : drop obsolete option for later versions
ianwclarkb: ^ still a bit of a wip but will update as i go along05:39
*** jpena|off is now known as jpena07:50
fricklerinfra-root: some people are reporting changes not getting merged after zuul +2. this isn't the usual "change needs rebase" situation. looking at zuul logs, I see some gerrit connection failures, but haven't been able to correlate those with failed merges08:58
fricklerthere's a lot of errors like but those seem likely not related08:59
fricklerexample patch is
tkajinamthese are some examples. just for your reference :-)09:00
tkajinamseems the change is not submitted to git repo so I guess something failed before that
fricklerI'm suspecting some relation with the submit requirements changes, but no idea how to debug this further. no obvious errors in gerrit log, so likely gerrit just assumes there is some unmet condition yet09:03
frickler2023-03-09 08:12:30,007 INFO zuul.GerritConnection: [e: 5428cb4fff314ed686e889946db11667] Conflict submitting data to gerrit, change may already be merged09:03
yoctozeptofrickler: maybe need to establish who is affected and start from there? and also: rechecks do not help? (to eliminate temporary issues)09:24
fricklerat least not all changes are affected, e.g. merged fine09:26
yoctozeptokolla rlz09:35
yoctozeptoanything interesting on the merge requirements diff between heat and kolla?09:36
hemanthyoctozepto: i have the same problem with gate +2 and not merged. I did recheck and situation is same. For reference,
yoctozeptouh-oh, so it's persistent10:23
yoctozeptoso we have charms and heat affected at least10:23
yoctozeptoand kolla unaffected10:23
tkajinamI have one patch I have to merge to unblock requirement bump but seems I have to wait for a while10:31
frickleryes, I'd rather have some of my co-roots check the situation before touching anything. things like another gerrit restart might as well make the situation worse than better11:02
tkajinamI'll leave now and come back later in a few hours.11:09
tkajinamto check the status11:10
tkajinamfrickler, thanks again !11:10
frickleractually zuul does report the issue in the buildset results
fricklerfirst of the current list seems to have been at 00:28 (start time + buildset duration) . nothing too obvious to me in terms of the set of affected projects11:14
fricklerin the gerrit UI, all submit requirements are shown as unsatisfied, even with all the necessary +1/+2 in place11:25
fricklerto double-check I made myself admin shortly and verified that I don't get shown the merge button, either11:25
yoctozeptohmm, worrying11:27
yoctozeptoI quickly checked that there were seemingly no relevant project-config updates that could break it so much11:28
yoctozeptounless they hit some gerrit bug11:28
yoctozeptolike, if the changes here caused gerrit to misbehave11:29
yoctozeptoit seems it merged at a time likely to be a cause of the disruption11:30
fricklereven abandoning and restoring a patch doesn't help11:31
fungii'm a little fuzzy on the submit requirements stuff, but does it indicate satisfied submit requirements on any changes that haven't merged yet? are we just talking about zuul putting verified +2 on a change but then not merging it, or not enqueuing approved changes into the gate pipeline?12:36
fungithe copycondition change is unlikely to be the cause, the primary acl (called all-projects) which every normal project's acl inherits from is not updated from changes in git, so ianw updated it manually in preparation for the gerrit 3.7 upgrade. see the most recent entry at the top of
fungithat happened just before 2023-03-08 22:22:18 UTC12:40
fungicould there be some connection between changes with an "n/a" status and changes zuul isn't merging?12:41
fungiif so, forcing an online reindex might be the next step. we didn't do that because it was thought the n/a status was purely a cosmetic thing12:42
fungiokay, i'm caught up on the discussion in #openstack-infra now, so it seems zuul tests things in the gate pipeline and then adds +2 verified but fails to merge it12:43
fungiapparently it's possible to reindex a single change, so i suppose i could try that one one of the changes that didn't merge12:46
fungii did a `gerrit index changes 876726` just now12:47
fungiand i've rechecked 876726 to see if that's changed the situation12:50
fungiif it has, we can just force an online reindex of all changes13:03
fungiand then we can pretty easily query gerrit for unmerged changes with verified +2 and recheck them13:05
fricklerthe issue also doesn't seem to be related to specific projects, just merged without issues13:11
fricklerbut show green checkmarks for for CR and W while being processed in gate, for 876726 this still shows unsatisfied, so I fear the reindex has not helped13:13
bbezakI have similar case with running gate for 3-rd time, as I tried to fix it13:18
bbezakthis too -
fungi876726 got another verified +2 but still did not merge even after the reindex13:21
fungii wonder if i need to use the --force flag, maybe the index command won't reindex a change that's already been indexed otherwise?13:22
fungimmm, no --force is only for a full reindex of all changes13:25
bbezakaccording to, this change finished gate jobs but didn't add +2 - not sure if that's related13:26
fungiunrelated, the change ahead of it hasn't completed its jobs yet and might fail, so zuul doesn't technically know yet whether 876807 on top of the change ahead represents a valid state for the branch13:29
bbezakmakes sense13:29
fungii'm going to start a full online reindex, since it will likely be a while before clarkb is awake, and he and ianw have a much stronger grasp of the submit requirements bits in gerrit 3.6/3.7 so i wouldn't want to attempt rolling back the various edits until he's around anyway13:30
fungi2793 tasks in the queue now13:32
fungiit's already finished 1k of those13:38
fungiwaiting for it to get through all of the heat changes so i can see if that's changed the submit requirements satisfaction on 87672613:39
fungiunder 1k tasks remaining now, but hasn't gotten to heat yet13:49
bbezakgate started once more (on its own) for 876807 and 876910, and 876808 hangs on submit requirements13:55
fungiheat changes have finished reindexing but the submit requirements on 876726 still all show unsatisfied13:57
fungiso the forced full reindex doesn't seem to have made any difference either13:58
fungifrickler: i know you had samples which indicate the issue isn't project-specific, but have you noticed if the difference in behavior is maybe branch-specific? for example, i see 876726 is for stable/2023.114:01
fricklerfungi: I've seen failures for both stable/2023.1 and master, I haven't seen success for stable branches yet, but that's not necessarily statistically significant yet14:04
fungiright, thanks14:04
fungii've also asked in the gerritcodereview matrix/discord discussion channel14:12
fungiin hopes someone there has seen this before14:12
iurygregoryhey opendev, we are trying to understand why looks like is stuck, it has verified +2 but doesn't look like is merged... the chain patch says it failed to merge ( based on zuul comment)14:19
fungiiurygregory: it's something to do with the switch to submit requirements in preparation for upgrading to gerrit 3.7, i've just about exhausted my troubleshooting options so am asking the broader gerrit community for suggestions while waiting for people here who have a better grasp of the feature in order to discuss how to roll back the changes14:21
fungiit seems to be impacting random changes, not all changes for the same project/branch get blocked like that14:22
fungibut if a change is affected, there seems to be no point in retrying it until we have this fixed14:22
iurygregoryfungi, ack, tks for the heads-up!14:24
iurygregoryyeah, I will hold this in ironic patches till this the problem is solved14:24
fungilooking in the gerrit error_log, i see changes which are successfully merging logging warnings like this:14:29
fungi[2023-03-09T14:22:15.599Z] [HTTP POST /a/changes/openstack%2Freleases~master~I2f3f23b29a5e1b0f39c2bd61f2f5cd4f3ce4bc79/submit (zuul from [2001:4800:7819:103:be76:4eff:fe04:3df3])] WARN : Change 876611: No result found for project config submit requirement 'workflow' [CONTEXT SUBMISSION_ID="876611-antelope-tp-latest-1678371734954-563cca49"14:29
fungiproject="openstack/releases" request="REST /changes/*/submit" ]14:29
fungiit logs the same thing about code-review and verified labels too14:30
fungifor the changes which are unable to merge due to unsatisfied submit requirements, i find no mention of them in the error_log whatsoever14:41
*** sfinucan is now known as stephenfin14:44
fungithe forced reindex does seem to have addressed the condition in this bug ianw reported at least:
clarkbmy first hunch was that we weren't satisfying the requirement for some legit reason. Then perhaps a bug in the condition for the submit requirement. But both look fine (you can literally hover andview conditions on the labels in the UI and they all seem to match)14:54
fungiand conditions shouldn't differ for changes to the same project+branch either14:55
clarkbthough they can I don't think any of the config we pushed did that (this is what applicableIf is for you can use that to make rules appl to specific branches)14:55
fungioh, like the prolog rules?14:56
clarkball of this exists to replace the prolog rules but be more approachable14:56
clarkbbut ya basically you can write a rule. YOu write conditions to satisfy the rule and conditions for when the rule applies14:56
fungianyway, in this case the hover pop-up for those conditions says approximately "these conditions aren't met... [conditions which the applied vote actually meets]"14:58
clarkbyes if you expand further and look at the rule it seems to be evaluating the definitely should be satisfied14:58
clarkbfungi: maybe try pushing a new change to the project+branch combo and see if setting satisfying votes on it changes anything15:01
fungiyeah, i guess the example i'm looking at is a change which was pushed before the acl changes15:03
fungimaybe i should rebase an affected change and the set votes on it as an administrator15:04
clarkbya I'm wondering if this is change specific somehow15:04
clarkbI wouldn't rebase. I would create an entirely new change15:04
clarkband if that changes things then maybe rebase15:04
clarkb(the change metadata is collected in a per change ref so if it is change specific rebase is less likely to change anything)15:04
fungi DNM: Testing submit requirements [NEW]15:06
clarkbok doesn't appear change specific15:07
fungii added +2 code-review, +2 verified, +1 workflow as an admin15:07
fungistill shows those as unsatisfied15:07
fungiabandoning and will try on bindep master15:08
clarkb is a heat master change that was fine15:10
clarkbits almost like this is branch specific (which you started investigating previously)15:10
opendevreviewJeremy Stanley proposed opendev/bindep master: DNM: Testing submit requirements
fungimaster branch examples were pointed out on a project with one change affected and another not15:11
clarkboh interesting15:11
clarkbah yup
clarkbwhat is extremely confusing is that it accurately shows the rules I would expect. So the problem isn't that the rules aren't applying or the rules are wrong. Its that for whatever reason it has decided the rule isn't satisfied15:12
fungisame behavior15:13
fungii've abandoned it now15:15
clarkbthinking out loud here: there were a series of changes. The first of which we made directly to all-projects and seems like stuff was happy at that point. Could a followup change have somehow broken things (seems more likely that things were just inconsistently broken at the start but exploring ideas)15:15
clarkbyou can see gets a vote at 2300 UTC ish esterday that satisfies the requirement15:17
fungithe copy conditions removal was the only production acl change to merge after all-projects was updated, right?15:18
clarkbfungi: ^ can you use your extra superpowers to remove ianw's vote and then apply a +2 code review for yourself?15:18
clarkbfungi: with a pause in the middle so that we can confirm the requirement is unsaitisfed after ianw's vote is removed15:18
clarkbya appears to be as far as the stack got15:20
clarkband that didn't touch opendev/system-config which also seems to exhibit this15:20
fungiit immediately shows the requirement as unsatisfied when i delete ianw's vote15:20
fungiconfirm you see the same15:20
clarkbfungi: yup I confirm15:20
funginow adding my own15:20
fungimine shows up satisfying it now15:21
fungigoing to test again15:21
clarkbthis has me going back to theidea it is change specific :/15:21
fungiyeah, so tested that change with both the webui and gertty, my cr+2 shows up as satisfactory15:22
fungiregardless of how i applied my vote15:23
clarkbthe other odd thing is it affects all the submit requirements on he affected changes. This implies it isn't rule specific15:24
clarkb(for example some projects do weird things with code-review like infra-specs and openstack/governance, but that doesn't seem to be in play here since verified and workflow are also affected)15:24
clarkbUNSATISFIED specifically means "The submit requirement is applicable (applicableIf evaluates to true), but the evaluation of the submittableIf and overrideIf expressions return false for the change." according to the docs15:25
clarkbthe submittableIf value is what you see in the condition output15:26
bbezakpretty recent occurrence (if that's helpful) -
opendevreviewMark Fedorov proposed opendev/git-review master: Add CC similarly to reviewers
clarkbok if this is it then its going to be the silliest thing ever15:30
bbezakdoes this issue only applies to changes pushed in relation chain?15:30
clarkbfungi: New Gerrit (either 3.7 or master) allows for boolean operators to be lower or upper case. The docs for 3.6 only show upper case being valid15:30
clarkbfungi: shows lower case being used for 'and' and not 'AND'15:31
clarkbfungi: in our copy condition change 'OR' is used15:31
clarkbanyway I'm looking at this becaues the docs say our submittableIf condition is not evaluating truthfully15:31
clarkbI have no idea how this worked before if this is the proble, but maybe we try changing the 'and' to 'AND' in All-Projects ?15:31
clarkbbbezak: no15:32
fungiseems worthwhile (and fast) to try15:32
clarkbbbezak: fungi reproduced with branch new changes on both stable/2023.1 and master on different projects with no relations to other changes15:32
clarkbfungi: are you in a position to make that update? I think ou've already escalated privs. I haven't even loaded my ssh keys yet15:33
fungii already removed my account from the groups but can readd15:33
clarkbcompare to for the case difference I'm talking about15:33
fungijust need to remember the fetch ref dance for all-projects config15:33
clarkbfungi: I think if you clone all-projects the default head may be refs/meta/config15:34
clarkbbut you would checkout refs/meta/config edit to s/and/AND/ (though mabe do by hand to avoid extra edits  its like three lines? ) and then git push HEAD:refs/meta/config15:34
clarkbI'm trying to test the query now15:36
fungihad to fetch origin refs/meta/config and checkout FETCH_HEAD15:37
clarkbya so if you do you don't get any changes that are having the problem. Switch to and then I think you get all the ones that are unhappy15:38
clarkbso ya I think maybe this is it15:38
clarkbhow this worked even a little bit I have no idea15:38
fungithis is what i have:
clarkbsorry kids are heading to school had to say bye. looking now15:40
clarkbfungi: yes three updated rules replacing 'and' with 'AND' is what I expected.15:41
fungithe push command always confuses me when pushing detached to an arbitrary remote ref15:43
clarkbI think in this instance `git push HEAD:refs/meta/config` would work since you're doing a normal fast forward using the locally checked out state15:44
clarkbbut you don't want to use HEAD if the local state is not what you want ot push15:44
clarkbfungi: can you let us know when it is pushed?15:45
fungifatal: You are not currently on a branch. To push the history leading to the current (detached HEAD) state now, use: git push HEAD:refs/meta/config HEAD:<name-of-remote-branch>15:46
fungigit changed a bunch of stuff around how push works, which is part of what's confounding me15:46
clarkbya I've never seen that before. I like when git tries to be helpful but ends up making thigns worse for people who figured things out previously15:47
clarkbfungi: maybe if you just checkout a branch state you'll be good?15:47
clarkbbasically stop being detached15:47
clarkbgit checkout -b update-meta-config should put you on a non detached state and then you can retry?15:47
fungissh: Could not resolve hostname head: Name or service not known. fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists.15:48
clarkboh sorry git push origin HEAD:refs/meta/config15:48
clarkbor whatever remote is the one that is gerrit15:48
fungi`git push origin HEAD:refs/meta/config` gets me farther, though my fungi.admin account apparently lacks permissions to do this by default15:49
clarkbare you in bootstrappers? YOu need that for the push15:49
fungi[remote rejected] HEAD -> refs/meta/config (prohibited by Gerrit: not permitted: update)15:49
clarkbadmin gets you like 80% of what you need tpically but when doing git push you also need bootstrappers15:49
fungiyeah, that's where i was headed next15:50
fungi27bdea8..5c92d15  HEAD -> refs/meta/config15:50
clarkbI think that was it. I just refreshed a change I had +2'd and was previously unsatisfied15:51
clarkbit is satisfied now15:51
clarkbI'll restore your bindep change to check it15:51
frickler also looks green now15:51
clarkbyup satisfied on bindep now too15:51
clarkbI have no idea why this was working at all15:51
fricklerbut why would this have affected only a subset of all patches?15:51
clarkbyes indeed wtf.15:52
fungifrickler: might be related to the warnings in the log for the changes that were merging? somehow gerrit didn't bother to evaluate the conditions on those?15:52
clarkbthe query links I posted above show you the difference between the two results15:52
clarkbthey are definitely different, but I'm not sure why/how it worked at all for some things. Maybe there was a transition point using the old rules15:53
clarkband the full reindex essentailly said no to that?15:53
fungi(the log entry example i quoted at 14:29z)15:53
clarkbwe should make a change to our system-config docs15:53
clarkbfwiw I focused on the condition and why it may not be satisfied since that is what the docs said was going on. Got a bit lucky that I seemed to recall newer gerrit made the boolean operators more forgiving and once I confiemd that checked if we had used the wrong version which we had. Then I checked the queries for deltas which existed and ya15:56
fungiso as far as getting things back on track, someone probably should query for any unmerged changes with verified +2 and recheck or reapprove them depending on the tenant15:56
fricklerhmm, this query shows some reviews with CR+1, not +2
clarkbcan also just enqueue straight to zuul if we want to do that15:57
clarkbfrickler: if you open the change directly they have max votes too15:57
clarkbnot sure why the summary shows hte lower value15:57
clarkbfrickler: I think the query results are accurate but the UI is buggy15:58
fungioh, right we could reenqueue to gate with zuul-client15:59
clarkbfungi: I guess rechecking is fine for openstack though since they have verified +2's already15:59
clarkbit won't clean check them which is what i was worried about16:00
fricklerhmm, didn't get submitted even after it received a fresh V+2 from zuul :-(16:00
fungii dunno, when i rechecked that affected heat stable/2023.1 change it went back to check16:00
fricklerbuildset result still stays MERGE_FAILURE
fungii wonder if there's some cache involved? or whether we need to force another reindex16:02
clarkbI would check the zuul logs for that chagne first as it should have more details on what exactly failed16:02
clarkband maybe the gerrit logs but yes I suppose that is possible if the check for mergablity by zuul is failing16:02
clarkbthough now that I've typed this I suddenly wonder if zuul needs explicit compatibility with submit requirements for mergability checking16:03
clarkb(again how did this work for a short time for some changes if so)16:03
clarkbfungi: I do notice that on the listing pages some of the X of Y values are stale compared to what is shown in the change page so ya could be some index is stale?16:04
clarkbdefintiely start with the service logs and work from there though16:04
fungii do suspect it's zuul deciding they can't merge without trying, because the gerrit error_log makes no mention of 876676 whatsoever16:05
fricklernot much more then this on zuul02 2023-03-09 15:50:09,133 INFO zuul.GerritConnection: [e: c322f3cd7eba42eb8e7b2a43f9df3e28] Conflict submitting data to gerrit, change may already be merged16:07
fricklerso this sounds like a response from gerrit to me16:07
clarkbfrickler: I think that the gerrit connection will log at a debug level the request and response it got before that?16:07
clarkbya it did16:08
fricklerthat just shows the post and no response16:09
clarkbdata: {}16:09
frickler is the whole context 16:09
clarkbI think data is the post data16:09
fricklerbut so zuul told gerrit to submit it, but it didn't happen16:10
clarkbthat says gerrit is returning a 40916:10
clarkbI would have expected gerrit to log that. But maybe because it isn't a 5XX it odesn't?16:11
fricklerwhere do you see that 409?16:13
fungii'm not finding 409 in the zuul scheduler debug log on zuul0216:14
clarkbfrickler: its the code that zuul emits that message for. And I just confirmed in the gerrit httpd log it was a 40916:14
funginot around the merge attempt for the change anyway16:14
clarkb(note you have to grep by the change id not not number as that is what zuul posts to)16:14
clarkbfungi: ya zuul converts the 409 to Conflict submitting ...16:14
fungigot it16:14
clarkband documents why that may happen16:15
fungi2023-03-09 15:50:08,055 DEBUG zuul.GerritConnection: POST:
fungiis what it tried to post16:15
fricklerhmm, o.k., maybe try to submit manually in gerrit and see what happens then?16:15
clarkbfrickler: ya is it safe enough on that change (no other changes trying to merge on that branch that you know of ?)16:15
clarkbI guess it is a release note too so should be fairl independent16:16
clarkbfungi: ^ did you want ot try that if you still have your admin account ready?16:16
fricklerat least I see the submit button in the UI, but greyed out for lack of priviledge. but that's better than earlier, where it was not shown16:17
fungii don't leave my account escalated, but pulling up the commands for submitting via cli now16:17
clarkblooks like some changes have merged since All-Projects was updated16:17
clarkbits possible we've got two independent things happening here?16:17
clarkbactually hold on one sec16:19
clarkbfrickler: your submit post came ~40 seconds before fungi reported the push completed16:19
clarkbcan we try rechecking it again to make sure we just didn't race each other16:19
clarkbfungi: ^ fyi on the holding off16:19
fungii was still putting the submit command together anyway16:20
clarkbfrickler: I'll let you do that since removing and applying the workflow vote should be quicker than a recheck16:20
fungii have the manual submit via gerrit ssh api ready to go if not16:22
clarkboh its behind another change16:24
clarkbbut ya I suspect this may have just been a race between the rules getting updated and zuul trying to submit16:24
clarkbparticualrly since I see other changes merged since the rule update16:24
clarkb876795 should merge soon (it has a slow py37 node though)16:25
clarkbnevermind that slow job timed out16:27
clarkbthere are two openstack releases changes that we can monitor that should merge in a few minutes16:30
opendevreviewClark Boylan proposed opendev/system-config master: Fix boolean operator in submittableIf rules
clarkbthere is the docs update. I'm going to go chekc the project-config side ofthings more thoroughly now16:34
clarkbI fetched the end of ianw's stack and grepped for ' or ' and ' and ' in submittableIf and copyCondition lines and found now results. ' AND ' has no results either but ' OR ' does16:37
clarkbI think All-Projects was the only place this happened16:37
clarkbthose two releases chagnes merged:
clarkbI really need to resume normal morning bootstrapping. Do we want to manually enqueue things or do a notice indicating people should recheck? I'm reasonably confident things should work now16:39
fungii should be able to put together a query of affected changes and reenqueue them, i don't think it will take too long16:40
opendevreviewMerged opendev/system-config master: Fix boolean operator in submittableIf rules
fungithat ^ merged fine16:47
fungigerrit query doesn't return data about the patch set number16:53
fungithat's going to make building the enqueue list harder16:53
Clark[m]Rechecks should be fine then16:54
Clark[m]We don't need to overthink it16:54
fungiseems there are 31 possibly affected changes16:55
fungi`gerrit query 'is:open -is:wip label:Verified+2 after:2023-03-08' --no-limit`16:55
*** jpena is now known as jpena|off17:00
fungigoing to run that now17:06
fungilooks like all the affected changes i could find were in the openstack zuul tenant, so made the string manipulation simple17:07
fungiand the script is done17:07
fungii see the changes in the gate pipeline now17:08
fungiand now i'm going to see about getting a shower and whatever else i missed by looking at the computer first thing when i woke up17:10
yoctozeptowhat was the issue?17:10
yoctozepto(with that not merging)17:10
yoctozepto(just curious)17:10
fungiyoctozepto: apparently when we switched to submit requirements late yesterday, there was a subtle typo in the config where boolean operators should have been written in all capital letters17:11
fungiso the end result was that it parsed the word "and" as a required commit message substring and logically or'ed it with the conditions on either side of it17:11
yoctozeptoooh, interesting17:11
clarkbfungi: logically AND'd not OR'd17:12
fungioh, could we have left the and out completely in that case?17:12
clarkbso basically you had to have a code review +2 and no code review -2. But you also needed to have 'and' in the commit message17:12
clarkbfungi: yes AND is the default. I think being explicit here is a good thing though so that we avoid any changes to those defaults17:12
yoctozeptooh my17:12
yoctozeptobut how did the docs change fix the issue? :D17:13
clarkbsome changes did have 'and' in the commit message which is why they worked leading to all the confusion17:13
yoctozeptoI am getting the issue now, just not the fix17:13
clarkbyoctozepto: it didn't. All-Projectsis not automatically managed. fungi manually applied the documented change in the docs to All-Projects. But the docs capture the situation17:13
yoctozeptoyeah, that "and" is really nasty17:13
fungithe all-projects config is managed by manually pushing commits into gerrit, not through automation, that document is just a reflection of what we push17:13
yoctozeptooooh, I see17:14
fungianyway, we were doing this in preparation for upgrading to gerrit 3.7 which doesn't support the old way of specifying requirements, and 3.6 supports using the new syntax. 3.7 apparently is more lax about accepting lower-case boolean operators, but 3.6 unfortunately is not17:14
yoctozeptowhat is all-projects btw? some hidden repo?17:15
clarkbit is one of two central config repos for gerrit17:15
fungiyoctozepto: yes, it's the repository in gerrit which holds the master acl that all other projects inherit from17:15
clarkband yes, I don't think it is very visible anymore17:15
yoctozeptoI see, pretty mysterious17:15
yoctozeptothanks for your explanations17:16
fungimy pleasure17:16
clarkbI should find a shower too. I did manage to eat and drink something though so thats a win17:17
clarkbhow does this look #status notice Yesterday's change to Gerrit configs to use submit-requirements had a boolean logic bug. This has not been corrected and any changes that did not merge as a result can be rechecked. We have rechecked the changes we identified as being affected.17:19
clarkb*this has now been corrected17:19
clarkbNow we need to turn 'and' into a secret handshake somehow17:21
fungi/srechecked/reenqueued/ technically?17:21
clarkbfungi: ++17:21
clarkbhow does this look #status notice Yesterday's change to Gerrit configs to use submit-requirements had a boolean logic bug. This has now been corrected and any changes that did not merge as a result can be rechecked. We have reenqueued the changes we identified as being affected.17:21
clarkbsorry thats an edited version with those two eidts17:22
fungidon't want people confused by a lack of recheck comments on those changes17:22
clarkb#status notice Yesterday's change to Gerrit configs to use submit-requirements had a boolean logic bug. This has now been corrected and any changes that did not merge as a result can be rechecked. We have reenqueued the changes we identified as being affected.17:22
opendevstatusclarkb: sending notice17:22
-opendevstatus- NOTICE: Yesterday's change to Gerrit configs to use submit-requirements had a boolean logic bug. This has now been corrected and any changes that did not merge as a result can be rechecked. We have reenqueued the changes we identified as being affected.17:22
clarkb fyi this is likely to affect openstack's gerrit replication but also many of us have github accounts17:23
opendevstatusclarkb: finished sending notice17:25
clarkb did end up merging17:28
clarkbas did
clarkb somehow I have ended up here and find it really interesting that the word "and" isn't super common in ASL17:36
yoctozeptoclarkb: as it can also be omitted in gerrit and multiple other similar places - makes sense to me17:39
yoctozeptoseemingly humans default to "and"17:40
clarkbya I guess when you list things it is purely extra information that can be omitted17:40
fungithat is indeed truly interesting18:22
fungiapparently it's fine as a conjunction though18:23
clarkbI've managed to get myself distracted with passport things this morning after the submit-requirements stuff. I need to reset my day and will probably pop out ofr a long lunch but when I get back will start looking at gitea db stuff19:04
fungiyeah, i'm now quite behind on other things i meant to be working on, but that's life in opendev19:05
opendevreviewMerged opendev/system-config master: Update gerrit image builds for 3.6.4 and 3.7.1 tags
ianwomg so the only thing that merged yesterday had "and" in the commit msg?!  jeez sorry about that20:24
ianwnone of the follow-ons have binary conditions (
ianwi made a similar mistake with is:True as well, being a bit too pythonic -- that has to be is:true20:30
Clark[m]Yup we made a secret merge handshake :)20:30
yoctozeptobtw, any plans to merge the NebulOuS patches?20:37
Clark[m]I've been meaning to take a look then ran into gitea fun yesterday and Gerrit todaym I'll try to look again today after doing the gitea stuff20:38
Clark[m]But no need to wait for me either if others get to it first20:39
fungiin rax-ord news, things seem to have improved but there were still some ugly periods of what are probably more launch timeouts even with the 15-minute wait20:41
fungiseeing a correlation with ~0 in use/high building with increased incidence of error node launches and higher time to ready on the graphs20:43
opendevreviewClark Boylan proposed opendev/system-config master: Add gitea13 and 14 to Gerrit replication
opendevreviewClark Boylan proposed opendev/system-config master: Add gitea13 and gitea14 to the haproxy load balancer
clarkbinfra-root ^ that first change should be good to go (but please do take a minute to double check the hosts look good to you)22:24
fungiwhat do people use for listing listening sockets these days? seems netstat doesn't come installed any more22:24
clarkbI'll WIP the second to prevent it from merging before we complete replication22:24
clarkbfungi: `ss` is the new thing22:24
fungii see. unfortunate that it seems to assume people use a mile-wide terminal22:25
clarkbalso I transplanted the db from gitea09 to 13 and 14 to prove out the transitivity. Seems to have worked from what I can see22:27
fungiyeah, seems to be working based on my tests22:28
opendevreviewMerged openstack/project-config master: gerrit/acl : submit-requirements for deprecated NoOp function
clarkbianw: I meant to ask if the earlire manage-projects run for the system-config inventory update did udpate the acls or if it waited for a subsequent run22:28
ianwclarkb: it did seem to have to wait for the subsequent run22:29
ianw was where it eventually applied22:29
ianwi've approved the next one now and will watch that too22:30
clarkbcool fwiw I did double check that the project-config updates didn't have the same boolean fun22:30
ianwoh, it just merged, 87580422:30
clarkbI could only find evidence of the problem in the all-projects side of things22:30
ianwyeah, luckily no and's in that stack, i double checked too :)22:31
ianwthere's no queue for ^^^ to apply22:31
ianwwell just the last hourly job to finish22:32
clarkband the ors are all OR so we won't search foo AND or AND bar22:32
ianwi did write up a little follow-on
opendevreviewIan Wienand proposed opendev/system-config master: Refactor adns variables
clarkbI've approved the replication change. Hoping that if I can trigger that today then sometime tomorrow we can put them behind haproxy22:40
opendevreviewMerged opendev/system-config master: Add gitea13 and 14 to Gerrit replication
ianwsomething like seems right to me22:59
clarkbI +2'd the nebulous changes but didn't approve them yet. FIrst I wanted to double check if fungi wanted to review them (also we want the first to land and apply before landing the second). And I've got gitea13 and gitea14 bootstrapping right now and while I don't think adding a new project at this stage will create problems since those hosts are fully managed with the other giteas23:04
clarkbthey just dont' have data I wanted to see if anyone else thought that mght be a problem23:04
opendevreviewSteve Baker proposed openstack/diskimage-builder master: A new diskimage-builder command for yaml image builds
opendevreviewSteve Baker proposed openstack/diskimage-builder master: Switch from disk-image-create to diskimage-builder
opendevreviewSteve Baker proposed openstack/diskimage-builder master: Document diskimage-builder command
ianwclarkb: i feel like it will be fine, as you say the creation should happen23:23
clarkbI'm replicating bindep ti 13 and 14 now (its been my first replication project canary)23:33
clarkbThat appears successful. I've asked the plugin to replicate everything to those hosts now23:36
fungii can take a look shortly23:42

Generated by 2.17.3 by Marius Gedminas - find it at!