Friday, 2023-04-07

Clark[m]I see virtualpdu changes in search now00:09
fungimanage-projects succeeded
fungi2023-04-07 00:04:18,879: jeepyb.utils - INFO - Executing command: git --git-dir=/opt/lib/jeepyb/openstack/virtualpdu/.git --work-tree=/opt/lib/jeepyb/openstack/virtualpdu push ssh:// HEAD:refs/meta/config00:29
fungidid we expect that?00:29
Clark[m]Yes the file was updated in one of them iirc00:30
Clark[m]It added the meta openstack parent config iirc00:30
Clark[m]But double check. The change for that project?00:30
fungiyeah, i'm grabbing it00:31
Clark[m] ya it changed so that should be correct 00:31
fungithat looks like it matches00:34
fungii need to step away for a few00:35
fungii think i accidentally closed the screen session on bridge, but i think we were done in it anyway?00:36
fungialso `gerrit show-queue --by-queue` doesn't have any more replication tasks00:37
Clark[m]Ya I already detached00:37
fungiwhat's step 21 about removing autoholds?00:37
fungibut otherwise i think we're at post-maintenance fixups00:37
Clark[m]I think that's for the test node(s) that were held00:38
fungioh, for the upgrade testing00:38
fungigot it00:38
fungiokay, stepping away, but will be back soon00:38
opendevreviewMerged opendev/system-config master: Upgrade Gerrit to version 3.7
ianw^ that has deployed01:09
ianwok at the bottom of i've added everything i think we need to follow up on01:12
ianwi don't think anything is super urgent01:13
clarkbthanks! I'll take a look in the morning. Other than the todos everyhting is done at this point right? Looks like it from my read of scrollback01:36
fungiyep, that's my take on it01:37
JayFI have not read all backscroll, so apologies if this is a repeat, but it looks like was archived at some point, and is still showing old content/redirects01:51
JayFit's not urgent, and things look right on opendev, but something else to follow up on perhaps01:51
fungiJayF: this was a move out and then back into the openstack namespace i guess? i think some of the tc members have github accounts with access to do the unarchiving01:57
*** Trevor is now known as Guest1022401:57
fungioh, i see, it got transferred to
fungiso we'd need to un-transfer it maybe02:02
ianwJayF / fungi: hrm, i tried logging in as what i thought was our user that would own that, but doesn't seem they do.  so might have to do some digging to find the right way to unarchive that04:55
*** TheMaster is now known as Unit19308:54
*** elodilles is now known as elodilles_pto10:11
JayFIIRC that used to require a github support round trip13:27
JayFto get a repo redirected13:27
fungiwhen we added the redirects, we did it via github's feature to transfer repositories between orgs14:04
noonedeadpunkfolks, is it possible to get an auto-hold for 879732 on job openstack-ansible-deploy-aio_metal-rockylinux-9 ? GPG thing again, but now I can't reproduce that in any way localy and it looks related to the patch... And what's more interesting is that lxc job is passing, so sounds like smth specific to the image....14:55
clarkbnoonedeadpunk: ya I'm sitting down now and will load ssh keys and do that15:12
noonedeadpunkssh keys btw:
noonedeadpunkah. damn, just failed 15:14
clarkbnoonedeadpunk: do you want to recheck it?15:14
clarkbok the hold is placed with zuul15:16
noonedeadpunkyup, decided to leave this only job not to waste resources15:17
clarkbinfra-root one thing I've noticed on jammy static02 is that the ubuntu universe package for openafs-client is slightly newer than that in our ppa so we are not using our ppa client. I don't think this is a problem but wanted to call it out15:19
clarkbinfra-root I'm going to test all of the names I identified as being hosted by static with local /etc/hosts overrides and if that looks good I'd like to land the DNS updates to switch the CNAME over to the new server15:19
clarkbthen I'll start looking at gerrit things. I'll probably start by writing that upstream bug for the migration thing.15:19
clarkb is the list of names I'm checking15:21
clarkbok all of the names in ^ lgtm when overridden in local /etc/hosts. I would like to merge and
clarkbinfra-root ^ let me know if you think that is safe at this point. I think the main risks are that I've missed some names somewhere (there are a lot of them) or that the ubuntu upstream afs package is unhappy somehow (it has had a lot less trial by fire than our version but the versions are similar)15:36
clarkbI think if we land those three and I update records we can stop apache on the old server then wait a while before we start removing old stuff15:36
clarkbwhich means it is easy to rollback if necessary15:36
fungiusing newer openafs from distro sounds fine (even better) to me, since our ppa was to deal with too-old openafs in ubuntu lts15:45
fungiclarkb: spot checks with dns overridden look right to me15:45
opendevreviewClark Boylan proposed opendev/system-config master: Extra debugging around gerrit config and cache diffs
opendevreviewClark Boylan proposed opendev/system-config master: Fix gerrit 3.7 config diff
clarkbI've added ^ to the todo section of our etherpad to track that15:48
clarkbfirst change is made to get more info out of our checks and second one corrects the delta. I'm hoping we can fix whatever the issue is in the checking and then squash the two together for merging15:48
clarkbfungi: cool you're good with me approving the dns updates now then? or do you want to do it?15:49
noonedeadpunkclarkb: job has just failed :)15:50
fungithat was nice and fast15:50
fungiclarkb: i can approve them, just pulling them up now15:50
fungii think i won't be able to approve 879782 until 879780 merges (or do they share a queue?)15:52
clarkbfungi: ya may have to be one after the other15:53
clarkbI don't recall if they share queues15:53
clarkbnoonedeadpunk: looking15:53
opendevreviewMerged opendev/ master: Update static.o.o CNAME to point at static02
hasharclarkb: my warm congratulations for the Gerrit 3.7 upgrade \o/15:54
clarkbnoonedeadpunk: root@
fungithanks hashar!15:55
noonedeadpunkthat worked, thanks!15:56
fungiianw did all the hard work15:56
hasharwell congratulations to ianw as well so :-]]]15:56
clarkbthis is the first time we're running the latest version of gerrit since like 2013/2014 maybe?15:56
clarkb(if you exclude the 3.8 rc0 that was just made a few days ago)15:57
fungii'm definitely going to exclude that, so it doesn't ruin the party15:57
clarkbya I don't think release candidates count :)15:57
fungii've decided that they definitely don't15:58
hasharwell 3.8 is going to be out soon :]15:59
clarkbya I know so we'll enjoy this for a couple of weeks while it lasts :)15:59
opendevreviewMerged opendev/ master: Update zuul dns records to the new static02 server
clarkbfungi: since you did the previous All-Projects surgery to fix and/AND what are your thoughts on essentially reverting the copyConditions from the migration?16:06
clarkbsince trivial rebase inlcudes no change we don't need both of them under code-review and alraedy had trivial rebase. And  Idon't think we want the copy conditions for verified or workflow set. I think that means a straight up revert with a commit message indicating why is a good next step there16:06
fungiis it going to undo them, or was that a one-time "upgrade" of the config?16:06
clarkbfungi: like sql db migrations it tracks what version it is at and won't reapply that migration as long as our recorded version is >= to the current value16:07
clarkbeffectively making it a one time thing16:07
noonedeadpunkthat is really interesting thing...
noonedeadpunkHow gpg check might fail, but if retry - it will pass....16:07
clarkbif you look at the revert notes in etherpad one of the steps is resetting that migration version value in notedb16:08
clarkbfungi: thats line 273 if curious16:08
clarkbnoonedeadpunk: does dnf/yum have some sort of refresh command?16:08
clarkbI wonder if doing that would resync things16:08
noonedeadpunkwell, re-runing playbook doesn't fix this as well as `dnf clean all` doesn't help16:10
noonedeadpunkI will play around with that for a while I guess... As I'm quite o_O right now16:11
fungilooking around, it seems once upon a time ansible didn't check package signatures with dnf, and then maybe added their own checker to solve that?16:12
fungiah, no, they just enabled it in dnf's config i think:
noonedeadpunkwell, asnible tend to re-write package managers logic, true... But I kinda see issue even without ansible in that image16:14
fungiunfortunately a web search on the vague error there turns up every conceivable reason someone might encounter signature verification errors16:15
fungicentos was also hitting a similar error this week, right?16:18
noonedeadpunkWell, yes, it was. It was more consistent and caused by gnupg2-2.3.3-316:20
noonedeadpunkAnd here is gnupg2-2.3.3-216:20
fungithough looking at the paste, i wonder if there's something racing the key addition, such that accepting the key only causes it to be checked on subsequent runs. maybe adding the archive key before installing packages would work?16:21
fungifor adding external debian repositories that's basically the only way to do it, because they sign the package indices instead of individual packages, so you can't even successfully pull the package list for a repository if the key hasn't been added to the package manager's keyring in advance16:22
noonedeadpunkyeah, might be... Btw, while I can reproduce paste outside of the nodepool VM, it doesn't hurt ansible run somehow16:22
clarkb file for the copyConditions issue16:24
noonedeadpunkIt actually looks like even more progressive issue then centos had, as you could workaround gpg import with `rpm --import` and here it refuses to do so16:26
clarkbNeilHanlon: may have ideas16:27
noonedeadpunkyep, pinged him already16:27
clarkbthe diff of gerrit.config is empty in our test job
fungiinfra-root: i have this prepared i can push to all-projects:
clarkbfungi: that diff looks great to me and thank you for the extra context in the commit message should we ever need to spelunk this in the future16:37
fungii improved the wording in the commit message a little more for added clarity16:38
fungii feel like there's probably no other infra-root around to object, so i'll go ahead and push it. can always revert the revert if there are concerns raised later16:40
fungiand done16:42
fungii had to temporarily add my fungi.admin account to the project bootstrappers group16:43
clarkbthank you!16:43
fungiapparently it had perms to pull but not to push?16:43
clarkbya read access to admins but push access is bootstrappers iirc16:43
clarkbcan you update the etherpad with that paste too just to captuer that?16:43
opendevreviewClark Boylan proposed opendev/system-config master: Extra debugging around gerrit config and cache diffs
opendevreviewClark Boylan proposed opendev/system-config master: Fix gerrit 3.7 config diff
clarkbfungi: the git-review output from gerrit ssh push tells you that votes are removed too16:47
clarkbthey would've been removed before your all-projects update since I actually made changes but thats neat16:47
clarkboh wait no this was on the child change which had no delta so that confirms your update is working16:47
fungiclarkb: for any other acls that may have acquired that addition, i expect a full manage-projects run will correct it, yeah?16:48
clarkbfungi: yes16:48
clarkbI think before we run that it might be worth spot checking to see what impact there is if any then plan to do that next week? I know ianw did it recently so should be familiar with the process16:49
fungijust a reminder, i won't be around to help next week, not that it sounds all that complicated16:50
clarkbya I think we've got it. Just need to spend a bit more time understanding the impact outside of all-projects16:50 static records are updated as are the zuul domain records. has not updated. fungi want to approve too?16:52
fungican do16:52
clarkband as far I can tell things are working16:52
fungialso, here's the openstack/releases acl migration update for an example:
fungilooks like indentation has changed too?16:53
clarkbI'll get records updated at some point today as well then put static01 in emergency file and shutdown its apache process16:53
fungi-requireContributorAgreement = true16:53
clarkbfungi: it added it back with indentation16:54
fungioh, so it did. i cancelled the pager without scrolling16:54
clarkbI think the only functional chagne in that file is the changekind:NO_CHANGE addition which is a noop16:54
clarkbconsidering that gerrit seems to think the tabbed version is canonical we might want to switch to that so that future migrations that may occur have easier to read diffs16:55
fungibut yeah, mainly confirming that it did alter the acls16:55
clarkbbut I think as far as functionality goes for right now we are ok16:55
fungiso definitely some impact outside all-projects but yes i doubt it's going to be significant16:55
clarkbI'll need to think through what a convert everything to tabbed indentation update looks like16:57
clarkbI think we will want ot incorporate that into the run mangae-projects by hand run so that we don't have to do that multiple times16:57
fungii think we can just change the normalizer and then accept the next normalization change16:57
clarkbbasically update our "linter" and all the files then do manual run16:57
fungior use it to generate the diff16:58
fungiall the original acl cleanup i did was via the normalizer, for precisely that reason16:58
clarkbcool so do that then do the manual run to fixup the copyConditions16:58
clarkbshould be something we can put together next week16:58
fungii would maybe alter transformation #1 since it's what currently unindents everything17:00
fungii'll see if i can hack that together real quickly17:01
clarkbcool I've put some notes about this into the etherpad too17:01
opendevreviewJeremy Stanley proposed openstack/project-config master: Indent Gerrit ACL options
clarkbfungi: what is interesting is that our all-projects config seems to be space indented not hard tabs. And it didn't undo that so whatever formatter it uses seems to respect tabs vs spaces but seems to default to tabs if there are none?17:10
funginow that i've written that ^ i'm not sure whether it's a good idea. lots of people are probably going to struggle with the enforced hard tab indentation, using editors which may end up using spaces magically instead17:10
fungioh! interesting. i'll switch to four space tabs17:10
clarkbfungi: I'm sad that this is a liekly problem :/17:10
clarkbI've configured my editor so that I can be specific about it and get proper annotations to distinguish between the two. But ya especially if you open the jenkins code base its a huge mess of mixed tabbing for example17:11
clarkbfungi: oh wait your paste for the rever tand the paste from yesterday showing the change differ in tabs vs spaces17:12
clarkbso now I'm more confused17:12
clarkbmaybe ianw's copy pasting converted them for us but your rever is actually tabs?17:12
clarkbthis probably need more investigating and testing unfrotunately17:12
fungii pushed the change as wip regardless17:12
fungibecause it will still need to be run against all the acl files to create a diff if/when we're ready to do that17:13
clarkbhonestly, it shouldn't be difficult for people to use hard tabs or spaces depending on which is required. But I agree that certainly there will be trouble17:13
noonedeadpunkclarkb: you can remove hold now, many thanks!17:17
clarkbnoonedeadpunk: will do. Did you manage to sort it out?17:18
noonedeadpunkwell... not the mistery why thi key didn't work, but I realized that gpg key from other mirror was matching as well (somehow)17:18
noonedeadpunkso just using gpg from repo a to verify repo b... or well, since it's packages that are signed, might be it's not that a big deal... but still nasty17:20
clarkbhrm i don't know if updated dns properly. I don't see any pending jobs for it and if I dig against our nameservers I get the old values back17:20
noonedeadpunkalso Nail has grabbed data to reproduce the setup if needed17:20
clarkbautohold delete has been issued17:21
clarkbbased on my latest patchset to the gerrit config debugging thing the quotes are removed on the old side and the new side of the diff. I think this is because we write gerrit.confg and then start gerrit 3.6 then it normalizes things. Then we capture the config for the old side17:24
clarkbI think we need to capture the configs before starting the old gerrit side but sorting that out in a generic way that doesn't leak into production deployments might be a pain. Ill look into that though17:24
clarkbmaybe we can run the playbook without starting anything, capture the logs, run the playbook and start things, then do upgrade17:25
clarkb* capture the configs17:25
opendevreviewJeremy Stanley proposed openstack/project-config master: Indent Gerrit ACL options
opendevreviewClark Boylan proposed opendev/system-config master: Extra debugging around gerrit config and cache diffs
opendevreviewClark Boylan proposed opendev/system-config master: Fix gerrit 3.7 config diff
clarkbif that works it will make the job take a little longer but hsould be more robust diffing17:43
clarkbI don't think the change triggered jobs to update the nameservers. I suspect this will self correct during our daily job runs and I'm happy to followup on this next week since I'vegot enough other stuff going on I can focus on today17:51
clarkbjust a heads up to what I think happened there. The jobs didn't run at all not jobs failed17:51
clarkbthe bulk change update abilities in gerrit 3.7's web ui are a nice update. Gerrit is adding Gertty features now. corvus should be proud17:55
fungiclarkb: oh! the cross-tenant triggering problem i guess?17:56
clarkbfungi: either that or we don't have the jobs declared at all for some other reason?17:56
clarkbbut ya I think it will correct itself when daily jobs run?17:56
clarkbwhich means I can finish things up next week. I'm not in a massive rush17:56
clarkbok getting closer with the commit diffs. Capturing the file before starting gerrit 3.6 shows it with ""s then starting 3.6 removes them. But the diffing didn't fail the job for some reason18:17
clarkbI think I see it this is fun18:18
opendevreviewClark Boylan proposed opendev/system-config master: Extra debugging around gerrit config and cache diffs
opendevreviewClark Boylan proposed opendev/system-config master: Fix gerrit 3.7 config diff
clarkbset -o pipefail is necessary because of the tee capturing the diff for logging18:22
clarkbassuming the parent fails and the child succeeds I'll work on condensing those two into a mergable change18:22
opendevreviewJeremy Stanley proposed openstack/project-config master: Deploy DNS changes
fungiit was already in the openstack tenant apparently18:47
Clark[m]I'll take a look after lunch18:50
fungiit's in no way urgent, just thought i'd push a patch since i bothered to look19:02
opendevreviewClark Boylan proposed opendev/system-config master: Fix gerrit upgrade config diff checking
clarkbI think ^ is now in a mergeable state and that should address this going forward20:59
clarkbfungi: I didn't approve 879910 only because it wouldn't trigger the job we want now so waiting for a second reviewer doesn't seem like a big deal21:01
clarkbbut feel free to +A if you like to keep one thing off our backlog before being afk21:01
ianwthanks for looking at the all proejcts21:07
ianwone thing i couldn't determine, was how do you actually get into a situation of pushing something that is NO_CHANGE?21:07
clarkbianw: only updating the commit itself maybe? liek a cherrypick and push?21:08
clarkbmaybe upstream can help clarify that on the bug I filed21:08
ianwyeah, i mean you'd think cherry-picking and pushing would change the parent?21:08
ianwi agree on the tabs; it seems gerrit really likes those for config files21:10
clarkbbut also enjoy your weekend. I think everything is mostly happy at this point. I ran down the diff thing and got that into shape. fungi updated All-Projects. Gerrit seems happy. static02 is half into production now too tldr no reason to interrupt your weekend21:11
clarkboh someone else noted that they have observed similar with replication. Hoepfully we can run it down together. But they are enjoying the holiday weekend so won't get to it any quciker than we will21:12
ianwyep and i saw that the replication issue was half-known about 21:12
ianwheh, yeah :)21:12
ianw++ thanks!  this was certainly an interesting upgrade cycle :)21:13
clarkband thank you. I think that went reall well at the end of the day. We kept the outage period within the block we notified about. No accidental project (re)creations and everything that did go unexpected has been minor21:15
clarkbI'm noticing all the little UI updates in gerrit and I feel like for once I have yet to find one I don't like. The submit requirements view conditions expander now shows you how each individual condition evaluates23:32
clarkband bulk topic updates are going to be great23:32
opendevreviewMerged openstack/project-config master: Deploy DNS changes
fungii'm still not convinced the labels are being cleared correctly. looking at 879906 when i pushed a revision which just altered the commit message, gerrit added a note that says "outdated votes: verified+1"23:34
fungidoes that mean it was cleared by the upload, or carried over?23:34
clarkbI believe cleared23:36
clarkbits letting you know the vote was outdated and not applicable to the new patchset23:36
clarkbbut you'd have to catch it between when zuul applies its latest CI run to be sure. I am prett sure I checked on one of my patchset updates earlier today though23:36
fungiif so, then cool. just sort of vaguely communicated in the comment it added23:36

Generated by 2.17.3 by Marius Gedminas - find it at!