Thursday, 2025-05-15

opendevreviewOpenStack Proposal Bot proposed openstack/project-config master: Normalize projects.yaml  https://review.opendev.org/c/openstack/project-config/+/94982402:19
mnaserwent back and forth a bit with setuptools maintainers02:35
mnaserthey thought they had it fixed but missed our pbr use case :)02:35
mnaserso should be all good now02:35
mnaserand 80.7.1 was tagged 14 minutes ago02:36
JayFmnaser: maybe post an update in the ^ above linked PBR bug too? 02:40
Clark[m]JayF that linked PBR issue is unrelated03:45
Clark[m]The issue today was a bug on setuptools part that happens to affect pbr packaged projects but doesn't require PBR to break. It was just buggy code. The PBR issue you linked is for planned cleanup of deprecated code that will break PBR03:46
opendevreviewMerged openstack/project-config master: Normalize projects.yaml  https://review.opendev.org/c/openstack/project-config/+/94982404:40
opendevreviewMichal Nasiadka proposed openstack/project-config master: kolla: Allow kolla-core to remove RP votes  https://review.opendev.org/c/openstack/project-config/+/94984205:34
opendevreviewMichal Nasiadka proposed openstack/project-config master: kolla: Allow kolla-core to remove RP votes  https://review.opendev.org/c/openstack/project-config/+/94984205:43
opendevreviewMichal Nasiadka proposed openstack/project-config master: kolla: Allow kolla-core to remove RP votes  https://review.opendev.org/c/openstack/project-config/+/94984205:57
opendevreviewMichal Nasiadka proposed openstack/project-config master: kolla: Allow kolla-core to remove RP votes  https://review.opendev.org/c/openstack/project-config/+/94984207:02
opendevreviewMichal Nasiadka proposed openstack/project-config master: kolla: Allow kolla-core to remove RP votes  https://review.opendev.org/c/openstack/project-config/+/94984207:02
*** ykarel_ is now known as ykarel11:41
opendevreviewMerged openstack/project-config master: Add collections required by OpenStack-Ansible to github projects  https://review.opendev.org/c/openstack/project-config/+/94932513:29
opendevreviewMerged opendev/zuul-providers master: Build images daily  https://review.opendev.org/c/opendev/zuul-providers/+/94980613:56
opendevreviewJames E. Blair proposed opendev/zuul-providers master: Remove max-ready-age  https://review.opendev.org/c/opendev/zuul-providers/+/94989613:58
opendevreviewMichal Nasiadka proposed openstack/diskimage-builder master: CI: Use opendevmirror/ubuntu for functests  https://review.opendev.org/c/openstack/diskimage-builder/+/94989714:01
fungii just realized that i've not been on matrix since saturday when my shell server got unexpectedly rebooted in flex and the python-based weechat-matrix plugin started throwing an exception deep in pyca/cryptography14:05
fungii guess an update at some point broke it, but since that plugin is abandoned anyway i'm going to take this opportunity to try out the (wip) rust rewrite14:07
opendevreviewTony Breeds proposed openstack/diskimage-builder master: WIP: Add support for CentOS Stream 10  https://review.opendev.org/c/openstack/diskimage-builder/+/93404514:10
clarkbI'm starting to put the gerrit 3.11 upgrade process and notes together in https://etherpad.opendev.org/p/gerrit-upgrade-3.1115:31
clarkbGerrit 3.11 adds a rest api endpoint to bulk abandon changes based on their age and whether or not they are mergeable15:42
*** kevko6 is now known as kevko15:49
opendevreviewClark Boylan proposed opendev/system-config master: Test gerrit.requireChangeForConfigUpdate=false  https://review.opendev.org/c/opendev/system-config/+/94991416:06
clarkbI'm trying to test if we can make the new install test setup for Gerrit 3.11 match the current 3.10 behavior of not requiring code review for project config updates. That way we can get consistent behavior in testing and in prod before and after the upgrade (mostly as a belts and suspenders we won't run into problems with acls after the upgrade)16:12
fungiwouldn't be a bad idea to up the default gerrit version in git-review's tests too16:14
fungii'm approving some pending changes for it, and will then update the testing16:15
clarkblooking at the gerrit upgrade I think the upgrade process itself is actually the simplest one we've had like ever. There are no index or schema updates16:16
clarkbso all the effort is in considering all the changes to other things (like acl updates needing code review by default and new apis and so on)16:16
clarkbwhile thats running I think we should consider https://review.opendev.org/c/opendev/system-config/+/882900 (moving gerrit images to quay) and https://review.opendev.org/c/opendev/system-config/+/949778 updating gerrit images to the latest point releases. We can land them together and do a single restart. Or do them separately if we think there is sufficient concern to spread them16:31
clarkbout. Mostly looking for feedback on that. I think it wouldn't be crazy to split them up and do two restarts just due to the previous issues we had with moving to quay16:31
fungilooks like i'm already +2 on both of those16:33
fungioh, or i was until the rebase16:33
funginever mind, it carried over16:34
opendevreviewMerged opendev/git-review master: Allow specifying remote branch at listing changes  https://review.opendev.org/c/opendev/git-review/+/87298816:36
opendevreviewMerged opendev/git-review master: Show remote side colors when color is enabled  https://review.opendev.org/c/opendev/git-review/+/91400016:36
opendevreviewMerged opendev/git-review master: Fix TypeError when reporting old git version  https://review.opendev.org/c/opendev/git-review/+/94714516:36
opendevreviewJeremy Stanley proposed opendev/git-review master: Update higher Gerrit versions used in testing  https://review.opendev.org/c/opendev/git-review/+/94991616:37
corvusthe main annoyance i'm seeing for the zuul-launcher right now is the infinite loop trying to download images that have expired (so that it can upload them to new clouds).  that should be addressed by building new images periodically, or manually building new ones.16:42
corvusa change to build images daily has merged, which will help16:42
corvusi'm also slowly manually triggering image builds, which will help16:42
corvusand a change to zuul to allow more than one image build to be enqueued at a time should merge soon, and that should help speed up the process of manually building them.16:43
corvusfungi: https://zuul.opendev.org/t/opendev/build/25ccf5103d3342659319cd414f8fa0ad happened to notice that failure for your git-review/gerrit change16:45
opendevreviewMichal Nasiadka proposed openstack/diskimage-builder master: WIP: Add support for CentOS Stream 10  https://review.opendev.org/c/openstack/diskimage-builder/+/93404516:45
fungithanks! yeah, i was going to dig into that one16:48
fungii guess i need to pull a more recent jre in the job16:49
opendevreviewJeremy Stanley proposed opendev/git-review master: Add CC similarly to reviewers  https://review.opendev.org/c/opendev/git-review/+/84921916:56
opendevreviewJeremy Stanley proposed opendev/git-review master: Update higher Gerrit versions used in testing  https://review.opendev.org/c/opendev/git-review/+/94991616:56
fungiwe'll see if openjdk-21 is too new for gerrit 3.10, hopefully not16:57
clarkbfungi: I don't think it is16:58
clarkbsetting gerrit.requireChangeForConfigUpdate to false does not allow us to continue to directly push refs/meta/config on 3.11 as we did with 3.1017:02
clarkbI've asked on gerrit discord what the purpose of this config item is in that case17:03
fungithat could make project creation and acl updates challenging17:03
clarkbfungi: ya the workaround we were using was you update All-Projects acls to allow force push17:04
fungimmm, okay so not a regression then17:04
clarkbif you allow force push then you can still do the old thing. But they documented this config option and not adding force push perms so I figured I'd try to documented path (also its easier for us to apply)17:04
fungigot it17:04
clarkbso I'm a bit stumped as to why the config option exists at all17:04
fungiless worrisome and more head-scratching17:05
clarkbbut if we are concerned we can manually update our all-projects acls on 3.10 prior to upgrading. I've asked them to clarify if this new restriction will apply to installs that upgrade from 3.10 to 3.11 or if it is only for new installs as well (we can also test this as part of upgrade testing)17:05
clarkbthis is a situation much like the recent pbr struggles. We've had this functioanlity for over a decade and now that the world has decided we aren't crazy and it is useful they've implemented it in a different manner without asking for input from those of us that have been doing it forever and in the process have broken us17:06
clarkbat least in this case we know of the workarounds upfront its just a matter of reconciling that information with the documentation as I'm digging into it17:10
clarkbfungi: https://opendev.org/opendev/system-config/blame/branch/master/doc/source/gerrit.rst#L227 I think this may have us covered already fwiw17:12
clarkbthen in testing the issue is we're using the admin account which is part of administrators which doesn't come with force push to refs/* by default17:13
fungiyeah, we'd need something akin to our project-bootstrappers group17:14
clarkbthe workaround we have been using is to add force push to refs/* for administrators via a proposed change that we immediately approve17:15
clarkbI was tryign to stop needing to do that since we won't do that in production. But ya I guess an alternative is to setup bootstrappers in the ci jobs instead17:16
fungiinfra-root (and anyone else interested): i'm looking at trimming the content on our splash page at https://opendev.org/ and wondering what do you think about maybe moving the faq into infra-manual and linking to it? and what about making the "how is development different..." section of the page part of the faq (or at least some of it, and just have a shorter paragraph about gerrit,17:38
fungimaybe something similar for the ci/zuul section too)?17:38
clarkbI like the idea of trimming it down. Makes the page less daunting and also theoretically makes porting to future gitea releases easier if they change the landing page template17:39
fungifor me the important sections to keep there are "what is opendev," "free software needs free tools," and "cloud donors"17:40
fungioh, and "contact info" and "service vulnerabilities" of course17:40
fungii might also shift the donors higher up the page, like right after the "what is..." section17:41
tonybAll of that sounds good to me.17:41
fungistarting on a draft later today, just softballing the pitch as i brainstorm17:41
frickler+1 sounds like a good idea to me17:42
fungipart of the incentive is the blog post about rackspace flex and opendev, the foundation is looking at reaching out to our other cloud donors to write up similar stories and i'm hoping to link them with the donors on the page17:42
opendevreviewTony Breeds proposed zuul/zuul-jobs master: ensure-devstack:  Add the ability is specify a custom cpu model  https://review.opendev.org/c/zuul/zuul-jobs/+/94954718:00
opendevreviewJeremy Stanley proposed opendev/infra-manual master: Add frequently asked questions  https://review.opendev.org/c/opendev/infra-manual/+/94992418:15
fungithere's the first part ^18:16
fungiheading out to grab a late lunch and then i'll push up the corresponding content removal change for system-config18:16
michelthebeauHi, a peer was asking about "has dupicates" error for gerrit. The docs suggest "tell an admin" or create a bug report for gerrit:18:46
michelthebeauhttps://review.opendev.org/q/Ifb2f618f47ce403321c04f4eceeb7240758e71bc https://review.opendev.org/Documentation/error-has-duplicates.html18:46
michelthebeauIs this a good channel for that?18:46
tonybmichelthebeau: Is the first link the one you're trying to upload?18:47
michelthebeauThe first link is the example.  I've already had the user fix the review with a new change ID.  I was just intending to follow the documented instruction when I came here "Since this error should never occur in practice, you should inform your Gerrit administrator if you hit this problem and/or open a Gerrit issue."18:48
tonybmichelthebeau: Well those docs are a little ... simple, it's pretty easy for a user to include a Change-ID especially when using cherry-picks or similar workflows18:54
tonybDo you have any idea (docs?) on the process the user used to hit the problem?18:55
tonybmichelthebeau: Would you rather discuss this in the StarlingX matrix room?18:56
clarkbtonyb: in theory gerrit should update the existing change when pushing the same change id  to the same project + branch tuple though19:05
clarkbso it is a bit unexpected that gerrit would hae two distinct changes here. And yes understanding how the user is pushing changes to gerrit would be helpful. Without that info I don't think there is much we can do19:05
michelthebeauI had only hinted at "how did you do that" when talking to Gustavo.  I believe Gustavo understands the issue now.  "The docs are a little simple"  :)  that is fair.  19:05
tonybclarkb: That's true.   I wasn't considering that19:07
clarkbmichelthebeau: if you have information on how this happened that may be useful in filing a bug with gerrit19:07
michelthebeauquerying Gustavo.  will get git and git-review version too19:08
clarkbbut also the first chagne was createon april 4 and the second on april 719:08
clarkbso that info may not be in memory anymore I guess19:08
clarkbto be clear there is almost certainly a gerrit bug here, but in order to understand how that bug occurs we need more info. Without htat information we're likely stuck accepting this  situation then waiting to see if it happens again19:10
michelthebeaugit-review version 2.4.0 git version 2.34.;  says "just use git review" command19:10
michelthebeau"I just used the 'git review' command'19:10
michelthebeaugit version 3.24.1 (It  wasn't supposed to be an end of sentence dot)19:12
clarkbso on april 4 eb28e5e is pushed to https://review.opendev.org/c/starlingx/app-rook-ceph/+/946355 then on april 7 that commit is updated creating 0f7dd25 and pushed to https://review.opendev.org/c/starlingx/app-rook-ceph/+/946585/1 both using git review?19:13
michelthebeauThat's what he's indicating.19:13
clarkbI notice 946355 got no zuul testing either19:14
clarkbI wonder if the problem occurs on april 4 with the initial push19:14
tonybclarkb: I noticed that too19:15
clarkbmy notes say that is the day we upgrade gerrit to 3.10.519:15
clarkbhrm maybe its just the image though and we didn't do a restart there? I'm not seeing discussion of it in the irc logs19:16
clarkbhttps://review.opendev.org/c/opendev/system-config/+/94605019:17
clarkboh beacuse I'm looking at 2024 not 202519:17
clarkbyes 946355 appears to coincide with restarting gerrit19:19
clarkbhttps://meetings.opendev.org/irclogs/%23opendev/%23opendev.2025-04-04.log.html#t2025-04-04T17:18:1519:19
clarkbtheory: gerrit is restarted around 17:18. That change, 946355 is pushed at ~17:15. If that change was not properly indexed when we shut down gerrit it wouldn't be in the index until some later action indexes it. Not being in the index may break gerrits check for preexisting changes with the same changeid?19:20
clarkband this was before we changed the signal emitted to gerrit to shutdown19:22
clarkb(just to rule out any relationship to that change possibly impacting graceful shutdown)19:22
clarkbmichelthebeau: thank you for the report I think ^ with that theory above we have enough to put together a bug report for upstream19:23
tonybThat sounds plausible19:23
michelthebeaucool19:24
opendevreviewMichal Nasiadka proposed openstack/diskimage-builder master: WIP: Add support for CentOS Stream 10  https://review.opendev.org/c/openstack/diskimage-builder/+/93404519:28
clarkbmichelthebeau: the second change has 10 patchsets on it. Can you ask if there were any errors pushing those?19:30
clarkbmichelthebeau: I'm assuming not, but I want to be accurate in my bug report19:31
clarkbI suspect that the error started because for some reason the first change got reindexed and now the check failed. But that doesn't happen until all 10 of those new patchsets are pushed for some reason19:31
michelthebeauI'm also asking "you abandoned  the first review because gerrit created the second?"19:31
opendevreviewMichal Nasiadka proposed openstack/diskimage-builder master: WIP: Add support for CentOS Stream 10  https://review.opendev.org/c/openstack/diskimage-builder/+/93404519:32
michelthebeauclarkb: gustavo writes "Up until patchset 9 I had no problem. But when I tried to rebase the analysis (patchset 10) we had this problem, and after abandoning the first analysis, it was possible. However, it worked once."19:38
clarkbmichelthebeau: thank you for confirming19:39
michelthebeauclarkb: gustavo writes "[abandonded the first (apr 4) review] Because a team member tried to rebase the review and got this error. Trying to solve it (When I noticed this issue), I abandoned the first review."19:39
opendevreviewJeremy Stanley proposed opendev/git-review master: Update higher Gerrit versions used in testing  https://review.opendev.org/c/opendev/git-review/+/94991619:40
clarkbhttps://issues.gerritcodereview.com/issues/418025702 has been filed19:46
michelthebeauthanks, signing off19:47
opendevreviewJeremy Stanley proposed opendev/system-config master: Remove content duplicated in the Infra Manual FAQ  https://review.opendev.org/c/opendev/system-config/+/94993919:51
opendevreviewTony Breeds proposed openstack/diskimage-builder master: Add new openstack/devstack based functional testing  https://review.opendev.org/c/openstack/diskimage-builder/+/94994220:07
opendevreviewJames E. Blair proposed opendev/zuul-providers master: Use zuul-supplied image formats  https://review.opendev.org/c/opendev/zuul-providers/+/94994420:13
opendevreviewJames E. Blair proposed opendev/zuul-providers master: Use zuul-supplied image formats  https://review.opendev.org/c/opendev/zuul-providers/+/94994420:14
clarkbfungi: the faq update to manual lgtm. I think we can probably just alnd that20:17
fungii have questions about the continued relevance of a few items in there, but figured it was best to start with a verbatim forklift and then we can revise in-place after20:18
clarkbyes oen of them talks about it being early days in the opendev transition :)20:19
clarkbbut agree we can forklift then edit later20:19
clarkbon the gitea side I'm waiting for the screenshot to show me things render properly there20:19
clarkbthe linux kernel is improving the performance of swap20:40
clarkbmaybe in 5 years having jobs swap won't be quite as bad as it is today20:41
clarkbfungi: https://44a737a17d7e744da544-e8c985e942bf44b286d3f5e0d40a9d67.ssl.cf5.rackcdn.com/openstack/c6b69397f8f24c6aa2e4f293a612c23e/bridge99.opendev.org/screenshots/gitea-main.png20:46
fungilooks good for my first attempt20:46
clarkbyup +2 from me20:47
tonybLooks good!20:47
fungii'm noticing that the prose in the "free software needs free tools" section seems to build on the earlier removed sections and could use a bit of minor rewording20:47
opendevreviewJeremy Stanley proposed opendev/git-review master: Update higher Gerrit versions used in testing  https://review.opendev.org/c/opendev/git-review/+/94991620:52
fungiswitching that ^ to wip for now since i need to crank up the timeout significantly to get proper log collection for the gerrit service to figure out why all the tests are timing out connecting20:53
fungiprobably the jdk version is causing it to hang when starting or something20:53
clarkbfungi: gerrit 3.11 is going to require the project acl setup if we're doing any modifications to those20:59
clarkbfungi: I think I would reduce the scope to gerrit 3.10 with java 1720:59
fungiwell, 3.10 is also timing outy20:59
clarkbya I'm wondering if the upstream 3.10 war can't run under java 2120:59
fungiyeah, it may be that 3.10 doesn't work with java 2120:59
fungibut if i can get service logs that will help narrow it down21:00
clarkbya that would help quite a bit21:00
fungiwe were testing with 3.9 previously, so the update to 3.10 is not all that urgent as we weren't very far behind like we have been in the past21:00
clarkbthe other thing to consider is the tests run java -jar if that java is java11 and not java17 its possible neither 3.10 or 3.11 are happy21:02
clarkbI don't know how the packaging resolves java to a specific version21:02
fungifair. but the 3.10 test was working before i switched from installing openjdk-17 to openjdk-2121:03
fungileading me to suspect it just may not work with 2121:03
clarkbaha21:04
fungithe 3.11 test was failing with openjdk-17 and with an error that made it sound like it needed newer java21:04
fungii was hoping i could just get them both running with openjdk-21 and avoid having to pass custom bindep profiles for choosing java versions on the same platform21:05
clarkbfungi: if you look in the tests' _run_gerrit method it already attemps to attach the gerrit log files as details. Implying it never got far enough to write the logs files (a good indication something with the jvm is indeed at play)21:19
clarkboh ya its saying error_log not found21:21
clarkbetc21:21
clarkbeither the paths are wrong or the files never get written21:21
fungii'll dig deeper into it tomorrow, coming up on my eod at this point21:22
opendevreviewClark Boylan proposed opendev/git-review master: Add more gerrit startup logging  https://review.opendev.org/c/opendev/git-review/+/94995021:28
clarkbthat may be helpful not sure21:28
clarkbssh'ing into the test nodes running for ^ I don't get a prompt21:39
clarkbthat is unexpected21:39
clarkboh its just very slow?21:40
clarkbfungi: based on not getting output from ^ that change in the logs I suspect that gerrit.sh start isn't actually returning and so we're hanging there and not progressing21:42
clarkbya if I run ps there are a bunch of gerrit.sh start processes21:44
clarkbI need to put this down but two other things I notice the gerrit.config from the golden site is fairly disjoint from the config template that we write out. Its possible that we need additional config there. However when manually running a copy of the golden site it fails immediately for me (different than the other behavior which doesn't return from gerrit.sh start). Then the other21:59
clarkbthing I notice is that stracing the running gerrit.sh processes they seem to be running dirname . in a tightloop21:59
clarkbthis probably needs to be debugged more frmo the bottom up updating things for gerrit we know need updating then see what behaviopr we get since this could all be side effects of staleness22:01
clarkbok I've just had a conversation wtih Martin Fick on Gerrit's Discord about our two chagnes with one change id situation23:14
clarkbtl;dr is that yes if the index does not have up to date data then you can run into this problem so my hunch was a good one23:15
clarkbIt seems this is/was fairly common with the haplugin and restarting constituent nodes of the gerrit cluser where they would get out of sync with one another and if you pushed to the wrong server you could generate this problem23:15
clarkbhttps://gerrit-review.googlesource.com/c/plugins/high-availability/+/422559 was written to address that. Basically a solution that reindexes things if they are discovered to be stale23:15
clarkbIn our case we don't run the ha plugin we have a single server. It sounds like we can just manually trigger reindexing on gerrit startup to mitigate this problem23:16
clarkbbit of a brute force but one that we actually already apply when we do upgrades between gerrit releases most of the time since the index versions update23:16
clarkbthe reason we saw it this time is we went from 3.10.4 to 3.10.5 which had no index updates so we don't reindex then boom. I'll note the 3.10 to 3.11 upgrade also does not have index updates. Nor does the 3.10.5 to 3.10.6 update so we've got some upgrades that will need manual reindexing triggers23:17
clarkbbut that explains why we don't see this more often I think23:17
tonybSo the suggestion is to include the reindex as part of our general gerrit restart process?23:18
tonybbut also thanks clarkb!23:18
clarkbthen finally we can probably have gerrit track staleness of indexes relative to its git pushes inteernally and have it elect to reindex things on its own in a manner similar to the haplugin. But that requires new code and I'm not sure anyone is volunteering to do that (I indicated I could try to look at it if I find time but I'm low on that resource right now)23:18
clarkbtonyb: ya, I basically said "our indexes are small enough we can reindex on startup" and they said yes that should work for you but isn't a general solution because other gerrits are larger and reindex for them is much more costly23:19
clarkbwe can use the 3.10.6 update as the guinea pig for the process23:20
tonybOkay cool23:20
tonybSounds good23:21
clarkbI'm not familiar with the internals of Gerrit in this portion of the code (to be fair I'm not really familar with any of GErrit internals but I have read through other different parts of gerrit for random things)23:23
clarkbbut maybe taking bits from the haplugin I can slap something together23:23

Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!