| *** liuxie is now known as liushy | 02:15 | |
| *** mrunge_ is now known as mrunge | 06:32 | |
| opendevreview | Merged openstack/project-config master: Use ubuntu-noble for publish-openstack-artifacts https://review.opendev.org/c/openstack/project-config/+/963367 | 11:50 |
|---|---|---|
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Add debian-trixie-nested-virt-8GB https://review.opendev.org/c/opendev/zuul-providers/+/970115 | 12:03 |
| *** dhill is now known as Guest33132 | 12:30 | |
| clarkb | infra-root I'll plan to approve https://review.opendev.org/c/opendev/system-config/+/969847 and https://review.opendev.org/c/openstack/project-config/+/969846 shortly if no one beats me to them as general cleanups after the gerrit upgrade. Then I'mgoing to see if I can get gerritlib's integration testing running against 3.11 before we land the acl update test change | 15:46 |
| fungi | looks like i already reviewed those, happy to approve them now | 15:47 |
| clarkb | fungi: thanks! | 15:47 |
| fungi | and done | 15:47 |
| clarkb | and if I can't figure out the integration testing in a reasonable amount of time we can probably proceed with testing in production :/ | 15:49 |
| clarkb | anyway something to eat and drink then I'm digging into that | 15:49 |
| opendevreview | Merged openstack/project-config master: Update Jeepyb's Gerrit builds to Gerrit 3.11 https://review.opendev.org/c/openstack/project-config/+/969846 | 15:51 |
| opendevreview | Merged opendev/system-config master: Update infra-prod review and manage-projects deps for new Gerrit https://review.opendev.org/c/opendev/system-config/+/969847 | 15:56 |
| opendevreview | Merged openstack/project-config master: Add SeaweedFS App to StarlingX https://review.opendev.org/c/openstack/project-config/+/969527 | 16:02 |
| clarkb | note ^ will test manage-projects | 16:03 |
| clarkb | we can probably abandon my bindep.config update change | 16:03 |
| clarkb | deployment jobs are behind the hourlies so we have a minute to stop things if we want. But I guess its probably best to just rip the bandaid off and see what happens | 16:05 |
| opendevreview | Clark Boylan proposed opendev/gerritlib master: Update integration testing to test with Gerrit 3.11 https://review.opendev.org/c/opendev/gerritlib/+/970142 | 16:08 |
| clarkb | manage-projects is running now | 16:11 |
| clarkb | manage-projects reports success. I'm going to look at logs now | 16:15 |
| clarkb | the logs look ok to me and the project seems to exist with a config that looks correct: https://review.opendev.org/admin/repos/starlingx/app-seaweedfs,general (requiring signed off by is set to True for example which is not the default as far as I know) | 16:17 |
| clarkb | I've marked that item done on the etherpad and will abandon my noopy bindep config update change | 16:19 |
| fungi | yeah, i figured that was a more useful test | 16:20 |
| clarkb | double checking the All-Projects config for Blocked Users addition (or not) was on my todo list but that isn't super urgent so I'm going to focus on updating the gerritlib testing and filing the delete project deleting nonexistant file bug first | 16:21 |
| clarkb | happy for someone else to do that one too. But I should be able to get to it eventually and the only change on our side is the docs update if it did change in prod so not really a big deal to get done right away | 16:22 |
| opendevreview | Clark Boylan proposed opendev/gerritlib master: Update integration testing to test with Gerrit 3.11 https://review.opendev.org/c/opendev/gerritlib/+/970142 | 16:23 |
| clarkb | trixie is python3.13 too right? Once this is working with gerrit 3.11 I should update the test platform for trixie to get python3.13 to better mimic prod and then update the unittests to include 3.13 as well | 16:28 |
| fungi | yes | 16:29 |
| mnasiadka | clarkb: probably you’re busy with gerrit upgrade fallout - but it seems https://review.opendev.org/c/zuul/zuul-jobs/+/966187 is ready for another review | 16:29 |
| clarkb | that is more about coverage for the java 21 move on trixie than anything else but may as well get it sorted while I'm looking at this stuff | 16:29 |
| clarkb | mnasiadka: ack I'll try | 16:29 |
| mnasiadka | And if anybody can have a look in https://review.opendev.org/c/opendev/zuul-providers/+/970115 - I would be grateful (we use nested virt nodes for kvm based testing in Kolla-Ansible CI) | 16:30 |
| clarkb | heh and I immediately hit the docker hub rate limit | 16:33 |
| clarkb | our quay mirror only has gerrit:latest. But maybe that is good enough for now | 16:34 |
| clarkb | if ^ works then we can mirror 3.11, 3.12, 3.13 too and pull them from the mirror and cover them as well ro something | 16:34 |
| mnasiadka | 970115 now only needs +w, because it has 2 * +2 ;-) | 16:36 |
| opendevreview | Merged opendev/zuul-providers master: Add debian-trixie-nested-virt-8GB https://review.opendev.org/c/opendev/zuul-providers/+/970115 | 16:43 |
| mnasiadka | thanks | 16:44 |
| opendevreview | Clark Boylan proposed opendev/gerritlib master: Update Gerrit integration testing to test many Gerrit versions https://review.opendev.org/c/opendev/gerritlib/+/970142 | 16:44 |
| opendevreview | Clark Boylan proposed opendev/gerritlib master: Add Python 3.13 testing https://review.opendev.org/c/opendev/gerritlib/+/970144 | 16:46 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Add base py313 jobs for tox and nox https://review.opendev.org/c/zuul/zuul-jobs/+/970146 | 16:52 |
| opendevreview | Clark Boylan proposed opendev/gerritlib master: Add Python 3.13 testing https://review.opendev.org/c/opendev/gerritlib/+/970144 | 16:53 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Add base py313 jobs for tox and nox https://review.opendev.org/c/zuul/zuul-jobs/+/970146 | 16:58 |
| opendevreview | Clark Boylan proposed opendev/gerritlib master: Update Gerrit integration testing to test many Gerrit versions https://review.opendev.org/c/opendev/gerritlib/+/970142 | 16:59 |
| opendevreview | Clark Boylan proposed opendev/gerritlib master: Add Python 3.13 testing https://review.opendev.org/c/opendev/gerritlib/+/970144 | 16:59 |
| clarkb | the integration testing against gerrit:latest passed so I think we are good | 16:59 |
| clarkb | I'll let reviewers decide if we want to do the full 3.11, 3.12, 3.13 matrix and whether we want to mirror all of those to quay before we rely on them | 17:00 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Add base py313 jobs for tox and nox https://review.opendev.org/c/zuul/zuul-jobs/+/970146 | 17:12 |
| clarkb | infra-root ^ I believe all three of those changes should be ready for review. The zuul-jobs update is the only one that ahsn't come back happy yet but the latest patchset should update bindep to make it happier | 17:13 |
| clarkb | and none of those directly affect production gerrit so should be safe to land. They just help us ensure we're testing jeepyb, gerritlib, and gerrit together so that when we do upgrades there aren't surprises | 17:13 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Drop Gerrit 3.10 image builds https://review.opendev.org/c/opendev/system-config/+/970150 | 17:17 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Upgrade build and runtime for Gerrit to Java 21 https://review.opendev.org/c/opendev/system-config/+/970160 | 17:36 |
| clarkb | I don't think we need to be in a huge rush for ^ the bigger thing is getting it working in CI so that I can get Gerrit 3.12 also working in CI | 17:37 |
| clarkb | we probably want to schedule the java runtime swap for a time where we can pay attention to it etc | 17:37 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Add base py313 jobs for tox and nox https://review.opendev.org/c/zuul/zuul-jobs/+/970146 | 17:48 |
| fungi | clarkb: that ^ makes flake8 unhappy because of importing unused modules | 18:01 |
| fungi | i can think of a few ugly hacks to avoid that, or just silence its objection | 18:02 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Add base py313 jobs for tox and nox https://review.opendev.org/c/zuul/zuul-jobs/+/970146 | 18:05 |
| clarkb | fungi: ya I went with just silence its objection. I think doing the imports is the thing that most closely matches the code we are trying to test so the rule just dones't make sense here | 18:05 |
| fungi | that's what i'd have done too | 18:05 |
| fungi | i'm going to go grab lunch, bbiab | 18:06 |
| cardoe | So what I am doing wrong with "git review"... I get "! [remote rejected] HEAD -> refs/for/master (no new changes)" | 18:09 |
| cardoe | I've got 1 commit locally in my branch from master for keystone.git | 18:09 |
| clarkb | cardoe: (no new changes) means what you are trying to push already exists | 18:10 |
| cardoe | It doesn't though | 18:10 |
| cardoe | I literally just rewrote the doc locally. | 18:10 |
| cardoe | Unless I managed a sha1 hash collision? | 18:11 |
| clarkb | or forgot to git add or commit ot something? We'd need more info I think | 18:11 |
| clarkb | git log and git show HEAD may be useful. | 18:12 |
| cardoe | https://opendev.org/openstack/keystone/commit/50f0f023bdd23bea8b9c21823a569a159af9ebeb | 18:12 |
| clarkb | basically info that describse the staet of your local tree that you are trying to push | 18:12 |
| cardoe | That's my commit | 18:12 |
| cardoe | It's not in master I'm surprised its appearing on opendev.org though | 18:12 |
| clarkb | https://review.opendev.org/c/openstack/keystone/+/970166 | 18:13 |
| clarkb | we replicate unmerged commits | 18:13 |
| clarkb | is it possible you ran git review twice and the first one succeeded and the second failed? | 18:13 |
| cardoe | I mean I've run it a few times cause it's been erroring at me each time. | 18:13 |
| * cardoe shrugs. | 18:14 | |
| clarkb | ok, I'd need a better account of what you did locally in terms of git review execution to understand if there was actually a problem or if it was maybe instigated by an expected error (missing signed off by for example) then dobule tapped after without realizing the first succeeded adn second failed with no new changes | 18:15 |
| clarkb | its up to you if you want to do that. It does seem to have worked | 18:15 |
| clarkb | were you using claude to push? | 18:15 |
| clarkb | did claude try multiple times and only report the last failure? | 18:16 |
| cardoe | Nope. claude doesn't have that tool as permissions. | 18:16 |
| cardoe | The very first time I got a timeout error actually. | 18:16 |
| cardoe | Going back to the scroll back in my terminal. | 18:16 |
| * clarkb suddenly wonders how claude permissions are enforced. but agree that seems like a thread not worth pursuing unless all others are exhausted | 18:17 | |
| clarkb | ok my hunch is that initial timed out connection did enough to create the change but maybe didn't get back the "here's your change number" response in time | 18:17 |
| cardoe | yeah that's the most likely situation. | 18:17 |
| cardoe | I just assumed it didn't exist and if it had I would get back a different error message. | 18:18 |
| clarkb | and a likely explanation for that is: we upgraded gerrit yesterday and in the process deleted a bunch of caches because they grow unbounded and got too large. And we're hitting cold cache issues. Though it look like stephenfin pushed some keystone changes slightly before you | 18:19 |
| clarkb | no new changes is a telltale sign that what you're pushing exists fwiw. And gerrit is saying I don't want to bother updating something that has no updates | 18:19 |
| clarkb | cardoe: in the sshd_log for gerrit I see you connect from an ipv4 address when I think the timeout occurs. There are three connections around then one of which seems to work and one of which seems to be killed. I don't understand the other. Then your subsequent connections come over ipv6 | 18:23 |
| clarkb | oh wait no there are four. Three look normal. The fourth via ipv4 is killed. Then you come in via ipv6 and all seems well from there | 18:23 |
| clarkb | note git review may make more than one connection so this doen't mean you ran git review multiple times when this happened | 18:24 |
| cardoe | So I assumed that my VPN was messing with it so I stopped the VPN and I bet that's when it switched to IPv6. | 18:24 |
| clarkb | My new hunch is that git review ran and did enough to make the chagne but then some subroutine in git review failed beacuse its connection timed out/got killed | 18:24 |
| cardoe | user error is a good enough answer for me. :-D | 18:26 |
| cardoe | I didn't think to check that it was created. | 18:26 |
| cardoe | sorry for the noise. | 18:26 |
| clarkb | ya the only traceback I can find is "channel already closed" | 18:27 |
| clarkb | cardoe: git review does have a verbose flag (-v iirc) which might eb useful if you find this happening frequently. That might help us narrow down where things go sideways | 18:28 |
| clarkb | but also looks like it may have been on the order of ~15 seconds? | 18:28 |
| clarkb | git can be slow | 18:29 |
| cardoe | corporate VPN also likes to fiddle with SSH | 18:29 |
| clarkb | 15 seconds isn't necessarily a long time to wait depending on the repo. As both sides have to negotiate what commits they know about along the common dag path then construct and exchagne the missing data | 18:29 |
| opendevreview | Merged opendev/system-config master: Drop Gerrit 3.10 image builds https://review.opendev.org/c/opendev/system-config/+/970150 | 18:30 |
| clarkb | https://zuul.opendev.org/t/openstack/build/3730086c8554461aa74ec28f7966c055/log/review99.opendev.org/logs/error_log#32 our first Gerrit 3.11 run on Trixie with Java 21 | 18:31 |
| clarkb | Looking at the logs we're using java 21 to build a war that targets java 17 still | 18:33 |
| clarkb | which I think is fine | 18:33 |
| clarkb | just want to note it as we aren't fully converted to 21 I guess until we get to 3.12 where java 17 is dropped entirely and they update the build target | 18:33 |
| clarkb | also I don't think we've converted anything else to trixie first. It probably isn't the worst idea to convert something else first like irc bots maybe? | 18:44 |
| clarkb | thoughts on that? | 18:44 |
| clarkb | also, I raelize I never got a clear answer on whether or not yall wanted me to followup with EMS to ask if we can use duplicate email addresses. Should I go ahead and do so? | 18:45 |
| *** jgwentworth is now known as melwitt | 19:01 | |
| fungi | or we could just test the theory ourselves | 19:24 |
| clarkb | my concern with doing that is associating an email address we'd want billing or whatever to go to with a specific account and breaking that system | 19:25 |
| clarkb | but we could use a single plus addressing address for all the bots and test if duplicatse of that one are ok | 19:25 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Add Gerrit 3.12 image builds and testing https://review.opendev.org/c/opendev/system-config/+/970173 | 19:36 |
| clarkb | of the pile of changes I've pushed so far I believe that https://review.opendev.org/c/zuul/zuul-jobs/+/970146 https://review.opendev.org/c/opendev/gerritlib/+/970144 and https://review.opendev.org/c/opendev/gerritlib/+/970142 should be mergable | 19:40 |
| clarkb | and I just double checked that bookworm doesn't have openjdk-21 in backports (it doesn't appear to: https://packages.debian.org/search?suite=bookworm-backports&searchon=names&keywords=openjdk) So I think we do have to bump up to trixie | 19:41 |
| clarkb | ok I rtfs'd that delete project exception we hit: https://gerrit.googlesource.com/plugins/delete-project/+/refs/tags/v3.11.7/src/main/java/com/googlesource/gerrit/plugins/deleteproject/fs/DeleteTrashFolders.java is the relevant code. The way that plugin works is it iterates through directories looking for directories that match one of three patterns. In this case the path that hit | 19:47 |
| clarkb | the NoSuchFileException does not match the paths so it wouldn't try to delete anything. Instead something else mus've been writing and removing that file/dir and this plugin saw it and they raced each other | 19:47 |
| clarkb | Pretty sure this should be a totally benign issue for us, but I'll file an issue with what I've learned after lunch because unnecessary exceptinos are annoying | 19:48 |
| clarkb | 970173's build did build targetting java 21 so I think that confirms it | 20:12 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Add Gerrit 3.12 image builds and testing https://review.opendev.org/c/opendev/system-config/+/970173 | 21:00 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Test Gerrit 3.11 to 3.12 Upgrades https://review.opendev.org/c/opendev/system-config/+/970180 | 21:00 |
| clarkb | https://issues.gerritcodereview.com/issues/467103128 issue is filed | 21:22 |
| clarkb | I diagnosed it to the point where I could probably fix it, but I don't have my google cla signed yet so won't do that yet. However, I'm hoping that gets done soon as I have requested it. If it does happen I should be able to determine which of the two theories I posted in the issue is the correct one and fix it | 21:23 |
| clarkb | I'm going to work on putting together our meeting agenda shortly. Let me know if there is anything you'd like added or edited. I plan to recap the gerrit upgrade and cover the next steps. I'll add a note about fungi's plan for renaming that one starlingx project Friday at 2100 UTC (I think that was the proposed time). And then add in Matrix updates | 21:25 |
| clarkb | Anything else? | 21:25 |
| fungi | nothing on my end | 21:27 |
| fungi | i do need to send out a (fairly short notice) announcement about the rename maintenance on friday, but shouldn't be a big deal since those are typically <5min downtime for gerrit | 21:28 |
| clarkb | The last major task on the post upgrade todo list that I haven't done anything to start yet is checking the all-projects refs/meta/config log to see if we added Blocked Users to the acl. In the command I've got I run a git log -p against the git repo on the host. It occusr to me now that I'm not positive that is 100% safe to run against a git repo that gerrit is operating on. I think | 21:30 |
| clarkb | it is safe and I'm pretty sure I tested it with gerrit running on the test node | 21:30 |
| clarkb | any concerns with that? It is a read only command against the repo and we run git gc against the repos outside of gerrit which is read write so I think just checking the log should be fine | 21:30 |
| fungi | no concerns on my part, seems like it should be perfectly safe yes | 21:31 |
| clarkb | ok I'm going to run it now | 21:31 |
| clarkb | 76473b67b78389fadef983291787e88cc1cb80f9 is HEAD and is the change fungi pushed in July to cleanup cla config | 21:32 |
| clarkb | this means it did not update the acl for us which means we don't need to update our docs | 21:32 |
| clarkb | I'll leave a comment on my docs change indicating it wasn't automatic for us and if we want to apply that change we will need to manually update the acls too | 21:33 |
| fungi | thanks! | 21:33 |
| clarkb | comment posted to https://review.opendev.org/c/opendev/system-config/+/968228 and WIP is removed | 21:35 |
| clarkb | I could go either way. Since we allow anonymous access people would be able to logout and see/fetch everything so really all this would prevent is them writing things | 21:36 |
| clarkb | and in that case maybe we're better off simply disabling their account which is what we've done for years | 21:37 |
| clarkb | good news: looks like the entire java 21, gerrit 3.12 image, 3.11 -> 3.12 upgrade stack is going to pass. I think the main considering with merging those is doing so when we can restart to pick up the trixie + java 21 switch so not something for today though reivews are welcome any time | 21:41 |
| clarkb | I think personally I'd like to update something other than gerrit to trixie before switching gerrit | 21:41 |
| clarkb | just as a sanity check that trixie as we build it in our container images is generally functional | 21:42 |
| fungi | yeah, that seems like a safe plan | 21:42 |
| clarkb | I need to go apply hashtags to changes as I've been bad about that. I'll link to that list here when done and then work on our meeting agenda | 21:42 |
| clarkb | https://review.opendev.org/q/hashtag:%22upgrade-gerrit-3.11%22+status:open | 21:44 |
| clarkb | ok the agenda is updated. I'll send that later today in case there are any more updates we can think of | 22:06 |
| opendevreview | Goutham Pacha Ravi proposed opendev/infra-manual master: Alter the repo retirement README template https://review.opendev.org/c/opendev/infra-manual/+/970191 | 22:17 |
| clarkb | infra-root do you want to weigh in on https://review.opendev.org/c/opendev/zuul-providers/+/970042 and my comment there? specifically the proposal is that we stop booting alma on all our clouds and restrict it to the v3 capable clouds like rocky and centos stream. I'm not sure if that sort of redundancy makes sense if I'm honest | 22:19 |
| opendevreview | Merged opendev/infra-manual master: Alter the repo retirement README template https://review.opendev.org/c/opendev/infra-manual/+/970191 | 22:30 |
| opendevreview | Goutham Pacha Ravi proposed openstack/project-config master: Cleanup Monasca infra https://review.opendev.org/c/openstack/project-config/+/970193 | 22:30 |
| *** melwitt is now known as jgwentworth | 22:31 | |
| *** jgwentworth is now known as melwitt | 22:32 | |
| *** dhill is now known as Guest33169 | 22:35 | |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!