| @mnasiadka:matrix.org | The yellow active ,,badge'' looks funny :) | 04:32 |
|---|---|---|
| @mnasiadka:matrix.org | Ok, I don't know if that's by purpose - by now project maintainers lost the ability to do a rebase via GUI - Gerrit now requires the rebase to be done by the committer and checks email | 04:55 |
| @mnasiadka:matrix.org | * Ok, I don't know if that's by purpose - but now project maintainers lost the ability to do a rebase via GUI - Gerrit now requires the rebase to be done by the committer and checks email | 04:56 |
| @harbott.osism.tech:regio.chat | I'm also seeing some weird UI stuff, https://review.opendev.org/c/openstack/requirements/+/984232 merged yesterday shortly after the gerrit upgrade was done, but it is still being shown in the "Your turn" list in my "Your Changes" page with just "CR+2 and W+1" flags, so likely the state from before the update. clicking the "Remove yourself from the attention set" also has no effect. | 07:31 |
| @harbott.osism.tech:regio.chat | the wrong state is also shown for me in the https://review.opendev.org/q/project:openstack/requirements overview page, might be interesting to learn whether others can reproduce that | 07:36 |
| @harbott.osism.tech:regio.chat | hmm, I can reproduce in an unauthenticated window in another browser, so looks like it may be a global issue indeed. maybe this happened because the merge was done in parallel to gerrit recreating the caches? do we need to trigger another update run? | 07:38 |
| @mnasiadka:matrix.org | It's the same for me (on the overview page) - so something somewhere did not refresh/recalculate/reindex? | 08:02 |
| @harbott.osism.tech:regio.chat | I added another comment on the review now and it seems to have updated the cache, so we can use that as a workaround if more similar issues pop up | 08:36 |
| -@gerrit:opendev.org- Michal Nasiadka proposed: [openstack/project-config] 978566: propose-updates: Add pcu target https://review.opendev.org/c/openstack/project-config/+/978566 | 12:39 | |
| -@gerrit:opendev.org- Michal Nasiadka proposed: [openstack/project-config] 978566: propose-updates: Add pcu target https://review.opendev.org/c/openstack/project-config/+/978566 | 12:59 | |
| @fungicide:matrix.org | is that stale attention set problem happening to all changes or just a handful? do we know yet | 13:01 |
| @fungicide:matrix.org | for rebasing existing changes, that was allowed for all registered users before, maybe they added a permission with a restrictive default | 13:03 |
| @fungicide:matrix.org | if so, we can revert it with a config entry | 13:03 |
| -@gerrit:opendev.org- Michal Nasiadka proposed: [openstack/project-config] 978566: propose-updates: Add pcu target https://review.opendev.org/c/openstack/project-config/+/978566 | 13:06 | |
| @fungicide:matrix.org | https://review.opendev.org/Documentation/access-control.html#category_rebase | 13:14 |
| @fungicide:matrix.org | it's possible revert is restricted now too? | 13:14 |
| @fungicide:matrix.org | (listed immediately below rebase) | 13:15 |
| -@gerrit:opendev.org- Zuul merged on behalf of Dmitriy Rabotyagov: [openstack/project-config] 981944: Stop Venus gating https://review.opendev.org/c/openstack/project-config/+/981944 | 13:23 | |
| @clarkb:matrix.org | fungi: our documented all projects acl says we should allow revert against refs/heads/* for registered users | 13:23 |
| @clarkb:matrix.org | So maybe that isn't working due to a bug or isn't actually applied as expected? | 13:23 |
| @clarkb:matrix.org | Re attention set: if leaving a comment later fixed it then I suspect this is fallout from things happening during the reindexing period | 13:23 |
| @fungicide:matrix.org | Clark: well, rebase is what the original complaint was for | 13:24 |
| @clarkb:matrix.org | So hopefully the impact is to changes updated during that ~45 minute period | 13:24 |
| @fungicide:matrix.org | i was just theorizing revert could also be affected, but probably not if we configure it | 13:24 |
| @clarkb:matrix.org | fungi: yes, maybe the acl doesn't match our documentation or Gerrit is buggy? | 13:24 |
| @clarkb:matrix.org | Oh sorry I said revert and meant rebase | 13:25 |
| @clarkb:matrix.org | Our docs says rebase should be allowed for registered users. But maybe the docs are out of sync or the acl is buggy | 13:25 |
| @clarkb:matrix.org | Another idea: the upgrade logged everyone out is it possible that the issue is viewed in a logged out tab and need to log in? | 13:26 |
| @fungicide:matrix.org | yeah, seeing that in https://docs.opendev.org/opendev/system-config/latest/gerrit.html#access-controls | 13:26 |
| @fungicide:matrix.org | so i agree, something has gone wrong there | 13:26 |
| @fungicide:matrix.org | i'll get logged in shortly to double-check | 13:27 |
| @mnasiadka:matrix.org | Well, currently the only way to rebase somebody elses patch is to fetch it locally, rebase it locally and run git review - which is fine if we want to limit drive-by-rebases, but I think we would need to inform everyone :) | 13:31 |
| @fungicide:matrix.org | mnasiadka: do you have an example change that can't be rebased? our configuration says to allow anyone to rebase any change (regardless of whether they're the owner or have other privs) | 13:33 |
| @mnasiadka:matrix.org | Here's a DNM change - https://review.opendev.org/c/openstack/kolla-ansible/+/983994 | 13:34 |
| @fungicide:matrix.org | i see a `rebase` button on changes owned by other users | 13:34 |
| @fungicide:matrix.org | i see a rebase button on that change | 13:34 |
| @mnasiadka:matrix.org | I can't rebase it: | 13:34 |
| An error occurred | ||
| Could not perform action: Change is already based on the latest patch set of the dependent change. | ||
| @mnasiadka:matrix.org | sorry | 13:34 |
| @fungicide:matrix.org | oh, that means it doesn't need to be rebased | 13:34 |
| @mnasiadka:matrix.org | let me find something stale | 13:35 |
| @mnasiadka:matrix.org | huh | 13:35 |
| @fungicide:matrix.org | maybe we can work up a mock scenarion in the sandbox repo | 13:35 |
| @mnasiadka:matrix.org | now I can rebase a patch I couldn't earlier | 13:35 |
| @mnasiadka:matrix.org | it wrote that rebase can be done only by comitter/uploader and my email doesn't match (it's a pity I didn't do a screenshot) | 13:36 |
| @fungicide:matrix.org | okay, so there was maybe some temporary error? if you know the approximate time we might be able to track down a corresponding backtrace in the gerrit `error_log` | 13:36 |
| @mnasiadka:matrix.org | 06:55 CEST based on my message here - which is 04:55 UTC? | 13:37 |
| @fungicide:matrix.org | yeah, 0455 is what my orc client shows too | 13:37 |
| @fungicide:matrix.org | what was the change number? | 13:37 |
| @fungicide:matrix.org | alternatively, if you want to dig in the log, it's on review03 in `/home/gerrit2/review_site/logs/error_log` | 13:38 |
| @fungicide:matrix.org | timestamps in it are utc, as usual | 13:39 |
| @mnasiadka:matrix.org | I think it was 959853 - but I rebased it locally around ~2 hours ago | 13:39 |
| @mnasiadka:matrix.org | Let me have a look | 13:39 |
| @fungicide:matrix.org | the only thing i find around the same time is an entry at 2026-04-13T04:53:44.587Z but it's for nova | 13:40 |
| @mnasiadka:matrix.org | Yeah, it's not in the error log | 13:42 |
| @fungicide:matrix.org | maybe something to do with a stale version of the webclient still loaded in your browser? | 13:42 |
| @fungicide:matrix.org | or other pathological situation from right when you started trying to interact with the upgraded server | 13:43 |
| @mnasiadka:matrix.org | [2a01:112f:4409:cd00:7946:e06e:c3d4:d3a6] [HTTP-180] - mnasiadka [2026-04-13T04:53:49.547Z] "POST /changes/openstack%2Fkolla~951971/revisions/10/rebase HTTP/1.1" 409 144 9 - "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.3.1 Safari/605.1.15" 9 10 790928 - 1776056029540-b50ddf81 | 13:45 |
| [2a01:112f:4409:cd00:7946:e06e:c3d4:d3a6] [HTTP-175] - mnasiadka [2026-04-13T04:54:00.667Z] "POST /changes/openstack%2Fkolla~951971/revisions/10/rebase HTTP/1.1" 409 144 7 - "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.3.1 Safari/605.1.15" 6 - 775304 - 1776056040662-148e2e99 | ||
| [2a01:112f:4409:cd00:7946:e06e:c3d4:d3a6] [HTTP-181] - mnasiadka [2026-04-13T04:55:00.115Z] "POST /changes/openstack%2Fkolla~951971/revisions/10/rebase HTTP/1.1" 409 144 7 - "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.3.1 Safari/605.1.15" 6 - 750088 - 1776056100110-3982dfc2 | ||
| @mnasiadka:matrix.org | That was probably this (409) - if I stumble across a similar problem I'll post it here, but it might have been the only case :) | 13:46 |
| @fungicide:matrix.org | yeah, i guess let's keep an eye out for any similar reports and maybe we can figure out a commonality | 13:47 |
| @mnasiadka:matrix.org | But no trace of this in error_log 🤷♂️ | 13:48 |
| @mnasiadka:matrix.org | On the mirror front - probably https://review.opendev.org/c/opendev/zone-opendev.org/+/983860 is fine to merge and I'll double check mirror04.gra1.ovh and re-raise the promotion patch (with a Depends-On this time) | 13:50 |
| @mnasiadka:matrix.org | mirror03.ord.rax system-config patch https://review.opendev.org/c/opendev/system-config/+/983911 is fine as well (it has proper Depends-On to the dns patch) | 13:52 |
| @mnasiadka:matrix.org | The revert from Friday - should I revert the revert so we keep history - or just raise a new patch? | 13:53 |
| @clarkb:matrix.org | I think either way is fine. The changes are simple enough anyone looking through history should be able to understand | 13:54 |
| -@gerrit:opendev.org- Michal Nasiadka proposed: [opendev/zone-opendev.org] 984393: Revert^2 "Promote mirror04.gra1.ovh" https://review.opendev.org/c/opendev/zone-opendev.org/+/984393 | 13:55 | |
| @mnasiadka:matrix.org | re^2 was easier :) | 13:55 |
| @clarkb:matrix.org | https://review.opendev.org/c/opendev/system-config/+/983474 does not show up in my turn listings so I think that issue frickler observed may be due to the confluence of reindexing and merging during the initial startup. I don't expect this is a bigger issue given my change seems to have been handled properly | 13:57 |
| @clarkb:matrix.org | re the docker issue https://zuul.opendev.org/t/openstack/buildset/49b60407e3bc47b28216620acaa307c8 doesn't seem to be reflecting that. Maybe `docker image prune` doesn't require a newer api like docker image list? I'll try to look into that more closely today. But It appears things may be more complicated than I first thought | 13:58 |
| -@gerrit:opendev.org- Zuul merged on behalf of Michal Nasiadka: | 13:59 | |
| - [opendev/zone-opendev.org] 983860: Add entries for mirror03.ord.rax https://review.opendev.org/c/opendev/zone-opendev.org/+/983860 | ||
| - [opendev/zone-opendev.org] 984030: Replace whitespaces with tabs https://review.opendev.org/c/opendev/zone-opendev.org/+/984030 | ||
| -@gerrit:opendev.org- Michal Nasiadka proposed wip: [opendev/zone-opendev.org] 984393: Revert^2 "Promote mirror04.gra1.ovh" https://review.opendev.org/c/opendev/zone-opendev.org/+/984393 | 14:01 | |
| -@gerrit:opendev.org- Michal Nasiadka proposed wip: [opendev/zone-opendev.org] 984393: Revert^2 "Promote mirror04.gra1.ovh" https://review.opendev.org/c/opendev/zone-opendev.org/+/984393 | 14:06 | |
| -@gerrit:opendev.org- Michal Nasiadka marked as active: [opendev/zone-opendev.org] 984393: Revert^2 "Promote mirror04.gra1.ovh" https://review.opendev.org/c/opendev/zone-opendev.org/+/984393 | 14:07 | |
| @fungicide:matrix.org | mnasiadka: question for you on 984393, see comments | 14:20 |
| @fungicide:matrix.org | maybe raced the indent normalization change? | 14:20 |
| -@gerrit:opendev.org- Michal Nasiadka proposed: [opendev/zone-opendev.org] 984393: Revert^2 "Promote mirror04.gra1.ovh" https://review.opendev.org/c/opendev/zone-opendev.org/+/984393 | 14:22 | |
| -@gerrit:opendev.org- Michal Nasiadka proposed: [opendev/zone-opendev.org] 984393: Revert^2 "Promote mirror04.gra1.ovh" https://review.opendev.org/c/opendev/zone-opendev.org/+/984393 | 14:23 | |
| @fungicide:matrix.org | thanks! much better. i was only expecting to see two lines changed (the serial and the cname), which is now what you have | 14:23 |
| @mnasiadka:matrix.org | fungi: well, unexpand -a did a tab and a couple of spaces, which is probably fine - but my brain probably wanted to just have double tabs and I've overdone it :) | 14:24 |
| @mnasiadka:matrix.org | fungi: https://review.opendev.org/c/opendev/system-config/+/983911 is ready to merge now - mirror03.ord.rax is resolving in DNS | 14:32 |
| @mnasiadka:matrix.org | Ok so: | 14:36 |
| 1. mirror04.gra1.ovh is ready to promote -> 984393 | ||
| 2. mirror03.ord.rax needs to be added to system-config -> 983911 | ||
| 3. Clark looking at https://etherpad.opendev.org/p/opendev-server-upgrade-planning and the list of servers in DFW Rackspace region- it seems it's safe to remove mirror02.dfw.rax? I see mirror03.dfw.rax in https://opendev.org/opendev/zone-opendev.org/src/commit/70f2ea484a211c64c99645702526e650b94ec171/zones/opendev.org/zone.db#L434 | ||
| -@gerrit:opendev.org- yatin proposed: [zuul/zuul-jobs] 984404: [ensure-python] Fix rpm_python_pkg_name For CentOS/RHEL https://review.opendev.org/c/zuul/zuul-jobs/+/984404 | 14:52 | |
| -@gerrit:opendev.org- yatin proposed: [zuul/zuul-jobs] 984404: [ensure-python] Fix rpm_python_pkg_name For CentOS/RHEL https://review.opendev.org/c/zuul/zuul-jobs/+/984404 | 14:53 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Michal Nasiadka: [opendev/system-config] 983911: Add mirror03.ord.rax https://review.opendev.org/c/opendev/system-config/+/983911 | 15:12 | |
| @fungicide:matrix.org | since i have a bit of a break between meetings i'm going to go find lunch and grab some sink drain parts from the hardware store, but should only be an hour or so. bbiab | 15:17 |
| -@gerrit:opendev.org- Michal Nasiadka proposed on behalf of Tony Breeds: [opendev/system-config] 921321: Add an opendev specific build of mediawiki https://review.opendev.org/c/opendev/system-config/+/921321 | 15:57 | |
| @clarkb:matrix.org | mnasiadka: re mirror02.dfw.rax yes I think my initial investigation showed we had replaced the server with mirror03, but did not clean up the old server so we just need to clean that up. We should double check that mirror03 is the mirror in use, then plan to delete the old mirror and any volumes it has attached. One thing to keep in mind is that rax.dfw is where we run many control plane services so we awnt to be extra careful to not delete the wrong things there | 16:11 |
| @clarkb:matrix.org | Gerrit H2 v2 cache sizes look reasonable and mostly stable | 16:11 |
| @clarkb:matrix.org | which is good news | 16:11 |
| @clarkb:matrix.org | I've also done some spot checking of the docker cli talking to podman socket and I think it is command specific. `docker ps -a` seems to work but `docker image ls` does not | 16:21 |
| @clarkb:matrix.org | so maybe this isn't a drastic problem and we can continue to roll forward as is and work around it with podman commands as things come up | 16:21 |
| @clarkb:matrix.org | infra-root https://review.opendev.org/c/opendev/gerritlib/+/983476 https://review.opendev.org/c/opendev/jeepyb/+/983482 and https://review.opendev.org/c/openstack/project-config/+/983475 are all followups to the Gerrit upgrade that would be good to land. I think the last one is the most important and the one we should land first | 16:24 |
| @clarkb:matrix.org | mnasiadka: the other thing that occurred to me re the rebase button is that sometimes browsers don't properly reload the new Gerrit js blob after an upgrade. We've seen this affect the ability to edit changes within the browser after previous upgrades. I wonder if the problem was imply a stale set of application js code and loading the new stuff fixed it? | 16:26 |
| @mnasiadka:matrix.org | It might be that, Safari is picky and it might be me doing something on a mobile version as well - don't really remember | 16:51 |
| @mnasiadka:matrix.org | huh, infra-prod-base failed on 983911 - https://zuul.opendev.org/t/openstack/buildset/8096d47a75284e5ea06b561ab51cd477 | 16:53 |
| @clarkb:matrix.org | mnasiadka: the deployment logs are stored on bridge in /var/log/ansible/ in this case I think it will be /var/log/ansible/base.yaml.log for the latest one. Then tehre will be files with timestamp suffixes if you need to look at older runs as well | 16:54 |
| @clarkb:matrix.org | we don't put them directly in zuul logs due to the potential for leaking info | 16:55 |
| @clarkb:matrix.org | mirror02.sjc3.raxflex.opendev.org says unreachable=1 | 16:56 |
| @clarkb:matrix.org | and ya I seem to not be able to reach it | 16:56 |
| @mnasiadka:matrix.org | fatal: [mirror02.sjc3.raxflex.opendev.org]: UNREACHABLE! | 16:57 |
| @mnasiadka:matrix.org | yup | 16:57 |
| @clarkb:matrix.org | ok the openstack api says that node is in an ERROR state. But I dont see a reason | 16:58 |
| @clarkb:matrix.org | first thing we should check is that region enabled in opendev/zuul-providers and disable it if it is | 16:58 |
| @clarkb:matrix.org | https://opendev.org/opendev/zuul-providers/src/branch/master/zuul.d/providers.yaml#L274-L276 it is not | 16:59 |
| @clarkb:matrix.org | mnasiadka: so I think what we can do next is put that server in the /etc/ansible/hosts/emergency.yaml file to add it to the disabled hosts list while we investigate that should allow your buildset to deploy as it will skip over the node | 16:59 |
| @clarkb:matrix.org | mnasiadka: do you want to edit that file? it should be straightforward and have examples in there you can follow. I can check it after too | 17:00 |
| @clarkb:matrix.org | that file is on bridge sorry I wasn't clear about that | 17:00 |
| @mnasiadka:matrix.org | yeah, let me edit it | 17:00 |
| @clarkb:matrix.org | then we can reenqueue your change to the deploy pipeline as long as nothing else has merged to system-config since that change merged | 17:00 |
| @mnasiadka:matrix.org | added | 17:01 |
| @clarkb:matrix.org | and I think that is the only thing that has merged today to system-config (previous merge was the gerrit 3.12 change yesterday) | 17:01 |
| @mnasiadka:matrix.org | event list on that server in SJC3 OpenStack doesn't show anything at all for 2026 | 17:01 |
| @clarkb:matrix.org | mnasiadka: and this error must be somewhat recent as I think the base job has succeeded at leasti n the last month | 17:02 |
| @clarkb:matrix.org | mnasiadka: one thing to note is the value in emergency.yaml must match the value in system-config/inventory/base/hosts.yaml for the host entry name. It does in this case so all is well but I call that out because it has been a cause of problems in the past (mismatched values) | 17:02 |
| @mnasiadka:matrix.org | I thought so - so double checked with the inventory | 17:03 |
| @clarkb:matrix.org | so now the next step is reenqueuing your buildset for deployment. There are two ways of doing that. One is through the Zuul web ui if you create a keycloak account and login to zuul then when loggedin you'll get a button to reenqueue a buildset on the buildset page. Or we can use the zuul-client commands on either zuul01 or zuul02 | 17:03 |
| @mnasiadka:matrix.org | Probably the better option is having a keycloak account? | 17:04 |
| @clarkb:matrix.org | Its definitely easier I find so ya if you want to work on that that is fine. I can also do this last step if you don't have time | 17:04 |
| @mnasiadka:matrix.org | I think you can reenqueue that job in the meantime - and if there's some docs on keycloak - I'm happy to follow them | 17:05 |
| @clarkb:matrix.org | yup the docs are here: https://docs.opendev.org/opendev/system-config/latest/keycloak.html which point to https://docs.opendev.org/opendev/system-config/latest/sysadmin.html#zuul-admins | 17:05 |
| @clarkb:matrix.org | mnasiadka: ok I will login now and reenqueue that buildset | 17:05 |
| @mnasiadka:matrix.org | Should I raise a change to disable sjc3 in zuul-providers? | 17:06 |
| @clarkb:matrix.org | mnasiadka: it is already disabled. We did that due to problems with floating Ips back in like December. I should followup on that again | 17:07 |
| @clarkb:matrix.org | https://opendev.org/opendev/zuul-providers/src/branch/master/zuul.d/providers.yaml#L274-L276 shows that | 17:07 |
| @mnasiadka:matrix.org | ah, so that provider is special :) | 17:07 |
| @fungicide:matrix.org | Clark: yeah, that's what i meant by stale webclient, basically still using the old 3.11 javascript loaded and running in an old browser tab, trying to interact with a 3.12 server | 17:08 |
| @clarkb:matrix.org | ok that buildset has been reenqueued | 17:09 |
| @clarkb:matrix.org | fungi: got it | 17:12 |
| @clarkb:matrix.org | mnasiadka: base jobs succeeded so the other jobs are running now. I think you also need teh letsencrypt job to run as a dependency of the mirror job so one more hurdle to clear | 17:19 |
| @mnasiadka:matrix.org | Ok, zuul recognizes me as admin | 17:24 |
| @mnasiadka:matrix.org | seems the letsencrypt job is running right now, so given it's 7:30pm here - I'll probably recheck tomorrow morning | 17:25 |
| @clarkb:matrix.org | looks like LE is failing now | 17:26 |
| @clarkb:matrix.org | but you should enjoy your evening. This isn't urgent | 17:26 |
| @clarkb:matrix.org | I'll check the logs and if it looks like a reenqueue is useful I can reenqueue it again | 17:27 |
| @mnasiadka:matrix.org | DNS problem: NXDOMAIN looking up TXT for _acme-challenge.mirror03.ord.rax.opendev.org - check that a DNS record exists for this domain | 17:28 |
| @clarkb:matrix.org | did we not update DNS again? | 17:29 |
| @mnasiadka:matrix.org | oh well, I didn't somehow add these entries | 17:29 |
| @mnasiadka:matrix.org | let me fire off a patch | 17:29 |
| @clarkb:matrix.org | ack | 17:29 |
| -@gerrit:opendev.org- Michal Nasiadka proposed: [opendev/zone-opendev.org] 984421: Add missing mirror03.ord.rax ACME entries https://review.opendev.org/c/opendev/zone-opendev.org/+/984421 | 17:31 | |
| -@gerrit:opendev.org- Michal Nasiadka proposed: [opendev/zone-opendev.org] 984421: Add missing mirror03.ord.rax ACME entries https://review.opendev.org/c/opendev/zone-opendev.org/+/984421 | 17:31 | |
| @clarkb:matrix.org | oh we had the internal record set up but not the public record | 17:31 |
| @clarkb:matrix.org | cool +2 from me. I think we can let fungi or frickler review that just to doule check and then let daily runs take care of it | 17:33 |
| @clarkb:matrix.org | rather than continuing to reenqueue things. And that way you can enjoy your evening | 17:33 |
| @mnasiadka:matrix.org | ACK | 17:34 |
| @fungicide:matrix.org | oops, i missed that when reviewing the other change | 17:46 |
| -@gerrit:opendev.org- Zuul merged on behalf of Michal Nasiadka: [opendev/zone-opendev.org] 984421: Add missing mirror03.ord.rax ACME entries https://review.opendev.org/c/opendev/zone-opendev.org/+/984421 | 17:53 | |
| @fungicide:matrix.org | it's not often i see this, but dict.org (the backend for the `dict` command-line dictionary utility) is offline. i wonder if they're battling the same llms we have been | 18:15 |
| @clarkb:matrix.org | fungi: if you get a chance can you review https://review.opendev.org/c/openstack/project-config/+/983475 I think that is the most critical of the followup changes after the gerrit upgrade. basically get jeepyb gerrit jobs in sync with what we deploy now. I'm about to grab lunch but when I get back I'll also work on changes to start cleaning up the 3.11 stuff whcih we can land as soon as a revert seems unlikely | 19:13 |
| @clarkb:matrix.org | also did you see my note about docker commands? it seems the specific command matters as docker ps -a works fine but docker image list does not. So I think we may mostly be ok for now | 19:13 |
| @fungicide:matrix.org | looking at 983475 | 19:14 |
| @fungicide:matrix.org | 983911 deploy failed twice in a row, that's the one we were just going to wait on the periodic jobs to eventually take care of? | 19:28 |
| @clarkb:matrix.org | fungi: yes. It failed the first time on the unreachable sjc3 mirror. We added that server to the emergency file then I reenqueued the buildset and it failed this time on the lack of the acme redirect dns record | 19:31 |
| @clarkb:matrix.org | If that record has been created in prod we may be able to reenqueue that buildset a third time if nothing else has merged to system-config which might conflict. I figured periodic was good enough | 19:32 |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [openstack/project-config] 983475: Bump jeepyb gerrit jobs up to gerrit 3.12 https://review.opendev.org/c/openstack/project-config/+/983475 | 19:33 | |
| @clarkb:matrix.org | excellent. I'm going to work on those gerrit 3.11 cleanups next and then I should look at meeting agenda updates. Let me know if you have any meeting agenda edits that i need to make | 20:06 |
| @fungicide:matrix.org | i didn't have any | 20:06 |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/system-config] 984468: Drop Gerrit 3.11 jobs https://review.opendev.org/c/opendev/system-config/+/984468 | 20:12 | |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/system-config] 984468: Drop Gerrit 3.11 jobs https://review.opendev.org/c/opendev/system-config/+/984468 | 20:14 | |
| @clarkb:matrix.org | https://review.opendev.org/q/hashtag:%22upgrade-gerrit-3.12%22+status:open pushing changes for post gerrit upgrade tasks to this hashtag | 20:15 |
| @clarkb:matrix.org | for 984468 the timing for landing that one is a bti of a judgement call. Basically we want to wait long enough that a revert to 3.11 is unlikely | 20:15 |
| @clarkb:matrix.org | I think the other changes can merge now as they aer just improving our test coverage of more gerrit versions in various tools | 20:15 |
| -@gerrit:opendev.org- Clark Boylan proposed: | 20:23 | |
| - [opendev/git-review] 984279: Add py313 testing against Gerrit 3.12 https://review.opendev.org/c/opendev/git-review/+/984279 | ||
| - [opendev/git-review] 984470: Drop global statement for unassigned var https://review.opendev.org/c/opendev/git-review/+/984470 | ||
| @clarkb:matrix.org | I'm hoping that fixes the problems with git-review that were exposed yesterday | 20:23 |
| @clarkb:matrix.org | note that none of them affect git review in the wild. Just our test jobs I think | 20:23 |
| @clarkb:matrix.org | I put some initial edits on https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting but it has been the sort of weeks where I wouldn't be surprised if I overlooked something important. Feel free to add/edit/cleanup content or let me know if anything needs that effort. I've got to do a school run shortly but will send the agenda when I return | 20:53 |
| @clarkb:matrix.org | fungi: the git-review change is still failing 3 test cases, but it passes most of them and actually runs gerrit successfully now | 20:55 |
| @iwienand:matrix.org | https://github.github.com/gh-stack/ | 22:12 |
| Break large changes into small, reviewable pull requests that build on each other — with native GitHub support and the gh stack CLI. | ||
| 🙄 | ||
| @mnaser:matrix.org | Where have I seen this before... | 22:14 |
| @iwienand:matrix.org | ... seen before and didn't require a special client to do it | 23:01 |
| @clarkb:matrix.org | I'm upgrading my home fileserver and it looks like Jammy may have gotten the same update that noble got that broke `docker image ls` against a podman socket | 23:18 |
| @clarkb:matrix.org | we just notice it on noble bceause its the only place where we pretend that docker is actually podman | 23:18 |
| @clarkb:matrix.org | we also install docker from upstream outside of noble iirc so don't need to wait for ubuntu to do these updates | 23:19 |
| -@gerrit:opendev.org- Steve Baker proposed: [openstack/diskimage-builder] 984479: WIP: Add ublk block device plugin as loopback alternative https://review.opendev.org/c/openstack/diskimage-builder/+/984479 | 23:20 | |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/git-review] 984279: Add py313 testing against Gerrit 3.12 https://review.opendev.org/c/opendev/git-review/+/984279 | 23:47 | |
| @clarkb:matrix.org | ianw: oh wow github is doing that first party now | 23:50 |
| @clarkb:matrix.org | https://github.github.com/gh-stack/introduction/overview/ this has concrete info on how it works and its basically the gerrit design | 23:51 |
| @clarkb:matrix.org | after years of people screaming about how gerrit forces you to rebase and is bad I guess they decided it was worthwhiel afterall | 23:51 |
| @clarkb:matrix.org | https://github.github.com/gh-stack/introduction/overview/#thinking-about-stack-structure like its literally Gerrit | 23:52 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!