| tkajinam | hmm. it seems reset author works. wondering if something can be fixed at infra side or I should update number of patches by my side... | 01:28 |
|---|---|---|
| tkajinam | hmmm. we may face a same problem when we attempt to backport a change but its author's email can't be validated so I think we have to fix it if the change in email address is the cause | 02:35 |
| frickler | tkajinam: the issue is a bug in gerrit, I mentioned a patch yesterday we could likely apply, but we'll have to wait for people to recover from stuffing turkeys it seems, so if you need some patch to merge soon, better do the email amending | 07:05 |
| tkajinam | frickler, yeah I did reset-author for some patches we may want to merge soon, but will leave the other non-urgent ones until people come back from nice thanksgiving holidays and we get a conclusion. | 07:16 |
| mrunge | How would one look at CI results these days, like https://zuul.opendev.org/t/openstack/build/32da70b1a6894ce7a9faed381828aafc for https://review.opendev.org/c/openstack/aodh/+/896091 | 10:46 |
| ykarel | until the issue is fixed can check log url in https://zuul.opendev.org/api/tenant/openstack/build/32da70b1a6894ce7a9faed381828aafc | 10:48 |
| mrunge | thank you ykarel! This is good to know | 10:49 |
| opendevreview | Takashi Kajinami proposed openstack/project-config master: Redirect irc notifications from os-(apply|collect|refresh)-config https://review.opendev.org/c/openstack/project-config/+/901916 | 12:28 |
| opendevreview | Takashi Kajinami proposed openstack/project-config master: Move os-(apply|collect|refresh)-config to heat's queue https://review.opendev.org/c/openstack/project-config/+/901917 | 12:32 |
| opendevreview | Takashi Kajinami proposed openstack/project-config master: Move os-(apply|collect|refresh)-config to heat https://review.opendev.org/c/openstack/project-config/+/901916 | 12:39 |
| dpanech | Hello, we are getting a Zuul error, it just says "Something went wrong" with no further information, could someone help? Here: https://zuul.opendev.org/t/openstack/build/0a10c781f0434f9c9fe77bcc9456dde8 | 14:59 |
| fungi | dpanech: yes, an upgrade over the weekend brought in a new version of a javascript library that has exposed a bug in the dashboard. we're working on fixing it now: https://review.opendev.org/901945 | 15:25 |
| dpanech | fungi: ok thanks | 15:46 |
| fungi | dpanech: looks like the fix we have will solve it based on the preview build in check, so it's been approved. once it merges and new container images appear, we'll restart the web front-ends to get it into production asap | 16:00 |
| dpanech | fungi: thank you | 16:01 |
| clarkb | tkajinam: frickler: this feels like somethnig that upstream should definitely fix rather than us carrying a local backport as others will be affected too | 16:07 |
| clarkb | frickler: I can bring it up with them this morning | 16:07 |
| clarkb | first thing to confirm is that the identified fix is not on stable-3.8. It is not so upstream probably does need a backport | 16:13 |
| clarkb | I've asked in discord if there is a reason to not backport and how far back it needs to be backported. It does cherry pick cleanly | 16:28 |
| opendevreview | Merged opendev/zone-opendev.org master: Add DNS records for mirror02.bhs1.ovh and mirror03.gra1.ovh https://review.opendev.org/c/opendev/zone-opendev.org/+/901627 | 16:51 |
| clarkb | infra-root are we ready to clean up gerrit iamges? https://review.opendev.org/c/opendev/system-config/+/901467 I can reapprove that one if so | 17:08 |
| fungi | i think so | 17:09 |
| clarkb | also there are new gerrit point releases so I can actually stick a new chagne int he middle to update the versions and make 3.9 the proper version from the start | 17:09 |
| fungi | seems pretty much no chance we'll roll back the upgrade at this point | 17:09 |
| tonyb | clarkb: I don't think we've seen anything that would cause a rollback | 17:09 |
| tonyb | #makeitso | 17:09 |
| clarkb | ya the issue that tkajinam and frickler debugged is probably the biggest one and we have a workaround for that and a presumed fix we can backport | 17:09 |
| clarkb | I'll approve it | 17:10 |
| clarkb | and now to update the rest of that stack with the latest releases | 17:10 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Add gerrit 3.9 image builds https://review.opendev.org/c/opendev/system-config/+/901468 | 17:21 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Add gerrit 3.8 to 3.9 upgrade testing https://review.opendev.org/c/opendev/system-config/+/901469 | 17:21 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update Gerrit 3.8 images to 3.8.3 https://review.opendev.org/c/opendev/system-config/+/901992 | 17:21 |
| clarkb | ok I updated the 3.9 stuff to 3.9.0 from 3.9.0-rc5 and then stacked the 3.8.3 upgrade on top of that so that we don't have to get prod updated to 3.8.3 before adding 3.9 testing (which I thik we can just go ahead and approve now if people want) | 17:21 |
| clarkb | tkajinam: frickler: upstream indicated that the identified fix should be backported to stable-3.8. I have done so here: https://gerrit-review.googlesource.com/c/gerrit/+/395016 | 17:29 |
| clarkb | if we get lucky we'll pull it in when https://review.opendev.org/c/opendev/system-config/+/901992 lands (depends on timing) | 17:30 |
| opendevreview | Merged opendev/system-config master: Cleanup Gerrit 3.7 image jobs and disable Gerrit upgrade job https://review.opendev.org/c/opendev/system-config/+/901467 | 17:31 |
| tonyb | In https://review.opendev.org/c/opendev/system-config/+/901628/comment/02cfd5fc_c1a754a3/ frickler asked "Just wondering if we should keep increasing the ids for new mirrors or recycle deleted ones, i.e. can we go back to mirror01 instead?" I don't have a strong preference. | 17:31 |
| tonyb | I can see arguments for either option. It isn't a blocker but something for us to think about in prep for the upcoming jammy->noble transition | 17:33 |
| clarkb | one upside to increasing the numbers is we avoid weird ansible caching problems | 17:34 |
| clarkb | ansible isn't great about knowing a new host has the same name and yiou have to do manual cleanup | 17:34 |
| fungi | we often have reused the lowest available number when replacing servers, unless it would cause confusion for some historical data tracking. for mirrors it seems safe enough, but also nothing i'd worry about redoing work over | 17:34 |
| clarkb | personalyl I like avoiding those problems in the frist place and just have new numbers | 17:34 |
| fungi | and yes, it got more problematic when we switched to ansible | 17:34 |
| fungi | since you may have to clear the ansible fact cache on the bridge | 17:35 |
| clarkb | infra-root in addition to the gerrit image updates the base python image updates for python3.12 should be ready to go now https://review.opendev.org/c/opendev/system-config/+/898756 and parent | 17:37 |
| tonyb | That's a good point about fact caching. It is only a cache, so we could just remove it whenever we delete a host (although just removing that host would be better) as it'll just recreate on the next run. | 17:39 |
| tonyb | The only change where it's relevant now is the mirror03 one and it isn't worth rebuilding that server. | 17:40 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update gitea to 1.21.1 https://review.opendev.org/c/opendev/system-config/+/897679 | 17:43 |
| clarkb | I'm not going to bother with update the hold node yet since we want to do gerrit key rotation first | 17:43 |
| clarkb | but I wanted to keep the change up to date with upstream updates | 17:43 |
| clarkb | the zuul web fix has landed. | 18:20 |
| clarkb | once images promote we should probably pull and mnaully update the zuul-web services on zuul01 and zuul02? | 18:21 |
| corvus | yeah, i'll go ahead and restart zuul-web | 18:21 |
| tonyb | clarkb, fungi: On my todo list for today is to boot the new mirrorXX.dfw did either/both of you want to do a meetup+screen session | 18:21 |
| clarkb | tonyb: I should be able to. What time works for you? | 18:22 |
| clarkb | I can do nowish or after lunch, but late afternoon I need to go and refill my bins full of leaves since they just collected them | 18:24 |
| fungi | thanks again corvus! | 18:25 |
| fungi | https://zuul.opendev.org/components is so convenient for monitoring restarts | 18:26 |
| clarkb | corvus: thanks! | 18:27 |
| corvus | #status log restarted zuul-web to fix js errors | 18:27 |
| opendevstatus | corvus: finished logging | 18:27 |
| fungi | https://zuul.opendev.org/t/zuul/build/abf219ddd8d5491fa0f6e0af2a17b033 works now! | 18:28 |
| clarkb | I can browse builds from the builds list to their logs | 18:28 |
| clarkb | maybe we should do a #status Notice Zuul build urls should be working again | 18:28 |
| clarkb | s/Notice/notice/ | 18:28 |
| fungi | frickler: mrunge: ykarel: dpanech: ^ | 18:28 |
| corvus | status notice zuul build urls should be working again (browser refresh may be required) | 18:29 |
| corvus | how about that ? | 18:29 |
| clarkb | corvus: lgtm | 18:30 |
| corvus | #status notice Zuul build urls should be working again (browser refresh may be required) | 18:30 |
| opendevstatus | corvus: sending notice | 18:30 |
| -opendevstatus- NOTICE: Zuul build urls should be working again (browser refresh may be required) | 18:30 | |
| clarkb | frickler: tkajinam upstream has merged the stable-3.8 backport of the identified fix. That means when we land https://review.opendev.org/c/opendev/system-config/+/901992 and then restart gerrit on the updated version this problem should go away | 18:30 |
| opendevstatus | corvus: finished sending notice | 18:33 |
| clarkb | hrm the gerrit 3.8 to 3.9 upgrade fails now... | 18:43 |
| fungi | connection refused from the rest api | 18:46 |
| clarkb | ya its happening due to a lucene issue. I think this may be an actual problem wtih gerrit's upgrade path so I'm reporting it to them now | 18:47 |
| fungi | exciting! | 18:47 |
| fungi | "This index was initially created with Lucene 7.x while the current version is 9.8.0 and Lucene only supports reading the current and previous major versions. This version of Lucene only supports indexes created with release 8.0 and later by default." | 18:48 |
| clarkb | yup thats the error | 18:48 |
| tonyb | clarkb: anytime 11:15 - 15:00 your time. | 18:48 |
| fungi | yeah, looks like they need some sort of intermediate lucene migration, or to blow away the existing index | 18:49 |
| clarkb | fungi: ya either the bug is in their lucene updates or in their release notes saying online reindexing is possible. I hope they stick to online reindexing myself and fix lucene | 18:49 |
| clarkb | tonyb: how about 11:30? I'm working on early lunch/late breakfast right now | 18:49 |
| tonyb | clarkb: Yup. No rush. | 18:50 |
| clarkb | I'm super happy we have this gerrit upgrade test job. Its been super useful | 18:51 |
| tonyb | Yeah it's really cool. Does upstream have anything like it? | 18:52 |
| clarkb | I'm not sure. Luca was saying they do run the gatling load tester against release candidates before doing releases as one of their pre release tasks | 18:52 |
| fungi | i like that he referred in their channel to "...our ci jobs that test the upgrade..." | 18:53 |
| fungi | maybe that'll get them wanting something similar | 18:53 |
| tonyb | where is said channel? | 18:53 |
| fungi | https://matrix.to/#/#gerritcodereview:matrix.org | 18:54 |
| clarkb | tonyb: it is the gerrit discord channel which is mirrored to ^ | 18:54 |
| fungi | though it's a mirror of... yeah what clarkb said | 18:54 |
| clarkb | I'm actually connected to both discord and matrix .... but I really only use the discord server for the monthly community meeting otherwise I prefer to use matrix | 18:54 |
| clarkb | I'm going to stack the 3.8.3 image update under the upgrade change now so that it is mergeable | 18:55 |
| fungi | sounds good, no reason to hold it up since we're not going to be upgrading to 3.9 soon anyway | 18:56 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update Gerrit 3.8 images to 3.8.3 https://review.opendev.org/c/opendev/system-config/+/901992 | 18:56 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Add gerrit 3.8 to 3.9 upgrade testing https://review.opendev.org/c/opendev/system-config/+/901469 | 18:56 |
| fungi | tonyb: clarkb: i'll probably skip the call today, i've got to take a gardening break to ready some plants for freezing temperatures predicted tomorrow night, and had to wait until the rain let up so that's basically nowish | 18:58 |
| tonyb | fungi: okay. | 18:58 |
| tonyb | fungi: I expect it'll be pretty quick/easy I just want extra eyes because it's dfw. | 19:00 |
| fungi | makes sense, and yeah it should be straightforward | 19:02 |
| corvus | i'm going to restart zuul-web a second time in order to run the schema migration that should fix periodic build queries | 19:23 |
| clarkb | tonyb: ready when you are | 19:29 |
| tonyb | clarkb: https://meetpad.opendev.org/opendev-root-bootstrap | 19:30 |
| tonyb | I figure we can reuse that one | 19:30 |
| corvus | https://zuul.opendev.org/t/openstack/builds?project=openstack%2Fneutron&branch=master&pipeline=periodic works now | 19:39 |
| frickler | I like how gerrit is making mode changes more obvious now https://review.opendev.org/c/openstack/designate/+/901594 | 20:16 |
| fungi | ooh, that's a nice little warning sign | 20:42 |
| clarkb | looks like gerrit confirms the lucene issue is a problem | 20:49 |
| fungi | apparently they didn't mean to increase the lucene version quite that far in 3.9 | 20:50 |
| fungi | though now that they have... i wonder what their options are | 20:50 |
| clarkb | they are discussing it in a differetn discord channel that isn't bridged to matrix | 20:50 |
| fungi | clearly downgrading lucene for people who have already installed 3.9 would be tricky | 20:50 |
| clarkb | seems like all of the options are not great so will be interesting to see what they end up with | 20:50 |
| fungi | even having not seen the discussion i could pretty much guess that | 20:51 |
| clarkb | the good news if there is any is that I was lucky enough to catch it less than a week since the 3.9.0 release | 20:52 |
| clarkb | hopefully that means the number of people who may have hit issues due to it is small | 20:53 |
| fungi | the sad news is that if they had an upgrade test like ours in their ci, it could have blocked the change that dragged in that unintentional increase | 20:54 |
| fungi | i notice that their merged-as and parent links in the change info have a link to gitiles instead of linking a gerrit search query like we do (so it also works with merge commit parents). i wonder how they do that? | 20:56 |
| fungi | we could in theory put gitea overrides there | 20:57 |
| clarkb | fungi: I think if you use gitiles its part of the plugin | 20:57 |
| fungi | aha, so we'd need a custom plugin for that | 20:57 |
| corvus | they do have a zuul, but no one is really dedicated to writing jobs for it. i think they would welcome contributions if someone wanted to set up an upgrade test. | 20:57 |
| clarkb | I'm letting them know that a 25 minute offline reindex for us isn't the end of the world for upgrading | 20:59 |
| clarkb | as a datapoint for them to consider their options | 20:59 |
| fungi | yeah, as long as we know to plan for it, i'd be fine with that | 20:59 |
| clarkb | ya the problem they have is anyone that has newly deployed 3.9.0 or blazed ahead with an offline reindex will get stranded if they revert. if they don't revert the rest of us have to do an offline reindex too | 21:02 |
| opendevreview | Tony Breeds proposed opendev/zone-opendev.org master: Add DNS records for mirror02.dfw.rax https://review.opendev.org/c/opendev/zone-opendev.org/+/902007 | 21:33 |
| opendevreview | Tony Breeds proposed opendev/system-config master: Add a helper script for doing the LVM setup on mirror nodes. https://review.opendev.org/c/opendev/system-config/+/901504 | 21:35 |
| opendevreview | Tony Breeds proposed opendev/system-config master: Add inventory/LE records for mirror02.bhs1.ovh and mirror03.gra1.ovh https://review.opendev.org/c/opendev/system-config/+/901628 | 21:35 |
| opendevreview | Tony Breeds proposed opendev/system-config master: Add inventory/LE records for mirror02.dfw.rax https://review.opendev.org/c/opendev/system-config/+/902008 | 21:35 |
| opendevreview | Tony Breeds proposed opendev/zone-opendev.org master: Add DNS records for mirror02.dfw.rax https://review.opendev.org/c/opendev/zone-opendev.org/+/902007 | 21:38 |
| *** dmellado206 is now known as dmellado20 | 22:08 | |
| tonyb | I am okay with: https://review.opendev.org/q/topic:%22gerrit-3.7-cleanup%22+is:open+label:Verified%3E%3D0 but I'm hesitant to +W them because I'm not certain what will happen once 901992 merges. | 22:36 |
| tonyb | I *think* it will update the gerrit-compose to the new tags after building and publishing the images but it will *not* restart gerrit. | 22:37 |
| tonyb | and an infra-root will need to do that manually at a "safe time" | 22:37 |
| fungi | yeah, we usually hold off approving that sort of change until we're ready to perform a controlled restart of the service, just in case something happens to the server that forces it to be rebooted | 22:41 |
| fungi | safer for unexpected reboots from provider outages to result in running a version of gerrit we've been using for a while than one we haven't tried outside of testing | 22:41 |
| tonyb | fungi: Thanks for confirming. | 22:45 |
| clarkb | the 3.9 image creation is probably good to hold off on now too | 22:55 |
| clarkb | upstream is talking about pulling the release | 22:55 |
| clarkb | I think I'll resync on all that tomorrow after it has time to settle and update the changes if necessary | 22:55 |
| tonyb | Okay. | 22:56 |
| fungi | oh fun | 22:56 |
| clarkb | I've been trying to do any last debugging on this laptop before I call lenovo for warranty stuff. Booting nomodeset (so disabling the gpu entirely) works but at lower (fuzzy resolution) | 22:57 |
| clarkb | one suggestion I found was disabling the dynamic power management might help but doing that I get no screen after askign grub to boot | 22:57 |
| clarkb | the issue is present in jammy though. I think I might try focal? | 22:58 |
| clarkb | its fun that this is so complicated and breaks often enough in linux that you can't really tell if it is hardware or software | 22:58 |
| tonyb | that sounds terrible | 22:59 |
| tonyb | I think I'm going to step away for the night. I'll figure out why the mirror testing is failing tomorrow | 23:00 |
| clarkb | ya about the only good news is that it is broken on ubuntu jammy so maybe I have some hope someone else has run into this if it isn't a hardware issue | 23:00 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!