| opendevreview | Clark Boylan proposed opendev/system-config master: DNM holding and testing a hacky gerrit build https://review.opendev.org/c/opendev/system-config/+/936986 | 00:01 |
|---|---|---|
| clarkb | ok theres the hacky thing assuming I got my use of patch correct. Now to hold the node and update the issue | 00:01 |
| opendevreview | Clark Boylan proposed opendev/system-config master: DNM holding and testing a hacky gerrit build https://review.opendev.org/c/opendev/system-config/+/936986 | 00:04 |
| clarkb | forgot to git add the patch file | 00:04 |
| clarkb | paladox: updated | 00:13 |
| paladox | 👍 | 00:13 |
| clarkb | the captcha in front of that issue tracker is serious :) | 00:13 |
| clarkb | but I also noticed there was a lot of spam on the issue tracker so that is probably why | 00:13 |
| fungi | or maybe a sign the captcha is useless for anything except annoying meat-based entities | 00:14 |
| clarkb | I had to do like 6 different captchas | 00:15 |
| clarkb | find all the cars, now find all the busses, now find all the squares with a traffic light in them, now find all the squares with a bus in them, now find all the sqaures with a bicycle | 00:15 |
| fungi | that happens to me a lot. i suspect it's because i use linux and so every site assumes my ua is made-up | 00:15 |
| clarkb | I feel like there was one more but I can't remember what it was | 00:15 |
| fungi | mice enjoying a pleasant meal of multi-mode fiber jacketing | 00:16 |
| clarkb | also now that I know about bazelisk I appreciate how easy it makes bootstrapping a gerrit build env | 00:16 |
| fungi | no wait, those were rats! see, to err is meat | 00:17 |
| clarkb | I can't believe I spent years without it | 00:17 |
| clarkb | that said I do wish that bazelisk and bazel to a less extent felt less like I'm one step away from a supply chain attack | 00:17 |
| fungi | be the supply chain attack you want for others | 00:17 |
| * fungi is full of motivational quips tonight | 00:18 | |
| clarkb | you do still have to install a c++ compiler and maybe make though | 00:18 |
| fungi | unless you use gentoo, i guess ;) | 00:20 |
| fungi | one of my favorite things: that debian-installer can set up and install into luks-encrypted block devices out of the box; one of my least favorite things: waiting for debian-installer to write random data to the entirety of a 2tb physical device before using it for encrypted storage... sigh | 00:23 |
| fungi | at least i'm giving the nvme heatsink and cooling fans on this little thing a workout | 00:24 |
| fungi | it does say i can cancel the operation and proceed, "albeit at the expense of a slight reduction of the quality of the encryption" | 00:26 |
| clarkb | I was going to say do you need a faster drive or better source of entropy? | 00:52 |
| clarkb | I have a held node with my hacky openid patch now (at least in theory the patch is in there) I'll have to test that tomorrow though as I've got a family thing tonight | 00:53 |
| opendevreview | Karolina Kula proposed openstack/diskimage-builder master: WIP: Add support for CentOS Stream 10 https://review.opendev.org/c/openstack/diskimage-builder/+/934045 | 09:37 |
| *** tosky_ is now known as tosky | 15:40 | |
| clarkb | fungi: I'm going to push up a change to purge those backups from the backup server then dive into testing my openid hack in gerrit and finally retesting the gerrit 3.9 -> 3.10 upgrade with new images. I think we can still land the gitea upgrade though and keep an eye on that as we do other things if we are still up for it | 15:50 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Purge previously retired backups on vexxhost backup server https://review.opendev.org/c/opendev/system-config/+/937040 | 15:55 |
| clarkb | thats the first thing. I don't think landing that is urgent but may be worth doing while everything is still paged in | 15:55 |
| Ramereth[m] | I wish I knew. I'm trying to get in contact with the people behind it. I'll let you know when I find out | 16:01 |
| fungi | Ramereth[m]: thanks! not super important, more just professional curiousity | 16:03 |
| clarkb | didn't take long for me to run into a type error. I guess java's type system couldn't protect me well enough. I'll respin that patch and then look at other things while that builds | 16:09 |
| fungi | rackspace opened a ticket a few minutes ago to let us know that the host underlying our zuul-v3 trove instance had an unexpected outage | 16:15 |
| opendevreview | Clark Boylan proposed opendev/system-config master: DNM holding and testing a hacky gerrit build https://review.opendev.org/c/opendev/system-config/+/936986 | 16:15 |
| clarkb | fungi: we're not using that db anymore I don't think | 16:16 |
| fungi | looks like we no longer rely on trove for that, and instead have a zuul-db01 server for that since ~8 months | 16:17 |
| clarkb | we can probably claen it upat this point as we're using the dedicated insance | 16:17 |
| fungi | yeah, that's what i'm thinking too | 16:17 |
| fungi | corvus: ^ opinion? (not urgent) | 16:17 |
| fungi | i went ahead and closed the ticket | 16:18 |
| clarkb | fungi: https://review.opendev.org/c/opendev/system-config/+/936198 is the gitea change if you think we should still proceed with that | 16:19 |
| clarkb | the changelog for 3.10.3 specifically looks very similar to the one for 3.9.8 so I suspect we're getting a consistent set of backports from 3.11/master | 16:19 |
| clarkb | which is good it means I don't have a bunch of extra stuff to evaluate before we upgrade. I just need to rerun through the actual upgrade process | 16:19 |
| fungi | yep, approving 936198 now | 16:19 |
| corvus | hey ho trove (for zuul) can go | 16:19 |
| fungi | thanks corvus! | 16:20 |
| fungi | good thing i haven't logged out of rackspace yet, i'll just clicky-clicky it away | 16:20 |
| corvus | fungi: maybe take a look to see if there are any other zuul-ish ones too | 16:20 |
| fungi | oh, in fact, we have two. there was a zuul-v10 trove instance | 16:21 |
| corvus | i think at some point we had more than one as we experimented with newer instances | 16:21 |
| fungi | yep | 16:21 |
| corvus | yeah that | 16:21 |
| corvus | i think that can go too | 16:21 |
| fungi | thanks, both will be zapped | 16:21 |
| fungi | #status log Deleted two no-longer-used Trove database instances previously storing Zuul DB reporter results | 16:22 |
| opendevstatus | fungi: finished logging | 16:24 |
| fungi | mildly amusing, rackspace's e-mail service that we use for our root contact alias flagged two automated messages from the rackspace ticket system as spam/junk (not the first time i've seen it happen, but the first time it's dawned on me they're ironically misclassifying their own messages) | 16:24 |
| clarkb | fungi: swinging back to list bounce processing we should have the verp probes enabled now so any removals at this point are legit. But we still need to either manually reenable any that were deactivated in openstack-discuss via the ui or script something up to do that? | 16:29 |
| clarkb | I don't know how many there are in total nor how tedious the manual steps are. I wonder if the web ui has a select all and then you cna hit enable on them setting enabled to enabled and disabled to enabled | 16:30 |
| clarkb | but I haven't logged in to check so that is probably wishful thinking | 16:30 |
| fungi | clarkb: already done 2 days ago: https://meetings.opendev.org/irclogs/%23opendev/%23opendev.2024-12-02.log.html#t2024-12-02T19:27:21 | 16:35 |
| clarkb | fungi: oh cool I had completely missed that. What process did you end up using for that? | 16:36 |
| fungi | i used the trivial client script mentioned in the mailman-users post linked just above there in discussion | 16:36 |
| clarkb | gotcha | 16:36 |
| fungi | there was a minor typo in the post, easily corrected after seeing the traceback | 16:36 |
| clarkb | as an expert in creating minor typos I feel I can't complain. Its cool that worked | 16:37 |
| fungi | well, the usual warning about mailercode is implicit, at least to me | 16:38 |
| clarkb | this softawre comes with no warranty | 16:40 |
| corvus | this sysadmin comes with no warranty | 16:41 |
| fungi | this warranty comes with no warranty comes with no warranty comes with no... | 16:42 |
| fungi | the openeuler rsync hit a timeout error: | 16:47 |
| fungi | [generator] io timeout after 600 seconds -- exiting | 16:47 |
| fungi | rsync error: timeout in data send/receive (code 30) at io.c(204) [generator=3.1.3] | 16:47 |
| fungi | i'll restart it | 16:47 |
| clarkb | ya that seems like a network timeout and restarting makes sense | 16:48 |
| fungi | it's resumed | 16:48 |
| fungi | looks like it probably died around 01:00 utc, but i didn't see the error for some reason until i entered screen's scrollback buffer viewing mode and exited that again | 16:51 |
| fungi | so it really only ran for 3-4 hours before it terminated | 16:52 |
| fungi | hopefully it'll stick it out this time | 16:52 |
| clarkb | success! my hack to the openid callback method in gerrit gets me the behavior we see with upstream gerrit and the developer any login is a good login auth type | 16:59 |
| clarkb | and the logging I added seems to trace out what I suspected. I'll put a paste together and post it to that chagne so people can see what it does | 17:00 |
| clarkb | ok log paste posted to https://review.opendev.org/c/opendev/system-config/+/936986 | 17:03 |
| clarkb | paladox: ^ fyi that may or may not be interesting to you. But I think this confirms my prior suspicion that openid makes the sign in problem worse due to this issue in the callback method | 17:04 |
| clarkb | I suspect a proper fix does someting more like "if (!(rdr.endsWith("/") && nextToken == "/")) { rdr.append(nextToken); } | 17:06 |
| opendevreview | Clark Boylan proposed opendev/system-config master: DNM holding and testing a hacky gerrit build https://review.opendev.org/c/opendev/system-config/+/936986 | 17:16 |
| clarkb | ok I've updated to something that I can probably propose upstream. Goign to rehold and retest that though | 17:17 |
| clarkb | considering this is auth code though being cautious is probably a good idea? | 17:18 |
| clarkb | I don't think we should try and rush that in as part of or before the 3.10 upgrade | 17:18 |
| fungi | yeah | 17:25 |
| clarkb | I even remembered you can't use strfoo == strbar in java to test string equality (that checks reference equality iirc) | 17:29 |
| fungi | oh, weird, like the "is" operator in python then? | 17:30 |
| clarkb | ya | 17:32 |
| clarkb | with that pretty well understood I'm going to take a break but should be back in time to watch the gitea upgrade then retest the gerrit upgrade with new images | 17:40 |
| fungi | yeah, i'm around, just making pizza dough for this afternoon | 17:40 |
| fungi | will keep an eye on its progress | 17:40 |
| BlaisePabon[m] | I have been baking a lot of sourdough lately and curious about pizza dough. | 17:48 |
| fungi | i have made sourdough pizza dough many times, but it takes a lot of skill to work it into shape because sourdough tends to be a lot more sticky than doughs levened with directly added yeast | 17:51 |
| fungi | it does work well for neopolitan-style crusts particularly though, really any higher-heat baking is perfect for sourdough | 17:52 |
| fungi | i cheat by shaping the dough directly onto parchment rounds and then slip the parchment out from under the crust about 5 minutes into the bake, otherwise it's really hard to move around with a peel (dusting the work surface with flour and cornmeal only gets you so far) | 17:53 |
| fungi | zuul thinks the gitea upgrade is 5 minutes from merging | 18:07 |
| opendevreview | Merged opendev/system-config master: Upgrade gitea to 1.22.4 https://review.opendev.org/c/opendev/system-config/+/936198 | 18:10 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update Gerrit image tag to 3.10 (from 3.9) https://review.opendev.org/c/opendev/system-config/+/937051 | 18:10 |
| clarkb | infra-root ^ I've marked that WIP in preparation for the upgrade. There is https://review.opendev.org/c/opendev/system-config/+/934075 which should be safe to land now so that when we update to 3.10 we pick up those config updates for log rotation though | 18:11 |
| clarkb | but reviews on both of them in preparation for the upgrade are apprecaited | 18:11 |
| fungi | promote completed, install is pending | 18:14 |
| fungi | looks like it's stuck behind an hourly set, but that's not got much longer | 18:15 |
| clarkb | cool I'll jump on gitea09 and prepare to check them in series as they update | 18:17 |
| clarkb | I should note that 934075 is not critical for the upgrade either we can always apply that later | 18:19 |
| clarkb | its just a minor optimization that gerrit 3.10 allows us to do. Up to us if we want to deal with that post upgrade or as part of the upgrade | 18:20 |
| tonyb | 934075 looks good to me, I'm happy for it to land $whenever works for y'all. | 18:23 |
| clarkb | tonyb: I think we can approve it as soon as the gitea upgrade is ocmplete and happy (it shouldn't affect gerrit 3.9 it shoudl ignore it which our test jobs also confirm) | 18:24 |
| clarkb | and only delaying for gitea to be done so we don't have multipel things in flight to distract us | 18:24 |
| clarkb | gitea09 has updated | 18:26 |
| clarkb | https://gitea09.opendev.org:3081/opendev/system-config/ seems to load for me | 18:26 |
| clarkb | I'll start a clone too | 18:26 |
| fungi | seems to be working for me | 18:26 |
| fungi | Powered by Gitea Version: v1.22.4 | 18:27 |
| clarkb | git clone is of the above link is happy for me too | 18:27 |
| fungi | i was browsing on 3000 but 3081 is also working | 18:27 |
| clarkb | gitea10 is done now too | 18:27 |
| clarkb | and now 11 is done. Halfway there | 18:29 |
| clarkb | all 6 are done now and reaching them directly I see the expected versions | 18:33 |
| clarkb | if the job completes successfully I think the last thing I'd like to double check is replication | 18:33 |
| clarkb | anyone have a change to push or a new patchset? | 18:34 |
| clarkb | usually we can wait for a random patchset to be pusehd too and just check that | 18:34 |
| clarkb | https://zuul.opendev.org/t/openstack/build/c93f7dfc60614f30890413f53f842f08 job was successful /me looks for something that has been pushed to check replication | 18:34 |
| clarkb | nothing yet | 18:35 |
| opendevreview | Ghanshyam proposed openstack/project-config master: Move test-release-openstack job to Noble https://review.opendev.org/c/openstack/project-config/+/937053 | 18:40 |
| clarkb | https://review.opendev.org/c/openstack/ovn-bgp-agent/+/936968 got a new patchset but also ^ | 18:40 |
| clarkb | I'll check gmann's change since I already have that repo cloned | 18:40 |
| clarkb | `git fetch origin refs/changes/53/937053/1` worked for me where origin https://opendev.org/openstack/project-config (fetch) | 18:41 |
| clarkb | so I think this is all looking good. Anything else we want to check before moving onto the next thing? | 18:42 |
| clarkb | fungi: as I'm going back through https://etherpad.opendev.org/p/gerrit-upgrade-3.10 I notice there are a few steps that should happen at least an hour before the maintenance starts (items 5 and 6 in the list). Any chance you'd be willing to get those out of the way since it will be slightly less early in the morning for you then? | 18:46 |
| clarkb | #5 is a new one in reaction to feedback from the last upgrade | 18:48 |
| clarkb | people awnt to be reminded with ~enough time to quickly finish what they are doing in gerrit before it goes down and an hour is probably a reasonable amount of time for that | 18:48 |
| fungi | clarkb: sure, i'll set myself a reminder and take care of those bits, no problem | 18:51 |
| clarkb | thanks! | 18:53 |
| fungi | i'll do the emergency file additions a couple of hours ahead just to be on the safe side | 18:57 |
| clarkb | sounds good | 19:01 |
| clarkb | since gitea seems happy should we +A https://review.opendev.org/c/opendev/system-config/+/934075 as mentioned above? | 19:01 |
| clarkb | if we do +A that I'll incorporate it into the manual upgrade pass I'm about to do | 19:03 |
| clarkb | otherwise if we want to do it post upgrade it can wait | 19:03 |
| fungi | i've approved it | 19:05 |
| fungi | it's fairly innocuous, no point in putting it off to after | 19:06 |
| clarkb | thanks I'm updating the held test node now then will run through an upgrade and a downgrade | 19:21 |
| clarkb | ok I've upgraded a gerrit 3.9.8 with the log rotation config in place on the held node noted in the etherpad up to a 3.10.3 gerrit and then downgraded back to a 3.9.8 to test the entire up and down process | 19:42 |
| clarkb | as expected nothing really changed process wise. I did update the etherpad with new log outputs and versions and so on but the process itself was unmodified and the outputs don't change other than versions and timestamps really | 19:43 |
| clarkb | at this point I think I've gone over most stuff short of really exercising the gerrit versions but our ci jobs do that pretty well already | 19:43 |
| clarkb | (things like make a change post a comment etc are all covered in ci) | 19:44 |
| clarkb | if you have time to look over https://etherpad.opendev.org/p/gerrit-upgrade-3.10 for anything obviously wrong that might be the onl remaining task | 19:44 |
| clarkb | now to do the review on docker image mirroring role I promised | 19:45 |
| clarkb | I might have time to retset the openid change after lunch and propose that upstream if it still looks good after my edits | 19:50 |
| *** elodilles is now known as elodilles_pto | 19:54 | |
| opendevreview | Merged opendev/system-config master: Configure native Gerrit log rotation and cleanup https://review.opendev.org/c/opendev/system-config/+/934075 | 20:08 |
| clarkb | that ^ appears to haev deployed successfully. And now to go back to finishing lunch | 20:32 |
| clarkb | I have tested my lastest hacked up patch for openid on a held node it is seems to work for both an initial login with the registration pane and a normal login of an existing user. So now I just need to figure out how to run gerrit's java formatter check and I'll push that upstream | 21:20 |
| clarkb | you grab this jar https://github.com/google/google-java-format/releases/download/v1.24.0/google-java-format-1.24.0-all-deps.jar and run it against the files | 21:24 |
| clarkb | oh actually I just realized I need to also check the case where redirect back to the start point works | 21:32 |
| clarkb | I'll do that before pushing | 21:32 |
| clarkb | yup that still works phew | 21:34 |
| clarkb | https://gerrit-review.googlesource.com/c/gerrit/+/445041 | 21:37 |
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: Protect hostvars iterations from implicit localhost https://review.opendev.org/c/zuul/zuul-jobs/+/937071 | 21:59 |
| fungi | openeuler mirror pull is still in progress, currently at 50% quota for the volume | 22:39 |
| opendevreview | Merged openstack/project-config master: Move test-release-openstack job to Noble https://review.opendev.org/c/openstack/project-config/+/937053 | 22:56 |
| clarkb | coming in gerrit 3.11: "Submitting changes with unresolved comments is blocked for new sites by default" I'm glad they don't just switch that for everyone I think we'd have a revolt on our hands if they did | 22:57 |
| JayF | That's a bold change :\ | 23:00 |
| opendevreview | Merged zuul/zuul-jobs master: Protect hostvars iterations from implicit localhost https://review.opendev.org/c/zuul/zuul-jobs/+/937071 | 23:02 |
| mordred | yeah. E_DONOTWANT | 23:04 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!