Wednesday, 2023-03-01

clarkbcool that is what I expected00:00
opendevreviewIan Wienand proposed opendev/system-config master: make-tarball: add some extraction instructions
ianwone interesting this is in the 1-abc... 2-def 3-... output of the ssss key split, the X- part is important00:03
ianwwe want to make sure we store the whole thing00:03
ianw(i'm glad i actually spent the time validating that so i realised that before it was too late :)00:07
ianwok, thoughts on gerrit @
Clark[m]ianw left a note about the ssh command thing might affect zuul01:04
Clark[m]Otherwise lgtm01:04
*** promethe- is now known as prometheanfire02:47
opendevreviewIan Wienand proposed openstack/project-config master: gerrit/acl : remove deprecated NoOp function
opendevreviewJake Yip proposed opendev/irc-meetings master: Update Magnum meeting chair
opendevreviewMerged opendev/system-config master: install-docker: remove apt-key cleanup
*** ralonsoh_ is now known as ralonsoh07:36
*** jpena|off is now known as jpena08:45
*** ralonsoh__ is now known as ralonsoh10:18
opendevreviewEspen Flage-Larsen proposed openstack/diskimage-builder master: Updated serial configuration of the bootloader element
opendevreviewEspen Flage-Larsen proposed openstack/diskimage-builder master: Updated serial configuration of bootloader
opendevreviewEspen Flage-Larsen proposed openstack/diskimage-builder master: Updated serial configuration of bootloader
opendevreviewEspen Flage-Larsen proposed openstack/diskimage-builder master: Updated serial configuration of bootloader
opendevreviewMerged opendev/irc-meetings master: Update Magnum meeting chair
clarkbinfra-root I've got two hours of meetings that just started now. Towards the end of that I'll approve to update our nine gitea to 1.18.516:06
clarkbfungi: it looks like someone on gerrit discord has said you can still lose replication events even with the disk backing16:07
clarkbfungi: what happens when you reload the config is you stop the plugin and then start it again and there is a small window during that process where you can miss new refs showing up in gerrit that need replication16:07
clarkbI think that this risk is still low particularly since we understand that it can occur. And that autoreloading the config will greatly simply the process of replacing gitea backends compared to restarting gerrit everytime we need to add or remove a backend16:08
clarkbwe just need to periodically run full syncs after the config updates16:08
clarkband then when we've replaced all the giteas we can remove autoreload config again to avoid the footgun16:09
fungiclarkb: yeah, saw that (i also follow the gerrit channel via matrix). seems like a small window compared to the old behavior where it dropped all queued replication tasks on reload16:45
clarkbright it will preserve what it already knew about via the disk location which is a big improvement16:45
clarkbbut still a small risk is present16:45
fungihow long does a reload of that plugin take?16:45
clarkbI don't know hard numbers ut if you look at the error log on startup I expect it is subsecond16:46
clarkbit should be a very small window.16:46
clarkbIf that small window and active mitigation after auto reloads is ok with other infra-root my next task for gitea replacements is likely to land the gerrit config update and the bind bound update16:46
clarkbBut I want to get gitea upgraded first since we're in a good spot to pause for that16:47
fungiyeah, i'm okay with it. odds of short-term impact are very low16:48
clarkbone concern with the bind mount change (that I've already checked but will probably triple check) is we have a data dir on review02 host side already. But it is empty so bind mounting it should be fine16:50
clarkbsince we could possibly bind mount in ancient data and cause plugins to take bad actions. But its empty so that shouldn't happen16:51
*** jpena is now known as jpena|off17:35
clarkbI've approved the gitea 1.18.5 change18:08
clarkbThats about an hour away I think. I'll keep an eye on it as it deploys18:19
fungii need to pop out and run some errands now that meetings have wrapped up, but should be back before then18:22
fungiokay, i'm back19:24
clarkbfungi: and its just about to merge.19:24
clarkbI took the opportunity to do expense reports and check on my passport renewal.19:25
fungii still need to do expense reports, probably saving that for the weekend19:26
opendevreviewMerged opendev/system-config master: Update gitea to 1.18.5
clarkbthe deployment job has started19:36
clarkbI'm on gitea01 waiting for its services to get restarted19:38
clarkbok it has restarted. experiencing the slow repo load at the beginning of startup. I expect that to last a minute or so before things are good19:39
clarkbyup first load of system-config was 67 seconds. Second was 95ms19:41
clarkband git clone works too19:41
clarkbthis looks good. Now we have to wait for the other 8 giteas19:41
clarkbone thing I like about the url warning in newer gitea is it makes it easy to ensure that I actually refreshed the page off of the gitea I expected to19:47
clarkbit shows the url in the page content which confirms the X value in gitea0X19:47
clarkb2/3 are done19:48
clarkbnow all of the original servers are done. Last one is the new one19:52
clarkbit looks happy too. I think this went well19:53
fungiyeah, all lgtm20:23
ianwanother successful upgrade :)20:36
opendevreviewAmy Marrich proposed opendev/irc-meetings master: Correction to the Diversity meeting
opendevreviewMerged opendev/irc-meetings master: Correction to the Diversity meeting
clarkbI'm going to try and work on some zuul reviews this afternoon since this is a good pause point before picking up the gerrit/gitea work again21:03
clarkbfungi: re gerrit/gitea, did you want to review the replication autoload and data dir bindmount changes?21:07
fungii'm about freed up and can look in a sec21:12
fungiclarkb: approved those and the jdk change too21:18
clarkbthank you. I guess once all of those are rolled out we'll want to down then up gerrit to pick them up21:20
clarkband ya triple checking /home/gerrit2/review_site/data exists and is empty which should make it safe to bind mount as is21:21
opendevreviewMerged opendev/system-config master: Enable Gerrit replication autoreload
clarkbfungi: your virtualpdu email got results.22:36
clarkbI'm logging in to moderate the email through22:38
clarkbI discarded the one to -announce and accepted the one to -discuss22:39
opendevreviewMerged opendev/system-config master: Bind mount Gerrit's review_site/data dir
opendevreviewMerged opendev/system-config master: Switch gerrit container from jre to jdk packages
clarkbafter ^ all three of those deploy I think we want to double check the docker compose file has the new bind mount and that the dir is still looking good in review_site. Then we should be ready to do a gerrit restart at any time22:41
clarkbnote it needs to be a down up I believe to pick up the changes to the mounts and stuff22:41
fungiclarkb: thanks! and yeah, i set reply-to and mail-followup-to service-discuss on that one22:50
clarkbfungi: related, we end up with a bunch of rxwholesale spam. Are we able to just reject all email from them? Or are we concerne they might one day have a legit reason to email us?23:26
clarkbI know they just want to get a piece of the fungi hair, but I suspect we're happy as is :P23:27
fungimm2 had a checkbox in the moderation view to automatically add a sender to a "drop from" list, but mm3 seems to not have that implemented. i think we can manually add that address in the list config though23:28
clarkbok the bind mount change appears to have done what I expected in docker-compose.yaml and the replication autoreload chnage too. Both will need gerrit to be down'd them up'd to take effect which will also pull in the jdk packgae change23:33
clarkbthe gate is pretty busy right now so I'm inclined to wait on that for a bit23:33
clarkb(release time for openstack so being careful to not create big impacts is a good thing)23:33

Generated by 2.17.3 by Marius Gedminas - find it at!