Tuesday, 2024-02-06

opendevreviewJeremy Stanley proposed opendev/system-config master: Upgrade to Keycloak 23.0  https://review.opendev.org/c/opendev/system-config/+/90714101:52
opendevreviewJeremy Stanley proposed opendev/system-config master: Upgrade to Keycloak 23.0  https://review.opendev.org/c/opendev/system-config/+/90714103:37
opendevreviewJeremy Stanley proposed opendev/system-config master: Upgrade to Keycloak 23.0  https://review.opendev.org/c/opendev/system-config/+/90714105:33
ykarelfrickler, it still shows similar error but for different sig packages05:41
ykarelany idea how to get past to it? not sure if changing to some other mirror would help here05:41
ykarelmay be if you have access can try running the sync manually to understand what's actual issue05:42
ianwsigh, not sure there's much we can do if the remote end is hanging up on us ... 06:23
ianwthere's nothing in the mirror-update logs ~ 2024-02-06T00:08:08 that would suggest local instability06:26
ykarelianw, so can try other mirror?06:28
ianw... we can, i mean just look at the git log for plenty of examples :)06:28
ykarellike revert of https://review.opendev.org/c/opendev/system-config/+/86839206:29
ianwwhat would be ideal is an upstream mirror with someone we can actually contact when it fails.  it feels like we are basically an excellent free monitoring system for someone06:29
ianwwe flip between the rax mirror, some facebook one, some university one ... 06:30
opendevreviewyatin proposed opendev/system-config master: Revert "Revert "Revert "Use rackspace mirror to sync centos stream repos"""  https://review.opendev.org/c/opendev/system-config/+/90792306:30
*** ykarel_ is now known as ykarel06:46
opendevreviewTakashi Kajinami proposed openstack/project-config master: Retire heat-cfnclient: End Project Gating  https://review.opendev.org/c/openstack/project-config/+/90795107:03
opendevreviewTakashi Kajinami proposed openstack/project-config master: Retire puppet-qdr: Remove Project from Infrastructure System  https://review.opendev.org/c/openstack/project-config/+/90795407:09
opendevreviewTakashi Kajinami proposed openstack/project-config master: Retire puppet-qdr: End Project Gating  https://review.opendev.org/c/openstack/project-config/+/90795107:10
opendevreviewTakashi Kajinami proposed openstack/project-config master: Retire puppet-qdr: Remove Project from Infrastructure System  https://review.opendev.org/c/openstack/project-config/+/90795407:10
ildikovHi All, I'm moderating the starlingx-discuss mailing list and got a question from a subscriber who has issues with his mails to the ML getting bounced. Has anyone experienced that before?08:21
opendevreviewDr. Jens Harbott proposed openstack/project-config master: Update reno update to check whether series exists  https://review.opendev.org/c/openstack/project-config/+/90796308:25
fricklerildikov: if you DM me the email address of the subscriber I can check the logs of the mail system08:27
fricklerildikov: just for clarity, are the bounces happening for their submission before they are accepted for the list? or on delivery to some of the subscribers?08:34
ykarelfrickler, hi can you check https://review.opendev.org/c/opendev/system-config/+/90792308:34
fricklerykarel: yes, I saw that, but I don't think that randomly flipping mirror sources is a sustainable task, will let other admins do that if they feel like it08:36
ykarelfrickler, yeap agree, but may be atleast can get it sync for now. it would also match the mirror used for 8-stream which is in sync08:39
*** liuxie is now known as liushy09:42
apevecykarel: we should get access to the primary CS rsync, we'll just need to provide public IP where openinfra is rsyncing from, it is not wide-open for capacity reasons10:27
apevecfrickler: is mirror-update running on a specific machine so you could get me whatismyip from there or does it run in multiple machines/clouds providers?10:28
tonybI'm pretty sure it runs from a single machine that changes infrequently (every couple of years)10:29
apevectonyb: cool, please get public IP so we can discuss this further on CS infra side!10:33
ykarelapevec, thx yes that should help10:33
tonybapevec: can do.10:35
tonybapevec: I'm 95% certain it'd be https://opendev.org/opendev/system-config/src/branch/master/inventory/base/hosts.yaml#L33310:36
fricklertonyb: apevec: I'll add the remaining 5%, just confirmed the addresses on the host itself10:36
apevecmirror-update02.opendev.org makes sense to run mirror-update :)10:37
apevecthanks frickler ++ tonyb ++ 10:37
fungiildikov: frickler: did the mailing list post bounce problem get sorted out? forwarding a copy of the non-delivery report the user received to one of us might also help in tracking down the cause13:48
fricklerI didn't get any further response, so nothing I could do from my side13:51
fungii'll try to follow up later14:14
opendevreviewAlfredo Moralejo proposed opendev/system-config master: Use centos hosted mirror to sync CentOS content  https://review.opendev.org/c/opendev/system-config/+/90803014:26
opendevreviewAlfredo Moralejo proposed opendev/system-config master: Use centos hosted mirror to sync CentOS content  https://review.opendev.org/c/opendev/system-config/+/90803014:29
amoralejapevec, tonyb ^14:32
fungiamoralej: already looking at it. are msync and rsync two different hosts?14:33
amoralejthere are separated hosts for stream-9 and previous releases14:33
fungigot it, and we use the latter for stream-9 i see14:33
fungii was going to test this out from the server to confirm connectivity14:33
amoralejyes, please14:33
fungiwill take me a few minutes14:33
amoralejactually, will it try to actually sync from some other ci server?14:34
amoralejin that case it will likely fail14:34
fungithose scripts aren't really tested, i don't think, which is part of why i'm going to manually test a patched copy from an interactive shell on the server to make sure14:34
amoralejgood, that'd be great to confirm connectivity and the directories layout14:35
Clark[m]Note we haven't done that in the past because the existing rules state only public mirrors can sync from those locations. I'm somewhat wary of being an exception to those rules14:36
fungiyeah, i figure it helps to confirm this works regardless of which direction the discussion ends up going14:39
amoralejWe explained the issues we've had in the past and the kind of private usage opendev are doing and they agreed on it. We may not be the first case they hit a similar case, let us know your concerns and we can discuss it14:41
amoralejhttps://pagure.io/centos-infra/issue/1354 is the ticket for centos for the record14:43
Clark[m]The main concern I have is that it will stop working in a few months when someone discovers we broke the rules and we will be right back where we started. It would be better to address the underlying issues14:43
Clark[m]Which means more curation of the second level mirrors which the centos project may not have time and resources for14:44
fungiamoralej: not working, i added inline comments with the error messages14:51
Clark[m]One problem with rsync is you can't easily browse to figure problems like this out. Many of the public mirrors serve http content too and make that less of an issue but msync doesn't seem to?14:53
fungithey do also print a big, bold warning (the gist of which we already know, but for the record): "This service is intended for the sole use of the CentOS worldwide mirror network to synchronize mirrors. Unless you are running or intending to run a listed public CentOS mirror use a mirror listed at https://centos.org/download/mirrors If you intend to populate a mirror for public use please read14:54
fungithe notes at https://wiki.centos.org/HowTos/CreatePublicMirrors If you do use this service then it is implied that you are providing a mirror for public use and giving us authority to publicise such mirror."14:54
amoralejlemme check14:55
fungiamoralej: also, ftr, the servers rsync ended up getting the responses back from were centosy8.centos.org and mref1.ue2.stream.centos.org according to the banners they printed14:56
fungino idea if that matters14:56
fungiwe could make the argument that our mirrors technically are for public use, just not safe for *direct* public use; we operate a public ci/cd system and the mirrors are operated in support of the worker nodes running within that context14:58
amoralejthat's how we explained the use of this mirror15:00
amoralejfungi, may it be using ipv6 ?15:02
amoralejyep, they found the ipv6 attempt in the log, will take a while to update it15:04
fungiit probably prefers ipv6 unless the remote host only publishes v4 records15:04
fungian alternative would be to see if rsync has a v4-only option or something in the interim15:05
fungi--ipv4, -4: prefer IPv415:06
fungii'll test that in the meantime15:06
fungilooks like it's working for now if i force it to go over ipv4, will see if it's able to complete15:08
amoralejalso, acl for ipv6 address should be in few next minutes15:09
funginon-stream rsync worked but was a no-op because it was already in sync15:12
fungistream rsync is running now15:12
fungii'll test again without -4 afterward to confirm the v6 address access is in place15:14
fungistream rsync does seem to be working, pulling some new/changed packages15:15
fungiso this could be a viable option for us, i expect, if reviewers reach a consensus. and in the meantime we'll have the mirrors updated at least15:15
fungithough whether or not we'll get the disconnects we observed earlier remains to be seen, of course15:16
amoralejok, sure, let's keep the discussion on the review15:17
amoralejthanks fungi! 15:17
fungiwill do. i plan to comment once testing has finished15:18
amoralejping me if you find any other issue15:20
opendevreviewJeremy Stanley proposed opendev/system-config master: DNM: Try future Keycloak 24.0  https://review.opendev.org/c/opendev/system-config/+/90725315:22
opendevreviewJeremy Stanley proposed opendev/system-config master: DNM: Fail keycloak testing for an autohold  https://review.opendev.org/c/opendev/system-config/+/90660015:23
fricklerhmm, is there actually a way to redact comments in gerrit? other than doing weird database stuff?16:08
Clark[m]Yes I think there is an API for it. Sounded like corvus was going to test it on a local test gerrit 16:12
Clark[m]The API can delete a comment and replace it's message. So not fully an edit but close enough16:13
fungiamorin: ovh sent us a notification that they couldn't "process our payment" which probably means our time-based credits have run out. anyone recall who we reach out to in order to ask them to refresh that?16:27
frickleroh, that time of year again. I think there was someone else at ovh, let me grep logs16:31
fricklerno, amorin fixed it last on 2023-01-0416:34
fungioh, thanks for confirming! i'll give him a while to respond, and try reaching out by e-mail soon also if necessary16:36
tkajinamHi. May I ask someone to add me to https://review.opendev.org/admin/groups/d4fc65b8e79a0c041e0803bb30a98a3e2664decb,members ?16:43
tkajinamcontext: https://review.opendev.org/c/openstack/project-config/+/90597616:43
clarkbfrickler: tonyb: if you get a chance reviewing https://review.opendev.org/c/opendev/system-config/+/907472 to update gitea to the latest bugfix release would be good16:48
opendevreviewJay Faulkner proposed openstack/diskimage-builder master: Re-enable voting on Gentoo DIB CI job  https://review.opendev.org/c/openstack/diskimage-builder/+/90790416:51
fungitkajinam: done!16:54
tkajinamfungi, thanks !16:55
fungiany time!17:01
opendevreviewMerged opendev/engagement master: Record raw responses  https://review.opendev.org/c/opendev/engagement/+/90429817:35
opendevreviewJames E. Blair proposed opendev/system-config master: Add a tool to delete (redact) gerrit comments  https://review.opendev.org/c/opendev/system-config/+/90818117:46
opendevreviewJames E. Blair proposed opendev/system-config master: Add a tool to delete (redact) gerrit comments  https://review.opendev.org/c/opendev/system-config/+/90818118:01
opendevreviewJames E. Blair proposed opendev/system-config master: Remove obsolete delete-gerrit-spam.py script  https://review.opendev.org/c/opendev/system-config/+/90818418:01
opendevreviewMerged opendev/system-config master: Add a tool to delete (redact) gerrit comments  https://review.opendev.org/c/opendev/system-config/+/90818118:15
opendevreviewJames E. Blair proposed opendev/system-config master: Fix gerrit-delete-comment script  https://review.opendev.org/c/opendev/system-config/+/90818618:29
opendevreviewMerged opendev/system-config master: Remove obsolete delete-gerrit-spam.py script  https://review.opendev.org/c/opendev/system-config/+/90818418:33
opendevreviewMerged opendev/system-config master: Fix gerrit-delete-comment script  https://review.opendev.org/c/opendev/system-config/+/90818618:50
ildikovfungi: frickler: apologies, I didn't get the notifications of your pings and got distracted. I just checked and the person's emails didn't get through to the ML, so they bounced before getting posted.19:59
clarkbildikov: and they didn't end up in moderation?19:59
fungiildikov: if they can get you (or one of us) a copy of the bounce message, that may help track down the cause19:59
fungiit's possible delivery was rejected by some intermediate mta before reaching us20:00
fungimaybe an office mail gateway or something20:00
clarkbfungi: thoughts on sending in the gitea upgrade?21:05
clarkbI have a school run in an hour or so but otherwise expect to be around21:05
clarkbhapyp to wait for others to review it first though21:05
fungii can approve it21:06
fungigoing to be cooking dinner soon, but nothing too distracting21:06
clarkbfungi: looks liek your figured out the env var for keycloak was KC_DB_PASSWORD?21:06
clarkband the new test case should confirm that is working so thats all good21:07
fungii wasn't sure originally because they have separate build-time and run-time envvars for some stuff, but if you don't use the --optimized switch with kc.sh start then it does a build before startintg21:07
fungiso the build-time envvars end up baked into the auto-built runtime, basically21:08
fungiit's so funky, and it does slow down container restarts, but it keeps us from having yet another container we need to build ourselves i guess21:08
clarkbthat is weird21:11
fungiyeah, so in summary, there seems to be no runtime envvar to set the database password, but there is a buildtime envvar to bake it into the image, and by not using --optimize so that it rebuilds at every container start, the buildtime envvar effectively acts as a runtime envvar21:31
fungithis is also why i increased the start wait for the service in ansible from one minute to five, the keycloak autobuild at start can take some time to complete, and while it usually finished in under a minute in testing, there were some timeouts i could only attribute to the ansible task giving up too early21:43
clarkbits weird that a startup that has to optimize things is faster than a startup that doesnt21:50
clarkbIwonder what otpimize actually does here21:50
opendevreviewTristan Cacqueray proposed zuul/zuul-jobs master: Remove the periodic-weekly pipeline  https://review.opendev.org/c/zuul/zuul-jobs/+/82736921:53
opendevreviewMerged openstack/diskimage-builder master: Fix various minor issues with Gentoo; make CI pass  https://review.opendev.org/c/openstack/diskimage-builder/+/90775721:55
clarkbfungi: looks like the gitea change will merge right around midway through my school run22:02
clarkbfungi: I can help test the deployment when I get back or you can -W it I suppose if the timing doesn't work22:03
fungii'm done with dinner and keeping an eye on it22:04
fungilooks like the last non-paused job just finished uploading logs now, so the registry job is unpausing and wrapping up22:38
opendevreviewMerged opendev/system-config master: Upgrade gitea to 1.21.5  https://review.opendev.org/c/opendev/system-config/+/90747222:38
fungiand there it is22:38
fungideploy is free of any delays at the moment too22:38
fungiand infra-prod-service-gitea deployment is running now22:39
fungihttps://gitea09.opendev.org:3000/ is back up and looking good22:42
fungi"Powered by Gitea22:42
fungiVersion: v1.21.5"22:42
fungiand i can clone a repo from it22:43
fungigitea10 is done22:43
fungiseems similarly fine22:43
fungiit's bringing up gitea14 now22:48
fungiand deploy job is done. going through the load balancer to test, everything seems consistent still. should be all set!22:49
fungibasically 10 minutes from merge to deploy. that's pretty amazing22:50
fungiespecially since the server upgrades are all serialized22:51
clarkbyup looks good from here23:09
opendevreviewClark Boylan proposed openstack/diskimage-builder master: Write fedora download redirect info to stderr  https://review.opendev.org/c/openstack/diskimage-builder/+/90820723:28

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!