Friday, 2022-01-21

*** rlandy|ruck is now known as rlandy|out00:09
clarkb exists now but our mirrors don't seem to have it yet00:14
clarkbI suspect that iputils is pulled in by our image builds too (but it may not be) and if that is the case we likely need to also rebuild our centos 8 images once our mirrors update00:14
clarkbMaybe that means tomorrowe we'll be happy00:14
clarkber wait its systemd not iputils we need to update and that definitely gets pulled at image build time :)00:16
clarkb hasn't updated yet so we are waiting on that, then our mirror, then our images00:21
clarkb hasn't updated either which means switching mirrors is unlikely to help us right now00:22
clarkbtime to practice patience. Something I'm nto very good at00:22
opendevreviewSteve Baker proposed openstack/diskimage-builder master: Rename existing BLS entry with the new machine-id
clarkbre the reload4j chagnes to gerrit. It looks like they are pushing those changes to stable-3.3 and rolling up to 3.4 and 3.5 and master. We should be careful with our upgrade that we don't miss it if we go with the upstream version (they may update 3.3 and then not 3.4 for some additional time)00:43
fungialso we've seen them release a x.y.z with a commit that isn't merged to stable-x.y00:44
clarkbwell and more that they seem to batch up the merge forwards and I'm slightly worried we might miss that 3.3 is fixed upstream but not 3.4. For that reason we might want to just go with our local fixing anyway. But I'll look at where things are tomorrow morning and go from there00:48
*** ysandeep is now known as ysandeep|away01:22
fungiwhere's our backup pruning documentation?02:02
fungioh, never mind, i didn't read far enough02:02
fungiEach backup server has a script /usr/local/bin/prune-borg-backups which can be run to reclaim space. This should be run in a screen instance as it can take a considerable time. It will prompt when run; you can confirm the process with a noop run; confirming the prune will log the output to /opt/backups.02:03
fungiit's in process on in a root screen session02:06
fungii'll check up on it periodically02:06
opendevreviewSteve Baker proposed openstack/diskimage-builder master: Fix the root device in the default BLS entry
opendevreviewSteve Baker proposed openstack/diskimage-builder master: Rename existing BLS entry with the new machine-id
opendevreviewSteve Baker proposed openstack/diskimage-builder master: Fix the root device in the default BLS entry
*** tkajinam is now known as tkajinam|lunch03:55
*** tkajinam|lunch is now known as tkajinam03:56
opendevreviewSteve Baker proposed openstack/diskimage-builder master: Fix the root device in the default BLS entry
*** hashar_ is now known as hashar05:51
*** ysandeep|away is now known as ysandeep06:03
*** frenzy_friday is now known as frenzyfriday07:05
chkumar|roverianw: hello, please have a look when free, thanks!07:23
*** amoralej|off is now known as amoralej08:03
*** jpena|off is now known as jpena08:04
*** ysandeep is now known as ysandeep|lunch08:12
*** johnsom_ is now known as johnsom09:03
*** bhagyashris_ is now known as bhagyashris|ruck09:03
*** Tengu_ is now known as Tengu09:07
*** kopecmartin_ is now known as kopecmartin09:09
*** melwitt is now known as Guest131209:32
*** ysandeep is now known as ysandeep|coffee10:48
*** melwitt is now known as Guest132010:50
*** ysandeep|coffee is now known as ysandeep11:09
*** rlandy is now known as rlandy|ruck11:13
*** dviroel|out is now known as dviroel11:25
fzzf[m]Hello, I run "ssh -p 29418 gerrit stream-events", result shows "Received disconnect from port 29418:12: Too many concurrent connections (96) - max. allowed: 96",How to deal with this.11:27
fzzf[m]zuul CI how to limit listen events. I use it for manila project.11:44
rlandy|ruckfungi: hello ... to follow up on the mirror issues:
rlandy|ruckplease see last two comments from Alfredo Moralejo (amoralej)  and the link to issue
rlandy|ruckalso patch still open for w+
fricklerrlandy|ruck: I still fail to find any indication in all of this that the facebook mirror is better than the one we are currently using to sync from12:19
rlandy|ruckfrickler: there is no confirmation of that12:29
rlandy|ruckwe have had some better stats on c9 this week12:29
rlandy|ruckbut that is only one week to go by12:29
rlandy|ruckwhere we switch to the facebook mirror or not is unrelated to
rlandy|ruckwhich is what I said I would follow up on for fungi12:30
fricklerrlandy|ruck: maybe if redhat is to interested in this they can provide us direct access to the original repo to mirror from, that would avoid all issues with public mirrors that tend to come and go and be broken once in a while12:32
fricklerthe c9 mirror was syncing from rackspace originally, not leaseweb, so that's no indication for anything12:34
rlandy|ruckI'll look into that suggestion12:37
*** rcastillo|out is now known as rcastillo12:56
fungi#status log Pruned backups on
opendevstatusfungi: finished logging12:57
fungialso the problem we ran into with rackspace's mirror is that it started rejecting rsync connections out of the blue, it wasn't related to the content of their mirror13:00
fungithough more generally, i am curious why centos insists on looking for different indices than are present on mirrors from time to time. it seems very sensitive to mirror states older than whatever exists on disk. are we not using our mirrors when building our centos images?13:01
*** rcastillo is now known as rcastillo|rover13:02
*** amoralej is now known as amoralej|lunch13:11
*** jentoio_ is now known as jentoio14:02
*** amoralej|lunch is now known as amoralej14:09
*** anbanerj is now known as frenzyfriday14:41
*** ykarel_ is now known as ykarel14:51
*** dviroel is now known as dviroel|lunch14:58
clarkbfzzf[m]: you need to ensure you have fewer connections made to gerrit on port 29418. One thing to double check is that your clients are properly closing connections15:41
fungizuul prior to 2.5.2 had an issue with that, right?15:48
fungileading to
*** promethe- is now known as prometheanfire16:03
*** dviroel|lunch is now known as dviroel16:15
frickleroh, might that be an issue with recent paramiko versions again? 2.9.x broke connections to cirros, which might be a similar scenario. although then I'd expect connections to fail, not to pile up16:17
fungior it might just be a very, very old zuul which doesn't have that fix16:18
fricklerhmm, gerrit show-connections is very slow, because it tries to reverse resolve every remote IP. if I avoid that with -n, at the same time it also only shows user ids instead of names. not very helpful, thx.16:29
frickleranyway I'm not seeing more than a handful of parallel connections currently, so it must only be a spurious problem16:30
clarkbI think in the past we've seen firewall ungracefully kill connections that idle too. Then zuul reconnects but gerrit doesn't know it can close the old connection. But ya if it isn't a persistent problem may be more difficult to debug16:31
clarkbwhen I looked at our mirrors yesterday for centos 8 stream both the facebook and limestone upstream seemed to be roughly up to date at the same level.16:57
clarkbAre we sure that facebook will be any better for stream? I suspect some underlying problem with the mirror updates and not the mirrors we pull from16:57
*** jpena is now known as jpena|off17:04
fungifrickler: yeah, i end up using alternative means to reverse the numeric ids to account names (rest api is probably easiest)17:09
fungiclarkb: if you follow the copy of the irc discussion copied in pagure, it sounds like there's more than one backend the mirror sites might get balanced between, so if both aren't kept in sync then downstream mirrors can end up rewinding to an earlier state17:14
fungibut there was some assertion that facebook validates their mirror consistency to prevent that from occurring on their copy? that much was not entirely clear to me17:15
clarkbah I see17:15
clarkbbut also that seems problematic if you are a centos 8 stream user and you can pull from constantly reverting systems17:16
fungialso i haven't checked to see if the leaseweb site fell off the official centos mirrors list in the ~2 years since we switched to it17:16
clarkbthis isn't just an opendev CI problem is what I'm saying :)17:16
fungithere was some indication that mirror operators are supposed to have specific blackout windows for updates in order to avoid copying inconsistent state too17:17
fungibut it also sounded like there may have been an update to the primary content outside that window17:18
fungiwhich, if so, yeah it's not the mirror site's fault17:18
fungigmann just updated one of the threads on openstack-discuss to say there are more 404 errors for centos stream 8 mirrors17:20
fungigmann: do you have a recent example?17:20
gmannfungi: yeah, let me check17:20
fungithere was a ~6 hour window yesterday where we'd copied a "bad" state from a centos mirror and were serving that, but it should have cleared up around 13:00z17:21
gmannthis is last i have seen 17:22
clarkbI've just remembered I totalled spaced on doing the limnoria update yesterday. Oh well.17:31
fungiaccording to we synced in AppStream/x86_64/os/repodata/6cf23d2ab4fded7416b05bddd463d0fba08f77965b091c59a9e057654b2cef99-filelists.xml.gz from leaseweb's mirror and published it at 2022-01-19T06:58:12, then deleted it because it had disappeared from leaseweb and published that state at 2022-01-20T06:55:09, then back again17:34
fungi2022-01-20T12:57:29, refreshed it 2022-01-20T18:45:42 (maybe there was a timestamp change?), deleted 2022-01-21T00:45:0117:34
clarkbis there a change to swap to the facebook mirror? we can test its consistency verification by using it and observiing if it does ^ ya?17:35
fungiyet there's a job trying to retrieve it at 2022-01-21 04:01:1317:35
fungigmann's latest example is for a job which started 2022-01-21 06:14:1917:36
clarkbthey may have cached index content via the image builds?17:38
fungiyeah, that's why i was looking at the job start time, trying to correlate it to image updates17:39
gmannfungi: clarkb I can see latest run is passing (at 08:13) 17:39
fungii wonder if we're not using our mirrors when building the centos-8-stream images17:40
fungiand at times end up with nodes that think the mirrors have gone backwards in time17:40
fungithat build ran in rax-dfw which got a new centos-8-stream image moments ago, but before that a little over a day ago17:42
fungiso whatever caused things to start failing or clear up doesn't seem to have been related to an image update17:43
fungithe mirror-update log shows a fairly massive stream 8 update which we published at 2022-01-21T06:57:37 so that's probably when it cleared up17:45
fungiseems to correlate fairly closely with gmann's observations17:45
fungii've also confirmed is still on the list of official mirrors at which claims to cover centos linux and stream 817:53
*** amoralej is now known as amoralej|off18:09
*** ysandeep is now known as ysandeep|out18:20
clarkbfungi: should I go ahead and approve ? I've been distracted by other things so not completely sure where you've landed on that18:28
opendevreviewClark Boylan proposed opendev/system-config master: Rebuild gerrit images to pick up slf4j fix
clarkbinfra-root ^ if we land that today I can try to help restart gerrit later today though I do have some appointments today18:33
fungiclarkb: i don't think 825446 is urgent, but if you're in favor of it then feel free to approve18:40
clarkbI imagine the sooner we can rule in or out the flip flopping of the facebook mirror the sooner centos can make reliable updates to their mirroring system?18:42
clarkbmostly my concern is that facebook is just as broken and we're pretending it isn't. hard data would be useful18:42
clarkbI've approved it18:46
fungithanks, and yeah that's my thinking as well18:52
opendevreviewMerged opendev/system-config master: Use facebook mirror for CentOS Stream 8
*** Guest1320 is now known as melwitt19:18
noonedeadpunkfwiw irc bot seems stuck19:49
fungiopendevreview is in channel still, it commented in here at 19:12 (a little over half an hour ago)19:51
fungidid it fail to report something more recently than that?19:51
fungimaybe it lost contact with gerrit19:51
fungii'll take a look in its logs19:51
noonedeadpunkYeah, I did bunch of patch updates and like 1 out of 10 was reported19:54
noonedeadpunkcommented for us more then 2 hours ago last time19:55
noonedeadpunklike has been just merged minutes ago19:56
clarkbfungi: re 825873 I wonder if we should abandon and restore that so that we hopefully get enqueued to a different cloud for the image builds (currently airship cloud and that one has less capacity hence the queuing)20:00
fungithe gerritbot saw the change-merged event for that at 19:54:1420:00
clarkbya I think I'm going to go ahead and do that now since I don't think it will be any slower20:00
clarkbbut has a good chacne of being faster20:00
noonedeadpunk824567 has been jsut reported though20:01
fungiPotential channels to receive event notification: set()20:02
clarkbprobably not subscribed to events for that repo20:02
noonedeadpunkthat could be the case:)20:03
funginoonedeadpunk: i don't see it in the config?20:03
fungithat repo, i mean20:03
noonedeadpunkI guess I took reporting as granted:) will fix that20:03
noonedeadpunksorry for taking your time20:03
funginoonedeadpunk: no worries, normally you'd add it to gerritbot/channels.yaml in openstack/project-config at the same time you created the project20:04
noonedeadpunkyeah, I was relying that it was done :)20:04
fungibut it's easy to overlook even though we have it documented together20:04
opendevreviewDmitriy Rabotyagov proposed openstack/project-config master: Add ansible-role-qdrouterd IRC reporting
*** dviroel is now known as dviroel|out20:53
opendevreviewGhanshyam proposed openstack/project-config master: Add openstack-skyline irc channel in access an gerrit bot
opendevreviewGhanshyam proposed opendev/system-config master: Add openstack-skyline channel in statusbot/meetbot/logging
opendevreviewMerged openstack/project-config master: Add ansible-role-qdrouterd IRC reporting
opendevreviewEduardo Santos proposed openstack/diskimage-builder master: Fix openSUSE images and bump them to 15.3
opendevreviewEduardo Santos proposed openstack/diskimage-builder master: General improvements to the ubuntu-minimal docs
opendevreviewEduardo Santos proposed openstack/diskimage-builder master: Don't run functional tests on doc changes
opendevreviewMerged opendev/system-config master: Rebuild gerrit images to pick up slf4j fix
clarkbfungi: looks like promote for new gerrit images completed successfully. Should we go ahead and restart gerrit real soon now? I'm not sure if you are still around22:55
fungiyeah, i'm still around and happy to help with a restart22:56
clarkbcool. I'd like to get that done then I'll probably shift gears to zuul reviews22:56
clarkbdo you want to drive or should I? I'm happy either way22:56
fungii can22:57
clarkbcool if you start a screen I can join that22:57
fungii have one going and pulled in it22:58
clarkbI've joined22:58
fungiopendevorg/gerrit   3.3       8aaea7eace5d   56 minutes ago   790MB22:58
clarkblet me double check against docker hub22:58
clarkbthat one looks correct to me when I inspect it and compare the sha against what is tagged on docker hub22:59
fungiokay, i'm downing the container in that case23:01
fungi#status notice The Gerrit service on is being restarted briefly to apply a bugfix23:01
opendevstatusfungi: sending notice23:01
-opendevstatus- NOTICE: The Gerrit service on is being restarted briefly to apply a bugfix23:01
fungiand it's on its way back up now23:02
fungi[2022-01-21T23:02:13.999Z] [main] INFO : Gerrit Code Review 3.3.9-17-g3724a74167-dirty ready23:02
clarkbI've got a web ui now23:02
clarkbit shows me all the work I should be doing :)23:02
clarkbloading changes is slow but that is normal23:02
fungiit's friday. you can always get your work done on saturday! ;)23:03
clarkbI hopped off the screen and will let you decide when it is safe to close it. But from what I see so far it seems happy23:04
fungiyeah, lgtm, i'll wrap it up23:04
clarkband thank you!23:05
clarkbI need some nourishment then diving into zuul reviews23:05

Generated by 2.17.3 by Marius Gedminas - find it at!