Monday, 2022-05-16

*** ysandeep|out is now known as ysandeep|rover05:08
*** ysandeep|rover is now known as ysandeep|rover|lunch08:02
opendevreviewA R proposed openstack/diskimage-builder master: Preserve local mirrors when using Ubuntu element
*** ysandeep|rover|lunch is now known as ysandeep|rover10:01
*** rlandy|out is now known as rlandy10:15
*** dviroel|out is now known as dviroel11:21
*** pojadhav is now known as pojadhav|afk12:01
*** ysandeep|rover is now known as ysandeep|rover|afk12:45
*** pojadhav|afk is now known as pojadhav13:06
*** ysandeep|rover|afk is now known as ysandeep|rover13:11
*** ysandeep|rover is now known as ysandeep|out15:02
clarkbmy gcc package has updated again and I have a ton of rebuilt packages. I wonder what broke :)15:22
*** dviroel is now known as dviroel|lunch15:25
fungii'm still hoping to find time to dig into since i currently can't use openafs with linux 5.1715:29
clarkbthat reminds me I should go and update afs quotas today15:30
clarkbits the last piece of the cleanup of distro mirrors15:30
*** marios is now known as marios|out15:37
clarkbI'm going to help someone get set up to talk to gerrit and they are likely on OS X. Is the best way to get git and git-review via homebrew?16:12
odyssey4meclarkb I think git is included in dev tools, and git-review can just be installed into a venv via pip... so brew isn't required, but it does make it easier to maintain and get current versions.16:14
odyssey4mes/dev tools/xcode/16:15
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Make note of python_version being a string value
clarkbodyssey4me: good to know, thanks16:17
*** dviroel|lunch is now known as dviroel16:24
clarkb#status log Updated OpenAFS quotas for distro mirrors to better reflect current usage.16:36
opendevstatusclarkb: finished logging16:36
clarkbI aimed for round numbers that resulted in 70-80% utilization16:36
opendevreviewMerged zuul/zuul-jobs master: Switch py3.10 testing to Ubuntu Jammy
fungiclarkb: how much potential quota did that free up?16:59
fungior i guess it's not really clear, since you can oversubscribe quotas17:00
clarkbfungi: I'm not sure. Ya exactly17:00
clarkbwe are currently at 72% ish utilization of 4TB?17:00
fungiit didn't free up space, but it does help us avoid some volume running away with a lot more utilization than we intend17:00
opendevreviewMerged zuul/zuul-jobs master: Make note of python_version being a string value
*** rlandy is now known as rlandy|mtg18:14
clarkbfungi: exactly minimizes blast radius if anything goes crazy18:31
clarkbalso success on setting up gerrit pushing on OSX. We ended up running `xcode-select --install` to install the git and python toolchains then with that made a venv to install git-review18:32
clarkbI've learned a lot18:32
opendevreviewClark Boylan proposed opendev/system-config master: Upgrade gitea to v1.16.8
mnaserI suspect this is something that might have been brought up, but what would be a good way for an opendev tenant to maintain a “downstream” of an openstack project?19:28
mnaserExample being: we have some patches that are taking longer than expected to land upstream in openstack, so we want to build our images with those patches on top19:28
mnaserI haven’t been able to come up with a clean way of maintaining this since we can’t really (easily) have say: an upstream branch and a downstream branch in gerrit19:29
mnaserMaintaining a bunch of patch files can also be a bit unclean as well since it might be hard to go about maintaining a bunch of files in system and “unwrapping” them19:30
mnaserJust wondering if anyone has any ideas19:30
*** rlandy|mtg is now known as rlandy19:30
clarkbmnaser: the deb package repos simply hard forked and kept a copy of all the repos which isn't ideal from a disk consumption standpoint but worked19:33
clarkbthe biggest issue is likely goign to be CI config more than anything else. If you convince openstack to have extra branches that introduces CI weirdness. If you hard fork on a copy that also introduces CI weirdness19:34
clarkbIf however you just want to apply some patches to docker containers you can do what OpenDev does and manage them liek distro pacakges do and use patch to apply them19:34
clarkbwe've done this for gerrit and I think etherpad in the past19:35
clarkbas far as cleanliness goes I think that is largely solved by the distro packagers if you need more than a couple. They get numbered and tracked and applied in order19:35
clarkbinfra-root is a gitea upgrade to a new bugfix release. Mostly about keeping up more than fixing any particular issue if you get a chance19:36
clarkbI just trimmed the meeting agenda way down. Removed the PPA topic and the review cleanup topic. Now is the time to add anything new20:32
clarkbianw: when you start your day is probably good to look at. If you think that is worthy of a +A soon I should be around to babysit20:51
*** dviroel is now known as dviroel|out20:51
ianwo/ looking21:54
clarkblooks like it got approved, thanks21:59
ianwyep i'm happy to keep an eye on deployment22:03
ianwclarkb: thanks for the upgrade comments.  i'm going to validate the downgrade path and also have a look at keeping mariadb up-to-date as well22:05
clarkbianw: ++ for downgrade path. Re mariadb I don't think that is urgent. The current version is supported for some time iirc. It was just something I have in the back of my mind and have thought about and noticed the env var flag you can set to have it check for updating from an old version22:06
clarkbbut also the mariadb for review is pretty low impact if something goes wrong so oddly upgrading it first with review is maybe the easiest thing to do :)22:06
ianwyeah, that was my thinking; once we have a process for it, it's much simpler if we do need to move it for some more pressing reason22:07
ianw ... glean still writes ifcfg-* files.  although the plugin still exists i guess that still working is on borrowed time22:26
clarkbiirc the NM support in glean relies on those entirely22:27
clarkbit doesn't write native configs yet22:27
clarkbf36 almost dropped bios support too. Good thing they didn't :)22:27
ianw"The RPM database has been moved from its previous location in /var/lib/rpm to a new location in /usr/lib/sysimage/rpm"22:28
ianwi don't think that will matter ... but stranger things have happened22:28
opendevreviewMerged opendev/system-config master: Upgrade gitea to v1.16.8
clarkbgitea01 has updated and looks fine23:02
fungii concur23:02
*** rlandy is now known as rlandy|out23:05
ianwam i crazy or are we missing periodic runs of ?23:13
clarkblooks like they stopped on the 12th23:15
clarkb and are the two system-cnofig changes that merge in roughly the time frame to cause that to happen but I don't see how either could cause that23:17
clarkbinfra-prod-base has failed recently do we require infra-prod-base to succeed?23:18
clarkbya infra-prod-base is a soft dependency so I think that is it23:19
ianwthat must be it23:19
ianwgpg log download ftw :)23:20
ianwinterrupted dpkg on kdc04 ...23:21
clarkbthe gitea upgrades seem to be complete and I'm not seeing anything weird there23:21
ianwi'm fixing the kernel install but i don't know how it got into this state23:23
clarkbis the fix to autoremove the older installs that are confusing it?23:24
clarkbas far as getting into that state maybe an unexpected reboot during updates?23:24
ianw23:24:56 up 5 days23:25
ianwthat seems recent, i don't think we did reboot in an expected fashion23:25
ianwthere was some volume maintenance that fungi prepped for23:25
clarkbya though that was for the afs servers not kdcs iirc23:25
clarkband the prep work should've meant we bypassed it entirely by being on volumes that are unaffected23:26
ianwit's booted 5.4.0-109-generic but 110 is now installed23:26
ianwthis is redundant so i might reboot it just to be sure it's happy23:26
fungiyes, i didn't reboot anything for the volume swaps23:26
clarkbianw: maybe see what apt-get autoremoev has to say about cleanups?23:27
clarkbjust to limit the number of moving parts fi we can trim further?23:27
ianwyep ran that :) ->
ianw#status log manually fixed kernel install and rebooted kdc04.openstack.org23:31
opendevstatusianw: finished logging23:31
ianwthat should get the base job going again ... i'll look out for it now.  i noticed because i expected the tiny (r) that un-centres the openstack logo on the gitea org page to have gone, but manage-projects hasn't run to update it23:32
clarkbianw: earlier today I lowered distro mirror quotas so that most of them are using 70-80% of their quota while keeping round numbers. One thing I noticed is that all the wheel mirrors have very large quotas compared to consumption any concern with dropping those from 100 GB each to say 20GB? (some are almost 10GB due to time they've been around so double that seems reasonable)23:38
ianwclarkb: yeah, no problem with that.  i half think i might have meant to make them 10gb and added a zero, and we've just set the rest the same23:39
clarkbheh ok. I'll do that tomorrow morning then. The older ones which accumulate more pacakges have grown to almost 10GB so they aren't tiny if given time to accumulate packages but they also are nowhere near 100GB23:40

Generated by 2.17.3 by Marius Gedminas - find it at!