Tuesday, 2022-09-20

opendevreviewIan Wienand proposed opendev/system-config master: install-ansible: remove stub install for ARA  https://review.opendev.org/c/opendev/system-config/+/85247800:07
opendevreviewIan Wienand proposed opendev/system-config master: bootstrap-bridge: drop pip3 role, add venv  https://review.opendev.org/c/opendev/system-config/+/85659300:07
opendevreviewIan Wienand proposed opendev/system-config master: run-selenium: Use latest tag on firefox image  https://review.opendev.org/c/opendev/system-config/+/85780300:07
opendevreviewIan Wienand proposed opendev/system-config master: afs-release: better info when can not get lockfile  https://review.opendev.org/c/opendev/system-config/+/85800900:07
opendevreviewIan Wienand proposed opendev/system-config master: Run jobs with a jammy bridge.openstack.org  https://review.opendev.org/c/opendev/system-config/+/85779900:07
opendevreviewIan Wienand proposed opendev/system-config master: testinfra: Update selenium calls  https://review.opendev.org/c/opendev/system-config/+/85800300:07
ianwjeez we have hardcoded "bridge.openstack.org" in a lot of places00:37
opendevreviewIan Wienand proposed opendev/system-config master: Abstract name of bastion host for testing path  https://review.opendev.org/c/opendev/system-config/+/85847602:00
diablo_rojoSo... it seems the meetbot hasn't appeared in the envirosig channel despite the patch being merged? 02:37
fungidiablo_rojo: i think we need to manually restart its container, it doesn't restart automatically so that we don't accidentally interrupt any running meetings03:07
fungii'll do it now03:07
diablo_rojooh thats a good safety mechanism03:15
diablo_rojothank you fung!03:16
fungiany time03:25
opendevreviewIan Wienand proposed opendev/system-config master: install-ansible: remove testinfra version install workaround  https://review.opendev.org/c/opendev/system-config/+/85247503:29
opendevreviewIan Wienand proposed opendev/system-config master: testinfra: install with ansible extras  https://review.opendev.org/c/opendev/system-config/+/85247603:29
opendevreviewIan Wienand proposed opendev/system-config master: install-ansible: remove stevedore workaround  https://review.opendev.org/c/opendev/system-config/+/85247703:29
opendevreviewIan Wienand proposed opendev/system-config master: install-ansible: remove stub install for ARA  https://review.opendev.org/c/opendev/system-config/+/85247803:29
opendevreviewIan Wienand proposed opendev/system-config master: bootstrap-bridge: drop pip3 role, add venv  https://review.opendev.org/c/opendev/system-config/+/85659303:29
opendevreviewIan Wienand proposed opendev/system-config master: run-selenium: Use latest tag on firefox image  https://review.opendev.org/c/opendev/system-config/+/85780303:29
opendevreviewIan Wienand proposed opendev/system-config master: afs-release: better info when can not get lockfile  https://review.opendev.org/c/opendev/system-config/+/85800903:29
opendevreviewIan Wienand proposed opendev/system-config master: Run jobs with a jammy bridge.openstack.org  https://review.opendev.org/c/opendev/system-config/+/85779903:30
opendevreviewIan Wienand proposed opendev/system-config master: testinfra: Update selenium calls  https://review.opendev.org/c/opendev/system-config/+/85800303:30
opendevreviewIan Wienand proposed opendev/system-config master: Abstract name of bastion host for testing path  https://review.opendev.org/c/opendev/system-config/+/85847603:30
tkajinamhello. it seems jobs using nodes with nested-virt-centos-9-stream labal is currently failing because of node failure. do we need to make any change in job definition or is that some issue within infra ?04:14
tkajinamexample: https://zuul.opendev.org/t/openstack/build/e3cab8e04c4a4d668d5029b40bf307b904:14
tkajinamthe current nodeset definition can be found here: https://github.com/openstack/puppet-octavia/blob/master/.zuul.yaml#L15-L2504:15
opendevreviewIan Wienand proposed opendev/system-config master: Abstract name of bastion host for testing path  https://review.opendev.org/c/opendev/system-config/+/85847604:23
*** ysandeep|out is now known as ysandeep05:04
*** pojadhav is now known as pojadhav|PTO05:09
opendevreviewIan Wienand proposed opendev/system-config master: Abstract name of bastion host for testing path  https://review.opendev.org/c/opendev/system-config/+/85847605:38
opendevreviewIan Wienand proposed opendev/system-config master: Convert production playbooks to bastion host group  https://review.opendev.org/c/opendev/system-config/+/85848605:38
opendevreviewlotorev vitaly proposed opendev/project-config master: Update link to zuul gating docs  https://review.opendev.org/c/opendev/project-config/+/85845405:42
vlotorev[m]<fungi> "vlotorev: looks like 858454 is..." <- fungi: I forget to publish edited file via web. Done now.05:46
vlotorev[m]I cdn modify pipeline description in other tenants if 858454 gets positive review.05:46
vlotorev[m]* fungi:  I forget to publish edited file via web. Done now.05:46
vlotorev[m]I can modify pipeline description in other tenants if 858454 gets positive review.05:46
*** ysandeep is now known as ysandeep|lunch07:24
*** jpena|off is now known as jpena07:30
*** ysandeep|lunch is now known as ysandeep08:04
*** ysandeep is now known as ysandeep|afk09:00
*** ysandeep|afk is now known as ysandeep09:36
*** ysandeep is now known as ysandeep|afk10:26
*** rlandy|out is now known as rlandy|rover10:34
*** ysandeep|afk is now known as ysandeep10:58
rlandy|roverhello ... we have a NODE_FAILURE showing up on the showing up tripleo-ci-centos-9-scenario010-standalone11:23
rlandy|roversince 09/1711:23
rlandy|roverthere are no logs so had to tell what is happening11:24
rlandy|roverthis job uses nodeset: single-centos-9-node-nested-virt11:24
akahat|ruckwe are testing to see if 8 is also problem.11:24
rlandy|roverso assumption is that it's the nest-virt nodeset11:24
rlandy|rovercan you see anything in the logs? if not, would it be possible to hold a node and investigate11:25
rlandy|roverhttps://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-scenario010-standalone+&skip=0 - looked ok until 15th11:28
rlandy|roverakahat|ruck will check it now11:28
opendevreviewMerged opendev/project-config master: Update link to zuul gating docs  https://review.opendev.org/c/opendev/project-config/+/85845411:41
fungitkajinam: rlandy|rover: akahat|ruck: still waking up, but i was looking into a similar issue with fedora-36 nodes yesterday as well. seems like both problems may have started around the same time11:43
fungiare you seeing similar problems with non-nestedvirt centos-9-stream labels?11:43
rlandy|roverfungi: hi - good morning  ... and no those are running fine11:44
rlandy|roverie: non-nested run everywhere - so those seems ok 11:44
rlandy|roveridk if c8 ones are impacted yet11:44
rlandy|roverthe last run on c8 was on 09/1511:45
rlandy|roverso before the start of trouble11:45
rlandy|roverjust created https://bugs.launchpad.net/tripleo/+bug/1990269 to start tracking and report gate blocker to openstack-discuss11:47
rlandy|roverakahat|ruck: ^^ fyi11:47
akahat|ruckrlandy|rover, ack11:47
fungijust checking up on the fedora-36 situation first, it's still what i was seeing yesterday. pulling the instance console log from nova it's booted up to a login prompt on tty1 and the boot output says networkmanager brought up networking, but nodepool is never able to reach the sshd on it11:49
funginow to see if the nested virt stream 9 is at all similar11:49
rlandy|roverfungi: thanks11:50
fungiinterestingly, nl04 which provides the bulk of those nodes (handles ovh) has no debug logs since 01:22 utc on the 17th12:00
Clark[m]fungi: nl04 is the only launcher providing those nodes. Sjc1 and limestone have max servers 0.12:01
fungiand the launcher daemon isn't running on it12:01
*** rlandy|rover is now known as rlandy|rover|mtg12:01
fungidocker-compose logs is full of tracebacks12:01
fungiKeyError: 'rockylinux-9'12:02
fungilet's see what config change rolled out around that time12:02
*** rlandy|rover|mtg is now known as rlandy|rover12:03
Clark[m]Or changes to nodepool12:04
funginothing merged over the weekend, so must be cropping up from a restart yes12:04
*** amorin_ is now known as amorin12:07
fungihttps://paste.opendev.org/show/816812 is the cleaned up traceback from the docker-compose logs output12:07
fungithe /etc/nodepool/nodepool.yaml file on the server last updated 2022-09-01, so long before the container failed to start12:09
Clark[m]Need to add that image to https://opendev.org/openstack/project-config/src/branch/master/nodepool/nl04.opendev.org.yaml#L19712:10
Clark[m]The other launcher file diskimage lists appear more complete. I wonder why nl04 is out of sync12:11
fungioh indeed. thanks for spotting that12:11
fungipatch inbound12:11
Clark[m]fungi: I think rockylinux-8 is missing too. May need to cross check all the images listed in the pools12:13
*** tosky is now known as Guest101212:14
*** tosky__ is now known as tosky12:14
fungithey're all out of sync12:14
funginl01 has one more entry than nl02 for that matter12:14
fungiyeah, openeuler only got added to nl0112:16
*** tosky is now known as Guest101312:30
*** tosky__ is now known as tosky12:30
opendevreviewJeremy Stanley proposed openstack/project-config master: Add missing rockylinux and openeuler entries  https://review.opendev.org/c/openstack/project-config/+/85852312:35
fungiinfra-root: that's ^ fairly urgent since we've basically not been using ovh or vexxhost for nodes so far this week (which makes nested virt flavors completely unavailable too)12:36
fungii'll get the config on nl04 patched locally by hand so i can start the service in the meantime12:36
rlandy|roverfungi: thanks for the quick action on this :)12:40
fungi#status log Manually patched configuration on nl04 in order to get the nodepool-launcher service running again while we wait for change 858523 to be reviewed12:41
opendevstatusfungi: finished logging12:41
fungiyou're welcome12:41
Clark[m]fungi: that change lgtm but I'm not logged into Gerrit on this device12:41
Clark[m]Note the hourly infra-prod-service- ode pool will likely undo the hand fix12:42
fungiturns out i +2'd https://review.opendev.org/854431 which introduced this error, so i feel partly responsible for it being broken12:42
fungiClark[m]: undo yes, but not restart the container probably?12:43
fungiseems like it broke on saturday because we added nodepool to the zuul restart playbook12:43
fungi(the configuration had been broken on that server since the first of the month)12:44
fungiso while ansible will probably undo the temporary addition, we'll hopefully deploy the fix before the next launcher restart12:44
Clark[m]Nodepool rereads it's config on each pass through the run loop. No restart required12:45
fungiyeah, but seems it only crashed when it was restarted?12:45
Clark[m]I suspect something changed in nodepool to make this more fatal than before. And that did require a restart (which we do automatically when nodepool has new images)12:46
fungioh, perhaps it will crash again due to newer nodepool. i'll keep an eye out for that12:46
Clark[m]Now that the new code is running I suspect a revert of the on disk config will cause it to crash again12:46
fungii can test that easily now12:46
fricklerI thought I had done a check for the config error in https://review.opendev.org/c/zuul/nodepool/+/852264 , or was that a different scenario?12:47
fungifrickler: this was the launcher crashing, not the builder. but similar i think, yes12:48
fungiand it was present in the provider pools labels list not the provider diskimages list (but still missing from the top level diskimages list)12:56
fungii think the pools is what the launcher uses, it doesn't seem to care about the provider diskimages list12:57
fungiClark[m]: so it looks like undoing my edit does cause the launcher to start complaining that it's unable to reload the new config, but it doesn't terminate13:05
fungiif i restart the container i expect the service will fail to come up13:05
Clark[m]Ah ok13:06
fungibut for now it's just stuck using the working config because it's unable to parse the broken one13:06
Clark[m]Re the validator update we could add similar checks to that for the launchers13:06
*** dasm|off is now known as dasm13:30
*** ysandeep is now known as ysandeep|dinner14:12
*** soniya is now known as soniya|dinner14:14
opendevreviewMerged openstack/project-config master: Add missing rockylinux and openeuler entries  https://review.opendev.org/c/openstack/project-config/+/85852314:20
clarkbfungi: it looks like ovh may actually be able to occasionally boot fedora 36 (I decided to take a quick look in case the fedora issues were related to ovh being down)14:52
clarkbwhich makes this problem more subtle and probably more difficult to debug. Looks like 2/3 of the ovh gra boots for the one request I checked failed then the third succeeded14:53
fungiand explains why the node_failure got worse when nl04 was out of commission15:05
clarkbnetwork manager is failing to start checkout the console log for clarkb-test-36 in iweb mtl0115:18
clarkbthe logs for that are in journald though according to the console log. Not sure how we're going to get those out. I guess chroot on a rescue instance15:19
fungiodd, the console logs i looked at said that nm started and systemd switched to network-ready or whatever15:19
clarkbyes, it does that eventually after marking it failed15:19
clarkbwhy it marks network online when NM fails I do not know15:19
clarkbhttps://src.fedoraproject.org/rpms/NetworkManager/c/22b2a6c081e8f3c3d30d459703de185857e3fca1?branch=f36 perhaps but that doesn't give a detailed changelog. Maybe nm does15:21
clarkbhttps://github.com/NetworkManager/NetworkManager/blob/1.38.4/NEWS nothing super noteworthy there15:22
clarkbI can't do a rescue in that cloud15:24
fungiclarkb: this is what i saw on one i caught failing in that same region: https://paste.opendev.org/show/81682215:24
fungi(pulled from my terminal buffer)15:25
clarkbyup line 16815:25
clarkbshows it failed15:25
clarkbI think its a best effort ot bring up networking before it gives up and then proceeds with booting15:25
fungiyeah, i completely overlooked that line15:25
fungimy eye was drawn to "Network is Online."15:26
fungiso anyway, yes your reproduction seems to match15:26
clarkbits possible this never worked and we didn't really notice until ovh's launcher stopped. OVH is a dhcp cloud so any sane fallback there with NM would end up dhcping and be happy15:27
fungiclarkb: i saw a similar failure in rackspace, so presumably we could rescue a node there15:27
clarkbthat said I did boot a node in ovh and nm seemed to be happy with the glean config. Not likely that it is falling back there and instead maybe something directly wrong with static network config15:28
clarkboh good point15:28
clarkbok I'll clean this one up and try in rax15:28
*** ysandeep|dinner is now known as ysandeep15:35
clarkbit looks like NM is attempting DHCP which of course returns no responses on the non DHCP clouds15:45
fungithat would seem to match the places we see it fail15:46
clarkb/etc/sysconfig/network-scripts/ifcfg-enX0 has BOOTPROTO=static and IPADDR set15:47
clarkbhttps://docs.fedoraproject.org/en-US/fedora/latest/release-notes/sysadmin/Networking/ I think that is our answer15:47
clarkbI can write a change to dib to add that package when glean is used15:48
clarkbit would be nice if that doc stated the package name though15:48
fungiso i guess 36 was never working outside dhcp clouds15:49
clarkbfungi: yes I think that is correct15:51
fungiindeed, that's not a package name i would have easily guess from the release note saying "include the ifcfg subpackage"15:53
fungiyou know, *the* ifcfg subpackage. NetworkManager-initscripts-ifcfg-rh, duh!15:53
opendevreviewClark Boylan proposed openstack/diskimage-builder master: Install Fedora ifcfg NM compat package  https://review.opendev.org/c/openstack/diskimage-builder/+/85854716:00
clarkbsomething like that maybe16:00
*** marios is now known as marios|out16:00
clarkbI'll leave the rescued node in rax dfw for now so that anyone else can chroot into /mnt and journalctl -u NetworkManager16:04
clarkbyou do have to ssh as the 'ubuntu' user but your keys should work16:05
*** ysandeep is now known as ysandeep|out16:15
clarkbif anyone has a pagure account for fedora it looks like you can edit https://pagure.io/fedora-docs/release-notes/blob/f36/f/modules/release-notes/pages/sysadmin/Networking.adoc to add the package name to that doc16:20
clarkboh wait its a third party system thta apparently write sthose files. THat explains the extreme line length16:21
opendevreviewMichal Nasiadka proposed openstack/project-config master: Add rockylinux-9-arm64  https://review.opendev.org/c/openstack/project-config/+/85855416:30
fungii'll make sure i triple-check that ^ one for missing parts :/16:33
fungiclarkb: for _the_ ifcfg subpackage, we could also temporarily stuff it into infra-package-needs if we're in a hurry (i'm not, but just pointing it out)16:35
*** jpena is now known as jpena|off16:37
mnasiadkafungi: I'm doing that for first time (adding arm64) if there's anything else that needs to be done first - give me a shout, happy to do everything is needed16:38
mnasiadkaWhile we're at it - I don't think there are Rocky Linux mirrors, right?16:38
fungimnasiadka: that one looks right. at least it doesn't miss the thing that nested-virt-rockylinux-9 did16:39
fungimnasiadka: no, we don't normally mirror every distro, we've been trying to keep it to the ones which are more heavily used, though if we see rockylinux approach centos levels of utilization in our jobs we should probably consider it16:39
mnasiadkafungi: oops, sorry for missing what I missed (wasn't aware)16:40
fungimnasiadka: no need to apologize, i should have caught it when i reviewed the change originally16:40
mnasiadkafungi: Kolla projects group is mainly going to utilise Rocky Linux 9 in CI after Zed release, so probably the usage will raise in couple of months16:41
fungimnasiadka: yeah, i have a feeling antelope will have a lot of demographic shift in jobs, and we should probably look at it again in a few months to see where we stand16:41
fungione preliminary bit of info though, does rocky have public rsync access to any mirrors?16:42
fungiall our other rpm-based distros are mirrored via rsync, so consistency there would be nice16:42
mnasiadkaA lot of mirrors on their website mention rsync (https://mirrors.rockylinux.org/mirrormanager/mirrors) but I don't know if it's public16:43
mnasiadkaNeilHanlon should know a lot more than me16:43
fungino worries, there's plenty of time to find out16:43
mnasiadkayup, thanks16:44
NeilHanlonyou can pull from msync.rockylinux.org via rsync; we'll be restructuring soon, though, so that'll line up with the opendev mirrors too, probably17:39
fungiNeilHanlon: good to know, thanks!17:40
*** dhill is now known as Guest103918:19
clarkbfungi: that is a good point. I think we can wait until ianw has a chance to look later today. If we think dib update can be done relatively quickly that is good enough18:43
chris7dohmenHello folks! I went to use `pbr==5.10.0` which had the symbol fix for squashed messages. I saw that it was not working and trying to track down why, and it looks like the commit did not make it into the build, although it was in master. The commit in question is https://opendev.org/openstack/pbr/commit/45d4c9f02fafb425e61a18bbd7a20ef456293a56 which was merged Aug 5th with the build on Aug 11th. Did I do something wrong or was it just not picked up b19:10
fungichris7dohmen: sorry, we're having our weekly meeting at the moment, but i can take a look when it wraps up in about 50 minutes19:11
clarkblooks like the commit that was tagged is before the commit you want19:13
clarkbsituation normal as far as git and the build system are concerned19:14
slittle1hi all, has opendev ever hosted a copy of the Linux kernel git?19:15
clarkbslittle1: no19:15
slittle1Would there be any concerns?19:16
clarkbyes, we'd need to make sure gerrit and our replica giteas (and possibly zuul depending on how it is used) will handle it19:16
fungii expect it would be very large. cloning it repeatedly in ci jobs would also likely be taxing on our servers19:16
clarkbthe gerrit mailing list is full of users struggling to manage that repo in gerrit19:16
clarkbit is doable but require fiddling19:16
clarkbhttps://groups.google.com/g/repo-discuss/c/TJ4Pdg3KHUI for example19:18
slittle1One of my StarlingX colleagues is looking for a place to host the kernel.  Not sure if Vefa has joined the channel yet or not.  Might still be getting his IRC set up19:19
clarkbslittle1: is there some reason that the existing kernel hosting doesn't work?19:19
clarkbI think OSUOSL hosts it and they let people have forks on it. But I'm not sure what the criteria for that are19:19
slittle1There are a few drivers.  One was a desire for a uniform code inspection framework, i.e. gerrit19:20
clarkbif we ignore that the linux kernel is huge and presents unique problems with git hosting, I think it is a good idea to avoid using opendev to host forks of projects that already have homes19:20
chris4dohmenfungi: thanks, no rush! I was updating our tooling and was rather confused by the version I was getting out after this change. After reviewing the files I had downloaded from pypi I noticed and confirmed the issue (or my mistake when I made the review)19:21
clarkbchris4dohmen: https://opendev.org/openstack/releases/src/branch/master/deliverables/_independent/pbr.yaml#L231 is what was tagged. Which is the sixth commit listed at https://opendev.org/openstack/pbr/commits/branch/master (and I think you want the first?)19:23
slittle1The second was an ability to create branches and tags on the Linux git.19:24
slittle1If OSUOSL allows branches/tags, that might be an option19:24
fungilinus mirrors his tree into github, so forking there could also be an option i suppose19:26
clarkbya for that I'm not sure. I know some people have their kernels hosted there, but that may just be for maintainers? For us this wuold be non trivial and isn't something I think we can commit to now. It would require significant investigation. (like stand up a whole new gerrit and gitea do benchmarking)19:27
clarkbhttps://discourse.gitea.io/t/issues-cloning-a-big-repository/63 gitea may have resolved those problems as this is a really old post, but it definitely struggled at one time too19:33
chris4dohmenclark: Ah ok I see what happened then. commit `1eadf23b94846a0ebc8402e09190ed1de6dbffd0` is the head of `pbr==5.10.0`, not what was in the latest master at the time. That makes sense why it was not included then as I was just going off dates. thanks for the clarification! what is the typical release schedule for new releases?19:33
clarkbchris4dohmen: OpenStack is working on putting its "Zed" release out the door right now. I suspect that you could propose a PBR release to openstack/releases and have it land as soon as that is done in a few weeks. https://releases.openstack.org/zed/schedule.html They likely won't want to release PBR until afterwards unless it is a release breaking fix19:34
chris4dohmenclark: nope, just a bugfix. I will talk with our devops team and try to convince them to use a dev build for now. When you say propose a change, you mean submit a review for an update to that file, or is there something else involved? thanks for all of your help! 19:37
clarkbchris4dohmen: yes push an update to https://opendev.org/openstack/releases/src/branch/master/deliverables/_independent/pbr.yaml#L231 with the new version and its associated commit hash. That gets reviewed and when it lands the build system makes the release19:38
chris4dohmenclarkb: awesome, thanks for clarifying that and answering all of my questions!19:39
ianwi can make a dib release with the f36 fix.  not sure how around to be i'll watch it by the time builds make it to production19:41
fungii don't think it's too urgent, mainly f36 nodes will keep failing to boot in non-dhcp providers and lead to delays filling node requests and a bit of wasted resources in those providers, and could start leading to new node_failure results if ovh is unusable for us19:44
*** rlandy|rover is now known as rlandy|rover|biab20:02
opendevreviewMerged openstack/diskimage-builder master: Install Fedora ifcfg NM compat package  https://review.opendev.org/c/openstack/diskimage-builder/+/85854720:26
*** rlandy|rover|biab is now known as rlandy|rover20:30
ianwfungi: if you have a sec to double-check things in https://review.opendev.org/c/openstack/diskimage-builder/+/855589 ; that is a linters fix as i think we dropped dib-lint20:31
ianwmnasiadka / fungi: i'm not fully -1 on https://review.opendev.org/c/openstack/project-config/+/858554 but i also don't think we know if it works.  i've put a suggestion in there about func test builds in dib.  it would need monitoring either way20:41
zigoHi there!20:51
zigoThere's no release notes for keystone for Zed.20:51
Clark[m]zigo: I think they need to push a change to add a file like this one for zed https://opendev.org/openstack/keystone/raw/branch/master/releasenotes/source/yoga.rst21:03
fungiClark[m]: zigo: https://review.opendev.org/c/openstack/keystone/+/85756521:05
fungii think that's it21:05
opendevreviewMerged openstack/diskimage-builder master: Allow flake8 version 5  https://review.opendev.org/c/openstack/diskimage-builder/+/85467021:28
opendevreviewIan Wienand proposed openstack/diskimage-builder master: Start running dib-lint again  https://review.opendev.org/c/openstack/diskimage-builder/+/85558921:57
ianwfungi: ^ that needed some fixes after a rebase and a flake8 update change merging before it21:58
ianwfungi: another for when you have time is https://review.opendev.org/q/topic:configure-mirrors-centos to add the flag to enable that repo to our configure-mirrors job.  there's a couple underneath it to keep the test job in sync22:27
*** dhill is now known as Guest104822:35
ianw"usermod: user zuul is currently used by process 749"22:39
ianwthis is i think a new one on system-config tests ... maybe it's something i've done22:39
ianwit must be specific to https://review.opendev.org/c/opendev/system-config/+/858476/; we've had good runs of system-config-run-base this  morning22:40
ianwahh it is my fault; from the shuffling i'm doing.  i've included zuul as an extra user in the bridge bringup22:42
opendevreviewMerged opendev/base-jobs master: Revert "Switch base-test to test-prepare-workspace-git"  https://review.opendev.org/c/opendev/base-jobs/+/85847323:22
opendevreviewMerged opendev/base-jobs master: base-test: add descriptive names  https://review.opendev.org/c/opendev/base-jobs/+/85847423:27
opendevreviewMerged opendev/base-jobs master: configure-mirrors: enable extras-common for 9-stream  https://review.opendev.org/c/opendev/base-jobs/+/85825523:27
*** rlandy|rover is now known as rlandy|out23:40

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!