Friday, 2022-01-28

corvusi'd like to rolling-restart zuul again.  :)00:29
corvus(this is so cool -- we're like converging on CD here :)00:29
corvusi'm going to restart the scheduler on zuul01 now00:30
clarkbsounds good. I'm around for a bit longer too00:30
corvusproblem preventing 01 from starting; looking into it00:32
corvusyou will be amused00:33
clarkbIt can't find the api model version for the other components?00:35
corvusi could have sworn we landed that change00:35
clarkbat least that is an easy fix00:35
corvuseasy but a surprising amount of typing....00:38
opendevreviewJames E. Blair proposed opendev/system-config master: Move Zuul SQL connection to "database"
corvusinfra-root: ^ our zuul config is broken and needs ^ before we can (re-) start any components00:47
ianwfixing that seems useful...00:49
clarkbcorvus: for did we merge in private vars over the top of that somehow?00:50
clarkbJust noting it doesn't ahve a uri or user/passwd info00:50
clarkb(wonderinf if we need to clean anything else up)00:50
corvusclarkb: yes, zuul_connection_secrets -- it is an empty list in system-config so no visible change.00:51
corvusi also have not removed the entry from that in secret hostvars, so as not to break the current system, but we can drop it there after that merges00:51
opendevreviewJames E. Blair proposed opendev/system-config master: Remove gearman from Zuul
corvuslow-priority followup ^00:55
corvusinfra-root: any objection to me throwing 826790 straight into gate?00:56
ianwno, please have it merged in case I ever need to restart it! :)00:58
clarkbhaha ya no objection from me00:58
corvusi'll be back in ~20m01:00
clarkbthe system-config-run-review jobs for fungi's gitea testing change show as failed and clicking on the link says the build doesn't exist. I wonder if zuul01 tried to process stuff despite not having a proper db config? Its not a big deal for those changes but calling it out here in case that is something to look into closer01:13
clarkbI need to go help wit hdinner now though01:13
corvusclarkb: it never started01:23
corvusthe build uuid from 3.4 cb2813b2a2704273920fba7ac310f936 doesn't show up in any zuul component logs01:34
corvusclarkb: fungi it's a bit hard to tell from the logs, but i suspect something related to container images; like it may not have found the required artifacts or something.01:38
corvus2022-01-28 01:00:46,727 INFO zuul.QueueItem: [e: f8775b510faa4cefbc3c0d149cb3e566] Job system-config-run-review-3.4 requires artifact(s) gerrit-3.4-container-image provided by build 7a2820c9b3934619a761a7a5092e0f5a (triggered by change 825337 on project opendev/system-config), but that build failed with result "FAILURE"01:39
corvusclarkbfungi ^ yeah that's it.  the UI is misleading because that build doesn't exist, but it doesn't represent an operational problem.01:41
corvusi think that will get reported in the message to gerrit01:44
opendevreviewMerged opendev/system-config master: Move Zuul SQL connection to "database"
corvuswaiting on deployment of that now01:48
*** rlandy|ruck|bbl is now known as rlandy|ruck02:02
Clark[m]corvus: aha I guess maybe we need to report skipped or something along those lines to reduce confusion02:03
*** ysandeep|out is now known as ysandeep02:07
opendevreviewMerged opendev/system-config master: Rebuild Gerrit images particularly for 3.4
*** rlandy|ruck is now known as rlandy|out02:16
corvusinfra-root: the deploy base job failed02:40
corvusi'm not in a position to debug that now02:41
ianwi'll take al ook02:42
ianwfatal: []: FAILED!02:43
ianwE: dpkg was interrupted, you must manually run 'dpkg --configure -a' to correct the problem.02:43
ianwi'll do a manual run of it to confirm02:48
ianwbase has deployed now03:25
ianwsha256:867785204c26492af92bee4f769c36421a77ba9e17bf94c7fd0d823610fb91b9 is the gerrit image promoted by
ianw was pushed by 826761 and that's what i've just ensure is pulled onto gerrit03:33
ianwi'm going to take gerrit down, upgrade docker and restart it, with that image03:37
ianw... and back03:41
*** ysandeep is now known as ysandeep|away03:52
opendevreviewIan Wienand proposed openstack/diskimage-builder master: yum-minimal: don't strip -* from releasever
opendevreviewIan Wienand proposed openstack/diskimage-builder master: Switch 9-stream testing to use opendev mirrors
opendevreviewIan Wienand proposed openstack/diskimage-builder master: Add 9-stream ARM64 testing
opendevreviewEduardo Santos proposed openstack/diskimage-builder master: Fix openSUSE images and bump them to 15.3
*** anbanerj is now known as frenzyfriday05:42
*** marios is now known as marios|ruck06:15
fricklerfungi: hrw: did some further debugging on the py27 oauthlib issue. seems the culprit is our wheel mirror, pip fails to detect that it should not install 3.1.1, likely because it is still an universal wheel06:24
fricklerthe reason that the issue only pops up now is that wheels hadn't been released since end of november due to the broken arm jobs
frickler.tox/py27/bin/pip install -U oauthlib --extra-index-url
fricklerthat shows the failure, without our mirror everything is fine06:25
fricklersee also which isn't in 3.1.1 yet06:27
opendevreviewMerged zuul/zuul-jobs master: pin oauthlib version for python2.7
opendevreviewIan Wienand proposed openstack/diskimage-builder master: yum-minimal: Document why we strip -stream from $releasever
opendevreviewIan Wienand proposed openstack/diskimage-builder master: Switch 9-stream testing to use opendev mirrors
opendevreviewIan Wienand proposed openstack/diskimage-builder master: Add 9-stream ARM64 testing
*** amoralej|off is now known as amoralej07:48
opendevreviewIan Wienand proposed openstack/diskimage-builder master: centos: do not use $releasever in .repo files
opendevreviewIan Wienand proposed openstack/diskimage-builder master: Switch 9-stream testing to use opendev mirrors
opendevreviewIan Wienand proposed openstack/diskimage-builder master: Add 9-stream ARM64 testing
*** bhagyashris_ is now known as bhagyashris08:01
*** jpena|off is now known as jpena08:13
dpawlikfungi, clarkb: hey, is it ok to add to our softwarefactory infra? I mean I would like to monitor the host state with prometheus node-exporter + check if services are alive. If it is ok, I will do a account on that host: "sf" or "zuul-sf" and it will be configuring additional things on that hos. 08:18
dpawlikfungi, clarkb: ah, I forget to mention: if it is fine to monitor with node exporter, could you open the firewall on pot 9100 for host please ?08:19
fungidpawlik: we don't manage any external firewall there, just update iptables on the server08:48
dpawlikack fungi08:51
*** bhagyashris_ is now known as bhagyashris08:53
*** ysandeep|away is now known as ysandeep09:29
*** dviroel_ is now known as dviroel11:03
*** rlandy|out is now known as rlandy|ruck11:12
*** amoralej is now known as amoralej|lunch14:02
corvuszuul01 is up14:10
*** rcastillo|rover is now known as rcastillo14:13
fungithanks for the quick fix!14:19
*** amoralej|lunch is now known as amoralej14:30
*** ysandeep is now known as ysandeep|dinner14:34
opendevreviewNeil Hanlon proposed openstack/diskimage-builder master: Add new container element - Rocky Linux
corvusfurther rollout is stalled pending
*** dviroel is now known as dviroel|lunch15:31
*** ysandeep|dinner is now known as ysandeep15:44
*** ykarel_ is now known as ykarel15:54
*** dviroel|lunch is now known as dviroel16:23
clarkbI've approved that change now16:27
clarkbianw: thank you for getting that new gerrit image installed16:28
corvusianw: and thanks for the base playbook fix :)16:29
corvusclarkb: thanks; i'll be afk for a while today, but i'll be around this evening/tomorrow to roll that out16:29
clarkbsounds good. I think I may need to restart zuul executors, I can work with fungi for that since he has done a couple of those recently16:31
fungiwell, we can do a graceful restart of the schedulers if we prefer16:34
fungiall the ones i did recently were hard restarts for the sake of expediency/urgency16:34
*** jpena is now known as jpena|off17:06
*** marios|ruck is now known as marios|out17:08
fungiclarkb: if you're cool with 826734 i can go ahead and get that server deleted17:10
clarkbfungi: approved17:11
opendevreviewMerged opendev/system-config master: Drop wiki-dev03 from inventory
fungicool, i'll delete it now17:37
fungiand done17:38
*** ysandeep is now known as ysandeep|out17:42
*** amoralej is now known as amoralej|off19:07
*** dviroel is now known as dviroel|brb20:46
clarkbok the system-config-run-review-3.4 job is failing to build because it is executing before the image it depends on has been built21:53
clarkbthe review-3.3 job is waiting properly21:54
clarkbthe good news is that zuul seems to be checking out the depends on in the imgae build properly which means we should be able to do this depends on thing with upstream gerrit changes if we figure out why the system-config-run-review-3.4 job is unhappy21:54
clarkbwe list system-config-build-image-gerrit-3.4 as a soft dependency21:56
priteauHello. Was I the only one who received email an hour ago from Storyboard, but for things that happened yesterday?21:58
clarkbok its a transitive issue. one of the parent chagne failed to build the image so no we can't build the image in later changes. I'll recheck the bottom of the stack and work our way up I guess22:00
clarkbpriteau: I haven't received recent emails from storyboard. fungi may be subscribed to more stuff22:00
clarkbpriteau: I do wonder if that means your mail servers were rejecting storyboard emails for a bit22:07
clarkbsmtp is a protocol that will retry with backoffs22:07
clarkbfungi: fyi I only rechecked the first change in your gerrit gitea stack because I realized that stack is making prod changes and the bottom of the stack is approved. The first one should be fine it is just a docker image update to remove the gitweb stuff explicitly from the image. But worried about not being ble to watch the others go in22:08
corvuszuul01 is restarted; i'm going to restart zuul02 now, which will cause a web outage23:07
corvusit's up, now the mergers23:22
corvus#status log restarted all of zuul on 930ee8faa3076233614565fcfbf55a4ee74551a723:25
opendevstatuscorvus: finished logging23:25
corvusi'm going to restart nodepool now23:27
corvus#status log restarted all of nodepool on 1a73a7a33ed63ad919377fae42c14390d8fb9eb523:31
opendevstatuscorvus: finished logging23:31

Generated by 2.17.3 by Marius Gedminas - find it at!