Thursday, 2022-05-05

*** rlandy|bbl is now known as rlandy|out00:55
opendevreviewIan Wienand proposed zuul/zuul-jobs master: add-gpgkey: trust incoming key
opendevreviewIan Wienand proposed zuul/zuul-jobs master: add-gpgkey: trust incoming key
opendevreviewIan Wienand proposed zuul/zuul-jobs master: add-gpgkey: trust incoming key
*** ysandeep|out is now known as ysandeep05:01
*** pojadhav- is now known as pojadhav05:07
*** marios is now known as marios|ruck05:15
*** jpena|off is now known as jpena07:39
*** ykarel_ is now known as ykarel07:48
*** ysandeep is now known as ysandeep|lunch08:07
*** pojadhav is now known as pojadhav|lunch09:01
opendevreviewIan Wienand proposed openstack/project-config master: Add opendev/infra-openafs-deb
opendevreviewIan Wienand proposed openstack/project-config master: Add opendev/infra-openafs-deb
*** pojadhav|lunch is now known as pojadhav|afk09:27
ianwfungi: ^ i completely admit your were right :)  i'm thinking that if we keep xenial -> jammy branches in a separate project, what we've already committed can be reworked to a generic job to upload to the PPA09:29
*** ysandeep|lunch is now known as ysandeep09:36
*** rlandy|out is now known as rlandy10:25
*** pojadhav|afk is now known as pojadhav10:36
*** dviroel|out is now known as dviroel11:21
*** ysandeep is now known as ysandeep|afk11:46
fungii'm not right all that often, so i'll take it! ;)12:25
fungiand yeah, ultimately this is closer to how the debian packaging is being done anyway (per the README.source file)12:26
*** soniya29 is now known as soniya29|afk12:27
frickleranother nice thing about phased updates: it seems reprepro only mirrors the update initially when it contains "Phased-Update-Percentage: 10" and doesn't update when this value gets updated12:27
fricklerour mirror is still at 10% for systemd while upstream has reached 70%12:28
fricklerbut the devstack jammy job worked fine after yesterdays patches12:29
*** ysandeep|afk is now known as ysandeep12:29
fungiinteresting. i wonder if newer reprepro behaves differently12:32
fricklerseem no:
TheJuliaanyone around who can hold nodes from a CI job so I can investigate the lost mysteries of why we can't seem to ssh from one node to another ?13:21
fricklerTheJulia: sure, patch id and job name?13:41
fungimysteries of the (ci) universe13:41
TheJulia  840530,3  ironic-grenade-multinode-multitenant <-- as requested. I just kicked the job with recheck but it is waiting at the moment. Likely should start in a minute or three and fail in ~1.5 hours13:44
fricklerTheJulia: ok, hold is set. it is only evaluated at the moment the job is failing, so that should be fine. but you could let us know our ssh key beforehand already13:51
*** soniya is now known as soniya|afk14:15
*** soniya|afk is now known as soniya15:11
opendevreviewSandeep Yadav proposed openstack/diskimage-builder master: Revert "Set machine-id to uninitialized to trigger first boot"
clarkbinfra-root looks like the 0600 backup on lists.o.o has failed today and yesterday. But the other backup isn't complaining?15:28
clarkbhas anyone looked at that yet?15:28
fungii have not yet, no15:28
clarkbwe have an arm64 jammy image now so I've approved
clarkbthat change adds the nodes. Then once node config is in place we can land the nodeset addition15:31
fricklerTheJulia: root@ +
clarkbfrickler: re reprepro I guess even more reason to disable use of phased updates on our side15:32
opendevreviewClark Boylan proposed opendev/system-config master: Borg ignore ansible tmp files
clarkbI think ^ is the fix for the lists backup failures. It is a  race against ansible updating tmp files and trying to back them up. We should be able to safely ignore those files15:38
fungioh, fun!15:40
opendevreviewMerged openstack/project-config master: Add Jammy arm64 test nodes
clarkbthe log files on the host for the vexxhost backup target show the error if anyone wants to dobule check it15:42
fricklerclarkb: ack15:42
TheJuliafrickler: thanks!15:43
*** dviroel is now known as dviroel|lunch15:53
*** marios|ruck is now known as marios|out16:01
TheJuliafrickler: I've pushed a 4th revision of that patch. The prior nodes can be reclaimed, and if the new nodes can be held for the job that is launching now, that would be appreciated16:20
TheJuliatl;dr change accidently squashed system state16:20
*** ysandeep is now known as ysandeep|out16:28
clarkbok I think the old hold is still there with a max count of 1 so it won't catch the new ps. I'll delete it then create a new one16:37
clarkbTheJulia: frickler  ^ fyi since I think frickler's day is ending now16:37
opendevreviewPierre Riteau proposed opendev/irc-meetings master: Bring Blazar meeting one hour forward
opendevreviewPierre Riteau proposed opendev/irc-meetings master: Schedule Blazar meeting one hour earlier
TheJuliaclarkb: thanks16:54
*** dviroel|lunch is now known as dviroel17:01
clarkbcorvus: I'm looking at that nl02 error and I think I may understand it. There is a node in inmotion cloud that failed to delete. Nodepool must be treating it as a leaked node because it has a record of it in zk without any information like node.type (which generates the error). I think that means it generated a simple record for it to clean up the leak17:10
clarkbThe correct way to fix this is to delete the instance from the cloud. I have no idea what that will entail right now. On the nodepool side I wonder if we may need to add some messaging around those errors indicating that leaked nodes may not have sufficient info to calculate quota from and that generates the error?17:11
corvusclarkb: perhaps -- but at this point every other driver handles leaked nodes differently, so that would be an openstack-driver-specific issue.  just something to keep in mind.17:21
corvus(ie, i think openstack may be the only one that creates bogus nodes)17:22
clarkbhow do the others handle it? Just attempt to delete it directly and try again later via a listing against the cloud to make the leaked list?17:22
clarkbanyway I'm not too worried about it currently as the problem seems straightforward and nodepool should continue to handle this just fine (it may get more quota errors than usual though)17:23
*** jpena is now known as jpena|off17:24
corvusclarkb: yep exactly17:43
corvushopefully we can move the openstack driver to that system at some point17:43
clarkbI'm manually booting a jammy arm64 instance now and if it comes up and is ssh'able I'll approve the nodeset change17:50
clarkbit boots!17:51
*** artom__ is now known as artom17:52
opendevreviewMerged opendev/base-jobs master: Add Jammy Arm64 nodeset
*** dviroel is now known as dviroel|out20:43
ianwlinaro should be back now, i'll double check on that and do a revert of the disable21:31
clarkbianw: kevinz: did the cert get updated too? I think it expires soon21:31
ianwnotBefore=May  5 12:48:39 2022 GMT21:34
ianwnotAfter=Aug  3 12:48:38 2022 GMT21:34
ianwlooks like yes21:34
fungimust have been replaced today once he got back from holidays22:04
*** rlandy is now known as rlandy|bbl22:36
opendevreviewMerged openstack/project-config master: Add opendev/infra-openafs-deb
clarkbTheJulia: I realize that I've spaced and not given you access to those nodes you wanted held. One second22:47
clarkbTheJulia: root @ and
clarkbianw: is probably interesting to you (noticed that when debugging the lists.o.o backup failures and this should fix it)23:06
ianw++ thanks, good find23:07
opendevreviewMerged opendev/system-config master: Borg ignore ansible tmp files
opendevreviewIan Wienand proposed opendev/infra-openafs-deb master: [wip] make jammy packages
TheJuliaclarkb: thanks, you guys can reclaim those machines now23:54
TheJuliaconfirmed I wasn't (too) insane too23:55

Generated by 2.17.3 by Marius Gedminas - find it at!