Tuesday, 2021-12-07

opendevreviewIan Wienand proposed opendev/dstat_graph master: Add basic Zuul job  https://review.opendev.org/c/opendev/dstat_graph/+/82064200:00
clarkbits up. I can ssh in and the webserver is serving mirrory things00:02
clarkbhopefully the daily letsencrypt run will be happy and renew any certs that need that00:02
clarkbour first gentoo build in a long time is running now. It hasn't failed yet (I looked for logs for the previous ones and they are all months old)00:05
opendevreviewIan Wienand proposed opendev/dstat_graph master: Add .gitreview and basic Zuul job  https://review.opendev.org/c/opendev/dstat_graph/+/82060900:12
opendevreviewIan Wienand proposed opendev/dstat_graph master: Update dependencies  https://review.opendev.org/c/opendev/dstat_graph/+/82063000:12
opendevreviewIan Wienand proposed opendev/dstat_graph master: Trim styles  https://review.opendev.org/c/opendev/dstat_graph/+/82063100:12
opendevreviewIan Wienand proposed opendev/dstat_graph master: Conver to bootstrap 5  https://review.opendev.org/c/opendev/dstat_graph/+/82063200:12
opendevreviewIan Wienand proposed opendev/dstat_graph master: Trim margin of overview panel  https://review.opendev.org/c/opendev/dstat_graph/+/82063300:12
opendevreviewIan Wienand proposed opendev/dstat_graph master: Fix walking elements of input list  https://review.opendev.org/c/opendev/dstat_graph/+/82063400:12
opendevreviewIan Wienand proposed opendev/dstat_graph master: Use list-group-item for graphs  https://review.opendev.org/c/opendev/dstat_graph/+/82063500:12
opendevreviewIan Wienand proposed opendev/dstat_graph master: Filter out blank lines  https://review.opendev.org/c/opendev/dstat_graph/+/82063600:12
opendevreviewIan Wienand proposed opendev/dstat_graph master: Replace remove .size() with .length  https://review.opendev.org/c/opendev/dstat_graph/+/82063700:12
opendevreviewIan Wienand proposed opendev/dstat_graph master: Fix focus panel chart selection for pcp-dstat  https://review.opendev.org/c/opendev/dstat_graph/+/82063800:12
opendevreviewIan Wienand proposed opendev/dstat_graph master: Update generate_page.sh  https://review.opendev.org/c/opendev/dstat_graph/+/82063900:12
opendevreviewIan Wienand proposed opendev/dstat_graph master: Add a sample.csv and update docs  https://review.opendev.org/c/opendev/dstat_graph/+/82064000:12
opendevreviewIan Wienand proposed opendev/dstat_graph master: Fix refresh for embedded csv  https://review.opendev.org/c/opendev/dstat_graph/+/82064100:12
tristanCclarkb: nice, that's good to hear :-)00:16
ianwclarkb: as the ptl can you quickly add infra core to https://review.opendev.org/admin/repos/opendev/dstat_graph or should i use my admin account?00:26
fungii expect he has to use his admin account to do that too. either way, it can be done via the ssh api, i think i added an example to the system-config docs00:27
ianwyeah i just wasn't sure what was quicker.  i'll pull up my admin details00:28
Clark[m]Yup it's the same process for both of us. I think you can go for it00:38
ianwdone, thanks :)00:38
opendevreviewMerged opendev/system-config master: Add comments to manage-projects about project-config syncing  https://review.opendev.org/c/opendev/system-config/+/81504900:39
opendevreviewMerged opendev/dstat_graph master: Add .gitreview and basic Zuul job  https://review.opendev.org/c/opendev/dstat_graph/+/82060900:42
opendevreviewMerged opendev/dstat_graph master: Update dependencies  https://review.opendev.org/c/opendev/dstat_graph/+/82063000:54
opendevreviewMerged opendev/dstat_graph master: Trim styles  https://review.opendev.org/c/opendev/dstat_graph/+/82063100:54
opendevreviewMerged opendev/dstat_graph master: Conver to bootstrap 5  https://review.opendev.org/c/opendev/dstat_graph/+/82063200:54
opendevreviewMerged opendev/dstat_graph master: Trim margin of overview panel  https://review.opendev.org/c/opendev/dstat_graph/+/82063300:54
opendevreviewMerged opendev/dstat_graph master: Fix walking elements of input list  https://review.opendev.org/c/opendev/dstat_graph/+/82063400:54
opendevreviewMerged opendev/dstat_graph master: Use list-group-item for graphs  https://review.opendev.org/c/opendev/dstat_graph/+/82063500:54
opendevreviewMerged opendev/dstat_graph master: Filter out blank lines  https://review.opendev.org/c/opendev/dstat_graph/+/82063600:54
opendevreviewMerged opendev/dstat_graph master: Replace remove .size() with .length  https://review.opendev.org/c/opendev/dstat_graph/+/82063701:57
opendevreviewMerged opendev/dstat_graph master: Fix focus panel chart selection for pcp-dstat  https://review.opendev.org/c/opendev/dstat_graph/+/82063801:57
opendevreviewMerged opendev/dstat_graph master: Update generate_page.sh  https://review.opendev.org/c/opendev/dstat_graph/+/82063902:01
opendevreviewMerged opendev/dstat_graph master: Add a sample.csv and update docs  https://review.opendev.org/c/opendev/dstat_graph/+/82064002:01
opendevreviewMerged opendev/dstat_graph master: Fix refresh for embedded csv  https://review.opendev.org/c/opendev/dstat_graph/+/82064102:01
opendevreviewIan Wienand proposed zuul/zuul-jobs master: ensure-dstat-graph: clone from opendev  https://review.opendev.org/c/zuul/zuul-jobs/+/82064502:06
opendevreviewIan Wienand proposed openstack/project-config master: cache-devstack: use dstat_graph from opendev  https://review.opendev.org/c/openstack/project-config/+/82064602:08
ianwthe gentoo build finished and is uploaded03:03
fungii'll recheck my changes in that case03:04
fungirechecked 782339 and 77110603:05
fungithey'll need a rebase, apparently03:07
ianwi think since https://review.opendev.org/c/zuul/zuul-jobs/+/771105 we also won't try it on gentoo03:08
wxy-xiyuanfungi: https://review.opendev.org/c/openstack/project-config/+/818723 can you take a look at this patch once you're free? Thanks03:15
opendevreviewJeremy Stanley proposed zuul/zuul-jobs master: WIP: Set Gentoo profile in configure-mirrors  https://review.opendev.org/c/zuul/zuul-jobs/+/78233903:17
opendevreviewJeremy Stanley proposed zuul/zuul-jobs master: Revert "Temporarily stop running Gentoo base role tests"  https://review.opendev.org/c/zuul/zuul-jobs/+/77110603:17
ianwthe gentoo images built, but do not seem happy04:07
ianw!!! Your current profile is invalid. If you have just changed your profile04:07
opendevmeetianw: Error: "!!" is not a valid command.04:07
ianw!!! configuration, you should revert back to the previous configuration.04:07
opendevmeetianw: Error: "!!" is not a valid command.04:07
*** pojadhav|out is now known as pojadhav|rover04:14
opendevreviewMerged openstack/diskimage-builder master: Update keylime-agent and tpm-emulator elements  https://review.opendev.org/c/openstack/diskimage-builder/+/81025404:29
opendevreviewIan Wienand proposed opendev/system-config master: infra-prod: setup system-config on bridge in bootstrap job  https://review.opendev.org/c/opendev/system-config/+/82032005:24
opendevreviewIan Wienand proposed opendev/system-config master: infra-prod-base: parent to infra-prod-setup-keys  https://review.opendev.org/c/opendev/system-config/+/82065105:24
opendevreviewIan Wienand proposed opendev/base-jobs master: opendev-infra-prod-base: remove unused job  https://review.opendev.org/c/opendev/base-jobs/+/82065205:24
opendevreviewIan Wienand proposed opendev/system-config master: Update bridge playbook match  https://review.opendev.org/c/opendev/system-config/+/82028105:28
opendevreviewIan Wienand proposed opendev/system-config master: Rename install-ansible to bootstrap-bridge  https://review.opendev.org/c/opendev/system-config/+/82028205:28
opendevreviewIan Wienand proposed opendev/system-config master: infra-prod: setup system-config on bridge in bootstrap job  https://review.opendev.org/c/opendev/system-config/+/82032005:28
opendevreviewIan Wienand proposed opendev/system-config master: infra-prod-base: parent to infra-prod-setup-keys  https://review.opendev.org/c/opendev/system-config/+/82065105:28
*** ysandeep|out is now known as ysandeep05:28
opendevreviewIan Wienand proposed opendev/system-config master: infra-prod-base: parent to infra-prod-setup-keys  https://review.opendev.org/c/opendev/system-config/+/82065105:44
*** marios is now known as marios|ruck06:23
opendevreviewMichal Nasiadka proposed openstack/diskimage-builder master: Add dnf versionlock support  https://review.opendev.org/c/openstack/diskimage-builder/+/81194506:44
*** ysandeep is now known as ysandeep|lunch07:39
*** pojadhav|rover is now known as pojadhav|rover|afk09:14
*** ysandeep|lunch is now known as ysandeep09:19
*** sshnaidm|afk is now known as sshnaidm09:20
fricklersorry for the german link, but TIL that the current snow in Germany is brought to us by ... Gerrit ;) https://www.wetteronline.de/fotostrecken/2021-12-07-ws09:29
ianwgerrrr-iiittt the snow man, was a jolly happy soul09:31
ianw(i have no idea if frosty the snowman is a thing in Germany :)09:33
ianwsomehow he's a thing in Australia, where it would be more like frosty the puddle09:33
*** pojadhav|rover|afk is now known as pojadhav|rover09:51
*** redrobot8 is now known as redrobot10:20
*** ysandeep is now known as ysandeep|afk11:00
*** rlandy|out is now known as rlandy|ruck11:08
*** ysandeep|afk is now known as ysandeep11:46
*** bhagyashris_ is now known as bhagyashris12:31
*** pojadhav|rover is now known as pojadhav|rover|brb12:44
*** pojadhav|rover|brb is now known as pojadhav|rover13:14
fungiperhaps it's a compellingly exotic concept for au13:48
noonedeadpunkfeels like zuul either super overflowed or a bit stuck?14:03
noonedeadpunkand most likely because of osa branching :(14:03
noonedeadpunktons of post jobs waiting for hour or so14:05
noonedeadpunkand new ones are not queued for check14:05
ykarelfungi, clarkb hi can you help in getting latest resource usage by projects14:17
ykarelmay be for last 15 days, an output like https://paste.opendev.org/show/jD6kAP9tHk7PZr2nhv8h/14:18
ykarelseems https://opendev.org/zuul/zuul/src/branch/master/tools/node_usage.py is the script for collecting that14:18
ykarelmainly looking for neutron projects, would like to see how much it differs from past report14:19
ykarelas we are currently trying to improve it further so wanted to have checkpoint so we have something to compare14:19
ykarelor if i need to contact someone else for this please redirect14:19
*** odyssey4me is now known as Guest786614:59
opendevreviewJeremy Stanley proposed opendev/system-config master: Copy Exim logs in system-config-run jobs  https://review.opendev.org/c/opendev/system-config/+/82089915:16
opendevreviewJeremy Stanley proposed opendev/system-config master: Block outbound SMTP connections from test jobs  https://review.opendev.org/c/opendev/system-config/+/82090015:16
corvusi'm going to do a rolling restart of zuul15:21
corvuswell, the schedulers and web at least15:22
funginoonedeadpunk: yeah, i think all the releasing/branching for openstackansible is causing delays. the executors are spending a good chunk of time not accepting new builds while they work through the ones they've already started15:25
fungiit does seem to be burning those counts down though, builds are being performed and completing15:25
fungicorvus: thanks for the heads up15:27
corvustell me about this branching?15:34
*** ysandeep is now known as ysandeep|dinner15:34
fungicorvus: https://review.opendev.org/c/openstack/releases/+/820644 Create Xena branch for OpenStack-Ansible roles15:37
fungithere's a bunch of jobs in the openstack tenant's post pipeline from that because openstackansible has i don't know how many repos but probably well over a hundred15:38
corvusare they all created now?15:38
fungii haven't looked, but probably so... it's the jobs triggered by the branch creation events which make up the current pileup15:38
corvusright, but the reason the system seems slow is that it's continuously reconfiguring the openstack tenant because of a bunch of new branches15:39
corvusso i was wondering if they were all created already, or is that still in progress15:40
fungiyeah, looks like the tag-releases job for that change has completed15:40
fungihttps://zuul.opendev.org/t/openstack/build/d29249c37338448bb98b7a90b4734e1015:40
fungifinished almost an hour ago15:41
fungionly took a couple minutes15:41
fungioh, that was a later change i think15:41
clarkbykarel: this came up in #openstack-infra a couple of weeks ago. As mentioned then that script doesn't work with modern zuul anymore. In particular the log formats have changed and the logs have moved from a centralized scheduler (though we have decentralized that too) to all of the executors. This means it is possible to collect similar data using that tooling but much more15:42
clarkbdifficult. What I did do back then was hack up the script to get a sample from a single executor and I should be able to do that again. But I suggested that openstack/neutron/etc look at using the graphite data to build a grafana dashboard for this information instead15:42
fungicorvus: it was this one, finished something like 5.5 hours ago: https://zuul.opendev.org/t/openstack/build/ab256275292a4c0cbcf9f025d7cca21c15:42
fungilooks like it created 55 branches15:43
fungiwe seem to have a strangely high count of available (ready) nodes15:46
fungiaccording to grafana15:46
fungiyeah, nodepool list confirms 441 nodes in "ready" state15:47
corvusthat's due to 2 reasons: the constant reconfigurations and at this moment there is no scheduler running15:47
fungiahh, okay15:47
ykarelclarkb, Thanks, data from 1/2 executor should be fine to start with15:48
ykarelif possible can you run that custom script again, that will be helpful15:49
ykarelwrt graphite/grafana tha needs to be explored 15:50
fungiykarel: to repeat, "that script doesn't work with modern zuul anymore"15:51
clarkbfungi: I mean I did hack it up15:52
ykarelfungi, yeap i got that, but afaiu clarkb said it's possible with hack to get  datafrom an executor15:52
clarkbthe script that is committed doesn't work ebacuse it assumes the old scheduler log format15:53
clarkbthere is a new executor format and it changes hosts15:53
fungioh, i get what you meant by "custom script" now15:53
clarkband ya I can run it on a single executor15:53
corvusokay, there is an issue with the data format in the new zuul; i would normally be tempted to just fix it and roll forward, however, if we also delete all the state data, we will effectively collapse all of the branch creation/reconfig events into one.  so i'm inclined to perform a delete-state and full restart in order to address both things.15:55
clarkbcorvus: the branch creation stuff from osa?15:57
corvusyep15:57
fungiseems like we'd probably get back to building things faster again with the full restart than waiting for however many reconfigs are pending from all the branch creation events15:57
corvusyeah.  i'm not sure why they weren't collapsed already; i'll have to look into that.15:57
clarkbykarel: https://paste.opendev.org/show/811516/ that is from ze01.opendev.org15:58
ykarelclarkb, Thanks can you run with may be 15 days and on seperate executor too like ze02?15:59
ykareljust to have some more data15:59
clarkbykarel: no, the executors only log 7 days. and I'd prefer to not need to mantain this on a bunch of hosts. We have the data aggregated in graphite, it just needs to be queried there16:00
ykarelclarkb, Thanks16:00
ykarelclarkb++16:01
ykarelwill check graphite too, but may be will ping here probably next week as i am not much aware about it16:01
clarkbcorvus: you're also talking about landing a fix and then doing a full restart on the fix? Or rolling back with the delete state?16:02
clarkbykarel: let me see if I can find logs from the #openstack-infra discussion16:02
corvusclarkb: sorry, issue was a non-backwards-compat change16:02
corvusso just a delete+restart should address it16:02
clarkbgot it16:02
clarkbykarel: https://meetings.opendev.org/irclogs/%23openstack-infra/%23openstack-infra.2021-11-15.log.html16:04
clarkbthat includes pointers to the graphite stuff16:04
ykarelclarkb, Thanks noted, will have a look16:06
clarkbinfra-root I updated https://etherpad.opendev.org/p/xvu2oKUQVLiHIHsvGRUt slightly for gentoo. Does that email look good to send out now?16:09
*** ysandeep|dinner is now known as ysandeep16:10
fungiclarkb: thanks, lgtm16:11
corvus1 scheduler up, web and 2nd scheduler on their way16:22
*** artom__ is now known as artom16:22
corvusre-enqueing16:29
clarkbci image email sent16:39
corvusfungi: can you take a look at https://review.opendev.org/820276 and https://review.opendev.org/820277 ?16:48
fungisure!16:48
corvuswe should probably merge those and test that out soon :)16:48
fungiooh, thanks i missed these16:48
opendevreviewMerged openstack/project-config master: Add REST api auth rules  https://review.opendev.org/c/openstack/project-config/+/82027717:02
opendevreviewLajos Katona proposed openstack/project-config master: Fix Neutron periodic dashboard  https://review.opendev.org/c/openstack/project-config/+/82091217:03
*** marios|ruck is now known as marios|out17:06
opendevreviewMerged opendev/system-config master: Add local auth provider to zuul  https://review.opendev.org/c/opendev/system-config/+/82027617:54
*** ysandeep is now known as ysandeep|out18:23
corvusre-enqueue complete.  (it really took that long)18:29
fungiwow, i guess it was all the post pipeline refs in the openstack tenant?18:30
fungiand thanks!18:30
fungithe failure on https://zuul.opendev.org/t/openstack/build/d11e7de3c5bc4d179bf1f36d661a9f66 is baffling me. the rules seem to include what it's testing for according to https://zuul.opendev.org/t/openstack/build/d11e7de3c5bc4d179bf1f36d661a9f66/log/focal/rules.v4.txt#1918:47
clarkbfungi: maybe try an explicit print(rules) to see what is in there?18:50
clarkbfungi: also I think that ruleset needs to add openstack-OUTPUT to the OUTPUT chain?18:53
fungioh, yep!18:54
fungithanks18:54
opendevreviewJeremy Stanley proposed opendev/system-config master: Block outbound SMTP connections from test jobs  https://review.opendev.org/c/opendev/system-config/+/82090018:57
clarkbit may be getting optimized out due to that lack of attachment18:57
fungiyep18:59
fungiworth confirming before i debug further anyway. thanks for spotting it18:59
opendevreviewJeremy Stanley proposed opendev/system-config master: Block outbound SMTP connections from test jobs  https://review.opendev.org/c/opendev/system-config/+/82090019:57
opendevreviewJeremy Stanley proposed opendev/system-config master: Copy Exim logs in system-config-run jobs  https://review.opendev.org/c/opendev/system-config/+/82089919:57
fungii'm trying to reverse-engineer the testinfra iptables validation, but suspect i'm still missing something20:01
clarkbI can take a look after some lunch and a bike ride. But need to get out now before it gets rainy and dark20:04
clarkbthough it is wetter than I had hoped already :/20:04
fungiyeah, there's no rush. enjoy!20:13
opendevreviewMerged zuul/zuul-jobs master: Update Fedora 34 to 35  https://review.opendev.org/c/zuul/zuul-jobs/+/81870220:30
ianwinfra-root: low priority but https://review.opendev.org/c/zuul/zuul-jobs/+/820645 and https://review.opendev.org/c/openstack/project-config/+/820646 update the dstat graph generation to use our opendev fork instead of github20:32
opendevreviewMerged zuul/zuul-jobs master: ensure-dstat-graph: clone from opendev  https://review.opendev.org/c/zuul/zuul-jobs/+/82064520:54
opendevreviewMerged opendev/system-config master: Update bridge playbook match  https://review.opendev.org/c/opendev/system-config/+/82028120:57
fungii have a feeling i'm going to need to set an autohold on 820900 and check whatever command the testinfra iptables module is running, because the output it quotes is uselessly truncated21:04
fungibut i'll wait to see if anyone else can spot the issue before i go to that length21:05
ianwfungi: to start with probably adding some info to that assert would help, should be able to just do assert rule in rules, "Rule %s not seen in rules" % rule ... ish21:12
corvusi'm going to restart zuul-web21:17
* TheJulia drifts in21:22
TheJuliao/21:22
ianwsigh, gnome has stopped accepting clicks in my work vm, so i'm going to reboot :/21:25
TheJuliacorvus: any luck?21:25
TheJuliaokay, its back21:28
corvusyep, just takes a while to start now; we need to add a second one and LB21:29
opendevreviewJames E. Blair proposed opendev/system-config master: Add zuul-client config to schedulers  https://review.opendev.org/c/opendev/system-config/+/82095121:40
corvusinfra-root: ^ i manually did that on zuul02 for development purposes.  you can run something like `zuul-client --tenant zuul autohold-list` there now and see it in action21:41

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!